A ferromagnetic sphere can support optical vortices in the form of whispering gallery modes and magnetic quasivortices in the form of magnetostatic modes with nontrivial spin textures. These vortices can be characterized by their orbital angular momenta. We experimentally investigate Brillouin scattering of photons in the whispering gallery modes by magnons in the magnetostatic modes, zeroing in on the exchange of the orbital angular momenta between the optical vortices and magnetic quasivortices. We find that the conservation of the orbital angular momentum results in different nonreciprocal behavior in the Brillouin light scattering. New avenues for chiral optics and optospintronics can be opened up by taking the orbital angular momenta as a new degree of freedom for cavity optomagnonics.