We experimentally implement a system of cavity optomagnonics, where a sphere of ferromagnetic material supports whispering gallery modes (WGMs) for photons and the magnetostatic mode for magnons. We observe pronounced nonreciprocity and asymmetry in the sideband signals generated by the magnon-induced Brillouin scattering of light. The spin-orbit coupled nature of the WGM photons, their geometrical birefringence, and the time-reversal symmetry breaking in the magnon dynamics impose the angular-momentum selection rules in the scattering process and account for the observed phenomena. The unique features of the system may find interesting applications at the crossroad between quantum optics and spintronics.