We report on the fabrication of reshaped Ag nanoparticles (NPs) embedded in a Nd:YAG crystal by combining Ag ion implantation and swift heavy Xe ion irradiation. The localized surface plasmon resonance (LSPR) effect is proved to be efficiently modulated according to the phenomenon of polarization-dependent absorption. The LSPR peak located at 448 nm shows red shift and blue shift at 0 degrees and 90 degrees polarization, respectively, which is in good agreement with calculation by discrete dipole approximation. Based on the near-field intensity distribution, the interaction between reshaped NPs shows a non-ignorable effect on the optical absorption. Furthermore, the polarization-dependence of the photoluminescence (PL) intensity is analyzed, which is positively related to the modulated LSPR absorption. It demonstrates the potential of the enhancement of PL intensity by embedded plasmonic Ag NPs. This work breaks the conventional view of the quenching effect of NPs by ion irradiation and opens a new way to realize the modulation of optical dichroism.