To investigate the origin of a phase transition at T-Q = 0.06 K simultaneously occurring with a superconducting transition in a cage compound PrRh2Zn20, we carried out ultrasonic measurements on a single-crystalline sample. The transverse modulus (C-11 - C-12)/2 is intimately coupled to the non-Kramers ground doublet Gamma(3), and elastic softening is observed at low temperatures. Below T-Q, the softening stops, suggesting the disappearance of quadrupole degrees of freedom. We clarified the negative quadrupole-quadrupole coupling constant and reentrant behavior of T-Q(H) in a magnetic field H. These results reveal that the phase transition at T-Q is antiferroquadrupolar ordering. The anisotropic magnetic field-temperature phase diagram is determined for H parallel to [100], [110], and [111]. A magnetic-field-induced phase transition is newly found at high fields in all three field directions. We also observed ultrasonic dispersion at around 50 K owing to the rattling motion of Zn atoms at the 16c site, and pointed out the strong electron-phonon coupling in PrRh2Zn20.