We have performed ultrasonic measurements at low temperatures in order to investigate vacancy in single crystal silicon. The longitudinal elastic constants of non-doped and boron-doped silicon grown by a floating zone method exhibit appreciable softening with decreasing temperature down to 20 mK. The softening of boron-doped silicon is easily suppressed in applied magnetic field up to 2 T, while the softening of non-doped silicon is robust in fields even up to 16 T. The softening of elastic constants in high-purity crystalline silicon is certainly caused by the coupling of elastic strains of the ultrasonic waves to electric quadrupoles of the vacancy orbital.