Recently, Goto et al. have discovered an anomalous elastic softening due to the vacancy in the crystalline silicon at low temperature by using the ultrasonic measurements. To investigate the quantum state of the silicon vacancy, we introduce a cluster model which includes both the Coulomb interaction between electrons in the dangling bond and the coupling between the electrons and Jahn-Teller phonons. We solve the model by using the numerical diagonalization method and find that the V-0 state with neutral charge shows a transition from S=0 to S=1 in the strong coupling regime. Remarkably, the V+ state with S = 1/2 becomes stable due to the trigonal phonon while unstable due to the tetragonal phonon. The obtained result with the trigonal phonon is consistent with the elastic softening in the B-doped silicon.