For $N=Z$ odd-odd nuclei, a three-body model assuming two valence particles
and an inert core can provide an understanding of pairing correlations in the
ground state and spin-isospin excitations. However, since residual core-nucleon
interactions can have a significant impact on these quantities, the inclusion
of core excitations in the model is essential for useful calculation to be
performed. The effect of core excitations must be included in order to gain a
detailed understanding of both the ground state and spin-isospin properties of
these systems. To this end, we include the vibrational excitation of the core
nucleus in our model. We solve the three-body core-nucleon-nucleon problem
including core vibrational states to obtain the nuclear ground state as well as
spin-isospin excitations. The spin-isospin excitations are examined from the
point of view of SU(4) multiplets. By including the effect of core excitation,
several experimental quantities of $N=Z$ odd-odd nuclei are better described,
and the root mean square distances between proton and neutron and that between
the center of mass of proton and neutron and core nucleus increase. Large
$B$($M1$) and $B$(GT) observed for $^{18}$F and $^{40}$Ca were explained in
terms of the SU(4) symmetry. The core nucleus is meaningfully broken by the
residual core-nucleon interactions, and various quantities concerning
spin-isospin excitations as well as the ground state become consistent with
experimental data. Including the core excitation in the three-body model is
thus important for a more detailed understanding of nuclear structure.