In mode-division multiplexing systems, the transmission quality is restricted by the differential mode delay (DMD). We previously proposed a spatial mode exchange technique using volume holograms (VHET) to reduce the DMD. VHET can be used for the exchange of multiple spatial modes with different transmission speeds using a single device. However, in VHET, incidents of cross-talk (XT) caused by non-target holograms severely degrade the exchange performance. To address this issue, we propose a spatial mode exchange technique that uses volume holograms in combination with a random optical diffuser. By scattering the intensity distribution of the signal beam uniformly using a random optical diffuser, high exchange performance can be attained due to the reduction in the influence of non-targeted holograms. We confirmed the basic operation of the proposed scheme by considering the crosstalk component coupled to the few-mode fiber in a mode-division multiplexing system. The simulation results show that the maximum XTs are significantly suppressed to - 15.8 dB.