It is important to precisely set and keep the phase and amplitude of an rf signal in the accelerating cavity of modern accelerators, such as an X-ray Free Electron Laser (XFEL) linac. In these accelerators an acceleration rf signal is generated or detected by an In-phase and Quadrature (IQ) modulator, or a demodulator. If there are any deviations of the phase and the amplitude from the ideal values, crosstalk between the phase and the amplitude of the output signal of the IQ modulator or the demodulator arises. This causes instability of the feedback controls that simultaneously stabilize both the rf phase and the amplitude. To compensate for such deviations, we developed a novel compensation method using a two-dimensional Discrete Fourier Transform (DFT). Because the observed deviations of the phase and amplitude of an IQ modulator involve sinusoidal and polynomial behaviors on the phase angle and the amplitude of the rf vector, respectively, the DFT calculation with these basis functions makes a good approximation with a small number of compensation coefficients. Also, we can suppress high-frequency noise components arising when we measure the deviation data. These characteristics have advantages compared to a Look Up Table (LUT) compensation method. The LUT method usually demands many compensation elements, such as about 300, that are not easy to treat. We applied the DFT compensation method to the output rf signal of a C-band IQ modulator at SACLA, which is an XFEL facility in Japan. The amplitude deviation of the IQ modulator after the DFT compensation was reduced from 15.0% at the peak to less than 0.2% at the peak for an amplitude control range of from 0.1 V to 0.9 V (1.0 V full scale) and for a phase control range from 0 degree to 360 degrees. The number of compensation coefficients is 60, which is smaller than that of the LUT method, and is easy to treat and maintain. (C) 2016 Elsevier B.V. All rights reserved.