When efficient power amplification is considered in digital wireless communications, even the nonlinear region of an amplifier is used. In such cases, the symbol error rate deteriorates due to AM-AM and AM-PM distortion. This paper studies the spectral expansion, intersymbol interference, and symbol error rate when a nonlinear amplifier is used for transmission in a multivalued amplitude and phase modulation scheme. Multiple-symbol differential detection using the RLS (Recursive Least Squares) algorithm is compared with coherent detection in order to eliminate the effect of nonlinear amplification and fading. The transmission characteristics when these detection methods are applied to transmission of the star QAM scheme in a white Gaussian noise channel and a Rayleigh fading channel are studied. It is confirmed that multiple-symbol differential detection provides better symbol error rates than coherent detection. (C) 2003 Wiley Periodicals, Inc.