The optical ZCZ code, which is a set of pairs of binary and biphase sequences with zero correlation zone, can provide code division multiple access (CDMA) communication system without co-channel interference. The M-ary amplitude shift keying spread spectrum (M-ary/ASK-SS) system using this code can detect a desired sequence without interference of undesired sequences. We have proposed the compact construction of a matched filter bank for this code, which is called the compact-type matched filter bank. However, M-ary/ASK-SS system using this filter bank might go down the bit error rate (BER) performance because input-output characteristics of this filter bank for this code with zero correlation zone 4n-2 are nonlinear characteristics. In this paper, we clarify that BER performance in the M-ary/ASK-SS system over the additive white Gaussian noise (AWGN) channels is not influenced by nonlinear characteristics of this filter bank by the theoretical formula of BER characteristics and the computer simulation. The BER performance in the ASK-SS system using compact-type matched filter under the AWGN environment go down compared to that using direct-type matched filter as expected. But the BER performance in the M-ary/ASK-SS system using this compact-type matched filter bank under the AWGN environment is almost equal compared to that using parallel-type matched filter bank. In fact, this compact-type matched filter bank is effective in the M-ary/ASK-SS system. © 2011 IEEE.