© COPYRIGHT SPIE. This paper tries improving image recognition accuracy with Convolutional Neural Networks (CNNs). CNNs are one of state-of-the-art image recognition frameworks, and have used the Rectified Linear Unit (ReLU) as the activation function. However, the ReLU rectifies negative values to zero. This paper applies the Sign-to-Position (S/P) format conversion after convolutional procedures to eliminate the rectification loss. Experimental results show that the proposed method improves the recognition accuracy of the MNIST and Fashion-MNIST data set by 0.50% and 1.30% compared with a conventional CNN respectively. The S/P format conversion also contributes to negative image recognition, and results in 12.58% and 3.66% higher accuracy.