The submucosal invasion depth predicts prognosis in early colorectal cancer. Although colorectal cancer with shallow submucosal invasion can be treated via endoscopic resection, colorectal cancer with deep submucosal invasion requires surgical colectomy. However, accurately diagnosing the depth of submucosal invasion via endoscopy is difficult. We developed a tool to diagnose the depth of submucosal invasion in early colorectal cancer using artificial intelligence. We reviewed data from 196 patients who had undergone a preoperative colonoscopy at the Osaka University Hospital and Osaka International Cancer Institute between 2011 and 2018 and were diagnosed pathologically as having shallow submucosal invasion or deep submucosal invasion colorectal cancer. A convolutional neural network for predicting invasion depth was constructed using 706 images from 91 patients between 2011 and 2015 as the training dataset. The diagnostic accuracy of the constructed convolutional neural network was evaluated using 394 images from 49 patients between 2016 and 2017 as the validation dataset. We also prospectively tested the tool from 56 patients in 2018 with suspected early-stage colorectal cancer. The sensitivity, specificity, accuracy, and area under the curve of the convolutional neural network for diagnosing deep submucosal invasion colorectal cancer were 87.2% (258/296), 35.7% (35/98), 74.4% (293/394), and 0.758, respectively. The positive predictive value was 84.4% (356/422) and the sensitivity was 75.7% (356/470) in the test set. The diagnostic accuracy of the constructed convolutional neural network seemed to be as high as that of a skilled endoscopist. Thus, endoscopic image recognition by deep learning may be able to predict the submucosal invasion depth in early-stage colorectal cancer in clinical practice.