Machine vision is a non-destructive grading technology and cost-effective method with high accuracy that can be used to predict length, width, and mass, as well as defects of both interior and exterior of a sample by employing different cameras, such as color, multispectral, or hyperspectral cameras. To obtain certain data, which relates to sample quality in the 3D space (thickness, volume, and surface gradient distribution) and mass prediction, a novel method was developed and the obtained appearance quality was graded utilizing a new image processing algorithm for depth images. In this study, we recorded the depth images of 110 potatoes using a depth camera, including samples with uniform shapes or with deformations (e.g., bumps, bent shape, and divots). Length, width, thickness, and volume were calculated respectively, and used as key factors for detecting potato deformity, such as bent shape, bumps, and hollow. Experimental results indicate that mass prediction based on a volume model for both normal and deformed potato samples showed high accuracy, thus 90% of the samples were graded for the correct size group using the volume model. In addition, the appearance quality grading reached 88% of a correct percentage for bent shape, bump, and hollow defect detection by combining the surface data in 2D and 3D space. In addition, a potato virtual reality model rebuilding algorithm was developed for sample quality tracing and rechecking based on 3D shape and color images. This model redisplays the potato color and 3D shape data in multi-views and supports 360-degree rotation in both horizontal and vertical directions to simulate the in-hand examination experience. This depth image processing is an effective potential method for future non-destructive post-harvesting grading, especially for products where size, shape, and surface condition are important factors.