© 2020 IEEE. Judgment of the appropriate time for harvesting high-quality fruits is performed by visual inspection, which is a qualitative evaluation, and the criteria are not quantitatively determined. The purpose of this study is to develop a system for quantitatively determining the optimal harvest time for tree fruits. The judgment value is calculated by comparing the fruit photographed using the color chart with the chart in the image. The color spaces used in this study are the L ∗ a ∗ b ∗ and HSV color spaces, which are robust to lighting changes. By comparing the measurement with the internal quality and the proposed method, the consistency with the measurement was confirmed relative to the internal quality. With respect to the proposed method, the ratio of the difference below the standard between the eye measurement and the proposed method was 75.7% for Shinano Sweet and 81.8% for Yume Akari, which ware roughly equivalent to the expert's judgment results. Therefore, the proposed method can be useful for determining the appropriate harvest time for tree fruits. In the future, we plan to investigate an automatic setting method for the fruit color analysis area and a method to determine the appropriate harvest time without requiring a color chart.