© 2019 The Society of Instrument and Control Engineers - SICE. Many traditional cultural assets in Kumamoto City were damaged by the Kumamoto earthquakes in 2016. The Kumamoto Castle, which recently celebrated its 400th anniversary, was disastrously damaged by the earthquakes. It will be completely repaired because it is one of the most important cultural assets in Japan. In this study, we propose a database design method, called 'ISHIGAKI,' after the Japanese name for stone castle walls, for tracking the broken stones of Kumamoto Castle to effect the complete repair of the walls. It is required to determine an accurate position for each stone before the Kumamoto Earthquakes struck. However, it is difficult to match the stones because photographs of each stone are plagued with variable shooting conditions. Consequently, we constructed the ISHIGAKI database for matching the correct position of the stones. We focus on the location of the fallen stones to improve the ability to find the correct original location. Moreover, we propose the ISHIGAKI matching algorithm based on not only image matching but also on the location information of the stones.