In this paper, the method of controlling the position of the quad-rotor was shown aiming structure inspection. The data obtained from an ultra-wide band (UWB) module capable of measuring a distance, an inertial measurement unit (IMU), and a distance sensor were fused and passed through a Kalman filter to estimate three-dimensional coordinates. All calculation of the position coordinates was performed in the on-board computer of the quad-rotor, and the computer from the outside gave only the start command of position control. In order to verify the position estimation accuracy and the position control performance, we carried out the position hold hovering experiment at one point and experiment to move multiple points. The possibility of the position estimation and the position control using the UWB module and the IMU was shown.