We proposed a supporting system for location-based AR that enables user to handle virtual object without AR markers and hand mounting instruments. We also carried out design and implementation of the proposed system. It makes possible to calculate global coordinates of hands and recognize hand gestures through a combination of depth sensor, motion capture system, and head mounted display. To evaluate the performance of the proposed system, we carried out the taskes of moving, generating, and eliminating of virtual object. The recognition rates for all tasks are above 90%. Furthermore, because it is possible to handle virtual object with hand gestures by adopting depth sensor, the supporting system improved the disadvantages of AR marker method for location-based AR.