Referential communication is common in physical remote collaboration. To successfully transfer instructions, remote instructors have to refer objects in the local worker’s environment. However, it is known that nonverbal behaviors are hard to be transferred correctly through modern remote collaboration systems. We focused on enhancing verbal communication and developed an AR-based remote collaboration system. Past research has shown that annotation and labels can support communication in collaboration. Thus, we proposed combining the two functions to achieve a smooth remote collaboration. Concerning the labeling function, we improved Chang et al’s work [1] and introduced semantic virtual labels that were attached to the objects in local workers’ environment. Besides, concerning the annotation function, we introduced a 3D pointer and the instructor can easily draw workers’ attention by moving the 3D pointer. In addition, the 3D pointer can highlight the virtual labels to improve the mutual understanding between the instructors and the workers. Later, a simple experiment was conducted to evaluate the usability of the proposed system.