Recording manipulation tasks are very useful for teaching-by-demonstration and digital archiving of manipulation skills. In this paper, we propose a method for recording hand-object relationship during tasks using first-person RGB-D vision. First-person vision expands the recording area even though only one camera is used. Detection of changes from 3D information relieves complicated recognition by an RGB image. By detecting appearance/disappearance of an object under the region occluded by a hand, we estimate action of release/grasp and appearance of the target object. We verify the effectiveness of the proposed method through an experiment of a simple pick-and-place task.