We propose an interactive optical 3D-touch user interface (UI) by using a holographic light-field (LF) 3D display and a color detection system of the scattered light from the touched 3D image. In the proposed system, color information embedded in the LF is used to realize the 3D position identification and movement detection of the interaction point in 3D space only with a single RGB camera. We demonstrate the real-time interactive implementation of the interface working at 12 frames per second, which verifies the feasibility of our proposed concept.