A robot system cooperating with gesture instructions using the Kinect sensor is proposed in this research. Specifically, coordinates of right hand is measured by the Kinect sensor, and the robot (omni-directional mobile robot) is moved based on the position data of the right hand obtained from the Kinect sensor. The robot system introduces operation modes of "flick mode" and "following mode". In the flick mode, the algorithm to move the object with the mass and the viscosity by virtual power corresponding to the velocity of the hand is proposed, and the robot follows the calculated position of the object. In the following mode, the robot follows the motion of the hand cooperatively through the "kansei transfer function". In addition, the neural network is used for switching of the two modes. A flick motion and following motion is input into a neural network for the learning of the characteristics of both modes. After the learning, the mode is identified by the neural network from the motion of the hand. The experiment using a suggested system was performed. As a result, a mode was able to be changed from the movement of the hand automatically. In addition, the movement characteristic is changed when the virtual viscosity was changed in the flick motion.