This paper describes a new type of motion capture system for virtual reality (VR) that uses rotating magnetic field and differential magnetic field. This system can provide enough resolution, accuracy and response speed for VR with low cost compared with typical motion capture systems that uses CCD cameras, magnetic field or mechanical links. Our method uses a cubic exciting coil that consists of six square shaped coils and pickup coils that are arranged on three axes of Cartesian coordinate fixed on target. Angular posture of target is detected by measuring phase of induced signal on pickup coils that are attached on the target in rotational magnetic field. The linear position of target is detected by measuring amplitude of magnetic field at target as differential transformer. Cross-correlations between induced signals and exciting current signals are calculated to obtain phase difference and amplitude with good tolerance to noises. These three dimensional angular and linear detection are performed simultaneously by multiple frequencies method. A compensation method for angular position that uses redundancy of detection and an idea to compose each output of pickup coils are also proposed to reduce errors. Experiments were carried out to confirm effectiveness of our method. We have obtained resolution of 0.1[deg] in angular detection and 0.5[mm] in linear detection.