The dynamic characteristics of mid-story isolated buildings and seismic response reduction due to a semi-active control system were investigated using a three-lumped-mass model that simplified the sixteen story building with an isolation layer in the sixth story. A semi-active control method using a rotary inertia mass damper filled with magnetorheological fluid (MR fluid) was proposed. The damper shows both mass amplification effect due to rotational inertia and variable damping effect due to the MR fluid. The damping force is controlled by the strength of the magnetic field that is applied to the MR fluid. It is determined by using the electric current, which is calculated by the proposed semi-active control method based on the velocity of the isolation layer relative to the layer just underneath it. Real-time hybrid tests using an actual damper and simulations using a building model were conducted to check the damper model; the test results were in good agreement with the simulation results. The simulation results suggest that the response displacement of the structure above the isolation layer is significantly reduced, without increasing the response acceleration of the entire structure against near-fault pulse and long-period ground motions. The proposed semi-active control using an MR rotary inertia mass damper was confirmed to be effective for mid-story isolated buildings.