We developed a fast X-ray detector system for nuclear resonant scattering (NRS) experiments. Our system employs silicon avalanche photo-diode (Si-APD) as a fast X-ray sensor. The system is able to acquire both timing and energy of a single X-ray photon simultaneously in a high rate condition, 106 counts per second for one Si-APD. The performance of the system was investigated in SPring-8, a synchrotron radiation facility in Japan. Good time resolution of 120 ps (FWHM) was achieved with a slight tail distribution in the time spectrum by a level of 10(-9) at 1 ns apart from the peak. Using this system, we successfully observed the NRS from the 26.27-keV level of mercury-201, which has a half-life of 630(50) ps. We also demonstrated the reduction of background events caused by radioactive decays in a radioactive sample by discriminating photon energy. Published by AIP Publishing.