Temporal variations in ambient dose rates in a restricted area designated as "difficult-to-return" for residents of Tomioka Town, Fukushima Prefecture were evaluated in a car-borne survey during 2018-2019. The median dose rates in the "Decontaminated area" in the difficult-to-return zone decreased rapidly from 1.0 μSv/h to 0.32 μSv/h; however, the median dose rates in the "Non-decontaminated area" and "Radioactive waste storage area" fluctuated between 1.1-1.4 μSv/h and 0.46-0.61 μSv/h, respectively. The detected rate of the cesium-137 (137Cs) (137Cs-detected points per all measuring points) in the "Decontaminated area" also decreased rapidly from 64% to 6.7%, accompany with decreasing in ambient dose rates. On the other hand, the detection of 137Cs in the "Radioactive waste storage area" and "Non-decontaminated area" decreased from 53% to 17% and 93% to 88%, respectively. We confirmed that the dose rates in the Decontaminated area dramatically decreased due to decontamination work aiming to help residents return home. Moreover, the estimated external exposure dose of workers during the present survey was 0.66 mSv/y in the Decontaminated area and 0.55 mSv/y in the Radioactive waste storage area, respectively. This case of Tomioka Town within the "difficult-to-return zone" may be the first reconstruction model for evaluating environmental contamination and radiation exposure dose rates due to artificial radionuclides derived from the nuclear disaster.