Climate resilience of crops is critical for global food security. Understanding the genetic basis of plant responses to ambient environmental changes is key to developing resilient crops. To detect genetic factors that set flowering time according to seasonal temperature conditions, we evaluated differences of flowering time over years by using chromosome segment substitution lines (CSSLs) derived from japonica rice cultivars “Koshihikari” × “Khao Nam Jen”, each with different robustness of flowering time to environmental fluctuations. The difference of flowering times in 9 years’ field tests was large in “Khao Nam Jen” (36.7 days) but small in “Koshihikari” (9.9 days). Part of this difference was explained by two QTLs. A CSSL with a “Khao Nam Jen” segment on chromosome 11 showed 28.0 days’ difference; this QTL would encode a novel flowering-time gene. Another CSSL with a segment from “Khao Nam Jen” in the region around Hd16 on chromosome 3 showed 23.4 days” difference. A near-isogenic line (NIL) for Hd16 showed 21.6 days’ difference, suggesting Hd16 as a candidate for this QTL. RNA-seq analysis showed differential expression of several flowering-time genes between early and late flowering seasons. Low-temperature treatment at panicle initiation stage significantly delayed flowering in the CSSL and NIL compared with “Koshihikari”. Our results unravel the molecular control of flowering time under ambient temperature fluctuations.