Chalcogenide perovskites with optimal band gap and desirable light absorption are promising for photovoltaic devices, whereas the absence of ferroelectricity limits their potential in applications. On the basis of first-principles calculations, we reveal the underlying mechanism of the paraelectric nature of Ba3Zr2S7 observed in experiments and demonstrate a general rule for the appearance of ferroelectricity in chalcogenide perovskites with Ruddlesden-Popper (RP) A(3)B(2)X(7) structures. Group theoretical analysis shows that the tolerance factor is the primary factor that dominates the ferroelectricity. Both Ba3Zr2S7 and Ba3Hf2S7 with large tolerance factor are paraelectric because of the 0 suppression of in-phase rotation that is indispensable to hybrid improper ferroelectricity. In contrast, Ca3Zr2S7, Ca3Hf2S7, Ca3Zr2Se7, and Ca3Hf2S7 with small tolerance factor exhibit in-phase rotation and can be stable in the ferroelectric Cmc2(1) ground state with nontrivial polarization. These findings not only provide useful guidance to engineering ferroelectricity in RP chalcogenide perovskites but also suggest potential ferroelectric semiconductors for photovoltaic applications.