In the lattice QCD formalism, we investigate the relation between confinement and chiral symmetry breaking. A gauge-invariant analytical relation connecting the Polyakov loop and the Dirac modes is derived on a temporally odd-number lattice, where the temporal lattice size is odd, with the normal (nontwisted) periodic boundary condition for link variables. This analytical relation indicates that low-lying Dirac modes have little contribution to the Polyakov loop, and it is numerically confirmed at the quenched level in both confinement and deconfinement phases. This fact indicates no direct one-to-one correspondence between confinement and chiral symmetry breaking in QCD. Using the relation, we also investigate the contribution from each Dirac mode to the Polyakov loop. In the confinement phase, we find a new "positive/negative symmetry" of the Dirac-mode matrix element of the link-variable operator, and this symmetry leads to the zero value of the Polyakov loop. In the deconfinement phase, there is no such symmetry and the Polyakov loop is nonzero. Also, we develop a new method for spin-diagonalizing the Dirac operator on the temporally odd-number lattice modifying the Kogut-Susskind formalism.