The dressed state formalism enables us to define the infrared finite S-matrix for QED. In the formalism, asymptotic charged states are dressed by clouds of photons. The dressed asymptotic states are originally obtained by solving the dynamics of the asymptotic Hamiltonian in the far past or future region. However, there was an argument that the obtained dressed states are not gauge invariant. We resolve the problem by imposing a correct gauge invariant condition. We show that the dressed states can be obtained just by requiring the gauge invariance of asymptotic states. In other words, Gauss's law naturally leads to proper asymptotic states for the infrared finite S-matrix. We also discuss the relation between the dressed state formalism and the asymptotic symmetry for QED.