Purpose: The Hospital Anxiety and Depression Scale (HADS) is a self-report questionnaire widely used to assess anxiety and depression. To the best of our knowledge, only four studies have examined the factor structure of the HADS for assessing pregnant women, with conflicting results. This study aimed to assess the factor structure and measurement invariance of the HADS for use with pregnant Japanese women. Participants and Methods: A total of 936 pregnant Japanese women completed the HADS questionnaire at three time points: the first and third trimester of pregnancy, and postpartum. We examined the factor structure of the HADS in Group 1 (n = 466) using exploratory factor analysis (EFA). We then compared the models identified in Group 1 with those from previous studies using confirmatory factor analysis (CFA) in Group 2 (n = 470). We performed multiple-group CFA for Group 2 to test the measurement invariance of the best-fit model across the three time points. Results: The EFA for the Group 1 data at the three time points revealed a two-factor model. In the CFA, the two-factor model from Group 1 showed the best fit with the data at the three time points. In the multiple-group CFA for Group 2, we confirmed the configural and metric invariance of the two-factor model across the three time points. Conclusion: Our findings provide evidence for a two-factor structure and weak measurement invariance of the HADS in pregnant Japanese women during the peripartum period.