The coronavirus disease 2019 (COVID-19) pandemic has profoundly affected the mental health of both infected and uninfected people. Although most psychiatric disorders have highly overlapping genetic and pathogenic backgrounds, most studies investigating the impact of the pandemic have examined only single psychiatric disorders. It is necessary to examine longitudinal trajectories of factors that modulate psychiatric states across multiple dimensions. About 2274 Japanese citizens participated in online surveys presented in December 2019 (before the pandemic), August 2020, Dec 2020, and April 2021. These surveys included nine questionnaires on psychiatric symptoms, such as depression and anxiety. Multidimensional psychiatric time-series data were then decomposed into four principal components. We used generalized linear models to identify modulating factors for the effects of the pandemic on these components. The four principal components can be interpreted as a general psychiatric burden, social withdrawal, alcohol-related problems, and depression/anxiety. Principal components associated with general psychiatric burden and depression/anxiety peaked during the initial phase of the pandemic. They were further exacerbated by the economic burden the pandemic imposed. In contrast, principal components associated with social withdrawal showed a delayed peak, with human relationships as an important risk modulating factor. In addition, being female was a risk factor shared across all components. Our results show that COVID-19 has imposed a large and varied burden on the Japanese population since the commencement of the pandemic. Although components related to the general psychiatric burden remained elevated, peak intensities differed between components related to depression/anxiety and those related to social withdrawal. These results underline the importance of using flexible monitoring and mitigation strategies for mental problems, according to the phase of the pandemic.