This study aimed to evaluate the association between work-related changes caused by COVID-19 and psychological distress among Japanese workers. The study was conducted from August 25 to September 30, 2020. The participants were 15,454 employees who were registered as panelists with an online survey company. The Kessler psychological distress scale with a 13-point cutoff was used to measure psychological distress. Multiple logistic regression was performed. Of the respondents, 8.9% were evaluated as having severe psychological distress. Among five examined work-related changes, being laid off and changing jobs (adjusted odds ratio [aOR] = 5.43; 95% confidence interval [CI]: 4.18-7.05), experiencing temporary workplace closure (aOR = 1.94; 95% CI: 1.67-2.25), being forced to visit the workplace for paperwork (aOR = 1.84; 95% CI: 1.58-2.15), and starting telework from home (aOR = 1.18; 95% CI: 1.01-1.37) were associated with increased psychological distress; no significant association was found for participation in work-related online meetings. The impact on psychological distress was greater among men, especially for being laid off and changing jobs because of COVID-19. It is important to assess and reduce negative mental health effects among workers experiencing work-related changes caused by the COVID-19 pandemic, taking gender differences into account.