(1) Background: Preventive measures to control the spread of COVID-19 are essential, but they often cause social isolation and diminish the physical and mental health of older adults. In cognitively impaired individuals, the pandemic has worsened behavioral and psychological symptoms of dementia (BPSD). Here, we explored the factors contributing to the worsening of BPSD during the COVID-19 pandemic. (2) Methods: Potential patients were identified at a memory clinic in Japan between June 2017 and June 2021. Eligible patients had a diagnosis of mild cognitive impairment (MCI) or dementia during the study period. The outcome was BPSD, as assessed by using the Dementia Behavioral Disorders Scale. Information on patients’ lifestyle habits and use of care services was obtained for use as primary explanatory variables; multiple regression analysis was performed to examine the relationship between BPSD and care services use or lifestyle habits. The model was adjusted for sociodemographic factors, and the interaction terms of the pandemic period with lifestyle and service use were included to evaluate the effects of COVID-19. (3) Results: We identified 977 participants with MCI and 1380 with dementia (MCI group: 69.8% age 75 years or older, 54.2% female; dementia group: 79.8% age 75 years or older, 64.8% female). After adjustment for possible confounders, significantly worse BPSD was demonstrated in those who used daycare services during COVID-19 (both MCI and dementia patients; p = 0.007 and p = 0.025 respectively) and in those with poor nutritional function (dementia patients; p = 0.040). (4) Conclusions and Implications: During COVID-19, poor nutritional status and use of daycare services were associated with BPSD in those with cognitive decline. These findings indicate the need to fully examine the quantity and quality of care services for people with cognitive decline during emergencies and to continue to provide effective services.