Dementia and cognitive impairment are considered to be one of the biggest social and medical problems. While there is a definite relationship between vitamin B and cognitive decline, this has yet to be fully assessed with regard to sex differences. Thus, the present study investigated the relationship of vitamin B1 or vitamin B12 with dementia in accordance with the sex in 188 patients who visited the Memory Clinic at Showa University Hospital in Japan from March 2016 to March 2019. Cognitive function was tested by the Japanese version of the Mini-Mental State Examination (MMSE) and Hasegawa Dementia Scale-Revised (HDS-R). Blood tests were performed to measure the vitamin levels. Logistic regression analysis was used to calculate the odds ratio (OR) for dementia and the 95% confidence interval (CI). Compared to the highest vitamin group (third tertile), the lowest vitamin group (first tertile) exhibited a significantly increased OR for dementia defined by MMSE for vitamin B1 (OR:3.73, 95% CI:1.52-9.16) and vitamin B12 (2.97, 1.22-7.28) among women. In contrast, vitamin levels were not significantly associated with dementia determined by MMSE in men. These findings were similar even when dementia was defined by HDS-R. The present study suggests that vitamin B1 plays a role in preventing development of dementia in women. Future longitudinal studies will need to be undertaken in order to examine whether decreasing vitamin levels occur before or after cognitive impairment, and whether maintaining a higher vitamin level can prevent a worsening of cognitive function and the development of dementia.