Although heavy alcohol consumption has been identified as a risk factor for adverse cognitive functioning, it currently remains unclear whether moderate alcohol consumption exerts similar effects. Observational studies previously reported the potential benefits of moderate alcohol consumption on cognition, particularly in the elderly; however, these effects have not yet been demonstrated in Asian populations. The aim of the present study was to investigate the relationship between alcohol consumption levels and global and domain-specific cognitive functions in cognitively intact elderly Japanese men. Cross-sectional data from the Shiga Epidemiological Study of Subclinical Atherosclerosis (SESSA), an ongoing prospective, population-based study in Shiga, Japan, were used to examine the relationship between alcohol consumption and cognitive function. Men (n = 585) aged ≥65 years provided information on their weekly consumption of alcohol, and the data obtained were used to construct categories of never, ex- (quit before interview), very light (<14 g/day), light (14-23 g/day), moderate (>23-46 g/day), and heavy (>46 g/day) drinkers. Cognitive function was measured using the Cognitive Abilities Screening Instrument (CASI). A fractional logistic regression model adjusted for age, education, body mass index, smoking, exercise, hypertension, diabetes, and dyslipidemia showed that the CASI scores for global and domain-specific cognitive functions were not significantly different between all subgroups of current drinkers and never-drinkers. However, the CASI score of ex-drinkers (multivariable adjusted mean CASI score [SD]) was significantly lower than that of never-drinkers in the global [never vs. ex: 90.16 (2.21) vs. 88.26 (2.58)] and abstraction and judgment domain [never vs. ex: 9.48 (0.46) vs. 8.61 (0.57)]). The present results do not suggest any beneficial or adverse relationship between current alcohol consumption levels and cognitive functioning (both global and domain specific) in elderly Japanese men; however, low cognitive function among ex-drinkers warrants future investigations to identify the factors causing drinkers to quit.