Caffeine has been described as a sports performance-enhancing substance. However, it is unclear whether it can increase the level of physical activity (PA) in nonathletic individuals. This study investigates the relationship between daily caffeine consumption and (a) daily PA/fitness or (b) intervention-induced changes in PA in women and men. On the basis of responses to a dietary habit questionnaire, which included items on caffeinated beverages, 1,032 Japanese adults, were categorized into lower or higher caffeine consumption groups (relative to the median caffeine consumption). In each group, daily step count; sedentary time; and light, moderate, and vigorous PA outcomes were objectively measured. Physical fitness, including peak oxygen consumption, was also evaluated. The relationship between daily caffeine consumption and the change in the levels of PA was investigated in a subgroup of 202 subjects who participated in a 1-year PA counseling intervention. Women in the higher caffeine consumption group presented higher moderate-to-vigorous PA and step count compared with their counterparts in the lower consumption group (4.0 ± 2.1 vs. 3.3 ± 2.1 MET-hr/day, p < .001; 10,335 ± 3,499 vs. 9,375 ± 3,527 steps/day, p < .001). A significant positive correlation was noted between caffeine consumption and peak oxygen consumption among women (r = .15, p < .001). No caffeine-related effects were noted in men. The lower and higher caffeine consumption groups showed no significant differences in their levels of PA at the end of the 1-year intervention. Therefore, caffeine consumption appears to be associated with higher levels of PA in Japanese women. Further studies are needed to clarify this association.