The aim of this study was to analyze correlations among the annual rate of gray matter volume change, age, gender, and cerebrovascular risk factors in 381 healthy community-dwelling subjects with a large age range by applying a longitudinal design over 6 years using brain magnetic resonance images (MRIs). Brain MRI data were processed with voxel-based morphometry using a custom template by applying diffeomorphic anatomical registration using the exponentiated lie algebra procedure. The annual rate of regional gray matter volume change showed significant positive correlations with age in several regions, including the bilateral temporal pole, caudate nucleus, ventral and dorsolateral prefrontal cortices, insula, hippocampus, and temporoparietal cortex, whereas significant negative correlations with age were observed in several regions including the bilateral cingulate gyri and anterior lobe of the cerebellum. Additionally, a significant age-by-gender interaction was found for the annual rate of regional gray matter volume change in the bilateral hippocampus. No significant correlations were observed between the annual rate of regional gray matter volume change and body mass index or systolic blood pressure. A significant positive correlation between the annual rate of gray matter volume change and age indicates that the region shows not linear but accelerated gray matter loss with age. Therefore, evaluating the annual rate of the gray matter volume change with age in healthy subjects is important in understanding how gray matter volume changes with aging in each brain region and in anticipating what cognitive functions are likely to show accelerated decline with aging.