The aim of this study was to investigate the relationship between olfactory recognition and morphological changes in olfactory brain regions including the amygdala, hippocampus, rectus, parahippocampus, orbitofrontal cortex, and medial frontal cortex in 27 elderly subjects and 27 younger healthy controls. The specific aim of the study was to determine which brain areas are associated with the initial decline of olfaction in elderly subjects, which occurs before the onset of dementia. All subjects underwent magnetic resonance imaging to measure anatomical brain volume and cortical thickness, and subjects were assessed using tests of olfactory acuity and cognitive function measured with the Montreal Cognitive Assessment. Overall brain volume reductions were observed in elderly subjects compared with young healthy controls, but only reduction in the volume of the left hippocampus was associated with decreased olfactory ability. The parahippocampus of elderly subjects was not different from that of controls; the extent of the reduction of parahippocampus volume varied among individuals, and reduction in this region was associated with olfactory decline. Similarly, parahippocampus thinning was associated with decreased olfactory function. The path analysis showed direct and indirect effects of hippocampus and parahippocampus volume on olfactory ability and that volume reductions in these areas were not associated with cognitive function. Parahippocampus volume reduction and thinning exhibited individual variation; this may be the first appearance of pathological changes and may lead to dysfunction in the connection of olfactory memory to the neocortex. Parahippocampus change may reflect the first sign of olfactory impairment prior to pathological changes in the hippocampus, amygdala and orbitofrontal cortex.