Family socioeconomic status (SES) is an important factor that affects an individual's neural and cognitive development. The two novel aims of this study were to reveal (a) the effects of family SES on mean diffusivity (MD) using diffusion tensor imaging given the characteristic property of MD to reflect neural plasticity and development and (b) the sex differences in SES effects. In a study cohort of 1,216 normal young adults, we failed to find significant main effects of family SES on MD; however, previously observed main effects of family SES on regional gray matter volume and fractional anisotropy (FA) were partly replicated. We found a significant effect of the interaction between sex and family income on MD in the thalamus as well as significant effects of the interaction between sex and parents' educational qualification (year's of education) on MD and FA in the body of the corpus callosum as well as white matter areas between the anterior cingulate cortex and lateral prefrontal cortex. These results suggest the sex-specific associations of family SES with neural and/or cognitive mechanisms particularly in neural tissues in brain areas that play key roles in basic information processing and higher-order cognitive processes in a way females with greater family SES level show imaging outcome measures that have been associated with more neural tissues (such as greater FA and lower MD) and males showed opposite.