The APOE epsilon 4 allele is associated with a risk of Alzheimer's disease in the elderly, with the association being pronounced in females. Conversely, findings of the effects of the APOE epsilon 4 allele in young adults are mixed. Here, we investigated the sex-genotype interaction effects of the APOE epsilon 4 allele on cognitive functions as well as brain structures among 1258 young adults. After adjusting for multiple comparisons, there were significant effects of the interaction between sex and the number of APOE epsilon 4 allele on some speed tasks (e.g., simple processing speed tasks and the reverse Stroop task) as well as on regional white matter volume (rWMV). The observed sex-genotype interaction conferred better cognitive performance and greater rWMV in the anterior frontal and precentral white matter areas in females having more APOE epsilon 4 alleles and reduced rWMV in the same areas in male having more APOE epsilon 4 alleles. These findings support the long-debated antagonistic pleiotropic effects of the APOE epsilon 4 allele in females.