Appearance of a color stimulus is significantly affected by the contrast between its luminance and the luminance of the background. In the present study, we used stimuli evenly distributed on the CIE-xy chromaticity diagram to examine how luminance contrast affects neural representation of color in V4 and the anterior inferior temporal (AITC) and posterior inferior temporal (PITC) color areas (Banno et al., 2011). The activities of single neurons were recorded from monkeys performing a visual fixation task, and the effects of luminance contrast on the color selectivity of individual neurons and their population responses were systematically examined by comparing responses to color stimuli that were brighter or darker than the background. We found that the effects of luminance contrast differed considerably across V4 and the PITC and AITC. In both V4 and the PITC, the effects of luminance contrast on the population responses of color-selective neurons depended on color. In V4, the size of the effect was largest for blue and cyan, whereas in the PITC, the effect gradually increased as the saturation of the color stimulus was reduced, and was especially large with neutral colors (white, gray, black). The pattern observed in the PITC resembles the effect of luminance contrast on color appearance, suggesting PITC neurons are closely involved in the formation of the perceived appearance of color. By contrast, the color selectivities of AITC neurons were little affected by luminance contrast, indicating that hue and saturation of color stimuli are represented independently of luminance contrast in the AITC.