Perceptual color space is continuous; however, we tend to divide it into only a small number of categories. It is unclear whether categorical color perception is obtained solely through the development of the visual system or whether it is affected by language acquisition. To address this issue, we recruited prelinguistic infants (5- to 7-mo-olds) to measure changes in brain activity in relation to categorical color differences by using near-infrared spectroscopy (NIRS). We presented two sets of geometric figures to infants: One set altered in color between green and blue, and the other set altered between two different shades of green. We found a significant increase in hemodynamic responses during the between-category alternations, but not during the within-category alternations. These differences in hemodynamic response based on categorical relationship were observed only in the bilateral occipitotemporal regions, and not in the occipital region. We confirmed that categorical color differences yield behavioral differences in infants. We also observed comparable hemodynamic responses to categorical color differences in adults. The present study provided the first evidence, to our knowledge, that colors of different categories are represented differently in the visual cortex of prelinguistic infants, which implies that color categories may develop independently before language acquisition.