Both adults and infants are faster at discriminating between two colors from different categories than two colors from the same category, even when between- and within-category chromatic separation sizes are equated. For adults, this categorical perception (CP) is lateralized; the category effect is stronger for the right visual field (RVF)-left hemisphere (LH) than the left visual field (LVF)-right hemisphere (RH). Converging evidence suggests that the LH bias in color CP in adults is caused by the influence of lexical color codes in the LH. The current study investigates whether prelinguistic color CP is also lateralized to the LH by testing 4- to 6-month-old infants. A colored target was shown on a differently colored background, and time to initiate an eye movement to the target was measured. Target background pairs were either from the same or different categories, but with equal target-background chromatic separations. Infants were faster at initiating an eye movement to targets on different-category than same-category backgrounds, but only for targets in the LVF-RH. In contrast, adults showed a greater category effect when targets were presented to the RVF-LH. These results suggest that whereas color CP is stronger in the LH than RH in adults, prelinguistic CP in infants is lateralized to the RH. The findings suggest that language-driven CP in adults may not build on prelinguistic CP, but that language instead imposes its categories on a LH that is not categorically prepartitioned.