Infants are slower to disengage from faces than non-face patterns when distracted by novel competing stimuli. While this perceptual predilection for faces is well documented, its universality and mechanisms in relation to other aspects of attention are poorly understood. We analysed attention disengagement times for faces and non-face patterns in a large sample of 6-to 9-month-old infants (N = 637), pooled from eye tracking studies in socioculturally diverse settings (Finland, Malawi, South Africa). Disengagement times were classified into distinct groups of quick and delayed/censored responses by unsupervised clustering. Delayed disengagement was frequent for faces (52.1% of trials), but almost negligible for patterns (3.9% of trials) in all populations. The magnitude of this attentional bias varied by individuals, whereas the impact of situational factors and facial expression was small. Individual variations in disengagement from faces were moderately stable within testing sessions and independent from variations in disengagement times for patterns. These results point to a fundamental dissociation of face and pattern processing in infants and demonstrate that the bias for faces can be robust against distractors and habituation. The results raise the possibility that attention to faces varies as an independent, early-emerging social trait in populations.