Studying ethnically diverse groups is important for furthering our understanding of biological mechanisms of disease that may vary across human populations. The ε4 allele of apolipoprotein E (APOE ε4) is a well-established risk factor for Alzheimer's disease (AD), and may confer anatomic and functional effects years before clinical signs of cognitive decline are observed. The allele frequency of APOE ε4 varies both across and within populations, and the size of the effect it confers for dementia risk may be affected by other factors. Our objective was to investigate the role APOE ε4 plays in moderating brain volume in cognitively normal Chinese older adults, compared to older white Americans. We hypothesized that carrying APOE ε4 would be associated with reduced brain volume and that the magnitude of this effect would be different between ethnic groups. We performed whole brain analysis of structural MRIs from Chinese living in America (n = 41) and Shanghai (n = 30) and compared them to white Americans (n = 71). We found a significant interaction effect of carrying APOE ε4 and being Chinese. The APOE ε4xChinese interaction was associated with lower volume in bilateral cuneus and left middle frontal gyrus (Puncorrected<0.001), with suggestive findings in right entorhinal cortex and left hippocampus (Puncorrected<0.01), all regions that are associated with neurodegeneration in AD. After correction for multiple testing, the left cuneus remained significantly associated with the interaction effect (PFWE = 0.05). Our study suggests there is a differential effect of APOE ε4 on brain volume in Chinese versus white cognitively normal elderly adults. This represents a novel finding that, if verified in larger studies, has implications for how biological, environmental and/or lifestyle factors may modify APOE ε4 effects on the brain in diverse populations.