Coffee consumption is associated with cerebral hypoperfusion that may contribute to the development of cerebral white matter hyperintensities (WMH). We investigated the effect of lifetime coffee consumption on the volume of WMH (VWMH) in late life, and compared the effect between men and women since caffeine clearance may be different between sexes. We enrolled 492 community-dwelling cognitively normal elderly individuals (73.4 ± 6.7 years old on average) from the Korean Longitudinal Study on Cognitive Aging and Dementia. We evaluated their patterns and amounts of coffee consumption using a study-specific standardized interview and estimated cerebral VWMH by automatic segmentation of brain fluid-attenuated inversion recovery sequence magnetic resonance images. Higher cumulative lifetime coffee consumption was associated with higher logVWMH in both sexes (p = 0.030). The participants who consumed more than 2 cups of coffee per day on average in their lifetime showed higher logVWMH in late life than those who consumed less. When both sexes were analyzed separately, these coffee-logVWMH associations were found only in women, although the volumes of brain and white matter of women were smaller than those of men. Our findings suggest that prolonged high coffee consumption may be associated with the risk of WMH in late life.