Cognitive decline in degenerative dementia is paralleled by progressive brain atrophy, with the localization of atrophy reflecting specific cognitive impairment. Confrontation naming deficits are frequently observed in dementia across etiologies. In this study we aimed to identify the brain regions underlying this deficit. In patients with clinically diagnosed dementia or mild cognitive impairment (MCI) we investigated the relationship between gray matter volume (GMV) and performance on a standardized confrontation naming test. 268 patients with one of three probable etiologies were included: Alzheimer's Dementia (AD), AD with signs of cerebrovascular pathology, and frontotemporal dementia. Applying voxel-based morphometry using a diffeomorphic registration algorithm we contrasted GMV of patients performing within the normal range with those of patients with pathological performance. Further, differential effects of gray matter atrophy on impaired performance in AD versus MCI of AD type were investigated. Results revealed significantly reduced GMV in the left anterior temporal lobe (ATL) in pathological performers compared to normal performers. The subgroup analysis confined to MCI of AD type and AD patients confirmed this relationship. While left ATL atrophy is known to be implicated in naming deficits in semantic dementia, our data confirm the same in AD and MCI of AD type.