Long-term memory integrates the multimodal information acquired through perception into unified concepts, supporting object recognition, thought, and language. While some theories of human cognition have considered concepts to be abstract symbols, recent functional neuroimaging evidence has supported an alternative theory: that concepts are multimodal representations associated with the sensory and motor systems through which they are acquired. However, few studies have examined the effects of cortical lesions on the sensory and motor associations of concepts. We tested the hypothesis that individuals with disease in auditory association cortex would have difficulty processing concepts with strong sound associations (e.g., thunder). Human participants with the logopenic variant of primary progressive aphasia (lvPPA) performed a recognition task on words with strong associations in three modalities: Sound, Sight, and Manipulation. LvPPA participants had selective difficulty on Sound words relative to other modalities. Structural MRI analysis in lvPPA revealed gray matter atrophy in auditory association cortex, as defined functionally in a separate BOLD fMRI study of healthy adults. Moreover, lvPPA showed reduced gray matter density in the region of auditory association cortex that healthy participants activated when processing the same Sound words in a separate BOLD fMRI experiment. Finally, reduced gray matter density in this region in lvPPA directly correlated with impaired performance on Sound words. These findings support the hypothesis that conceptual memories are represented in the sensory and motor association cortices through which they are acquired.