Studies of older adults with depressive disorders indicate greater cognitive deficits and brain alterations than would be expected for their age. There is some evidence that these findings are present after a single episode of depression, but this work has been cross-sectional in nature. We investigated both cross-sectional and longitudinal associations between a history of elevated depressive symptoms (HDS), frontal lobe volumes, and cognitive performance within the context of normal age-related changes over time in the Baltimore Longitudinal Study of Aging. After controlling for age, HDS was associated with smaller total frontal gray matter volumes and with smaller regional volumes in the cingulate gyrus and orbitofrontal cortex. Men, but not women, with HDS showed deficits in auditory attention span at older ages. Results confirm previous reports that even a single episode of depression is associated with adverse outcomes in older adults but suggest that HDS does not affect longitudinal trajectories of cognitive and brain volume change.