Late-life depression (LLD) is associated with an increased risk of developing dementia; however, it is not known whether individuals with a history of LLD exhibit a more rapid rate of cognitive decline. We aimed to determine whether those with LLD experienced faster cognitive decline compared with never-depressed control (NDC) participants from the community and whether stratification of LLD into early-onset depression (EOD) and late-onset depression (LOD) subtypes revealed differing rates and domain-specific expression of cognitive decline. We conducted a prospective, longitudinal study where 185 participants with LLD (remitted) and 114 NDC were followed for 5 years on average. EOD was defined as having first lifetime depressive episode at <60years and LOD at ≥60years. Every year, participants underwent comprehensive neuropsychological assessment. Composite scores for each cognitive domain were calculated through averaging standardized scores across tests. LLD compared to NDC demonstrated significant baseline impairment but did not decline more rapidly. EOD were significantly impaired in attention/processing speed and global cognitive function at baseline but did not experience more rapid decline as compared to NDC. Those with LOD compared to both NDC and EOD performed worse in all domains at baseline and experienced more rapid decline in verbal skills and delayed memory ability. Our findings suggest that baseline impairment may lower the threshold for those with LLD to develop dementia. EOD and LOD may represent distinct phenotypes of cognitive impairment with differing neural substrates. LOD may represent a distinct phenotype with a more rapid decline in verbal skills and delayed memory.