Brain aging is a complex process influenced by various lifestyle, environmental, and genetic factors, as well as by age-related and often co-existing pathologies. MRI and, more recently, AI methods have been instrumental in understanding the neuroanatomical changes that occur during aging in large and diverse populations. However, the multiplicity and mutual overlap of both pathologic processes and affected brain regions make it difficult to precisely characterize the underlying neurodegenerative profile of an individual from an MRI scan. Herein, we leverage a state-of-the art deep representation learning method, Surreal-GAN, and present both methodological advances and extensive experimental results that allow us to elucidate the heterogeneity of brain aging in a large and diverse cohort of 49,482 individuals from 11 studies. Five dominant patterns of neurodegeneration were identified and quantified for each individual by their respective (herein referred to as) R-indices. Significant associations between R-indices and distinct biomedical, lifestyle, and genetic factors provide insights into the etiology of observed variances. Furthermore, baseline R-indices showed predictive value for disease progression and mortality. These five R-indices contribute to MRI-based precision diagnostics, prognostication, and may inform stratification into clinical trials.