Understanding which biological pathways are specific versus general across diagnostic categories and levels of symptom severity is critical to improving nosology and treatment of psychopathology. Here, we combine transdiagnostic and dimensional approaches to genetic discovery for the first time, conducting a novel multivariate genome-wide association study of eight psychiatric symptoms and disorders broadly related to mood disturbance and psychosis. We identify two transdiagnostic genetic liabilities that distinguish between common forms of psychopathology versus rarer forms of serious mental illness. Biological annotation revealed divergent genetic architectures that differentially implicated prenatal neurodevelopment and neuronal function and regulation. These findings inform psychiatric nosology and biological models of psychopathology, as they suggest that the severity of mood and psychotic symptoms present in serious mental illness may reflect a difference in kind rather than merely in degree.