Although secondary succession has been studied extensively, we have little knowledge of the succession of soil-borne microbial communities. In this study, we therefore examined the structures of the microbial communities across two separate chronosequences of chalk grasslands in Limburg, the Netherlands, which are at different stages of secondary succession after being abandoned for between 17 and >66 years. Arable fields were also included in the investigation as non-abandoned references. Changes in the soil-borne microbial communities, as determined by phylogenetic microarray and quantitative PCR methodologies, were correlated with the prevailing environmental conditions related to vegetation and soil biochemistry. We observed clear patterns of microbial secondary succession related to soil age, pH and phosphate status, as exemplified by the overrepresentation of Verrucomicrobia, Acidobacteria, Gemmatimonadetes, and alpha-, delta- and epsilon-Proteobacteria at late successional stages. Moreover, effects of secondary succession versus changes in soil pH could be resolved, with pH significantly altering the trajectory of microbial succession.