Crop rotation schemes are believed to work by preventing specialist soil-borne pests from depressing the future yields of similar crops. In ecology, such negative plant-soil feedbacks may be viewed as a type of Janzen-Connell effect, which promotes species coexistence and diversity by preventing the same species from repeatedly occupying a particular site. In a controlled greenhouse experiment with 24 plant species and using soils from established field monocultures, we reveal community-wide soil-based Janzen-Connell effects between the three major functional groups of plants in temperate European grasslands. The effects are much stronger and more prevalent if plants are grown in interspecific competition. Using several soil treatments (gamma irradiation, activated carbon, fungicide, fertilizer) we show that the mechanism of the negative feedback is the buildup of soil pathogens which reduce the competitive ability of nearly all species when grown on soils they have formerly occupied. We further show that the magnitude of the change in competitive outcome is sufficient to stabilize observed fitness differences between functional groups in reasonably large communities. The generality and strength of this negative feedback suggests that Janzen-Connell effects have been underestimated as drivers of plant diversity in temperate ecosystems.