In tropical forests, resource-based niches and density-dependent mortality are mutually compatible mechanisms that can act simultaneously to limit seedling populations. Differences in the strengths of these mechanisms will determine their roles in maintaining species coexistence. In the first assessment of these mechanisms in a Congo Basin forest, we quantified their relative strengths and tested the extent to which density-dependent mortality is driven by the distance-dependent behavior of seed and seedling predators predicted by the Janzen-Connell hypothesis. We conducted a large-scale seed addition experiment for five randomly selected tropical tree species, caging a subset of seed addition quadrats against vertebrate predators. We then developed models to assess the mechanisms that determine seedling emergence (three months after seed addition) and survival (two years after seed addition). As predicted, both niche differentiation and density-dependent mortality limited seedling recruitment, but predation had the strongest effects on seedling emergence and survival. Seedling species responded differently to naturally occurring environmental variation among sites, including variation in light levels and soil characteristics, supporting predictions of niche-based theories of tropical tree species coexistence. The addition of higher densities of seeds into quadrats initially led to greater seedling emergence, but survival to two years decreased with seed density. Seed and seedling predation reduced recruitment below levels maintained by density-dependent mortality, an indication that predators largely determine the population size of tree seedlings. Seedling recruitment was unrelated to the distance to or density of conspecific adult trees, suggesting that recruitment patterns are generated by generalist vertebrate herbivores rather than the specialized predators predicted by the Janzen-Connell hypothesis. If the role of seed and seedling predation in limiting seedling recruitment is a general phenomenon, then the relative abundances of tree species might largely depend on species-specific adaptations to avoid, survive, and recover from damage induced by vertebrate herbivores. Likewise, population declines of herbivorous vertebrate species (many of which are large and hunted) may trigger shifts in species composition of tropical forests.