The first stars are predicted to have formed within 200 million years after the Big Bang, initiating the cosmic dawn. A true first star has not yet been discovered, although stars with tiny amounts of elements heavier than helium ('metals') have been found in the outer regions ('halo') of the Milky Way. The first stars and their immediate successors should, however, preferentially be found today in the central regions ('bulges') of galaxies, because they formed in the largest over-densities that grew gravitationally with time. The Milky Way bulge underwent a rapid chemical enrichment during the first 1-2 billion years, leading to a dearth of early, metal-poor stars. Here we report observations of extremely metal-poor stars in the Milky Way bulge, including one star with an iron abundance about 10,000 times lower than the solar value without noticeable carbon enhancement. We confirm that most of the metal-poor bulge stars are on tight orbits around the Galactic Centre, rather than being halo stars passing through the bulge, as expected for stars formed at redshifts greater than 15. Their chemical compositions are in general similar to typical halo stars of the same metallicity although intriguing differences exist, including lower abundances of carbon.