In this paper, we report a three-dimensional synthetic aperture imaging method with pulsed terahertz waves realized by a terahertz time-domain spectrometer. In contrast to synthetic aperture imaging systems operating at microwave or millimeter-wave frequencies where the frequency of the transmitter is scanned in the frequency domain, in our imaging system, all the frequency components are contained in a single terahertz pulse that can be generated and detected by photoconductive antennas. The image algorithm was analyzed theoretically and confirmed numerically using the finite-difference time-domain method. A key with plentiful detailed structures was used as the object to be imaged to demonstrate the three-dimensional imaging capabilities of this method. The resolution of the imaging system is 0.3 mm for the linear dimension and 0.1 mm for the circular dimension, as tested by the experimental setup. Finally, an optically opaque plastic pen with and without the cartridge was imaged, and the shape and location of the cartridge could be observed from the reconstructed three-dimensional terahertz images, demonstrating the non-destructive evaluation capabilities of this imaging method. Benefiting from the improvements in the experimental setup in this study, the imaging speed was significantly improved compared with that of the step-by-step scanning method commonly used in terahertz imaging systems with a single transmitter/receiver pair. This imaging method avoids the image degradation caused by specular reflections in active quasi-optical focal plane imaging and the lack of semiconductor devices working at several THz frequencies for synthetic aperture imaging, and may be used for non-destructive evaluation of objects with complex surfaces and internal structures.