A planar laser pulse propagating in vacuum can exhibit an extremely large ponderomotive force. This force, however, cannot impart net energy to an electron: As the pulse overtakes the electron, the initial impulse from its rising edge is completely undone by an equal and opposite impulse from its trailing edge. Here we show that planarlike "flying focus" pulses can break this symmetry, imparting relativistic energies to electrons. The intensity peak of a flying focus-a moving focal point resulting from a chirped laser pulse focused by a chromatic lens-can travel at any subluminal velocity, forward or backward. As a result, an electron can gain enough momentum in the rising edge of the intensity peak to outrun and avoid the trailing edge. Accelerating the intensity peak can further boost the momentum gain. Theory and simulations demonstrate that these dynamic intensity peaks can backwards accelerate electrons to the MeV energies required for radiation and electron diffraction probes of high energy density materials.