Nonreciprocal single-photon scattering in a one-dimensional waveguide coupled to a giant two-level atom assisted with a time-modulated single-mode cavity is investigated. The analytic expressions of the single-photon scattering amplitudes are derived by using an effective Floquet Hamiltonian in real space. The scattering characteristics are discussed detail in both the Markovian and the non-Markovian regimes, and the corresponding conditions for achieving perfect nonreciprocal single-photon transmission are obtained. In the Markovian regime, a frequency-tunable single-photon diode with an ideal transmission contrast ratio can be realized by adjusting the frequency of the cavity mode, the local coupling phase difference, and the accumulated phase between the two coupling points. Furthermore, the influence of the intrinsic energy dissipations on the photon transport is discussed in detail. It is found that the dissipations of the cavity and the giant atom affect discriminatively the nonreciprocal single-photon scattering process. In the non-Markovian regime, the influence of the non-Markovian retarded effect induced by the time delay on the nonreciprocal single-photon scattering is discussed in detail. The results reveal that, although the retarded effect leads to a complex nonreciprocal scattering spectrum, dynamic tunable perfect nonreciprocal transmission with more abundant physical phenomena suitable for photons with different frequencies within a larger range can also be achieved. Such a nonreciprocal single-photon device can be used as an elementary unit for various quantum information processing and may have potential applications in quantum network engineering.