Dynamical processes on time-varying complex networks are key to understanding and modeling a broad variety of processes in socio-technical systems. Here we focus on empirical temporal networks of human proximity and we aim at understanding the factors that, in simulation, shape the arrival time distribution of simple spreading processes. Abandoning the notion of wall-clock time in favour of node-specific clocks based on activity exposes robust statistical patterns in the arrival times across different social contexts. Using randomization strategies and generative models constrained by data, we show that these patterns can be understood in terms of heterogeneous inter-event time distributions coupled with heterogeneous numbers of events per edge. We also show, both empirically and by using a synthetic dataset, that significant deviations from the above behavior can be caused by the presence of edge classes with strong activity correlations.