We propose an efficient sensitivity analysis method for a wide class of colored-noise-driven interacting particle systems (IPSs). Our method is based on unperturbed simulations and significantly extends the Malliavin weight sampling method proposed by Szamel [Europhys. Lett. 117, 50010 (2017)0295-507510.1209/0295-5075/117/50010] for evaluating sensitivities such as linear response functions of IPSs driven by simple Ornstein-Uhlenbeck processes. We show that the sensitivity index depends not only on two effective parameters that characterize the variance and correlation time of the noise, but also on the noise spectrum. In the case of a single particle in a harmonic potential, we obtain exact analytical formulas for two types of linear response functions. By applying our method to a system of many particles interacting via a repulsive screened Coulomb potential, we compute the mobility and effective temperature of the system. Our results show that the system dynamics depend, in a nontrivial way, on the noise spectrum.