The control of optical fields on the nanometer scale is a central theme of plasmonics and nanophotonics. Methods for characterizing localized optical field distributions are necessary to validate theoretical predictions, to test nanofabrication procedures, and to provide feedback for design improvements. Typical methods of probing near fields (e.g., single molecule fluorescence and near-field microscopy) cannot probe both the complex-valued and vectorial nature of the field distributions. We demonstrate that a nanoparticle probe with isotropic polarizability in combination with polarization control of excitation and detection beams provides access to this information through the interaction tensor. For a sample consisting of a single nanoparticle we show that the recorded images correspond to maps of the local Green's function tensor elements that couple the probe and sample. The tensorial mapping of interacting nanoparticles is of interest for optical sensing, optical antennas, surface-enhanced Raman scattering, nonlinear optics, and molecular rulers.