It has often been proposed that regions of the human parietal and/or frontal lobe may modulate activity in visual cortex, for example, during selective attention or saccade preparation. However, direct evidence for such causal claims is largely missing in human studies, and it remains unclear to what degree the putative roles of parietal and frontal regions in modulating visual cortex may differ. Here we used transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI) concurrently, to show that stimulating right human intraparietal sulcus (IPS, at a site previously implicated in attention) elicits a pattern of activity changes in visual cortex that strongly depends on current visual context. Increased intensity of IPS TMS affected the blood oxygen level-dependent (BOLD) signal in V5/MT+ only when moving stimuli were present to drive this visual region, whereas TMS-elicited BOLD signal changes were observed in areas V1-V4 only during the absence of visual input. These influences of IPS TMS upon remote visual cortex differed significantly from corresponding effects of frontal (eye field) TMS, in terms of how they related to current visual input and their spatial topography for retinotopic areas V1-V4. Our results show directly that parietal and frontal regions can indeed have distinct patterns of causal influence upon functional activity in human visual cortex.