Lateral inhibition is a central principle for sensory system function. It is thought to operate by the activation of inhibitory neurons that restrict the spatial spread of sensory excitation. Much work on the role of inhibition in sensory systems has focused on visual cortex; however, the neurons, computations, and mechanisms underlying cortical lateral inhibition remain debated, and its importance for visual perception remains unknown. Here, we tested how lateral inhibition from PV or SST neurons in mouse primary visual cortex (V1) modulates neural and perceptual sensitivity to stimulus contrast. Lateral inhibition from PV neurons reduced neural and perceptual sensitivity to visual contrast in a uniform subtractive manner, whereas lateral inhibition from SST neurons more effectively changed the slope (or gain) of neural and perceptual contrast sensitivity. A neural circuit model identified spatially extensive lateral projections from SST neurons as the key factor, and we confirmed this with anatomy and direct subthreshold measurements of a larger spatial footprint for SST versus PV lateral inhibition. Together, these results define cell-type specific computational roles for lateral inhibition in V1, and establish their unique consequences on sensitivity to contrast, a fundamental aspect of the visual world.