Behavioral and electrophysiological studies of visual processing routinely employ sine wave grating stimuli, an approach that has led to the development of models in which the first stage of cortical visual processing acts as a bank of narrowband local filters whose responses vary with the contrast of preferred structure falling within their receptive fields. The relevance of this approach to natural vision is currently being challenged. We examine the contrast response of the human visual system to natural scenes. The results support a narrowband approach to visual processing but require its elaboration. Unlike grating patterns, the contrast response to natural scenes depends on the phase structure at remote spatial scales, but over a limited spatial region. The results suggest that contrast gain control acts within, but not across, cortical hypercolumns and serves to reduce the difference between the responses of detectors in regions of high and low contrast. This process tends to normalize the response of the visual system across natural scenes, which contain uneven contrast distributions.