Human visual performance is better below than above fixation along the vertical meridian—a phenomenon we refer to as vertical meridian asymmetry (VMA). Here, we used fMRI to investigate the neural correlates of the VMA. We presented stimuli of two possible sizes and spatial frequencies on the horizontal and vertical meridians and analyzed the fMRI data in subregions of early visual cortex (V1/V2) that corresponded retinotopically to the stimulus locations. Asymmetries in both the spatial extent and amplitude of the fMRI measurements correlated with the behavioral VMA. These results demonstrate that the VMA has a neural basis at the earliest stages of cortical visual processing and imply that visual performance is limited by the pooled sensory responses of large populations of neurons in the visual cortex.