To explore the factors limiting the development of visual sensitivity, we constructed an ideal observer model for the infant macaque visual system. We made measurements of retinal morphology in infant and adult macaque monkeys, and used the data in combination with published optical data to formulate the model. We compared the ideal observer’s ability to detect low-contrast gratings presented either in isolation or in spatiotemporal noise with behavioral data obtained under matched conditions. The ideal observer showed some improvement in visual performance up to the age of 4 weeks, but little change thereafter. Behavioral data show extensive changes over the ages 5-50 wk, after the ideal observer’s performance has become asymptotic. We conclude that the development of visual sensitivity in infant monkeys is not limited by changes in the front-end factors captured by the ideal observer model, at least after the age of 5 weeks. Using noise masking, we also estimated the variability of neural processing in comparison with the photon noise-limited ideal. We found that both additive and multiplicative components of this variability are elevated in infant monkeys, and improve (though not to ideal levels) during development. We believe that these changes all reflect maturation of visual processing in cortical circuits, and that no aspect of visual performance in the regime we studied is limited by the properties of the retina and photoreceptors, either in infant or in adult animals.