Every eye fixation brings to our retinae a new visual scene, from which the visual system must extract the most relevant information. Clearly, not all objects from a typical scene will be consciously registered (Rensink et al., 1997; O’Regan et al, 1999; Simons & Levin, 1998). Among those that will, many will not be consolidated into visual memory, and will be rapidly forgotten (Sperling, 1960; Baddeley, 1986). The visual system must therefore continuously and actively select at different stages the properties or objects relevant to current behavior and higher cognitive functions. How does this selection occur? What determines, and what is the relation between what we see, what we almost see, and what we fail to see?

There is increasing evidence that at least some form of high-level representation of the visual scene can be accessed very rapidly (Thorpe et al., 1996; VanRullen & Thorpe, 2001), in an automatic and possibly unconscious way (Ohman & Soares, 1994, 1998; Esteves et al., 1994; Dehaene et al., 1998; Bar et al, 2001; VanRullen & Koch, in press). This representation can be detailed enough to allow subjects to detect an animal in a briefly flashed image, or to categorize a scene in rapid serial visual presentation (RSVP; Potter & Levy, 1969; Potter, 1976; Bar & Biederman, 1998; Coltheart, 1999). In contrast, consciously recognizing an object probably requires some form of attention to be drawn selectively to this object (Rensink et al., 1997; Simons & Chabris, 1999; Mack & Rock, 1998). Further selection might be required in deciding what objects should be consolidated in memory, and what objects can be forgotten. Figure 1 illustrates this continuous selection process among successive levels of representation.

The capacity of these different levels of visual representation (preconscious, conscious, short-term memory) can be assessed with specific paradigms. Free recall is typically used to access the contents of immediate working memory, in general found to contain around 4 objects (Sperling, 1960; Broadbent, 1975; Pylyshyn & Storm, 1988; Yantis 1992; Luck & Vogel, 1997; Cowan 2001). Implicit measures, such as performance in forced-choice recognition, or visual priming, can be used to determine which objects were perceived, even when they are not explicitly remembered (e.g. Biederman & Cooper, 1991; Bar & Biederman, 1998, 1999).

However, estimates of capacity obtained by different studies with different paradigms, and at different levels of representation, are very unlikely to be comparable. Here we apply a combination of three such paradigms (free recall, forced-choice recognition, and visual priming) on the same complex natural scenes and for the same subjects. Immediately after a large natural scene containing 10 different objects was briefly presented, subjects had to report the objects that they had perceived. They could also “guess” an additional number of objects. Subsequently, these same objects were presented among other unfamiliar ones in a word-picture matching task. Reaction times were analyzed to reveal visual priming. Surprisingly, the objects that the subjects could neither explicitly report nor guess elicited a significant negative priming effect, suggesting that they had been suppressed at a rather late stage of visual processing.

It is necessary to stress that negative priming has been known for over 20 years as a reflection of active attentional suppression of ignored objects (Neill 1977; Tipper 1985; Fox 1995). However, it is typically observed in situations where a unique target (attended) object competes with another unique overlapping distractor (ignored) object, and the to-be-attended property (e.g. color) is defined in advance. Here negative priming is reported under “realistic” conditions of stimulation, where different objects of a natural scene compete for attentional resources and selection, and observers have no a priori bias as to what object or property they should attend to.

Each of 10 stimulus scenes (Figure 2A), containing 10 objects, was presented for 250 ms, immediately followed by a strong contrast color mask (a situation designed to approximate an average single fixation). The mask was obtained by superimposing many different samples of white noise that were band-pass filtered at particular spatial frequencies, so that the resulting mask would display a power spectrum resembling that of natural images (i.e. 1/f). The scene and mask subtended 16 degrees of visual angle in width. Immediately after each scene, subjects were presented with a list of 20 object names, including the 10 target objects. Distractor object names were carefully chosen so that they could have normally been present in the context of the scene. Subjects were asked to report the objects that they had consciously perceived with full confidence (free recognition). After signaling that they were not confident anymore, they had to select a further number of objects (forced-choice recognition), so that the overall number of selected objects, including the ones reported with full confidence, was exactly 10. Note that the term “forced-choice recognition” generally refers to a situation where the number of alternatives is determined by the experimenter. In our case, the number of alternatives is determined by the subjects’ performance in the previous “free recognition” task.

R^*, the corrected number of objects reported in free recognition (correct reports that can not be explained by chance), is defined as:

However, R^- is small enough in our case, and the difference between these 2 methods can be neglected.

In the forced-choice recognition part of the report task, the a priori distributions of probability for target and distractor objects are not equal but depend on the previous responses (R⁺, R^-) of each subject for each scene. Therefore, the number G^* of “above-chance guesses” can be defined as:

Immediately after the forced-choice recognition task for each scene, the subjects had to perform a block of 40 trials of a word-picture matching task, in which some of the stimuli were target objects that had appeared in the previous scene. In each trial, an object name was presented for 500 ms and, following an inter-stimulus interval of 1 second, an object image was flashed at fixation for 250 ms. The subjects held down the mouse button continuously, and had to release it as fast as possible, within 1 second, if and only if the object picture matched the previous word. The objects were presented on a uniform grey background of the same luminance as the rest of the screen. Object size was variable, between approximately 2 and 10 degrees of visual angle. The objects that were extracted from the scene were always presented with their original size, at the fixation point. The average change in eccentricity for a given object between its presentation in the scene and its presentation in isolation was around 3.5 degrees. All 20 object names from the free recognition and forced-choice recognition tasks (10 targets and 10 distractors) were presented in this block. The 10 target objects always matched the target names (“match” trials). Five of the 10 distractor words were paired with a matching object, and five with a non-matching object. Finally, in an additional 20 trials of the same block, both the object name and the object picture were totally new (15 “match” and 5 “non-match” trials). Note that the familiarity of the written name (i.e. whether it belonged or not to the list of 20 objects in the previous task) did not predict the status (match/non-match) of the following object, since in both cases the probability of a match trial was 75%. The order of the trials was randomized in each block. Reaction times (RT) were recorded for each trial, and were used as a measure of visual priming.

Ten subjects in each group (test and control) participated in the experiment. They were seated in a dark room, 120 cm from a computer screen connected to a SGI (O2) workstation. They were first trained on 2 examples of simple scenes and the corresponding word-picture matching task blocks. The group of control subjects performed the experiment in reverse order, viewing the word-picture matching task before they were presented with each scene and had to report their contents. The reaction times from these subjects in the word-picture matching task were used as a reference (no priming). Furthermore, their performance in the report task (free and forced-choice recognition) allows us to determine if and how object recognition is facilitated by a prior single exposure to target objects.

To summarize, the test subjects were presented with a scene, asked to report (or guess) its contents, then performed the corresponding word-picture matching task; conversely, the control subjects were first asked to perform this word-picture matching task, then viewed the scene, and finally reported its contents. This sequence was repeated 10 times for each group.

On average, subjects explicitly report 2.28 objects per scene (corrected for guessing; see Methods and Table 1). This number is dependent upon the particular scene, and upon individual subjects. The number of reported objects varies between 1.7 and 3 for different scenes (averaged across subjects), and between 1.8 and 2.7 for different subjects (averaged across scenes).

The group of control subjects, who have been presented once with the target objects, performs reliably better (paired t-test, d.f.=9, t=5.55, p<.001). On average, these subjects report 3.31 objects per scene (corresponding to a 45% increase in recognition performance). This increase is paralleled by a corresponding increase of about 40% of the d’. Here again, performance varies across individual subjects (from 2.7 to 4.8) and scenes (from 2.1 to 4.2). Interestingly, the number of errors (R^-) is not higher for these control subjects than for the test subjects (0.21 errors per scene versus 0.33 errors per scene), indicating that this improvement truly reflects a facilitation of object recognition, and not simply a higher degree of confidence, or a change in report strategy.

The average (corrected) number of correct “guesses” for the group of test subjects is 2.28 (see Table 1). This number varies between 0.46 and 4.29 for individual scenes (averaged across subjects), and between 0 and 3.68 for individual subjects (averaged across scenes). For control subjects, the average number of correct guesses is 2.72, ranging from 1.1 to 4.5 for individual scenes and from 1.67 to 3.5 for individual subjects. Because control subjects had already reported more correct objects than test subjects in the free recognition task, they had fewer target and more distractor objects to choose from in the forced-choice recognition task. Taking into account these a priori probabilities for each group, this corresponds to a 36% increase in recognition probability for control subjects versus test subjects. Note that the d' measure also parallels this increase of about 40% (Table 1).

Figure 3 presents the combined results from the free recognition and forced-choice recognition tasks for each of the 10 scenes that were used as stimuli. The number of objects correctly “perceived” by test subjects (i.e., either explicitly reported, or guessed in the forced-choice recognition paradigm) varies between 2.3 and 6.1. After a single prior exposure to target objects, control subjects correctly perceive between 4.1 and 7.5 objects per scene.

It is not entirely clear how many of these objects have reached a conscious level of representation. A lower bound of around 4 objects can be recalled from visual short-term memory. This number is compatible with previous measurements of the capacity of short-term memory, generally believed to contain between 4 and 6 individual items (Sperling, 1960; Broadbent, 1975; Pylyshyn & Storm, 1988; Yantis 1992; Cowan 2001). Among the remaining objects, a certain number (and possibly all) might have accessed visual awareness, but without leaving a strong enough trace for later recall.

It is important to note that the number of objects perceived can depend on the particular scene presented, and probably on specific properties of each target object, such as its overall saliency. Among the factors that might determine whether an object will be reported or not, retinal eccentricity (that is, distance from fixation point) and size seem to be of particular importance. As compared to an average over all objects, the distance from fixation point is 15% smaller (t-test, d.f.=9, t=5.39, p<.001) for the objects reported by test subjects, and 11% smaller (t=4.25, p<.005) for those reported by control subjects. The size of the objects reported by test subjects is also 25% larger (t>10, p<.0001) than the average size of all objects, and the objects reported by control subjects are 22% larger (t>10, p<.0001). Finally, the objects that were guessed during forced-choice recognition show a significant (t>4, p≤.001) trend in the other direction, being 8% smaller for test and 10% smaller for control subjects than the average size for all objects. Conversely, the objects that were missed (i.e., neither explicitly reported, nor guessed during forced-choice recognition) are 17% smaller than average (t=7.15, p<.0001) for test subjects and 20% smaller (t>10, p<.0001) for controls, while their distance from fixation is roughly 6% higher than the average (although this number is only significant for test subjects, at the p<.001 level, t=5.32).

To summarize the results described so far, up to 7.5 objects from a complex natural scene can be identified in a single fixation, although 6 would be a more reliable (and conservative) estimate. Up to 4 of these objects can be consolidated into visual short-term memory and are reported by subjects with high confidence as having been “seen”. We now turn to the question of the remaining objects, those that were neither reported in free recognition nor guessed in forced-choice recognition (the “missed” objects). Whereas these objects obviously did not access a conscious level of representation, it is still possible that they could have reached some “high” level of representation, i.e., been recognized before being filtered out. In other words, does the observed limitation occur at the level of visual awareness or visual short-term memory, or is this limitation a consequence of a low-level selection, occurring earlier on in the visual system?

When a particular stimulus (hereafter called the “prime”) is presented to the visual system, even under conditions where it is not consciously perceived or remembered, it elicits a specific trace of neural activity, that can modify the processing of a subsequent repetition of the same stimulus (hereafter the “probe”). This phenomenon, known as visual priming, can take two distinct forms: either a stimulus-specific facilitation (Biederman & Cooper, 1991; Bar & Biederman, 1998, 1999), or a stimulus-specific impairment of subsequent visual processing (Neill, 1977; Tipper, 1985). While the former effect (positive priming) usually occurs for the objects that are selected by visual attention (or under conditions of low attentional load), the latter (negative priming) is generally thought to reflect the suppression of ignored objects during attentional selection (e.g. Tipper & Driver, 1988; Fox, 1995; Moore, 1996), although alternative theories have been proposed (Neill et al, 1992; Park & Kanwisher, 1994). Visual priming has been shown to be invariant to low-level picture manipulations (translation, reflection; Biederman & Cooper, 1991), and specific to higher-level properties of the stimulus, such as its semantic category (Allport et al, 1985; Tipper & Driver, 1988).

In order to determine whether objects of a particular group (e.g., missed objects) were perceived when the scene was presented, a block of 40 trials of a word-picture go/no-go matching task was performed after each entire report sequence (i.e. only once, after both free and forced-choice recognition were completed for a scene). The target objects from the previous scene were extracted from their background and presented in this task, among other trials containing “new” objects that had not been present in the scene. On average, the delay between the presentation of the whole scene and the presentation of one of these 40 word-picture matching trials was around 2 minutes, that is, well under the reported duration of visual priming (Bar & Biederman, 1998; DeSchepper & Treisman, 1996).

We reasoned that if an object was positively (resp. negatively) primed, the actual reaction time should be shorter (resp. longer) than the reaction time of a control subject, viewing the same object for the first time. In order to make reaction times comparable between the test and control subject groups, we normalized the RTs of each test subject so that their mean and standard deviation for the set of new objects would match the mean and standard deviation of RTs of control subjects on these new objects. We then compared the RT obtained for each target object (i.e., an object that was present in the original scene) to the median RT of control subjects on the same object (in other words, this median RT was considered as a reference). If there was no significant priming effect, on average 50% of the RTs would fall below this reference, and 50% above (since there could have been no priming for the control subjects group). This is what we observed for the set of objects that were guessed in the forced-choice recognition task: 49% of these objects elicited RTs below the reference, and this proportion was not significantly different from 50% (χ² test, 396 observations, d.f.=1, χ²=.09, p=0.8). On the other hand, 55% of the objects that were explicitly reported in the free recognition task elicited RTs that were shorter than the reference, suggesting a non-significant (261 observations, χ²=2.39, p=.1) positive priming effect, whereas 57.5% of the RTs on missed objects were longer than the reference, indicating a significant (343 observations, χ²=7.58, p=.005) negative priming effect for these objects. Whereas the former effect (positive priming) can be naturally expected to occur for objects that the subjects explicitly reported (because these objects have obviously been identified), the latter effect is more surprising. Indeed, when a subject reliably fails to report certain objects from the scene, it would be rather intuitive to conclude that these objects were not perceived. However, the negative priming effect suggests that these objects were in fact represented in the visual system, but that this representation was eventually suppressed.

This negative priming effect is also significant when comparing mean RT (paired t-test, t(9)=3.27, p=.01) and error rate (t(9)=3, p=.015) between the set of missed objects and the set of new objects (Figure 4). These latter effects are not significant (t(9)=2.2, p=.055 for RTs; t(9)=1.48, p=.17 for error rates) for the group of control subjects, indicating again that the priming effects are indeed due to the prior perception of target objects in the scene. Additionally, the magnitude of this negative priming (calculated as the difference between error rates for “missed” vs. “new” objects) was stronger for test than control subjects (t(9)=2.96, p=.016). This effect is in fact strong enough (and in particular, stronger than the positive priming observed for explicitly reported objects) to be observed when we average over the entire set of target objects (whether explicitly reported, guessed, or missed): the overall error rate in the word-picture matching task is significantly (paired t-test, t(9)=2.4, p=.04) higher for target objects (6.2%) than for “new” objects that do not belong to the original scenes (4.0%). Once again, this comparison is not significant for control subjects (t(9)=1.13, p=.29).

This observation is particularly important because it rules out alternative explanations based on the correlational nature of our analysis. Indeed, our subjects select by their performance which objects belong to the class of reported, guessed or missed objects for which priming will later be tested. One could therefore argue that our analysis only reveals correlations between bad performance in both the report task and the reaction time task. However, this is not true in our case because the group of test subjects actually performs worse on the overall set of target objects, independent of the correlation among images drawn from these three categories.

One could also argue that subjects could recognize written names as part of the previous list, and use this information to bias their response in the word-picture matching task. In that case, the same “negative priming” should also be observed for “distractor” names, those that were actually presented in the previous list but not in the scene (indeed, from the subject’s point of view, there is no way to tell these objects from the “missed” objects). However, reaction times obtained for these distractor objects in the priming task are significantly shorter (paired t-test, t(9)=2.55, p=.03) than the ones for “missed” objects, and the error rates significantly lower (paired t-test, t(9)=2.49, p=.035). These RTs and error rates for “distractor” objects are not significantly different (t(9)=1.37, p=.2 for RTs; t(9)=.12, p=.9) from those obtained for “new” objects . In other words, the fact that a name is recognized as part of the previous list, but not part of the scene, cannot by itself account for the observed negative priming.

Yet another possible interpretation of this result could be that the difference between test and control subjects arises from a form of interference between the two tasks. For example, when presented with a missed object in the word-picture matching task, a test subject could realize that he (or she) failed to report this object as part of the previous scene. This in turn might interfere with the generation of the motor response. There could be no such effect for control subjects, who have not yet viewed the scene at the time of the word-picture matching task. However, because such an error judgment would require not only the identification of the object, but also access to the memory of responses from the previous task, one would expect it to mostly affect the longest RTs, i.e., those for which the subject has enough time to make this sort of judgment. In contrast, the shortest RTs would most probably reflect an automatic object recognition process. We find that the probability of generating a motor response for a missed object before 400 ms post-stimulus is already significantly (paired t-test, d.f.=9, t=4.15, p<.005) smaller than the probability of responding to a new object (15% in the former case versus 26% in the latter), suggesting that object recognition itself, and not (only) later cognitive judgments, is impaired in the case of missed objects. In other words, this impairment is certainly a true negative priming effect, indicating that missed objects from the scene have indeed accessed a high level of representation, even if the resulting neural activity was too weak, or did not last long enough, to allow these objects to be consciously reported.

When a novel natural visual scene is presented to our retinae, we almost immediately and automatically extract its overall meaning, its “gist” (Wolfe, 1998). In addition, a certain number of individual objects usually complement this representation. When asked to describe what these objects are, observers will usually report 2 or 3 objects with confidence. If they have been exposed to the target objects shortly before, they will most likely be able to report around 4 objects. Even without full confidence, if forced to choose from a list of possible objects, observers can select the correct objects well above chance. This brings the total number of perceived objects up to 6, although some of them might not be explicitly remembered. Prior exposure to the target objects can even increase this total to almost 8 objects. How many of these objects are represented in visual awareness remains unclear, but this number is certainly greater than 4, since in many cases 4 objects or more are explicitly remembered by the observer. Finally, a subject will completely fail to report between 2 and 4 out of 10 objects, depending on the particular scene. Note that, for such a failure to occur, the subject must judge other distractor objects more likely to have been present in the scene. In other words, the observer must be confident to a certain degree that they have not perceived the target objects in the scene. However, when viewing these same objects in a following task, the subject will tend to respond slower and make more mistakes than for a set of completely new objects (negative priming). Therefore, these objects must have been processed to a certain extent by the visual system, before being filtered out.

This sequence of selection among different levels of representation can be better understood in terms of the underlying neural mechanisms. The early representation that is mediated by neural populations in striate and early extrastriate visual areas (i.e., V1, V2...) most probably describes the scene in a spatially uniform way, except for an enhanced resolution towards the center of the visual field, and a degradation towards the periphery, due to retinal and cortical magnification factors. The competition taking place between neurons at this level is unlikely to account for object-based selection, since the receptive fields will in general be too small, and the selectivities too coarse, to allow the representation of individual objects. In consequence, most if not all of the objects present in the visual scene will be represented (at least partly and/or temporarily) at the level of V4 and in its postsynaptic target areas in the inferior temporal cortex (IT) and in the equivalent regions of the human temporal lobe (e.g., fusiform gyrus), where neural populations as well as individual neurons have been found to code specifically for certain object categories such as faces, houses or chairs (Allison et al, 1999; Aguirre et al, 1998; Epstein and Kanwisher, 1998; Ishai et al, 1999; Chao et al, 1999). A recent electrophysiological study in the macaque by Sheinberg and Logothetis (2001) indicates that objects in natural cluttered scenes such as the ones used here can activate selective neurons in infero-temporal cortex in a manner very similar to an isolated presentation of the same objects. There is supportive experimental evidence that some degree of object-based competition within and between neurons takes place at this level. For example, 2 objects falling inside the same neuronal receptive field are known to compete for attentional resources in order to dominate the neuronal response (Moran & Desimone, 1985; Desimone & Duncan, 1995; Reynolds et al, 1999). As a result of this competition, a certain number of objects (around 4.5 or more in light of the present results) will be selected to receive attentional resources, while the representation of the remaining objects (between 2 and 4 in a scene containing 10 objects) will be actively inhibited, so as to avoid interference.

Neurons coding for “ignored” objects will not participate in the following stages of this sequence of processing. However, because they are not passively but actively suppressed or inhibited (either in IT or its post-synaptic targets), the neural activity resulting from a subsequent presentation of the same object will first need to overcome the long-lasting effects of this suppression before the neurons can be made to respond again. This might constitute the neural basis of the negative priming phenomenon (Tipper, 1985). What is remarkable here from a biophysical point of view is that a single exposure of an image, with an associated neural activity most likely lasting less than one second in duration (Kreiman et al., 2000) must give rise to some sort of long-lasting synaptic effect that can lead to a less effective neural representation many minutes later when the same image is flashed on again.

Similarly, a single prior presentation of a target object in isolation (such as when the control subjects performed the word-picture matching task before viewing the scene) will trigger some sort of facilitation in the neurons coding specifically for this object, that can last long enough to enhance later selection of this object, when presented in the context of the scene. This corresponds to a positive priming effect. The number of selected objects can be enhanced in such a way (approximately from 4.5 to more than 6), suggesting that the capacity limitation at this level is not a “hard” limitation, but one that can be overcome in particular situations.

It is striking to notice that the negative priming effect obtained here can be much stronger than the corresponding positive priming observed for selected objects. Indeed, the “net” effect observed on all target objects (whether correctly reported, guessed or rejected) is a significant negative priming one. In contrast, most psychophysical studies more readily appear to observe positive priming (e.g. Biederman & Cooper, 1991; Bar & Biederman, 1998). This discrepancy might arise from the fact that in our case, the “prime” stimulus is not presented in isolation, but in a cluttered scene containing many objects. This might force the visual system to activate attentional selection mechanisms, inhibiting the representation of certain objects which would otherwise (if presented in isolation) receive full attentional resources. By comparison, other studies do not in general require the visual system to actively select among many different competing stimuli.

What is the fate of the objects whose representation survived the competition at this stage? How do they finally give rise to an explicit percept or a vivid memory? The first obvious conclusion is that the activity of the neural populations representing these objects will last longer or be stronger in some manner than for “ignored” objects, because in the case of selected objects, neural activity has been facilitated, while it was suppressed for the other objects. It is possible that a prolonged duration of neural activity in the inferior temporal cortex could be a necessary condition for the neural correlates of visual consciousness (Crick & Koch, 1990; Subramaniam et al, 2000; Bar et al, 2001). If this hypothesis were true, our results suggest that a possible role for such an integration period could be to leave enough time for competition between stimuli to be resolved, before the selected objects can enter visual awareness. Alternatively, this information could be transmitted to higher-level neuronal areas, such as parahippocampal structures or prefontal areas, which receive direct connections from inferotemporal cortex and whose neurons are selective to visual categories (Distler et al, 1993; Suzuki, 1996; Miyashita & Hayashi, 2000; Kreiman et al., 2000). Such areas could then mediate awareness and/or working memory of the objects selected by visual attention (Suzuki, 1999; Crick & Koch, 1998). The present results however do not allow to differentiate between these two hypotheses.

This research was supported by grants from the NSF-sponsored Engineering Research Center at Caltech, the National Institutes of Health and the W.M. Keck Foundation Fund for Discovery in Basic Medical Research. R.V. is supported by a fellowship from the California Institute of Technology. The initial idea for this experiment emerged from discussions with Francis Crick and Nikos Logothetis. The authors wish to thank L. Reddy and P. Wilken, as well as Dan Simons and one anonymous reviewer for useful comments on the manuscript. Commercial relationships: None.