In the real world, survival is contingent upon the rapid, accurate recognition of objects. Despite extensive research in both neuroscience and computer science, how this task is accomplished remains a mystery.

One view is that object recognition constitutes a chicken-egg type problem between lower and higher levels of image analysis. The low-level shape features that are useful for identifying an object - edges, contours, surface curvature and the like - are typically ambiguous in natural scenes, so they cannot be computed directly based on a local analysis of the image. Rather, they must be inferred based on global context and higher-level knowledge. However, the global context itself will not be clear until there is some degree of certainty about the presence of low-level shape features. A number of theorists have thus argued that recognition depends on information circulating through cortico-cortical feedback loops in order to disambiguate representations at both lower and higher levels in parallel (Mumford, 1994; Ullman, 1996; Lewicki & Sejnowski, 1997; Rao & Ballard, 1999; Lee & Mumford, In Press).

However, a large body of both neurophysiological and psychophysical data suggests that recognition is performed so rapidly that it must be accomplished in one feedforward sweep of activity propagated through the visual system (Fukushima, 1980; Mel, 1997; Riesenhuber & Poggio, 1999; VanRullen & Thorpe, 2001a; VanRullen & Thorpe, 2002). For example, rapid serial visual presentation (RSVP) techniques (Intraub, 1999) and other masking paradigms (Breitmeyer, 1984) reveal behavioral selectivity to images presented for 100 ms or less. Single unit recordings in macaque superior temporal sulcus show that face-selective activity can be elicited by masked images presented for as little as 14 ms (Keysers, Xiao, Földiák & Perrett, 2001) and human functional magnetic resonance imaging (fMRI) studies have shown activations in object recognition areas to be correlated with masked image presentations as brief as 40 ms (Grill-Spector, Kushnir, Hendler & Malach, 2000). This sort of result is difficult to reconcile with models requiring feedback because the original image should be replaced by a subsequent one by the time signals from higher level areas are fed back to lower levels.

Further, it has been argued based on known biophysical and anatomical constraints (e.g. speed of spike propagation, number of cortical processing stages) that the earliest responses in macaque higher-level, object recognition areas must be based on one or at most two spikes per neuron in the pathways leading up to them (Thorpe & Imbert, 1989). This hypothesis is supported by latencies of about 100 ms reported for neurons in macaque inferotemporal cortex (IT) (Nowak & Bullier, 1997). Response latencies and reaction times in recognition tasks are a bit later in humans than macaques (Fabre-Thorpe, Richard & Thorpe, 1998), so that in humans, intracranial EEG studies show the earliest response latencies in the facial recognition areas of the fusiform gyrus to be 130-140 ms (Allison, Puce, Spencer & McCarthy, 1999). This 30-40 ms delay is presumably due to the increased conduction times associated with larger head size.

A study of the timecourse of object recognition in humans has revealed a component in the electroencephalogram (EEG) arising approximately 150 ms after image presentation that is related to the detection of objects in complex natural scenes (Thorpe, Fize & Marlot, 1996). Importantly, this signal is presentation-locked, meaning that the timing of its onset is constant across trials and is not correlated with the subsequent reaction time. The latency of this component is also constant for both novel and previously learned images (Fabre-Thorpe, Delorme, Marlot & Thorpe, 2001). Thus, EEG differences seen in object recognition tasks are nearly as fast as the earliest known responses in higher-level areas. These results seem to leave little or no opportunity for recurrent activity in cortical feedback loops to have an effect on the earliest responses in IT, and suggest that object recognition is performed in a rapid, feedforward manner without regard to the particular object present, the context, or familiarity.

One difficulty in using methods such as EEG or other physiological signals to measure the timecourse of object recognition arises from the fact that various visual features are not equally probable across image categories. For example, certain spatial frequencies, textures, colors, and simple spatial patterns may be associated with a given category of images without necessarily being diagnostic for the objects themselves. Low-level features such as these could be detected in parallel and might give rise to a purely stimulus-related pre-recognition EEG signal that varies across categories of images. This sort of signal could be easily confused with target-related signals that depend upon the completion of object recognition. As such, when comparing the average EEG waveforms obtained from different categories of images, any differences could be due to non-specific visual processing as well as the task relevance of the stimuli. Interestingly, experiments that control for featural differences by mixing together images from different categories (VanRullen & Thorpe, 2001b) also show differences in the EEG which rise to significance around 150 ms, but the topographical distribution and overall timecourse of these difference signals are not the same as those obtained when different categories are compared. This raises the question as to whether there are different underlying neural processes at work in the two cases.

Here we address this question by examining the nature of the EEG signals obtained in two types of categorization tasks. In the first task, subjects categorize stimuli into animal and nonanimal categories, so that there are potential low-level feature differences between target and nontarget images, similar to previous experiments (Thorpe, Fize & Marlot, 1996; Fabre-Thorpe et al., 1998; Fabre-Thorpe et al., 2001). In the second task, the target categories change from trial to trial and the images assigned to these categories are arranged so as to ensure that the pool of target and nontarget images is identical across subjects. Thus, any EEG differences between targets and nontargets in the second task must be contingent upon recognition rather than due to image features. We show that the earliest differences reliably contingent upon recognition appear between 152 and 300 ms after presentation and have an onset which covaries with the subsequent reaction time. Earlier differences, which appear by 137 ms and are presentation-locked, only occur when comparing signals evoked by images with featural differences, suggesting that they are due to differences in early visual processing rather than the result of recognition. These results suggest that the timecourse of object recognition, while still quite rapid, is not as fast as previously thought. We discuss the implications of these results for models of recognition.

A total of thirty-nine adult subjects (10 males and 29 females, aged 18 to 41 years) participated in the four experiments reported in this study. Seven subjects participated in Experiment 1, twelve in Experiment 2, twelve in Experiment 3, and eight in Experiment 4. All participants had normal or corrected to normal vision. All participants gave informed consent and the UC Davis Human Subjects IRB approved all studies.

Sample images used in the four experiments described here are shown below (Figure 1). Images used in Experiment 1 were taken from a commercially available collection (Corel Photo CDs, out of production) and included images of animals, flowers, and other outdoor scenes. The images used in Experiments 2 and 4 were created by digitally centering a cutout image of an object (Hemera Photo-Objects) against one of thirty artificially created backgrounds. The backgrounds were created to match the images from Experiment 1 in terms of both their spatial and color second order statistics (pairwise correlations) by generating 1/f noise for each of the principal components in color space and scaling each by the square root of the variance along each component. Experiment 2 also used these background images without a superimposed object (“Background-Only”). Images from Experiment 3 were collected from the internet. The following links provide additional, full-sized examples of the images used in Experiment 1, Experiments 2 and 4, and Experiment 3.

Two types of experiments were used in this study, a “single-category” experiment and three “cued-target” experiments.

In the single-category case, subjects were given one category (“animal”) which served as the target for the duration of the experiment. Images were presented for 40 ms and were not masked. There was a delay of 3300-3700 ms between images (Figure 2). The timing of the image sequence did not depend on the subject’s responses. Subjects were asked to make button-press responses as quickly as possible and to delay their blinks for about 2 seconds after each image presentation.

In the cued-target case, a word (target cue) was presented on the screen before each image to inform subjects of the target category for that trial. The target cue remained on the screen until the subject was ready to go and performed a button press to initiate the trial. After an additional 500-700 ms delay, the image was presented for 40 ms, not masked. The next target cue appeared 1700 ms after the previous image (Figure 2). Subjects were asked to make button-press responses as quickly as possible and to delay their blinks after each image until the subsequent target cue appeared.

All images were centrally presented on a CRT monitor. Image presentation was controlled by a PC running the software Presentation (NeuroBehavioral Systems). Viewing distance was 75 cm except for Experiment 3 where the viewing distance was 65 cm.

Experiment 1: Single-category experiment. This experiment had two conditions: “forced choice”, where subjects pressed one button if the image contained a target and another button if the image contained no target, and “go/no-go”, where subjects pressed one button if the image contained a target and did nothing if the image contained no target. Counterbalances were made across subjects for button presses and for given image appearance in the forced choice or go/no-go condition. Six subjects viewed 500 images in the forced choice condition and 500 images in the go/no-go condition. One subject viewed 1000 images in the forced choice condition only. Targets and nontargets appeared equally in both forced choice and go/no-go conditions. Results from nontarget images containing flowers were analyzed separately and are not included in this study.

Experiment 2: Cued-target, 1/f Backgrounds experiment. Target cues were all formulated for entry level classification (Rosch, Mervis, Gray, Johnson & Boyes-Braem, 1976; Jolicoeur, Gluck & Kosslyn, 1984). Subjects viewed 990 images and made a forced choice response. Target objects, nontarget objects, and background-only images (which required a ‘nontarget’ response) appeared equally frequently. Counterbalances were made across subjects for button presses and target status of individual images (except background-only images). Results from background-only images were analyzed separately and are not included in this study.

Experiment 3: Cued-target, Natural Backgrounds experiment. Target cues were presented at two levels of abstraction, entry level and superordinate level. Cues of both levels of abstraction were randomly interleaved. Subjects viewed a total of 1000 images. Superordinate level targets, superordinate level nontargets, entry level targets, and entry level nontargets appeared equally frequently. Because some superordinate categories were less intuitive than others, subjects were briefed on the superordinate categories before the experiment. Counterbalances were made across subjects for target status and required level of classification of individual images.

Experiment 4: Cued-target, Motor Related Control experiment. 1/f background images used in Experiment 4 all contained objects; no background-only images were used. This experiment had two conditions, Respond-Target and Respond-Nontarget. Both conditions were go/no-go and consisted of 500 images. Targets and nontargets appeared equally frequently. The Respond-Target condition was run in its entirety before the Respond-Nontarget condition. Subjects were not informed of the upcoming change in condition until it occurred. Counterbalances were made across subjects for target status and response mode condition (forced choice or go/no-go) of individual images.

The following links provide full lists of the target cues used in Experiment 2 ( entry level) and Experiment 3 (entry level and superordinate level), along with some associated data including number of times used, and average reaction time. The target cues used in Experiment 4 were identical to those in Experiment 2, and are not included.

Subjects were fitted with a 19-channel electrode cap (Electro-Cap International, Eaton, Ohio) and were prepared for EEG recording according to standard techniques. Recorded channels (FP1, FP2, F7, F3, FZ, F4, F8, T7, C3, CZ, C4, T8, P7, P3, PZ, P4, P8, O1, O2) were selected from the International 10-20 set of electrode positions (American Electroencephalographic Society, 1994). In addition to the cap electrodes, facial electrodes were attached to record horizontal and vertical electrooculogram (EOG). All recordings were referenced to the right mastoid. Subjects performed the experiment in a darkened, sound-dampened, electrically shielded booth. EEG signals were amplified (SA Instrumentation, San Diego) with a high-pass cutoff of 100 Hz and a low-pass cutoff of 0.01 Hz, then sent through an analog-to-digital converter before being recorded at 256 samples/sec on a PC running Digitize (Arthur Jones, LBNL).

Raw data were normalized, artifact rejected, and analyzed using Matlab software developed in-house. Software for the display of scalp topographies was developed by Scott Makeig (Salk Institute, San Diego). Data were artifact rejected on a trial-by-trial basis for eyeblink and on a channel-by-channel basis for drift, blocking and excessive alpha wave.

To create the ERPimages, individual correct-response EEG trials were assigned to 3.9 ms (one sample) wide bins on the basis of reaction time (RT) and an average was calculated for each bin. Bins with RTs between 300 and 600 ms were sorted by RT. Each averaged bin was then vertically expanded, with an expansion factor proportional to the number of EEG trials underlying the average. In the case of the difference ERPimages, the averaged nontarget bin was subtracted from the averaged target bin to create one difference wave at each RT. These difference waves were then sorted by RT and expanded, with an expansion factor proportional to the lesser of the number of EEG trials, target or nontarget, underlying the original RT bins before subtraction. All ERPimages were then smoothed vertically with a Gaussian filter having a standard deviation of 1/50 the height (number of expanded trials) of the plot.

To perform the spatial frequency image analysis, the center 512x512 pixel region of each image was extracted and windowed with a 1-cycle cosine function (to remove boundary artifacts). The power spectrum of each windowed image was calculated by the discrete Fourier transform, and then averaged over all images for a particular class. The average power spectrum was then bandpass filtered into 9 one-octave bands spaced by 1/2 octave, at 8 orientations each. This resulted in an 8x9 element array containing the power in each band, which was then interpolated in 2D with a cubic spline.

In the "single-category" paradigm, target and nontarget categories were composed of different sets of images – “animal” images and “nature” images. These two sets of images could potentially differ in terms of low-level image statistics. For instance, textures or shapes associated with fur, eyes and other animal parts, the horizon, sky, trees, land terrain, water and other image features are likely to be unevenly distributed across the animal and nature categories. This could potentially give rise to measurable neural processing differences unrelated to object recognition in striate or early extrastriate cortex.

While there are a host of potential differences in low-level features and textures between the two sets of images, differences in the power spectrum are the easiest to demonstrate. A spatial-frequency analysis of the animal and nature images (excluding images containing flowers) used in our single-category experiment reveals substantial differences in the power spectrum – namely, scenes containing animals have approximately equal power in all directions whereas those without animals have much more power in the vertical spatial frequencies (Figure 3). Similar differences in the average power spectra between major categories of images have been shown previously (Oliva & Torralba, 2001).

The design of the "cued-target" paradigm ensured that each image appeared equally in both the target and nontarget conditions, across subjects. Since the set of target images and the set of nontarget images were identical, there were no differences in low-level features, spatial frequency or otherwise, between the images. The only difference between target and nontarget images was in their conceptual status, not in their featural content. Two types of images were used in the cued-target experiments, one with cutout photos of objects digitally placed atop an artificial 1/f background (“1/f BG”), and another with objects in their natural context (Figure 1).

The trial-averaged EEG waveform – known as the event-related potential (ERP) – was computed separately for target and nontarget stimuli. These two ERPs were directly compared by subtracting the nontarget ERP from the target ERP. For both single-category and cued-target tasks, the resulting difference waveform shows an early positive divergence centered around frontal and central electrodes.

The grand average ERP waveforms for the forced choice, single-category task at electrode FZ are plotted in Figure 4a. The first differences between targets and nontargets arise in the first positive deflection, which begins about 100 ms after image onset. The ERP difference waveform for this task (Figure 4f, black) has two separate peaks, an early peak rising before 150 ms and a late peak starting around 300 ms. Both peaks have a maximum amplitude of about 4 μV. To determine when the ERPs recorded for targets and nontargets first significantly differ, we used a two-sample t-test for difference of means. In order to avoid counting spurious deflections from baseline, we considered the onset of difference to be the first sample point for which the calculated p-value was less than 0.01 and for which the following 9 samples (10 consecutive points) also reached the same criterion. By this definition, targets and nontargets differed as early as 137 ms after presentation. The timecourse of this difference signal is similar, but slightly earlier than that reported in previous single-category studies (Thorpe, Fize & Marlot, 1996).

Because previous single-category studies were done with go/no-go responses instead of forced choice responses, we also ran the single-category experiment in a go/no-go response mode. The waveforms for the go/no-go experiment are shown in Figure 4b and the corresponding difference between targets and nontargets in Figure 4f, dark blue. The ERPs and difference waveforms are quite similar, especially in the first 300 ms, to those from the forced choice version of the single-category task. Here, targets and nontargets first differed 148 ms after presentation.

The ERPs for the cued-target task using 1/f backgrounds (Figure 4c) show that the first positive deflection has a much lower amplitude than in the single-category condition. Moreover, while targets and nontargets are clearly different by the peak of the first positivity in the single-category task, they are only beginning to differ at the corresponding time in the cued-target task. The difference waveform obtained in this task (Figure 4f, red trace) rises much more slowly (first significance 187 ms) than those in the single-category task, and has only one clear peak with a much higher late maximal amplitude of 10 μV. This positive, posterior signal during target detection is similar in latency, though not in amplitude, to previous reports of target-related signals (VanRullen & Thorpe, 2001b), and bears similarity to those found in many other target detection studies (Sutton, Braren, Zubin & John, 1965; Picton, 1992).

Aside from the fundamental change in task, there were two other changes between the single-category experiment and the 1/f BG cued-target experiment which might have affected the nature of the difference waveforms. The first was that in the 1/f background cued-target experiment, pictures of cutout objects appeared against artificial backgrounds rather than in their natural context. To determine if this caused the changes in the difference waveforms, a subsequent cued-target experiment was run with objects in their natural context (see Figure 1). In this case, both the ERPs (Figure 4d) and the difference waveforms (Figure 4f, green) resemble the 1/f BG cued-target task. They feature a single peak with a maximal amplitude of 7.75 μV and the difference is first statistically significant 172 ms after presentation. Despite the disparity of the image backgrounds in the two versions of the cued-target task no qualitative differences are evident between the two, suggesting that the signals that we are recording are relatively insensitive to the nature of the image background.

A second change between the single-category and 1/f BG cued-target experiments was the level of abstraction of the requested categorization. In the single-category experiment, the target category was at a superordinate level of abstraction (“animal”), whereas in the 1/f BG cued-target experiment, the target categories were at a lower, entry level of abstraction (e.g. “cat”, “chair”) (Rosch, et al. 1976, Jolicoeur, Gluck & Kosslyn, 1984). In the natural image cued-target experiment, participants also categorized some images at the superordinate level (e.g. “animal”, “furniture”) to match the categorizations made in the single-category task. Again, the ERPs for these trials resembled those in the 1/f BG cued-target condition (Figure 4e). The difference waveforms (Figure 4f, light blue) maintain the single peak typical of cued-target tasks, and have an even slower onset (first significance 207 ms). However, the maximal amplitude of the difference drops to 3.5 μV, similar to that of the single-category difference.

Scalp topographies reveal dramatic differences between the distribution of these potentials across both space and time. The scalp topography of the single-category (forced choice) difference captures the two peaks seen in the ERP difference waveform (Figure 5a). The first peak consists of a frontal positivity coupled with a strong bilateral occipito-temporal negativity, while the second peak is a central positivity and lacks the negative occipito-temporal component. The single peak of the 1/f BG cued-target experiment (Figure 5b) has a time course and scalp distribution which closely resemble those of the second peak of the single-category experiment, and lacks the bilateral occipito-temporal negativity characteristic of the first peak.

To further investigate the differences between the single-category and cued-target conditions, we utilized a modified version of the ERPimage (Jung, Makeig, Westerfield, Townsend, Courchesne & Sejnowski, 1999; Makeig, Westerfield, Jung, Enghoff, Townsend, Courchesne & Sejnowski, 2002). The ERPimage shows the EEG waveforms from all trials sorted by reaction time, and thus clearly reveals the presentation-locked and reaction-time related components contributing to the ERP. Individual EEG trials were separated by target status, and each group was sorted and binned by reaction time to create an ERPimage. We then compared the binned ERPimages for targets and nontargets by subtracting the latter from the former. This difference ERPimage reveals the trial-by-trial differences in the EEG for trials having the same reaction time (see methods for more detail on ERPimage creation). ERPimages were created for both the single-category (forced choice) experiment and the 1/f BG cued-target experiment.

The target and nontarget ERPimages at electrode FZ (Figures 6a, 6b) show presentation-locked activity as vertical structure. Activity that is correlated with the reaction time has a diagonal structure, often closely following the displayed reaction time (RT) curve (solid black line).

The difference ERPimage for the single-category task (Figure 6c, left) shows that the earliest differences arise from presentation-locked components. The dashed vertical line indicates the onset of statistical significance for these trials (see Figure 4) and the target minus nontarget positivity is evident at the onset of statistical significance regardless of reaction time. Difference ERPs (Figure 6d, left) created for fast-RT trials (300-400 ms response) and slow-RT trials (400-500 ms response) do not differ greatly in amplitude or onset time in the single-category task.

The difference ERPimage for the 1/f BG cued-target task (Figure 6c, right) shows that the earliest differences are not only slower but are also dependent on the reaction time for a given trial rather than presentation-locked. The target minus nontarget differences on fast-RT trials rise more quickly than on slow-RT trials. In contrast to the single-category task where all trials appeared to contribute equally to the statistical significance, in the cued-target task only the fastest trials appear to be responsible for the earliest statistically significant differences. In fact, in the slowest trials shown in Figure 6c (right), the difference does not appear to arise before 300 ms. Difference ERPs for fast- and slow-RT trials (Figure 6d, right) also reflect the RT-dependent component of this difference.

We calculated the time of first statistical significance for fast- and slow-RT trials for both the single-category and cued-target tasks shown in Figure 6d. Using the same statistical criterion as before (p < 0.01, ten consecutive samples) we see a time lag for first significance in the cued-target task, with fast-RT trials first differing at 187 ms and the slow-RT trials first differing at 219 ms, a delay of 32 ms. If we adopt a stricter statistical criterion (p < 10^-4, five consecutive samples), this delay extends to 63 ms (fast-RT = 195 ms, slow-RT = 258 ms). There was a small 11 ms delay in the single-category results under the original criterion (fast-RT = 137 ms, slow-RT = 148 ms) that disappeared using the stricter criterion (fast-RT = slow-RT = 152 ms).

The onsets of the earliest differences seen in the cued-target case are correlated with the subsequent reaction time, while the onsets of those in the single-category case are, if anything, only marginally correlated with RT. The development of RT-dependence, taken together with changes in overall latency, number of peaks, peak amplitude, and scalp topography, very strongly suggest that the earliest ERP differences in the cued-target case are not simply a delayed form of the same signal seen in the single-category case. Since the fast, presentation-locked differences are seen only in the single-category case - where the images contain demonstrable low-level feature differences - but not in the balanced cued-target case, it seems likely that they are due to differences in early visual processing rather than the completion of object recognition.

In order to determine if the target-related signal we found could be due to motor effects (motor planning, motor execution, motor countermand) rather than target awareness, we performed a task swapping experiment. Subjects were given a cued-target task with 1/f BG images, but asked to respond in a go/no-go fashion only to targets for the first half of the experiment, then only to nontargets for the second half. Subjects were not informed of the impending reversal in response mode until the time came to change. The target minus nontarget difference in the first half of the experiment corresponded to a response minus no-response condition. In the second half of the experiment, the target minus nontarget difference corresponded to the reverse case: a no-response minus response condition. If the signals we see are due to differences in motor preparation or output between target and nontarget responses, we should expect them to reverse in sign and follow the motor response rather than the target status of the image in the second half of the experiment. The ERP waveforms and difference waveforms (Figure 7) are similar to those for cued-target conditions in duration, amplitude, and scalp topography. They clearly do not reverse sign as would be expected for a motor-related signal suggesting that they are not due to motor effects but rather to the conceptual status of the image as target or nontarget.

In the respond-target condition, the difference becomes significant 152 ms after presentation. This is the earliest difference seen in any cued-target task in this study. The difference waveforms rise somewhat later during the more difficult respond-nontarget condition, becoming significant at 176 ms.

One possible concern in comparing results from the single-category and cued-target tasks is that the cued-target task, with its switching targets, may be inherently more difficult that the single-category task. A comparison of the reaction times and accuracy for targets in the single-category task (Table 1, 427 ms, 94.1%) and the corresponding superordinate categorizations in the cued-target task (520 ms, 86.9%) might suggest that the cued-target response mode is more difficult than the single-category response mode. While this could be due to the switching of target category, another possibility is that the required categorizations are more difficult in the cued-target case. However, if we look only at cued-target trials for which the target cue was “animal”, target responses (438 ms, 95.8%) are slightly more accurate and only 11 ms slower than those seen in the single-category experiment. This suggests that the cued-target paradigm per se is not significantly more difficult than the single-category paradigm, but rather that the single-category target of “animal”, though nominally superordinate, is easier than the other superordinate categorizations it was grouped with in Experiment 3. In addition, when the same images are categorized at both superordinate and entry levels, we find a 56 ms delay for superordinate categorization, similar to reports of a 50 ms delay in a similar task using line drawings (Rosch, et al., 1976). This suggests that the superordinate categories we chose were not unusually difficult ones.

Two other observations merit comment. First, go/no-go categorizations appear to be easier than forced choice categorizations, as we would expect from the easier mapping of task to motor output. This change in difficulty seems to be manifested mostly in reaction time in the single-category case (399 ms vs. 427 ms) and in accuracy in the cued-target case (97.3% vs. 90.1%). Second, there is a difference between performance in the entry level cued-target task for natural images (464 ms, 93.9%) and images with artificial backgrounds (432 ms, 90.1%). This might be accounted for by a speed/accuracy tradeoff, but certain characteristics of the natural images (potential ‘distractor’ objects, object not always centered at fixation, etc.) may also play a part in the slower reaction times in this case.

We have shown that the earliest components in the EEG that are correlated with recognition have an onset that varies between 150 and 300 ms and is correlated with the subsequent reaction time. These results stand in stark contrast to previous EEG studies showing early, fixed latency components correlated with object recognition arising around 150 ms (Thorpe, Fize & Marlot, 1996; Fabre-Thorpe et al., 1998; Fabre-Thorpe et al., 2001). These previous studies are equivalent to our "single-category" paradigm, in which we show that the early, fixed-latency components (in our case, arising ~135 ms) are most likely due to low-level feature differences in the images. We therefore conclude that the early 135-150 ms onset, fixed-latency component in the EEG corresponds to differences in early visual processing among images, and that the neural signatures actually corresponding to object recognition occur later and with variable latency.

These studies also demonstrate that a population-wide p-value of the ERP is not necessarily representative of the average signal latency. While the population-wide p-value is useful for determining the latency of a stimulus-locked signal, as seen in the single-category experiment, it appears from our data to be highly biased by the fastest reaction time trials when estimating the latency of an RT-dependent signal. As such, it is difficult to draw conclusions about the correspondence of ERP differences across experiments on the basis of p -value latencies alone. The ERPimage (Jung, et al., 1999) has the advantage of showing both stimulus-locked and RT-dependent components simultaneously, making it more valuable than a simple global p-value for the analysis and interpretation of EEG components that may not have a constant latency.

It should be noted that although we have attributed the earliest differences in neural activity in the single-category case to low-level feature differences, this does not preclude these features playing a role in recognition. For instance, Ullman has argued (Ullman, Vidal-Naquet, & Sali, 2002) that pictorial image features of “intermediate complexity” are the most useful features for the discrimination of objects. In addition, it has been shown that natural images can be classified into animal and nonanimal categories at a success rate of 80% using nothing but measures of global image statistics such as the power spectrum (Torralba & Oliva, In Press), suggesting that some low-level features are fairly consistent within categories of images. Thus, the visual system might be able to build a good template of the features associated with a category and use this template to make preemptive categorizations with reasonable accuracy. This sort of template could range from a simple list of features which are expected to be present to a complex mental image of the object in question. If such a template is created, the early differences we ascribe to low-level visual processing may in fact represent an interaction between the bottom-up processing of the image and a top-down target template.

The results from this study can be interpreted to support a top-down template model if it is assumed that in the cued-target case, subjects were not able to adequately create the necessary template. The subjects were forced to change their target for every image in the cued-target experiments, but had only one target for the duration of the single-category experiment. Although the cued-target trials were self-paced (i.e., the subjects were not rushed into a subsequent image presentation before they deemed themselves ready), it is possible that top-down template priming requires a longer time - on the order of tens to hundreds of seconds - to establish. Alternatively, in the single-category case the visual system might learn, over the course of many animal and nature scenes, its own template of features which are commonly associated with each category. This sort of learning would be difficult in the cued-target case because of the large number of different target categories and the low number and non-consecutive presentations of each. If either of these scenarios holds, it is possible that the earliest target minus nontarget differences in single-category tasks represent a special form of recognition based on low-level features of the images rather than simply processing of the image features themselves. However, this sort of feature-based recognition would require a long-term preparatory strategy (as evidenced by the lack of differences in the cued-target case) that is unlikely to be of use in everyday object recognition.

The experiments reported here also provide new insights into the neural signatures of superordinate level categorization. We find that the amplitude of target-related differences in the cued-target, natural images experiment depends greatly on the requested level of categorization, with the amplitude of entry level differences on the order of twice as great as that of superordinate level differences. The amplitudes of these differences compare well with those seen at the same levels of categorization in Experiments 1 and 2 (Figure 4f), and may reflect a greater level of certainty of categorization. Another possibility is that the entry level cues allow a more precise target template to be created, leading to a better match (as discussed above, albeit this time applying to the later, RT-dependent signal, not the earlier stimulus-locked signal). Previous electrophysiological studies (Tanaka, Luu, Weisbrod & Kiefer, 1999) have suggested that superordinate level categorization occurs as late as 340 ms after image presentation. Given that entry level categorization appears to occur in as little as 152 ms, this would imply that the additional processing required for superordinate categorization might take as much as 200 ms. However, we have found evidence of target/nontarget differences at the superordinate level arising as early as 207 ms, which suggests that superordinate processing takes much less time than previously reported.

While is it not possible to discern whether the recognition-related component we observe in the EEG reflects the act of object recognition, the target decision or yet another process, it seems clear that this component can arise only after the target status of the image is known. Since the knowledge of target status is dependent on successful recognition, our measurements serve as an upper bound for the time of recognition itself. Thus, there is a variable 20-170 ms delay between the hypothetical physiological lower bound of 130 ms in a purely feedforward scheme (Thorpe & Imbert, 1989; Allison, et al., 1999) and our measured upper bound of 150-300 ms.

One hypothesis for this delay is that it reflects the extra time needed for recurrent activity to circulate in cortico-cortical feedback loops. A number of theorists have argued that recurrent processing between higher and lower levels of visual cortex is a necessary aspect of perception. Some have emphasized the role of binding of features (Grossberg, 2001; Knoblauch & Palm, 2001), while others have proposed that it sends the predictions of higher levels to lower levels to “explain away” (Rao & Ballard, 1999), or disambiguate (Lewicki & Sejnowski, 1997; Lee & Mumford, 2003) representations at lower levels. There are numerous examples from neurophysiological studies in animals (Hupe, James, Girard, Lomber, Payne & Bullier 2001; Supèr, Spekreijse & Lamme, 2001), human event related potentials (Murray, Wylie, Higgins, Javitt, Schroeder & Foxe 2002; Noesselt, Hillyard, Woldorff, Schoenfeld, Hagner, Jancke, Tempelmann, Hinrichs & Heinze, 2002) and fMRI (Murray, Kersten, Olshausen, Schrater & Woods, 2002) suggesting that cortico-cortical feedback loops play an active role in perception. Our results, while not a direct demonstration of feedback processes at work, imply that there is at least enough time for considerable recurrent activity to occur prior to recognition. The fact that target-related differences are correlated with the subsequent reaction time suggests that the first feedforward wave of activity through cortex may not always be sufficient to perform reliable recognition and that the routine use of feedback information could be a fundamental component of everyday visual processing.

Yet another possibility is that the delay reflects the extra time required to simply integrate weak or ambiguous signals at various stages along the feedforward pathway. For example, evidence has been shown for neural signals that ramp up during the decision process with a rate that depends on the strength of the stimulus (Gold & Shadlen, 2000), suggesting a neural mechanism for integrating evidence. Although such integration would involve recurrent activity, it is typically hypothesized to occur only within each level of cortical hierarchy, not between them.

One problem for both of these temporal-integration hypotheses, however, is the fact that behavioral performance remains good under strictly masked conditions (Breitmeyer, 1984; Intraub, 1999; Grill-Spector, et al., 2000; Keysers, et al., 2001). A mask that wipes clean the cortical or LGN image should disrupt any obligatory feedback or integrative processes. However, if representations are sufficiently sparse (Olshausen & Field, 1996; Vinje & Gallant, 2000), it may be possible for the representation of the mask or subsequent image to co-exist with the previous activity pattern without destructive interference. This idea is supported by the fact that masks must be individually tailored to different types of images for maximum effectiveness. Thus, it is possible that feedback processes could still operate on persistent, sparse activity left untouched by subsequent visual information.

Despite the somewhat slower timecourse for recognition measured in our experiments, it should be emphasized that the speed with which recognition occurs is still quite impressive. If cortico-cortical feedback processes are indeed a necessary prerequisite to recognition, then the number of iterations in these loops must be fairly small. The real challenge, however, is to understand what exactly is happening in these feedback loops at the level of neuronal representation and how they operate in more realistic, dynamic situations where both the eye and objects in the world are moving. In order to properly design experiments to investigate this, it will be necessary to develop more detailed, neurobiologically-based models of feedback and its role in scene analysis.

This work was supported by grant MH57921 (B.A.O.) from the National Institutes of Health. A preliminary version of this work was presented at the 2002 Annual Meeting of the Vision Sciences Society, May 10-15, 2002, Sarasota, FL. Thanks to Charan Ranganath and Scott Murray for draft comments.

Commercial relationships: none.