Although the model was removed for subjects in Group 2 and the design was longer in the case of Group 3, there were no significant differences between groups in the proportion of occurrence of the four main categories, as can be seen on the right column of Table 1.

Complex search was the most commonly occurring category of search pattern. On these occasions it seems unlikely that subjects are using spatial memory to target a specific piece, but rather go to some random location in the resource and then search for a particular piece on the basis of visual features (see Movie 1). In the local search category, subjects could be using either visually based search or memory. In the latter case they may target a remembered location, and then correct the movement (see Movie 3). However, for a significant proportion of the pickups during no swap trials (around 25%), subjects landed directly on the piece to be picked up after a single large saccade into the resource area (direct movement), and it is tempting to assume the saccade was programmed on the basis of spatial memory. We examined the video records in each of these movements to find if the landing point was in fact visible in the peripheral retina at the point of initiation of the saccade. We found that in about 49% of the movements the target was not visible, so these are clearly programmed on the basis of spatial memory (see Movie 2). In the remaining cases, the target was visible, so current retinal image information, or some combination of memory and current visual information, may have been used to program the movement. It is also possible that subjects were not targeting a piece of a particular color, and simply happened to pick up the piece they landed on by chance.

In general this seems unlikely because subjects copied the patterns in a very reproducible order, suggesting that each visit to the resource area was for the purpose of locating a piece of a particular color and shape. Subjects’ tendency to construct the copy in the same order is shown in Figure 7. The proportion of times each piece was placed at a specific moment over trials is plotted against order of placement. As can be seen in the plot, average performance in our experiments was very close to always placing the pieces in the same order. The calculation of Goodman and Kruskal’s Gamma as a measure of ordinal association showed that in almost 80% of the cases it was possible to predict correctly which piece would be placed at that specific moment based on previous performance (Gamma = .798, p < .001).

The interpretation of the old to new category search pattern, when the resource pieces moved around in the swapping condition, is similarly ambiguous. As in the case of the direct movements during no swap trials, a large proportion of these saccades were made when the initial landing point was out of the visual field. Inspection of the video records revealed this was the case for 77% of the old to new movements. Note that this is a somewhat higher proportion than for the direct movements. These movements must have been programmed on the basis of information from prior views. Of the remaining old to new movements, when the landing point was visible, we cannot definitively conclude that the saccade was programmed on the basis of memory information. However, it was often observed that the hand movement targeted the same location, and arrived at about the same time as the eye, and made the same corrective movement to the new piece, which strengthens the suggestion that a particular spatial location was being targeted (see Movie 4). Note that if this interpretation is correct, the saccade target selection process must use memory information exclusively, even though inconsistent visual information is present in the retinal image because a different piece is now at the old location.

As described above, fixations in the resource area, on trials just before and just after the swapping manipulation was introduced, were also analyzed to see whether the introduction of changes in swap trials had any effect in their frequencies or durations.

An analysis of the fixations in the resource area showed that actions while finding the next piece to be moved could be organized primarily in two different phases: First, subjects search the area until they localize the next piece to be moved, and then they fixate it while controlling the hand to pick it up. Both fixations while locating next piece and while picking it up are the most frequent, and together account for 85% of the fixations made on the resource area. Other fixations (e.g., fixations on other pieces after location but before pickup) only appeared in 13.6% of the cases and their frequency differed between subjects. Because of the higher between-subjects variability, these fixations were not analyzed further.

As can be seen in Table 2, the mean frequencies of fixations locating and picking up a piece are quite similar, with a mean value of around 12 fixations per trial; that is, a little more than one fixation, on average, of each kind for every piece that was moved. However, fixation durations are very different depending on their function: Fixations made while picking up have a duration of 2000 ms on average, whereas fixations made while locating the next piece are shorter (around 300 ms on average). This result presumably reflects the different requirements of locating the piece and guiding the pickup.

Epelboim et al. (1995) also segregated fixations into search (or locating) fixations and those guiding the tapping movement, with a similar difference in durations. Given that there is no haptic feedback, and contact with the piece is signaled by a color change, it is to be expected that the process of picking up the piece in the virtual environment should be quite long. Dependence of fixation duration on the specific visual information required has also been described in other experiments using natural tasks (Hayhoe, Shrivastava, et al., 2003; Land, Mennie, & Rusted, 1999).

During swap trials, multiple changes were introduced in the resource area every time participants made a saccade away from it. If saccades into the resource area preparatory to picking up a piece are planned using spatial information from prior fixations in the resource area, then such a manipulation should affect those saccades made after a swap. To analyze this effect, frequencies and durations of fixations in the resource area were obtained and compared between the last two trials before swap and the first swap trial. Only fixations made while locating and while picking up were analyzed. Long bars were excluded when analyzing locating fixations because they did not vary their position in swap trials.

As can be seen in Figure 8, the introduction of changes in the position of the pieces in the resource area produced an increase in the number of fixations made while locating the next piece, but not while picking it up. This increase in the frequency of locating fixations with the introduction of swap reached significance (see Table 3). Within-subjects contrasts showed that the first trial after swap was significantly different from the two trials prior to swapping (trial 4 vs. trial 6: p =.005; trial 5 vs. trial 6: p =.002), whereas the last two trials before swap did not differ (trial 4 vs. trial 5: p = 1). Although the increase in the number of locating fixations was higher for Group 2, there were no significant differences between the groups in the increase in frequency after swap (F₂ = 2.042, p = .164; tested by a one way ANOVA on the effect of group over the differences in frequency between the last trial without swap and the first one with swap).

Thus, the introduction of changes in the environment significantly increased the number of fixations made by the subjects while looking for the next piece to be picked up, but did not affect fixations made while controlling the hand movements needed to pick up that piece, or once the next piece had been located. This means that when the pieces were in a stable location in the resource area, subjects took advantage of that to aid search. Interestingly, the introduction of changes only affected the frequency of the fixations made during the locating phase, but not their duration (see Table 3).

As can be seen in Figure 8, all three groups of subjects showed the same effect with the introduction of changes during swap trials. In all cases, after the introduction of changes in the environment, the frequency of fixations while locating pieces increased. However, as can be seen in Table 3, there were also significant differences between groups in the frequency of fixations while picking up the next piece. Specifically, subjects of Group 3, who experienced 10 trials in the task before swap was introduced, showed significantly fewer fixations while picking the next piece than the other two groups [post hoc analysis using Tukey contrast showed significant differences between Group 3 and Groups 1 (p = .015) and 2 (p = .002)]. There were no significant interactions between the effects of group and swapping. This result suggests that practice decreased the number of fixations during pickup. However, there were no differences between groups in the mean duration of fixations, which suggests that practice does not reduce the amount of time needed to process the contents of each fixation.

After finding that the introduction of changes in the environment (swap condition) produced an increase in the number of fixations made in the resource area while locating the next piece, we were interested in analyzing how such an effect was related to the different patterns of eye movements that subjects could use. To do so, all fixations made while locating the next piece were classified based on the category that was used to describe the sequence they were in. This analysis showed that the new fixations in the resource area that appeared during swap trials resulted from the use of the “old to new” strategy to find the next piece. Figure 9 plots the same data as in Figure 8, for frequency of fixations made while locating a piece (black line), together with that value with the fixations resulting from the old to new category removed (red line). Thus it can be seen that all the extra fixations resulting from introducing swaps came from those sequences in which subjects went to the old, remembered position of the piece, and then to the new one after the change. This result strongly supports the interpretation of the old to new fixations as being based on memory. A paired samples t test also confirmed a significant difference in the mean number of fixations between both cases, when the fixations resulting from the old to new category were or were not included in the total (T₁₇ = 4.873, p < .001).

As discussed above, previous experiments using a model copying task have shown that subjects often rely on multiple fixations in the model to both pick up and drop a piece from the resource area (Ballard et al., 1995, 1997; Hayhoe, 2000; Hayhoe et al., 1998; Karn & Hayhoe, 2000). This behavior suggested that subjects prefer a minimal memory strategy, where they fixate the model in preference to using visual memory of its properties.

The present experiment differed from those experiments in that the same model pattern was repeated in each trial, thus giving subjects longer exposure to the same patterns. We measured the frequency with which subjects looked at the model for each trial of the experiment (but without calculating how many fixations took place in each look). This analysis showed that over the trials of the experiment subjects of all groups needed to look less and less often to the model (see Figure 10). This decrease appeared in both possible cases, when looking at the model before picking up a piece (that is, prior to locating it), and when looking at it after a pick up (in the way to the work space). At the beginning of the experiment, subjects needed to look at the model around 16 times (taking together before and after pickup looks) in the course of copying it: that is almost twice per piece. After 5 trials, subjects looked at the model about 6 times, or less than one time per piece. After 10 trials, they were looking at the model about one time per each two pieces.

This result shows that subjects clearly learn and remember the properties of the model and that in fact they prefer to perform the task at least partly by memory. For that reason, the disappearance of the model in the case of Group 2 did not have any obvious effect in performance. Subjects still looked at the empty model area occasionally, but their performance was similar to that of the subjects of the other two groups. These data are consistent with the data on fixations in the resource in showing that subjects accumulate spatial information across fixations and also over several trials.

The current experiment provides evidence that memory of the spatial structure of a scene is retained across gaze positions and is used in saccadic targeting in the course of natural behavior. When selecting a target for gaze changes into the resource area, for the purpose of picking up a piece needed for copying, observers frequently made saccades that fell directly on the piece they then picked up (see Movie 2). This occurred on about 25% of the gaze changes. Although these gaze changes cannot be definitively identified as memory guided, there are several arguments for this interpretation. First, subjects were quite consistent in the order with which they copied the model. This suggests that the piece they landed on was explicitly targeted, as opposed to a situation where observers simply picked up the piece that they accidentally landed on. Second, about half the movements were initiated when the landing point was outside the field of view. Such movements must rely on spatial memory for target selection. For the rest of the movements, the target was usually close to the edge of the visual field (about 25-deg eccentricity) at the point when the saccade was initiated, usually as a consequence of an ongoing head rotation toward the resource area. In these cases it is possible that the visual features of the object played a role in target selection. A stronger argument for the use of spatial memory comes from the old to new category of gaze movements in the latter part of the experiment when the pieces changed locations every time the observers looked away (see Movie 4). The fact that observers do not pick up the piece they land on, but instead move immediately to fixate and pick up the piece that had previously been in that location, strongly suggests the use of spatial memory in targeting.² This strategy accounted for about 20% of the gaze movements into the resource area during swap trials. A final argument implicating spatial memory is the increase in fixations required to locate a piece when the pieces changed location following each pickup. About 2 to 5 additional fixations were made in the resource area in the course of the trial when the pieces were changing locations (that is, about 0.3-0.8 extra fixations per piece; see Figure 8). Importantly, all these additional fixations came from those sequences in which observers landed on the old location and then had to find the piece in the new location.

One of the goals of this experiment was to evaluate the demands that natural behavior places on memory from prior fixations. Although it seems fairly clear that observers used memory from prior fixations to target movements back to the resource area, it was by no means the most common strategy. Only about 25% of movements in the first 5 trials were direct movements to the pieces, and only about 20% of movements went to the old location during the subsequent swapping trials. On about half the movements, observers made multiple fixations around the resource area before locating a piece for pickup (complex search). Thus, while observers commonly use spatial memory to program the movements, they are more likely to make a large saccade to the general region, and then to search for the piece in a local region, presumably using visual features. On the other hand, on those trials when memory-based targeting is implicated (direct and old to new movements), the spatial precision is quite impressive, because the gaze changes were 20-to-30 deg in magnitude, and the targeting precision approximately 2-3 deg. Land et al. (1999) have also noted excellent accuracy in very large gaze changes to regions outside the field of view (greater than 90 deg) that land within a few degrees of the target. The need to orient to regions outside the field of view in natural vision (e.g., moving around within a room) provides a rationale for storing information about spatial layout. The fact that many of the direct and old to new saccades were actually to regions currently visible in the retinal image suggests that spatial memory information is not used exclusively for locations outside the field of view. Consistent with this, Edelman, Cherkasova, and Nakayama (2002) and Kristjánsson and Nakayama (2003) have observed that subjects are able to locate and saccade to targets that are unresolvable in the peripheral retina, provided they have been fixated previously. This suggests that spatial memory aids target selection for objects within the field of view.

An advantage of a strategy that uses memory information, whether or not the target is within the field of view, is that it may minimize the number of movements (and time) required to locate a piece. Although minimizing the time to locate a piece might not always be particularly critical, another and possibly more important advantage is that it allows early planning of head and hand movements. We have found that, in this experiment, observers initiate the hand movement to the resource area on average about 400 ms before the eye movement. The head movement is initiated about 200 ms before the eye (Hayhoe, Aivar, Gaines, & Jovancovic, 2003). Typically, in response to a visually presented target, head- and hand-movement initiation lag behind the eye by 100 ms or more (Abrams, Meyer, & Kornblum, 1990). The early initiation of the movements has the consequence that the head and hand both arrive close in time to the arrival of the eye in the resource area. Thus a significant role for memory for the spatial layout of a scene is probably for early planning and coordination of the eye, head, and hand movements.

The time course of the memory for spatial structure observed in this experiment is difficult to evaluate. The reduction of the frequency of complex search patterns over the first 5 trials is consistent with observers building up long-term memory representation of the layout of the resource over a period of tens of minutes and of the order of 100 fixations. A similar reduction in the number of fixations required to locate items for tapping was observed by Epelboim et al. (1995). However, the reduction in complex search frequency was not accompanied by an increase of the direct movements that explicitly implicate memory use. The frequency of the direct strategy remained fairly stable across the first 5 trials (see Figure 5). The decrease in frequency of complex search patterns might therefore reflect other aspects of learning, such as learning what piece to select. Another possibility is that subjects may learn general aspects of the spatial structure, such as the location of the resource relative to the workspace, the general layout of the resource, and potential locations for pieces, rather than the location of specific pieces. This information may aid search without necessarily contributing to the frequency of direct or old to new movements. Alternatively, the reduction of complex search frequency might reflect subjects’ use of some amalgam of current visual information with memory signals that facilitates search but does not clearly implicate spatial memory as the direct and old to new strategies do.

Although the overall reduction in number of fixations as a function of trial number in both model and resource areas points toward some kind of accrual in long-term memory, the memory revealed by the occurrence of old to new fixations may be of shorter duration. Because all pieces change position every time the subject looks away from the resource area, it is only possible to use location information from the immediately prior visit to the resource area. If observers depended only on long-term memory, one might expect to see no evidence for memory-based targeting at all. However, the old to new strategy accounted for an extra 2-5 fixations per trial, and about 20% of the movements. This suggests that subjects base their targeting on memory from the immediately prior visit to the resource area. Because several fixations and about 5 s intervene before the return to the resource area, it seems that the information is not rapidly decaying, but nonetheless it does not appear to reflect accumulation over the entire experiment: When a recognition task was presented at the end of the experiment to the subjects in Group 3, none of them were able to select the picture showing the position of the pieces at the beginning of the trial. In the tea-making task, Land et al. (1999) also noted a number of instances where objects were found more easily when they had been fixated a few seconds previously. The current observations provide further evidence that memory across fixations is needed as a basis for motor planning and coordination.

It is also a little surprising that the frequency of the old to new strategy remains fairly constant across the 5 trials where the pieces changed every trial (see Figure 5). Subjects do not appear to reject or move away from the memory-based strategy over time, despite the incidence of landing on the wrong piece. It is also interesting to note that the incidence of the complex search strategy does not appear to increase (in fact, it decreases) when the pieces change locations in the swap trials. This suggests that the memory responsible for the overall reduction in complex search as a function of trial number is not the same as that responsible for targeting the pieces in the direct search and old to new movements. That is, the memory used for targeting the resource pieces for the most part is short term, though there may be some undetermined facilitation from long-term accrual. In this respect the time course is more comparable to priming of pop-out, which is effective over a few trials (Maljkovic & Nakayama, 1994, 1996, 2000), than to contextual cueing, which accrues over blocks of trial (Chun & Jiang, 1998).

Although the use of memory to guide saccadic targeting in this experiment is not the dominant strategy, it is clearly a significant aspect of performance. Movements to the resource area are frequently planned using visual information acquired several seconds previously during prior fixations in the region. This means that memory representations integrated across saccades must include precise spatial information that can be used for saccade planning, in addition to scene gist and a small number of object files, as previously proposed (e.g., Irwin 1991; Irwin & Andrews, 1996; Irwin, Zacks, & Brown, 1990; O'Regan, 1992; O'Regan & Levy-Schoen, 1983). Other evidence shows that information about the spatial organization of scenes is preserved across fixations. For example, De Graef and Verfaille show encoding of spatial relationships of "bystander" objects that are not the target of a saccade (De Graef, Verfaille, & Lamote, 2001; Verfaille, De Graef, Germeys, Gysen, & Van Eccelpoel, 2001). Hayhoe et al. (1992) also showed integration of very precise spatial information across saccades that served as a basis for spatial judgments. O'Regan (1992) and Irwin (1991) have postulated that there is some integrated representation of the scene, but suggest that the representation of spatial information is imprecise and that the representation is semantic in nature. The evidence presented here, however, supports the suggestion of Chun and Nakayama (2000) that the spatial information cannot be imprecise, but must be able to support high precision movements.  Other evidence also suggests that the original proposals of O'Regan (1992) and Irwin and Andrews (1996) probably underestimate the extent of the memory across saccades. For example, Hollingworth and Henderson (2002), Irwin and Zelinsky (2002), Melcher (2001), Melcher and Kowler (2001), and Tatler, Gilchrist, and Rusted (2003) have all demonstrated robust visual memory representations of multiple objects and their locations in images of complex scenes. However, it is difficult to determine the precision of the spatial information retained in these experiments because a partial report technique was used to explore memory of the objects in the scene. In our experiments, the accuracy of both eye and hand movements in the direct and old to new strategies suggests that the spatial information retained is quite precise. However, more research is needed to more directly assess the precision of the spatial information retained in memory.

In summary, despite compelling evidence from change blindness studies, our results suggest that there are implicit memory mechanisms implied in saccadic guidance and movement control that can retain precise spatial information about the objects in the scene. Such mechanisms are useful for the adequate coordination of eye and hand movements, and so need to be studied under complex paradigms that try to get closer to the conditions of vision and action in natural behavior. Thus using simple tasks to measure transsaccadic memory does not reveal the extent to which memory is required for ordinary behavior.

This research was supported by National Institutes of Health Grants EY 05729 and RR 06853. MPA was supported by a doctoral FPU grant from the Ministry of Education and Culture of Spain (AP97-10903352) and by a research grant from the University of Oviedo. Portions of this article are based on a dissertation submitted by MPA in fulfillment of the requirements for the Ph.D. degree at the University of Oviedo.

We would like to thank Brian Sullivan and Diane Kucharczyk for their assistance with data collection and analysis; Tomás R. Fernández and Jose Carlos Sánchez for helpful discussions on these issues; and all subjects that participated in this experiment for their collaboration.

¹While doing the task subjects habitually maintained only one or two of the areas inside the visual field of view. To see the whole configuration subjects needed to move their heads backward so that the three areas will fit the visual field.

² Note that there are a small proportion of direct fixations during the swap condition. These fixations could result from subjects picking up a long bar (long bars did not change position during swap trials), or could also appear in those sequences during swap trials in which swap did not occur. (Although it was set to happen every time, swaps only occurred on 80% of the sequences.) It is also possible that subjects may have opted to pick up the new piece that occupied the position they landed on after a direct saccade.