When human subjects are asked to localize flashes that are shown to them shortly before or during an eye movement, they make systematic errors. Such errors can be understood by assuming that visual information from the retina is combined with information concerning eye orientation at a moment that does not exactly correspond with the moment of the retinal stimulation. This is a reasonable assumption because transmission delays make it difficult for the brain to determine the orientation of the eyes exactly at the moment that the light strikes the retina. The mislocalizations might, therefore, reveal how information concerning eye orientation is combined with information from the retina.

Most studies in which subjects had to judge the position of targets that were flashed prior to a saccade, or during smooth pursuit eye movement, show that subjects misjudge the position of the flashes in the direction of the eye movement (for a review, see Schlag & Schlag-Rey, 2002). A possible explanation for this is that retinal information is combined with the commanded eye orientation without accounting for transmission delays (Brenner, Smeets, & van den Berg, 2001). If so, the mislocalization of targets during pursuit should be highly correlated with the eye movement during some interval after the flash. On the other hand, there are also studies that do not involve eye movements in which motion signals were found to influence the perceived position of briefly visible targets (Nishida & Johnston, 1999; Whitney & Cavanagh, 2000; Honda 2001; Watanabe, Sato, & Shimojo, 2003). This suggests that the target motion might be the main factor in mislocalization. If so, the mislocalization should be highly correlated with some aspect of target movement. In the studies in which targets were flashed during pursuit eye movements, the eye movements were so strongly related to the target motion that it is impossible to distinguish between the two when interpreting the data. It is not even clear whether it is the eye (or target) movement after the flash or before the flash that determines the errors, because the movement after the flash was very similar to that before the flash in these studies.

To be able to distinguish between the use of information from before and after the flash, we presented a stimulus that ensured that the information before and after the flash was different. This was achieved by having the pursuit disk follow a path that has turns in it, and flashing targets around the moment of the turns. The subjects were asked to follow the pursuit disk with their eyes. By doing this, we got the subjects to make eye movements that changed direction somewhere close in time to the flash. Because the change in eye movement direction occurs later than that in the direction of target movement, this also allows us to determine whether the eye movement or the target movement is critical. We measured the movement of the subjects’ eyes, head, and hand and compared the direction of the localization errors with the movement directions of gaze and target both before and after the flash. Because we did not want to have to deal with possible influences of retinal eccentricity (Rotman, Brenner, & Smeets, 2004), we always flashed the target where we expected the subject to be looking (i.e., at the same position as the pursuit disk). Flashing targets at the same position as the pursuit disk also encourages subjects to pursue the disk. To minimize possible effects of memory, we asked the subjects to tap the flashed target with their index finger as soon as they saw the flash. Subjects were completely unrestrained, so they were free to pursue the target naturally, with any combination of eye and head movements.

Subjects were seven members of our department, including one of the authors. Except for the author, the subjects were naïve about the purpose of the experiment. Stimuli were projected on a large screen (113 x 84 cm) that was tilted 20º with respect to horizontal. A CRT projector (Sony, VPH 1271QM, 800 x 600 pixels, 120 Hz) projected the stimuli via a mirror from the rear onto this screen. We used only the central part of the screen (70 x 55 cm, 500 x 400 pixels). The projector received its input from an Apple Macintosh G4. The subject was standing in front of the screen (Figure 1). The room was dimly lit so the subjects could see the screen and their hands. Having a visible background has the additional advantage of ensuring that subjects will not dramatically underestimate their pursuit velocity (e.g., Festinger, Sedgwick, & Holtzman, 1976; Mack & Herman 1978).

A red 15-mm diameter pursuit disk (2 cd/m²) moved along a path of connected line segments. The subject was asked to pursue this disk with his eyes. We varied the parameters of the disk’s motion to make its movement unpredictable for the subject. The disk traveled along each segment for a random period between 500 and 700 ms. The change in movement direction from one line segment to the next was chosen at random for half of the turns. For the other half, the pursuit disk made a right angle turn (either in the clockwise or in the counter-clockwise direction) to have a repeated event that we could analyze. The right angle and random angle turns alternated. The speed of the pursuit disk changed when the pursuit disk made a random angle turn. It was chosen at random from between 16 and 32 cm/s. When the pursuit disk made a right angle turn, its speed did not change. If the combination of the randomly drawn period and direction would lead the pursuit disk outside the central part of the screen, a new period and direction were chosen at random until a pair was found that fit.

A target was flashed around the time of the right-angle turn. Flashes were presented for one frame either 200, 100, or 50 ms before the pursuit disk made its turn, at the time that the pursuit disk made its turn, or 50, 100, or 200 ms after the pursuit disk had made its turn. There were 30 flashes for each of these seven moments. The flashes were green 30-mm diameter disks (8 cd/m²). They were centered at the position of the pursuit disk, so they looked like a green ring surrounding the red pursuit disk. The subjects were instructed to quickly tap the position of the flash with their index finger.

The position of the tip of the subject’s right index finger was monitored at 250 Hz by a movement analysis system (Optotrak 3010; Northern Digital) that tracked an infrared emitting diode (IRED) that was attached to the nail of the subject’s index finger. On one of its analog input channels, the Optotrak measured the blue video signal from the computer. This was done to synchronize the measured IRED positions with the flashes: The flashes were drawn in green as well as in blue but only the green output was connected to the projector.

Eye movements were measured with an EyeLink system (EyeLink I; SensoMotoric Instruments, Teltow, Germany). Three IREDS were attached to the EyeLink’s headbands, so we could measure the head’s position and orientation in space using the Optotrak. This was needed to convert the EyeLink’s (eye in head) data into gaze positions on the screen (i.e., to determine where the subject was looking).

To determine the spatial relationship between the EyeLink’s measurements and those of the Optotrak, a calibration procedure was conducted before each experiment. The calibration consisted of two steps. First, we determined the vector between each eye and the IREDS on the head, so that we could later calculate the position of each eye on the basis of the measured IRED positions. Once we knew the position of the eyes, we could determine the function that relates the direction of gaze in the EyeLink’s reference system to a direction of gaze relative to the positions of the IREDS on the head. With this we could later transform EyeLink data and measured IRED positions to a gaze position on the screen.

To determine the temporal relationship between the EyeLink’s measurements and those of the Optotrak, we used a pulse generator. The pulses from the pulse generator were measured by one of the analog input channels of the Optotrak and via the parallel port of the “operator PC” of the EyeLink system. The relative timing of these synchronization signals was calibrated using a model eye: a cylinder with a hole (simulated pupil) in it. The model eye was connected to a potentiometer. Rotating the model eye (by hand) changed the voltage over the potentiometer. An analog input channel of the Optotrak measured this voltage. At the same time, the EyeLink measured the changing position of the simulated pupil. The data measured by the EyeLink were shifted in time by various amounts and correlated with data measured by the Optotrak. The shift of the EyeLink data that gave the highest correlation coefficient told us how to synchronize the measurements. We found that the data point at the moment of the pulse in the Optotrak data file corresponded to the data point 5 ms after the pulse in the EyeLink data file.

We defined the tapped position as the first position after the start of the movement at which the velocity of the finger came below 6 cm/s and the IRED was less than 2 cm from the screen (the finger was closer because the IRED was attached to the nail). In some cases no tapped position could be determined because the subject did not move (presumably because he missed the flash) or because the subject turned his hand so that he Optotrak could not see the IRED. To quantify the mislocalization, we calculated tapping errors: vectors connecting the positions of the flashes to the tapped positions.

We determined the position of gaze on the screen for both the left and the right eye, and then averaged them. Sometimes the subject turned his head so that the Optotrak could not see one of the IREDS. Sometimes the eye could not be seen by the EyeLink (presumably because the subject blinked). In both cases, the missing parts of the gaze path were not used in the analysis, but the parts that were not missing were used.

Parts of the gaze path were synchronized with respect to the moment at which the pursuit disk changed movement direction (right angle turns). The parts with a flash at the same moment relative to the turn were averaged. Because the pursuit target moved in a random direction and at a random velocity during each part, we could not simply average these parts. Before averaging them, the paths were rotated so that motion in the direction of the pursuit disk’s movement before the turn was to the right. If the disk turned clockwise, we also flipped the paths, so that the direction of the pursuit disk’s movement after the turn was always upward. Finally, each gaze path was scaled by the velocity of the pursuit disk during that time interval (distances from the turn were divided by the velocity to give a “distance” in time), and these scaled paths were averaged. The position that the subject tapped was obviously rotated, scaled, and flipped in the same way as the corresponding gaze path.

To get a measure of the direction and speed of the gaze movement, the trace of gaze positions (after having been scaled, turned, and/or flipped) was convolved with the first derivative of a normalized Gaussian. This procedure removed noise and gave us (smoothed) gaze velocity vectors for each sample. The amount of smoothing depends on the width of the Gaussian. We used a width of 8 ms.

To find out what kind of saccades the subjects made, we determined the direction and the amplitude of all the saccades that were made between 400 ms before and 700 ms after the pursuit disk’s turn. We defined saccades on the basis of an angular velocity threshold of 40º/s. We added 8 ms of eye movement before and after the periods during which the angular velocity exceeded 40º/s to be sure to include the beginning and the end of the saccade, and consider the total change in gaze during this period as the saccade. To calculate the mean smooth pursuit component of the eye movement, we averaged all remaining eye movements.

The tapped position could be determined for 94.5% of the taps. For one subject, the right eye had not been calibrated correctly, so only the data of the left eye were used; 99.8% of the gaze data could be used.

On average, gaze turned 130 ms later than the pursuit disk (Figure 2a; at that time the average direction of gaze movement was 45 deg). This delay means that there is more than 100 ms during which gaze and the pursuit disk move in different directions. The subjects showed quite consistent pursuit behavior, as can be seen from the abrupt change of the average direction of gaze movement. Although we averaged all 1,470 pursuit movements of the seven subjects, with targets moving in various directions and at various angular velocities, the transition from 0º to 90º still occurred within a period of only about 60 ms. The speed of gaze is shown in Figure 2b; there are differences between the speed of gaze and the speed of the pursuit disk. This was the case because the pursuit movement consisted of two components, a smooth component and a saccadic component. The smooth component was consistently slightly lower than the velocity of the pursuit disk (the ratio was between 0.4 and 0.8; see Figure 2b). The saccadic component made gaze move much faster than the pursuit disk so that both components together prevent the position of gaze from moving too far from the position of the pursuit disk. The scaled total gaze velocity is on average larger than one because the eye moves over a longer distance than the target due to the overshoot at the turn.

Subjects made systematic errors when tapping the flashed targets. There were some differences between the amplitudes of tapping errors (the distance from the flash to the tap) for targets flashed at different moments (Figure 3a, p = .003 repeated measures ANOVA with subject as repeated measure). The variability in the magnitude of the tapping error also depended on the moment of the flash (Figure 3b, p = .03 repeated measures ANOVA with subject as repeated measure). The average interval between the flash and the tap was 540 ms. This interval depended on the moment of the flash relative to the turn of the pursuit disk (Figure 3c, p < .0001 repeated measures ANOVA with subject as repeated measure): Subjects were slower if the target appeared long before the turn, perhaps because the flash was then sooner after the previous tap.

A positive linear correlation that was found between the amplitude of the errors in spatial units and the disk's velocity could explain 6% of the variance in the errors. A negative correlation between the amplitude of the errors in temporal units and the disk's velocity could explain 1% of the variance. Thus, it is not completely certain that the mislocalization should be interpreted as a temporal error, but because the dependency on velocity was considerably stronger when the errors were expressed in spatial units, a large part of the tapping errors is probably the result of temporal errors. For this reason, the paths and errors were expressed in time units before averaging (scaling described in “Data analysis”). The average error expressed in time units was 133 ms.

Figure 4 shows the averages of the scaled gaze and pursuit disk paths together with all the tapped positions. This figure shows that the taps are more evenly distributed around the gaze paths than around the pursuit disk path: The distribution of the taps is shifted from the pursuit disk’s path in the same direction as the gaze paths.

Figure 5 shows the relation between the pursuit disk and gaze paths and the average tapping error. The average of the tapped positions is closer to the gaze paths than to the pursuit disk’s path. Various aspects of the timing are also indicated in Figure 5. From this, one can see that the position in the gaze path that is closest to the average tapped position is not always at the same moment after the flash. This point is usually about 200 ms after the flash, but in Figure 5b and 5c, it is considerably later.

Even when the flash occurred 200 ms before the pursuit disk’s turn, the tapping error was biased slightly in the direction of the pursuit disk’s movement after the turn (Figure 5a). As the flash gets closer to the turn of gaze, the direction of the average tapping error gets closer to the direction of gaze movement after the turn. When the flash is after the turn in gaze direction, the error is in the direction of gaze movement after the turn.

If the velocity of gaze at some instant after the flash determines the tapping error, one would expect errors distributed around 0 or 90 deg, because the eyes only move in those two directions. If the change in gaze position over a substantial time interval determines the tapping error, the errors would be distributed around intermediate values because the errors would be an average of movement in the 0º direction and movement in the 90º direction. To see whether the direction of the tapping errors shifts gradually or whether the gradual shift seen in Figure 5 is caused by averaging different combinations of tapping errors with directions of 0º and 90º, we made histograms of the direction of the tapping errors (Figure 6). Figure 6 shows that the tapping errors are distributed unimodally around the average directions that is indicated by the arrows in Figure 5.

Because subjects pursued the disk with a combination of smooth eye movements and saccades, part of the mislocalization might be related to saccadic eye movements. Figure 7 shows the number of saccades that started at specific points in time relative to either the turn of the pursuit disk, the flash, or the tap. Many saccades started between 160 and 210 ms after the turn of the pursuit disk (Figure 7a). Thus, if saccadic eye movements had a strong influence on the tapping errors, we would expect to see a large change in tapping errors for targets that were flashed some time after the turn of the pursuit disk. We do not see large differences between the tapping errors for targets flashed at different times relative to the turn, but rather a smooth transition in the direction of the errors and small differences in their magnitudes.

There were no additional saccades around the time of the taps (Figure 7b). Moreover, the diamonds in Figure 5 indicate that on average the subjects did not look at the position that they tapped when they tapped it. It seems that the subjects never looked at the position that they were going to tap when they tapped. To see whether subjects made any saccades to look at the tapped position at all, we computed the directions of all saccades. Figure 8 shows the distribution of saccade directions and amplitudes separately for the saccades that started earlier than 100 ms after the turn of the pursuit disk, and for the ones that started later. The saccades that started more than 100 ms after the turn of the pursuit disk were almost all in the direction of the pursuit disk’s movement after the turn, whereas the ones that started earlier were almost all in the direction of the pursuit disk’s movement before the turn. The amplitudes of both groups of saccades are modest and have a similar distribution. The subjects never made saccades back to the position that they were going to tap. Almost all of the saccades appear to have been made to compensate for imperfect pursuit.

The tapped positions were not distributed around the flash but around positions that gaze and the pursuit disk had after the flash. They were closer to the path of gaze than to the path of the pursuit disk. The errors that were made when tapping the targets that were flashed before the eyes or the pursuit disk had made the turn were deflected in the direction of the movement after the turn. Thus, information acquired after the flash is required to account for the tapping errors.

Without additional assumptions, the hypothesis that the subjects aimed for a position of gaze at a constant time after the flash cannot explain the errors. This is evident from the fact that the time between the flash and the moment that gaze reached the position that was closest to the average tapped position was not the same in all conditions.

The condition in which the flash appears when the pursuit disk makes the turn is critical for making the distinction between the use of gaze movement or pursuit disk movement. In that condition, the pursuit disk and gaze moved in different directions for more than 100 ms after the flash. If the movement of the pursuit disk after the flash had been critical, the average tapping error would have been “upwards” in Figure 5d. It clearly is not. Even when the flash came 50 ms after the pursuit disk’s turn, the average tapping error had a clear component in the original direction of pursuit (Figure 5e). When the flash came shortly (on average 32 ms) before the eyes made the turn (Figure 5f), the average tapping error was largely in the new direction of pursuit. When the target was flashed after the eyes had made the turn, the tapping error was “upwards” (Figure 5g). Thus, the eye movement after the flash is critical.

Brenner et al. (2001) suggested that a commanded eye orientation signal is combined with retinal information without considering neural delays. This would result in a timing error that is equal to the time that it takes for information about the flash to reach the brain, plus the time that it takes for a motor command to make the eyes move. The tapping error depends on the velocity and direction of the movement during the timing error. In our current experiment, the average tapping error (133 ms) is indeed close to the average response latency of the eyes (130 ms, Figure 2). However, the timing error should not be exactly equal to the tapping error because the direction of the eye movement changes. The amplitude of the tapping error that is predicted by the hypothesis that a commanded eye orientation signal is used, ignoring neural delays, is shown by the curve in Figure 9a. The amplitudes of the tapping errors that we found (dots in Figure 9a, corresponding to the lengths of the arrows in Figure 5) do not fall on this curve. They do bear some resemblance to the curve, but they are clearly larger than predicted.

The curve in Figure 9b shows predictions based on the above-mentioned hypothesis for the direction of the tapping error. Again, there is some similarity with the data, but the directions that we measured do not fall on the curve. In particular, when examining the direction of the average tapping error, one can see that even when the target was flashed more than 300 ms before the turn of gaze, the direction of the tapping error was deflected a bit in the direction of the gaze movement after the turn (also see Figure 5a). In that case, the interval between the flash and the change in direction of gaze movement is much longer than the sum of the time that it takes for the flash to be detected and for a motor command to travel from somewhere within the brain to the extra ocular muscles and make the eyes move (less than 130 ms, Figure 2). Thus, ignoring neural delays when combining eye orientation information with retinal information (Brenner et al., 2001) cannot fully explain the tapping errors.

The time course of the directions of the tapping errors seems to correspond to a “damped” version of the predicted directions. The use of a ”damped” representation of the eye orientation for visual localization has been proposed to explain the results of experiments in which subjects had to judge the position of targets that were flashed near saccades (Honda 1991, 1993; Dassonville, Schlag, & Schlag-Rey, 1992, 1993; Bockisch & Miller 1999). The suggestion is that localization uses a coding of the (impending) saccade that is not exactly equivalent to the saccade itself. However, the time window over which the changes in gaze direction would have to be averaged to explain the error directions that we measured (Figure 9b) would have a width of about 400 ms, which predicts error lengths that are between 283 and 400 ms, which is much larger than what we found (Figure 9a). Thus, although we could get a better fit of the direction of the timing error by assuming a severe damping of the relevant eye movement signals, doing so would result in poor prediction of the amplitude of the errors. Moreover, a “damped eye orientation signal” cannot account for the tapping errors that we found, because the turn is not predictable in our experiment, so subjects could not have planned the eye movement long in advance (as they could have done in the experiments that flashed targets around the moment of a saccade).

The shallow slope of the change in the direction of the tapping error (Figure 9b) could result from variability in the timing of signals that are involved in the response (Boucher, Groh, & Hughes, 2001). It is not clear what factors vary enough within our study to possibly result in such a damped representation. Here, we briefly discuss some obvious possibilities.

The mislocalization of flashes during pursuit is reduced when there are visible references (Brenner et al., 2001) and is increased when the flashes are hard to detect (Mita, Hironaka, & Koike, 1950). However, neither the availability of references nor the detectability of the flash varied much in our study (for stimuli that are well above detection threshold, factors such as luminance hardly appear to matter; Boucher et al., 2001). Predictability of the flash has also been shown to influence the localization error under some conditions (Mitrani, Dimitrov, Yakimoff, & Mateeff, 1979; Mateeff, Yakimoff, & Dimitrov, 1981; Rotman, Brenner, & Smeets, 2002), but that too hardly differed between flashes in our study, and would be expected to give rise to smaller errors for flashes after than for ones before the turn if it were an important factor (which is not what we see in Figures 3b and 9a). However, if the subjects were anticipating the unpredictable targets, this may have caused some of the variability in the results. Another factor that could introduce some variability is the retinal position of the flash (Mitrani & Dimitrov, 1982; van Beers, Wolpert, & Haggard, 2001). The different velocities of pursuit probably also gave rise to some variability, because the responses did depend on the velocity to some extent (see “Tapping movements”).

Although variability in the signals that are involved in the timing of the response could cause a shallow slope, as in Figure 9b, none of the above-mentioned sources of variability can explain the early onset. Possibly two effects lead to the total pattern of the errors: one effect being responsible for the shallow slope, and another for the early onset.

Another interesting finding from this experiment is that the subjects did not look at the position that they were tapping. The eyes did not return to the position of the flash; they kept pursuing the disk. Almost all of the saccades that the subjects made seem to be made to catch up with the pursuit target. This shows that ocular gaze does not always have to stay at the target of an intended hand movement, as suggested by Neggers and Bekkering (2000, 2001, 2002). In the studies of Neggers and Bekkering, as well as in ours, the subjects were given instructions about the eye movements that they had to make. In more natural tasks, without eye movement instructions, both types of behavior are found. In an experiment by Pelz, Hayhoe, and Loeber (2001), subjects had to pick up and position colored blocks to copy a given model consisting of similar blocks. They found that in certain cases the eyes kept gazing at the target location until the hand had reached it, whereas in other cases, gaze left the target location 100 to 150 ms before the hand had reached it. Johansson, Westling, Bäström, and Flanagan (2001) found that when grasping a small bar, subjects direct their gaze somewhere near the grasp site before starting the grasp, but often deviate their gaze from this position 163 ms before the hand contacts the bar. In our study, subjects had always been looking at the position that they were going to tap only about 500 ms before the tap (Figure 3a). Apparently, looking at the target of a hand movement shortly before the hand’s arrival is not a necessity. Thus, the coordination between the eyes and the hand is task dependent.

In the above, we regarded the tapping errors as temporal errors. Can we disregard the possibility that purely spatial errors play a role? The analysis of the eye movements showed that subjects were not looking where they tapped. This might have influenced the errors, because pointing toward eccentric positions can lead to systematic errors. Bock (1986) found that subjects overestimate the retinal eccentricity of a target when asked to point at it with the unseen hand. Enright (1995) found that pointing movements toward targets that were presented eccentrically relative to the head ended at more eccentric positions when the subject kept fixating straight ahead than when the subject made a saccade toward the target position. This happened irrespective of whether the target was still visible when the saccade was made or not. Exactly the same was found by van Donkelaar and Staub (2000). Henriques, Klier, Smith, Lowy, and Crawford (1998) also found that subjects overestimate the eccentricity when pointing toward targets seen in the retinal periphery. The Henriques et al. (1998) study contained a condition in which the target was presented briefly at the fovea, after which the subjects made a saccade to some eccentric position and then pointed to the remembered position of the target. In that condition, they found the same amount of overestimation of the eccentricity as when the target was presented eccentrically. That condition is the most similar to ours because in our experiment the eyes were also directed at the target when it flashed and also moved away from the position of the target before the subjects pointed at it.

The results from the four studies discussed in the previous paragraph suggest that pointing movements toward a retinally eccentric position will reach a position that is too far from the gaze position. We see an opposite trend in our data. Thus, we cannot explain the errors by assuming that the retinal eccentricity at some moment after the flash is overestimated, because if so the tapping errors in Figure 5 should have pointed away from the gaze trace after the flash and not toward it. Thus, although such effects in pointing could introduce variability, and thus contribute to the smooth transition of the tapping error, they cannot explain the bias that we find. If we assume that our subjects also have this tendency to point too far in the retinal periphery, we have to explain an even larger effect.

To explain the bias in terms of retinal eccentricity alone, we would have to assume an influence opposite to that mentioned above, namely that subjects underestimate the eccentricity. Underestimation of retinal eccentricity has been proposed to account for the results of other studies (van der Heijden, van der Geest, de Leeuw, Krikke, & Müsseler, 1999; Müsseler, van der Heijden, Mahmud, Deubel, & Ertsey, 1999; Kerzel, 2001). Such underestimation could account for the smooth transition in our data. However, the studies that proposed underestimation of the eccentricity were less similar to ours than the ones that found overestimation, because the tasks that they used were not pointing tasks. Thus, before such a mechanism can be considered to be an explanation, we would have to understand why subjects would underestimate eccentricity in this study, whereas they overestimated it in other studies in which subjects responded by pointing. Perhaps pointing is different during pursuit.

We cannot yet fully account for the time course of the errors, but our results clearly suggest that the mislocalization depends on the eye movement rather than the target movement, and on the movement after the flash rather than on the movement before the flash.

This work was financed by the Research Council for Earth and Life Sciences (ALW; grant 809-37.006) of the Netherlands Organisation for Scientific Research.