Journal of Vision - Comparing perceptual learning tasks: A review, by Fine & Jacobs

Psychophysical and neurophysiological evidence has made it increasingly obvious that the adult visual system is plastic at almost all stages of processing, from the photoreceptors (Smallman, MacLeod, & Doyle, 2001) to extrastriate areas concerned with object recognition (Kobatake, Wang, & Tanaka, 1998; Zohary, Celebrini, Britten, & Newsome, 1994). Here we examine the effect of task complexity in 16 tasks, ranging from studies of simple orientation judgments to studies of object recognition. We chose studies that were homogenous in as many methodological details as possible, and therefore only included a restricted subset of the growing number of studies on perceptual learning. We used seven main criteria for including studies. First, we limited our review to studies that examined relatively long-term learning processes by requiring that training was carried out for at least four sessions, with training sessions lasting at least 30 minutes, and only one session carried out each day. Although remarkably specific (and long lasting) learning effects have been found to take place within an hour or two of training (e.g., Fiorentini & Berardi, 1980, 1981; Shiu & Pashler, 1992; Fahle, Edelman, & Poggio, 1995; Liu & Vaina, 1998), we chose to focus on slow learning processes that take place over a number of sessions. Because of possible fatigue effects (e.g., Beard, Levi, & Reich, 1995) and the role of sleep in consolidating learning (Karni, Tanne, Rubenstein, Askenasy, & Sagi, 1994), we excluded studies where significantly more than an hour of training was carried out per day (e.g., Vogels & Orban, 1985). Second, tasks were carried out foveally or with free fixation. Data allowing comparisons between learning in the periphery and the fovea have been obtained only for low-level tasks (e.g., Johnson & Leibowitz, 1979; Fendick & Westheimer, 1983; Bennett & Westheimer, 1991; Westheimer, 2001). Because the current state of the literature did not provide enough data to determine how learning interacts with task and eccentricity, we excluded studies carried out using stimuli that extended into the periphery (e.g., Beard et al., 1995; Westheimer, 2001; Ahissar & Hochstein, 1996). Third, we only included studies where improvements did not seem to be limited by ceiling effects (i.e., performance did not exceed 95% correct by the end of training). Fourth, we excluded studies where observers were given any significant pretraining. Fifth, we only included studies where error feedback was given after each trial because some studies show stronger learning effects when feedback is given after every trial than when no feedback is given (Herzog & Fahle, 1997; Shiu & Pashler, 1992). Sixth, tasks had to be purely perceptual. For example, in the Gauthier, Williams, Tarr, and Tanaka (1998) study on object recognition, the task involved learning arbitrary names (e.g., “vali” and “pimo”) for “greebles” and parts of “greebles” (e.g., “boges” and “dunth”), and, therefore, involved a substantial semantic memory component. Finally, we only included studies where percent correct, d’, or thresholds were used as a performance measure. Studies using reaction time (e.g., Vidyasadar & Stuart, 1993) were excluded, as were studies using a combination of percent correct and reaction time (e.g., Gauthier et al., 1998), because encouraging subjects to respond as quickly as possible might result in a speed-accuracy trade-off.

Despite these restrictions, the studies we included still varied significantly in their methodology. Training sessions could last anywhere between 30 and 60 minutes. In some studies, subjects were trained till asymptote, whereas in other studies, they were trained for a fixed number of sessions. Some studies used fixed stimuli that did not vary across training sessions, whereas in other studies, stimulus intensity depended on the performance of the observer. Some studies used only naïve observers, whereas others included experienced psychophysical observers (occasionally the authors). A wide variety of tasks were used, including same-different, yes-no, 2-alternative forced choice (AltFC), 4AltFC, and match to sample.

In all studies, we converted performance into measures of d’ before and after learning (see the attached source code for further details). Signal detection theory (SDT) has been used to interpret subjects' performances in a wide variety of perceptual and cognitive tasks, and d’ can be calculated for a variety of measurement techniques (e.g., percent correct and threshold) and psychophysical procedures (e.g., yes-no or forced choice) as described in Green and Swets (1966) and Macmillan and Creelman (1991). Moreover, d’ tends to be robust to violations of its assumptions (in some circumstances this may be due to the central limit theorem and SDTs frequent use of normal distributions). For these reasons, we chose d' as a reasonable candidate for a common metric that could be used to compare learning across studies. We used a learning index, L, as our measure of improvements in performance with practice, , where s is the session number. Similar indices have been used to measure attentional effects in neurophysiology and fMRI studies (Treue & Maunsell, 1996; Gandhi, Heeger, & Boynton, 1999). The larger the learning index, the greater the amount of learning: a learning index remaining near 1 implies that observers showed no improvement in performance with practice. Learning is generally modeled with an exponential function, because at some point performance necessarily asymptotes. However, in many studies, performance never approached asymptote, and over the first four sessions, we found that learning, measured using d’ was better fit by a linear rather than by an exponential function. We estimated the slope of learning (slope_L), by fitting a line to the learning indices L_s for s={1, 2, 3, 4}. No learning would result in a slope of 0, whereas d’ doubling across each session would result in a slope of 1. We based our estimation of the slope on data from the first four sessions for two reasons. First, all of the studies included took place over at least 4 days, and second, in a few studies, observers’ behavior seemed to begin to be asymptote by the fifth session. Unfortunately, the data available to us made it impossible to reliably compare asymptotes between studies (see Figure 3).

Though some of the studies in this paper may not have strictly complied with the assumptions made by signal detection theory, our estimates of learning were remarkably robust to deviations in the assumptions that we made. For example, simulations showed that our estimates of learning were very robust to variation in our estimates of the relative standard deviations of signal and noise. Simulations also showed that, within reasonable limits, our estimates of learning were reasonably robust to deviations from the assumption that observers always used the best possible criterion. When calculating changes in d’ with practice, we chose stimulus intensities well within the mid range of the psychometric curves describing performance before and after practice. When converting threshold measures to d’, we chose a stimulus intensity where d’ was between 0.5 and 1 at the beginning of training (corresponding to a stimulus intensity resulting in performance between 59.9%-68.7% correct in a yes-no task). Figure 1A and 1B show hypothetical curves for percent correct and d’ as a function of stimulus intensity before (solid line) and after (dashed line) training in a yes-no task. The red arrows indicate changes in percent correct and d’ for a stimulus intensity corresponding to d’=0.5 at the beginning of training; at the end of training, d’ was 2.3, corresponding to L=4.7. The blue arrows indicate changes in percent correct and d’ for a stimulus intensity corresponding to d’=1 at the beginning of training; by the end of training, d’ was 4.1, corresponding to L=4.1. Conveniently, simulations showed that provided thresholds and slopes fell within reasonable limits, our estimation of the learning index was fairly robust to the choice of the intensity value for which we calculated changes in d’, especially for smaller values of L, for example, when L≈1 estimates vary by 0.5% to 1% depending on whether d’=0.5 or d’=1 was chosen as a starting point. For L≈2, estimates vary by about 3% to 6%, for L≈4, estimates vary by about 15%, and for L≈6, estimates vary by about 20%.

(b) Matthews and Welch (1997) carried out a very similar study in which observers were trained to discriminate differences in the direction of motion for a field of random dots moving within a 10-degree aperture. The direction of motion was 0° or 90° and the dot traveled at 2, 10, or 16 degrees/s (the data presented here are averaged over all three speeds). Observers were presented with a moving field of dots in each of two temporal intervals, and were asked to indicate whether the direction of motion in the second interval was rotated clockwise or counterclockwise compared to the first. Performance is averaged across six observers. Observers showed only a small amount of learning; the slope of learning, slope_L, was 0.0261. Slope_L, averaged across both studies (3a, 3b), was 0.1046.

(b) Similarly, Matthews and Welch (1997) carried out a study in which observers were trained to discriminate differences in the direction of motion for a field of random dots moving within a 10-degree aperture. The direction of motion was 45° or 135° and the field of dots traveled at 2, 10, or 16 degrees/s (the data presented here are averaged over all three speeds). Observers were presented with a moving field of dots in each of two temporal intervals, and were asked to indicate whether the direction of motion in the second interval was rotated clockwise or counterclockwise compared to the first. Performance is averaged across six observers. Observers showed only a small amount of learning; the slope of learning, slope_L was 0.0727. Slope_L, averaged across both studies (7a, 7b) was 0.2269.

Fine and Jacobs (2000) asked observers to discriminate changes in spatial frequency within a complex plaid pattern using a 4AltFC task. The “wicker” texture contained two orthogonal signal gratings masked by four noise gratings. One signal grating was centered on 3 cpd, had an orientation of -45°, and a contrast of 1.5% to 12.8%. The other signal grating was centered on 9 cpd, had an orientation of 45°, and a contrast of 5.5% to 44%. The first noise component had a frequency of 9 cpd, an orientation of -45 degrees, and a contrast of 11.2%. The second noise grating had a frequency of 3 cpd, an orientation of 45 degrees, and a contrast of 3.2%. The third noise grating had a frequency of 4.3 cpd, an orientation of 0 degree, and a contrast of 7.1%. The fourth noise grating had a frequency of 6.2 cpd, an orientation of 9 degrees, and a contrast of 7.1%. Phases were randomized in each presentation interval. Observers were asked to discriminate in which of four temporal intervals both signal gratings in the plaid shifted slightly in spatial frequency. Observers showed more improvement than when asked to discriminate small changes in spatial frequency within simple plaids (study 5), suggesting that integrating information across a wide range of spatial frequencies and orientations is a relatively plastic process. Observers showed relatively large improvements in performance as they learned to base their responses on the spatial frequencies and orientations that are relevant for the task. Slope_L averaged across five observers was 0.2517.

It is still not clear what sort of neuronal changes underlie these improvements in performance found with practice. One suggestion is that perceptual learning might be mediated by changes in the tuning of the sensitivity functions of the relevant neurons: neural tuning functions might shift, sharpen, or broaden with practice depending on the stimulus and the task. Alternatively, it has been suggested that learning might be a consequence of selective reweighting of the neurons that contribute to the psychophysical response, so that the neurons best tuned for optimal performance are given more weight (e.g., Saarinen & Levi, 1995). We believe that these two explanations are consistent with each other, because selective reweighting of neurons will necessarily result in changes in the tuning functions of all mechanisms (including decision mechanisms) subsequent to the reweighting. It seems likely that this reweighting or retuning as a function of practice may not result in permanent changes in the tuning properties of neurons, but may instead be context dependent. The lack of transfer across stimuli and tasks found psychophysically (e.g., Beard et al., 1995), as well as the context-dependent learning effects found by Crist, Li,, and Gilbert (2001), suggests that even at very early stages of processing, reweighting may be task specific and mediated by higher-level cognitive feedback and attention.

Obviously the neuronal changes underlying performance improvements may well differ substantially depending on the task. For example, the process of reweighting of inputs (and the consequent shifts in tuning) may take place sequentially throughout the visual system. Consistent with this, observers often seemed to show more learning for the stimuli that intuitively might be considered more complex (Green & Swets, 1966). Figure 2 shows the stimuli from the different tasks, ranked in order of the learning slope, with the tasks that showed least learning at the top. As can be seen from Figure 2, tasks involving relatively simple stimuli (plaids, bars, moving dots) and judgments along a single perceptual dimension, such as a spatial frequency, orientation, or direction of motion, tended to show only small amounts of learning.

We classified tasks as low level if they involved a judgment along a “basic” perceptual dimension, such as a single spatial frequency, orientation, direction of motion, or position. In none of the tasks described above were the stimuli corrupted by external noise (external noise paradigms have tended to be carried out in the periphery where learning effects are larger). Eleven tasks were classified as low level: resolution limit thresholds (study 2), direction of motion discrimination for a single dot (studies 1, 8a & 8b), direction of motion discrimination for a field of dots (studies 3a, 3b, 7a & 7b), orientation discrimination (studies 4a, 4b & 10), and Vernier offset discrimination (study 11). Performance on low-level tasks showed fairly limited improvement with practice; after four sessions, the slope of the learning index averaged across these tasks (treating different studies using the same task as separate studies) was slope_L= 0.1658. This limited learning is a little surprising given the growing literature on low-level plasticity. However, many low-level learning studies finding strong learning effects have examined improvements in performance within one or two sessions (e.g., Fiorentini & Berardi, 1980, 1981; Shiu & Pashler, 1992; Fahle et al., 1995) or have examined learning in the periphery (Crist, Kapadia, Westheimer, & Gilbert, 1997; Dosher & Lu, 1998, 1999; Mayer, 1983) where, as described above, learning effects seem to be larger (Fendick & Westheimer, 1983; Bennett & Westheimer, 1991; Westheimer, 2001), at least for low-level tasks.

Often learning for these low-level stimuli is very specific for orientation, spatial position, size, and, occasionally, eye of origin. It has, therefore, been argued that learning must be taking place in neurons, situated early in processing, that are selective for these properties. However, it is possible that neurons normally unselective for properties like orientation and spatial position might become more selective with training (Mollon & Danilova, 1996). Unfortunately, the neurophysiological evidence for changes as a function of practice at early stages of visual processing is still fairly weak. Although Schoups, Vogels, Qian, and Orban (2001) have evidence for small changes in the slope of neural tuning within V1 as a function of practice using an orientation discrimination task in the periphery, other studies have not found evidence for significant changes in population-tuning properties using a bisection task (Crist et al., 2001) and an orientation-discrimination task very similar to that of Schoups et al. (Ghose, Yang, & Maunsell, 2002). However, Crist et al. did find context-dependent surround interactions within V1 after training, suggesting that practice modifies task-dependent feedback from higher visual areas.

We found that the five tasks using stimuli that contained external noise (studies 9, 12, 13, 14, and 16) showed, on the whole, more learning than low-level tasks, with the slope of the learning index averaged across studies containing external noise being slope_L= 0.5709. This observation is not particularly surprising because several studies have found greater learning effects when external noise is added to a stimulus (Dorais & Sagi, 1997; Dosher & Lu, 1998, 1999; Saarinen & Levi, 1995; Gold et al., 1999; though curiously in the Gold et al. studies, more learning was demonstrated in low- than in high-noise conditions). Reweighting or retuning of neurons would help to exclude external noise (by reducing the weighting of those neurons for which tuning does not match the stimulus well, or for which responses are particularly sensitive to external noise) as well as to reduce internal noise by excluding neurons with a low signal-to-noise ratio. Improvements in performance on a variety of tasks have been shown to be due to a combination of external noise exclusion and, in some studies, to suppression of internal noise. For example, Dorais and Sagi (1997), Dosher and Lu (1998, 1999), and Saarinen and Levi (1995), using orientation discrimination, contrast detection, and Vernier acuity tasks, have found that learning for stimuli masked with external noise is consistent with external noise exclusion as a major factor in perceptual learning. Gold et al. (1999), using a similar external noise technique on face recognition, also found that a significant amount of learning seemed to be due to a reduction in external noise, with training seeming to have little effect on internal noise.

We classified tasks as high level if they involved identifying or discriminating real-world natural objects. Three tasks were categorized as high level: familiar object recognition (study 6) and novel face recognition with low- and high- contrast noise (studies 14 & 16). The average slope of learning across these three tasks was slope_L= 0.6. Although observers showed large amounts of learning in an unfamiliar face-identification task, they showed much less learning for familiar objects. One possibility is that previous experience of observers with the familiar objects used in the Furmanski and Engel (2000) experiment may have limited the extent of further learning within the experiment. Observers may have already had mechanisms that were optimally (or close to optimally) tuned for identification of objects at the basic level of categorization used in the study. Consequently, performance may have been limited mainly by factors such as irreducible internal noise, limiting the potential for further improvement. Tasks using familiar stimuli generally demonstrate less learning than those using less familiar stimuli: for example, Ball and Sekuler (1982, 1987; see above, studies 3a and 7a); Matthews and Welch (1997; see above, studies 3b and 7b); and others (Mayer, 1983; Vogels & Orban, 1985; McKee & Westheimer, 1978) have found more learning for oblique as opposed to cardinally oriented stimuli.

Consistent with the tendency for more complex tasks to show more learning, neuronal tuning at higher stages of processing has been shown to be highly experience dependent. This experience-dependent plasticity may help alleviate the trade-off between the need to have highly specific neurons, and biological limitations on the number of neurons that can be devoted to visual processing. Despite the fact that it would require a prohibitive number of neurons to represent every possible stimulus (because the more selective a neuron is, the smaller the number of possible stimuli it can represent), neuronal tuning in extrastriate cortex is remarkably specific. It seems that rather than representing every possible stimulus, neurons only represent a subset of possible stimuli. Especially at higher stages of processing, selectivity seems to be strongly shaped by experience, with neurons preferentially representing stimuli that have been frequently encountered, or behaviorally important in the past. Given that past experience is a good predictor of future experience, adaptability allows neurons to selectively represent an ecologically important subset of all possible stimuli. For example, neurons in inferotemporal cortex (IT) are not strongly tuned for retinotopic position, but are tuned for particular shapes: for example, neurons in macaque IT respond to particular objects and shapes, including hands and faces (Desimone, Albright, Gross, & Bruce, 1984; Logothetis, Pauls, & Poggio, 1995). These responses seem to be strongly shaped by experience with objects particular to that animal’s environment. For example, monkey face selective cells in IT show different responses to different faces, with their responses carrying identity information. The tuning of these cells seems also to be dependent on factors other than physical similarity, such as familiarity or social hierarchy (Young & Yamane, 1992; Rolls & Tovee, 1995).

P(S|s) is the probability of a hit: saying yes when the signal was present. P(S|n) is the probability of a false alarm: saying yes when only noise was present. P(N|s) is a miss, and P(N|n) is a correct rejection. A receiver-operating curve (ROC) shows how the probability of hits and false alarms change as an observer bases her responses on different criteria. As the observer lowers her criterion, the number of hits increases, but so do the number of false alarms. Because an observer only has the choice of responding yes or no, P(S|s)+P(N|s)=1 and P(S|n)+P(N|n)=1. The ROC curve, therefore, also describes the number of misses and correct rejections. If signal and noise are equally likely, and the observer chooses a criterion that maximizes the probability correct, then the probability correct is simply p(c)=P(S|s) or equally p(c)=P(N|n).

In many psychophysical procedures, correct decisions (hits and correct rejections) are equally rewarded, and errors (false alarms and misses) are equally penalized. In this case, the optimal decision rule is to choose a criterion that maximizes the number of hits and minimizes the number of false alarms, i.e., maximizes P(S|s)-P(S|n) (where noise and signal are equally likely). The best strategy is to choose S if and only if l_sn(e)>=β. Where false alarms were not measured, we assume in our analysis that observers weight hits and correct rejections equally and false alarms and misses equally. In all the studies we reviewed, error feedback did not distinguish between hits and correct rejections or between false alarms and misses.

Threshold studies measure what stimulus intensity is necessary to achieve a fixed level of performance (usually ~75% in a 2-alternative forced-choice task). Studies measuring d’ and percentage correct, on the other hand, measure performance for a fixed stimulus intensity level. Figure 1A shows two idealized psychometric functions for a yes-no task, measured before (solid) and after (dashed) training. Each value of percent correct in both psychometric functions can easily be converted into d’, as described above. Figure 1B shows d’ as a function of stimulus intensity for both psychometric functions. The change in d’ with practice for a particular stimulus intensity corresponds to the vertical separation between the two curves at that intensity value. As shown by the red and blue lines in Figure 1, the change in d’ with practice depends on the particular stimulus intensity that is chosen. We calculated changes in d’ choosing stimulus intensities well within the mid range of each psychometric curve. Where possible, we used a stimulus intensity where d’ was 0.5 at the beginning of training. A d’ of 0.5 corresponds to a stimulus intensity resulting in performance at 59.9% correct in a yes-no task. Conveniently, within reasonable limits, our estimate of the value of the learning index remained fairly robust to the choice of the intensity value for which we calculated changes in d’ with practice.

% ExampleMain.m
% example code showing how to calculate d prime from psychometric functions
% other necessary functions are
% FitdvPercent.m
% dvpercent.m
% normcdf.m
% contrast values for the psychometric function
contrast=0:.2:1;
task='YESN';
% theoretic percent correct for each contrast, before and after training
per_correct_before=[0.5000 0.5371 0.6326 0.7500 0.8542 0.9271];
per_correct_after= [0.5000 0.6326 0.8542 0.9688 0.9964 0.9998];
%initial estimate of separation
init_dprime=1;
%find dprime for each contrast
for i=1:length(contrast)
% IF YOUR MACHINE DOESN'T HAVE FMINS TRY USING FMINSEARCH %- EQUIVALENT FUNCTIONS
if(1)
dprime_before(i)=fmins('FitdvPercent', init_dprime, [], [],per_correct_before(i), task);
dprime_after(i)=fmins('FitdvPercent', init_dprime, [],[],per_correct_after(i), task);
else
dprime_before(i)=fminsearch('FitdvPercent', init_dprime,[],per_correct_before(i), task);
dprime_after(i)=fminsearch('FitdvPercent', init_dprime, [],per_correct_after(i), task);
end
end
% find the contrast for which dprime=0.5 before training
init_dprime=0.5;
interp_contrast=interp1(dprime_before, contrast, init_dprime); %the contrast for which d'==.5
% find the dprime value after training for the contrast at which dprime=0.5
% before training).
new_dprime=interp1(contrast, dprime_after, interp_contrast);
subplot(1, 2, 1)
plot(contrast, per_correct_before, 'k', contrast, per_correct_after, 'k--');
xlabel('contrast')
ylabel('percent correct')
legend('before training', 'after training')
subplot(1, 2, 2)
plot(contrast, dprime_before, 'k', contrast, dprime_after, 'k--');
xlabel('contrast')
ylabel('dprime')
%*****************************************%
function L=FitdvsPercent(dprime, correct, task);
% finds the d prime separation for a given percent correct
% uses maximum likelihood function minimization
bestper=dvPercent(dprime,task);
L=(correct-bestper)^2;
%*****************************************%
function bestper=dvPercent(dprime,task);
% finds the percent correct (assumining an optimal criterion etc.) for a given
% dprime
% creates signal and noise distributions, assuming signal and noise
% have equal standard deviations of 1
% distributions are scaled by the standard deviation of the noise
sigS=1; %standard deviation of signal
sigN=1; %standard deviation of noise
x=linspace(-10, 10, 1000)./sigN;
dprime=dprime/sigN;
sigS=sigS/sigN;
%calculate the hit/false alarm rate
hit=1-NormalCumulative(x, dprime, sigS^2);
fa=1-NormalCumulative(x, 0, sigS^2);
if (task=='YESN') %yes-no
correctvals=hit+(1-fa); %assuming the criterion is that yes and no equally likely
beta=find(correctvals==max(correctvals));
bestper=hit(beta(1));

elseif (task=='2ALT') %2-alt FC
bestper=0;
bestper=sum((hit(1:length(x)-1)-hit(2:length(x))).*(1-fa(2:length(x))));

elseif (task=='3ALT')
bestper=0;
bestper=sum((hit(1:length(x)-1)-hit(2:length(x))).*((1-fa(2:length(x))).^2));

elseif (task=='4ALT')
bestper=0;
bestper=sum((hit(1:length(x)-1)-hit(2:length(x))).*((1-fa(2:length(x))).^3));

elseif (task=='SMDF')% same different
correct1=hit+(1-fa);%assuming the criterion is that same and different equally likely
beta=find(correct1==max(correct1));
beta=beta(1);
bestper=2*(hit(beta))^2-2*hit(beta)+1;
end
%*****************************************%
function prob = NormalCumulative(x,u,var)
% function prob = NormalCumulative(x,u,var)
% Compute the probability that a draw from a N(u,var)
% distribution is less than x.
% Taken from the psychophysics toolbox
% http://www.psychtoolbox.org//
% 6/25/96 dhb Fixed for new erf convention.
[m,n] = size(x);
z = (x - u*ones(m,n))/sqrt(var);
prob = 0.5 + erf(z/sqrt(2))/2;

We would like to thank the authors who so generously shared their data with us, in particular Merav Ahissar, Karlene Ball, Patrick J. Bennett, Heinrich Bultoff, Shimon Edelman, Stephen A. Engel, Manfred Fahle, Christopher S. Furmanski, Bard J. Geesaman, Charles D. Gilbert, Jason M. Gold, Erik D. Herzog, Shaul Hochstein, Avi Karni, Zili Liu, Nestor Matthews, Ning Qian, Allison B. Sekuler, Dov Sagi, Robert Sekuler, Mariano Sigman, and Leslie Welch. We would also like to thank Geoffrey M. Boynton, Zhong-Lin Lu, and two anonymous reviewers for helpful comments on the manuscript. This work was supported by National Institutes of Health Research Grants R01-EY13149 and EY01711, National Science Foundation Grant SBR-9870897, and a La Jolla Interfaces in Science postdoctoral fellowship. Commercial relationships: None.