|
|
|
 |
Classification image weights and internal noise level estimation
Abstract
For the linear discrimination of two stimuli in white Gaussian noise in the presence of internal noise, a method is described for estimating linear classification weights from the sum of noise images segregated by stimulus and response. The recommended method for combining the two response images for the same stimulus is to difference the average images. Weights are derived for combining images over stimuli and observers. Methods for estimating the level of internal noise are described with emphasis on the case of repeated presentations of the same noise sample. Simple tests for particular hypotheses about the weights are shown based on observer agreement with a noiseless version of the hypothesis.
History
Received November 20, 2001; published March 22, 2002
Citation
Ahumada, A. J., Jr. (2002). Classification image weights and internal noise level estimation.
Journal of Vision, 2(1):8, 121-131,
Keywords
discrimination, detection, vision, noise
In 1965, a frustrated graduate student in physiological
psychology was looking for a thesis topic in the auditory research laboratory of
E. C. Carterette and M. P. Friedman, the editors to be of the
Handbook of Perception. They
recommended that he tape record the stimulus of the traditional tone-in-noise
yes-no detection experiment and analyze the sounds in the four different types
of trials to determine whether correlates could be found in the stimuli relating
to the observer responses. The noise masker was continuous wide-band noise, and
marker tones were recorded on a second track to keep track of the signals
presented. The tapes were digitized and analyzed, but the signal-to-noise level
at threshold was so low that no trace of the signals could be found in the
digitized records. To ensure earning a degree in the foreseeable future, the
student made several changes in the experiment. To improve the signal-to-noise
ratio on the tape, the noise bandwidth was narrowed, and the noise was turned on
only during the short interval when the signal might be present. To reduce the
effects of observer noise, the tape was repeatedly presented to the observer to
get average ratings of signal presence. To minimize degrees of freedom in the
stimulus measurement, the stimulus was reduced to the energy passed by a filter
tuned to the signal tone frequency. This combination of changes allowed the
student to find that on signal trials, very narrow filter outputs correlated
best with observer ratings, whereas on noise trials, wider filter outputs
correlated best, contradicting the prediction of single linear filter models for
auditory tone detection (Ahumada, 1967).
To gain better control of the masking noise and avoid
the limitations of tape recording,
Ahumada and Lovell (1971) used
computer-generated tones and noises defined by their Fourier component
amplitudes and reported linear regressions on the component energies with
average observer ratings. These results were essentially auditory classification
images that again demonstrated results contrary to simple linear filter theory:
frequency components were weighted differently on signal trials from noise-only
trials and negative weights were frequently observed. The results of both
experiments seemed to be consistent with models with multiple linear channels
that were being nonlinearly combined.
Ahumada, Marken, and Sandusky (1975) extended
the experiment to the combined time and frequency domains with similar
results.
Our first visual
classification images (Ahumada, 1996) were done
to see whether the method we had used in audition could be used to elucidate the
features used by observers to accomplish a vernier acuity task. Figure 1 shows a
raw classification image and the same image smoothed and quantized so only
weights that are significantly different from zero are colored differently from
the gray background. The ideal observer would have only weights on the right
side, the side of the line that was either even with or one pixel higher than
the left line. Spatial position uncertainty was presumably responsible for the
observer needing to compare the two lines and for blurring the image more than
optical blurring would predict. Theories that postulate that the response would
be determined by the output of a single best-discriminating Gabor-like filter
(Findlay, 1973;
Foley, 1994) are not supported by the
appearance, but were not tested statistically.
Beard and Ahumada (1998) wanted to see
whether observer performance was best characterized as orientation
discrimination based on an oriented filter output or a local position
measurement
(Waugh, Levi, & Carney, 1993). The
question was left unanswered; the linear classification functions obtained from
the abutting stimuli were consistent with possible implementations of either
theory.   Figure 1.
A raw classification image (top) and the same image smoothed and quantized
(bottom), so only weights significantly different from zero are colored
differently from the gray background. The black squares on the sides show the
heights and positions of the fixed line (left) and the variable line offset
(right). The dark lines on the top and bottom show the lengths and positions of
the lines. The observer was A.J.A., who ran 1,600 trials
(Ahumada, 1996).
The first visual classification images were linear
combinations of four averaged noise images, one for each of the four
stimulus-response categories. For a given stimulus, the average of all the added
noises has zero mean, so the sum of noises from one response class has an
expectation equal to the negative of the expectation of the sum of the noises
from the other response class, so we knew to combine the two response noise
images with opposite sign. It appeared in the initial images that the error
images were clearer than the correct response images, so we took the difference
of the averages rather than the sums, realizing that this was an arbitrary
decision. We also arbitrarily combined the images from the two stimuli with
equal weight to get a single overall image. By symmetry, this must be the right
weighting to use if the observer is making the same number of errors to equal
numbers of each kind of stimulus, which was approximately the case. In the next
section, there is an analysis showing that for a simplified theoretical
situation, it is possible to show that the averaging is nearly optimal and to
find expressions for good weighting functions for the cases of unequal stimulus
presentation rates and unsymmetrical response biases. The beginning of the
section introduces notation for a standard signal detection experiment as
analyzed by Green and Swets (1966).
s 0
and
s 1
are 1 by m signal vectors,
presented with probabilities
p 0
and
p 1
= (1 −
p 0)
for N trials. On each trial,
a random noise sample vector
n is added
to the signal, so the trial
stimulus
or
n
is a 1 by m vector of
independent samples of identically distributed Gaussian variables
n(i)
with
and
where
E[·] is the averaging or
expectation operator and
Var[·] computes the
variance. Without loss of generality, we can assume that the noise has been
normalized by its standard deviation so
that
The linear observer classifying the noisy signals by
responding R = 0
or R = 1 or would use
a vector w
and respond R = 1 if and only
if
where β is a response criterion and
T indicates the matrix
transpose operator, so that
Also, without lack of generality, we will
assume that
w has unit
length (w
and β have already been divided by the length of
w) so
that
The performance of an observer is characterized by the
error
rates
the probability of signal
s 0
being followed by response
R = 1, and
the probability of signal
s 1
being followed by response
R = 0.
For the linear classifier with vector
w and
criterion β,
w
n
T is Gaussian with mean zero and
unit variance.
Hence
and
where
Φ(·) is the
cumulative standard Gaussian distribution
function. If we define sensitivity and bias
parameters
and
then the error rates are
and
These parameters can be found from the error rates as
and
where
is the functional inverse of the cumulative
standard normal distribution function
Φ(·).
The ideal observer classifying the noisy signals as
R = 0 or
R = 1 would use the
linear classifier
For the ideal
observer,
The efficiency of a non-ideal linear classifier is
given by
the square of the correlation between the
actual and the ideal classifier coefficients, sometimes called the sampling
efficiency.
Human observers classify the same images different ways
on different presentations. This is modeled here by assuming that the observer's
criterion
β 0, H
(corresponding to β
0) is a normally distributed random variable with
and
independent of the noise
n. It does
not matter whether the variability is added to the criterion or the
classification function value. Because the noiseless criterion
β 0 was
defined as the criterion for a variable with unit variance, the parameter
1 +
γ2
can be interpreted as the total variance of the classification variable and
as the proportion of variance in the
classification variable that arises from the external noise
n. The
error probabilities are
now
and
If we define the observer's sensitivity and biases
as
then we can compute these parameters from the
human model observer error rates as
and
The efficiency of the human observer model
is
Because
ρ
2
≤ 1, a lower
bound for α is given
by
and an upper bound for
γ 2 is given
by
These bounds are reached when
w is
w I,
and the inefficiency is only the result of the internal or criterion
noise.
The classification image components are the four
average noises
a s, R,
the averages of the noises
n for the
trials segregated by signal
s
s
and detection response
R. We would like to find the
mean and the variance of the pixels of
a s, R
as a function of the parameters
(s 1,
s 0,
w,
β 0,
and γ or α).
In the single pixel
(m=1) case, we seek the mean
of a single Gaussian variable
n that has been truncated by
a random criterion
β H.
Let
n s, R
be the truncated variable when
s was the
stimulus and R was the
response
and
Then, because
||w||
= 1, w = ±1. We can assume without loss of generality that
s 1
is greater than
s 0,
and the sign of w is set to
maximize correctness, so that
w = 1.
Hence,
if and only
if
So in the case that
s
= R =
0,
and for the other
cases
Consider now the single pixel case when there is no
noise in the criterion
(β 0, H = β 0).
where
φ(z) is the standard
normal density function and the integration of
Similarly,
The Gaussian criterion case can be reduced to the fixed
criterion case by a change of variables. Let
z be the standard Gaussian
used to form the criterion
β 0, H,
so
that
Then
If we
let
and
the new variables
x and
y are independent
(E[x y] = 0), standard
(E[x] = E[y] = 0, Var[x] = Var[y] =
1) Gaussian variables. These variables have the properties
that
and
that
if and only
if
So
The effect of the criterion noise on
v 0,
0 is to reduce it by the factor α.
Similarly,
because
p 0,
1 =
1−
p 0, 0
and
If false alarms are less frequent than correct
rejections
(p 0, 1
<
p 0,
0 ),
then
a larger absolute expected value on a false
alarm than a correct rejection trial. The signal case is the same with the
criterion changed to β 0,
H −
d
0' so
that
and
Again, if misses are less frequent than hits
(p 1,
0<p 1,1),
then
a larger absolute expected value on a miss than
a hit trial. Regardless of the signal, the expected value depends only on the
response proportion and the criterion
variability.
Another independent variable transformation allows the
single pixel case result to solve the multiple pixel case. Let us first examine
the means and variances of the pixels of
n 0,
0. For any vector
w of unit
length, it is possible to construct an orthonormal transformation
U whose
first row is
w, that
is
such
that
where
I is the
identity transformation (the transpose of
U is its
inverse). When this transformation is applied to
n T,
we get a new noise
U n T
whose distribution is the same as that of
n T,
but whose first pixel is
w n T.
On an
s 0
trial, a noise vector
n will be
classified as
n 0, 0
if and only if the first pixel of
U n T,
The rest of the pixels
(z 2, ..., z m)
of
U n
T are independent standard
Gaussian variables (with mean zero and variance
1).
A similar argument for the other cases leads to the
general result
that
The mean of a classified noise is proportional
to the classifying vector
w. The
variance of individual elements of
n s, R,
which is bounded by
Var[z 1
|
s s
, R] and one. Truncation of a Gaussian can only decrease the variance,
so
Because
||w||
= 1, if there are very many significant weights in
w, they
will have to be small, so
that
Let
a s, R
be the average value of a number
N s, R
of
n s, R.
Any combination of the
form
with positive weights
k s,R
will be an estimate of
w times a
positive constant.
If we have two independent estimates
b and
c of the same quantity
(having the same expected value,
E(b)=E(c)) with variances
σ b2
and
σ c2
, the linear combination of the two estimates with the same expected value and
the smallest variance
is
That is, each estimate should be weighted
inversely by its variance. To obtain a minimum
variance estimate of
w(i)
from a sample with
N s, R
approximately independent samples of each type, the individual estimates of
w(i),
n s, R (i)/ v s, R
should be weighted inversely by their variances, which are approximately
so we should weight each
n s,
R (i)
by
v s, R
. Because the weights do not depend on
i, we can
then just weight
n s, R
by
v s, R
. A good un-normalized estimate of the classifier
w is thus
given by the
v s, R
weighted sums
N s, R
a s, R
,
If we replace
p s, R
in
v s, R
of Equations
(2.5.3.7, 8, 11, 12)
by
p s, R
=
N s, R
/
N s,
where
N s
=
N
s, 0
+
N
s,
1 , and take
advantage of the fact that
we
obtain
The more frequent stimulus should
be given more weight and, because for
p < .5,
φ(Z(p)) increases
monotonically, the more error-prone stimulus should be given more weight. These
weights take into account all the parameters assumed to determine the observer's
performance. If both stimuli are equally frequent and the error rates are equal,
the formula is proportional to
the combination rule originally used by Ahumada
and Beard (Ahumada, 1996; Ahumada & Beard,
1998, 1999;
Beard & Ahumada, 1997,
1998,
2000). In the
next section, we add a third subscript to refer to the observer. The good
weighting scheme for combining average classification images
a s, R,
O over responses, stimuli, and observers can be described as a
sequential process. To combine over responses, just take the
difference,
To then combine images for different stimuli, weight by
factors involving the relative frequencies of the stimuli and the extremeness of
the error proportions for the
stimuli
where
If estimates are to be combined over
M observers, they need to be
weighted by the square root of the observer's proportion of decision variance
due to the external noise,
α2 = 1/(1 + γ2),
and the number of trials run by the observer (which is already included here in
the
N s, O).
Estimates of the internal noise (α or γ) are
needed in order to use the above formula
(2.5.5.10) to combine
estimates over observers. Internal noise estimates are also needed to compute
the variance of an estimate of
w to plan
the number of trials that need to be run (see Equation
2.5.5.2). For this section,
we are using the model of section
2.4, relaxing the linearity
assumptions about the classification function and the external noise to the
assumption that the term
e = w n T
is a standard Gaussian.
The subscripts
i and
j are
added to indicate two separate trials. The response agreement probability for a
particular signal and response with the same noise is denoted
by
To obtain this probability, we will compute it
conditional on the value of e
and then average over the possible values of
e.
Conditional on the value of
e, for
s = 0,
the probability of a response
R = 0 is given
by
For another response to the same signal and the
same noise, the criterion variability is independent, so the probability of two
R = 0 responses is
given
by
The probability of two correct responses
R
i
= R
j
= 0 to the same noise is
then
and the probability of two incorrect responses
R I = R j = 1
to the same noise
is
The equations for arbitrary
s can be
written
Either of these equations can be used to solve for an
estimate of γ (the same estimate results by using either
one).
Ahumada and Beard (1998)
also derived other estimates for γ based on the assumption that
e comes from a known model
(and is distributed as a Gaussian) and that Gaussian internal noise is added by
the observer. This allows falsification of the model if the estimates of γ
are not consistent. The model they tested was a particular parametric linear
filter estimated by
Barth, Beard, and Ahumada (1999), but any
model can be tested using their scheme if the performance of the noiseless model
can be computed for the same noises presented to the observer. The noiseless
model's performance index,
d 0',
is needed as is the trial by trial agreement of the observer and the
model.
One estimate of γ is based on the ratio of the
performance of the model,
d 0',
to that of the observer,
dH'
From (2.4.6) above we
have
so
|
![[home]](/images/article/home.gif)
![[e-mail]](/images/article/mail.gif)
