6 T
.S. Wallsten, A. Diederich Mathematical Social Sciences 41 2001 1 –18 Table 2
Possible assumptions about inter-judge relationships among the X ; and their association with the DM’s
j
uncertainty about the kind of shared information DM assumes X identically distributed
j
Yes No
Level of dependence Constant for
Conditionally Conditionally
Conditionally event i
independent pairwise
dependent independent
Case 1
2 3
4
proportion of events judge j assigns to the categories that are true. With a few assumptions, one can expect averaging to improve the discrimination index, DI. Case 4
is the most interesting and realistic case and awaits further work. Next, we present our results for Case 1 and then for Cases 2 and 3.
4. A mathematical framework
Throughout the following we consider the overt estimate and covert confidence of judge j with respect to one particular event, say i. However, it becomes rather messy to
i i
include the event in the notation, e.g., Eq. 1 becomes R 5 h X ,E . As no
j j
j j
confusion will arise, we omit explicitly referencing the event. Further, to allow the DM to transform response scales, we introduce a function f that may convert the overt
responses, for example, from probability into log-odds. f may be any continuous strictly increasing finitely bounded function.
4.1. Case 1 Our result in this case suggests that averaging multiple estimates or their transforma-
tions will improve their accuracy depending on the response function, h .
j
Define •
ER ; 1 Jo E fR , j 5 1, . . . ,J, as the average expected transformed response
J j
j
of J judges to event i; •
M ; 1 Jo fh x ,e , j 5 1, . . . ,J with e [ E , as the observed mean trans-
J j
j j
j j
j
formed response of J judges to event i. For the model presented in Eq. 1 consider the following assumptions:
A1. For a given item i the x 5 x are constant for all judges j, j 5 1, . . . ,J;
j 2
A2. E , j 5 1, . . . ,J, are independent random variables with EE 5 0 and s ;
j j
A3. fR , j 5 1, . . . ,J, are random variables with finite mean and finite variance.
j
T .S. Wallsten, A. Diederich Mathematical Social Sciences 41 2001 1 –18
7
` 2
Theorem 1. Case 1 If A1– A3 hold and if o
Var[ fR ] j , ` then as J →
`
j 51 j
M →
ER a.s.
J J
Proof. Since the x are constant and E are independent, R 5 h x,E are independent,
j j
j j
j 5 1, . . . ,J. The proof is completed by Kolmogorov’s strong law of large numbers. 4.2. Cases 2 and 3
Although Theorem 1 applies to Cases 2 and 3 as well, it is not useful because in these cases events are unique and therefore only either true or false. For convenience the states
of the event are denoted as S 5 1 and S 5 0, for true and false events, respectively. As
i i
will be shown, with suitable assumptions, averaging leads to highly diagnostic estimates. Let:
• l, l 5 0,1, . . . ,N , index the N 1 1 underlying probability categories used by judge j,
j j
with N being even;
j
• X be a discrete random variable taking on values x
in 0,1, representing the
j jl
proportion of events judge j assigns to category l that are true. Without loss of generality let 0 , x , x , ? ? ? , x
, 1.
j 0 j 1
j,N
j
The claim that covert confidence is categorical with an odd number of categories is descriptively compelling, but we could structure the proof without it. The descriptive
merit of defining X in such a way arises from the fact that most judges provide
j
probability estimates in multiples of 0.1 or 0.05 see, for example, Wallsten et al., 1993. However, we include the claim not because of its cognitive content, but as a
mathematical convenience that simplifies the statements of A4 and A5 below and to some degree the proof. Moreover, the claim itself is not very restrictive. First, note that
it explicitly allows judges to have different numbers of categories. Second, the requirement that the number of categories be odd is of no substantive consequence; the
probability of the center category or of symmetric pairs of categories, for that matter may be 0.
Now define: •
E ; 1 Jo E fR uS 5 1 as the average expected transformed response over
J,S 51 j
j i
i
J judges to event i, given the event is true; •
E ; 1 Jo E fR uS 5 0 as the average expected transformed response over
J,S 50 j
j i
i
J judges to event i, given the event is false; •
M ; 1 Jo fh x ,e with e [ E as the observed mean transformed response
J j
j j
j j
j
of J judges to event i. For the model presented in Eq. 1 consider the following assumptions:
A19. For a given event state, S 5 0 or S 5 1, X , j 5 1, . . . ,J, are pairwise
i i
j
independent random variables;
8 T
.S. Wallsten, A. Diederich Mathematical Social Sciences 41 2001 1 –18
A4. The values x of X have the following properties for judge j:
jl j
1. they are symmetric about 0.5, i.e. x 1 x 5 1 for l 5 0, . . . ,N 2 2 1, and
jl j,N 2l
j
j
x 5 0.5; and
j,N 2
j
2. the probabilities are equal for symmetric pairs, i.e. PX 5 x 5 PX 5 x for
j jl
j j,N 2l
j
l 5 0, . . . ,N 2 2 1; with, of course
j N
j
0 PX 5 x , 1 and
O
PX 5 x 5 1.
j jl
j jl
l 50
A5. The expected transformed response conditional on underlying confidence over trials, E fR
uX 5 x , obeys the following properties:
j j
jl
1. E fR
uX 5 x 5 f0.5w 1 1 2 w fx , 0 , w , 1;
j j
jl jl
jl jl
jl
2. E fR
uX 5 x 2 f0.5 5 f0.5 2 E fR uX 5 x ⇔
E fR uX 5 x
j j
jl j
j j,N 2l
j j
jl
j
1 E fR uX 5 x
5 2f0.5, for l
j j
j,N 2l
j
N
j
] 5 0, . . . ,
2 1; and E fR uX 5x
5f0.5.
j j
j,N 2
j
2 Let us first discuss the reasonableness of the axioms.
6
We consider A19 and A2 together. A2, that the E are independent, is reasonable. A19,
j
however, is unlikely to be fully met in practice. All we can ever observe are measures of conditional dependency among the R as indices of possible violations of A19 and A2
j
taken jointly. The practical question is, how robust are the effects of averaging with respect to violations? This is an empirical issue to which the data in the Introduction are
relevant, and which we will discuss subsequently. A4 and A5 are symmetry conditions. A4 says that each judge’s confidence categories
are structured symmetrically around the central one and are used with symmetric probabilities. A5 says that the effects of variation are symmetric for complementary
confidence categories and, moreover, that the variation is regressive with regard to expected transformed responses. Despite the extensive literature on subjective prob-
ability estimation, pure evidence on A4 and A5 does not exist. The reason is that both axioms refer to the latent and therefore unobservable variable, X , whereas evidence by
j
its very nature is observable, in this case distributions of R . A specific theory with its
j
6
A reviewer of an earlier version of this paper raised the possibility that judges’ errors are correlated, perhaps due to overlapping information sets, common training, or the like, thereby violating A2. Note, however, that
in our formulation, such dependencies are absorbed in X , not in E .
j j
T .S. Wallsten, A. Diederich Mathematical Social Sciences 41 2001 1 –18
9
own possibly questionable axioms would be necessary to make inferences about X
j
from R . Nevertheless, it is reassuring that empirical distributions of R tend to be
j j
symmetric and that complementary events elicit complementary confidence categories Wallsten et al., 1993; Tversky and Koehler, 1994; Ariely et al., 2000, suggesting that
A4 and A5 are reasonable. Moreover, A4 is likely to be true for the full population of events in a domain. This is because event populations are closed under complementa-
tion, so that for each true event in the domain, there is also a complementary false event obtained by negation or rephrasing of the statement. A4 follows as a consequence.
Similarly, A4 will hold for any sample that includes true and false versions of each item.
´ Such conditions can be arranged in experimental contexts, as Wallsten and Gonzalez-
Vallejo 1994 did, but do not necessarily obtain in real-world situations. Indeed, Lemma 1, which is required for the theorem and is proved in Appendix A,
states that if A4 holds, then PS 5 1 5 PS 5 0 5 0.5. In other words, if A4 holds
i i
exactly, the set being judged must contain an equal number of true and false items. And conversely, if it does not, then A4 must be violated. Because event complementarity is
so easy to arrange, this is not in principle an onerous requirement. Let us now turn to the theorem itself.
Theorem 2. Cases 2 and 3 If A19, A2, A3, A4, and A5 hold, then for a given event, i, for J judges,
1. E . f0.5 and E
, f0.5; and if, in addition
J,S 51 J,S 50
i i
a sup E
u fR 2 E[ fR ]u , `
j .0 j
j `
2
b o
Var[ fR ] j , `
j 51 j
2. then as J →
` 1,
if M . f0.5,
J
PS 5 1 uM
→
H
i J
0, if M , f0.5.
J
Proof. The proof is in Appendix A.
Note that with Assumption A19 the theorem applies directly to Case 3. But any result for Case 3 also holds for Case 2. On the face of it, Theorem 2 is quite remarkable: given
the axioms, the expected mean transformed probability estimates of true events are greater than the transformation of 0.5; and those of false events are less than the
transformation of 0.5. Consequently if the fR satisfy the indicated boundedness
j
conditions, as the number of judges increases, the probability that an event is true approaches 1 if its mean transformed estimate exceeds the middle of the scale and
approaches 0 if it is below. Or, in other words, if the axioms apply, then all the DM requires to be virtually certain that an event is true or false is a sufficient number of
judges.
5. Discussion
