Effect of Majority Rule and Initial Bias
on Information Aggregation by Groups
R. D. Sorkin,
S. Luan, & J. Itzkowitz
University of Florida
This research was partially supported by the Air
Force Office of Scientific Research.
The problem: You (an art dealer) must decide whether
to buy a possible ‘find’; a sketch by an early
Impressionist.
Your decision is ‘assisted’ by 3 experts who examine
the picture (further tests/opinions are impractical)-
1
Louise
Expertise:
High
Correlation:
with Louis
Bias: vy conserv
Vote:
NO
Louis
Willy
Yourself
Medium
High
Medium
with Louise
less conserv.
YES
none
very liberal
YES
none
medium
-
Expertise
Louise
high
Louis
med
Terrence
high
Yourself
med
Bias
vy consrv
less cons.
vy. liberal
neutral
Correlation 1st vote
with Louis
N
with Louise
Y
uncorrelated
Y
uncorrelated
-
Is there an optimal way to combine their opinions?
You hold a brief teleconference to obtain their final
recommendations about purchasing the art…
-
2
Optimal integration of a group’s information is a
potentially difficult problem.
In some cases, there could be person-by-person
optimization of criteria and responses.
Most people just use a majority rule to aggregate the
information (which may describe some common ‘biases’).
Suppose that repeated votes are taken…
Repeated votes and a majority rule are properties of the
standard American jury.
How do the accuracy & bias of decisions depend on the
parameters of such a system?
-
Quick review of signal detection theory:
µN
Normalized separation =
detection index, d’
µSN
z
zc
z is a likelihood ratio statistic based on the input, x, e.g.,
z = log [λ(x)] = log [f(x|sn)/f(x|n)]
The detector should respond ‘yes’ if z ≥ zc where zc is a
function of the payoffs & prior probabilities.
-
3
Receiver Operating Characteristic
1
All
Hit Probability
P(Conviction|Guilty)
0.8
on
pts
e=
hav
C
RO
d’
Negative zc
Higher d’ & percent correct
0.6
Positive zc
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
False Alarm Probability
P(Conviction|Innocent)
A Distributed Detection
Model of the Jury
σ 12
di' =
µS1 − µS 0
+
s0 or s1 event
2
σ COMMON
σ i2
x1
+
2
σ COMMON
σ N2
2
σ COMMON
+
xN
xi
2
σ i2 + σ COMMON
d’1
d’i
d’N
lnλ ( xi ) = ln[ f ( xi | s1 ) / f ( xi | s0 )]
lnλ(x1)
βi = V ⋅
p( s0 | r1, j r2, j ⋅⋅⋅ rN , j ) j ≠i
p ( s1 | r1, j r2, j ⋅ ⋅⋅ rN , j ) j ≠ i
LDM 1
λ(x
) i)
lnλi(x
lnλ(x1) ≥
lnβ1 ?
r1,0 or r1,1
LDM i
lnλ(x
λ(xNN))
lnλ(xi) ≥
lnβi ?
ri,0 or ri,1
lnλ(xN) ≥
ln βN ?
rN,0 or rN,1
Decision Center
ln[λ( r1,j , r2,j ,…, rN,j)] ≥ ln βC or K of N rule
Optimal rule
“yes if n1 and y2 but not n3”
LDM N
Feedback
Information
about
{d’i , βi}
Jury rule
r0 or r1 response
4
How does the vote feedback influence your vote?
You are detector number 3.
You have a detection index d´3 that depends on
(a) the signal-to-noise properties of the display, and
(b) your own detection expertise
You have an initial criterion c3 that depends on
(a) the prior odds ratio,
(b) the payoff matrix for the vote,
Your observation results in z > c3 so your first vote is yes
-
Having heard the other members’ votes, you now need to make
a second vote.
Members 1, 2, and 4 voted { y1, n2, n4 }
Calculate a new value for the ‘prior’ odds ratio and use that
ratio to calculate a revised criterion.
From Bayes’ theorem:
p ( y1 | n) p (n2 | n) p(n4 | n) p(n)
p (n | y1 , n2 , n4 )
p ( y1 , n2 , n4 | n) p(n)
=
=
p ( sn | y1 , n2 , n4 ) p ( y1 , n2 , n4 | sn) p( sn) p ( y1 | sn) p(n2 | sn) p(n4 | sn) p( sn)
The values for p(yi|n), p(yi|sn), p(ni|n), etc. can be calculated from
your knowledge of the members’ { d1´, d2´, d4´, } and { c1, c2, c4, } -
5
Given several no votes, and few yes votes, the criterion would shift right
to a more ‘conservative’ value of zc.
Bayesian Jury Updating Strategy
f(x|sn), f(x|n)
input (x)
λ(x)=
f(x|sn)
f(x|n)
β=
(Vcorrect − no + V false − alarm ) p(n)
⋅
(Vhit + Vmiss )
p ( sn)
logλ(x)>logβ?
initial vote (yes, no)
6
Bayesian Jury Updating Strategy
f(x|sn), f(x|n)
input (x)
λ(x)=
β=
f(x|sn)
f(x|n)
(Vcorrect − no + V false − alarm ) p(n)
⋅
(Vhit + Vmiss )
p ( sn)
logλ(x)>logβ?
initial vote (yes, no)
other votes {y1,n2,n4}
βF =
(Vcorrect − no + V false − alarm ) p (n | y1 , n2 , n4 )
⋅
(Vhit + Vmiss )
p( sn | y1 , n2 , n4 )
logλ(x)>logβ?
next vote (yes, no)
d´1, d´2, d´4, β1, β2 , β4
repeat
(knowledge of other voters’ expertise & bias)
Delphi Updating Strategy
f(x|sn), f(x|n)
input (x)
λ(x)=
f(x|sn)
f(x|n)
β=
(Vcorrect − no + V false − alarm ) p(n)
⋅
(Vhit + Vmiss )
p ( sn)
logλ(x)>logβ?
initial vote (yes, no)
other votes {y1,n2,n4}
βF =
(Vcorrect − no + V false − alarm ) p (n | y1 , n2 , n4 )
⋅
(Vhit + Vmiss )
p( sn | y1 , n2 , n4 )
d´1 = d´2 = d´4 = µd’ β1 = β2 = β4 = µβ
(knowledge of average voters’ expertise & bias)
logλ(x)>logβ?
next vote (yes, no)
repeat
7
Conforming Strategy
f(x|sn), f(x|n)
input (x)
λ(x)=
f(x|sn)
f(x|n)
β=
(Vcorrect − no + V false − alarm ) p(n)
⋅
(Vhit + Vmiss )
p ( sn)
logλ(x)>logβ?
initial vote (yes, no)
other votes {y1,n2,n4}
If sum of yes votes > no votes,
decrease logβ else increase by some
amount
d´1, d´2, d´4, β1, β2 , β4
(no specific knowledge of other voters’
expertise & bias)
logλ(x)>logβ?
next vote (yes, no)
repeat
Review of Possible Strategies
Rational Strategy
Revise your decision criterion according to the
Bayes’ rule that optimally incorporates knowledge of other
members’ sensitivity, criterion, and vote.
Limited Knowledge (Delphi) Rational Strategy
Revise your decision criterion according to the
Bayes’ rule but use other members’ votes and average
member sensitivity and criterion.
Conforming Strategy
Always shift your criterion in the direction of the
majority vote; if tie, go with the majority of other
members.
8
Expectations About Juror Parameters
Jury size
More jurors -> higher performance
More jurors -> effect on hang rate?
Juror expertise (incl. weight of evidence)
Higher expertise -> higher performance
Initial bias of Jury
Deviation from neutral -> effect on performance?
-> effect on decision bias
Juror diversity:
Between-juror correlation
lower correlation -> higher performance
Variance in member expertise
higher variance -> higher performance
Variance in member bias
higher variance -> effect on performance?
Expectations About Jury Process Variables
Majority required for decision
½, 2/3, ¾, unanimous
-> effect on accuracy of initial (predeliberation) vote
-> effect on accuracy of final vote
-> effect on bias of initial vote
-> effect on bias of final vote
-> effect on hang rate
9
Decision accuracy as a function of group size & majority rule
(Ideal combines continuous ests.)
Group percent correct
1
0.95
Majority rule
0.9
0.5
0.85
0.666
0.8
0.75
0.75
0.7
0.995
0.65
ideal
0.6
0.55
With deliberation,
the unanimous majority
is best & 0.5
the simple majority worst.
3 4 5 6 7 8 9 10 11 12
mean c=0
Group size
µd' =1.0
σd' µc σ c =µd' σd'
= 0.33
G ro u p p ercen t co rrect
Comparison of optimum (Bayes’) updating and
non-deliberation group groups (0.5 is the same).
1
0.95
0.9
0.85
0.8
0.75
0.7
0.65
0.6
0.55
0.5
Majority rule
0.5
0.666
0.75
0.995
0.5
0.666
0.75
0.995
3
4
5
6
7
8
9
10 11 12
With NO deliberation, the unanimous
Group size
majority is worst & the simple
best.
mean c=0
Bottom three curves are for initial vote only.
10
Group percent correct
Effect of extreme initial juror bias…
1
0.95
0.9
0.85
0.8
Majority rule
0.5
0.666
0.75
0.7
0.65
0.6
0.55
0.5
0.75
0.995
3 4
5 6 7
8 9 10 11 12
Group size
mean c=-3
or +3
A unanimous rule reduces the negative effect of juror bias.
Group percent correct
Comparison of optimum and Delphi updating rules.
1
Majority rule
0.95
0.666
0.9
0.75
0.5
0.995
0.85
0.5
0.8
0.666
0.75
0.75
0.995
0.7
3 4 5 6 7
8 9 10 11 12
mean c=0
Group size
The Delphi and Conforming rules are below
opt andthe
delphi
performance of the Bayes’ updating rule.
11
Hang prob. as a function of size, majority rule, & juror bias.
-3
(Hangs defined as
More than x ballots)
-2
m=12
0.015
-1.5
-1
0.01
-0.5
0.005
0
0
0.5
0.5
0.6
0.7
0.8
0.9
1
1
Majority rule
1.5
2
-3
0.02
Prob. of Hang
Few hangs; more
hangs with
smaller juries and
unanimous rules.
Parameter is
initial µ c
Prob. of Hang
0.02
-2
m=6
0.015
-1.5
-1
0.01
-0.5
0.005
0
0.5
0
0.5
0.7
1
0.9
1.5
Majority rule
2
Accuracy as a function of majority rule and juror bias.
4
Initial µ c
3.5
-2
Group d'
3
-1.5
0
±0.5
2.5
-1
-0.5
±1
2
0
0.5
1.5
1
±1.5
1
Best performance
is with neutral bias and strict majorities.
1.5
A strict0.5
rule mediates the negative effect of juror bias. 2
0
0.5
0.6
0.7
0.8
Majority rule
0.9
1
size=12
12
Decision bias as a function of majority rule and juror bias.
Initial µ c
5
-3
4
-2
Group criterion
Group
criterion
3
3
2
2
-1.5
-1
1
-0.5
0
-1
0
0.5
0.6
0.7
0.8
0.9
1
1
-2
-3
-2
-3
0.5
1.5
2
-4
3
-5
Majority rule
size=12
Stricter majorities mediate the effects of non-neutral biases.
Juries & Strict Majorities
The major consequence of a strict majority rule is improved
accuracy (not a more conservative decision). A strict rule
may accomplish this by fostering deliberation (consideration
of other members’ opinions).
The second effect of a strict majority rule is to reduce the
effect of juror bias on the accuracy and bias of the jury’s
decision.
It appears to be a near optimal system, so long as (a) jurors
employ a sensible updating strategy, (b) jurors’ estimates are
not correlated (in a non-uniform way), and (c) jurors are
motivated to make the most accurate decision.
If it ain’t broke…
13
References
Pete, A., Pattipati, K.R., & Kleinman, D.L. (1993). Optimal team and
individual decision rules in uncertain dichotomous situations. Public
Choice, 75, March, 205-230.
Sorkin, R. D. & Dai, H. (1994). Signal detection analysis of the ideal group.
Organizational Behavior and Human Decision Processes, 60, 1-13.
Sorkin, R. D., Hays, C.J., & West, R. (2001). Signal detection analysis of
group decision making. Psychological Review, 108, 183-203.
Swaszek, P. E., & Willett, P. (1995). Parley as an approach to distributed
detection. IEEE Transactions on Aerospace and Electronic Systems, 31, 1,
447-457.
Viswanathan, R. & Varshney, P. K. (1997) Distributed detection with
multiple sensors: Part I—Fundamentals, Proceedings of the IEEE,
85, 54-63.
Questions?
Effect of bias in juror variance on percent correct
1
0.95
Group percent correct
0.9
0.99
0.75
0.85
0.8
0.5
0.75
0.7
σc = 0.33
σc = 0.66
0.65
0.6
0.55
0.5
3
4
5
6
7
8
9
10
11
12
Group size
14