Approval-rating systems that never
reward insincerity
COMSOC ’08
3 September 2008
Rob LeGrand
Washington University in St. Louis
(now at Bridgewater College)
[email protected]
Ron K. Cytron
Washington University in St. Louis
[email protected]
Approval ratings
2
Approval ratings
• Aggregating film reviewers’ ratings
–
–
–
–
Rotten Tomatoes: approve (100%) or disapprove (0%)
Metacritic.com: ratings between 0 and 100
Both report average for each film
Reviewers rate independently
3
Approval ratings
• Online communities
–
–
–
–
Amazon: users rate products and product reviews
eBay: buyers and sellers rate each other
Hotornot.com: users rate other users’ photos
Users can see other ratings when rating
• Can these “voters” benefit from rating insincerely?
4
Approval ratings
5
Average of ratings

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0.4, 0.7, 0.8, 0.8, 0.9
outcome:

f avg (v )  0.72
0.72
1
0
data from Metacritic.com: Videodrome (1983)
6
Average of ratings

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0.7, 0.8, 0.8, 0.9
outcome:

f avg (v )  0.64
0.64
1
0
Videodrome (1983)
7
Another approach: Median

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0.4, 0.7, 0.8, 0.8, 0.9
outcome:

f med (v )  0.8
0.8
1
0
Videodrome (1983)
8
Another approach: Median

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0.7, 0.8, 0.8, 0.9
outcome:

f med (v )  0.8
0.8
1
0
Videodrome (1983)
9
Another approach: Median
• Immune to insincerity
– voter i cannot
obtain a better result by voting vi  ri 

– if f med (v )  vi , increasing vi will not change f med (v )


– if f med (v )  vi , decreasing vi will not change f med (v )
• Allows tyranny by a majority

v  0, 0, 0,1,1,1,1
–

– f
med (v )  1
– no concession to the 0-voters
10
Declared-Strategy Voting
[Cranor & Cytron ’96]
cardinal
preferences
rational
strategizer
ballot
outcome
election
state
11
Declared-Strategy Voting
[Cranor & Cytron ’96]
sincerity
cardinal
preferences
strategy
rational
strategizer
ballot
outcome
election
state
• Separates how voters feel from how they vote
• Levels playing field for voters of all sophistications
• Aim: a voter needs only to give sincere preferences
12
Average with Declared-Strategy Voting?
• Try using Average protocol in DSV context
cardinal
preferences
rational
strategizer
ballot
outcome
election
state
• But what’s the rational Average strategy?
• And will an equilibrium always be found?
13
Rational [m,M]-Average strategy
• Allow votes between m  0 and M  1
• For 1  i  n, voter i should choose vi to move
outcome as close to ri as possible

f avg (v )  ri
v
• Choosing vi  ri n 
would
give
j
j i
v , m), M )
• Optimal vote is vi  min(max( ri n 
j i j


• After voter i uses this strategy, one of these is true:

– f avg (v )  ri and vi  M

– f avg (v )  ri

– f avg (v )  ri and vi  m
14
Equilibrium-finding algorithm

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0.4, 0.7, 0.8, 0.8, 0.9
0.72
1
0
Videodrome (1983)
15
Equilibrium-finding algorithm

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0, 0, 0, 0
0
1
16
Equilibrium-finding algorithm

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0, 0, 0,1
0.2
0
1
17
Equilibrium-finding algorithm

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0, 0,1,1
0.4
0
1
18
Equilibrium-finding algorithm

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0,1,1,1
0.6
0
1
19
Equilibrium-finding algorithm

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0.5,1,1,1
equilibrium!
0.7
0
1
• Is this algorithm guaranteed to find an equilibrium?
20
Equilibrium-finding algorithm

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0.5,1,1,1
equilibrium!
0.7
0
1
• Is this algorithm guaranteed to find an equilibrium?
• Yes!
21
Expanding range of allowed votes

r  0.4, 0.7, 0.8, 0.8, 0.9

v  1, 1, 2, 2, 2
1
0.8
2
• These results generalize to any range
22
Multiple equilibria can exist

r  0.4, 0.7, 0.7, 0.8, 0.9

v  0, 0.5,1,1,1

v  0, 0.6, 0.9,1,1

v  0, 0.75, 0.75,1,1
outcome in each case:

f avg (v )  0.7
• Will multiple equilibria always have the same average?
23
Multiple equilibria can exist

r  0.4, 0.7, 0.7, 0.8, 0.9

v  0, 0.5,1,1,1

v  0, 0.6, 0.9,1,1

v  0, 0.75, 0.75,1,1
outcome in each case:

f avg (v )  0.7
• Will multiple equilibria always have the same average?
• Yes!
24
Average-Approval-Rating DSV

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0.4, 0.7, 0.8, 0.8, 0.9
outcome:

f aveq (v , 0,1)  0.7
0.7
1
0
Videodrome (1983)
25
Average-Approval-Rating DSV

r  0.4, 0.7, 0.8, 0.8, 0.9

v  0, 0.7, 0.8, 0.8, 0.9
outcome:

f aveq (v , 0,1)  0.7
0.7
0
1
• AAR DSV is immune to insincerity in general
26
Evaluating AAR DSV systems
• Expanded vote range gives
 wide range of AAR
DSV systems:  a ,b (v ) 0  a  1 0  b  1
• If we could assume sincerity, we’d use Average
• Find AAR DSV system that comes closest
• Real film-rating data from Metacritic.com
– mined Thursday 3 April 2008
– 4581 films with 3 to 44 reviewers per film
– measure root mean squared error
• Perhaps we can come much closer to Average than
Median or [0,1]-AAR DSV does
27
Evaluating AAR DSV systems
b  0.5
RMSE a , 0.5
a
minimum at
a  0.3240
28
Evaluating AAR DSV systems: hill-climbing
b  0.4820
RMSE a , 0.4820
a
minimum at
a  0.3647
29
Evaluating AAR DSV systems: hill-climbing
a  0.3647
RMSE 0.3647,b
b
minimum at
b  0.4820
30
Evaluating AAR DSV systems

 0.3647,0.4820 (v )

f avg (v )
31
AAR DSV: Future work
• New website: trueratings.com
– Users can rate movies, books, each other, etc.
– They can see current ratings without being tempted to
rate insincerely
– They can see their current strategic proxy vote
• Richer outcome spaces
– Hypercube: like rating several films at once
– Simplex: dividing a limited resource among several uses
– How assumptions about preferences are generalized is
important
Thanks! Questions?
32
What happens at equilibrium?
• The optimal strategy recommends that no voter
change
• So (i) v  ri  vi  1
• And (i) v  ri  vi  0
– equivalently,
(i) vi  0  v  ri
• Therefore any average at equilibrium must satisfy
two equations:
– (A)
– (B)
v n  i : v  ri 
i : v  ri   v n
33
Proof: Only one equilibrium average
A( )  n  i :   ri 
B( )  i :   ri   n
• Theorem:
A(1 )  B(1 )  A(2 )  B(2 )  1  2
• Proof considers two symmetric cases:
– assume
– assume
1  2
2  1
• Each leads to a contradiction
34
Proof: Only one equilibrium average
case 1:
1  2
(i) 2  ri  1  ri
i : 2  ri   i : 1  ri 
i : 2  ri   i : 1  ri 
2n  i : 2  ri 
A(2 )
B(1 )
i : 1  ri   1n
2n  i : 2  ri   i : 1  ri   1n
2n  1n
2  1 , contradicting 1  2
35
Proof: Only one equilibrium average
Case 1 shows that
1  2
Case 2 is symmetrical and shows that
Therefore
1  2
2  1

Therefore, given r , the average at equilibrium is unique
36
An equilibrium always exists?

• At equilibrium, v must satisfy
(i ) vi  min(max( ri n   j i v j , m), M )

Given a vector r , at least one equilibrium indeed
always exists.
A particular
algorithm
will
always
find
an
equilibrium

for any r . . .
37
An equilibrium always exists!
Equilibrium-finding algorithm:

• sort r so that (i  j ) ri  rj
• for i = 1 up to n do
vi  min(max( ri n  k i vk  (n  i)m, m), M )
(full proof and more efficient algorithm in dissertation)
• Since an equilibrium always exists,
average at

equilibrium is a function, f aveq (r , m, M ) .


• Applying f aveq to v instead of r gives a new
system, Average-Approval-Rating DSV.
38
Average-Approval-Rating DSV
• What if, under AAR DSV, voter i could gain an
outcome closer to ideal by voting insincerely
( vi  ri )?
• It turns out that Average-Approval-Rating DSV is
immune to strategy by insincere voters.
•

,
M
)

v
Intuitively, if f aveq (v , m
i, increasing v i

will not change f aveq (v , m, M ) .
39
AAR DSV is immune to strategy
• If

f aveq (v , m, M )  vi  ri,
• If
f aveq (v , m, M )  vi  ri,

– increasing vi will not change f aveq (v , m, M ).

– decreasing vi will not increase f aveq (v , m, M ) .


– increasing vi will not decrease f aveq (v , m, M ) .

– decreasing vi will not change f aveq (v , m, M ) .
(complete proof in dissertation)
• So voting sincerely ( v  r ) is guaranteed to
i
i
optimize the outcome from voter i’s point of view
40
Parameterizing AAR DSV
• [m,M]-AAR DSV can be parameterized nicely using
a and b, where 0  a  1 and 0  b  1:
1
a
M m
m
b
1 M  m
b
1 b
m b
M b
a
a

b
1 b 

 a ,b (v )  lim f aveq  v , b  , b 

xa
x
x 

41
Parameterizing AAR DSV
• For example:


1,b (v )  f aveq (v , 0,1)


 1 1 (v )  f aveq v ,  1, 2
,
3 2 

 1 1 (v )  f aveq v ,  10,11
,
21 2


 1 (v )  f med v 
0,
2 

 0, 0 (v )  max v 


 0,1 (v )  min v 
42
Evaluating AAR DSV systems
• Real film-rating data from Metacritic.com
– mined Thursday 3 April 2008
– 4581 films with 3 to 44 reviewers per film
0  a 1
0  b 1


 2
SEa ,b v    a ,b v   f avg v 
RMSE a ,b V  


v  SEa ,b v 


vV

v

vV
43
Higher-dimensional outcome space
• What if votes and outcomes exist in d  1
dimensions?
2


x
,
y


: 0  x  1 0  y  1
• Example:
• If dimensions are independent, Average, Median
and Average-approval-rating DSV can operate
independently on each dimension


– Results from one dimension transfer
44
Higher-dimensional outcome space
• But what if the dimensions are not independent?
– say, outcome space is a disk in the plane:
x, y  

: x2  y2  1
• A generalization of Median: the Fermat-Weber point
2
[Weber ’29]
– minimizes sum of Euclidean distances between outcome
point and voted points
– F-W point is computationally infeasible to calculate
exactly [Bajaj ’88] (but approximation is easy [Vardi ’01])
– cannot be manipulated by moving a voted point directly
away from the F-W point [Small ’90]
45