Crowdsourcing
and All-Pay Auctions
Milan Vojnović
Microsoft Research
Joint work with Dominic DiPalantino
UC Berkeley, July 13, 2009
Examples of Crowdsourcing
• Crowdsourcing = soliciting solutions via open calls to
large-scale communities
– Coined in a Wired article (’06)
• Taskcn
– 530,000 solutions posted for 3,100 tasks
• Innocentive
– Over $3 million awarded
• Odesk
– Over $43 million brokered
• Amazon’s Mechanical Turk
– Over 23,000 tasks
2
Examples of Crowdsourcing (cont’d)
• Yahoo! Answers
– Lunched Dec ’05
– 60M users / 65M answers (as of Dec ’06)
• Live QnA
– Lunched Aug ’06 / closed May ’09
– 3M questions / 750M answers
• Wikipedia
3
Incentives for Contribution
• Incentives
– Monetary
$$$
– Non-momentary
Social gratification and publicity
Reputation points
Certificates and “levels”
• Incentives for both participation and quality
4
Incentives for Contribution (cont’d)
Contest duration
Number of submissions
Number of registrants
Number of views
• Ex. Taskcn
Reward range (RMB)
100 RMB  $15 (July 09)
5
Incentives for Contribution (cont’d)
• Ex. Yahoo! Answers
Points
Source: http://en.wikipedia.org/wiki/Yahoo!_Answers
Levels
6
Questions of Interest
• Understanding of the incentive schemes
– How do contributions relate to offered rewards?
• Design of contests
– How do we best design contests?
– How do we set rewards?
– How do we best suggest contests to players and
rewards to contest providers?
7
Strategic User Behavior
User Strategies on Taskcn.com
User Strategies on Taskcn.com
• From empirical analysis of Taskcn by Yang et al (ACM EC ’08) –
(i) users respond to incentives, (ii) users learn better strategies
– Suggests a game-theoretic analysis
8
Outline
• Model of Competing Contests
• Equilibrium Analysis
– Player-Specific Skills
– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion
9
Single Contest Competition
c1
c2
R
c3
c4
players
contest offering
reward R
ci = cost per unit effort or quality produced
10
Single Contest Competition (cont’d)
Outcome
c1
b1
c2
b2
c3
c4
-c1b1
b3
b4
R - c2b2
R
-c3b3
-c4b4
11
All-Pay Auction
Outcome
v1
b1
v2
b2
v3
v4
-b1
b3
v2 - b2
-b3
b4
-b4
Everyone pays their bid
12
Competing Contests
1
R1
2
...
R2
u
vu , j
...

vu  (vu ,1 ,,vu , J )
...
...
Rj
N
RJ
users
contests
13
Incomplete Information Assumption
Each user u knows
N

vu
= total number of users
= his own skill
F
= skills are randomly drawn from F
We assume F is an atomless distribution
with finite support [0,m]
14
Assumptions on User Skill
1) Player-specific skill

v  (vu ,,vu )
vu
random i.i.d. across u
(ex. contests require similar skills or skill
determined by player’s opportunity cost)
2) Contest-specific skill

v  (vu ,1 ,,vu , J )
vu , j
random i.i.d. across u and j
(ex. contests require diverse skills)
15
Bayes-Nash Equilibrium
• Mixed strategy
 u , j (v)
bu , j
= prob. of selecting a contest of class j
= bid
• Equilibrium
Select contest of highest expected profit
where expectation with respect to “beliefs”
about other user skills
Contest class = set of contests that offer same reward
16
User Expected Profit
• Expected profit for a contest of class j
v


g j (v)  R j  1  p F (x)
c
j j
N 1
dx
0
p j = prob. of selecting a
contest of class j
Fj () = distribution of user skill
conditional on having
selected contest class j
g j (v)  E(g j (v , M))
v


n
n
g j (v , n)  R j vu , j Fj (v)   xFj (dx )
0


M ~ Bin(N  1, p j )
17
Outline
• Model of Competing Contests
• Equilibrium Analysis
– Player-Specific Skills
– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion
18
Equilibrium Contest Selection
m
1
2
1
v2
v3
3
4
v4
2
3
4
0
5
skill
levels
contest
classes
19
Threshold Reward
• Only K highest-reward contest classes selected
with strictly positive probability
1
 



1
~

N 1
H[1,i ] (R )
K  max i : Ri  1 





 J[1,i ] 


  J  1  1
H A (R )    J Ak Rk N 1 
 kA

Jk = number of contests of class k
J A   Jk
kA
20
Partitioning over Skill Levels
• User of skill v is of skill level l if
v  [v l ,v l 1)
where
N 1


R
~
l



F (v l )  1  J[1,l ] 1 
, for l  1,, K
 H[1,l ] (R ) 


1
~
v l  0, for l  K,, K
21
Contest Selection
• User of skill l, i.e. with skill v  [v l ,v l 1) selects a
contest of class j with probability
 R j N 1
 l

 j (v )    Rk N11
 k 1
 0
1
j  1,, l
j  l  1,, K
22
Participation Rates
• A contest of class j selected with probability

 
 H[1,K~ ] (R )
1

1  1 
1


J
~
pj   
[1,K ] 
R jN 1


0
~
j  1,, K
~
j  K  1,, K
• Prior-free – independent of the distribution F
23
Large-System Limit
• For positive constants , k , k , k  1,,K
N
limN   
J
Jk
limN 
k
J
limN  Npk  k
where K is a finite number of contest classes
R1  R2    RK
24
Skill Levels for Large System
• User of skill v is of skill level l if
v  [v l ,v l 1)
where
l
 [1 ,l ]
F (v l )  1 
log

 Rk k
 / [ 1 ,l ]
k 1
Rl
, for l  1,, K~
v l  0, for l  K~  1,, K
25
Participation Rates for Large System
• Expected number of participants for a contest of class j
Rj
 
 log K~

 [1,K~ ]
 k /  [ 1,K~ ]
j  
Rk

k 1


0
~
j  1,, K
~
j  K  1, ,K
i


 k / [ 1 ,i ]   [1 ,i ] 
~
K  max i : Ri    Rk
e

 k 1



• Prior-free – independent of the distribution F
26
Contest Selection in Large System
• User of skill l, i.e. with skill v  [v l ,v l 1) selects a
contest of class j with probability
 1

 j (v)   J[1,l ]

 0
j  1,, l
j  l  1,, K
m
1
2
1/3
1
1/3
2
3
1/3
4
0
3
4
5
• For large systems, what
matters is which contests
are selected for given skill
27
Proof Hint for Player-Specific Skills
g1(v)
g2(v)
R1  R2  R3  R4
g3(v)
g4(v)
0
v3
v2
v1
m
v
• Key property – equilibrium expected payoffs as showed
28
Outline
• Model of Competing Contests
• Equilibrium Analysis
– Player-Specific Skills
– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion
29
Contest-specific Skills
• Results established only for large-system limit
• Same equilibrium relationship between
participation and rewards as for playerspecific skills
30
Proof Hints
• Limit expected payoff – For each
v [0, m]

limN  g j (v)  R j e j v
• Balancing – Whenever  j  0
Rje
 j
 Rk e k , for all k
• Asserted relations for
(1 ,, K )
follow from above
31
Outline
• Model of Competing Contests
• Equilibrium Analysis
– Player-Specific Skills
– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion
32
System Optimum Rewards
SYSTEM
K


 kUk (k (R))   kC k (R)
K
maximise
k 1
over
k 1

R R
K
subject to

k 1
k
k  
• Set the rewards so as to optimize system welfare
33
Example 1: zero costs
(non monetary rewards)
Assume Uk () are increasing strictly concave functions. Under
player-specific skills, system optimum rewards:
 U ( ) 

R j  c 1 

N


' 1
j
 (N 1)
, j  1,, K
for any c > 0 where  is unique solution of
K
' 1

U
 k k ( )  
k 1
• Rewards unique up to a multiplicative constant – only
relative setting of rewards matters
34
Example 1 (cont’d)
• For large systems
Assume Uk () are increasing strictly concave functions. Under
player-specific skills, system optimum rewards:
R j  ce
U 'j1 (  )
, j  1,, K
for any c > 0 where  is unique solution of
K
' 1

U
 k k ( )  
k 1
35
Example 2: optimum effort
• Consider SYSTEM with
Utility:


U j ( j (R))  Vj ( j ( j (R)))
{

   j (R )
 j ( j (R))  R j m(1  (1   j (R))e
)
exerted effort
{
{
Cost:


  j (R )
C j (R)  (1  e
)D j (R j )
prob. contest cost of
attended giving Rj
(budget constraint)
36
Outline
• Model of Competing Contests
• Equilibrium Analysis
– Player-Specific Skills
– Contest-Specific Skills
• Design of Contests
• Experimental Validation
• Conclusion
37
Taskcn
• Analysis of rewards and participation across
tasks as observed on Taskcn
– Tasks of diverse categories: graphics, characters,
miscellaneous, super challenge
– We considered tasks posted in 2008
38
Taskcn (cont’d)
reward
number of
views
number of
registrants
number of
submissions
39
Submissions vs. Reward
Graphics
Characters
Miscellaneous
• Diminishing increase of submissions with reward
linear regression
40
Submissions vs. Reward
for Subcategory Logos
• Conditional on the rate at which users submit solutions
any rate
once a month
every fourth day
every second day
• Conditioning on the more experienced users, the
better the prediction by the model
model
41
Same for the Subcategory 2-D
any rate
model
once a month
every fourth day
every second day
42
Conclusion
• Crowdsourcing as a system of competing contests
• Equilibrium analysis of competing contests
– Explicit relationship between rewards and participations
• Prior-free
– Diminishing increase of participation with reward
• Suggested by the model and data
• Framework for design of crowdsourcing / contests
• Base results for strategic modelling
– Ex. strategic contest providers
43
More Information
• Paper: ACM EC ’09
• Version with proofs: MSR-TR-2009-09
– http://research.microsoft.com/apps/pubs/default.
aspx?id=79370
44