Research
Prediction Markets and the
Wisdom of Crowds
David Pennock
Yahoo! Research NYC
Research
Outline
• Prediction Markets Survey
• What is a prediction market?
• Examples
• Some research findings
• The Wisdom of Crowds:
A Story
Research
Bet = Credible Opinion
Hillary Clinton will win the election
“I bet $100 Hillary will win at 1 to 2 odds”
• Which is more believable?
More Informative?
• Betting intermediaries
• Las Vegas, Wall Street, Betfair, Intrade,...
• Prices: stable consensus of a large
number of quantitative, credible opinions
• Excellent empirical track record
Research
Example: March Madness
Research
More Socially Redeemable
Example
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
http://intrade.com
Screen capture 2007/05/18
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Research
A Prediction Market
• Take a random variable, e.g.
Bird Flu Outbreak US 2007?
(Y/N)
• Turn it into a financial instrument
payoff = realized value of variable
I am entitled to:
$1 if
Bird Flu
US ’07
$0 if
Bird Flu
US ’07
Research
Why?
Get information
• price  probability of uncertain event
(in theory, in the lab, in the field, ...more later)
• Is there some future event you’d like
to forecast?
A prediction market can probably help
Research
A Prediction Market
• Take a random variable, e.g.
2008 CA
Earthquake?
US’08Pres =
Dem?
=6?
• Turn it into a financial instrument
payoff = realized value of variable
I am entitled to:
$1 if
=6
$0 if
6
Research
Aside: Terminology
• Key aspect: payout is uncertain
• Called variously: asset, security, contingent
claim, derivative (future, option), stock,
prediction market, information market,
gamble, bet, wager, lottery
• Historically mixed reputation
• Esp. gambling aspect
• A time when options were frowned upon
• But when regulated serve important social
roles...
Research
Getting Information
• Non-market approach: ask an expert
I•amHow
entitledmuch
to:
$1 if
would you pay for this?
=6
$0 if
6
• A: $5/36  $0.1389
•
•
•
•
caveat: expert is knowledgeable
caveat: expert is truthful
caveat: expert is risk neutral, or ~ RN for $1
caveat: expert has no significant outside stakes
Research
Getting Information
• Non-market approach: pay an expert
• Ask the expert for his report r of the probability
P(
=6 )
• Offer to pay the expert
• $100 + log r
• $100 + log (1-r)
if
if
=6
6
“logarithmic scoring rule”,
a “proper” scoring rule
• It so happens that the expert maximizes expected
profit by reporting r truthfully
•
•
•
•
caveat: expert is knowledgeable
caveat: expert is truthful
caveat: expert is risk neutral, or ~ RN
caveat: expert has no significant outside stakes
Research
Getting Information
• Market approach: “ask” the public—experts & nonexperts alike—by opening a market:
I am entitled to:
$1 if
=6
$0 if
6
• Let any person i submit a bid order:
an offer to buy qi units at price pi
• Let any person j submit an ask order:
an offer to sell qj units at price pj
(if you sell 1 unit, you agree to pay $1 if
= 6)
• Match up agreeable trades (many poss. mechs...)
Non-Market Alternatives vs. Markets
 Opinion poll
Sampling
No incentive to be
truthful
Equally weighted
information
Hard to be real-time
 Ask Experts
Identifying experts can
be hard
Incentives
Combining opinions
can be difficult
 Prediction Markets
Self-selection
Monetary incentive
and more
Money-weighted
information
Real-time
Self-organizing
Non-Market Alternatives vs. Markets
 Machine
learning/Statistics
Historical data
Past and future are
related
Hard to incorporate
recent new information
 Prediction Markets
No need for data
No assumption on past
and future
Immediately incorporate
new information
Function of Markets 2:
Risk Management
 If
is bad for me,
I buy a bunch of
$1 if
$0 otherwise
 If my house is struck by lightening, I am
compensated.
Risk Management Examples
Insurance
I buy car insurance to hedge the risk of
accident
Futures
Farmers sell soybean futures to hedge the
risk of price drop
Options
Investors buy options to hedge the risk of
stock price changes
Financial Markets vs. Prediction Markets
Financial Markets
Prediction Markets
Primary
Social welfare (trade)
Hedging risk
Information aggregation
Secondary
Information aggregation
Social welfare (trade)
Hedging risk
Giving/Getting Information
• What you can say/learn • Where
% chance that
– Hillary wins
– GOP wins Texas
– YHOO stock > 30
– IEM, Intrade.com
– Intrade.com
– Stock options market
– Duke wins tourney
– Oil prices fall
– Heat index rises
– Las Vegas, Betfair
– Futures market
– Weather derivatives
– Hurricane hits Florida
– Rains at place/time
– Insurance company
– Weatherbill.com
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
http://intrade.com
http://tradesports.com
Screen capture 2007/05/18
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Research
Intrade Election Coverage
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Research
http://www.biz.uiowa.edu/iem
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
http://www.wsex.com/
http://www.hedgestreet.com/
Screen capture 2007/05/18
Screen capture 2007/05/18
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Play money;
Real predictions
http://www.hsx.com/
http://www.ideosphere.com
Cancer
cured
by 2010
Machine Go
champion
by 2020
http://us.newsfutures.com/
Research
Yahoo!/O’Reilly Tech Buzz Game
http://buzz.research.yahoo.com/
More Prediction Market Games
• BizPredict.com
• CasualObserver.net
• FTPredict.com
• InklingMarkets.com
• ProTrade.com
• StorageMarkets.com
• TheSimExchange.com
• TheWSX.com
• Alexadex, Celebdaq, Cenimar, BetBubble, Betocracy, CrowdIQ,
MediaMammon,Owise, PublicGyan, RIMDEX, Smarkets, Trendio,
TwoCrowds
•
http://www.chrisfmasse.com/3/3/markets/#Play-Money_Prediction_Markets
Catalysts
• Markets have long history of predictive accuracy:
why catching on now as tool?
• No press is bad press: Policy Analysis Market
(“terror futures”)
• Surowiecki's “Wisdom of Crowds”
• Companies:
– Google, Microsoft, Yahoo!; CrowdIQ, HSX,
InklingMarkets, NewsFutures
• Press: BusinessWeek, CBS News, Economist,
NYTimes, Time, WSJ, ...
http://us.newsfutures.com/home/articles.html
Does it work?
Yes, evidence from real markets, laboratory
experiments, and theory
Racetrack odds beat track experts [Figlewski 1979]
Orange Juice futures improve weather forecast [Roll 1984]
I.E.M. beat political polls 451/596 [Forsythe 1992,
1999][Oliven 1995][Rietz 1998][Berg 2001][Pennock 2002]
HP market beat sales forecast 6/8 [Plott 2000]
Sports betting markets provide accurate forecasts of
game outcomes [Gandar 1998][Thaler 1988][Debnath
EC’03][Schmidt 2002]
Market games work [Servan-Schreiber 2004][Pennock 2001]
Laboratory experiments confirm information aggregation
[Plott 1982;1988;1997][Forsythe 1990][Chen, EC’01]
Theory: “rational expectations” [Grossman 1981][Lucas 1972]
 More later …
[Source: Berg, DARPA Workshop, 2002]
Research
Example: IEM 1992
[Source: Berg, DARPA Workshop, 2002]
Research
Example: IEM
[Source: Berg, DARPA Workshop, 2002]
Research
Example: IEM
[Source: Berg, DARPA Workshop, 2002]
Example: IEM
AAAI’04 July 2004
MP1-35
[Source: Berg, DARPA Workshop, 2002]
Example: IEM
AAAI’04 July 2004
MP1-36
[Source: Wolfers 2004]
Speed: TradeSports
Contract: Pays $100 if Cubs win game 6 (NLCS)
Price of contract
(=Probability
that Cubs win)
Fan reaches over
and spoils Alou’s
catch. Still 1 out.
Cubs are winning
3-0 top of the 8th
1 out.
The
Marlins
proceed to
hit 8 runs
in the 8th
inning
Time (in Ireland)
AAAI’04 July 2004
MP1-37
Research
Does money matter?
Play vs real, head to head
Experiment
• 2003 NFL Season
• ProbabilitySports.com
Online football forecasting
competition
• Contestants assess
probabilities for each game
• Quadratic scoring rule
• ~2,000 “experts”, plus:
• NewsFutures (play $)
• Tradesports (real $)
•
Results:
• Play money and real
money performed
similarly
• 6th and 8th respectively
• Markets beat most of the
~2,000 contestants
• Average of experts
came 39th (caveat)
Used “last trade” prices
Electronic Markets, Emile ServanSchreiber, Justin Wolfers, David
Pennock and Brian Galebach
Research
Prediction Accuracy
Market Forecast Winning Probability and Actual Winning Probability
100
Prices: TradeSports and NewsFutures
100
Fitted Value: Linear regression
45 degree line
TradeSports: Correlation=0.96
NewsFutures: Correlation=0.94
90
75
TradeSports Prices
80
70
60
50
40
50
25
30
20
10
0
0
0
10
20
30
40
50
60
70
Trading Price Prior to Game
80
90
0
100
20
40
60
NewsFutures Prices
n=416 over 208 NFL games.
Correlation between TradeSports and NewsFutures prices = 0.97
Data are grouped so that prices are rounded to the nearest ten percentage points; n=416 teams in 208 games
Rank
Prediction Performance of Markets
Relative to Individual Experts
0
20
40
60
80
100
120
140
160
180
200
220
240
260
280
300
NewsFutures
Tradesports
1
2
3
4
5
6
7
8
9 10 11 12 13 14
Week into the NFL season
80
100
Research
Does money matter?
Play vs real, head to head
ProbabilityFootball Avg
TradeSports
(real-money)
NewsFutures
(play-money)
Difference
TS - NF
0.443
0.439
0.436
0.003
(0.012)
(0.011)
(0.012)
(0.016)
0.476
0.468
0.467
0.001
(0.025)
(0.023)
(0.024)
(0.033)
Average Quadratic Score
9.323
12.410
12.427
-0.017
= 100 - 400*( lose_price2 )
(4.75)
(4.37)
(4.57)
(6.32)
Average Logarithmic Score
-0.649
-0.631
-0.631
0.000
= Log(win_price)
(0.027)
(0.024)
(0.025)
(0.035)
Mean Absolute Error
= lose_price
[lower is better]
Root Mean Squared Error
= ?Average( lose_price2 )
[lower is better]
[higher is better]
[higher (less negative) is better]
Statistically:
TS ~ NF
NF >> Avg
TS > Avg
Real markets
vs. market games
IEM
HSX
average
log
score
arbitrage
closure
AAAI’04 July 2004
MP1-41
Real markets
vs. market games
HSX
FX, F1P6
probabilistic
forecasts
actual
forecast source
F1P6 linear scoring
F1P6 F1-style scoring
betting odds
F1P6 flat scoring
F1P6 winner scoring
100
expected
value
forecasts
50
20
10
5
489 movies
2
1
estimate
1
AAAI’04 July 2004
2
5
10
20
50
100
MP1-42
avg log score
-1.84
-1.82
-1.86
-2.03
-2.32
Research
1/7 Story
Survey
Research
A Wisdom of Crowds Story
• ProbabilitySports.com
• Thousands of probability judgments
for sporting events
• Alice: Jets 67% chance to beat Patriots
• Bob: Jets 48% chance to beat Patriots
• Carol, Don, Ellen, Frank, ...
• Reward: Quadratic scoring rule:
Best probability judgments maximize
expected score
Opinion
Research
Individuals
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
• Most individuals are poor predictors
• 2005 NFL Season
• Best: 3747 points
• Average: -944
Median: -275
• 1,298 out of 2,231 scored below zero
(takes work!)
Research
Individuals
• Poorly calibrated
(too extreme)
• Teams given < 20%
chance actually won
30% of the time
• Teams given > 80%
chance actually won
60% of the time
Research
The Crowd
• Create a crowd predictor by simply
averaging everyone’s probabilities
• Crowd = 1/n(Alice + Bob + Carol + ... )
• 2005: Crowd scored 3371 points
(7th out of 2231) !
• Wisdom of fools: Create a predictor by
averaging everyone who scored below
zero
• 2717 points (62nd place) !
• (the best “fool” finished in 934th place)
Research
The Crowd: How Big?
More:
http://blog.oddhead.com/2007/01/04/the-wisdom-of-the-probabilitysports-crowd/
http://www.overcomingbias.com/2007/02/how_and_when_to.html
Research
Can We Do Better?: ML/Stats
[Dani et al. UAI 2006]
• Maybe Not
CS “experts algorithms”
Other expert weights
Calibrated experts
Other averaging fn’s (geo mean, RMS,
power means, mean of odds, ...)
• Machine learning (NB, SVM, LR, DT, ...)
•
•
•
•
• Maybe So
• Bayesian modeling + EM
• Nearest neighbor (multi-year)
Research
Can we do better?: Markets
Rank
Prediction Performance of Markets
Relative to Individual Experts
0
20
40
60
80
100
120
140
160
180
200
220
240
260
280
300
NewsFutures
Tradesports
1
2
3
4
5
6
7
8
9 10 11 12 13 14
Week into the NFL season