Collective Revelation: A Mechanism
for Self-Verified,
Weighted, and Truthful Predictions
Sharad Goel, Daniel M. Reeves, David
M. Pennock
Presented by: Nir Shabbat
Outline
• Introduction
– Background and Related Work
• The Settings
• A Mechanism For Collective Revelation
– The Basic Mechanism
– A General Technique for Balancing Budgets
• Summary
Introduction
• In many cases, a decision maker may seek to elicit
and aggregate the opinions of multiple experts.
• Ideally, would like a mechanism that is:
1. Incentive Compatible - Rewards participants to be
truthful
2. Information Weighted - Adjusts for the fact that
some experts are better informed than others
3. Self-Verifying - Works without the need for
objective, “ground truth” observations
4. Budget Balanced - Makes no net transfers to agent
Background and Related Work
• Proper Scoring Rules – Rewards the agent by
assessing his forecast against an actual
observed outcome.
• Agents are incentivized to be truthful.
• For example, using the Brier scoring rule:
We get:
Background and Related Work
• Peer Prediction Methods – Can be Incentive
Compatible and Self-Verifying.
• For example, the Bayesian truth serum asks
agents to report both their own prediction
and their prediction of other agents’
predictions.
• Relies on the general property of informationscores, that a truthful answer constitutes the
best guess about the most “surprisingly
common” answer.
Background and Related Work
• With the appropriate reward structure , this
framework leads to truthful equilibria:
• 𝑥𝑘𝑟 = 1 iff response of agent 𝑟 is 𝑘
• 𝑦𝑘𝑟 - prediction of player 𝑟 for the frequency of 𝑘
• 𝑥𝑘 , 𝑦𝑘 - geometric average of predictions (or
opinions) and meta predictions
Background and Related Work
• Predictions Markets – encourage truthful
behavior and automatically aggregate
predictions from agents with diverse
information.
• For example, a prediction market for U.S.
presidential elections. Agents buy and sell
assets tied to an eventual Democratic or
Republican win. Each share pay $1 if the
corresponding event occurs.
Background and Related Work
• Delphi Method – generates consensus
predictions from experts generally through a
process of structured discussion.
• In each round, each expert anonymously
provides his forecast and reasons, at the end
of the round, market maker summarize the
forecasts and reasons.
• Process is repeated until forecasts converge.
Background and Related Work
• Competitive Forecasting – elicits confidence
intervals on predictions, thereby facilitating
information weighting. Like prediction
markets, competitive forecasting rewards
accuracy, though is not rigorously incentive
compatible and relies on benchmarking
against objective measurements.
Background and Related Work
Incentive
Compatible
Proper Scoring Rules
●
Prediction Markets
○
Peer Prediction
○
Information
Weighted
●
●
Delphi Method
Competitive
Forecasting
●
○
●
Polls
Collective Revelation
SelfVerifying
●
○
●
●
The Settings
1. Common prior – Agents & Market Maker have a
common prior on the distribution of 𝑋
2. Independent Private Evidence – Each agent 𝑖
privately observes 𝑛𝑖 ≥ 1 independent
realization of the random variable 𝑋
(independent both of each other and across
agents).
3. Rationality – Agents update their beliefs via
Bayes’ rule.
4. Risk neutrality – Agents act to maximize their
expected payoff.
A Mechanism For Collective Revelation
• Agents are scored against one another’s
private information (Self Verification).
• Agents report their subjective expectation of
𝑋, and their revised estimate in light of new
hypothetical evidence (HFS).
• From these 2 reports the subjective posterior,
private information and also this agent’s
prediction of other agents’ prediction is
constructed.
A Mechanism For Collective Revelation
The Basic Mechanism
LEMMA 1
1. 𝑋1 , … , 𝑋𝑛 𝑖. 𝑖. 𝑑 Bernoulli(𝑝)
2. An agent has subjective distribution 𝐹 for 𝑝
It is possible to elicit the kth moment of 𝐹 via a
proper scoring rule that pays based on the
outcome ( 𝑋1 , … , 𝑋𝑛 ) iff 𝑘 ≤ n
The Basic Mechanism
• Simply put, scoring against a single Bernoulli
observation can only reveal the agent’s
expectation and not the uncertainty of it’s
prediction (as quantified by the variance).
• So in order for a mechanism to be
Information-Weighted (for Bernoulli outcome)
it must score agents against multiple
observations.
The Basic Mechanism
• For example, say we want to elicit an agent’s
prediction regarding the distribution of
𝑋~Bernoulli(p).
• We want to reward the agent using a proper
scoring rule based on a single outcome 𝑋1 :
𝑈 𝑟 = 𝐶 − 𝑟 − 𝑋1 2
• To elicit the confidence, we must use another
outcome 𝑋2 . For example, like this:
𝑈 𝑟, 𝑐 = 𝐶 − 𝑟 − 𝑋1 2 − 𝑐 − ∑ 𝑋𝑖 − 𝑋 2 2
The Basic Mechanism
The Beta distribution is the “conjugate prior” of
the Bernoulli distribution.
That is, given 𝑋~Bernoulli(𝑝) with a prior
p~Beta 𝛼, 𝛽 the posterior distribution of 𝑋,
given 𝑛 independent trials out of which 𝑠 are
successful, is:
𝐹 𝑝 𝑛, 𝑠 ~Beta 𝛼 + 𝑠, 𝛽 + 𝑛 − 𝑠
The Basic Mechanism
LEMMA 2
• Let 𝑌 ~ Bernoulli 𝑝
• 𝑝~Beta 𝛼, 𝛽 a prior on 𝑝
• Fix 𝑛 > 0 (trials) and 0 ≤ 𝑠 ≤ 𝑛 (successes)
integers
• Define:
That is:
𝑔𝑠,𝑛
𝛼
𝛼+𝑠
𝛼, 𝛽 =
,
𝛼+𝛽 𝛼+𝛽+𝑛
The Basic Mechanism
LEMMA 2 (Cont’d)
Then:
• 𝑔𝑠,𝑛 is a bijection from R + × R + to
• And
The Basic Mechanism
LEMMA 3
• Let 𝑋1 , … , 𝑋𝑛 𝑖. 𝑖. 𝑑 Bernoulli 𝑝 (𝑛 ≥ 2)
• Let 𝑝~Beta 𝛼, 𝛽 a prior on 𝑝
• Let 𝑆 = ∑𝑛𝑖=1 𝑋𝑖
• Define:
The Basic Mechanism
LEMMA 3 (Cont’d)
Then:
• ℎ𝑛 is a bijection from R + × R + to
• And:
The Basic Mechanism
THEOREM 1
• Consider a Bayesian game according to the
settings shown before with 𝑁 ≥ 3 players,
𝑋~Bernoulli(𝑝).
• Let 𝑝~Beta 𝛼0 , 𝛽0 a common prior on 𝑝
• Let 𝐹𝑖 be the posterior dist. of agent 𝑖 for 𝑝
after updating according to its private info.
• Fix 𝑛 > 0 and 0 ≤ 𝑠 ≤ 𝑛 integers
The Basic Mechanism
THEOREM 1 (Cont’d)
• Suppose that each agent 𝑖 plays action
𝑎𝑖 , 𝑏𝑖 ∈ Ω where
The Basic Mechanism
THEOREM 1 (Cont’d)
• Define:
Where 𝑃𝑘 is projection into the kth component.
The Basic Mechanism
THEOREM 1 (Cont’d)
• For arbitrary constants 𝐶𝑖 , set the reward of
agent 𝑖 to be:
The Basic Mechanism
THEOREM 1 (Cont’d)
• Then:
(*)
is a strict Nash Equilibrium.
The Basic Mechanism
PROOF
• Fix attention on agent 𝑖, and suppose all
agents 𝑗 ≠ 𝑖 play according to (*).
−1
• By Lemma 2 - 𝑔𝑠,𝑛
𝑎𝑗 , 𝑏𝑗 = 𝛼𝑗 , 𝛽𝑗
parameters of 𝐹𝑗 .
The Basic Mechanism
PROOF (Cont’d)
• Let 𝑠𝑗 , 𝑛𝑗 be private observations of agent 𝑗.
Then:
𝛼𝑗 = 𝛼0 + 𝑠𝑗
𝛽𝑗 = 𝛽0 + 𝑛𝑗 − 𝑠𝑗
Consequently:
The Basic Mechanism
PROOF (Cont’d)
• In particular, 𝑠−𝑖 = ∑𝑗≠𝑖 𝑠𝑗 and 𝑛−𝑖 = ∑𝑗≠𝑖 𝑛𝑗
• If 𝑖 plays according to (*) then:
Since we are using the Brier Proper scoring rule,
strategy (*) maximizes 𝑖’s expected reward.
The Basic Mechanism
PROOF (Cont’d)
−1
• Moreover, since ℎ𝑛−1 ○ 𝑔𝑛,𝑠
is an injection,
this is 𝑖’s unique best response, and so
strategy (*) is a strict Nash equilibrium.
The Basic Mechanism
CORROLARY
• Suppose all agents play the equilibrium
strategy.
• Define:
The Basic Mechanism
CORROLARY (Cont’d)
• Define the aggregate information-weighted
prediction as:
Let 𝐹 𝑝 𝑋𝑖,𝑗 be the posterior dist. Resulting from
the cumulative private evidence of all agents.
Then 𝑝 = E𝐹 𝑋.
A General Technique for Balancing
Budgets
• The idea is to reward agents via a Shared scoring
rule.
A simple application would be:
And this is clearly budget balanced since
∑𝑛𝑖=1 𝑓 𝑟𝑖 , 𝑥𝑖 = 0
A General Technique for Balancing
Budgets
• In our setting, the “observations” 𝑥𝑖 are
determined precisely by agents’ reports –
simple application won’t work!
• The solution proposed is to decouple scoring
from benchmarking, e.g. agent 𝑖 is rewarded
according to its performance relative to a set
of agents whose score cannot be affected by
agent 𝑖’s action.
A General Technique for Balancing
Budgets
DEFINITION
• Let 𝐺 be a Bayesian game with 𝑛 players
• A 𝒌-projective family of 𝑮 is a family of games
𝐺𝑣 such that for each 𝑉 ⊆ 1, … , 𝑛 with 𝑉 =
𝑘, 𝐺𝑣 is a Bayesian game restricted to the players
𝑉 that preserves type spaces, action spaces, and
players’ beliefs about types. In particular, 𝐺𝑣 is
determined by a family of reward functions 𝑈𝒗
that specifies the expected reward for each
player in V resulting from any given strategy
profile of those players.
A General Technique for Balancing
Budgets
THEOREM 2
• G is a Bayesian game with n players
• 𝑟 = 𝑟1 , … , 𝑟𝑛 is a (strict) Nash equilibrium of G
• Suppose 𝐺𝑣 is a k-projective family of G such
that 2 ≤ 𝑘 < 𝑛/2 + 1
• Suppose that for each 𝑉 = 𝑗1 , … , 𝑗𝑘 , 𝑟𝑗1 , … , 𝑟𝑗𝑘
is a (strict) Nash equilibrium for 𝐺𝑣 .
A General Technique for Balancing
Budgets
THEOREM 2 (Cont’d)
• Then for any constant 𝐶 ∈ R, there are player
rewards 𝑈𝑖 for the n-player game G such that:
1. For any strategy profile s, ∑𝑛𝑖=1 𝑈𝒊 𝑠 = 𝐶
2. The original equilibrium q is still a (strict) Nash
equilibrium for the modified game
In particular, by setting 𝐶 = 0, one can alter the
rewards so that the game G is strongly budget
balanced.
A General Technique for Balancing
Budgets
PROOF
• For 𝑖 ∈ 𝑉, denote:
– 𝑈𝑖 - reward function of player 𝑖 in 𝐺
– 𝑈𝑉,𝒊 - reward function of player 𝑖 in the game 𝐺𝑉
– 𝑈𝑉,𝒊 𝑠 - reward of player 𝑖 in game 𝐺𝑉 , when 𝑠 is
restricted to game G𝑉 .
A General Technique for Balancing
Budgets
PROOF (Cont’d)
• Denote the player sets Vi = 𝑖, 𝑖 + 1, … , 𝑖 +
A General Technique for Balancing
Budgets
PROOF (Cont’d)
• Summing over all players we get:
A General Technique for Balancing
Budgets
PROOF (Cont’d)
• Further more:
And since 𝑘 < 𝑛/2 + 1, we have 2 𝑘 − 1 < 𝑛,
and so 𝑖 ∉ 𝑉𝑖+𝑗 for 1 ≤ 𝑗 ≤ 𝑘 − 1.
A General Technique for Balancing
Budgets
PROOF (Cont’d)
• Consequently, if 𝑠𝑖 is the strategy of player 𝑖
and 𝑠−𝑖 is the strategy profile of all other
players, then:
Where 𝐶𝑖 is a function that does not depend on
𝑠𝑖 .
A General Technique for Balancing
Budgets
PROOF (Cont’d)
• Since (𝑟𝑖 , 𝑟−𝑖 ) is a (strict) Nash equilibrium for
𝐺𝑉𝑖 :
And so, using the new reward functions, the
original equilibrium 𝑟 is still a (strict) Nash
equilibrium.
Summary
• We’ve seen a mechanism, i.e. Collective
Revelation, for aggregating experts opinions
which is:
– Incentive Compatible
– Information Weighed
– Self-Verifying
• We’ve seen a general technique for constructing
budget balanced mechanisms that applies both
to collective revelation and to past peerpredictions method.
THE END