A
Eliciting Predictions for Discrete Decision Making
YILING CHEN, Harvard University
IAN A. KASH, Microsoft Research
MIKE RUBERRY, Harvard University
VICTOR SHNAYDER, Harvard University
We consider a decision maker who can select one of a finite set of possible actions, each of which will result
in one of a finite set of possible outcomes. If the decision maker has preferences over these outcomes, it will
naturally prefer taking some actions to others, but may be uncertain of the result of each action. In this
paper we describe how a decision maker can elicit expert predictions about the outcome of each action,
allowing the decision maker to make an informed decision. We show that strictly proper decision making,
where experts have an incentive to accurately reveal their beliefs about the outcome of each action, allows
the decision maker to take a preferred action with probability arbitrarily close—but not equal—to one;
with positive probability, the decision maker must take an action at random. Requiring a decision maker
to sometimes act randomly is clearly undesirable, so we also describe an alternative where a single expert
directly reveals a preferred decision instead of predicting each action’s outcome.
Categories and Subject Descriptors: F.0 [Theory of Computation]: General; J.4 [Computer Applications]: Social and Behavioral Sciences
General Terms: Economics, Theory
Additional Key Words and Phrases: Market design, decision markets, prediction markets, decision making,
information elicitation, scoring rules
ACM Reference Format:
Chen, Y., Kash, I., Ruberry, M., and Shnayder, V. 2012. Elicting Predictions for Discrete Decision Making.
ACM V, N, Article A (January YYYY), 23 pages.
DOI = 10.1145/0000000.0000000 http://doi.acm.org/10.1145/0000000.0000000
1. INTRODUCTION
When making a decision we are often interested in achieving a specific outcome, but instead
of picking outcomes we are limited to picking actions. Worse, we may not know the result
of each action: will donating to a politician’s campaign improve their odds of winning? Will
our company make a profit if we open a new store in Springfield?
To make an informed decision, a decision maker needs to understand the mapping from
its actions to the outcomes of interest. Hanson [1999] proposed using experts to predict
this relationship. These predictions, which provide a conditional probability distribution
over outcomes for each action, can be represented by an action-outcome matrix like the one
Parts of this research initially appeared in Chen and Kash [2011] and Chen et al. [2011].
This work is partially supported by NSF grant CCF-0953516. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors alone. The authors are grateful to Craig
Boutilier for helpful discussions about this work.
Author’s addresses: Y. Chen, M. Ruberry, and V. Shnayder, School of Engineering and Applied Sciences,
Harvard University, {yiling, mruberry, shnayder}@seas.harvard.edu; I. A. Kash, Microsoft Research, Cambridge, UK, [email protected].
Permission to make digital or hard copies of part or all of this work for personal or classroom use is
granted without fee provided that copies are not made or distributed for profit or commercial advantage
and that copies show this notice on the first page or initial screen of a display along with the full citation.
Copyrights for components of this work owned by others than ACM must be honored. Abstracting with
credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any
component of this work in other works requires prior specific permission and/or a fee. Permissions may be
requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA,
fax +1 (212) 869-0481, or [email protected].
c YYYY ACM 0000-0000/YYYY/01-ARTA $10.00
DOI 10.1145/0000000.0000000 http://doi.acm.org/10.1145/0000000.0000000
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:2
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
in Figure 1. The prediction in Figure 1 contains all the information relevant to making a
Outcomes O
Actions A
Profit
Loss
2
3
2
5
1
3
3
5
Springfield
Greenville
Fig. 1. A prediction of how likely each of two possible actions, building a new store in Springfield or
Greenville, would result in a profit or a loss. Entries in the matrix represent the conditional likelihood of
their column’s outcome given their row’s action; the upper-left value of the matrix, for example, indicates
a 23 chance the store will make a profit in Springfield. We will assume outcomes are mutually exclusive and
exhaustive of all possible futures, so each row is a probability distribution.
decision. And if the decision maker believes the prediction is accurate it will prefer opening
a store in Springfield to Greenville.
Hanson [1999] suggested soliciting expert predictions of action-outcome matrices using a
prediction market-like structure, which he termed a decision market. Prediction markets are
an effective method of aggregating expert opinion [Wolfers and Zitzewitz 2004; Berg et al.
2001; Chen and Pennock 2010], but are typically used only to predict the likelihood of events
independent of the market. A prediction market can, for example, be used to predict how
likely a horse is to win a race, or whether it will rain on a particular day1 . A strictly proper
prediction market provides an incentive for experts to accurately reveal how likely they
think these events are, and (subject to a technical caveat) eventually leads to a consensus
prediction that (in theory) reflects the pooled information of every expert [Ostrovsky 2009;
Chen et al. 2012].
Unlike a typical prediction market where no decision is made, when we choose a decision
based on a market’s predictions the realized outcome directly depends on the predictions.
This dependency means that the techniques that make prediction markets strictly proper
break down for decision markets. Figure 2 is an example of the failure of straightforwardly
adapting a strictly proper prediction market to decision making.
Since strict properness is usually considered a critical property of a well-designed prediction market, we describe how to extend it to decision making, providing a characterization
of decision strict properness analogous to characterizations of strict properness [Gneiting
and Raftery 2007]. We show that strictly proper decision making allows the decision maker
to take a preferred action with probability arbitrarily close—but not equal—to one; the rest
of the time the decision maker must take an action at random. This is an undesirable and
likely non-credible restriction in practice, so we also present a new mechanism where a single
expert directly reveals a preferred decision instead of describing each action’s outcome.
2. BACKGROUND AND RELATED WORK
There has been extensive prior work on scoring rules and prediction markets, and we will,
by necessity, only describe a small part. While prediction markets can be operated using
continuous double auctions [Forsythe et al. 1992; Berg and Rietz 2003], automated market
makers [Othman and Sandholm 2010a; Othman et al. 2010], and other wagering mechanisms [Plott et al. 1997; Pennock 2004; Mangold et al. 2005], we are interested in prediction
markets that use scoring rules. It is known that such markets can be equivalently implemented as automated market makers [Chen and Pennock 2007; Chen and Vaughan 2010],
but we restrict our discussion to the former for technical tractability.
1 See
intrade.com for active public prediction markets like these.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:3
(a) Most Accurate Prediction
Profit Loss
Springfield
Greenville
3
4
1
10
Profit
Springfield
Greenville
1
20
1
10
1
4
9
10
Expected score = 1 +
3
4
3
4
+
1
4
1
10
+
9
10
log
log
1
4
≈ .44
(b) Higher Scoring Prediction
Loss
19
20
9
10
Expected score = 1 +
1
10
log
log
9
10
≈ .86
Fig. 2. A decision market adapted from a strictly proper prediction market is not strictly proper. In this
example, the decision maker selects the action more likely to result in a profit and scores the prediction for
the selected action using a strictly proper logarithmic scoring rule, 1 + log po , where po is the prediction
for the realized outcome given the selected action (see Section 2 for strictly proper scoring rules). The
expert, who is assumed to be the last participant in the market, expects Springfield to result in a profit and
Greenville a loss as the action-outcome matrix in (a) indicates but scores higher in expectation by claiming
that Springfield is worse than Greenville as shown by the action-outcome matrix in (b). Since the decision
maker never builds in Springfield, this deception is never caught.
Scoring rules are fitness functions for predictions. Originally suggested by Brier [1950]
to describe the quality of weather forecasts, they are now used to score many kinds of
predictions. An in-depth summary of scoring rules appears in [Gneiting and Raftery 2007].
We use the following definition.2
Definition 2.1 (Regular Scoring Rule). A function
s : ∆(O) × O → R ∪ {−∞}
where O is a finite set of mutually exclusive and exhaustive outcomes, and ∆(O) is the
set of all probability distributions over outcomes. We often abbreviate s(p, o) as so (p), and
require
so (p) = −∞ =⇒ po = 0
that is, a (regular) scoring rule can assign a forecast p a score of negative infinity only if
that forecast assigned zero probability to the actual outcome o.
Regularity is a common assumption for scoring rules, and we will restrict our attention to
regular scoring rules in the rest of the paper. The interesting scoring rules are (strictly)
proper.
Definition 2.2 ((Strictly) Proper Scoring Rule). A scoring rule s is proper if and only if
X
q ∈ arg max
qo so (p), ∀q ∈ ∆(O)
p∈∆(O)
o
and strictly proper if and only if q is the unique maximizing argument.
A natural interpretation of a proper scoring rule is that an expert maximizes its expected
score for a prediction by predicting what it believes to be true, or, equivalently, predicting
as accurately as possible. If the scoring rule is strictly proper then this prediction uniquely
maximizes its expected score.
2A
note on mathemtical notation. For consistency we will use calligraphy (A) for sets, capitals (A) for
matrices, bold (a) for vectors and lowercase (a) for functions, individual actions and outcomes, and reals.
When we refer to the elements of matrix A or vector a we will use an index, like Ax,y or ay .
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:4
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
We use in our technical results the following characterizations of (strictly) proper scoring
rules, which are given in Gneiting and Raftery [2007] but credited to McCarthy [1956] and
Savage [1971].
Theorem 2.3 (Gneiting and Raftery [2007]). A scoring rule is (strictly) proper if
and only if
s(p, o) = g(p) − g ? (p) · p + go? (p)
where g : ∆(O) → R is a (strictly) convex function, g ? (p) is a subgradient of g at the point
p, and go? (p) is the o-th element of g ? (p).
Corollary 2.4 (Gneiting and Raftery [2007]). Any (strictly) proper scoring rule
s(p, o) = g(p) − g ? (p) · p + go? (p)
satisfies
X
po s(p, o) = g(q), ∀p ∈ ∆(O).
o
Our characterization of strictly proper decision making in Section 4 and Theorem 2.3 have
a similar relationship with convex functions.
One well-known strictly proper scoring rule is the logarithmic scoring rule, slog (p, o) =
ao + b log po with log 0 defined as equal to −∞ and b > 0 and ao being parameters. This
rule can be constructed via Theorem 2.3 by letting the convex function be the prediction’s
negative Shannon entropy. This scoring rule already made a brief appearance in Figure 2
with ao = b = 1.
Hanson adapted scoring rules for use in prediction markets [Hanson 2003; Hanson 2007],
where experts sequentially improve over previous predictions and are scored for the improvement. Such a mechanism is called a market scoring rule. A prediction market opens
with an initial prediction p0 and accepts a series of expert predictions p1 , p2 , . . .. Each
prediction receives a net score equal to the difference of its prediction’s score and that of
the immediately preceeding prediction. For example, if the outcome o∗ occurs the net score
for prediction pt is
s(pt , o∗ ) − s(pt−1 , o∗ ).
Since the market’s outcome o∗ and previous prediction pt−1 are fixed, maximizing the expected net score of a prediction is the same as maximizing its expected prediction score; thus
(strictly) proper scoring rules make (strictly) proper prediction markets. Markets cheaply
aggregate information from many experts, since each prediction except the last is paid for
by the following one.
Hanson [1999] suggested adapting prediction markets to decision making by running a
standard market scoring rule prediction market for each action and voiding markets for
actions that are not taken. Such markets, as shown in Figure 2, are not strictly proper,
however. We create strictly proper decision markets by extending market scoring rules to
account for a decision maker’s decision policy.
Othman and Sandholm [2010b] first formally developed the idea of using a single expert
to predict the results of a finite set of actions and two outcomes of interest, “good” and
“bad.” Their decision maker wishes to choose the action maximizing the likelihood of the
“good” outcome, and they developed quasi-strictly proper scoring rules that incentivized
the expert to accurately reveal both the action most likely to cause the “good” outcome
and how likely that action was to do so. The “quasi-” in quasi-strictly proper is due to
the fact that the scoring rule allows the score-maximizing prediction for the non-optimal
actions to be arbitrary. In Section 5.2 we generalize this result and characterize all types of
preferences that have right-action rules, which incentivize an expert to reveal the decision
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:5
maker’s most preferred action. As a corollary, we also characterize the set of preferences
that have quasi-strictly proper rules.
Like Othman and Sandholm [2010b], we only consider the decision maker as having
preferences over the outcomes. Our experts are indifferent to the action taken and the
eventual outcome, and they also have no means of changing what each action does. Other
work related to decision making has relaxed these assumptions. Shi et al. [2009] considered
a setting where experts can take actions to change the outcome of a prediction market,
and defined principal-aligned scoring rules that incentivize them to take only “helpful”
actions that are more likely to cause outcomes preferred by the decision maker. These
rules are extremely similiar to our right-action rules, though we consider a broader class of
preferences for the decision maker. Shi et al. [2009] also discuss how these rules might work
with multiple experts instead of the single expert we consider in Section 5.2, although this
discussion does not include decision markets.
More recently, Boutilier [2012] has described an expert with preferences over the outcomes
and a decision maker who, similar to Othman and Sandholm [2010b], always takes the “best”
reported action. To incentivize the expert to make an accurate prediction he introduces
compensation functions that compensate the expert for any loss of utility it may incur
when a less desirable outcome is achieved. He also details some realistic complexities of this
setting, e.g. the decision maker may not precisely know the expert’s utility function.
Adapting these additional considerations to strictly proper decision making is an interesting opportunity for future work.
3. A FRAMEWORK FOR DECISION MAKING
In this section we introduce our formal definition of strict properness for decision making,
first describing how predictions are elicited, a decision is made, and experts are scored. This
provides a basis for our adaptation of strict properness to decision making. In a standard
prediction market—where no decision is made—whether the market is strictly proper or
not depends only on its choice of scoring rule. Strictly proper decision making, on the other
hand, depends on the relationship between the decision maker’s method of making a decision
and its method of scoring predictions. Thus, a decision making strategy in our setting is
described by a pair (decision method, scoring method). Some such pairs are strictly proper
for a single expert, others are strictly proper only for a market, and some are strictly proper
for any number of experts. We will argue that restricting our attention to this last set of
pairs is without loss of generality when we consider decision markets, and with only a slight
loss of generality when working with a single expert. For completeness, we characterize strict
properness for an expert separately in Section 5.1 after characterizing strict properness (for
any number of experts) in Section 4.
3.1. Predictions and their Elicitation
We begin by describing how the decision maker elicits predictions and uses them to make
a decision.
When making a decision we naturally think of a helpful prediction describing what is
likely to occur for each choice. Concretely, if we let A be our finite set of possible actions,
and O a finite set of outcomes, these predictions can be represented by an A × O actionoutcomes matrix like the one in Figure 1. Each row is associated with an action and each
column an outcome, and a row describes how likely each outcome is to occur if that row’s
action is taken. An outcome can be any observable property of the future the decision
maker is interested in, but in this paper we assume the outcomes are chosen to be mutually
exclusive and exhaustive so each row of the matrix is a conditional probability distribution.
We let P be the set of all such matrices.
These predictions are elicited from either a single expert or many experts participating
in a decision market. A single expert makes a single prediction P , and experts in a market
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:6
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
make a series of predictions P 1 , P 2 , . . . after the market opens with an initial prediction
P 0 . We make no further assumptions about either process except that (1) experts can only
observe prior predictions before making their own and (2) both processes produce a final
prediction. This latter assumption is trivially true with a single expert. Whether a market
produces a consensus prediction with rational participants3 is beyond the scope of this
paper, and we make no additional assumptions about market dynamics.
After the final prediction is produced, the decision maker must choose a single action,
and we make two assumptions about this choice: first that the decision maker can draw
an action stochastically, second that its method of decision making can be described as a
function of the final prediction.
We define a decision rule as the function mapping from (consensus) predictions to distributions the action is drawn from, which we will also refer to as decision policies.
Definition 3.1 (Decision Rule). A function
d : P → ∆(A)
mapping predictions in P to probability distributions over actions in ∆(A). We describe a
decision rule as having full support if da (P ) > 0, ∀P, a, and write d ∈ d(·) for a decision
policy in the image of d.
Decision rules “with full support” always create policies with full support, and in Section 4
we show that a decision rule with full support is necessary and sufficient for a strictly proper
decision market. When working with a single expert a decision rule with full support is not
required for strict properness, but the decision rules of interest will usually create policies
with full support nonetheless; this caveat is discussed further in Section 5.1.
Once the action is picked an outcome is observed. As mentioned above, we assume that no
expert can influence what outcome occurs except indirectly by changing the action taken.
3.2. Scoring a Single Expert
After a single expert makes its prediction, the decision maker applies its decision rule and
an outcome is observed, the expert is scored. Instead of a scoring rule we use a decision
scoring rule to assign the prediction a score.
Definition 3.2 (Regular Decision Scoring Rule). A function
s : A × O × ∆(A) × P → R ∪ {−∞}
mapping an action, outcome, decision policy and prediction to the extended reals; as a
shorthand we let sa,o (d, P ) = s(a, o, d, P ). We also require
sa,o (d, P ) = −∞ → da = 0 or Pa,o = 0
As with scoring rules, we will restrict our attention to regular decision scoring rules for the
rest of the paper.
Given a decision rule d and decision scoring rule s, an expert with beliefs Q has an
expected score for a prediction P of
X
da (P )Qa,o sa,o (d(P ), P )
a,o
3 See
[Ostrovsky 2009; Chen et al. 2012] for a theoretical discussion of information aggregation in prediction
markets.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:7
the sum of possible scores (sa,o (d(P ), P )) weighted by how likely each score is to be realized
(da (P )Qa,o ). We now define strict properness4 for an expert.
Definition 3.3 (Strictly Proper for an Expert). A pair (d, s) is strictly proper for an expert if and only if a single expert always uniquely maximizes its expected score by predicting
its beliefs Q
X
{Q} = arg max
da (P )Qa,o sa,o (d(P ), P ), ∀Q ∈ P
P ∈P
a,o
We will continue to emphasize that this notion of strict properness applies only to a single
expert for easier reading, and we characterize all pairs that are strictly proper for an expert
in Section 5.1.
3.3. Scoring Decision Markets
In a typical prediction setting a strictly proper scoring rule is strictly proper for both a
single expert and many experts participating in a market, but a pair (d, s) that is strictly
proper for an expert is not necessarily strictly proper for a market because the market has
an additional complication—the decision rule is not applied to an expert’s own prediction
but the final prediction in the market. Experts in a market are rewarded for improving
over the previous prediction made, and to avoid overloading the term “score” we define an
expert’s expected net score as the difference
X
da Qa,o sa,o (d, P ) − sa,o (d, P 0 )
a,o
when an expert has beliefs Q, the decision policy is d, the prior prediction is P 0 and the
expert predicts P . We now define strict properness for a market.
Definition 3.4 (Strictly Proper for a Market). A pair (d, s) is strictly proper for a market
if and only if an expert in a market always uniquely maximizes its expected net score by
predicting its beliefs
X
X
da Qa,o sa,o (d, Q) − sa,o (d, P 0 ) ≥
d0 a Qa,o sa,o (d0 , P ) − sa,o (d0 ), P 0 ) ,
a,o
a,o
0
∀Q, P, P ∈ P, d, d0 ∈ d(·)
with the inequality strict if P 6= Q. A market using such a pair is called a strictly proper
decision market.
Like with the notion of strict properness for an expert we will continue to stress the caveat
that this notion of strict properness applies only for a market.
3.4. Strictly Proper Pairs
While a pair (d, s) may only be strictly proper for an expert or only strictly proper for a
market, we can drop these caveats and simply talk about strictly proper pairs with little loss
of generality. We begin by giving a formal definition of a strictly proper pair, then discuss
its properties.
4 We
focus on strict properness rather than properness in this paper because properness as a property can be
trivially achieved by some uninteresting scoring rules such as a constant scoring rule that assigns a constant
value to all predictions.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:8
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
Definition 3.5 (Strictly Proper Pair). A pair (d, s) is strictly proper if and only if a
prediction’s expected score is independent of the decision policy
X
X
da Qa,o sa,o (d, P ) =
d0a Qa,o sa,o (d0 , P ), ∀Q, P ∈ P, d, d0 ∈ d(·)
(1)
a,o
a,o
and uniquely maximized when an expert predicts its beliefs
X
{Q} = arg max
da Qa,o sa,o (d, P ), ∀Q ∈ P, d ∈ d(·)
P ∈P
(2)
a,o
The first condition of strict properness, Equation 1, implies the expected scores of prior
predictions in a decision market appear constant, and so independently maximizing a prediction’s expected score also maximizes its expected net score. As we will discuss at the
end of Section 4, the independence between a prediction’s expected score and the decision
policy also means that experts do not have to know the exact form of the decision rule
ex ante. The second condition, Equation (2), taken together with the first, implies that an
expert maximizes its expected score and expected net score by predicting its beliefs. The
following proposition formalizes our argument that strictly proper pairs are strictly proper
for an expert and for a market (the proof appears in the appendix).
Proposition 3.6. Every strictly proper pair (d, s) is strictly proper for both an expert
and a market.
As mentioned, there are pairs that are strictly proper for a market that are not strictly
proper as defined here, but these pairs are an uninteresting technical caveat. Every pair
that is strictly proper for a market can be replaced with a corresponding strictly proper
pair without changing the expected net score of any predictions.
Proposition 3.7. For every pair (d, s̄) that is strictly proper for a market, there exists
a strictly proper pair (d, s) such that every prediction has the same expected net score
X
X
da Qa,o s̄a,o (d, P ) − s̄a,o (d, P 0 ) =
da Qa,o sa,o (d, P ) − sa,o (d, P 0 ) ,
a,o
a,o
0
∀Q, P, P ∈ P, d ∈ d(·)
The proof appears in the appendix.
The set difference of strictly proper pairs and those that are strictly proper for an expert
is not so readily described, and in Section 5.1 we characterize all pairs that are strictly
proper for an expert separately. In short, while strictly proper pairs always have a decision
rule with full support, some pairs that are strictly proper for an expert do not. However,
pairs that are strictly proper for an expert create decision policies with full support for
most predictions, so this distinction is unlikely to be useful in practice. We focus on strictly
proper pairs as a technically incomplete but salient description of strictly proper decision
making.
4. STRICTLY PROPER DECISION MAKING
In this section we characterize all strictly proper pairs, describing most of strictly proper
decision making. It turns out that having full support is a necessary and sufficient condition
for a decision rule to be part of a strictly proper pair. Returning to our example, if our firm
runs a strictly proper decision market to decide whether to open a store in Springfield or
Greenville, this implies that it can make its preferred decision with probability arbitrarily
close—but never equal—to one. With some > 0 chance it must build in the less preferred
location. Even this risk may be unacceptable in practice, and in Section 5.2 we describe
how a preferred decision policy can be directly elicted from a single expert, without learning
the complete mapping from actions to outcomes.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:9
4.1. Strictly Proper Pairs have Decision Rules with Full Support
We begin by showing that full support is necessary for the decision rules of strictly proper
pairs.
Theorem 4.1 (Full Support is Necessary for a Strictly Proper Pair).
pair (d, s) is strictly proper, d has full support.
If a
Proof. Assume, for a contradiction, that d is a decision rule without full support and s
is a decision scoring rule such that (d, s) is strictly proper. Let P ∗ be a prediction such that
da0 (P ∗ ) = 0 for some action a0 , which must exist by our assumption that d does not have
full support, and let Q and Q0 be two action-outcome matrices differing only on action a0 .
Then we have
X
da (P ∗ )Qa,o sa,o (d(P ∗ ), P ) − sa,o (d(P ∗ ), P̄ )
a,o
=
X
da (P ∗ )Q0a,o sa,o (d(P ∗ ), P ) − sa,o (d(P ∗ ), P̄ ) , ∀P, P̄ ∈ P
a,o
implying the same prediction maximizes the expected value of an expert who believes Q
or Q0 , and since this prediction cannot be both Q and Q0 at the same time, the pair (d, s)
violates Equation (2) and so must not be strictly proper, a contradiction as desired.
Simply put, experts have no incentive to be accurate on actions that are never tested, so
a decision rule without full support cannot be strictly proper.
4.2. Constructing Strictly Proper Pairs
As mentioned, a decision rule with full support is a necessary and sufficient condition for it
to be part of a strictly proper pair. In fact, given any decision rule d with full support we
can construct a strictly proper pair (d, s) using a strictly proper scoring rule, s̄, and letting
sa,o (d, P ) =
1
s̄o (Pa ).
da
(3)
The expected score for a prediction is then
X
X
1
da Qa,o
s̄o (Pa ) =
Qa,o s̄o (Pa )
da
a,o
a,o
the same expected score as if an expert were participating in |A| independent and strictly
proper prediction markets, one for each action. The following formal characterization shows
that all strictly proper pairs can be described by a similar construction.
Some additional notation is needed before stating the theorem.
P We use a colon between
two matrices to denote their Frobenius inner product, A : B = i,j Aij Bij , and let g ? (P )
be a subgradient of the convex function g at P . The subgradient of a real-valued convex
function is usually considered a vector, but since P is a matrix we also index the subgradient
as a matrix.
Theorem 4.2 (Strictly Proper Pair Characterization). A pair (d, s) is strictly
proper if and only if d has full support and there exists a strictly convex function g such
that
?
ga,o
(P )
(4)
sa,o (d, P ) = g(P ) − g ? (P ) : P +
da
Proof. We begin by showing that given a decision rule d with full support and a strictly
convex g, defining a decision scoring rule s as in Equation 4 makes (d, s) a strictly proper
pair.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:10
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
An expert’s expected score for predicting P with beliefs Q and decision policy d is
X
da Qa,o sa,o (d, P )
a,o
=
X
=
X
da Qa,o g(P ) − g ? (P ) : P +
a,o
?
ga,o
(P ) da
{da Qa,o g(P ) − g ? (P ) : P } + Q : g ? (P )
a,o
= g(P ) − g ? (P ) : P + Q : g ? (P )
(since
P
a,o
da Qa,o = 1)
?
= g(P ) + (Q − P ) : g (P )
which is independent of the decision policy, and the expert’s expected score for accurately
predicting Q is then
g(Q) + (Q − Q) : g ? (Q) = g(Q)
and applying the subgradient inequality we have
g(Q) > g(P ) + (Q − P ) : g ? (P ), ∀P 6= Q ∈ P
implying (d, s) is a strictly proper pair.
Now we show that given a strictly proper pair (d, s) it is necessary that d have full
support and there exists a strictly convex g such that s is as defined in Equation 4. Since
Theorem 4.1 proved the necessity of d having full support, we only need prove the latter
condition.
As a shorthand, we define an expected score function
X
v(d, Q, P ) =
da Qa,o sa,o (d, P )
a,o
and recall from Definition 3.5 that
v(d, Q, P ) = v(d0 , Q, P ), ∀Q, P ∈ P, d, d0 ∈ d(·)
allowing us to write simply v(Q, P ); our strictly convex function g will be g(P ) = v(P, P ),
which is convex (and we will verify is strictly convex shortly), and we’ll use
?
ga,o
(P ) = d̄a sa,o (d̄, P )
where d is any decision policy in d(·), as our subgradient at P . We verify it is a subgradient
by checking the subgradient inequality:
g(P ) + (Q − P ) : g ? (P )
X
= v(P, P ) +
(Qa,o − Pa,o )d̄a Sa,o (d̄, P )
a,o
= v(P, P ) + v(Q, P ) − v(P, P )
= v(Q, P )
< v(Q, Q)
for all P 6= Q ∈ P. The strict inequality following since (d, s) is a strictly proper pair and
this strict inequality implies g is strictly convex [Hendrickson and Buehler 1971].
Before concluding, we note that since (d, s) is a strictly proper pair
da sa,o (d, P ) = d0a sa,o (d0 , P ), ∀d, d0 ∈ d(·), P ∈ P, a ∈ A, o ∈ O
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:11
(otherwise there exist beliefs Q such that v(d, Q, P ) 6= v(d0 , Q, P )), and we use this fact to
verify that g with subgradients as given is, in fact, equal to s
g(P ) − g ? (P ) : P +
= v(P, P ) −
?
ga,o
(P )
da
X
d̄sa,o (d̄, P )
{Pa,o d̄a sa,o (d̄, P )} +
da
a,o
= v(P, P ) − v(P, P ) +
1
d̄sa,o (d̄, P )
da
1
dsa,o (d, P )
da
= sa,o (d, P )
=
So from any strictly proper pair we can construct a strictly convex g satisfying Equation 4.
Theorem 4.2 shows that while a decision maker can take a preferred action with probability arbitrarily close to one, it cannot deterministically take its preferred action and still run
a strictly proper decision market. Note, however, that it is sufficient for experts to believe
they will be scored in a strictly proper fashion, and the decision maker does not have to ex
ante design its decision rule. Instead it can simply review the final prediction, construct any
decision policy with full support, and then score the experts using an appropriate decision
scoring rule to create a strictly proper decision market. This seems a preferable approach
in practice to accounting for every contingency in advance.
A good analogy to running a strictly proper decision market is to an overwhelmed teaching
assistant grading a midterm. The teaching assistant does not have the time to grade every
question, but must pick one from each test. If some questions are more likely to be graded
than others then students will spend more time on those and neglect the rest, biasing their
scores. Only by (1) possibly grading any question and (2) weighting that question’s score
by the inverse likelihood its graded will the teaching assistant create an unbiased estimator,
where the student’s expected grade is the same as if every question were reviewed. This
encourages students to pay equal attention to each question and not “game the system.”
5. WORKING WITH A SINGLE EXPERT
When there is only one expert making a prediction its prediction is also the last prediction,
and this coupling provides some additional structure to exploit. If we are still eliciting the
likely outcomes of each action, this distinction allows strictly proper pairs for an expert to
have a decision policy without full support. But more interestingly, it can allow a decision
maker to directly request and then take a preferred action. This latter technique doesn’t
reveal what would have happened if other actions were taken, but allows the decision maker
to deterministically take its preferred action instead of risking taking any action.
5.1. Pairs that are Strictly Proper for an Expert
As discussed in Section 3, we call a pair strictly proper if it is strictly proper for an expert
and strictly proper for a market. When describing strictly proper decision markets this
distinction is without loss of generality, but the same is not true when working with only
a single expert. In this section we characterize strict properness for an expert. These pairs,
unlike strictly proper pairs, can have decision rules without full support. Such decisions
rules will still tend to create decision policies with full support, however, suggesting that
these pairs are unlikely to be preferred over pairs that are simply strictly proper. Formally,
the set of predictions a pair that is strictly proper for an expert maps to a decision policy
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:12
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
without full support is nowhere dense in the set of all possible predictions. The proofs of
this section’s theorems appear in the appendix.
Theorem 5.1. For any pair (d, s) that is strictly proper for an expert, the set of actionoutcome matrices that d maps to distributions without full support is nowhere dense in the
set of all action-outcome matrices with their natural Euclidean topology.5
Intuitively, this means that for any d(P ) without full support, there is a P 0 arbitrarily
close to P , such that d(P 0 ) does have full support. Thus, while the set of pairs that are
strictly proper for an expert is larger the set of pairs that are just strictly proper, this
additional latitude is rarely useful.
We conclude with a complete characterization of pairs that are strictly proper for an
expert. The statement and its proof are similar to those of Theorem 4.2.
Theorem 5.2 (Strictly Proper for an Expert Characterization).
A pair
(d, s) is strictly proper for an expert if and only if there exist a strictly convex function g
and a subgradient g ? (P ) where ga? (P ) = 0 whenever da (P ) = 0 such that
sa,o (d(P ), P ) = g(P ) − g ? (P ) : P +
?
(P )
ga,o
, ∀da (P ) > 0.
da (P )
(5)
Following this characterization, if the decision maker was averse to actions with a particular distribution over the outcomes, p ∈ ∆(O), it may pick a strictly convex function
g : R|O| → R with a minima at p. It may then construct a decision scoring rule from g using
the simple construction (3) from the previous section, and if the expert predicts a conditional distribution of p for some action (or set of actions) avoid it completely. Note also that
Theorem 5.2 only restricts sa,o (d(P ), P ) when da (P ) > 0. When da (P ) = 0, sa,o (d(P ), P )
can take any value as long as the decision scoring rule is regular.
This concludes our discussion of strictly proper decision making, where a decision maker
solicits the complete mapping from actions to outcomes. In the next section we discuss
an alternative where, instead of this mapping, a decision policy is directly solicited. This
alternative allows the decision maker to deterministically take a preferred action, instead
of doing so with high probability.
5.2. Optimal Decision Making
As shown in Sections 4 and 5.1, strictly proper decision making (generally) requires the
decision maker risk taking any action, even if it learns that some actions will result in
undesirable outcomes. This is not ideal and in practice may be non-credible. It is hard
to imagine a real firm learning that Springfield is a better location for a new store than
Greenville, but still building in the latter because of a coin flip.
In this section we describe a new method for making a decision, still derived from scoring
rules, that can incentivize a single expert to directly reveal a preferred decision policy,
instead of describing the complete mapping from actions to outcomes. This allows the
decision maker to take deterministic actions, but not to learn what would have happened
if other actions were chosen (e.g. the expert tells our firm to build in Springfield, and not
what would happen if it built in Greenville). This approach works for many—but not all—
types of decision maker preferences, including the commonly considered preferences of an
expected value maximizer who assigns a value to each outcome.
5A
set is nowhere dense in a topological space if the interior of its closure, with respect to the topological
space, is empty.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:13
We assume the decision maker’s preferences can be represented by a utility function6
u : ∆(O) → R,
mapping probability distributions over the outcomes, or lotteries, to the reals. Figure 3
shows some example utility functions.
Fig. 3. Four utility functions of lotteries over two outcomes, > and ⊥. The x-axis represents the probability
of outcome > and the y-axis is the utility for the lotteries. At left is the utility function of an expected
utility maximizer, with the outcome utility of ⊥ zero and > one. Middle-left is an extremal utility function
that prefers “certainty,” middle-right is a utility function that values “uncertainty” or “fairness” and at
right is the utility function of a risk averse decision maker.
The decision maker’s goal is to elicit a decision policy whose resultant lottery (the distribution over outcomes conditional on that action being taken) maximizes its utility function.
Formally, if an expert has beliefs Q the decision maker wants to find a decision policy d∗
solving
max u(QT d)
d∈∆(A)
where QT d is the distribution over outcomes conditional on drawing an action according to
d (we use the transpose of the action-outcome matrix Q since we treat all vectors as column
vectors). Also, while a decision maker has preferences over lotteries of outcomes, we will
describe a decision policy d as preferred or utility maximizing, too, with the understanding
we are describing its conditional lottery QT d.7
Instead of predicting an action-outcome matrix we now have the expert report a decision
policy and a lottery (d, p) ∈ ∆(A) × ∆(O), and we let R be the set of such reports. After
the expert makes its report, the decision maker draws an action according to the reported
decision policy d, observes an outcome o∗ and scores the expert using so∗ (p), where s
is a scoring rule. What we will show is that if the decision maker’s preferences can be
ordered by a convex utility function, this function can be used to create a scoring rule that
aligns the expert’s incentives with the decision maker’s. The expert will want to be scored
for conditional distributions more preferred by the decision maker, and so will report an
optimal decision policy (although not necessarily the correct conditional distribution).
6 We
adopt the generic textbook definition of a utility function, implicitly assuming the decision maker’s
preferences are complete and transitive. See Mas-Colell et al. [1995] for details. This definition includes
the Von Neumann-Morgenstern utility function for expected value maximizers as a special case but is
more general.
The utility function of an expected utility maximizer who has value vo for outcome o is
P
u(p) = o∈O po vo for lottery p.
7 We consider eliciting a decision policy, and not just a single action, for generality, because some preferred
lotteries may require a convex combination of several actions to construct. We will see, however, that we
cannot elicit preferred decision policies for preferences like this, and when we can incentivize an expert to
reveal a preferred decision policy, a single action will always maximize the decision maker’s utility.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:14
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
5.2.1. Right-action Rules and Making an Optimal Decision. Before proving this result we need
some new notation. Given a scoring rule s, an expert with beliefs Q maximizes its expected
score by reporting in the set
X
Rs (Q) = arg max
da Qa,o so (p)
(d,p)∈R
Rsd (Q)
a,o
s
and we use
= {d|∃p (d, p) ∈ R (Q)} for the set of decision policies that are
part of such pairs. If an expert always maximizes its expected score by revealing a utilitymaximizing decision policy, we describe the scoring rule s as a right-action rule for the
utility function u.
Definition 5.3 (Right-Action Rule). A scoring rule s is a right-action rule for a utility
function u if
u(QT d∗ ) ≥ u(QT d), ∀Q ∈ P, d ∈ ∆(A), d∗ ∈ Rsd (Q)
This section’s primary result is that a right-action rule exists for a utility function u if and
only if u is convexible.
Definition 5.4 ((Strictly) Convexible). A function f is (strictly) convexible if there exists
a (strictly) convex function g such that if f (x) > f (y) then g(x) > g(y).
In other words, a right-action rule for a decision maker’s utility function exists if and only
if that function’s strict preference ordering can be represented by a convex utility function.
We stress that this convex utility function may not represent the decision maker’s actual
preferences because it does not need to preserve indifferences. Intuitively, if we are equally
happy with an apple or an orange we have the latitude of “preferring” one to the other
without loss of utility, and this flexibility may be useful in constructing a convex ordering of
our preferences. This allows, for example, a constant utility function to be strictly convexible
with any strictly convex function satisfying the criteria.
Many but not all preferences are convexible. All convex utility functions are trivially
convexible, and these include the utility functions of expected value maximizers. Some
strictly concave utility functions are convexible as well. Of the four example utility functions
in Figure 3, the two at left are convex and thus trivially convexible, but the rightmost and
concave utility function is convexible, too, since it’s strictly increasing. Only the middle-right
function is not convexible.
We can now state our result that a right-action rule exists if and only if the expert’s
utility function is convexible. The formal statement adds the caveat that the decision maker
consider at least two actions, since otherwise it is running a prediction market and any
scoring rule is trivially a right-action rule.
Theorem 5.5 (Right-Action Rule Characterization). If the decision maker is
considering at least two actions, a utility function u has a right-action rule if and only if u
is convexible.
Proof. We first show that any convexible utility function has a right-action rule. Since
u is convexible there exists a convex function g such that u(x) > u(y) implies g(x) > g(y).
We use this to construct a scoring rule
so (p) = g(p) − g ? (p) · p + go? (p)
and by Theorem 2.3 this scoring rule is proper (for a typical prediction market) with expected score function
X
g(p) =
po so (p)
o
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:15
Since s is proper an expert with beliefs Q maximizes its expected score by solving
X
max
(QT d)o so (p)
d,p
o
and since s is a proper, Corollary 2.4 and our construction imply
X
arg max max
(QT d)o so (p) ∈ arg max u(QT d)
d
p
(6)
d
o
implying s is a right-action rule for u.
Now assume, for a contradiction, that u is not convexible and has a right-action rule s.
The expert’s expected maximizing score function over lotteries is
X
v(q) = sup
qo so (p)
p
o
0
which is convex, implying there exists q and q such that
v(q) > v(q0 )
u(q) < u(q0 )
since u is not convexible. Because v is convex
v(q) ≥ v(αq + (1 − α)q0 ), ∀α ∈ (0, 1)
so an expert prefers to report policies with conditional distribution q, even when policies
with conditional distribution q0 , which is preferred by the decision maker, or convex combinations of the two are available. This implies that if the expert has beliefs Q with Qa = q
and Qa0 = q0 , ∀a0 6= a, then it maximizes its expected score by reporting a policy assigning
probability one to action a, even though the decision maker prefers policies that assign zero
probability to a, contradicting our assumption that s is a right-action rule for u.
There are two important points to be made about this proof. First, the result is not
constructive because it provides no means of finding a convex function that preserves the
decision maker’s strict preferences, and we leave this problem for future work. Second, the
reported lottery p may not be an accurate prediction of what will occur when an action is
drawn according to the report decision policy d. If, for example, the decision maker is an
expected value maximizer with a utility for each outcome, then simply paying the expert
a portion of the realized utility is a right-action rule. This rule ignores the lottery p when
scoring and so an expert can maximize its score while setting it arbitrarily.
While we have allowed the expert to report an entire decision policy, convexible preferences imply reporting a single action can always maximize a decision maker’s utility, too.
Proposition 5.6 (Reporting One Action is Enough). For any utility function u
with a right-action rule s, there exists an action a and decision policy d ∈ Rsd (Q), ∀Q ∈ P
where da = 1.
Proof. An expert with beliefs Q maximizes their expected score by reporting in
X
Rs (Q) = arg max
da Qa,o so (p).
(d,p)∈R
∗
∗
a,o
∗
Let (d , p ) be an element of this set. If d already has an action a where d∗a = 1 then
we’re done. Otherwise, for all actions a and a0 with d∗a > 0 and d∗a0 > 0, we will show that
X
X
Qa,o so (p) =
Qa0 ,o so (p)
o
o
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:16
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
which implies that (d0 , p∗ ) with d0a = 1 also maximizes the expert’s expected score and
completes the proof. Suppose the statement is false, then
X
X
Qa,o so (p) >
Qa0 ,o so (p)
o
0
o
∗
and reporting (d , p ) has a higher expected value than reporting (d0 , p∗ ) has a higher
expected value than reporting (d∗ , p∗ ), contradicting our assumption that the latter pair
maximized the expert’s expected score.
In short, although there may be optimal decision policies that assign positive probability
to multiple actions, the decision maker must be indifferent between them. E.g. if opening a
store in Springfield or Greenville will result in the same profit then any combination of the
actions has the same expected value for the decision maker, but the decision maker loses
nothing by requiring the expert specify one city or the other.
5.2.2. Quasi-Strict Properness. Taking a preferred action is good, but it may also be important to know the likely result of that action. When the decision maker’s utility function
is strictly convexible we can construct scoring rules that incentivize revealing not only the
preferred policy, but also an accurate conditional lottery. We call these scoring rules quasistrictly proper after Othman and Sandholm [2010b] (see Section 2). In contrast to strictly
proper pairs for an expert, which incentivize an expert to accurately reveal the likelihood of
the outcomes conditional on each possible action, quasi-strictly proper rules incentivize the
accurate revelation of the likelihood of the outcomes conditional on following the preferred
decision policy. This is a generalization of Othman and Sandholm [2010b]’s definition, where
they consider deterministic decision policies and hence their quasi-strictly proper rules accurately reveal the likelihood of the outcomes conditional on the action that would be taken
(but not necessarily for other actions).
Definition 5.7 (Quasi-Strictly Proper). A scoring rule s is quasi-strictly proper for a utility function u if it is a right-action rule for u and
p = QT d, ∀Q ∈ P, (d, p) ∈ Rs (Q)
Corollary 5.8 (Quasi-Strictly Proper Characterization). If u is a strictly
convexible utility function, there exists a quasi-strictly proper scoring rule for u.
The proof is immediate from the first part of Theorem 5.5, since a strictly convex function
implies Equation 6 is uniquely maximized when the prediction p is equal to the conditional
lottery QT d.
In practice quasi-strictly proper scoring rules may be interesting as they suggest the
decision maker can “look before they leap” and plan for the effect of their decision.
6. CONCLUSION
We described how a decision maker can create a strictly proper incentive for experts to
predict the mapping from its actions to some outcomes of interest. Understanding this
mapping allows the decision maker to make an informed decision, although it can only do
so with high probability—with some > 0 chance it must risk taking an action at random.
Even this small chance may be too much to accept in practice, so we also propose a method
where a single expert directly reveals a preferred decision policy; whether a decision maker
can take advantage of this technique depends on the convexibility of its preferences.
While we have discussed our work in the context of decision making, the techniques can
be applied to other settings where observations are incomplete or costly. For example, if we
would like experts to predict how many barrels of oil are underneath the ocean at five sites,
it is costly to drill in each one to verify the predictions (and defeats the purpose of asking
for them!). Using a decision market, the market maker can drill in only one location and
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:17
still expect an accurate prediction from the experts. If this location happens to be the one
with lots of oil, all the better; otherwise, the decision maker can simply drill there next,
only paying the costs of one additional borehole instead of five.
There are many possible variations for decision making. We restricted our attention to
discrete decision making, where a decision maker considers finite sets of actions and outcomes. We not address situations where experts are asked to predict a mapping from an
infinite number of actions to an infinite number of outcomes, a situation that might occur if
the decision maker is trying to locate a facility, for example. Our model might be extended
in other ways, too: as discussed in Section 2, prior work has already considered experts able
to take actions affecting the eventual outcome or with their own preferences among the
outcomes. Finally, we have left some more immediate questions open, such as describing
which types of preferences are convexible, and which are strictly convexible.
7. HISTORY
Portions of this paper originally appeared in [Chen and Kash 2011] and [Chen et al. 2011],
which appeared at the Tenth Conference on Autonmous Agents and Mulitagent Systems
(AAMAS 2011) and the Seventh Workshop on Internet and Network Economics (WINE
2011), respectively. In particular, Theorem 5.2 appeared in [Chen and Kash 2011] and
Theorems 4.1 and 4.2 in [Chen et al. 2011].
This paper’s discussion of strict properness for an expert and strict properness for a
market, as well as their formal unification, is new, as are Theorem 5.1 and the entirety of
Section 5.2. We have also updated the presentation of prior results for clarity and consistency.
REFERENCES
Berg, J. E., Forsythe, R., Nelson, F. D., and Rietz, T. A. 2001. Results from a dozen years of election
futures markets research. In Handbook of Experimental Economic Results, C. A. Plott and V. Smith,
Eds.
Berg, J. E. and Rietz, T. A. 2003. Prediction markets as decision support systems. Information Systems
Frontier 5, 79–93.
Boutilier, C. 2012. Eliciting forecasts from self-interested experts: Scoring rules for decision markets. In
Proceedings of the Eleventh International Conference on Autonomous Agents and Multiagent Systems
(AAMAS-12).
Brier, G. W. 1950. Verification of forecasts expressed in terms of probability. Monthly Weather Review 78, 1, 1–3.
Chen, Y., Kash, I., Ruberry, M., and Shnayder, V. 2011. Decision markets with good incentives.
In WINE’11: Proceedings of the 7th international conference on Internet and Network Economics.
Springer-Verlag, Berlin, Heidelberg, 72–83.
Chen, Y. and Kash, I. A. 2011. Information elicitation for decision making. In AAMAS ’11: Proceedings
of the 10th International Conference on Autonomous Agents and Multiagent Systems.
Chen, Y. and Pennock, D. M. 2007. A utility framework for bounded-loss market makers. In UAI ’07:
Proceedings of the 23rd Conference on Uncertainty in Artificial Intelligence. 49–56.
Chen, Y. and Pennock, D. M. 2010. Designing markets for prediction. AI Magazine 31, 4, 42–52.
Chen, Y., Ruberry, M., and Vaughan, J. W. 2012. Designing informative securities. In Proceedings of
the 28th Conference on Uncertainty in Artificial Intelligence (UAI).
Chen, Y. and Vaughan, J. W. 2010. A new understanding of prediction markets via no-regret learning.
In EC ’10: Proceedings of the 11th ACM Conference on Electronic Commerce. ACM, New York, NY,
USA, 189–198.
Forsythe, R., Nelson, F., Neumann, G. R., and Wright, J. 1992. Anatomy of an experimental political
stock market. American Economic Review 82, 5, 1142–1161.
Gneiting, T. and Raftery, A. E. 2007. Strictly proper scoring rules, prediction, and estimation. Journal
of the American Statistical Association 102, 477, 359–378.
Hanson, R. 1999. Decision markets. IEEE Intelligent Systems 14, 3, 16–19.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:18
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
Hanson, R. D. 2003. Combinatorial information market design. Information Systems Frontiers 5, 1, 107–
119.
Hanson, R. D. 2007. Logarithmic market scoring rules for modular combinatorial information aggregation.
Journal of Prediction Markets 1, 1, 1–15.
Hendrickson, A. D. and Buehler, R. J. 1971. Proper scores for probability forecasters. Annals of Mathematical Statistics 42, 6, 1916–1921.
Mangold, B., Dooley, M., Dornfest, R., Flake, G. W., Hoffman, H., Kasturi, T., and Pennock,
D. M. 2005. The tech buzz game. IEEE Computer 38, 7, 94–97.
Mas-Colell, A., Whinston, M. D., and Green, J. R. 1995. Microeconomics Theory. Oxford University
Press, New York, NY.
McCarthy, J. 1956. Measures of the value of information. PNAS: Proceedings of the National Academy of
Sciences of the United States of America 42, 9, 654–655.
Ostrovsky, M. 2009. Information aggregation in dynamic markets with strategic traders. In EC ’09:
Proceedings of the tenth ACM conference on Electronic commerce. ACM, New York, NY, USA, 253.
Othman, A. and Sandholm, T. 2010a. Automated market making in the large: The Gates Hillman prediction market. In EC ’10: Proceedings of the 11th ACM Conference on Electronic Commerce. ACM,
New York, NY, USA, 367–376.
Othman, A. and Sandholm, T. 2010b. Decision rules and decision markets. In Proceedings of the 9th
International Conference on Autonomous Agents and Multiagent Systems (AAMAS). 625–632.
Othman, A., Sandholm, T., Pennock, D. M., and Reeves, D. M. 2010. A practical liquidity-sensitive automated market maker. In EC ’10: Proceedings of the 11th ACM Conference on Electronic Commerce.
ACM, New York, NY, USA, 377–386.
Pennock, D. M. 2004. A dynamic pari-mutuel market for hedging, wagering, and information aggregation.
In Proceedings of the Fifth ACM Conference on Electronic Commerce (EC’04).
Plott, C. R., Wit, J., and Yang, W. C. 1997. Parimutuel betting markets as information aggregation
devices: Experimental results. Tech. Rep. Social Science Working Paper 986, California Institute of
Technology. Apr.
Savage, L. J. 1971. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association 66, 336, 783–801.
Shi, P., Conitzer, V., and Guo, M. 2009. Prediction mechanisms that do not incentivize undesirable
actions. In WINE ’09: Proceedings of the 5th International Workshop on Internet and Network Economics. Springer-Verlag, Berlin, Heidelberg, 89–100.
Wolfers, J. and Zitzewitz, E. 2004. Prediction markets. Journal of Economic Perspective 18, 2, 107–126.
Appendix
This appendix contains the proofs of Propositions 3.6 and 3.7 that appeared in Section 3
and the proofs of Theorems 5.1 and 5.2 that appeared in Section 5.1.
7.1. Proof of Proposition 3.6
Proposition 3.6. Every strictly proper pair (d, s) is strictly proper for an expert and a
market.
Proof. Let (d, s) be a strictly proper pair. Strict properness for an expert requires
X
arg max
da (P )Qa,o sa,o (d(P ), P ) = {Q}, ∀Q ∈ P
(7)
P ∈P
a,o
which is always satisfied since, from the definition of strict properness an expert’s true
beliefs Q always maximize an expert’s expected score with respect to any decision policy d
X
arg max
da Qa,o sa,o (d, P ) = {Q}, ∀Q ∈ P, d ∈ d(·)
P ∈P
a,o
and and the expected score is independent of the decision policy
X
X
da Qa,o sa,o (d, P ) =
d0a Qa,o sa,o (d0 , P ), ∀Q, P ∈ P, d, d0 ∈ d(·)
a,o
a,o
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:19
implying Q is also the unique maximizer for Equation 7.
Strict properness for a market requires
X
da Qa,o sa,o (d, Q) − sa,o (d, P 0 )
a,o
X
≥
d0 a Qa,o sa,o (d0 , P ) − sa,o (d0 , P 0 ) , ∀Q, P, P 0 ∈ P, d, d0 ∈ d(·)
a,o
with the inequality strict if P 6= Q. From the definition of strictly proper pair we have
X
X
da Qa,o sa,o (d, P ) =
d0a Qa,o sa,o (d0 , P ), ∀Q, P ∈ P, d, d0 ∈ d(·)
a,o
a,o
implying the difference
X
da Qa,o sa,o (d, Q) − sa,o (d, P 0 ) − d0 a Qa,o sa,o (d0 , P ) − sa,o (d0 , P 0 )
a,o
=
X
da Qa,o sa,o (d, Q) − da Qa,o sa,o (d, P )
a,o
and
X
da Qa,o sa,o (d, Q) ≥
a,o
X
da Qa,o sa,o (d, P ), ∀Q, P ∈ P, d ∈ d(·)
a,o
unless P = Q since Q is the unique maximizing argument to the expression. So the strictly
proper for a market inequality always holds.
7.2. Proof of Proposition 3.7
Proposition 3.7. For every pair (d, s̄) that is strictly proper for a market, there exists
a strictly proper pair (d, s) such that every prediction has the same expected net score
X
da Qa,o s̄a,o (d, P ) − s̄a,o (d, P 0 )
a,o
=
X
da Qa,o sa,o (d, P ) − sa,o (d, P 0 ) , ∀Q, P, P 0 ∈ P, d ∈ d(·)
a,o
Proof. We will prove there exists a decision policy d̄ ∈ d(·) and function cd̄ such that
da s̄a,o (d, P ) − cd̄ (a, o, d) = d̄a s̄a,o (d̄, P ), ∀P ∈ P.
(8)
In other words, there exists a function, independent of the prediction P , that translates
¯ From the
from the expected score under one decision policy to another (in this case, d).
definition of strict properness for a market we have
X
da Qa,o s̄a,o (d, Q) − s̄a,o (d, P 0 )
a,o
≥
X
=⇒
X
d0 a Qa,o s̄a,o (d0 , P ) − s̄a,o (d0 , P 0 ) , ∀Q, P, P 0 ∈ P, d, d0 ∈ d(·)
a,o
X 0
da Qa,o s̄a,o (d, Q) − s̄a,o (d, P ) =
da Qa,o s̄a,o (d0 , Q) − s̄a,o (d0 , P ) , ∀d, d0 ∈ d(·), Q, P ∈ P
a,o
X
a,o
a,o
Qa,o da s̄a,o (d, Q) −
d0a s̄a,o (d0 , Q)
=
X
Qa,o da s̄a,o (d, P ) − d0a s̄a,o (d0 , P ) .
a,o
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:20
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
Assuming, for a contradiction, that Equation 8 is false and there exists no such function
cd̄ . Then there exists P and P 0 , decision policies d and d0 and an action a and outcome o
such that
da s̄a,o (d, P ) − d0a s̄a,o (d0 , P ) 6= da s̄a,o (d, P 0 ) − d0a s̄a,o (d0 , P 0 )
implying there exists a linear combination (belief) such that
X
X
Qa,o da s̄a,o (d, P ) − d0a s̄a,o (d0 , P ) 6=
Qa,o da s̄a,o (d, P 0 ) − d0a s̄a,o (d0 , P 0 )
a,o
a,o
but from strict properness for a market it must be that
X
X
Qa,o da s̄a,o (d, Q) − d0a s̄a,o (d0 , Q) =
Qa,o da s̄a,o (d, P ) − d0a s̄a,o (d0 , P ) , ∀P
a,o
a,o
which cannot be simultaneously true for both P and P ’, a contradiction as desired, so there
exists a function satisfying Equation 8.
Equation 8 lets us define our new scoring rules as
s̄ (d, P ) − cd̄ (a, o, d)/da , da > 0
sa,o (P, P ) = a,o
0, o.w.
we need to prove (1) that using this scoring rule gives each expert the same expected net
score, and (2) that it is strictly proper. Substituting our new rule for the old in the expected
net score equation gives
X
da Qa,o sa,o (d, P ) − sa,o (d, P 0 )
a,o
=
X
da Qa,o s̄a,o (d, P ) − s̄a,o (d, P 0 )
a,o
so the two scoring rules have the same expected net score, as desired. We conclude by
proving that (d, s) is a strictly proper pair.
First, the expected score is independent of the decision policy, since substitution gives
X
X
da Qa,o sa,o (d, P ) =
d̄a Qa,o s̄a,o (d̄, P ), ∀d ∈ d(·), Q, P ∈ P
a,o
a,o
so it’s as if the decision policy is fixed. Second, the expected score is uniquely maximized
when an expert predicts its beliefs since (again, from the definition of strict properness for
a market)
X
arg max
da Qa,o s̄a,o (d, P ) − s̄a,o (d, P 0 ) = {Q}, Q, P 0 ∈ P, d ∈ d(·)
P ∈P
a,o
and the expected net score is the same using the new scoring rule s, so
X
arg max
da Qa,o sa,o (d, P ) − sa,o (d, P 0 ) = {Q}, Q, P 0 ∈ P, d ∈ d(·)
P ∈P
a,o
and the score of the previous prediction Qa,o da sa,o (d, P 0 ) is independent of P , so equivalently the maximizing argument of the above expression is
X
arg max
da Qa,o sa,o (d, P ) = {Q}, Q ∈ P, d ∈ d(·)
P ∈P
a,o
and the pair (d, s) is strictly proper.
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:21
7.3. Proof of Theorem 5.1
Theorem 5.1. For any pair (d, s) that is strictly proper for an expert, the set of actionoutcome matrices that d maps to distributions without full support is nowhere dense in the
set of all action-outcome matrices with their natural Eucdliean topology.
Proof. Let (d, s) be any pair that is strictly proper for an expert and assume, for a
contradiction, that the set of matrices that d maps to distributions assigning zero probability
0
to some action a0 , P a = {P ∈ P|da0 (P ) = 0}, is not nowhere dense in P. The set of all
matrices d maps to distributions without full support is a finite union of such sets (since
0
we assumed in Section 3 the set of actions was finite), so proving P a is nowhere dense in
P for arbitrary a0 proves the theorem.
0
We begin by finding two matrices in the interior of the closure of P a that differ only
on a0 and have different expected scores. The existence of points differing only on a0 in
0
the interior of the closure of P a is immediate from our assumption that the closure has
nonempty interior, while the expected score of a prediction given (d, s) can be written as a
function
X
v(Q, P ) =
da (P )Qa,o sa,o (da (P ), P )
a,o
0
Let P̂ and P̄ be matrices differing only on a0 in the interior of the closure of P a , without
loss of generality either v(P̂ , P̂ ) > v(P̄ , P̄ ) or v(P̂ , P̂ ) = v(P̄ , P̄ ). In the former case we’re
done so we assume, for a contradiction, the latter is true. Now consider the difference
v(P̂ , P̄ ) − v(P̄ , P̄ )
X
X
=
da (P̄ )P̂a,o sa,o (da (P̄ ), P̄ ) −
da (P̄ ), P̄a,o sa,o (da (P̄ ), P̄ )
a,o
=
X
a,o
P̂a0 ,o − P̄a0 ,o da0 (P̄ )sa0 ,o (da (P̄ ), P̄ )
o
This is a difference of two convex combinations of the terms da0 (P̄ )sa0 ,o (da (P̄ ), P̄ ). If these
terms are not all equal then, because of our assumption that P̄ is in the interior of the clo0
sure of P a there exists a P ∗ also differing from P̄ only on action a0 with v(P ∗ , P̄ ) >
v(P̄ , P̄ ), implying v(P ∗ , P ∗ ) > v(P̄ , P̄ ) too because we assumed (d, s) was strictly
proper for an expert. Alternatively all these terms are equal, da0 (P̄ )sa0 ,o0 (da (P̄ ), P̄ ) =
da0 (P̄ )sa0 ,ō (da (P̄ ), P̄ ), ∀o0 , ō ∈ O, and the argument applies symmetrically to P̂ so we
assume its corresponding terms are equal, and immediately from the difference and the
relationship v(P̂ , P̂ ) = v(P̄ , P̄ ) the terms for both predictions must all equal each other,
too. This is, however, a contradiction, since it implies v(P̂ , P̄ ) = v(P̄ , P̄ ) = v(P̂ , P̂ ) which
contradicts our assumption that (d, s) is strictly proper for an expert. Thus we conclude
0
there exist points P̂ and P̄ in the interior of the closure of P a differing only on action a0
with different expected scores, v(P̂ , P̂ ) > v(P̄ , P̄ ).
Next we show the function v(P ) = v(P, P ) is continuous. Assume, for a contradiction
and without loss of generality
lim v(Q) < v(P )
Q→P
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
A:22
Y. Chen, I. A. Kash, M. Ruberry, and V. Shnayder
But the difference
lim v(Q, P ) − v(P, P )
X
=
Qa,o − Pa,o da (P )sa,o (da (P ), P )
Q→P
a,o
= 0
implying (since we assumed (d, s) was strictly proper for an expert) that limQ→P v(Q) ≥
v(P ), a contradiction. Alternatively
lim v(Q) > v(P )
Q→P
and we have a symmetric argument, so the function v(P ) = v(P, P ) is continuous.
0
Now let {Q̂k } and {Q̄k } be sequences in P a converging to P̂ and P̄ respectively, and
consider the difference of the limits
lim v(P̂ , P̂ ) − v(P̄ , Q̂k )
k→∞
=
=
=
lim v(Q̂k , Q̂k ) − v(P̄ , Q̂k )
X
k
lim
da (Q̂k ) Q̂ka,o − P̄a,o
sa,o (Q̂k )
k→∞
k→∞
(by continuity)
a,o
lim da0 (Q̂k ) P̂a0 ,o − P̄a0 ,o sa0 ,o (Q̂k )
k→∞
= 0
0
(since da0 (Q̂k ) = 0, ∀k because Q̂k is always in P a )
Recalling that we assumed (without loss of generality) that v(P̂ , P̂ ) > v(P̄ , P̄ ), this implies
there exists a prediction P ∗ ∈ {Q̂k } such that v(P̄ , P ∗ ) > v(P̄ , P̄ ) which, again, contradicts
0
our assumption that (d, s) is strictly proper for an expert. We conclude P a must be nowhere
dense in P, and since a0 was chosen arbitrarily the set of all matrices d maps to distributions
with full support is also nowhere dense in P.
7.4. Proof of Theorem 5.2
Theorem 5.2. A pair (d, s) is strictly proper for an expert if and only if there exist a
strictly convex function g and a subgradient g ? (P ) where ga? (P ) = 0 whenever da (P ) = 0
such that
?
ga,o
(P )
sa,o (d(P ), P ) = g(P ) − g ? (P ) : P +
, ∀da (P ) > 0.
(5)
da (P )
Proof. Given a decision rule d, strictly convex function g, and subgradients g ? of g,
with ga? (P ) = ~0 whenever da (P ) = 0, we prove that the pair (d, s) with s written as in
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.
Elicting Predictions for Discrete Decision Making
A:23
Equation (5) whenever da (P ) > 0 is strictly proper for an expert.
X
da (P )Qa,o sa,o (P )
a,o
=
X
da (P )Qa,o g(P ) − g ? (P ) : P +
a∈{a|da (P )>0},o
= g(P ) − g ? (P ) : P +
X
?
ga,o
(P )Qa,o
a∈{a|da >0},o
?
?
?
ga,o
(P ) da (P )
(recalling da (P ) = 0 =⇒ ga? (P ) = 0)
= g(P ) − P : g (P ) + Q : g (P )
= g(P ) + (Q − P ) : g ? (P )
Exactly as in Theorem 4.2 and the same argument concludes this direction of the proof.
Given a (d, s) pair that is strictly proper for an expert, we can construct a strictly convex
function g and subgradients of g satisfying the theorem’s criteria. As in Theorem 4.2 we
consider the expected score function
X
v(Q, P ) =
da (P )Qa,o sa,o (d(P ), P ),
a,o
and let g(P ) = v(P, P ). Define the subgradients of g(P ) as
∗
ga,o
(P ) = da (P )sa,o (d(P ), P ).
We can verify that they satisfy the subgradient inequality:
g(P ) + (Q − P ) : g ? (P )
X
=
(Pa,o + Qa,o − Pa,o )da (P )sa,o (d(P ), P )
a,o
=
X
Qa,o da (P )sa,o (d(P ), P )
a,o
= V (Q, P ) < V (Q, Q) = g(Q)
for any Q 6= P , where the inequality holds because (d, s) is strictly proper for an expert.
This also implies g is strictly convex. Note also that da (P ) = 0 =⇒ ga? (P ) = ~0 for all
P , so it only remains to show that s can be written as in Equation (5) using g for any
a ∈ {a|da (P ) > 0}:
g(P ) − g ? (P ) : P +
?
(P )
ga,o
da (P )
?
ga,o
(P )
da (P )
da (P )sa,o (d(P ), P )
=
= sa,o (d(P ), P ).
da (P )
=
ACM Journal Name, Vol. V, No. N, Article A, Publication date: January YYYY.