Outline
Statement of Problem
Adaptive Utility
Example
Decision Making with Uncertain
Preferences
Brett Houlding
Department of Mathematical Sciences, Durham University, UK
Brett Houlding
Decision Making with Uncertain Preferences
Future Options
Outline
Statement of Problem
Adaptive Utility
Outline:
Statement of Problem
Decision Problems
Historical Treatment
Example
Adaptive Utility
Research Needed
Interpretation of Utility Parameter
Adaptive Utility Functions
Commensurable Utility Functions
Surprise and Selection Strategy
Trial Aversion and Value of Information
Example
Future Options
Brett Houlding
Decision Making with Uncertain Preferences
Example
Future Options
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Decision Problems
Decision Problems
I
A Decision Maker (DM) must select a decision from within a set of
available options D.
I
Initially there is uncertainty over what will be the resulting outcome
of each available decision.
I
Let R be the set of all possible outcomes.
I
Selecting a decision is to be seen as equivalent to selecting a
distribution over R.
How should the DM select her choice?
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Example
Example
A company currently uses service X , but has the option of instead using
a better service Y for an additional cost of c1 . Service Y can either lead
to an increase of s1 or s2 in performance, with prior beliefs that either
outcome is equally likely.
I
Possible outcomes (c, s) display changes to status quo in cost c and
performance s respectively.
I
Possible decisions are to use service X or to use service Y .
I
Service X leads to outcome (0, 0) for sure.
I
Service Y leads to outcome (−c1 , s1 ) with probability 0.5 and
outcome (−c1 , s2 ) otherwise.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Historical Treatment
St Petersburg Paradox
I
Consider the game where a coin is successively flipped.
I
For a throw of a head £2 is won and placed in a pot.
I
For each further throw of a head the money in the pot is doubled.
I
The pot is given to the player as soon as a tail is thrown.
I
How much would you pay to play this game?
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Historical Treatment
St Petersburg Paradox
I
I
There is (1/2)n chance of winning £2n for n = 1, 2, . . .
P∞
P∞
The expected value of game is n=1 (1/2)n × 2n = n=1 1 = ∞.
Maximising expected return is not always a good way to make decisions!
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Historical Treatment
Bernoulli and Utility Hypothesis
I
Consider measuring a person’s happiness just like we may do their
temperature.
I
Rather than in degrees Celsius we do this in utils.
I
The more utils a person has the happier they are.
I
Bernoulli (1738) suggested that giving someone twice as much
money will not necessarily give them twice as many utils.
I
A utility function u : R → R need not be linear.
Suggest choose decision maximising expected utility.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Historical Treatment
Notation
I
d1 d2 means d1 at least as preferable as d2 .
I
d1 d2 means d1 strictly preferred to d2 .
I
αd1 + (1 − α)d2 with d1 and d2 decisions means with probability α
d1 , otherwise d2 .
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Historical Treatment
Axioms
I
A1 Completeness: is a complete relation and the set of feasible
decisions D is a closed convex combination of lotteries.
I
A2 Transitivity: is a transitive relation.
I
A3 Archimedian: If d1 , d2 , d3 ∈ D are such that d1 d2 d3 , then
there is an α, β ∈ (0, 1) such that
αd1 + (1 − α)d3 d2 βd1 + (1 − β)d3 .
I
A4 Independence: For all d1 , d2 , d3 ∈ D and any α ∈ [0, 1],
d1 d2 ⇔ αd1 + (1 − α)d3 αd2 + (1 − α)d3 .
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Historical Treatment
VNM and Utility Theorem
Von Neumann and Morgenstern (1947) showed that axioms A1-A4 are all
that is required to prove there exists a unique utility function (up to a
positive linear transformation) with the properties that:
1. For all d1 , d2 ∈ D, u(d1 ) ≥ u(d2 ) ⇔ d1 d2 .
2. For all d1 , d2 ∈ D and any α ∈ (0, 1),
u(αd1 + (1 − α)d2 ) = αu(d1 ) + (1 − α)u(d2 ).
This proves we should choose decision with maximum expected utility
return.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Example
Example
In our example all that is needed to determine the solution is the utility
function for changes in cost and performance.
I
Assume u[(c, s)] = c + s.
I
Then u(X ) = 0 and u(Y ) = 0.5(−c1 + s1 ) + 0.5(−c1 + s2 )
I
Select Y if 0.5(s1 + s2 ) > c1 otherwise select X .
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Historical Treatment
CD and Uncertain Utility?
I
This is all fine and Anscombe and Aumann (1963) later allowed for
subjective probabilities over outcomes of decisions.
I
Accepted generally by academic community. Witsenhausen (1974)
of Bell Labs used utilities in considering how to make decision
policies when we accept future opinions may be different.
I
Yet can we always be certain of what our utilities will be?
I
Cyert and DeGroot (1975) argued that at times we may be unsure
of our utilities (and hence our preferences).
How then make decisions?
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Adaptive Utility
Adaptive Utility
I
Introduced by Cyert & DeGroot (1975) in the form of parametric
utility functions where DM could learn of parameter through
Bayesian updating.
I
Generalises Bayesian Statistical Decision Theory and coincides in
single decision case.
I
The unknown parameter θ could be used to represent unknown
trade-off weights or measure of risk-aversion etc.
I
Of interest in Marketing Theory where a new brand becomes
available and Clinical Trials with unknown drug attributes.
I
Also assumed of use in R&D and Experimental Design.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Surprise and Selection Strategy
Reliability and Testing
The effects of adaptive utility upon testing and reliability were considered
by Houlding and Coolen (2007):
I
Permits subjective cost of system failure to remain uncertain.
I
When then should systems be fixed if this involves a known cost but
unknown cost of faulty system?
I
Now observing a system failure can be beneficial as, although means
they are more likely, they might be seen to be not as disastrous as
was expected.
I
Decision to replace less reliable system can now be postponed as
DM learns of utility for that reliability level.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Research Needed
Research Needed
Since the initial work of Cyert & DeGroot in 1975 little attention has
been paid to developing adaptive utility. In particular there has been little
or no attention paid to:
I
What is the interpretation of a utility parameter and how is it
meaningful to compare different utility functions?
I
Can one develop a proof that adaptive utilities exist and that the
DM should seek to maximise adaptive utility?
I
What are the implications for decision selection strategy?
I
What are effects on diagnostics such as value of information and
risk aversion?
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Interpretation of Utility Parameter
Interpretation of Utility Parameter
I
Let a DM’s state of mind θ characterise the true utility function.
I
Assume θ uncertain so as to represent uncertainty of utility function.
I
Each possible value for θ will induce a different utility function and
hence represent different preferences over rewards and decisions.
I
Represent in notation as a conditioning argument u[·|θ].
I
Model beliefs over θ through a prior probability distribution.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Adaptive Utility Functions
Adaptive Utility Functions
I
We now have a set of possible utility functions u[·|θ1 ], u[·|θ2 ], . . .
I
Define an adaptive utility function ua (·) to be expected value of
u[·|θ] with respect to beliefs about θ.
Formally, ua (·) = Eθ u[·|θ]
I
I
Repeated application of von Neumann and Morgenstern proves
existence and uniqueness of function.
I
Furthermore, a DM should seek to choose decision with maximum
expected adaptive utility.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Commensurable Utility Functions
Commensurability
I
However, this requires it be meaningful to compare utility values
conditioned on differing values of θ.
I
A classical utility function is only unique up to a positive linear
transformation.
I
We need to scale classical utilities so they are commensurable, i.e.,
so that it is meaningful to make comparisons between terms such as
u(r1 |θ1 ) and u(r2 |θ2 ).
I
Boutilier (2003) showed this was possible under assumption of
extremum equivalence.
I
Demonstrated commensurability can even be achieved without
extremum equivalence.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Surprise and Selection Strategy
Surprise and Selection Strategy
The effects of adaptive utility and accepting preferences are uncertain,
but that they may be learned of, are:
I
No difference in a one-off decision.
I
Can model a situation where a DM is pleasantly surprised by
outcome or upset by outcome.
I
Depending on length of decision sequence and prior beliefs, DM
should select decisions that are likely to lead to unfamiliar outcomes.
I
This should be done even if DM expects that outcome will be less
favourable than another.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Trial Aversion and Value of Information
Trial Aversion and Value of Information
The interpretations of utility diagnostics will now be different:
I
How much should DM pay for information about likely outcome of a
decision if uncertain of preferences?
I
No general connection between value of information and level of
utility uncertainty, but depends on individual case.
I
Trial Aversion is analogous to Risk Aversion (preference or
avoidance of actuarially fair bets).
I
Consider u(r |θ) as a function of θ only.
I
Would call DM trial averse in region that u(r |θ) is concave.
I
Purely a diagnostic as never get to choose distribution over θ.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Example
Example
Return to Example and assume two possible utility functions:
I
u[(c, s)|θ] = c + θs
I
θ = θ1 = 1 with probability 0.5 otherwise θ = θ2 = 2.
I
Assume wish to maximise sum of utility over two periods.
I
Assume c1 = 2.3, s1 = 1 and s2 = 2 so that in a one off decision
DM should stick with service X .
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Example
Example
I
Information z about θ comes from observing either positive surprise
or negative surprise from utility return after first period.
I
Let z = 1 for positive surprise and z = 0 for negative surprise.
I
If service X is chosen utility known for sure and no information
gained.
I
If service Y is chosen we learn about θ.
I
Assume probability that Y truly leads to performance increase s if it
was previously seen to do so is 0.9.
I
If service Y chosen we thus also learn of its true performance
increase.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Example
Influence Diagram
(c,s)
X or Y
(c,s)
X or Y
z
q
Brett Houlding
Decision Making with Uncertain Preferences
u
Outline
Statement of Problem
Adaptive Utility
Example
Example
Through use of Bayes’ Theorem we find:
I
P(θ = 1|z = 1) = 0
I
P(θ = 1|z = 0) = 2/3
I
P(z = 1) = 1/4
I
P(z = 0) = 3/4
Brett Houlding
Decision Making with Uncertain Preferences
Example
Future Options
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Example
Example
Through dynamic programming we find:
I
Should choose service Y in first period even though under initial
beliefs service X seems better.
I
Should choose service Y again in second period only if it was seen
to lead to the larger increase in performance, otherwise choose
service X .
I
This trivial example is equivalent to solving a decision tree with 25
end nodes.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Future Options
Computational Issues
I
Adaptive Utility permits a DM to remain uncertain over preferences.
I
Bayesian sequential decision theory notoriously computationally
intensive.
I
Suffers from so-called “curse of dimensionality” associated with
numerical backward induction with large state space.
I
Involves solving a nested sequence of maximizations and
expectations over multi-dimensional spaces.
I
Now also must incorporate extra computations for utility parameter
and its associated learning.
I
Some integrals can not be found in close form.
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Future Options
Computational Issues
McDaid and Wilson (2001) on classical Bayesian sequential decision
theory for testing of software:
I
“other testing plans are more reasonable ... but often difficult to
implement because the expressions for the expected utility become
difficult to calculate”
I
“The most desirable testing plan is the sequential test ... but its
solution is not tractable”
Brett Houlding
Decision Making with Uncertain Preferences
Outline
Statement of Problem
Adaptive Utility
Example
Future Options
Future Options
Future Options
This work focuses on foundations and is not yet at application stage.
Due to intense computation cost for application on interesting problems
the further challenges include:
I
Identification of utility forms which are reasonable at modelling
preferences and lead to tractable computations.
I
Investigation of simple and accurate techniques for eliciting
necessary information concerning beliefs and possible preferences.
Brett Houlding
Decision Making with Uncertain Preferences