THE HONG KONG UNIVERSITY OF SCIENCE & TECHNOLOGY
CSIT 5220: Reasoning and Decision under Uncertainty
L09: Graphical Models for Decision Problems
Nevin L. Zhang
Room 3504, phone: 2358-7015,
Email: [email protected] Home page
CSIT 5220
Page 2
L09: Graphical Models for Decision Problems

Introduction

Extending BN to Include a Single Decision

Fundamentals of Rational Decision Making

Decision Trees

Influence Diagrams

Solving influence Diagrams

Value of information
CSIT 5220
Page 3
Probabilistic Reasoning and Decision


Method 1: Two-stage

In a BN, calculate posterior probabilities

Use the posteriors to make decisions
Method 2

Combine the two stages

Extend BN to include decisions
 Better reveal structure of decision problem
 Compute optimal decisions directly from model

Reasoning: Jensen & Nielsen, Sections 9.1-9.4, 10.2, 11.1
CSIT 5220
Page 4
L09: Graphical Models for Decision Problems

Extending BN to Include a Single Decision

Fundamentals of Rational Decision Making

Decision Trees

Influence Diagrams

Solving influence Diagrams

Value of information
CSIT 5220
Page 5
Poker

From Lecture 04

Extend the model so that I can calculate the probability that my hand
is better than the opponent’s hand

MH: My Hand

BH: Best Hand
CSIT 5220
Page 6
Fold or Call
CSIT 5220
Page 7
Fold or Call

Information that I have: FC, SC, MH
CSIT 5220
Page 8
Modeling One Action

Start with a BN

Add the decision node and utility nodes

What information we have when making the decision

What chance and utility variables will the decision influence
CSIT 5220
Page 9
Including More Decisions

Things become a bit more complicated.

Will see later.
CSIT 5220
Page 10
L09: Graphical Models for Decision Problems

Extending BN to Include Decisions

Fundamentals of Rational Decision Making

Decision Trees

Influence Diagrams

Solving influence Diagrams

Value of information
CSIT 5220
Page 11
Decision Theory

Normative decision theory


How people should decide. (Rational agent)
Descriptive decision theory

How people actually decide.
CSIT 5220
Page 12
Normative Decision Theory
CSIT 5220
Page 13
Are you rational?

Lottery A: [$1mill]

Lottery B: 0.5[$2mill] + 0.5[$0mill]

Which one do you choose?

Most people would choose A
U(1) > 0.5 U(2) + 0.5 U(0)

Most people are risk-averse, with concave utility function
CSIT 5220
Page 14
Are your rational?

Suppose that you are $2mill in debt

Lottery A: [$1mill]

Lottery B: 0.5[$2mill] + 0.5[$0mill]

Which one do you choose?

Probably B
U(1) < 0.5 U(2) + 0.5 U(0)

You are being risk-seeking, with convex utility function
CSIT 5220
Page 15
Utilities without Money
CSIT 5220
Page 16
Utilities without Money
CSIT 5220
Page 17
Marks as Utilities
CSIT 5220
Page 18
Other Considerations

2 is passing grade

If fail, can retake and hopefully get a better grade in transcript

In this case, 2 is the worst!
CSIT 5220
Page 19
L10: Graphical Models for Decision Problems

Extending BN to Include Decisions

Fundamentals of Rational Decision Making

Decision Trees

Influence Diagrams

Solving influence Diagrams

Value of information
CSIT 5220
Decision Trees
Page 20

Classical way to represent decision problems with multiple decisions

Explicitly show all possible sequences of decisions and observations.

Example: Oil Wildcatter
A wildcatter is a person who drills
wildcat wells, which are oil wells
drilled in areas not known to be oil
fields.
Test on Seismic structure
CSIT 5220
Page 21
Decision Tree for Oil Wildcatter
CSIT 5220
Page 22
Decision Trees

Decision nodes: Rectangles

Chance nodes: ellipses

Utility values: at leaves, some times inside diamonds

To be read from root to leaves

Branches from a decision node: possible actions

Branches from a chance node: possible outcomes and probs

A decision node follows a chance node:
 The chance node is observed before the decision is made

No-forgetting
 Decision-maker remembers all the labels from root to a decision node

Game between decision maker and nature
CSIT 5220
Page 23
Solution to a Decision Tree

Strategy: Which decision node to pick at each decision node
CSIT 5220
Page 24
Solution to a Decision Tree

Optimal Strategy: The strategy with the highest expected utility
CSIT 5220
Page 25
Solving Decision Trees
CSIT 5220
Page 26
Example
77.59
77.59
CSIT 5220
Page 27
CSIT 5220
Page 28
L09: Graphical Models for Decision Problems

Extending BN to Include Decisions

Fundamentals of Rational Decision Making

Decision Trees

Influence Diagrams

Solving influence Diagrams

Value of information
CSIT 5220
Page 29
Extending BN to Include one Decision

Start with a BN

Add the decision node and utility nodes


What information we have when making the decision

What chance and utility variables will the decision influence
To include multiple decision nodes,

Need to consider the interactions among the decisions
CSIT 5220
Page 30
Including Multiple Decisions

Two more decisions

MFC: my first change

MSC: my second change
CSIT 5220
Page 31
Representing the Decision Sequence


First representation

All nodes observed before a decision are parents of that decision.

Information arcs.
Assume that the decision maker doesn’t forget, then some links are
redundant.
CSIT 5220
Page 32
Representing the Decision Sequence

No-forgetting allows a more concise representation

Keep directed path going through all the decision node: Order of
decision.

Arrows into a decision node only from those nodes observed
immediately before that decision.

Implicit parents: parents of earlier decisions
CSIT 5220
Page 33
Influence Diagram

A DAG with three types of nodes

Chance nodes, decision nodes, and utility nodes

There is a directed path containing all the decision nodes.

The utility nodes have no children.

Each chance node is associated with the conditional distribution given
its parents.

Each utility node is associated with a utility function, a real-valued
function of its parents.
CSIT 5220
Page 34
Influence Diagram
CSIT 5220
Page 35
Influence Diagram

An influence diagram for the oil wildcatter problem

Decision: T: test = {y, n}; D: drill={y, n}

Utility: C: cost of test ; V: Benefit of drilling

Chance:
 O: Oil ={dry, wet, soaking}
 R: seismic structure
{no-structure, open-structure, closed-structure, no-result}
CSIT 5220
Page 36
L09: Graphical Models for Decision Problems

Extending BN to Include Decisions

Fundamentals of Rational Decision Making

Decision Trees

Influence Diagrams

Solving influence Diagrams

Value of information
CSIT 5220
Page 37
Strategy (Policy)

A policy specifies what to do for each decision

It is a function of observed variables

Different policies lead to different expected utility

Optimal policy: the Policy that yields the maximum expected utility.

How to find the optimal policy?
CSIT 5220
Page 38
Finding Optimal Policy

First idea:


Convert to decision tree and solve it
How to convert influence diagram into decision tree
1.
Draw tree
 Root: the thing that happens first
 Children of root: the thing that happens next
 …
2.
Figure out numerical information
CSIT 5220

Order of events

Tree structure

Numerical info

Prob for branches
from chance node

Utility for leaves
CSIT 5220
A Side Note

Two decision trees for Oil Wildcatter

First
 directly from problem specification.
 Asymmetric

Second
 from influence diagram
 Symmetric

Pro of ID: compact

Con of ID: cannot represent assymetry

Need to introduce artificial state R = no-result
CSIT 5220
Page 41
Finding Optimal Policy

First idea:

Convert to decision tree and solve it

Exponential still!

Next:

Variable Elimination Algorithm for solving influence diagrams

Note

BN inference: All orderings give correct result, but might have
different complexity

ID: Must use “strong elimination orderings”.
CSIT 5220
Page 42
Temporal Order among Decisions and Observations



Notations

Decision nodes have a temporal order: D1, D2, …, Dn

T0: Set of chance nodes observed prior to any decision

Ti: Set of chance nodes observed after Di is taken and before
Di+1 is taken
Oil Wildcatter

D1 = T; D2 = D

T0 = {}; T1 = {R}; T2={O}
Partial temporal order

T0, D1, T1, D2, T2, …., Dn, Tn

Oil Wildcatter: T, R, D, O
CSIT 5220
Page 43
Temporal Order

T0={}, T1={T}, T2={A, B, C}

Partial temporal ordering

D1, T, D2. {A, B, C}

No ordering among A, B, C
CSIT 5220
Page 44
Strong Elimination Ordering

Partial temporal order



T0, D1, T1, D2, T2, …., Dn, Tn
Strong elimination orders

First eliminate variables in Tn

Then eliminate Dn

Then eliminate variables in Tn-1

Then eliminate Dn-1

…..
Oil Wildcatter

Temporal order:
 T, R, D, O

Strong elimination ordering
 O, D, R, T
CSIT 5220
Page 45
Strong Elimination Ordering

T0={}, T1={T}, T2={A, B, C}

Partial temporal ordering


D1, T, D2. {A, B, C}

No ordering among A, B, C
Strong elimination orderings

A, B, C, D2, T, D1

B, C, A, D2, T, D1

C, A, B, D2, T, D1

….
CSIT 5220
Variable Elimination

Two set of potentials (factors):

Eliminate decision and chance nodes one by one according to a
strong elimination ordering.

When eliminate variable X
Page 46
CSIT 5220
Page 47
Variable Elimination on Oil Wildcatter

Strong Elimination Ordering: O, D, R, T
CSIT 5220
Page 48
Variable Elimination on Oil Wildcatter

Eliminate: O
CSIT 5220
Page 49
CSIT 5220
Page 50
CSIT 5220
Page 51
Potentials after Eliminating O
CSIT 5220
Page 52
Potentials after Eliminating O
CSIT 5220
Page 53
Eliminating D

No probability potential involves D

Optimal decision for D
CSIT 5220
Page 54
Potentials after Eliminating D
CSIT 5220
Eliminating R
Page 55
CSIT 5220
Page 56
Potentials after Eliminating R
CSIT 5220
Page 57
Eliminating T


Optimal decision for T
Results same as those by decision tree
CSIT 5220
Page 58
L09: Graphical Models for Decision Problems

Extending BN to Include Decisions

Fundamentals of Rational Decision Making

Decision Trees

Influence Diagrams

Solving influence Diagrams

Value of information
CSIT 5220
Page 59
Two types of Decisions

Action decisions

Result in significant state change
of variables of interest

Example:
 D: Drill or not to drill

Test decisions

Look for more evidence

Example:
 T: Test of Seismic structure
CSIT 5220
Page 60
Two types of Decisions

Typical scenario

Need to make one decision

Want to get more information before making the decision

Question
 Is it worthwhile to perform a particular test?
 Which test to choose if multiple tests are available?
CSIT 5220
Page 61
Value of Information


What is the value of a test?

Create two influence diagrams

Solve both

Compare their values
Example: Oil wildcatter

Is it worthwhile to perform the seismic test?

ID1: without the test

ID2: with the test
CSIT 5220
Page 62
Value of Information

Expected utility of ID2


U(ID2) = 22.55
What is the expected utility of ID1?
CSIT 5220
Expected Utility of ID1

Temporal ordering: D, O

Elimination ordering: O, D

Eliminate O:
CSIT 5220
Page 64
Expected Utility of ID1

Potentials after eliminating O

Eliminate D

Expected utility of ID1

U(ID1) = 20
CSIT 5220
Page 65
Value of Information

Difference in expected utility

U(ID2) – U(ID1) = 22.55 – 20 = 2.55

The expected value of the seismic test is 2.55

The test is worthwhile
CSIT 5220
Page 66
Value of Information

If there are multiple tests

T1, T2, T3, …

Compute the value of each test, pick the best one

If the value of the best is positive,
 Pick the test among remain tests

Stop when value of the selected test is not positive