Trick or treatment?
Evaluating the quality of structured risk
management decisions
Joe Arvai
The Ohio State University
Decision Research
SNR
1
Skunkwor
ks
Outline
Structured DM in brief
A practical example
Questions of quality (3 experiments)
Parting thoughts
SNR
2
Skunkwor
ks
SDM In Brief
• Structured decision approaches have
their roots in…
– …the observation that people tend to have
tremendous difficulty with making decisions
that involve multiple objectives and,
therefore, tradeoffs.
SNR
3
Skunkwor
ks
SDM In Brief
• Structured decision approaches have
their roots in…
– …studies of the constructive nature of
preferences in response to available cues.
SNR
4
Skunkwor
ks
SDM In Brief
• Structured decision approaches are
designed based on…
– …“value focused thinking.”
SNR
5
Skunkwor
ks
SDM In Brief
• Structured decision approaches are
designed based on…
– …the literature dealing with normative
decision making, specifically the steps
required for a complete analysis of a given
decision
SNR
6
Skunkwor
ks
SDM In Brief
• Structured decision approaches are
designed based on…
– …methods for decreasing the cognitive
burden associated with complex choices.
SNR
7
Skunkwor
ks
The Case of Water Use Planning
in British Columbia
• Work with B.C. Hydro on a
comprehensive, stakeholder-based
development of revised operating plans
at all major hydroelectric facilities.
• In response to increasing, competing
demands on provincial water resources
SNR
8
Skunkwor
ks
Scale of Study
SNR
9
Skunkwor
ks
The Case of Water Use Planning
in British Columbia
•
Multiple Objectives:
1. Electricity generation/trade
2. Environmental quality
•
Water, land, & air
3. Recreation opportunities
4. Cultural values
5. Learn over time; reduce uncertainty
SNR
http://www.bchydro.com/wup/
Skunkwor
ks
10
Water Use Planning in B.C.
The Basic Steps
•
The approach is based on work with stakeholders on:
1.
Eliciting objectives from various stakeholders.
2.
Identifying a series of workable options for
management.
3.
Establishing attributes of/measures for each objective.
4.
Generating a matrix across these objectives and options
and addressing the tradeoffs that selecting one option
over another entails.
SNR
11
Skunkwor
ks
Water Use Planning in B.C.
Participants
•
•
•
•
•
•
BC Hydro (Crown Corporation)
Federal Government
Provincial Government
Local Government
First Nations
Community Stakeholders
– Home owners, business operators, etc.
SNR
12
Skunkwor
ks
Water Use Planning in B.C.
Objectives, Attributes, Measures
Objectives
Attributes
Recreation
e.g., Weighted User Days
Erosion
e.g., Weighted Erosion Days
Flooding
e.g., Weighted Flood Days (flow level)
Fish
e.g., % Available Habitat, IBI
Water Supply
e.g., Water Quality Impact Rating
Cultural Heritage
e.g., Consistency Rating
Financial
e.g., Annual Revenues M$ / Year
SNR
13
Skunkwor
ks
Water Use Planning in B.C.
e.g., Financial Objectives & Attributes
MONITORING AND
MITIGATION PGMS
COST OF MONITORING
AND MITIGATION PGMS
NEW CAPITAL WORKS
CAPITAL COSTS
TOTAL COSTS
OPERATING COSTS
MARKET VALUE
OF POWER
TOTAL FINANCIAL
VALUE
WATER MANAGEMENT:
POWER GENERATION:




RESERVOIR LEVELS
FLOW RATES
DIVERSION FLOWS
RAMPING RATES


TOTAL REVENUES
AMOUNT OF
GENERATION
TIME OF GENERATION
THERMAL POWER
PRODUCED
GHG EMISSIONS
SNR
14
Skunkwor
ks
Water Use Planning in B.C.
e.g., Recreation Objectives & Attributes
RESERVOIR
LEVELS
ACCESS TO:



SHORELINE
EROSION
BEACH
BOAT LAUNCH
SHORELINE
VISUAL
QUALITY
DEBRIS
MANAGEMENT
DIVERSION
FLOWS
FLOW RATES
STANDING
DEBRIS
FLOATING
DEBRIS
BOATING &
SWIMMING


CONDITIONS
SAFETY
RECREATION


OPPORTUNITIES
QUALITY
NATURAL HISTORY
NON-CONFORMING
PARK USE
KAYAK & CANOEING
CONDITIONS
TOURISM
SNR
15
Skunkwor
ks
Water Use Planning in B.C.
Assessing Value
Objectives
Attributes
Mimic Natural
Hydrograph
Enhanced Summer
Releases
Enhanced Winter
Releases
Environment
% Viable
Spawning
Habitat
50
20
25
NPV ($)
$60 Million
$80 Million
$65 Million
Number of
User-Days
1400
1200
1500
Conserve
Salmon
Economic
Revenue
Generation
Social
Recreation
Opportunities
SNR
16
Skunkwor
ks
Water Use Planning in B.C.
Risk & Uncertainty
Objectives
Attributes
Mimic Natural
Hydrograph
Enhanced
Summer
Releases
Enhanced
Winter
Releases
Environment
Conserve
Salmon
% Viable
Spawning
Habitat
50
20
30
Economic
Revenue
Generation
NPV ($$)
$ 60 M
$ 80 M
$ 65 M
Social
Recreation
Opportunities
# of user
days
1400
1200
1500
SNR
Nat
Nat
Sum
60
40
100
80
10
140
Probability
1.0
Range of Values
Expected
Low
High
RISK PROFILES
Sum
0.8
0.6
0.4
0.2
20
60
100
140
Expected Value
17
Skunkwor
ks
Water Use Planning in B.C.
Objective
Upper Campbell / Buttle Lake
Erosion - Days / Year
Recreation - Days / Year
Effective Littoral Zone
Lower Campbell / McIvor / Fry
Erosion - Days / Year
Recreation - Days / Year
Spawning Habitat - Cutthroat
Spawning Habitat - Rainbow
Campbell River
Flooding - Total Days
Recreation - Days / Year
Total Spill Days - All Species
Spawning Habitat - All Species
Rearing Habitat - All Species
Salmon River
Canoe Route - Days / Year
All Habitat - All Species
System-Wide
Power / Financial
Attribute
E
F
Alternatives
G
H
I
J
weighted days (220 and 221 m)
37
weighted days (217.5, 218.5, 200m by season)
43
hectares
91
13
40
107
4
106
93
3
158
214
3
158
215
3
158
220
weighted days (177.4 and 178.3 m)
weighted days (175.75 - 177.8 by season)
% Available Habitat
% Available Habitat
3
115
78
26
27
43
18
3
13
83
95
49
0
167
79
49
0
170
79
47
0
167
78
50
weighted days (300, 453, 530 cms)
weighted days (28 cms - 80 cms)
days (Q>340cms, Sept 22 - April 15)
% successful redds (Chum as indicator)
"Average" risk index (scale 0 - 1)
34
66
118
55
0.53
48
83
214
89
0.48
24
51
102
78
0.53
59
81
176
59
0.50
59
79
177
59
0.49
59
81
176
59
0.49
days (Q<6cms, April 1 - Oct 22)
"Average" risk index (scale 0 - 1)
162
0.54
167
0.47
153
0.44
204
0.48
183
0.47
204
0.47
Annual Revenue M $ / Year
68.5
64.6
68.6
65.1
65.3
64.1
SNR
18
Skunkwor
ks
Water Use Planning in B.C.
Moving From Alternative:
E
To Alternative:
G
Results In:
Upper Campbell / Buttle Lake
G A decrease in Erosion - Days / Year of 33 weighted days
G An increase in Recreation - Days / Year of 63 weighted days
Y An increase in Effective Littoral Zone of 2 hectares
Lower Campbell / McIvor / Fry
R An increase in Erosion - Days / Year of 10 weighted days
R A decrease in Recreation - Days / Year of 32 weighted days
G An increase in Spawning Habitat - Cutthroat of 17 %
G
R
G
Y
Campbell River
A decrease in Flooding - Total Days of 10 weighted days
A decrease in Recreation - Days / Year of 16 weighted days
An increase in Spawning Habitat - All Species of 23 %
No change in Rearing Habitat - All Species of 0 in average risk
Salmon River
Y A decrease in Canoe Route - Days / Year of 9 days
G A decrease in All Habitat - All Species of 0.1 in average risk
System-Wide
Y An increase in Power / Financial of 0.1 M $ / Year
SNR
Denotes an improvement in the PM greater than the significant difference.
Denotes a worsening in the PM greater than the significant difference.
Denotes a change in the PM that is less than the significant differencfe.
19
Skunkwor
ks
A High Quality Process?
1.
Government Support
•
•
2.
Cost-effective
•
•
3.
SNR
Approach originally conceived as a pilot project at a single
site
Now implemented at 23 sites province wide (18 Water Use
Plans completed to date)
Original budget to complete all 23 plans = $25 Million
Revised budget = Approx. $14 Million
Participant Satisfaction
•
•
•
Historically adversarial groups work together
BC Hydro has a clear mandate
Consensus plans often achieved
20
Skunkwor
ks
Experiment 1
Quantitative Measures of Quality
• Designed to compare two approaches
for involving stakeholders in water use
planning in B.C…
– …small groups (7-10)
– …both conditions provided with the same
information
– …“Structured” vs. “Unstructured” process
SNR
21
Skunkwor
ks
Experiment 1
Quantitative Measures of Quality
UNSTRUCTURED
1. Self-rating questions
2. Technical Information
• newspaper article
• informative booklet
• audio-documentary
3. Group Discussion
4. Evaluation of policy alternatives
• referendum
• willingness to pay
5. Self-rating questions
STRUCTURED
1. Self-rating questions
2. Technical Information
–
–
–
newspaper article
informative booklet
audio-documentary
3. Decision structuring
–
–
–
characterize values & objectives
group discussion about values and
objectives.
objectives ranking/tradeoffs
4. Evaluation of policy alternatives
–
–
referendum
willingness to pay
5. Self-rating questions
SNR
22
Skunkwor
ks
Experiment 1
Results
Self-rating Item
UNSTRUCTURED
x Start sStart x End sEnd
STRUCTURED
p
x Start sStart x End sEnd
p
Knowledge about power
3.71 1.34 4.23 1.10 <0.0001
3.58 1.34 4.39 1.13 <0.0001
Knowledge about salmon
3.84 1.58 4.21 1.41
<0.01
3.63 1.48 4.30 1.30 <0.0001
Amount of information on
which to base an informed
answer
3.23 1.54 3.52* 1.28
0.08
3.11 1.59 3.93* 1.46
<0.001
*<0.01
Comfort with givi ng policy
advice
4.34 1.48 4.47* 1.30
0.06
4.44 1.69 4.88* 1.75
<0.01
*<0.01
5.88 1.13
<0.05
Choices reflect what
matters
5.45 1.15
SNR
23
Skunkwor
ks
Experiment 1
Results
0.5
0.4
F
0.3
Unstructured
Structured
0.2
0.1
0
Costs
Technical
Issues
Social
Issues
Cultural
Issues
SNR
24
Skunkwor
ks
Experiment 1
Conclusions
• Based on participants’ self-ratings and
an analysis of deliberation periods, we
conclude that the the structured
approach leads to higher quality
decisions.
• Arvai, J. L., R. Gregory, and T. McDaniels.
2001. Testing a structured decision approach:
Value-focused thinking for deliberative risk
communication. Risk Analysis, 21: 1065-1076.
SNR
25
Skunkwor
ks
Experiment 2
Analyzing Choices
• Decision structuring to alleviate
embedding in environmental valuation
– …when a good is assigned a higher value
on its own vs. when it’s part of a more
inclusive set.
• e.g., 5:1 differences in WTP for a single vs. a
set of disaster preparedness services
SNR
26
Skunkwor
ks
Experiment 2
Analyzing Choices
• Providing much-needed structure may
help to overcome the embedding
problem by helping decision makers to
think about the components of a
valuation problem.
SNR
27
Skunkwor
ks
Experiment 2
Analyzing Choices
VERSION A
1. Technical Information
–
–
–
newspaper article
informative booklet
audio-documentary
2. Decision structuring
–
–
–
characterize values & objectives
group discussion about values and
objectives.
objectives ranking/tradeoffs
VERSION B
1. Technical Information
–
–
–
newspaper article
informative booklet
audio-documentary
2. Decision structuring
–
–
–
characterize values & objectives
group discussion about values and
objectives.
objectives ranking/tradeoffs
3. WTP 1 Rivers
3. WTP 10 Rivers
4. WTP 10 River
4. WTP 1 River
SNR
28
Skunkwor
ks
Experiment 2
Analyzing Choices
CHOICE 1
CHOICE 2
VERSION A
WTP 1 River
W
WTP 10 Rivers
X
VERSION B
WTP 10 Rivers
Z
WTP 1 River
Y
SNR
29
Skunkwor
ks
Experiment 2
Conclusions
• Based on participants’ mean WTP
judgments, embedding was not
alleviated (according to the ratio
standard).
• McDaniels, T., R. Gregory, J. L. Arvai, and R.
Chuenpagdee. 2003. Decision structuring as a
means of alleviating embedding in
environmental valuation. Ecological Economics,
44: 33-46.
SNR
30
Skunkwor
ks
Experiment 3
2-Part Evaluation
• An experiment that would measure both
self-ratings of quality and subjects’
choices…
SNR
31
Skunkwor
ks
Experiment 3
2-Part Evaluation
UNSTRUCTURED
1. Self-rating questions
2. Technical Information
3. Choice Task
• Funding allocations across
three risk problems
4. Self-rating questions
STRUCTURED
1. Self-rating questions
2. Technical Information
3. Decision Structuring
•
•
Objectives ranking,
tradeoffs
Linking objectives with
management problems
4. Choice Task
•
Funding allocations
across three risk
problems
5. Self-rating questions
SNR
32
Skunkwor
ks
Experiment 3
Three Risk Problems
Problem
Affect Rating
Corresponding
Objective
Wildlife Disease
Affect Rich
Human Health
Deer
Overpopulation
Affect Neutral
Environmental
Health
Damaged Trails
Affect Neutral
Recreation
SNR
33
Skunkwor
ks
Experiment 3
Self-Rating Results
Unstructured
Self-rating item
Knowledge
(Trails)
Knowledge
(DOP)
Knowledge
(Disease)
Comfort with
providing input
Choices reflect
what matters
Satisfaction
with choice
Structured
x Start
SEStart
x End
SEEnd
p
x Start
SEStart
x End
SEEnd
p
3.1
0.25
4.1
0.22
<0.001*
3.7
0.21
4.4
0.23
<0.001*
3.8
0.24
4.4
0.22
<0.001*
3.7
0.22
4.4
0.17
0.001*
3.3
0.20
4.1
0.22
<0.001*
3.7
0.21
4.3
0.21
0.005*
4.5
0.25
5.1
0.23
<0.05
--
--
4.9
0.21
--
--
--
5.5
0.17
<0.05
--
--
4.5
0.25
--
--
--
5.1
0.23
<0.05
SNR
34
Skunkwor
ks
Experiment 3
Ranked Objectives
OBJECTIVE
RANK
Environmental
Health
1=Overpopulation
Human Health
1
2=Disease
2
Recreation
3=Trails
3
SNR
35
Skunkwor
ks
Experiment 3
Funding Allocations
SNR
36
Skunkwor
ks
Experiment 3
Conclusions
• Appears to be a disconnect between
self-ratings of quality and actual funding
choices
• Wilson, R.S. and J. L. Arvai. 2004. Evaluating
the quality of structured risk management
decisions. In Review.
SNR
37
Skunkwor
ks
Parting Thoughts
• Level of facilitation, time for deliberation,
increased attention to tradeoffs, etc. all
seem to be critical.
SNR
38
Skunkwor
ks
Parting Thoughts
• Anecdotal observations or evaluations
based on self-reports alone are likely
insufficient for evaluating the quality of
structured decision approaches.
SNR
39
Skunkwor
ks
Parting Thoughts
• Affective responses to stimuli exert
powerful influences on risk judgments
SNR
40
Skunkwor
ks
Parting Thoughts
• Not suggesting that the outcomes of all
structured decision making approaches
are suspect.
SNR
41
Skunkwor
ks
Thanks
Tim McDaniels
Robin Gregory
Ying Chuenpagdee
Robyn Wilson
Louie Rivers
Dan Ohlsen
SSHRC
NSF
OARDC
EPI
SNR
42
Skunkwor
ks