Analytical Decision
Making for Financial
Managers
Defense Resources Management Institute
Naval Postgraduate School
Monterey, California
How do you make decisions?
2
Analysis
The process of breaking a
complex topic or problem
into smaller parts to gain a
better understanding of it
3
Robert McNamara (1961)
• “Major decisions should be
made by choices among explicit,
balanced, feasible alternatives”
• “The Secretary should have an
active analytic staff to provide
him with relevant data and
unbiased perspectives”
• “Open and explicit analysis,
available to all parties, must
form the basis for major
decisions”
4
Decision Maker and Analyst
New problem (never encountered)
Decision
maker
Analyst
Experience
and
judgment
SOLUTION
5
Why Do Analysis?
• Analysis can be difficult, time consuming, and
expensive
• But analysis creates
Answers that are accessible to critical examination
Answers that can be retraced by others
Answers that account for different factors and
elements
• And it leads to better decisions
6
Elements of Analysis
• Goals: What the decision maker is trying to achieve
• Objectives: Outcomes that you want to occur to
achieve a goal
• Alternatives: Choices available to achieve goals
• Models: Tools for predicting and evaluating the
consequences of choosing an alternative
• Preferences: Rules for ranking the alternatives (best
to worst)
7
Process of Analysis
Formulation
(conceptual
phase)
Search
(research
phase)
Define issues
of concern
Develop
alternatives
Clarify
objective
Look for data
Scope
problem
Evaluation
(analytic phase)
Build
mathematical
models
Use models
to predict
consequences
Interpretation
(judgmental
phase)
Compare
alternatives
based on
model
predictions
Derive
conclusions
Identify
alternatives
Build mental
models
Indicate
courses of
action
8
Process and Elements of Analysis
Formulation
(conceptual
phase)
Search
(research
phase)
Goal
Alternatives
Objective
Data
Evaluation
(analytic phase)
Interpretation
(judgmental
phase)
Model
Model
output
Preferences
How well are we
doing on our
?
9
Formulation Phase
• Focus on the right problem
• Do not jump immediately to solving the
problem
• Think about
Goal(s)
Objective(s)
Scope
10
Work on the Right Decision Problem
•
•
•
•
•
Ask yourself why there is a problem
Ask what triggered the decision
Focus on the problem not the symptoms
Be creative about defining the problem
Turn problems into opportunities
What can you gain from the situation?
What are the opportunities here?
11
Goal
• Desired end state, what
are you ultimately trying to
achieve
• Binary condition: either
you achieve the goal or
you don’t
• Examples
Achieve 95% availability
Eliminate IED attacks
12
Objective
• Outcomes that will help you to achieve a goal
• Directionally oriented--usually maximize or
minimize
• Examples:
Maximize number of IEDs detected
Minimize IED production
Maximize reliability of radar
Maximize effectiveness
13
Identifying Objectives: First Steps
• Start with strategic objectives: review planning
and strategy documents
• Identify appropriate stakeholders and involve
them in the process
Decision makers
Superiors or commanding officers
Other leaders
Operators or customers
Community leaders
Government agencies
Legislative bodies
14
Generating Possible Objectives
• Expansive generation of objectives (pruning and
structuring comes later)
• Solicit others’ ideas but avoid group work initially
• Brainstorming: solicit objectives from
stakeholders without evaluating them
You may or may not want stakeholders in a room
together
Try to focus on objectives not positions or alternatives
15
Some Questions to
Generate Objectives
• What is your goal?
• Why is there a decision to be made?
• If you had no limitations or constraints, what
would your objectives be?
• What is your ideal outcome of the decision, and
what makes it so ideal?
• What is your nightmare scenario and what makes
it so bad?
• What consequences would be unacceptable?
16
Discovering Your Goal(s)
Descriptive
scenario
COMPARE
The way things are
Needs
Prescriptive
scenario
The way things
should be
Goals
Objectives
17
Military Healthcare Example
Healthcare for military veterans has been provided by military
hospitals. A recent study found that civilian hospitals generally
have more advanced medical technology and better trained
physicians than military hospitals. Additionally, an increase in the
number of veterans has led to longer wait times for some
services at military hospitals. The director of veterans affairs is
considering a plan under which the Ministry of Defense would
pay for veterans to receive services at civilian hospitals.
What should be the goal for the director of veterans affairs when
considering whether to adopt the new plan?
18
Cyber Security Example
The Army’s chief technology officer is concerned about the
increasing number of cyber attacks against the Army’s computer
network. He is unsure whether these attacks are coming from
individual hackers who want to cause mischief or from the
country’s primary rival for power in the region. Although the vast
majority of cyber attacks are unsuccessful at cracking the Army’s
firewall, some attacks have gotten through the firewall although
even these attacks have been discovered quickly and thwarted
thus far.
What should be the goal of the chief technology officer when
considering different cyber security alternatives?
19
Military Healthcare Example
What objectives support the primary goal in the
military healthcare example?
20
Scope
Extent or range of viewpoint or outlook of analysis
Scoping in
Narrowing the problem
Considering one specific
decision
More easily tackled
definition
Scoping out
Broadening the problem
Considering other related
decisions
Comprehensive definition
21
Military Healthcare Example
• Scoping out the problem
 How to measure quality of care?
 What to do with military doctors employed by veterans
affairs? Current military hospitals?
 Will decision impact current force structure of the
military?
• Scoping in the problem: Assume quality of care is
equal under either alternative and select lowest-cost
alternative
• Changing scope: Understand what the problems are
in the current structure and trying to fix those
problems
22
What Does This Mean?
Analysis can help you solve a problem and make
a better decision, but only if you are working on
the right problem
• Define the scope that you should be
considering
• Think about broadening the scope
• Define goals and objectives to correspond
with the scope
23
Process of Analysis
Formulation
(conceptual
phase)
Search
(research
phase)
Define issues
of concern
Develop
alternatives
Clarify
objective
Look for data
Scope
problem
Evaluation
(analytic phase)
Build
mathematical
models
Use models
to predict
consequences
Interpretation
(judgmental
phase)
Compare
alternatives
based on
model
predictions
Derive
conclusions
Identify
alternatives
Build mental
models
Indicate
courses of
action
24
Search Phase
• Asking questions
What are the alternatives?
What data do we need?
What are the important elements and relationships
of this decision?
• Designing alternatives
Identify all courses of action available to you
Select an appropriate set of possible alternatives to
examine
• Too few  may not find the best one
• Too many  may reduce quality of analysis
25
Decision Elements
Should be identified in formulation and search
phases
FUTURE
CONDITIONS
Courses of action
ALTERNATIVES
PREFERENCES
Result
Value
OUTCOMES
PAYOFF
26
Outcome Vs. Payoff
• Outcome: Result or
consequence that can occur
based on the alternative and
future condition
• Payoff
 How the decision maker “feels”
about the outcome
 Based on the decision maker’s
preferences
• Outcome and payoff can be the
same thing (example: money)
27
Mental Models
• Graphically depict decision elements using
shapes and arrows
Future Conditions
ALTERNATIVES
Preferences
OUTCOMES
PAYOFF
28
Infection
Control
Healthy
Soldiers
Transmission
Natural
Diseases
Bioterrorism
Hospital
Care
Sanitation
Treatments
Detection
Development
29
Hints for Mental Maps
• Begin simple and then make it more complex if
necessary
• Do not specify each alternative
 Decision node captures several alternatives
• Only include elements that impact your decision or
goal/objective
30
Nuclear Weapons Program Example
Decision
Interdiction
strategy
Political and
economic
environment
Future
nuclear
defense
Direct
security
threats
Regional
nuclear
environment
International
standing
Political
leader
Domestic
support
National
security
Nuclear
weapons
program
Outcome
Adapted from D.J. Caswell and M.E. Paté-Cornell, 2011, “Probabilistic analysis of a country’s program to
acquire nuclear weapons,” Military Operations Research 16(1): 5-20.
31
Importance of Mental Maps
• Structure the problem
Identify outcomes, future conditions, decisions
Examine relationships between variables
• Provide framework for multiple decision
makers
32
Why Structure Problems?
We impose structure (like mental
mappings) on decision problems to
• Gain insight into the problem
• Specify what we do and don’t “control”
• Better understand uncertainty
• Facilitate quantitative analysis
33
Process of Analysis
Formulation
(conceptual
phase)
Search
(research
phase)
Define issues
of concern
Develop
alternatives
Clarify
objective
Look for data
Scope
problem
Evaluation
(analytic phase)
Build
mathematical
models
Use models
to predict
consequences
Interpretation
(judgmental
phase)
Compare
alternatives
based on
model
predictions
Derive
conclusions
Identify
alternatives
Build mental
models
Indicate
courses of
action
34
Evaluation Phase
• Model: simplified
version of reality
• Mathematical
models to
 Predict
consequences
 Evaluate decisions
 Select alternatives
All models
Mental
models
Mathematical
models
M=B-X
Physical
models
Symbolic
models
Verbal
models
Amount of
money
available is
budget minus
expenses
35
Mathematical Model Components
1. Variables
Represent things that change, either
with our choice, or because they are
inherently unknown or uncertain
2. Relations
Represent the connections between
variables
3. Parameters
Represent the assumptions and
information available from data
𝑥
𝑦
𝑧
≤
+ =
𝑔 = 9.8 𝑚
𝑠2
36
Mathematical Models
• Mathematical models can be classified by
their representational form
Algebraic
𝑦 = 1 + 2𝑥
Tabular
𝒙
𝒚
1
3
2
5
3
7
7
Graphical
𝑦
1
3
𝑥
37
Mathematical Models
• Mathematical models can be classified by
their treatment of uncertainty
Deterministic
Probabilistic
• Mathematical models can be classified by
their treatment of time
Static
Dynamic
38
Mathematical Models
• Mathematical models can be classified by their
type of application
Optimization (prescriptive)
• Maximize or minimize an objective
• Subject to some constraints
• Calculate best alternative
Simulation (descriptive)
• What is the relationship between different variables?
• How often is an event likely to occur?
• Simulation models can also be used as inputs into
an optimization model
39
Importance of Models
• Allow us to create an ordering over all the
alternatives.
• Aid decision makers in ranking alternatives using
something universally understood
A4
A5
A1
A2
A6
A3
-9
5
15
21
34
42
REALLY BAD
BAD
O.K.
GOOD
BEST
40
Model Building Process
No
Simplify
Identify
the
problem
Make
assumptions
Can you
make the
model?
Yes
No
Simplify
Meaningful
output?
Yes
Test the
model
Maintain
the model
Implement
the model
Yes
Is the
model
useful?
No
41
Simplification and Assumption
• Most decision problems
have uncertainty and
change over time
• But these models are
difficult to build and to
solve
• Making assumptions
and simplifying is
necessary
Mathematical model
• Probabilistic
• Dynamic
Mathematical model
• Deterministic?
• Static?
42
Assumptions
• Decide which assumptions
 Are necessary in order to build and solve a model
 Make the model too simple to be useful
• Explain what assumptions are in the model
• Answer how the model output changes if
assumptions change
• Incorporate enough detail so that
 Results meet your needs
 Model is consistent with available data
 Model can be analyzed in the time available
43
Example: Renting a Car
• Goal: Select the cheapest rental
car company
• Objective: Minimize total weekly
costs
• Assumptions
1.
2.
3.
4.
All cars of equal size and type;
equally efficient and effective
No alternatives accept awards programs or other incentives
Available insurance options all equal
Services equal across all alternatives
44
Car Rental Company Selection
Mental map
Rental
company
Weekly
rental rate
Weekly
total cost
Mileage
rate
Mileage
cost
Miles
driven
45
Car Rental Company Selection
Three Rental Car Alternatives
1. Hurts
• $70 per week plus $0.30 per mile driven
2. Avion
• $100 per week plus $0.80 per mile
• But first 200 miles per week are free
3. Bottomdollar
• $160 per week with unlimited free miles
46
Car Rental Company Selection
Algebraic model
Hurts = $70 + $.30(mileage)
Avion = if mileage <= 200 then cost=$100
else cost = $100 + $.8(mileage - 200)
Bottomdollar = $160
47
Car Rental Company Selection
Tabular model
Hurts
Avion
Bottomdollar
50
85
100
160
Miles driven
100
200
300
100
130
160
100
100
180
160
160
160
375
183
240
160
48
Weekly Auto Rental Cost
$300
$250
Cost
$200
$150
$100
$50
$0
50
100
150
200
250
300
350
400
Miles Driven
Hurts
Avion
Bottomdollar
49
Model Building
• Decision rule becomes clear - when to
switch
• Reduces problem to its most important
element - mileage
• Aids in understanding the influence of
parameters
What happens if the flat rate on Bottom Dollar
drops to $120?
What happens if the milage rate on Hurts is
reduced to $0.60
50
Model Complexity
Error
Optimal complexity depends on the decision context!
Total
Oversimplification
Measurement
Complexity
51
Model Complexity
• A model always simplifies reality.
• Incorporate enough detail so that:
Results meet your needs
Model is consistent with available data
Model can be analyzed in the time available
Mental maps help you define this boundary
52
“Everything should
be made as
simple as possible - but no
simpler”
Albert Einstein
1879-1959
53
Process of Analysis
Formulation
(conceptual
phase)
Search
(research
phase)
Define issues
of concern
Develop
alternatives
Clarify
objective
Look for data
Scope
problem
Evaluation
(analytic phase)
Build
mathematical
models
Use models
to predict
consequences
Interpretation
(judgmental
phase)
Compare
alternatives
based on
model
predictions
Derive
conclusions
Identify
alternatives
Build mental
models
Indicate
courses of
action
54
Iteration
Formulation
(conceptual
phase)
Search
(research
phase)
Evaluation
(analytic phase)
Interpretation
(judgmental
phase)
• Analysis and model building are iterative processes
• Returning to formulation and search phases may be
beneficial
 Redefine goals and objectives
 Develop new alternatives
 Change assumptions
55
Strategic Air Command Air Base Study
Question
assumptions
Evaluate &
decide
Strategy
Test for
sensitivity
Logistics
Minimize
cost
Compare
outcomes
Predict
consequences
Predict
consequences
Build
models
Collect
data
Minimize
vulnerability
Identify
alternatives
Build
models
Collect
data
Identify
alternatives
56
Key Takeaways
• Before attempting to solve a problem
Think about your goals and objectives
Scope the problem so that it aligns with your goals
• Construct a mental map or use another
brainstorming tool to identify key variables
and decisions
• Don’t be afraid to go back and revisit
assumptions and/or revise your model
57
The
Analytical World
The
Real World
DM Preferences
Environment
?
Analytical Process
of
Decision Making
Decision
(alternative)
INTERACTION
of
DECISION
with
ENVIRONMENT
Outcome
VALUATION
of
OUTCOME
via
PREFERENCES
Payoff
“Market”
“Operations”
Combat
Humanitarian Relief
Peace Keeping
58
Our Model
of the
Real World
FUTURE
CONDITIONS
Environment
Decision
(alternative)
INTERACTION
of
DECISION
with
ENVIRONMENT
Outcome
GOAL
OBJECTIVE
Preference
VALUATION
of
OUTCOME
via
PREFERENCES
Payoff
59
Decision Environments
Certainty
Uncertainty
Complete
information
Incomplete
information
60
“… as we know, there are known knowns;
there are things we know we know.
We also know there are known unknowns;
that is to say we know there are some
things we do not know.
But there are also unknown unknowns -the ones we don't know we don't know …”
Secretary of Defense Donald H. Rumsfeld
http://www.defenselink.mil/news/Feb2002/t02122002_t212sdv2.html
61
Decision Environments
Certainty
Uncertainty
You know:
• all the alternatives
• the one future condition
• all the payoffs
62
Decision Matrix Under Certainty
Future Condition
Alternative 1
Alternative 2
:
Alternative 3
63
Decision Environments
Certainty
Uncertainty
Complete
information
Incomplete
information
You do not know one or more of:
• all the alternatives
• all the future conditions
• all the payoffs
• all the probabilities
of the future conditions
64
Decision Environments
Certainty
Uncertainty
Complete
information
Incomplete
information
You know:
• all the alternatives
• all the future conditions
• all the payoffs
• all the probabilities
of the future conditions
65
Static Decisions
Decision Matrix
Future
Condition 1
Future
Condition 2
•••
Alternative 1
Probability
Alternative 2
Outcome
Payoff
•
•
Alternative K
Future
Condition N
•
•
•••
66
Probability
• A measure of the likelihood of the
occurrence of a future event
• The quantification of uncertainty
67
Histogram
0.50
474
0.45
0.40
0.35
0.30
240
0.25
0.20
158
0.15
0.10
65
0.05
38
20
0.00
1.5 - 2.5
2.5 - 3.5
3.5 - 4.5
4.5 - 5.5
5.5 - 6.5
6.5 - 7.5
4
1
7.5 - 8.5
8.5 - 9.5
Relative Frequency
.5
Earthquake Problem
.474
.4
.3
.240
.2
.158
.1
0
Do
Nothing
0
New
Codes
10
& Retrofit
2.5
1.5
Implementation
Cost
New Codes
.065
F1
3.5
F2
4.5
F3
.038
5.5
F4
.020
7.5
6.5
F5
.004
F6
.001
8.5
F7
9.5
F8
50
69
Relative Frequency
.5
Earthquake Problem
.474
.4
.3
.240
.2
.158
.1
0
.065
2.5
1.5
Implementation
Cost
3.5
4.5
.038
5.5
.020
.004
7.5
6.5
F1
F2
F3
F4
F5
F6
.001
8.5
F7
Do
Nothing
0
25
250
2,500
25,000
250,000
2,500,000
New
Codes
10
18
74
522
4,106
32,778
262,154
2,097,162
50
51
58
106
472
3,214
23,780
178,029
New Codes
& Retrofit
9.5
F8
25,000,000 250,000,000
16,777,226
1,334,889
70
Relative Frequency
.5
Earthquake Problem
.474
Decision Rule:
.4
Most Likely
.3
.240
.2
.158
.1
0
.065
2.5
1.5
Implementation
Cost
F1
Do
Nothing
0
25
New
Codes
10
18
50
51
New Codes
& Retrofit
3.5
F2
4.5
F3
.038
5.5
F4
.020
7.5
6.5
F5
.004
F6
.001
8.5
F7
9.5
F8
71
Relative Frequency
.5
Earthquake Problem
.474
Decision Rule:
.4
Expected Value
.3
.240
.2
.158
.1
0
1.5
.065
2.5
3.5
4.5
Do
Nothing
411,592
New
Codes
32,030
New Codes
& Retrofit
.038
5.5
.020
6.5
.004
7.5
.001
8.5
9.5
2,730
72
Malaria Prevention
Mental Map
Exposed
to Malaria?
Take
Malaria
Pills?
Payoff
73
Malaria Prevention
(Decision Matrix)
F1
F2
Exposed Not exposed
to Malaria to Malaria
Take
Malaria Pills
Don’t Take
Malaria Pills
74
Dynamic Decisions
What happens when choice of an
alternative changes the matrix ?
• when the likelihoods change ?
• when the number of future
conditions change ?
• when there is a sequence of
decisions ?
75
Tree Model
NODES
BRANCHES
Decision node
Future condition node
76
Malaria Prevention
(Decision Tree)
Payoffs
Take
Malaria Pills
Exposed
to Malaria
Not Exposed
to Malaria
Don’t Take
Malaria Pills
Exposed
to Malaria
Not Exposed
to Malaria
77
Why use a decision tree?
(Sequential decisions)
• If you have been exposed to malaria, you
can take medications immediately after
exposure to prevent malaria
Lowers your chances of malaria
78
Malaria Prevention
Mental Map
Take
PostExposure
Pills?
Exposed
to Malaria?
Take
Malaria
Pills?
Develop
Malaria?
Payoff
79
Malaria Prevention
Malaria
Take PostExposure Pills
Take
Malaria Pills
Exposed
to Malaria
Malaria
No PostExposure Pills
Not Exposed
to Malaria
Don’t Take
Malaria Pills
Exposed
to Malaria
No Malaria
No Malaria
Malaria
Take PostExposure Pills
No Malaria
Malaria
Not Exposed
to Malaria
No PostExposure Pills
No Malaria
80
Which Alternative is Better?
Benefits
B
A
C
$ Cost
81
Which Alternative is Better?
Benefits
A
C
B
$ Cost
82
Which Alternative is Better?
Benefits
B
A
C
$ Cost
83
Guidance
(OMB Circular A-94)
“The goal is…to promote efficient
resource allocation through wellinformed decision-making by the
Federal Government.”
http://www.whitehouse.gov/omb/circulars_a094/
84
Decision Criteria
With UNLIMITED resources…if there is any
positive benefit, then no matter what it costs,
make the investment in:
National Security;
Transportation;
Education;
Health & Safety Regulations
Posner & Adler (Eds) 2001 Cost-Benefit Analysis
85
Why Worry About Cost?
• Any course of action, any decision, will exact
a cost
Cost is a measure of the consequences of our
decision
• As long as resources are limited, cost will be
a factor in our decision
86
Decision Criteria
• Equal Benefits
Minimize costs
• Equal Costs
Maximize benefits
• Different Costs and Benefits
If benefits can be monetized
• Net Present Value
If not…
• Need to make tradeoffs!
87
Getting Started
• Identify feasible, mutually exclusive
alternatives
• Define the planning horizon
• Develop cash flow profiles
• Specify the discount rate to be used
88
Planning Horizon
• The period of time over which the cash
flows of alternatives are compared
Must be the same for each alternative
Must consider useful life of alternatives
• Common approaches
Least common multiple of lives
Shortest life
Standard horizon
89
Specifying the Interest Rate
Circular A-94 sec 8
Benefit-Cost Analysis
• Benefits and costs can be monetized
• Real 7% (marginal pretax rate of return on an
average investment in the private sector)
– Market interest rates are nominal
Cost-Effectiveness Analysis
• Most DoD decisions fall in this category
• Treasury’s borrowing rates for comparable length of
maturity
– Published Treasury rates are nominal
90
OMB Guidance
• Cost-Effectiveness is appropriate whenever it
is unnecessary or impractical to consider the
dollar value of the benefits.
• Analysis of alternative defense systems often
falls in this category.
OMB Circular A-94, par. 5b
91
Cost-Effectiveness Analysis
• Benefits can not be quantified in monetary terms
 Define effectiveness based on desired
capabilities/characteristics
 Measure capabilities of alternatives and assign a
measure of effectiveness
• Identify "efficient frontier"
• Incremental analysis (tradeoffs)
  effectiveness vs.  cost
92
Risk is part of life
• Every decision in an uncertain world
involves some degree of risk.
There is the risk that the cost will be higher
than expected.
There is the risk that the benefit will be lower
than expected.
93
ASSESSING RISK
•
•
•
•
What can go wrong?
What is the likelihood?
What are the consequences?
How do we feel about the
consequences?
What do we mean by Cost Risk?
• What can go wrong?
The actual cost of a program exceeds the budget
for that program (cost overrun)
• What is the probability of a cost overrun?
• What are the consequences of a cost overrun?
Normal Distribution
Probabilities
68%
X
 3  2  1

 1  2  3
Normal Distribution
Probabilities
95%
X
 3  2  1

 1  2  3
Normal Distribution
Probabilities
99%
X
 3  2  1

 1  2  3
Thinking about the budget
Funding at the 50% level means there is a
50% chance of cost overrun.
Comparing the Risk of Alternatives
P(cost overrun)
= 9.63%
P(cost overrun)
= 97.76%
Acceptable Risk?
Risk cannot be spoken of as acceptable or not in
isolation, but only in combination with the costs and
benefits that are attendant to that risk. Considered in
isolation, no risk is acceptable! A rational person
would not accept any risk at all except possibly in
return for the benefits that come along with it.
Even then, if a risk is acceptable on that basis, it is still
not acceptable if it is possible to obtain the same
benefit in another way with less risk.
Kaplan and Garrick, “On The Quantitative Definition of Risk”,
Risk Analysis, Vol. 1, No. 1, 1981
101
Risk Tradeoff
Risk
Too Risky
Acceptable
Risk
Proposed Budget
Required Budget
Cost
102
Risk Management
Risk
old
new
Cost
103
Risk Management
Risk
Less Risk
new
Proposed Budget
Cost
104
Risk Management
Risk
Acceptable
Risk
new
Less Cost
Cost
105
Ultimately, policy makers must decide how much
the United States is willing to pay to lower the risks
associated with deploying forces abroad. But some
might argue that defense planners occasionally focus
on absolute requirements – the minimum number of
forces that they believe will meet DoD’s military
needs – without fully weighing the relative risks
and costs of alternative levels.
Moving U.S. Forces: Options for Strategic Mobility
Congressional Budget Office, Feb. 1997
106
Pitfalls in All Analysis
• Not enough time spent defining the
problem.
• Examining a restricted range of
alternatives.
• Too much time spent in the details of
the models.
107
Advantages of Analysis
• Answers are accessible to critical
examination
• Answers can be retraced by others
• Answers can be modified by others
108