NVVK, 7 October 2015
Getting effective change after incidents
John Kingston
Questions posed by NVVK
1. How do you get from analysis results to concrete
(operational) measures?
a)
What is the influence of your methods and models on the
measures that are derived from the results?
b)
Who do you involve in the process?
c)
How do you ensure that the measures are concrete?
2. What about the effectiveness of our measures?
a)
How do you keep track of the measures?
b)
How do you determine their effectiveness?
c)
Who do you involve in the process?
Questions posed by NVVK
1. How do you get from analysis results to
concrete (operational) measures?
a) What is the influence of your methods and models
on the measures that are derived from the results?
b) Who do you involve in the process?
c) How do you ensure that the measures are concrete?
2. What about the effectiveness of our measures?
a) How do you keep track of the measures?
b) How do you determine their effectiveness?
c) Who do you involve in the process?
Analysis is not truth
• Spurious Objectivity
– Systematic ≠ complete
Systematic does not always mean
comprehensive
– Ignorance2 : analysis and
analysts have limits—
• Imposed by paradigm
• Framing/scope
• Bounded rationality
• Biases
• ‘Superknowers’ vs. social
construction of meaning
• More information is needed to
decide preventative action
Questions posed by NVVK
1. How do you get from analysis results to
concrete (operational) measures?
a) What is the influence of your methods and models
on the measures that are derived from the results?
b) Who do you involve in the process?
c) How do you ensure that the measures are concrete?
2. What about the effectiveness of our measures?
a) How do you keep track of the measures?
b) How do you determine their effectiveness?
c) Who do you involve in the process?
HOW TO MOBILISE KNOWLEDGE?
OPENNESS TO LISTEN &
OPENNESS TO SPEAK
TACIT KNOWLEDGE
• People may know more
than they want to tell
• ‘Inclusive’ investigation
• Trust and ‘Just-Culture’
• People know more than
they can tell
• Involvement
– 1st and 2nd Stories
– Beware of hard-wiring
investigation to the discipline
process
– in ‘Sense-making’
– In the design and
implementation process (the
lifecycle)
Questions posed by NVVK
1. How do you get from analysis results to
concrete (operational) measures?
a) What is the influence of your methods and models
on the measures that are derived from the results?
b) Who do you involve in the process?
c) How do you ensure that the measures are concrete?
2. What about the effectiveness of our measures?
a) How do you keep track of the measures?
b) How do you determine their effectiveness?
c) Who do you involve in the process?
Recommendations and Actions
Recommendations
• Focus the risk owner (and
other stakeholders) on the
problem to be solved
• Tightly specify the goal, and
allow the risk owner space
to find a specific solution
Actions
• Specific selections from
various alternatives
• SMART
• (does the “A” stand for
Achievable/Attainable or
Agreed?)
BRAN is good for you
When designing a recommendation
or and action, ask:
•
•
•
•
What are the Benefits?
What are the Risks of doing it?
What are the Alternatives?
What about Not doing it?
“CONCRETE (OPERATIONAL) MEASURES”
• Concrete operational measures: just
downstream, or upstream too?
– Policy formulation and Corporate
Governance
Upstream
– Design and Risk management processes
– Implementation processes
– ‘Risk Control Systems’
– Materials, Mechanisms, Behaviours and
Conditions
Downstream
SOME CONCRETE IS HARD TO MIX!
•Very novel or challenging
•Strong stakeholder views
•Significant risk trade-offs or risk transfer
•Large uncertainties
•Perceived lowering of safety standards
Societal
External consultation
A
Values
Internal consultation
•Lifecycle implications
•Some risk trade-offs/transfers
•Some uncertainty or deviation
from standard or best practice
•Significant economic implications
Benchmarking
B
Peer Review
Verification
CS&R
Codes &
C
Standards
•Nothing new or unusual
•Well understood risks
•Established practice
•No major stakeholder implications
The x-axis ratio indicates how much each
source of knowledge can contribute to
informing a decision about a risk in the yaxis context
From “A Framework for risk Related Decision-Support”, UKOOA (1999). (UKOOA is now Oil & Gas UK)
Questions posed by NVVK
1. How do you get from analysis results to concrete
(operational) measures?
a) What is the influence of your methods and models on
the measures that are derived from the results?
b) Who do you involve in the process?
c) How do you ensure that the measures are concrete?
2. What about the effectiveness of our measures?
a) How do you keep track of the measures?
b) How do you determine their effectiveness?
c) Who do you involve in the process?
ISO 31000:2009, Section 5
“Risk Treatment involves a cyclical process of
⎯ assessing a risk treatment;
⎯ deciding whether residual risk levels are tolerable;
⎯ if not tolerable, generating a new risk treatment; and
⎯ assessing the effectiveness of that treatment.”
Other useful ideas in section 5
–
–
–
–
Secondary Risks, and their management
Documented risk treatment plan
Stakeholder involvement
Monitoring and Review
Evaluating Effectiveness
• Some considerations
1. Implementation ≠ Evaluation
2. Baseline measurements to allow comparisons?
3. Possibility of (quasi-) experimental trials?
4. Downstream is easier to evaluate than upstream
5. Prove vs. Improve
6. Effectiveness may be short-term
7. How is effectiveness measured and who decides?
8. Openness to unintended consequences
9. Steady state monitoring and review
10. Trevor Kletz’s wisdom