Guidelines for
Research and Evaluation
Global Health Cluster
Geneva
14 December 2011
World Association for Disaster and
Emergency Medicine
Sundnes KO, Birnbaum ML, O’Rourke AP, Daily EK
Global Health Cluster Meeting
Geneva - December 2011
Background
• Knut Ole Sundnes
• Task Force on Quality Control of Disaster
Medicine (WADEM)
• Expectation Template
• Results unexpected
• Breakthrough at Utstein Abbey
Global Health Cluster Meeting
Geneva - December 2011
Andre
w
91
9
1
ico
x
e
M
Haiti 05 February 2010
Bosnia
“Why have we not learned from what we
have learned?”
David Nabarro, 2005
WHO Assistant Director General
Health Actions in Crisis
Lessons Learned vs. Lessons Observed
Global Health Cluster Meeting
Geneva - December 2011
Sources of Information
• Peer-reviewed journals
• Books
• Gray literature (NLM—What do we do with
it?)
• After-action reports (unstructured)
• Media !!!!
• Interviews of……
Global Health Cluster Meeting
Geneva - December 2011
Event-Specific Literature Search
Smith, Wasiak, Sen, et al (PDM 2009;24:306-311)
75 journals, 789 papers
Number Publications/Journal
Percent Journals
Disasters Due to Natural Hazards (last 30 years)
70
60
50
40
30
20
10
0
64
17
1
2
11
3
2
3
1
1
1
4
5
6-10
11-20
>20
Number/Journal
Global Health Cluster Meeting
Geneva - December 2011
SE Asia Earthquake & Tsunami (2004)
Kohl, Ofrin, Salunke, Birnbaum (as of 16 Feb 2007)
Percent of Sources
SE Asia Earthquake & Tsunami (n =167)
70
60
50
40
30
20
10
0
66
13
1
2
6
2
2
5
3
3
4
5
6-9
>9
Number Publications/Journal
11% in Peer-reviewed Literature!!!!
Global Health Cluster Meeting
Geneva - December 2011
Major Issues
•
•
•
•
•
Gray literature
Not Indexed difficult access
No Structure
UN Documents
No Uniform Definitions
Global Health Cluster Meeting
Geneva - December 2011
Disaster Research and
Evaluation
1. Understand the epidemiology of disasters to:
• Know what to expect
• Identify strategies to enhance resilience to specific events
• Identify factors common to all disasters, regardless of type of event
2. Evaluate interventions to:
• Determine those with optimal effectiveness, efficacy, and benefit to the
affected/at-risk population
• Determine those with highest level of efficiency at lowest cost
• Determine “side effects”
• “Accountability”
3. Build Science of Disaster Health
Global Health Cluster Meeting
Geneva - December 2011
Information & Evidence
Evidence-based vs. Evidence-informed
Evidence-based vs. Best-available
Evidence-based vs. Eminence-based
Global Health Cluster Meeting
Geneva - December 2011
Evaluation and Research
Problems
• Although predictable, many strike
unexpectedly
• No two disasters are exactly the same
• Different geographic areas
• Population and cultural differences
• Different levels of preparedness for event
• Research a low priority in times of crisis
• Difficult/impossible to do RCT
• ?Prove Cause:Effect
• Media drives agenda
Global Health Cluster Meeting
Geneva - December 2011
Evaluation and Research
Barriers
1. Lack of uniformly accepted,
standardized definitions
2. Lack of a framework to provide a
structure
3. Lack of endorsed set of indicators
Global Health Cluster Meeting
Geneva - December 2011
Lack structured evidence
Unable to build the science
- Without science
-No standards/best practices
-? Competencies?curricula
-Unable to credential/accredit
Global Health Cluster Meeting
Geneva - December 2011
5 Frameworks
Epidemiological
1. Conceptual
2. Longitudinal
3. Transectional
Interventional
4. Operational
5. Preparedness
Global Health Cluster Meeting
Geneva - December 2011
1. Conceptual Framework
Global Health Cluster Meeting
Geneva - December 2011
Hazard
Event
Damage
Changes in Function
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Hazard
Event
Damage
Changes in Function
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Risk
The probability that something will
occur
•
•
•
•
“Probability” = likelihood, chance, prediction
Probability ranges 0 to 1
“Chance” = % something will happen
Problem: Difficult to quantitate predictions, 1 in
10, 1 in 1,000, 1 in a million!!!!
• Qualitative: “high”, “moderate”, “medium”, “mild”,
“slight”, “minimum”
Global Health Cluster Meeting
Geneva - December 2011
Probability vs Risk
“Risk Reduction” “Risk Management”
World Health Organization
International Strategy for Disaster Reduction (ISDR)
UN-Office for the Coordination of Humanitarian Affairs
(OCHA) and its Inter-Agency Standing Committee
(IASC)
Probability = Risk
Global Health Cluster Meeting
Geneva - December 2011
Hazard
RiskEvent
Event
Damage
Changes in Function
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Hazard
RiskEvent
Hazard Modification
Event
Damage
Changes in Function
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Hazard
RiskEvent
Hazard Modification
Event
RiskDamage
Damage
Changes in Function
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Hazard
RiskEvent
Hazard Modification
Event
RiskDamage
Absorbing Capacity
Damage
Changes in Function
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Absorbing Capacity
• The ability to absorb the free energy of an event without
sustaining damage
–
–
–
–
–
–
–
Levees
Building codes
Reforestation
Boarding windows
Bomb shelters
Bullet proof vests
Air bags
• Mitigation is anything done to increase absorbing
capacity or reduce exposure
–
–
–
Evacuation
Relocation
Isolation/quarantine
Global Health Cluster Meeting
Geneva - December 2011
Damage
Absorbing Capacity
Absorbing Capacity
Global Health Cluster Meeting
Geneva - December 2011
Hazard
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Changes in Function
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Hazard
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Buffering Capacity
• The ability of a society to cope with damage and
to function despite the damage
• The ability to maintain essential functions
despite a change in available resources
–
–
–
–
–
Backup Generators
Alternate sources of fuel
Shelters
Cross-trained personnel
Wells/rain water
Global Health Cluster Meeting
Geneva - December 2011
Buffering Capacity
Level of Function
High
Low
Time
Global Health Cluster Meeting
Geneva - December 2011
Hazard
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Local Response
Capacity
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Response Capacity
ability to intervene to prevent further
deterioration of functions associated with
the damage from an event, as well as
meet the needs associated with the
change(s) in function(s), and/or to restore
the functioning of the impacted society to
its pre-event levels of functions (recovery)
Global Health Cluster Meeting
Geneva - December 2011
Response Capacity
Local—without the need for assistance
from outside of the affected area. The
local responses to local needs are a
function of the local response capacity
(RC).
Outside—use of response capacity(ies)
from outside of the area directly affected
Global Health Cluster Meeting
Geneva - December 2011
Hazard
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Risk Emergency
Local Response
Capacity
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Hazard
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Risk Emergency
Risk Disaster
Local Response
Capacity
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Outside Response Capacity
Hazard
RiskEvent
Hazard Modification
Event
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Risk Emergency
Risk Disaster
Local Response
Capacity
Needs
Local Responses
Emergency
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Resilience
RiskDamage
Outside Response Capacity
Resilience =
Absorbing Capacity
+
Buffering Capacity
+
Local Response Capacity
Global Health Cluster Meeting
Geneva - December 2011
Examples of Resilience
- Construction of safe rooms (tornado
and severe wind shelters)
- Generators and related equipment
- Warning and alert notification
systems
- Fire-retardant materials
- Cross-training staff
- Surge capacity
Global Health Cluster Meeting
Geneva - December 2011
Resilience
Capacity Building
Global Health Cluster Meeting
Geneva - December 2011
Assessments  Functional Deficits
Functional Deficits  Needs
Needs = Goods and Services
Goods and Services  Improve
Function
Global Health Cluster Meeting
Geneva - December 2011
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Risk Emergency
Risk Disaster
Local Response
Capacity
Needs
Local Responses
Emergency
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Resilience
Risk Reduction
Preparedness
Hazard
Outside Response Capacity
What is depicted below?
A. Absorbing Capacity
B. Buffering Capacity
C. Response Capacity
D. Disaster
C. Response
Capacity
Global Health Cluster Meeting
Geneva - December 2011
What is depicted below?
A. Absorbing Capacity
C. Response Capacity
B. Buffering Capacity
D. Disaster
B. Buffering Capacity
Global Health Cluster Meeting
Geneva - December 2011
What is depicted below?
A. Absorbing Capacity
B. Buffering Capacity
C. Response Capacity
D. Disaster
A. Absorbing Capacity
Global Health Cluster Meeting
Geneva - December 2011
5 Frameworks
Epidemiological
1. Conceptual
2. Longitudinal
3. Transectional
Interventional
4. Operational
5. Preparedness
Global Health Cluster Meeting
Geneva - December 2011
Longitudinal Phases
for Definition of Epidemiology
Pre-event
Event
Damage
Change in Function
Relief
Recovery
time
Development
Pre-event
Event
Damage
Change in Function
time
Global Health Cluster Meeting
Geneva - December 2011
Relief
Recovery
Transectional “Snapshots”
Pre-event
Event
Damage
Change in Function
Relief
Recovery
time
1
2
Assessment
Global Health Cluster Meeting
Geneva - December 2011
3
Transectional Structure
of Society
Public Health
Economy
Medical Care
Communications
Water & Sanitation
Security
Society
Logistics &
Transport
Social Structure
Public Works
& Engineering
Global Health Cluster Meeting
Geneva - December 2011
Shelter & Clothing
Food & Nutrition
Education
Energy Supply
Transectional Structure
of Society
Public Health
Economy
Medical Care
Communications
Security
Water & Sanitation
Coordination
and Control
Logistics &
Transport
Social Structure
Public Works
& Engineering
Shelter & Clothing
Food & Nutrition
Education
Energy Supply
Management
Global Health Cluster Meeting
Geneva - December 2011
5 Frameworks
Epidemiological
1. Conceptual
2. Longitudinal
3. Transectional
Interventional
4. Operational
5. Preparedness
Global Health Cluster Meeting
Geneva - December 2011
4. Operational Framework
•
•
•
•
•
Evaluation of Responses
Outcome(s)
Impact(s)
Other effects
Process (including Performance Evaluations)
–
–
–
–
Costs
Efficiency
Critical Points of Success
Critical Points of Failure
Global Health Cluster Meeting
Geneva - December 2011
Logic Model for
Evaluations of Responses
Goal
Impacts
(-term)
Objective
Short
Current
Status
Intervention
Inputs
(Costs)
Global Health Cluster Meeting
Geneva - December 2011
O
u
t
p
u
t
s
Effect1
Effect2
Effectn
Outcome
Medium
Long
Impact of Intervention
Goal: Restore fishing industry
Objective: Build replacement boats
Need: Boats to
restore destroyed
livelihoods
Build Boats
Outcome
Impact
700 Boats
Boats not
Delivered
Useable!!
Money Supplies Equipment Personnel
Global Health Cluster Meeting
Geneva - December 2011
Impact of Intervention
Goal: Restore fishing industry
Objective: Build replacement boats
Need: Boats to
restore destroyed
livelihoods
Build Boats
Outcome
Impact
700 Boats
Boats not
Delivered
Useable!!
Money Supplies Equipment Personnel
Global Health Cluster Meeting
Geneva - December 2011
Achievement Indicators
“Accountability”
Global Health Cluster Meeting
Geneva - December 2011
Impact vs Process
• Impact = So What?
• Process
–Effectiveness
–Costs
–Efficiency
–Critical Points of Failure
–Critical Points of Success
Global Health Cluster Meeting
Geneva - December 2011
Where To Go
•
•
•
•
•
•
Structure all reports
Use logic model in design of responses
Educate donors
Standardize terminology
Force past into structure
Identify indicators of function
Global Health Cluster Meeting
Geneva - December 2011
Preparedness Interventions
• ?? Impact(s)
• ?? Funding
• Requires
– Standards
– Milestones
– Benchmarks
– Data
– Comparisons—what worked elsewhere?
Global Health Cluster Meeting
Geneva - December 2011
Where Are We?
• Workshops
• Validation of Frameworks
– Structure available information
– Use for future reports/papers
• Build consensus
• Educate/Train evaluators
• Build evidence/science
–
–
–
–
–
Standards
Best practices
Competencies
Curricula
Credentialing/accreditation
Global Health Cluster Meeting
Geneva - December 2011
Global Health Cluster Meeting
Geneva - December 2011
Hazard
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Risk Emergency
Risk Disaster
Local Response
Capacity
Needs
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Emergency
Outside Response Capacity
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Local Response
Capacity
Needs
RiskDisaster
Global Health Cluster Meeting
Geneva - December 2011
Local Responses
Resilience
Risk Reduction
Preparedness
Hazard
Emergency
Outside Responses
Disaster
Outside Response Capacity
Hazard
RiskEvent
Hazard Modification
Event
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Local Response
Capacity
Needs
RiskDisaster
Local Responses
Emergency
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Resilience
RiskDamage
Outside Response Capacity
RiskEvent
Hazard Modification
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Risk Emergency
RiskDisaster
Local Response
Capacity
Needs
Local Responses (Emergency)
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Resilience
Event
RiskEvent
Hazard Modification
Event
RiskDamage
RiskΔFunction
Absorbing Capacity
Damage
Buffering Capacity
Changes in Function
Risk Emergency Needs
RiskDisaster
Local Response
Capacity
Local Responses
Outside Responses
Disaster
Global Health Cluster Meeting
Geneva - December 2011
Resilience
Risk Reduction
Preparednes
Hazard
Emergency