3ie-LIDC Seminar Series 'What Works In International Development’
London School of Hygiene and Tropical Medicine
20th February 2013
Getting Evidence Into Policy
Philip Davies
International Initiative for Impact Evaluation [3ie]
Philip Davies
www.3ieimpact.org
Outline of Presentation
• How evidence is used in policy
• Factors other than evidence
• Some features of evidence
• Different notions of evidence
• Barriers to getting evidence into policy
• Overcoming barriers to getting evidence into policy
Philip Davies
www.3ieimpact.org
How Evidence is Used in Policy Making
Instrumental Use
Involves acting on research results in specific, direct ways.
Conceptual Use
Involves using research results for general enlightenment;
results influence actions, but in less specific, more indirect
ways than in instrumental use
Symbolic Use
Involves using research results to legitimate and sustain
pre-determined positions.
Source: Carol Weiss, 1982
Philip Davies
www.3ieimpact.org
How Evidence is Used in Policy Making
“Rarely does research supply an “answer” that policy
actors employ to solve a policy problem. Rather,
research provides a background of data, empirical
generalisations, and ideas that affect the way that policy
makers think about a problem.”
“But to acknowledge this is not the same as saying that
research findings have little influence on policy.”
Source: Carol Weiss, 1982
Philip Davies
www.3ieimpact.org
Factors Other Than Evidence
Values and
Decision Making
Pragmatics &
Context
Experience
Contingencies
&
Expertise
Lobbyists &
Pressure Groups
Bureaucratic
Culture
Philip Davies
Evidence
Judgement
Resources
www.3ieimpact.org
Some Features of Evidence
• Evidence is almost always probabilistic
• Evidence is often context specific
• Often disagreement agreement on what counts as ‘evidence’
• Evidence is always contestable/contested
• Evidence is rarely self-evident
• Not all research is of equal value/sufficient quality
• Single studies can misrepresent the balance of evidence
• Hence, the need for systematic reviews/synthesis of evidence
Philip Davies
www.3ieimpact.org
Different Notions of Evidence
Policy Makers’
Evidence
Knowledge
Translation and
Transfer
Researchers’
Evidence
• Colloquial (Narrative)
• ‘Scientific’ (Generalisable)
• Anything that seems reasonable
• Proven empirically
• Policy relevant
• Theoretically driven
• Timely
• As long as it takes
• Clear Message
• Caveats and qualifications
Source: J. Lomas et al, 2005
Philip Davies
www.3ieimpact.org
UK Policymakers’ Views of Evidence*
• Focus on the ‘end product’, rather than how the
information was either collected or analysed
• Use of ‘anecdotal’ evidence
• Drawing on such things as ‘real life stories’, ‘fingers
in the wind’, ‘local’ and ‘bottom-up’ evidence
But:
• “If we try and move anywhere without having the
scientific basis to do so we get fleeced in the House”
*Source. Campbell, S. et al, 2007: Analysis for Policy, London, GSR.
Philip Davies
www.3ieimpact.org
Where Do UK Civil Servants Go For Evidence?
Sharks
Academic/Evaluation Research?
Philip Davies
www.3ieimpact.org
Plankton
Barriers to the Use of Evidence
• Policymakers’ lack of familiarity with the research process
• Researchers’ lack of familiarity with the policy process
• Trust (lack of trust) of policymakers in researchers (vice
versa)
• Physical access to evidence
• Cognitive access to evidence (i.e. lack of understanding)
• Lack of clarity in the presentation of evidence
• Timeliness and availability of evidence
Sources: Lomas, 2000; Petticrew et al, 2004; Lavis et al, 2005; Dobbins et al, 2007;
Ouimet, et al, 2009; Rosenbaum, 2010
Philip Davies
www.3ieimpact.org
Overcoming Barriers to the Use of Evidence
• Early and ongoing involvement of relevant decision makers
• Interactions between researchers and policymakers increases the
prospects for research use by policymakers
• Identify and use interpersonal networks and face-to-face
interactions
• Identify willing and able knowledge brokers
• Separate strategic from operational demands for evidence
• Get policy makers to own the evidence – not just the policy
Sources: Lomas, 2000; Petticrew et al, 2004; Lavis et al, 2005; Dobbins et al, 2007;
Ouimet, et al, 2009; Rosenbaum, 2010
Philip Davies
www.3ieimpact.org
Improving Communication of Evidence
• Establish what the evidence says, and does not say
• Establish the policy messages and policy implications
• But avoid ‘recommendations’
• Provide information about the costs and benefits, harms and
risks of interventions/policies
• Present contextual factors that affect local applicability
• Be clear - plain English summary, unclouded by jargon
• Find the ‘influencing moment’
• Aligned to decision making timescales
• Use a 1:3:25 format
Philip Davies
www.3ieimpact.org
The ‘One’ in the 1:3:25 Format
•
•
•
•
•
A one page of main message bullets
The lessons decision makers can take from the research
Not a summary of findings
Suggest implications of findings
No details of methodology
Philip Davies
www.3ieimpact.org
The ‘Three’ in the 1:3:25 Format
• These are the key findings of the study
• The classic Executive Summary
• Condensed to serve the needs of the busy decision maker
• Focus on how the study may be useful for policy
• Some brief mention of methodology
• Some implications of policy and practice
Philip Davies
www.3ieimpact.org
The ‘25’ in the 1:3:25 Format
This should include:
• Context/Background
• Approach (Methodology in Appendices, not text)
• Results
• Implications
• Knowledge gaps
• References
• Additional resources
• Appendices
Philip Davies
www.3ieimpact.org
Summary
• The routes of evidence to policy are usually indirect
and delayed
•
•
•
•
•
•
•
There are factors other than evidence
Evidence is seldom definitive or invariant
Nor is it self-evident
There are different notions of evidence
Physical and cognitive access to evidence is important
Evidence has to be ‘worked’ into policy/practice
Clear communication evidence is essential
Philip Davies
www.3ieimpact.org
References
Campbell, S., et al, 2007
Analysis for Policy, London, GSR
Dobbins et al, 2007
Public Health Decision-Makers’ Informational Needs and Preferences for Receiving
Research Evidence, Worldviews on Evidence-Based Nursing, 3rd Quarter, 156-163,
New York, Wiley Inc.
Lavis et al, 2005
Towards systematic reviews that inform health care management and policy-making,
Journal of Health Services Research and Policy, 10, Suppl 1 July 2005 S1:35.
Lomas, J., 2000
Connecting Research and Policy, Canadian Journal of Policy Research, Spring, 140144
Lomas, J. Culyer, T., McCutcheon, C., McAuley, L., and Law, S., 2005
Conceptualizing and Combining Evidence for Health System Guidance, Final Report,
Canadian Health Services Research Foundation, Ottawa.
Philip Davies
www.3ieimpact.org
References (continued)
Ouimet, M., Landry, R., Ziam, S., and Bédard, P., 2009
The absorption of research knowledge by public servants, Evidence and
Policy, 5, 4, 331-350.
Petticrew, M., et al, 2004
Evidence for public health policy on inequalities: 1: The reality according
to policymakers, Journal of Epidemiology and Community Health
2004;58:811-816
Rosenbaum, S.E., et al, 2010
Evidence summaries tailored to health policy-makers in low- and middleincome countries, Bulletin of the World Health Organization, 89, 1, 54-61.
Weiss, C.A., 1982
Policy research in the context of diffuse decision making, Journal of
Higher Education, 53, 6, 619-639.
Philip Davies
www.3ieimpact.org
Thank you
Philip Davies
Email: [email protected]
+44 (0)207 958 8350
Visit www.3ieimpact.org
Philip Davies
www.3ieimpact.org