MEASURING IMPACT: USING QUANTITATIVE
RESEARCH
v
Eibhlín Ní Ógáin
London, November 2012
AGENDA
1 What is impact
measurement
2 NPC’s approach
3 Quantitative
approaches
4 Tips and things
to remember
TRANSFORMING THE CHARITY SECTOR
Increasing the
impact of charities
Charity
Increasing the
impact of funders
eg, effective
commissioning
Funder
We are a think
tank and
consultancy
working at the
nexus between
charities and
funders
eg, outcomes
frameworks
Strengthening the
sector
eg, collaboration
towards shared
goals
Sector
WHAT IS IMPACT
MEASUREMENT?
Impact: The difference an organisation or activity makes to the people or
problem it aims to help
Impact measurement: Planning, managing, tracking and reviewing how
much difference you are making
5
NPC’S APPROACH TO
MEASURING SOCIAL
IMPACT
NPC’S APPROACH TO MEASURING SOCIAL
IMPACT
Strategic
vision /
goals
Existing evidence
A welldeveloped
Theory of
Change
Appropriate measurement
tools
7
QUANTITATIVE RESEARCH
8
WHY USE QUANTITATIVE RESEARCH
• Qualitative: understand something in detail. Rich information and
insight.
• Quantitative: how many people? How large an effect? How much
impact and how robust?
• Use quantitative research when dealing with large numbers of people
and when you want to show a consistent effect.
• E.g does new type of psychotherapy lead to improved mental wellbeing? Need consistent measure and need large sample.
• Depends on type of ‘intervention’ and questions you are trying to
answer.
• Know a lot, measure a little, know a little, measure a lot.
9
DIFFERENT METHODS IN QUANTITATIVE
RESEARCH
• Ranges from easy to measure to hard to measure
• Easier: Hard and objective outcomes: move into employment, gain a
qualification, reduction in reoffending.
• Harder: Softer, difficult to observe outcomes: changes in feelings,
personal growth, well-being.
• NPC’s preference is for previously tested scales and questionnaires.
• If there is a gap, develop own questions.
10
TYPES OF QUANTITATIVE TOOLS AND THEIR
USES
Type of tool
Identify
needs of
group
Admin. data

?
?
?




?

Clinical tools





Scales
(developed by
researchers)





Single
questions





Hard
outcomes (eg,
statutory data)





Case work tool
(eg, Recovery
Star)
Measures
progress for
individuals
Aggregate
to show
change
Robust
measure
of change
Explain why
change has
happened
11
WHEN TO MEASURE
• Pre-post design: how outcomes change over time by collecting data
before (pre) and after (post) the programme.
Before
measure
Programme / service
After
measure
• When you think change happens influences your decision about:
– When to collect the before measure
– When to collect the after measure
12
THINK ABOUT ATTRIBUTION
• Is the programme the only factor influencing any changes? What
would have happened anyway?
• Control groups
• Compare before and after data for participants and non-participants,
ideally matching on their key characteristics
• Holy grail: Randomised Control Trial, where potential participants
randomly assigned to control group
13
13
THINK ABOUT SAMPLING
• Representativeness
– Does your sample represent your population?
• Response rates
– Is your response rate high enough to avoid bias?
• Statistical validity
– Are your sample sizes large enough for analysis?
• Oversampling (segmentation)
Biased/small sample => Avoid percentages
14
YOU DON’T NEED TO MEASURE EVERYTHING
15
WHAT: BEFORE YOU THINK ABOUT
COLLECTING DATA ASK YOURSELF:
Do you
already
collect this
data?
Yes
Use this
data (if it is
fit for
purpose)
No
Has anyone
already
proved the
causal link
you want to
make?
Yes
Is it really
important
No you need
data on
this
outcome?
Yes
Does anyone
else collect
this data?
Yes
Can you
access this
data?
No
No
Use research to
limit your data
collection
Yes
Don’t collect
unless you
decide it is
really
important
No
Develop
your own
data
source
16
PRIORITISE OUTCOMES THAT:
• you directly influence (rather than indirectly support)
• are important / material to your mission
• are not too costly to measure
• will produce credible data?
17
HOW ROBUST DOES EVIDENCE OF YOUR
SOCIAL IMPACT NEED TO BE?
• What do your target
stakeholders (investors?) think?
• What is possible, given your
resources etc?
18
LEVELS OF EVIDENCE
Credibility
Anecdotes /
quotes
Case Self-reported
studies
change
Basic
Nesta level 1
Before and
after survey
Control
groups
Randomised
control trial
Advanced
Nesta level 2
Nesta levels 3, 4, 5
19
6 TIPS FOR MEASURING YOUR IMPACT
1.
2.
Decide what’s important. Define your research question and be
clear what you want to achieve.
Keep it simple. Concentrate on what you really want to know and
don’t try to be too ambitious in the scope of your evaluation.
3.
Know your limits. You need to choose an approach that
matches your resources.
4.
Consider investing in training. If you are not confident, get
training or seek external help.
5.
Think about the future. Don’t build a system that will quickly go
out of date and look at the year-on-year costs.
6.
Use the information that you collect: If you don’t use it, don’t
measure.
20
RESOURCES
•
•
•
•
•
•
•
•
•
Review of evidence of ICT and learning:
http://educationendowmentfoundation.org.uk/toolkit/ict
A good website to look for measures and indicators:
http://wilderdom.com/tools/ToolsSummaries.html
How to communicate your results: Hedley, S et al (2010) Talking about results. New
Philanthropy Capital.
Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design.
Jossey-Bass: San-Francisco.
http://www.theoryofchange.org/
The innovation network’s logic model workbook:
http://www.innonet.org/client_docs/File/logic_model_workbook.pdf
The Centre for What works (http://www.whatworks.org/) has an outcome portal where
you can browse different outcomes and their corresponding indicators.
TRASI (http://trasi.foundationcenter.org/) has a database where you can search
different approaches to impact assessment.
NPC has a page on shared measurement where it outlines some indicators for different
sectors.
http://www.philanthropycapital.org/publications/improving_the_sector/measuring_togeth
er.aspx
21
RESOURCES
•
•
•
•
•
•
•
•
Survey question bank: A website where you can research widely used surveys and
single questions by theme or using key word searches. http://surveynet.ac.uk/sqb/
CES’s guide to theory of change: http://www.cesvol.org.uk/downloads/makingconnectionsusingatheoryofchangetodevelopplan-800808.pdf
Forum for Youth Investment’s guide to measuring soft skills in youth programmes:
http://www.forumfyi.org/files/Soft_Skills_Hard_Data.pdf
A list of the most common psychological scales and questionnaires:
http://www.ull.ac.uk/subjects/psychology/psycscales.shtml
Harvard Family Research Project’s tools for out-of-school time programmes:
http://www.hfrp.org/out-of-school-time/publications-resources/measurement-tools-forevaluating-out-of-school-time-programs-an-evaluation-resource2
Bradburn et al. (2004) Asking questions: the definitive guide to questionnaire design.
Jossey-Bass: San-Francisco.
Ritchie et al. (2003) Qualitative Research Practice – A Guide for Social Science
Students and Researchers. Sage Publications Ltd
Measurement and assessment tools for family and parenting interventions:
http://www.actionforchildren.org.uk/media/3986553/outcome_tools_1.pdf
22
v
Eibhlín Ní Ógá[email protected]