Designing a measurement
framework
For Elfrida Rathbone Camden (ERC), a charity supporting young disabled
people, carers and families in need.
Rose Drummond and Will Tong (with quality assurance from Jane Parkin)
Overview
• Introduction to the charity and purpose of project
• PHASE 1 – background research
• PHASE 2 – engaging with the charity
• PHASE 3 – finalising the detail and write-up
• Reflections on project
Overview
• Introduction to the charity and purpose of project
• PHASE 1 – background research
• PHASE 2 – engaging with the charity
• PHASE 3 – finalising the detail and write-up
• Reflections on project
Introduction- About the charity
• Elfrida Rathbone Camden (ERC) supports young disabled people, families in
need, parents of children with disabilities and young carers in Camden
• Small charity involved in diverse range of projects:
• Advocacy – for families of young carers
• Life skills courses - for young disabled people to bridge gap from adolescence to
adulthood
• Support and advice – for families with a range of problems. Debt and welfare advice,
family relationships, parenting, children’s emotional problems, education.
• Diverse range of funders for different projects
• Each project had its own monitoring and evaluation framework
Introduction- Project aim
• Exec wanted to measure the impact of the organisation as a whole
• Project aim: Develop measurement framework at organisational level
• This would bring together the diverse range of projects initially with:
• Different target populations with different needs
• Different funders
• Different monitoring and evaluation frameworks
Introduction - Project set-up
• The team:
• Alexis (Director of ERC)
• Will and Rose (pro-bono OR analysts)
• Jane (pro-bono quality assurance analyst)
• Set out scope and objectives in project initiation template
• Template provided by OR society
Reflection:
1. Get the right team together
• Director of charity: Meant we had buy-in from the top of the
organization. Alexis also in touch with the work of the projects, and
could tell us what was feasible across the organization.
• Experienced quality assurance analyst: Fantastic to tap into the
experience of someone who has run many such projects before.
Reflection:
2. Agree realistic objectives and scope up front
• We split the project in half - excluded funding applications.
• Needs to be realistic given that the work is being fitted into
volunteer’s spare time.
Overview
• Introduction to the charity and purpose of project
• PHASE 1 – background research
• PHASE 2 – engaging with the charity
• PHASE 3 – finalising the detail and write-up
• Reflections on project
PHASE 1 – Background research
1. Understanding the charity
• Charity Director explained about the charity and what they do
• Access to charity’s current monitoring and evaluation documents,
funding proposals and project records.
• Mapped out the indicators used on different projects – very different!
• Charity website and mission statement
Reflection:
3. To understand the charity, if you can, visit in person.
• For us this was not practical, so we used alternative sources.
PHASE 1 – Background research
2. Researching frameworks used in the charity sector
Lots of online resources!
Structure:
• Outcomes star (already used by ERC)
• ASCOT (same as outcomes star, but outcomes weighted)
Content:
• Outcome indicator banks - the evaluation cookbook for getting feedback
from children in the form of a game.
PHASE 1 – Background research
From our research:
• We realised it was unclear what outcomes to measure- the projects
were too diverse to simply combine existing outcome frameworks.
• We chose Charities Evaluation Service template to structure the
framework – flexible step-by-step guide to defining outcomes,
indicators, collection methods and frequency.
PHASE 1 – Background research
Monitoring and Evaluation Framework Template
Source: Charities Evaluation Service
PHASE 1 – Background research
Source: Charities Evaluation Service
Overview
• Introduction to the charity and purpose of project
• PHASE 1 – background research
• PHASE 2 – engaging with the charity
• PHASE 3 – finalising the detail and write-up
• Reflections on project
PHASE 2 – engaging with the charity
• We agreed that we needed to go back to the charity and work with
them to agree which were the important outcomes to measure.
• Wrote plan and timetable.
Reflection:
4. Beware of scope creep!
• Full of ideas after research stage – danger of scope becoming too
ambitious!
PHASE 2 – engaging with the charity
Data collection – key outcomes
• Initially just going to ask executive committee in a meeting.
Jane (QA) had 2 suggestions:
• Collect views of all stakeholders (also staff, volunteers, trustees,
service users, funders) in a workshop.
• First ask questionnaire - to inform workshop planning and get people
thinking.
PHASE 2 – engaging with the charity
Questionnaire
• Four variations of questionnaire, one for each group of respondents
•
•
•
•
ERC charity exec/trustees
Funders
Service users
Staff/volunteers
• Service users one went on the Facebook page, and was short and
simple. Director emailed funders who he had personal relationship
with
• Went to over 30 people – 4 responses (including 1 very detailed
response)
Reflection:
5. Get charity input – they know your target audience
• Minimize burden on respondents
• Right approach for each target group (Facebook, email, phone call)
• Director was sure funders would be too busy to answer questionnaire
– he was right!
PHASE 2 – engaging with the charity
Workshop
• In a facilitated 3 hour workshop agreed key outcomes among 12
people representing:
• ERC charity exec/trustees
• Service users
• Staff/volunteers
• A key success of the project – unique opportunity to hear from all
these groups and get buy-in, agreement and joint ownership.
PHASE 2 – engaging with the charity
"If ERC could only achieve one thing for its service users,
what should it be?"
Agreed main outcomes:
• Improved mental health
• Increased opportunity to access employment, volunteering, training
• Increased access to support and social networks
• Reduced risks
• Cultural shift - the community benefitting from the contribution of
disabled people
Reflection:
6. Running a workshop.
• Get the right people involved – personal approach from charity
• Questionnaire – got people thinking about key themes in advance,
helped plan structure of workshop, gave us something to fall back on
if room was silent.
• Plan, plan, plan –templates, needs of participants, extra activities
• Ice breaker – to get people talking
• Work in pairs – one facilitates, the other writes notes.
• Assigning discussion groups – ideal if someone who knows everybody
assigns people to suitable groups
• Group tasks should have clear objective and time limit – to keep
discussion focused.
• Be flexible.
Overview
• Introduction to the charity and purpose of project
• PHASE 1 – background research
• PHASE 2 – engaging with the charity
• PHASE 3 – finalising the detail and write-up
• Reflections on project
PHASE 3 – Finalising detail and write-up
• Analysts filling in the gaps in different colours so that you could see
the source:
•
•
•
•
Workshop
Questionnaire
ERC current measurements
Rose/Will’s input- using tested questions where possible (such as ONS wellbeing questions)
• Meeting with Director and 2 staff members. To remove overlap from
outcomes, and pick out most important details to measure.
• QA – incorporate comments from Jane (ie measuring numbers or %
increase, measuring too many things with one indicator)
Overview
• Introduction to the charity and purpose of project
• PHASE 1 – background research
• PHASE 2 – engaging with the charity
• PHASE 3 – finalising the detail and write-up
• Reflections on project
Reflections - Working with young people
• Workshop –avoid technical language and explain the language being
used (as you should with any non-technical audience)
• Final framework – if asking users to assess their own progress against
an indicator, questions must be easy to understand and answer.
Reflections on working pro-bono
• Great way to learn, meet people and make a positive impact!
• Choose a project/topic that you have a real interest in
• Be realistic about the time you are able to give up, and be prepared
for long gaps if people go on holiday etc.
• Consider whether distance will be a hindrance
• Having an experienced analyst for QA was fantastic
Any comments or questions?
Contact details for Rose Drummond:
[email protected]
PHASE 3 – Finalising detail and write-up
Outcome
Improved
Mental
Health
Sub-details
Indicators (measurable)
Service users have reduced feelings Number and percentage of
of anxiety and depression and
service users who report
improved sense of well-being.
reduced feelings of anxiety or
depression having used an
ERC service.
How to collect data
Self reported:
Questionnaire/
structured interview
before and after using
an ERC service.
Sample question (where appropriate)
On a scale of 1-10 where 10 is highest, how
happy were you feeling this time yesterday?
On a scale of 1-10, where 10 is very anxious, how
anxious were you feeling this time yesterday?
Source: Office for National Statistics Wellbeing
questions
Less isolation/loneliness
Increased
access to
support and
social
networks
Percentage of service users
whose loneliness score is
reduced by at least 2 points
having worked with ERC.
Questionnaire/intervie
w before and after
working with ERC
service
1. How often do you feel that you lack
companionship: Hardly ever, some of the time, or
often?
2. How often do you feel left out: Hardly ever,
some of the time, or often?
3. How often do you feel isolated from others?
(Hardly ever, some of the time, or often?)
For each question, hardly ever has score of 1,
some of the time has a score of 2, often has a
score of 3. Sum the answers for overall score out
of 9.
Source: 3-item UCLA loneliness scale
Collecting data:
when and who
Reporting data:
when and who