Breaking the Cycle
Embedding financial capability within debt advice at Plymouth
Citizens Advice Bureau
Acknowledgements
This research was funded by the Royal Bank of Scotland Group and written by Helen
Aynsley, Evaluation and Policy Manager on RBS Innovate at Toynbee Hall.
The author would like to thank Jane Guy, Training Manager at Plymouth Citizens
Advice Bureau, for coordinating and providing the data for this report and for her
on-going support and collaboration. Thanks also go to Sharon and Bill at the bureau
for providing additional information and allowing the evaluator to sit in on their
sessions.
2
Contents
Executive Summary…………………………………………………………………………..
4
1
Introduction to RBS Innovate………………………………………………………...
8
2
The Debt Gateway and Money Active at Plymouth CAB……………………..
10
Plymouth Citizens Advice Bureau…………………………………………………..
10
The Debt Gateway and Money Active…………………………………………...
11
Financial capability and debt advice………………….…………………………
12
Aims………………………………………………………………………………………
14
The Model……………………………………………………………………………….
14
Assessment……………………………………………………………........................
15
Money Active workshops……………………………………………………………..
15
Follow up…………………………………………………………………………………
17
Diagram: The Debt Gateway and Money Active ……………………………….
18
Evaluation……………………………………………………………………………….
19
The trouble with evaluation………………………………………………………….
19
Evaluation Stage 1: Set up of evaluation…………………………………………
21
Evaluation Stage 2: Methodology………………………………………………….
23
Debt Gateway and Money Active evaluation framework……………………
28
Findings……………………………………………………………………………………
29
Evaluation of the project…………..………………………………………………….
29
Evaluation of the model……………………………………………………………….
38
Evaluation of the evaluation………………………………………………………….
40
6
Conclusions and recommendations……………………………………………....
41
7
Appendix A……………………………………………………………………………….
43
8
References………………………………………………………………………………..
44
3
4
5
3
Executive Summary
1. Introduction to RBS Innovate
1.1
From 2005 to 2009 RBS provided funding for small projects to test out new
approaches to financial inclusion. This programme, called RBS Innovate and
run through Toynbee Hall, helped 13 organisations get new financial inclusion
initiatives off the ground. After three successful years the decision was taken
that, in order for projects to provide maximum learning, it was important to
focus on lessons learned and the long-term impact of interventions. The focus
therefore shifted to the evaluation of projects with emphasis on three aims:
1. Help projects to set up their own internal evaluation structures;
2. Share the resulting evaluation frameworks with the rest of the financial
inclusion sector and apply to other organisations where appropriate;
3. Showcase the work of the projects and offer models of replication.
1.2
RBS Innovate’s role was therefore restructured to work with four organisations,
each based in different areas of the UK. The decision of which four to choose
was based not only on the projects themselves but also on the combination of
the four projects – it was important to ensure a good mix of size, remit,
population served and geographical location. The four selected projects are:
1. Money Wise at Centrepoint (London)
2. Debt Gateway and Money Active at Plymouth CAB (Plymouth)
3. Save and Insure at Prince Bishops Credit Union (Consett)
4. Your Money Garden (Belfast)
2.
Plymouth Citizens Advice Bureau
2.1
As well as identifying what specialist advice debt clients need, the Debt
Gateway at Plymouth CAB identifies which clients could benefit from
additional financial capability training. Those that could benefit are referred
to a Money Active session – either a workshop or a one to one. The aim of
Money Active is to increase service users’ knowledge of budgeting,
borrowing, savings, debt and where to go for additional support and help.
There are three key aims to the Debt Gateway/Money Active model:
2.2
Increase financial capability: Plymouth CAB believes it is important to increase
the knowledge, skills and confidence of service users to manage their money
more effectively and take responsibility for personal financial decisions.
2.3
Decrease repeat appointments within the bureau: Debt service users often
come back to the bureau multiple times, sometimes even for the same issue. It
is hoped that by referring a number of service users to financial capability
training, it may be possible to decrease the amount of clients who come back
in for more advice, especially for the same issue.
4
2.4
Explore implementation: Consider different procedures to identify the best
way for Plymouth CAB to embed financial capability within debt advice
without taking away from debt service delivery.
3.
The model
3.1
There are three steps to the Debt Gateway/Money Active model:
3.2
Assessment and referral: All debt clients that come to Plymouth CAB are
assessed by the Debt Gateway. The advisor at the Debt Gateway assists
clients with queries that can be handled there and then and refers clients that
need specialist advice to the appropriate service. Clients that they feel could
benefit from money management training are referred to a Money Active
workshop or one to one.
3.3
Service delivery: Workshops are delivered in a group setting and cover four
key subject areas: budgeting, borrowing and saving, dealing with debts and
where to go for help. A4e sessions are delivered to clients facing barriers
getting into or returning to work and the bureau also holds training sessions for
advice staff who work in Plymouth but not in a CAB.
3.4
Follow up: Plymouth CAB do not usually follow up with clients who attend a
Money Active session but they did so as part of this evaluation. The results
were encouraging and the follow up not only provided clients with the
opportunity to reflect on their progress since attending a Money Active session
but provided useful feedback for the staff and volunteers who run the sessions.
4.
The evaluation
4.1
Key questions to ask when evaluating an internal project:
4.1.1
What are the key objectives of the project? Establish the purpose of the
project and its aims and objectives right at the start. In the case of RBS
Innovate, an initial meeting was held with Gill Tishler and Jane Guy at
Plymouth CAB in order to gain an understanding of the Debt Gateway and
Money Active and what they were seeking to achieve for participants
through Money Active. It is important to have this discussion at the start of the
evaluation as it lays the groundwork for subsequent evaluation stages.
4.1.2
How to strike the right balance between ‘perfect’ methodology and actual
capacity to deliver? In order to successfully embed evaluation activities within
an organisation, it is necessary to be realistic about what is achievable as
regards funding, staff, and available time. As Jane was the prime contact for
the duration of the evaluation we designed an evaluation framework that
would realistically fit in with her time and other commitments. This is an
important process for any organisation setting up an internal evaluation; you
could design the most robust evaluation framework that would guarantee a
wealth of rich information, but if it is not going to be realistic to gather that
information it will not be achievable.
5
4.1.3
What information is already being captured? Plymouth CAB already capture a
lot of data for their own use and to report back to funders. Rather than
starting from scratch it was useful to establish what information had to be
captured for other purposes (i.e. funder reports) and what information was
already available so that work was not replicated.
4.1.4
What kind of evaluation do we want to undertake? Do we want the evaluation
to be formative or summative? A formative evaluation asks “how can the
programme be improved?’ and is undertaken in order to provide feedback
with the goal of progress. Summative evaluations on the other hand ask “what
is the overall merit or worth of the programme? Should it be modified? Should
it be continued?” Gill and Jane decided that the evaluation at Plymouth CAB
should be both formative and summative.
4.1.5
What methods are we going to use? Sometimes quantitative data such as
numbers and figures will provide the most important information while other
times more descriptive, qualitative information will be most useful; sometimes it
will be both. Whilst an external evaluator can provide information about the
different kinds of data and their respective benefits, it is up to the organisation
to decide which method will give the richest data to suit their needs and help
them to inform their decisions going forward.
5.
Evaluation of Plymouth Citizens Advice Bureau
5.1
The most common reason a client was referred to a Money Active session
(36%) was so that they could learn to prevent and deal with their debts
followed closely by learn to budget their money and complete a financial
statement (30%).
5.2
When asked what they would change about the session 99% of respondents
said ‘nothing’.
5.3
100% of participants said that they felt their attitude towards money had
changed since attending the workshop, with 83% saying their attitude towards
budgeting had changed.
5.4
100% of respondents were able to give an example of an affordable source of
credit and 79% of respondents were able to give an example of a priority bill
weeks after attending a workshop.
6.
Evaluation of the model
6.1
Plymouth CAB found that if a client takes part in a Money Active session any
subsequent debt case work takes a third of the time it takes for clients who
have not been through a Money Active session.
6.2
Take up of workshops was low throughout the duration of the evaluation but
staff at Plymouth CAB were able to identify factors to mitigate this and have
adjusted the service accordingly.
6
7.
Evaluation of the evaluation
7.1
This model of evaluation provides a good balance between internal and
external; the evaluation framework was developed in partnership with the
external evaluator and Plymouth CAB and all major decisions were made
jointly. Balanced alongside this is the fact that the evaluator offers an outside
view and a different perspective.
7.2
Even though staff at Plymouth CAB followed up with Money Advice clients
after the session for evaluation purposes only, this proved to be such a useful
exercise for both service users and staff that it is recommended this be
continued going forward.
7.3
Literacy, mental health and confidence issues means that Plymouth CAB will
potentially want to explore other ways of evaluating the course in future that
are not reliant on paper-based tools. As some participants need help filling in
forms it means that the evaluation can take up a disproportionate amount of
time and this takes away from the time needed to actually deliver the service.
7
Part 1 – Introduction to RBS Innovate
From 2005 to 2009 RBS provided funding for small projects to test out new
approaches to financial inclusion. RBS called this programme RBS Innovate and
through it helped 13 organisations get new financial inclusion initiatives off the
ground. The overall aim of the RBS Innovate programme is to “increase financial
inclusion through the improvement of local financial inclusion services” and the
emphasis throughout the programme has been on innovative interventions,
meaningful work with long term impact, improvement and development and
dissemination of learning and best practice. The 13 projects that RBS Innovate
funded were very diverse – some helped individuals gain access to financial
products, some produced guides and resources for trainers, others supported
housing association tenants with specific needs – but all focused on helping people
experiencing financial exclusion.
As the programme progressed RBS decided that in order for financial inclusion
interventions to provide maximum learning for the sector, it was important to focus
on lessons learned and long-term impact. The focus of RBS Innovate therefore shifted
in the period 2009 – 2011 from funding interventions to the evaluation of projects with
the emphasis on three aims:
1. Help projects set up their own internal evaluation structures;
2. Share the resulting evaluation frameworks with the rest of the financial
inclusion sector and apply to other organisations where appropriate;
3. Showcase the work of the projects and offer models of replication.
Good projects are often under-evaluated so exploring how they can be scaled up
and how this can work across different geographies and with different clients groups
is key to promoting success in financial inclusion. RBS and Toynbee Hall therefore
restructured the role of RBS Innovate to work with four organisations based in
different areas of the UK over the course of one year. The four projects were selected
using an open tender process. In December 2009 RBS Innovate announced that
evaluation support was available for any organisation working in financial inclusion or
financial capability. This announcement was disseminated via Transact, the National
Forum for Financial Inclusion, the DWP Champions Initiative and the Citizens Advice
National Financial Capability Forums. The application form was posted on the
Transact web-site and the deadline was set for January 2010. In all, 33 applications
were received and 4 projects were selected. Using an open tender process
guaranteed that every organisation that applied was interested in setting up their
own internal evaluation frameworks, having their organisation evaluated and sharing
learning with the sector. Choosing only four organisations to work with was difficult as
there were many interesting, innovative organisations interested in RBS Innovate and
in the end the decision was based not only on the projects themselves but on the
combination of the four projects; it was important to ensure a good mix of size, remit,
population served and geographical location. The four organisations selected are
outlined below.
‘Moneywise’ at Centrepoint (London)
Centrepoint’s vision is to end youth homelessness and Moneywise fits into this vision
by providing training on financial capability to educate Centrepoint young people
8
about personal financial management. While advice services regarding financial
capability are available, young people find them difficult to access due to travel
costs, intimidation and their adult focus. Embedding financial capability into wider
work with homeless young people and young people at risk of homelessness helps to
create socially and economically viable citizens.
The objectives of Moneywise are achieved through AQA accredited workshops, one
to ones with young people, capacity building through staff training and signposting
to specific agencies where necessary. This holistic programme ensures that
Centrepoint residents have access to high quality advice and leave Centrepoint
services fully informed of the importance of financial capability.
Money Active at Plymouth Citizens Advice Bureau (CAB) (Plymouth)
At Plymouth CAB at least 70% of the issues staff deal with are related to financial
inclusion and their Debt Rights and Remedies Unit provides specialist money and
benefits advice to help tackle these issues. Through providing these services the
bureau has become aware that, unless they find a way of equipping their service
users with the skills and confidence to manage their financial affairs more effectively,
they are likely to return – possibly within a fairly short timescale – with the same issues.
To eradicate this, Plymouth CAB decided to refer a number of debt service users to
management sessions delivered within the Money Active programme. Some service
users are referred to a one to one session and some to a group session, both of which
help service users deal with issues such as budgeting, borrowing and saving, debt
and where to get help.
Save and Insure at Prince Bishops Community Bank (Consett)
Many people fall into financial difficulties when an unforeseen event such as a burst
pipe or burglary occurs and they have no insurance or savings to replace lost items.
As a result they often resort to doorstep lenders. Problems of loan sharks were
recently highlighted when a resident paid £88,000 interest on a £500 loan.
Through the Save and Insure scheme, tenants of Derwentside Homes in County
Durham have been encouraged to open a credit union account. If they save at
least £10 a month for six months they will receive a £20 credit to their account from
the scheme. In addition new members receive a voucher worth £40 towards
Derwentside Homes/RSA low cost contents insurance which covers a six month
premium, based on a £9000 sum insured. After saving for twelve weeks individuals
also have access to low cost loans from the credit union.
Your Money Garden (Belfast)
Your Money Garden is a project that provides financial capability education training
primarily to women across Belfast, Newtownabbey and surrounding areas. The
training is delivered across a series of workshops and aims to address gaps in
knowledge around finances. The training helps participants to develop their money
management, budgeting and practical skills so that they may increase their level of
financial knowledge and economic activity.
9
Financial learning sessions are delivered to existing groups (community women’s
centres, etc.) as an effective way of reaching people in need. In order to ensure that
participants have all the support they need, Your Money Garden works with partner
organisations such as local CAB and credit unions.
For the past year RBS Innovate has been working with the four projects to evaluate
the impact of their work, provide evaluation frameworks for the sector and use
lessons learned to make recommendations on how to replicate these models in
different areas of the UK.
Part 2 – The Debt Gateway and Money Active at Plymouth CAB
As well as identifying what specialist advice debt clients need, the Debt Gateway at
Plymouth CAB identifies which clients could benefit from additional financial
capability training. Those that could benefit are referred to a Money Active session –
either a workshop or a one to one. The aim of Money Active is to increase service
users’ knowledge of budgeting, borrowing, savings, debt and where to go for
additional help and support. It is hoped that providing this additional support will
increase the financial capability of service users and therefore reduce future debt
problems.
2.1 Plymouth Citizens Advice Bureau
Plymouth Citizens Advice Bureau (CAB) is an independent registered charity that
offers free, impartial, independent and confidential advice to their service users.
They are part of the national network of Citizens Advice Bureaux that can be found
across the UK – see www.citizensadvice.org.uk for your local bureau. Plymouth CAB
operates across four key principles:
1. Free services: All of the services of Plymouth CAB are free because they believe it
is everyone’s right to have information about laws and services that affect them.
They also believe they everyone should be able to make use of these laws where
necessary and have access to the services that they are entitled to.
2. Confidentiality: All information given to Plymouth CAB is kept strictly confidential.
3. Independence: The service provided by Plymouth CAB is independent. Because of
this, they are able to offer impartial advice to all enquirers and to take up any issue
with the appropriate authority on behalf of individuals or groups.
4. Impartiality: All services offered through Plymouth CAB are impartial and are
available to everybody. They believe it is important to value diversity, promote
equality and challenge discrimination.
Plymouth CAB gives information and advice to service users on a number of different
subjects. Advisors are trained to deal with queries relating to housing, consumer
issues, debt, welfare benefits, employment, family and personal issues, nationality
and immigration. They help people in the community in a number of different ways;
specific issues can be dealt with through face-to-face advice, on the telephone and
by e-mail and additional information is also available on-line at
www.adviceguide.org.uk. Covering such a wide range of issues means that
10
Plymouth CAB is able to take a rounded view of the problems people face. This is
important as they find that one significant change in circumstance can trigger a
whole raft of problems: for example, losing a job can lead to the loss of a home,
breakdown of a relationship and problems with debt. Falling ill can result in
complicated benefit applications and anxiety about employment and bills. As a
result, Plymouth CAB feel it is important to provide a holistic service to service users so
that all of their needs are met wherever possible. Helping service users with all of their
issues can also help tackle the root cause, thus preventing problems occurring in the
future.
As well as providing services under their roof, Plymouth CAB also provide the
secretariat for the Advice for All group (A4A) which was launched in September
2006. A4A coordinates advice services across Plymouth and focuses on improving
the quality and consistency of advice and also increasing the capacity to give
advice. As part of this they work on referral mechanisms and signposting and how to
work well in partnership. They also produce local fact sheets covering a wide range
of topics including debt, mental health, benefits, job seeking, employment, drug and
alcohol misuse, housing and energy efficiency. The full set of fact sheets can be
found at www.plymouthcab.org.uk and for further information on Advice 4 All visit
www.advice4all.org.uk.
Plymouth CAB has a good working relationship with the funder of its core service,
Plymouth City Council, who has a city-wide financial inclusion strategy which
supports the work of the CAB. The CAB has also developed an exceptional
relationship with the local water company, South West Water, with an independent
water gateway for their customers in debt across Devon & Cornwall which helps and
advises customers where savings can be made andcan offer grants to those in
hardship to clear their water debts.
2.2 The Debt Gateway and Money Active
Of the 20,000 issues dealt with by Plymouth CAB in 2008/2009, at least 80% were
related to financial inclusioni and they handled over £50 million worth of debt on
behalf of their service users. Over the course of the past 5 years they have built up
their Debt Rights & Remedies Unit which currently employs 12 staff. In order to assess
which service users are in greatest need and ensure that they are seen as quickly as
possible, service users who attend the drop in are given a short diagnostic interview.
This is referred to as their Debt Gateway and it enables the bureau to gather brief
details of the client’s enquiry, assess the complexity and urgency of the issue and
determine the best way to deal with it. Sometimes all a service user needs is some
further information on their query, such as a leaflet or signposting to a web-site,
whereas at other times they may need a referral to a specific service within the
bureau or even out to an external organisation.
A key role of the Debt Gateway is also to identify who could benefit from further
financial education and capability training; those who could benefit are referred to
a Money Active session. Money Active is a national scheme run through Citizens
Advice across the UK and is funded by Nationwide Building Society. The aim of
Money Active is to train and support staff and volunteers in Citizens Advice Bureaux
across the UK to carry out financial education work in their area. It is a three year
project, running from 2009 – 2012, that supports the recruitment and training of 1,300
11
volunteers to deliver financial education sessions. At Plymouth CAB these sessions
take the form of either a two hour workshop or a one to one session, both of which
cover basic skills around budgeting, banking, credit and saving. Clients are given the
option of attending a workshop or a one to one session in recognition of the fact
that they may not be comfortable sharing in a group session.
In December 2010 Citizens Advice released a statement with interim results of Money
Active. Two thirds of people who attended a Money Active session report feeling
more confident about handling their finances as a result. Over a third said they
would now open a savings account, and a third planned to set up a budget to help
them manage their money. A further 19% intended to get a better deal from their
energy supplier. It is estimated that Money Active has so far reached around 80,000
peopleii.
2.3 Financial Capability and Debt Advice
As mentioned above, debt advice is a core part of the service that Plymouth CAB
provides and they have a well-established team who provide case work to over
1,000 service users per year. However, they know that, unless they find a way of
equipping debt service users with the skills and confidence to manage their financial
affairs more effectively in the future, they are likely to return to the bureau with the
same issues. It is for this reason that Plymouth CAB place a particular emphasis on
delivering financial capability sessions to their own debt service users. But what is
financial capability? And how does it relate to debt advice?
The true origin of the term ‘financial capability’ is not entirely known but one of the
first places it is mentioned is in The Financial Services and Markets Act of 2000 where
it was stated that one of the four statutory objectives of the Financial Services
Authority was to “promote public understanding of the financial systemiii”. In 2003 the
FSA set the foundations for the National Strategy for Financial Capability in place
and established the Financial Capability Steering Group to oversee the work. The
aim of the new strategy was to “provide customers with the education, information
and generic advice needed to make their financial decisions with confidence”iv.
Financial capability also has its roots in financial literacy. Around the same time as
the National Strategy was set in place, the FSA joined forces with the Basic Skills
Agency, as well as a range of expert partners across the financial capability sector,
to develop a model for drawing the different elements of financial capability under
one banner. They did this through the development of the Adult Financial Capability
Framework: “Following the recommendations of the Adult Financial Literacy Advisory
Group report and the introduction of the new adult literacy and numeracy core
curricula we were keen to develop a framework that outlined the skills and
competencies that were deemed necessary for financial capability”. The new
framework broke financial capability down into three elements:
1. Financial knowledge and understanding
2. Financial skills and competence
3. Financial responsibility
In 2005 the Financial Services Authority conducted their baseline survey which
measured the financial capability of 5,300 adults in the UK. By this point,
12
understanding of financial capability had evolved and so the new survey measured
financial capability along five new components:
1.
2.
3.
4.
5.
Making ends meet
Keeping track of your finances
Planning ahead
Choosing financial products
Staying informed about financial matters
These five elements were deemed to cover the core skills necessary for an individual
to be considered ‘financially capable’ and continue to frame understanding of
financial capability today.
There are a number of guides for providing financial capability support, both one to
one and in a workshop environment. But how do you provide this work alongside the
case work that goes with advising a service user with debt? Or with a group of
people who are struggling to make ends meet? There are a few examples of
financial capability delivery, including within the Citizens Advice Bureau:
CAB Financial Capability Pilots: From August 2008 until July 2009 Citizens Advice was
funded by the Department for Business, Enterprise and Regulatory Reform (BERR) to
run a pilot project in 10 UK locations to people who were excluded from mainstream
financial services. The project also provided training to frontline workers who
delivered services directly to people experiencing financial exclusion. The ‘FinCap’
project aimed to improve clients’ financial capability and encourage them to pass
their learning on to friends and colleagues. In total, the FinCap project delivered 458
training events and trained 1,083 service users and 1,433 staff. The number of staff
who said they were fairly or very confident in supporting service users on financial
issues increased from 48% to 94% following the training. Service users who described
themselves as fairly or very confident in managing their finances increased from 60%
to 89% following the trainingv.
CAB Financial Skills for Life: Launched in 2002, the Financial Skills for Life project was a
joint venture between Citizen’s Advice and Prudential that piloted new ways of
delivering preventative financial education to a range of adults at risk of financial
exclusion. The pilot ran for three years and in total there were nine pilots based in
local bureaux that focused on developing good practice in delivering financial
education to adults. The evaluation of the project was conducted by ECOTEC and
identified some significant outcomes for service users. In the short term individuals
involved in the project displayed greater confidence around personal finance and
longer term benefits showed that these outcomes lasted; individuals surveyed
months later showed that they had changed their patterns of behaviour, improved
their economic wellbeing and increased their savingsvi.
Capitalise Financial Capability Pilot: Capitalise is London’s debt advice partnership
comprised of 17 organisations and managed by Toynbee Hall. 56 advisors from these
organisations deliver face to face debt advice to people experiencing social and
financial exclusion across the capital. The financial capability pilot was based on the
idea that increasing financial capability is more effective as a crisis intervention than
just dealing with specific debts and the aim was to improve the financial capability
of Capitalise service users, including their skills, knowledge, awareness, confidence,
13
attitude and motivation in relation to their finances. For 12 months Capitalise ran the
pilot, delivering financial capability to a sample of debt service users so that they
had a test group and a control group. The results were significant: 78% of the pilot
group felt more able to manage their money compared to 51% of the control group;
78% of the test group felt more able to plan ahead compared with 34% of the
control group and 47% of the test group felt more able to choose financial products
compared with 31% of the control groupvii.
These three examples illustrate the effect that financial capability can make on an
individual’s experience of debt as well as long term financial well being. Findings
from all three reports suggest that embedding this work within wider debt advice
structures is a key challenge; Capitalise, for example, found that the target pressures
and inflexible structures of debt advice meant that there was little time for any other
work. It is hoped that through this evaluation of the Debt Gateway and Money
Active at Plymouth CAB, some of these challenges can be addressed.
2.4 There are three key aims of the Debt Gateway and Money Active working
together:
1. Increase financial capability: While there is no one established definition of
financial capability, Plymouth CAB focuses on increasing the knowledge, skills
and confidence of service users to manage their money more effectively and
take responsibility for personal financial decisions. Effective financial
management is important for a number of areas in life, so the first key aim of
Money Active is to increase the financial capability of service users at
Plymouth CAB.
2. Decrease repeat appointments within the bureau: Debt service users often
come back multiple times, sometimes even for the same issue. It is hoped that,
by adding in financial capability training, it may be possible to decrease the
amount of clients who come back in for more advice, especially for the same
issue.
3. Explore implementation: Compare experiences of one to ones and group
sessions to identify which are most successful in tackling debt service users’
money problems. Explore different methods of working to uncover the best
way for Plymouth CAB to embed financial capability within debt advice
without taking away from debt service delivery.
Part 4 – The model
There are three key steps to the Gateway Service/Money Active model:
1. Assessment and referral
2. Service delivery
3. Follow up
The below model represents the current structure of the Debt Gateway/Money
Active model at Plymouth CAB as well as their plans going forward that have
resulted from this evaluation. For further information on the findings from this
evaluation, please see the section on ‘findings’.
14
4.1 Assessment and referral
Plymouth CAB is open from Monday to Friday, 9am to 5pm for visitors and interview
sessions. The Debt Gateway is open for those without an appointment from 10am to
1pm Monday to Friday and from 2pm to 5pm on Mondays. As a service run by a fulltime worker, the Gateway service has been running since November 2009. On
average they will see around 10 service users per morning and issues can be very far
ranging. Sometimes the issue will not actually be about debt – eviction proceedings
for example are often dealt with by those specialising in housing advice – in which
case the service user gets referred to another department. The Gateway advisor
records the reason why the service user is seeking advice and will do what they can
in the session but for any other specialist advice or for an extended advice session
they will refer the service user to a case worker. Sometime a service user just needs
signposting to a particular service or assistance in filling in forms in which case the
Gateway advisor helps them with that at the time. This ultimately saves the bureau
time in the long run as otherwise that service user would require a one to one
appointment with a case worker.
At the time of the evaluation Bill, the Gateway advisor, estimated that he referred
approximately 10% of all service users that came to the Debt Gateway to a Money
Active session. Referrals are made at the discretion of the Gateway advisor and
based on their assessment of the service users money management skills. If a service
user’s debts appear to be as a result of their poor money management, for example,
they will be referred to a Money Active session. The advisor records the reason for this
referral as well as what type of session they referred the service user to (group or one
to one). In most cases clients are referred to a workshop as this saves time and results
show that participants benefit from the group environment. If a client is insistent that
they do not want to attend a group session, or if their problem is very specific, they
are referred to a one to one. Because referrals are up to the discretion of the Debt
Gateway advisor, extensive training is very important.
Money Active workshops are also advertised through flyers in the main bureau
reception and in the interview rooms for anyone else who is interested in attending.
Clients may also be referred to a Money Active session by a frontline worker.
4.2 Service delivery
Plymouth CAB has been running Money Active sessions for approximately 4 years but
in 2009 they decided to look at further ways of developing these sessions to meet
the needs of their service users more effectively. They began this process by setting
up regular team meetings with staff that were directly involved in the project.
Through the course of these meetings they discussed how many of their service users
were in financial difficulties simply because they had never had an opportunity to
learn money management skills. A number of these service users were not even in
debt but could benefit from additional training to prevent money problems
developing in the future. As a result of these conversations a new training workshop
was developed using power point, and additional materials were developed and
collated for service users to take away with them e.g. budget sheets, money saving
tips, Money Made Clear leaflets, etc. The presentations and course materials are
updated on a regular basis to make sure that all information is current. At all of the
workshops at least two members of staff or two volunteers are present to facilitate
15
the session and support those attending. Service users will often need support in filling
in forms or completing course activities so it is useful to have more than one
facilitator present.
As the workshops are delivered in a group environment, the first part of the session
involves setting ground rules for conduct. For example ‘it’s fine to disagree with and
challenge each other but it’s important to challenge views and not individuals’. As a
lot of the topics covered in the sessions touch on sensitive issues, such ground rules
are important for engendering a safe and supportive environment. The workshop is
then delivered under the heading of ‘Money Management Skills’ and the key aim of
the session is stated as ‘to provide you with the skills to manage your money better’.
The facilitator then delivers the workshop along five main themes:
• Budgeting
• Borrowing and saving money
• Dealing with debts
• Signposting
• Questions and answers
During the session on ‘budgeting’ participants are asked to complete a financial
statement. A financial statement is a form containing boxes for ‘income’ and
‘expenditure’ that is filled out by an individual based on the details of their personal
financial circumstances. It is often used in debt advice as a tool for negotiating with
creditors; once income and expenditure are recorded, whatever money is left over is
used towards payment of debts. It can be a useful budgeting tool on its own
however and people are often surprised when they are faced with the full picture of
their income and expenditure. As one participant said in a Money Active workshop –
“it’s a real eye opener!”
Participants can be hesitant about staying at the start of the session when faced with
the group environment. The session’s coordinator will invite them to stay for a few
minutes and see how they feel, emphasising that they are welcome to leave at any
time. In actual fact, the group environment is an enormous strength of the service
delivery and participants will usually stay throughout the duration of the workshop
and contribute to the session. The workshops are very interactive and service users
are encouraged to join in and participate which facilitates their learning. For
example, during the session on ‘borrowing and saving money’ service users are
asked for examples of sources of credit that they use as well as sources of credit that
they will not use. During one session they added ‘loan sharks’ to the list as well as
‘family and friends’ while acknowledging that the latter can be emotional. They
talked about general distrust of banks – “banks are just organised mafia” – and
interestingly, only one person out of the whole group had heard of a credit union.
When they cited the Provident as a source of affordable credit the facilitator asked
them to guess how much the Provident charges in interest and they guessed 29%; in
actual fact the average Provident loan is charged at 272.2% APRviii. Delivering
sessions in this group environment means that participants can learn as they go and
ask the facilitator questions. It also means that they can learn from each other. One
participant shared that her grandson uses her car but does not contribute any
money towards maintenance or petrol and that the costs really add up. Another
participant said that she had experienced the exact same problem with a family
member and, in her case, she had asked for a financial contribution and that this
16
tactic had worked for her. This peer learning would not have arisen in a one to one –
it can only come from shared experiences.
Plymouth CAB also provides training through A4e for customers wishing to get into
work or to return to work. The workshop outline is almost exactly the same except
that it focuses on employment with the first session covering ‘things that stop
someone returning to work’. Aside from that, the workshops at A4e are delivered
using a power point presentation and participants are given resources to take home
with them, just like the other Money Active workshops at Plymouth CAB.
Plymouth CAB also provides training to the staff and volunteers that support clients
dealing with money problems. This training is delivered through interactive sessions
where staff discuss their various client groups and how they might deliver training to
them. For example, some clients have learning disabilities so a particular focus needs
to be on independent living and where they can go to for further support. Clients
with mental health issues need to know about other services in their area that can
help them, such as Mind or Plymouth Options. During the training staff are shown the
power point presentations that are used in the workshops with clients and are given
guidance on how to deliver each session. For the session on budgeting, for example,
it is important for staff to know how to challenge high expenditure on non-essential
items that clients write into their financial statements. Staff are also given information
on local doorstep lenders, local credit unions and how to make referrals.
Follow up
Currently Plymouth CAB do not follow up with service users who are referred to a
Money Active session. As part of this evaluation, however, they did a follow up
phone call to a sample of participants and the results show that participants found
the sessions very useful, retained knowledge and started putting their financial plans
into action. For further information on the findings of this follow up, see the section on
‘findings’. Going forward, it would be useful to continue to follow up with a sample of
clients who are referred to a Money Active session. This would have two benefits: it is
useful for evaluation purposes as it can demonstrate the impact that the sessions
make on participants, but it is also an opportunity for participants to reflect on what
they have learned.
17
The Debt Gateway and Money Active at Plymouth Citizens Advice Bureau
Money Active
Workshop
Debt Gateway
Further specialist
advice
Money Active
One to one
Follow up
(sample)
External agency
Staff training
Assessment and referral
• All debt clients come through the Debt Gateway
• Some clients simply need help filling in a form, information on their query or signposting to a different agency while others are
referred for specialist advice
• Approximately 10% of clients are assessed as needing money management training and are referred to a Money Active
session
Delivery
• The majority of debt clients who are referred to a Money Active session are referred to a workshop
• Clients who are not comfortable with the workshop setting or have specific needs are referred to a Money Active one to
one
• Staff are trained to deliver Money Active sessions through group training sessions
Follow up
• Going forward Plymouth CAB are planning to continue following up with a sample of clients after they have attended the
workshop to discuss their progress and what knowledge they have retained
18
Part 4 – Evaluation
Key questions to ask when evaluating an internal project:
What are the key objectives of the project?
How to strike the right balance between ‘perfect’ methodology and actual capacity
to deliver?
What information is already being captured (for funders, stakeholder, etc.)?
What kind of evaluation do we want to undertake?
What methods are we going to use?
4.1 The trouble with evaluation
A lot of organisations are required to evaluate their projects, often for the purpose of
demonstrating to funders that agreed objectives have been met. These objectives
are often demonstrated via ‘outputs’ – the number of clients seen, the number of
workshops held, the number of hits to a web-site, the number of calls received, etc.
This is useful information to have, but in order to demonstrate the true impact that an
organisation is making on their client group, community or audience, an evaluation
needs to go beyond outputs.
In his original work in 1959, Donald Kirkpatrick made recommendations for evaluation
that have laid the basis for a lot of thinking in the subject ever since. He argues that
programme evaluation should concentrate on four levels:
The four levels of programme evaluation:
Level 1: Reaction – evaluating the reaction of participants to the intervention
Level 2: Learning – measuring the knowledge, skills and attitudes gained from the
intervention
Level 3: Behaviour – measuring the changes in behaviour that resulted from the
intervention
Level 4: Results – relating the results of the intervention to wider organisational
objectives
Alongside outputs, many evaluations conducted in the financial inclusion and
capability sector only capture the first level of evaluation. This is with good reason:
reaction is the easiest to capture; it helps organisations to improve their programmes;
and it can be easily analysed. Capturing reaction means asking questions such as
“what did you like about the training”, “what did you not like about the training” and
“what suggestions do you have for the training going forward” and is often recorded
in an evaluation form after a course or an intervention has taken place. Just like
outputs, reaction is useful information to capture, but it does not provide the whole
picture. In order to capture real changes and attribute them to an intervention, it is
necessary to go beyond this first level, to look at changes in behaviour and learning
and, where possible, map these changes back to the activities of the organisation.
19
Confusion around how to capture outcomes is compounded by the fact that there is
no single accepted definition of evaluation, let alone one method. Here are a few
examples of definitions of evaluation that are widely used:
“Evaluation is about using monitoring and other information you collect to make
judgments about your project. It is also about using the information to make changes
and improvements” (Charities Evaluation Service)
“Evaluation involves the systematic collection of data about the characteristics of a
programme, product, policy or service. As part of this process, evaluation will often
explore what needs to be changed, the procedures that are most likely to bring
about this change, and whether there is evidence that change has occurred” (Warr
et al., 1970)
“Evaluation is the systematic assessment of the worth or merit of an object” (Joint
Committee on Standards for Educational Evaluation, 1994).
This lack of one accepted definition is in part due to the fact that evaluation is a
relatively new field; real interest in the area can only be dated as far back as the
1970s. The emphasis in early evaluations was on accuracy and specific measuring
techniques but in recent years this has shiftedix. With the rise of action learning and
self-development programmes, learning is now seen as arising within and through a
situation rather than just through formal evaluations and issues of subjectivity and
ethics are of prime concern in today’s evaluations.
Subjectivity is often at the heart of why many organisations struggle to evaluate their
services beyond the reaction level. What actually is the relationship between inputs
and outcomes? Can a change in learning or behaviour really be attributed to an
intervention? What does ‘success’ look like? These are some of the questions that
organisations struggle with when designing their own evaluation strategies because
they often involve making subjective judgments. This brings us to a key question: who
is ultimately responsible for making these judgments? Who has a right to?
While evaluation is a new field, financial capability is an even newer one, so there
are not yet many established examples of evaluation best practice in the sector.
Added to this is the fact that many outcomes in financial capability are ‘soft’
outcomes which can be difficult to capture and, again, sometimes require
subjective judgements to be made. Examples of soft outcomes include changes in
knowledge, confidence, motivation and behaviour; all of which are extremely
important and indicate that a real change has taken place but are not very
straightforward to capture.
So these are the key challenges that the sector faces when it comes to evaluating
work in financial capability:
• There is no single accepted definition or method of evaluation;
• Financial capability is a young sector so does not have one accepted
method of evaluation;
• Success in financial capability often means positive results in soft outcomes
which can be difficult to measure;
20
•
Measuring outcomes often means making subjective judgements which some
organisations may find difficult.
Understanding that evaluating work in the sector is not always a straightforward
process, a key aim of RSB Innovate is to provide the sector with evaluation
frameworks that can be applied to different settings. Different organisations were
deliberately chosen so that different approaches could be tested and best practice
established. There was no guarantee of how this would work in practice, but the
findings show that there are certain key identifiable themes across all four
organisations. While the four organisations evaluated through RBS Innovate all deliver
services directly to clients, the stages outlined below can also apply to other types of
services working in financial inclusion.
4.2 Evaluation Stage 1: Set up of evaluation
As described above, one of the three key aims of RBS Innovate was to design
evaluation frameworks that are applicable to the whole sector. Whilst this is not
necessarily going to be achieved through working with just four organisations, we
deliberately chose organisations that are different so as to test out different
approaches and establish common best practices. It is important to note that we did
not start the process wedded to one evaluation framework or methodology. Instead,
I (as the Evaluation and Policy Manager) started the evaluation process by visiting
each individual project and approaching each evaluation with an open mind. In this
way we were able to design an evaluation framework tailored to each organisation
while at the same time staying mindful of key themes that emerged from each one.
To start the evaluation process, an initial meeting was held in February 2010 with Gill
Tishler, then Director of Plymouth CAB, and Jane Guy, A4A Training Manager. The
purpose of this meeting was to learn more about Plymouth CAB and to gain an
understanding of what they were looking to achieve through Money Active. As a
well established organisation, they had already spent a lot of time thinking about
what outcomes they wanted to achieve for service users and how their services
could best enable those outcomes. As discussed previously, they had also been
thinking a lot about their financial capability work and what more they could do for
their service users. Their initial thinking had been to capture the cumulative impact of
all of their work but, given the time constraints on the evaluation and current interest
in embedding financial capability within debt advice, we decided to focus on this
one area. The meeting presented an opportunity to talk about the service in depth
and really hone down on the aims and objectives of the Debt Gateway and Money
Active. This represents the first stage in developing an evaluation framework: decide
what your key aims of your intervention are.
As discussed above, a key question in evaluation is who has the right to make
decisions on evaluation? With few exceptions it is the organisations themselves that
know their client group or target audience best, as well as their organisational
objectives. They also know the resource that has been invested in an intervention
and the resulting benefit to their target group so it is often the organisations
themselves that are best placed to decide the terms of an evaluation.
21
Aims
Increase financial capability
Decrease repeat
appointments
Explore implementation
Detail
Financial capability can relate to a number of
different factors but Money Active specifically seeks
to increase knowledge, skills, motivation and
confidence in the key areas of budgeting, borrowing
and saving money, dealing with debts, and where to
go for help. Increases are demonstrated by
participant responses as well as examples of positive
action.
In order to accurately assess decrease in repeat
appointments and attribute that decrease to the
Money Active sessions it would be necessary to
compare data captured during the evaluation
period to data captured outside of the evaluation
period controlling for particular factors (i.e. the wider
economic environment, other financial support
received by clients etc.). Unfortunately we did not
access to the necessary data to gather this
information so we instead focused on wider
organisational structures, the overall process and
anecdotal evidence. Going forward Plymouth CAB
might wish to establish a ‘baseline’ of data and
compare that to data gathered during an
evaluation period.
Explore how referrals from the Debt Gateway to
Money Active work in practice; examine whether
more people get referred to a one to one or a group
session and which one is more effective; gather
information on the overall process and what can be
improved going forward.
A key part of the initial conversation with Gill and Jane was also about capacity. In
order to embed evaluation activities within an organisation successfully, it is
necessary to be realistic about what is achievable as regards staff and available
time. Gill was the prime contact for the majority of the evaluation until she left the
bureau at which point Jane became the main contact. In order to make sure we
designed an evaluation framework that would be realistic, we discussed what their
time was like and what format they would be comfortable collecting and relaying
data in. This is a key stage for any organisation to undertake; you could create the
most robust evaluation framework that would guarantee a wealth of rich
information, but if it is not going to be realistic to gather that information it will not be
achievable. This represents another key step in designing an evaluation framework:
find the right balance between the ideal evaluation goals and the actual capacity to
deliver. This point is especially relevant to Plymouth CAB as any evaluation activity
takes time away from delivering the actual service. Evaluations are supposed to
enhance an intervention, not detract from them, so getting the right balance is very
important.
During the meeting we also discussed what information Plymouth CAB was already
collecting, both for their own use and for any other funders that they were reporting
to. This represents another important step when designing an evaluation framework:
22
establish what information is already being captured so as to utilise whatever
information is already available. This is especially pertinent for organisations that
have to provide detailed information to their funders. Funder reports will usually
contain key information on a programme or intervention so if you wish to conduct a
more in-depth evaluation internally, it is useful to take this information into account
when designing an evaluation framework for your organisation so that you are not
replicating any work. Some funders also require organisations to use specific
evaluation materials so it is important to take these requirements into account as
well. Plymouth CAB, as part of the national network of bureaux, are required by their
funder to use specific questionnaires to capture evaluation data in the Money Active
sessions. This meant that we were limited as to what additional information we could
gather in the actual workshops as inundating participants with evaluation forms
should be avoided. It is important to acknowledge restrictions such as this right at the
start however and build an evaluation structure around these existing activities.
4.3 Evaluation Stage 2: Methodology
After the first meeting I went away and worked on what the evaluation framework
for Plymouth CAB should look like. The whole process took about a month and
involved sending ideas back and forth between myself, Gill and Jane. Again, it was
very important to make sure that the ideas we were generating were achievable
and sustainable. As part of this work I also sat in on the Debt Gateway and a Money
Active session so I could understand how the service works.
Part of the work that I did during the process of designing an evaluation framework
for Plymouth CAB was determine what evaluation type and approach would be a
good fit. RBS Innovate is not about conducting formal, external evaluations as the
organisations being evaluated are involved in determining the evaluation terms and
responsible for supplying data. Because of this, it would be nearly impossible to claim
that RBS Innovate is providing an external, objective evaluation. What RBS Innovate is
focused on, however, is developing internal evaluation frameworks and showcasing
what works best, both in evaluation implementation and for actual service delivery.
Because RBS Innovate differs from a more formal evaluation in this way, adherence
to one type or approach did not matter so much but it is still useful to align the work
with evaluation models that are acknowledged in the wider field of evaluation, both
in the UK and abroad.
In order to figure out what type or approach of evaluation to take, it is useful to
review what exactly we are evaluating. The Debt Gateway identifies service users
who might benefit from a financial capability session and subsequently refers them
to a Money Active session – either a workshop or a one to one. In the Money Active
session service users are given information on how to budget and save and manage
their finances more effectively. The intended end result is that service users are able
to manage their money better (that is, increase their financial capability) and
subsequently do not need to come back to the CAB for debt advice as often. As
there are not many existing models of financial capability embedded in debt
advice, a key aspect of the evaluation needed to be a reflection on the process as
well as a report on its success. Some questions that address this learning are ‘how
have service users benefitted from their referral to Money Active?’, ‘what are some
of the unexpected outcomes?’, ‘what can be improved going forward?’ and ‘how
can such a model be replicated elsewhere?’
23
Evaluation Type: Formative or Summative?
One of the first points we discussed around methodology was whether Gill and Jane
wanted the evaluation to be formative or summative. A formative evaluation asks
“how can the programme be improved?’ and is undertaken in order to provide
feedback with the goal of progress. It is relatively informal with the evaluator working
alongside the practitioner in order to identify the strengths and weaknesses of a
programme or interventionx. It usually takes place during the lifetime of a project as
opposed to at the end of it and encourages an on-going, reflexive approach and
conversations between the organisation and the evaluator. Summative evaluations
on the other hand ask “what is the overall merit or worth of the programme? Should
it be modified? Should it be continued?” Overall, such evaluations aim to determine
how effective an overall programme or project is and whether or not it should
continue. Summative evaluations are usually more formal in character with the
evaluator operating in a more independent role. Summative evaluations, in contrast
to formative, take place at the end of a project. I asked Gill and Jane what type of
evaluation they were interested in – did they mainly want to determine whether the
programme had been a success or a failure? Or were they interested in improving
the course? They decided that they wanted the evaluation to be both formative
and summative so this was taken into account in the evaluation design. Simply put it
means that I worked with Gill and Jane throughout the project; we discussed the
data as it came through, the findings that really stood out and what it meant for the
project going forwards. This is another important step to consider when designing an
internal evaluation structure – what type of evaluation are you interested in
conducting? The benefit of this approach is that evaluating the process as it goes
means you can make changes where appropriate. It also meant that Gill and Jane
were on board with the evaluation methods.
Evaluation Approach
As for the type of evaluation approach we went for with Plymouth CAB, it is most
closely aligned with the decision- and accountability- oriented study which is a type
of improvement- and accountability-oriented evaluation approachxi. As stated
above, because the organisations are themselves involved in the evaluation in RBS
Innovate, the programme does not strictly adhere to any one evaluation approach,
but this is the one it most closely resembles in ethos and in practice. Decision and
accountability oriented studies stress the need to fully assess a programme’s value.
They look for all relevant outcomes, not just those keyed to programme objectives
and they generally use multiple qualitative and quantitative assessment methods in
order to provide cross-checks on findings. Following on from the discussion of
formative vs. summative above, this approach emphasizes that programme
evaluation should be used proactively to help improve a programme as well as
retroactively to judge its value. Under this approach an evaluations’ most important
purpose is not to prove but to improve. A key ethos of the approach is that an
evaluation can best effect change in a target groups’ behaviour by involving
members in planning, monitoring, and judging the enterprise. This active
participation by stakeholders not only gives them a voice in the determinations that
will affect them but also brings them on board with the enterprise from the start.
24
There are some key advantages associated with the decision and accountability
approach:
• It encourages program personnel to use evaluation continuously and
systematically to plan and implement programmes that meet beneficiaries’
targeted needs;
• It aids decision making at all programme levels and stresses improvement;
• It balances the use of qualitative and quantitative methods;
• It can provide the framework for both internal and external evaluations.
However, there are also some limitations:
• It may overemphasise formative evaluation and give too little time and
resource to summative evaluation;
• The collaboration between evaluator and stakeholders may impede the
evaluation or bias its results;
• The internal nature of evaluation may mean some of the benefits of an
independent, detached perspective are lost.
Given the objectives of RBS Innovate and that we are open about the fact that this is
not an external, objective evaluation, the advantages of this approach outweigh
the limitations.
Using Mixed Methods
As can be seen from the evaluation framework for Plymouth CAB, both quantitative
and qualitative data were captured in the evaluation and both were treated as
equally important.
Collecting both quantitative and qualitative data in this way is referred to as using
‘mixed methods’. Using mixed methods in one research study can serve a number of
purposes. One purpose, for example, is to confirm that a finding is true – if both the
quantitative and the qualitative data point to the same finding then confidence in
that finding is stronger than if it is discovered using just one method. This is referred to
as ‘triangulation’ which means using more than one approach to the investigation of
a research question in order to enhance confidence in the ensuing findingsxii. In the
case of Plymouth CAB, different methods are not used in order to enhance
confidence in the findings so much as to uncover information that could not be
found using only one method.
Quantitative Data Collection
The data gathered during the Debt Gateway is mainly quantitative. While some
questions asked of the clients are open ended (“what can we help you with
today?”) responses are recorded according to general categories. For example, the
reason for a client seeking debt advice is recorded according to seven categories:
• Living beyond means
• Reduction in household income
• Loss of employment/benefits
• Relationship breakdown
• Illness/accident
• Increase in household expenses
• Other
25
Recording answers and coding them according to category or theme allows for
ease in analysis later on. Using this method it is possible to uncover, for example, that
the majority of clients come to the bureau for debt advice because they are
experiencing a reduction in household income. This information can then be used to
target services more effectively. The Debt Gateway also captures information on
why a client was referred to a Money Active session and what type of session they
were referred to (workshop or one to one).
Workshop questionnaires
After the Money Active workshop service users are given a questionnaire to fill in. The
questionnaire has six questions and asks participants to score their answers on a
scale from ‘very unconfident’ to ‘very confident’. These responses were then
assigned numerical values so that they could be compared. The questions relate to
different information – for example some ask the respondent to compare their
confidence before the workshop to their confidence after completing the workshop.
Other questions relate to how the respondent found the workshop and what they
would change about it. In this way the questionnaires capture ‘reaction’ data as
well as ‘learning’ data (see section on ‘evaluation’ for more information). When
comparing changes in learning and behaviour before an intervention compared to
after, it is usually good practice to administer the questionnaire both before and
after the intervention has taken place. However, as time is limited in the workshop
and, as discussed previously, some service users have language and literacy issues,
this was not possible to do in this instance. Asking participants to code their responses
on a scale allows for quantitative analysis. A five point scale is used so that
respondents can give a neutral answer (3) if they do not feel strongly one way or
another. There is a body of research that states it is important to provide a neutral
answer and there is another body of research that states that prompting
respondents to make a choice (i.e. using a four point scale where there is no neutral
option) is preferablexiii. Often it will depend on the nature of the evaluation but, for
the purposes of this research, it was felt that forcing respondents to have an opinion
one way or another was false and that it was important to provide a neutral
response option.
Qualitative Data Collection
Telephone follow up data
Sharon and her volunteers followed up with 18% of service users who had attended a
MoneyActive workshop to find out what impact it had made on them, what
changes they had made and what information they had remembered. In particular,
they asked them:
•
•
•
•
•
If their attitude towards money had changed since the workshop
If they had taken any positive steps as regards managing money since the
workshop
Whether the information they had received on budgeting had made a
difference to how they manage their money
If they remembered examples of sources of low cost credit
If they could remember an example of a priority bill or debt
26
•
If they had any money saving hints or tips they could share with others
The questions were open-ended and responses were recorded exactly as they
respondents had given them. So that patterns could be established, the data was
then coded and converted into quantitative data.
Staff Interviews
As a final part of the evaluation process, staff at the bureau were Interviewed. One
interview was held with Jane Guy, Training Manager, and Ann Kinahan, Chief
Executive, to discuss how they felt the overall process had gone, issues they had
identified and what they would like to change about the Debt Gateway and Money
Active going forward. Another interview was held with Sharon who facilitates the
majority of the Money Active workshops to talk about the workshops in depth and a
final interview was held with Bill who runs the Debt Gateway. Each interview lasted
approximately an hour and the purpose of these was to gain an in-depth
understanding of the outcomes of the evaluation from the people who run the
programme on a day to day basis. The interviews were conducted with the use of a
topic guide (Appendix A).
Case studies
As part of the qualitative element of the evaluation, case studies were also collected
to illustrate the impact that Money Active and Plymouth CAB more widely has made
on the lives of participants. Case studies are a very effective way of making an issue
personal by looking at the lives of people affected by an intervention and identifying
the ways that that intervention has made a difference. They often arise through
contact with particular individuals but can be planned in advance so that certain
participants are targeted for a case study.
27
Plymouth Citizens Advice Bureau Evaluation Framework
When
May 1 2010 – August 1 2010
Tools/Method
Existing internal system
Details
For anyone referred to a Money Management
session (group or one to one) record:
• Reason for seeking debt advice
• Reason for referral
• Type of session referred to (group or one to
one)
• Did they attend the session?
Feedback questionnaires
August 2 2010 – November 1 2010
November 2 2010 – December 1
2010
Existing internal system
Monitor those who attended the money management
session to see
• If they come back to the CAB for further
money-related advice
• If so, what kind of advice
Phone questionnaire
Follow up with a sample of those who attended a
Money Active workshop to find out:
• Progress since attending the session
• If the session influenced their behaviour or
prompted them to take any action
Face to face interviews
Interview with staff about how they found the process
• Jane (overall process)
• Bill (Debt Gateway)
• Sharon (Money Active workshops
Existing internal system
Compare captured data to:
• Debt clients who aren’t referred to a money
management session
• Data from before the formal referral process
was established
28
Part 5 – Findings
Key Findings
The majority of debt clients who were referred to a Money Active session from the
Debt Gateway were referred to a workshop (90%) as opposed to a one to one
(10%).
The most common reason a client was referred to a Money Active session (36%)
was so that they could ‘learn to prevent and deal with their debts’ followed closely
by ‘learn to budget their money and complete a financial statement’ (30%).
The highest proportion of Money Active participants reported enjoying the session
on budgeting (26%).
When asked what they would change about the session 99% of respondents said
‘nothing’.
100% of participants said that they felt their attitude towards money had changed
since attending the workshop with 83% saying their attitude towards budgeting had
changed.
100% of respondents were able to give an example of an affordable source of
credit and 79% of respondents were able to give an example of a priority bill weeks
after attending a workshop.
Plymouth CAB found that if a client takes part in a Money Active session any
subsequent debt case work takes 1/3 the time it takes for clients who have not been
through a Money Active session.
5.1 Evaluation of the project
Basic information – who took part in Money Active?
In total, Plymouth CAB held 17 internal Money Active workshops that were attended
by 81 participants over the evaluation period. They also held 10 external Money
Active workshops attended by 58 participants. In total they trained 38 staff and 101
service users through Money Active.
While 101 service users were referred to a Money Active session, data on these clients
from the Debt Gateway is only available for 69, representing 68% of the total. This is
because a number of the clients were referred to Money Active via front line workers
so data on the reason for the referral was not recorded. According to the data from
the Debt Gateway, the most common reason for seeking debt advice was because
service users were living beyond their means. A high proportion also came to the
service because they had experienced a reduction in income:
29
Reason for seeking debt advice
Living beyond means
7%
22%
9%
Reduction in household
income
Loss of
employment/benefits
13%
Relationship breakdow n
Illness/accident
20%
13%
Other
16%
Increase in household
expenses
The most common reason a service user was referred to a Money Active session was
so that they could to learn how to prevent and deal with their debts. Another
common reason was because the advisor felt the service user could benefit from
completing a financial statement and learning how to budget:
Reason for referral to a Money Active session
Learn to
prevent/deal with
debts
6% 3%
Learn to budget
money/complete
financial
statement
Increase
awareness of
benefits/entitleme
nts
Increasing
income/reducting
outgoings
12%
36%
13%
30%
By far, the majority of service users (90%) were referred to a group session. Only 10%
of service users who were referred to Money Active were referred to a one to one
and this was usually because they were not comfortable taking part in a group
session or had specific issues that were better handled in a one to one setting.
Questionnaire data was captured from 33 service users at the Money Active
workshops representing approximately 33% of the total. 61% of respondents were
female, 33% were male (6% ‘no response’):
30
Gender
6%
Female
33%
Male
Unknow n
61%
Information on age range showed that respondents were mainly between the ages
of 16 to 44 although the age distribution was fairly even:
Age range
10%
25%
16 - 24
16%
35 - 44
25 - 34
55 - 64
45 - 54
23%
26%
100% of respondents reported that they were white British and 24% said that they had
a disability. The evaluation also showed that the majority of respondents were in
work, followed closely by on benefits:
31
Source of incom e
6%
9%
In w ork
9%
43%
On benefits
Benefits & employment
Other
Blank
33%
As discussed previously in the section on ‘evaluation’, to fully capture the impact of
an intervention it is important to capture evaluation data across four levels: reaction,
learning, behaviour, and results. The rest of the findings from the evaluation of the
Debt Gateway and Money Active at Plymouth CAB are therefore discussed along
these four levels.
Level 1: Reaction
Reaction captures information on how participants found an intervention; what they
liked, what they did not like and their suggestions for the intervention going forward.
As reaction data can help a project to grow and develop, it was included as part of
the Money Active evaluation.
Participants were asked to list what they most enjoyed about the Money Active
workshop and responses were fairly evenly split, with the biggest proportion of
respondents reporting that they enjoyed the session on budgeting:
What did you find m ost useful?
16%
21%
All
Budgeting
16%
Priority vs non priority
Group dynamic
26%
Advice/explanations
21%
As discussed previously, some participants were at first unsure of taking part in a
group workshop but it actually ended up being a strength. As demonstrated above,
32
roughly 16% listed the group dynamic as the aspect of the training that they found
most useful, without any prompting:
“I liked that everyone was in the same position as me and I didn't feel alone”
“The feedback from other people on the course”
“Looking at all areas of debt and discussing debt openly”
Participants were also asked ‘was there anything you didn't like about the training or
anything we could do better?’ Out of the sample group only one person gave a
response saying they felt it took too long. Other than that respondents either said
‘no’ or left their answer blank.
Levels 2 & 3: Learning and behaviour
Levels 2 and 3 go one step further by capturing the changes in learning and
behaviour that have occurred as a result of an intervention taking place. These are
the softer outcomes that can demonstrate the long term impact than an
intervention has on its participants. This part also helps us to assess the first aim of
Money Active; that is, how much the workshops have increased the financial
capability of participants.
Participants were asked if there were things that they learned during the session or
changes that they were planning to make. The majority of responses referred to
paying off debts and bills followed by skills related to budgeting:
Things you learned/changes you plan to make?
24%
29%
Budgeting
Debt/bills
Other
47%
Participants were also asked what they are planning to do about managing their
money following the Money Active workshop. They were given six options and were
allowed to tick as many options as they liked. ‘Set up a budget to help you manage
your money’ was the most popular, followed by ‘look for debt advice’:
33
What are you planning to do about managing your money?
30
25
20
15
10
5
0
Set up a
Look for debt Change how Change how Change how
budget to
advice
you save
you plan for you borrow
help you
money
future
money
manage your
spending
money
Claim
additional
benefits
Finally, participants were asked what they are planning to do about their accounts
and service providers. Once again they were given a number of options and
allowed to tick as many that applied. ‘Talk to a local credit union’ was most popular
followed by’ get a better deal with fuel supplier’:
What are you planning to do about your accounts and
service providers?
20
15
10
5
0
Talk to local
credit union
Get a better deal
with fuel
supplier
Change your
bank account
Open a savings
account
Open a bank
account
While this data shows that clients are still largely concerned with paying off existing
debts and bills, it does show that other, longer-term plans are a priority as well;
budgeting for example featured in a number of responses. This suggests that
delivering information on debt in conjunction with wider information related to
financial inclusion is beneficial as it relates the two together. The financial statement
for example is a tool used in negotiating with creditors but it is also a tool for
budgeting. Completing a financial statement made clients acutely aware of their
incoming and outgoings and how these impacted on their financial management
more widely.
As part of the evaluation case studies were also collected to illustrate the impact
that the course has made on the lives of participants, some of it unexpected:
34
Mrs Brown was referred to a financial capability session after coming in to Plymouth
CAB for support. She is 59 years old and her husband had suddenly left her after 37
years of marriage. She has suffered depression for many years. Mrs Brown’s husband
had always dealt with all finances so, when he left, she was not only emotionally
broken but was at a total loss financially. She discovered bills had not been paid and
that she had outstanding rent, council tax, water and overdraft debts totalling
£9,700. In addition, her husband had fraudulently signed her name and had claimed
housing benefit to which he was not entitled.
Mrs Brown attended a Money Active session and as a result felt much better able to
manage her finances. With ongoing support from the bureau she is sorting out the
debts. She now has the tenancy and council tax liability in her sole name and is not
being pursued for the overpayment or arrears. The bureau helped her to apply to
SWW special assistance fund for help with the arrears on the water debt and await a
decision. Overall Mrs Brown said she has benefited significantly from the financial
capability approach and feels much more in control of her finances and able to
manage them effectively.
Jason is divorced and has a dependent son who lives with his mum although Jason
is currently fighting for custody. He is a housing association tenant in receipt of
income support, housing and council tax benefit, and disability living allowance. He
has Aspergers syndrome and has long term mental health issues.
Jason has £6,500 of debt which he had been trying to manage over several years.
The debt was affecting his health and he did not feel he could manage any longer.
As a result he was referred to and attended a money management session and a
follow up appointment was made with a debt caseworker. The result of these steps
was:
• Jason successfully applied for a Debt Relief Order taking away the stress and
anxiety he was suffering. He is now opening post and answering his telephone
which he had not done in a long time.
• A referral to a welfare benefits adviser resulted in his income support increasing
and high rate care disability living allowance being awarded indefinitely.
• A referral to a mental health agency resulted in client having support in the
community.
• Attendance at the money management session means he now understands the
importance of paying priority bills and the consequences of not paying them. He
is also saving for his son’s future.
• Legal help has resulted in client winning custody of his son.
Both of these examples illustrate two key points:
• A number of clients who come to Plymouth CAB have multiple needs and
require multiple services;
• Embedding financial capability within the debt work at the bureau means
that it is provided as part of a holistic package of support.
Names have been changed to ensure confidentiality
35
While the Money Active workshop that Jason attended would not have fulfilled all of
his needs, it provided him with knowledge that may prevent financial problems
occurring in the future. For example, being able to identify priority versus non-priority
debts will hopefully mean that Jason is able to distinguish between them going
forward and prioritise his spending accordingly.
Plymouth CAB followed up with a sample of service users who had participated in
the Money Active workshops over the telephone. The purpose of this follow up was
to talk about how they had found the workshop, what steps they had taken as a
result of the workshop and what information they had retained. Sharon and her
volunteers attempted to reach 20 participants and succeeded in getting responses
from 18, representing nearly 1/5 of participants for the evaluation period.
Participants were asked whether they felt their attitude to money had changed
since attending the workshop to which 100% of respondents reported yes. 83% said
their attitude towards budgeting had changed and the remaining 17% said their
attitude towards debt and credit had changed.
Respondents were then asked what positive steps they had taken since attending
the MoneyActive workshop. The majority of respondents said that they had started to
settle their existing debts since the workshop:
Positive steps taken
5%
5%
5%
Settling debts and paying bills
5%
Set up and following a budget
Saving
57%
Seeking further advice
Job interview
23%
Opened up basic bank account
When asked to give an example of a source of affordable credit 100% of
respondents were able to do so and 100% cited credit unions.
When asked to give an example of a priority bill or debt, 79% of respondents were
able to do so with the majority of respondents citing mortgage or rent:
36
Example of a priority bill
6%
17%
Mortgage or rent
Utility bills
55%
Council tax
Gas or electric bill
22%
‘Utility bills’ cannot be considered correct as some utility bills are a priority (gas and
electricity) while others are not (water).
Finally, respondents were asked if they had any money saving hints or tips that they
could share with others:
“Shop around for the best deal”
“Pop a few pounds into a pot each week”
“Shop wisely and freeze your leftovers”
“Change utility provider”
“Cut your own hair”
“Stick to essential spending”
“Make a shopping list and stick to it”
Again, what these findings show is that there is still an emphasis on paying back
debts but that it is in conjunction with wider financial capability goals. Once again
budgeting featured heavily in responses and it is also encouraging that so many
respondents were able to give a correct answer to the questions on affordable
credit and priority debts. Going forward it might be useful to ask respondents these
questions before the start of the workshop and compare answers.
One of the volunteers helping to facilitate the Money Active sessions said the
workshop environment was far more powerful than ‘telling’ service users how to
manage money which is done a lot more in debt casework; by’ showing and
sharing’ the messages were sticking much better with service users. This is possibly
because the focus of the sessions were less on debt and more on overall financial
capability so clients were more relaxed.
37
Level 4: Results
The fourth level of evaluation is concerned with mapping the findings of the
evaluation back to the original aims and objectives of the intervention; that is,
whether the reactions of participants and their increased learning and behaviour are
achieving the desired results.
•
Increase financial capability: Plymouth CAB define financial capability as the
knowledge, skills and confidence to manage money effectively and take
responsibility for personal financial decisions. Participants reported in the
workshops that they were planning on tackling their existing debts and setting
up a budget. The follow up phone calls demonstrated that 57% had taken
steps to settling their existing debts and 23% had set up and were following a
budget. 100% were able to give an example of an affordable source of credit
and 79% were able to give an example of a priority bill.
•
Reduce repeat clients: Due to internal reporting structures there are no clear
numbers available yet to demonstrate that this model reduces repeat clients.
In order to gain this information it would be necessary to compare data from
the evaluation period to data from before the evaluation period and control
for external factors. Unfortunately that was not possible on this occasion.
Anecdotally, however, there is some evidence to suggest that embedding
financial capability within debt advice does reduce repeat visits. The case
study on Jason, for example, demonstrates how taking part in the Money
Active sessions can potentially reduce the likelihood of making the same
financial mistakes again. While changes in learning do not always equate to
changes in behaviour, there is an argument that service users are more likely
to take positive steps if armed with the necessary knowledge.
5.2 Evaluation of the Model
The evaluation of the model relates to the third aim of the Debt Gateway and
Money Active programme of exploring implementation. Specifically, this section
seeks to answer the question of what have we learned about embedding financial
capability within debt advice.
Plymouth CAB found that the main issue in implementation of the model related to
attendance at Money Active workshops. Specifically, while a number of service users
were referred to a workshop, not every one showed up. This is problematic for a
number of reasons: it is demoralising for the facilitator of the workshop and it means
that the service user misses out. As is demonstrated with the data above, those that
did attend a workshop reported significant learning and a number enjoyed the
group dynamic. Staff at Plymouth CAB discussed the reasons for the lack of
attendance and work out some possible solutions:
•
Who was running the Gateway service was crucial as at the beginning there
were a number of different people carrying out this function and not all of
them considered referrals to the Money Active workshops to be a priority.
Others found it difficult to persuade the service user to take action which
impacted on how well the service was promoted. To combat this, bureau
staff have decided to give one person the responsibility of running the
38
Gateway who is better briefed on the importance of the Money Active
workshops.
•
The information provided to the service user before the session also made a
difference. Many service users needed a lot of reassurance about what the
session would entail and how it would run. Some were happy about the
group set up whereas others were deterred by it. To combat this, the
Gateway advisor will be encouraged to spend some additional time talking
to service users, persuading them that the workshops are non-threatening
and will actually better prepare them for their debt interview. Service users
will also be encouraged to bring along a relative or friend if that helps them
feel more comfortable. Plymouth CAB are also producing a hand out on
what to expect from the workshop and have adjusted their advertising
leaflets accordingly.
•
The time of year, day of the week and time of day that the workshops were
held was also found to be important. Mondays and Fridays did not work as
well as mid-week. More service users were able to attend if times were fitted
in with school schedules and avoided school holidays. As a result, the bureau
has adjusted the times that they hold the Money Active workshops.
•
Debt problems resolved in the debt gateway often meant that service users
felt that they had no need to return for the Money Active workshop. To
combat this, the bureau will encourage the Gateway advisor to recognise
which service users need more encouragement: ‘although we have now
helped you to resolve your debt situation, it is very important to attend one
of our money management sessions so that you don’t have to come back to
see us again’.
•
Those who said they would come but did not turn up. This is a difficult
challenge as it is not possible to anticipate who is unlikely to show up on the
day. To resolve this issue, staff/volunteers at the bureau telephone or send a
text to service users the day before the workshop to remind them to attend.
Those that do not attend now receive a phone call to ask why they had not
attended. Many service users had good reasons, especially those with
chaotic lifestyles, but a few did not show up because they were anxious
about the workshop set up and unsure of what it would entail. These clients
were offered one-to-one sessions, 50% of whom turned up.
•
Some information was clearly not needed. Initially Money Active workers tried
to include all information that they thought was relevant in the workshops but
later decided that a lot of information was not necessary. For example, a
number of their service users live in social housing so do not need information
about mortgages. Topics that are of most interest include understanding the
cost of credit, budgeting, where to seek advice and help and how to
differentiate between priority and non-priority bills and the consequences of
not paying. As a result, Plymouth CAB have restructured the sessions to
include only the most relevant information.
There is one final point that is perhaps the most significant in encouraging service
users to attend a Money Active workshop and that is the realisation that attendance
39
at a workshop means that any follow up debt case work is significantly shortened. Bill
said that, on average, if a service user had attended a workshop, their debt one-toone took one third of the time it took to see other debt clients. A key reason for this is
that the Money Active sessions cover an exercise that is a common feature in debt
case work: the financial statement. By completing their financial statement in the
Money Active workshop, service users had already worked out their incomings,
outgoings and subsequently the money they had available to make debt
repayments. The acknowledgement that Money Active sessions save time in debt
case work ended up working in the bureau’s favour as regards incentivising service
users to attend the workshops. Service users who were reluctant to attend but keen
to get their debts sorted are informed that attending a workshop will ensure they are
seen by a debt advisor much faster. It works as an incentive but it is also factual:
debt case work sessions that are one third of the length of regular sessions are a lot
easier to fit into busy schedules.
5.3 Evaluation of the Evaluation
The type of evaluation conducted through RBS Innovate differs from other types of
evaluation. Halfway between an internal evaluation and an external evaluation, this
method provided a number of benefits as well as drawbacks. Some of the learning is
outlined below:
What worked well?
•
Having a balance between internal and external: Working with Gill, Jane,
Sharon and Bill meant that the original aims and objectives for the evaluation
of the Debt Gateway and Money Active came from them and were
grounded in their work. The evaluation framework was developed in
partnership with Plymouth CAB and the external evaluator and all major
decisions were made jointly which meant that everyone was on board with
the methodology. Balanced alongside this is the fact that the evaluation
findings were evaluated externally so were viewed from a different
perspective. This balance between internal and external provided a variety of
different information that would be difficult to obtain using just one
perspective.
•
Follow up phone calls: Plymouth CAB had not previously followed up with
Money Active clients before this evaluation and in fact there were two
benefits to this evaluation technique. The first is that it enabled service users to
reflect on the workshop and think about their progress. The second is that it
was a highly motivational exercise for staff. As mentioned above, the
sometimes poor attendance at Money Active workshops has been
demoralising for staff; hearing positive feedback and seeing how far some
service users have come since the session was highly motivational.
What did not work so well?
•
Paper-based evaluation tools: A number of the service users that come to the
Money Active sessions have literacy issues as well as other needs. This means
that Plymouth CAB will potentially want to explore other ways of evaluating
the workshops that are not reliant on paper-based tools in the future. As some
40
participants need help filling in the forms it means that the evaluation took up
a disproportionate amount of time and this can take away from the time
needed to deliver the workshop. One way to combat this could be to build
evaluation activities into the course curriculum. For example, knowledge at
the start of the course could be captured using an interactive group quiz that
could then be run again at the end of the course. Data could be captured in
other ways, for example via voice recorder that could be listened to and
transcribed later. There are positives and negatives associated with different
methods but they are worth exploring.
Part 6 – Conclusions and recommendations
There are a number of conclusions that can be drawn from the evaluation of the
Debt Gateway and Money Active at Plymouth CAB and recommendations for both
the organisation and the sector going forwards.
1. Financial education can save organisations time: Attendance at a Money
Active session meant that the time needed in subsequent debt one to ones
was reduced by two thirds. Clients who had been through a Money Active
session had already worked out their budget, completed a financial
statement and were aware of some of the options available to them.
Because this saved the case workers time, they were able to slot Money
Active clients into shorter appointments that they would not be able to
otherwise; in other words, they were able to see them sooner. Therefore, being
seen by a case worker faster became a powerful incentive for clients to
attend Money Active workshops.
2. Group settings are an effective way to deliver financial education: There is
sometimes a perception that group settings inhibit participants from
contributing and sharing but the opposite has been found through this
evaluation. A number of participants cited the group dynamic as their
favourite aspect of the course and said it actually encouraged them to
participate. A key reason for this was that it enabled participants to realise
that they are not alone and that others struggle with the same money worries.
Knowing they are not alone and learning from one another’s experience can
provide the confidence boost necessary to make a change. Organisations
should not shy away from the group delivery model with financial capability
work; it can be very effective.
3. Financial capability may just be one part of a holistic service: The results of the
evaluation show that debt is still very much at the forefront of any activity
resulting from the Money Active sessions. This suggests that financial capability
for these clients is still practiced in relation to debt but this is not necessarily a
negative finding. Most financial inclusion and financial capability interventions
find that it is important to make money matters relevant to participants.
Sessions will often be tailored to the needs of a particular client group so that
the information resonates with them and encourages them to take positive
action. Learning financial capability in relation to debt means that clients can
see the relevance of it and how it can help them to overcome existing
financial worries and prevent them in the future.
41
4. Evaluation activity can be challenging with particular client groups: Delivering
money management workshops to particular service users can be
challenging enough without also having to distribute evaluation forms.
Literacy problems, mental health issues and lack of confidence mean that
some clients need a lot of support with the actual workshop activities, let
alone filling in questionnaires at the end of it. At the same time, Plymouth CAB
are keen to see the impact of their services recorded and assessed so that
they can continue to make improvements going forward. Because of this they
will be looking at additional ways that they can evaluate their services going
forward.
5. It is important to follow up with participants: Following up with participants
after a workshop is important for a number of reasons. It helps to demonstrate
the long term impact that Money Active has had on service users – impact
that is not always immediately obvious. It also provides service users with the
opportunity to reflect on what they have learned and revisit the goals that
they set in the workshop, if they have not already done so. Finally, following up
with clients is good for staff and volunteer morale. Low turn out at workshops
can mean that facilitators question how much of an impact they are having
on service users. Staff and volunteers at Plymouth CAB found that following up
with service users helped them to see that the Money Active workshops have
made an impact and that the majority of participants found the sessions
extremely useful.
There are not currently many examples of organisations successfully embedding
financial capability within their wider debt advice work. The sometimes inflexible
structures of debt advice and the high targets required by funders often mean that
case workers are under pressure to support their clients with existing debts with no
time to think about their wider financial capability. A key strength of the model at
Plymouth CAB is that it actually saves staff and volunteers time in the long run. By
requiring all debt clients to go through the Debt Gateway the advisor is able to
support a proportion of clients then and there while referring others with specialist
needs to specific advisors. This means that case workers do not spend time with
clients who only require information or signposting. Referring clients who could
benefit from further financial capability training to a Money Active session saves case
workers even more time. These clients come to the session prepared with their
financial statements and information on their situation and possible options. While
additional resource is needed to run the Debt Gateway, this model clearly
demonstrates how it can pay off in the long run.
42
Part 7 – Appendices
Appendix A: Plymouth CAB – Staff interview topic guide
Jane Guy and Ann Kinahan (12.30pm)
• In general, how do you feel the debt gateway and the money management sessions
have gone?
• Can you send me copies of the evaluation forms from the money management
sessions?
• Were participants assigned a unique number as we discussed with Gill at the
beginning?
• What problems have you had around engagement? (I.e. getting people to attend)
How have you combated this?
• In your e-mail at the beginning of October you said you were going to have a team
meeting to talk about potential methods to improve attendance of the money
management sessions – how did that go?
• What would you recommend to boost engagement going forward?
• Do you have any more case studies?
• What worked really well?
• What’s not worked so well?
• What would you recommend changing or keeping going forward?
Bill (2pm)
• In general, how do you feel it’s gone?
• When did the debt gateway start? Was in November 2009?
• What are the criteria for deciding who gets referred to a money management
session?
• How many people were referred to a money management between May and August?
• On average, what percentage of clients do you refer to a money management
session?
• How many people were referred to a one to one from the money management
session?
• Have you noticed a difference in the amount of people who have gone through a
money management session coming back in versus those who have not gone
through a session?
• What’s worked really well?
• What’s not worked so well?
• What would you recommend changing or keeping going forward?
Sharon (3pm)
• In general, how do you feel it’s gone?
• How did the follow up phone calls go?
• How many people did you reach?
• What/where are the findings?
• Can I have copies of the responses?
• What problems have you had around engagement? (I.e. getting people to attend)
How have you combated this?
• What would you recommend to boost engagement going forward?
• How have you found running the sessions?
• There was feedback in the summer that people preferred the money management
sessions to the one to one’s – did that remain consistent?
• What’s worked really well?
• What’s not worked so well?
• What would you recommend changing or keeping going forward?
43
References
i
Plymouth Citizens Advice Bureau: Annual Review 2008/2009
ii
Citizens Advice press release http://www.citizensadvice.org.uk/press_20101215
iii
FSA (2009) ‘Financial capability and wellbeing: Evidence from the BHPS’ p. 4
iv
FSA (2003) ‘Towards a national strategy for financial capability’ p. 3
v
Bob Widdowson (2009) ‘Citizens Advice Financial Inclusion Fund Financial Capability
Evaluation Report’
ECOTEC (2006) ‘Evaluation of the Citizen’s Advice National Financial Capability Project: A
vi
full report to Citizens Advice and Prudential PLC’
vii
Transact web-site, ‘Evaluation Case Studies’, www.transact.org.uk
viii
http://www.providentpersonalcredit.com/
ix
Doing Research in the Real World, p. 279
x
Doing Research in the Real World, pp. 285 – 286
xi
Evaluation Theory, Models and Applications p. 198
xii
Bryman, Triangulation paper
xiii
Kalton, G., Roberts, J., and Holt, D. (1980) ‘The Effects of Offering A Middle Response Option
with Opinion Questions’, The Statistician, Vol. 29, No 1 http://www.jstor.org/pss/2987495
44