Kathy Moore, MSW
Family Violence Prevention
Services Act (FVPSA)
Reporting Requirements
Domestic Violence
Research & Evaluation Consultant
[email protected]
510.529.1514 (cell)
May, 2008
Agenda
I. Training Objectives IV. FVPSA Required
Outcomes (*)
II. New FVPSA
V. Strategies for
Program Mandate
Collecting this
for Outcome
Evaluation
Information
from Survivors
III. FVPSA’s Data
VI. Reporting the
Collection Work
Data (*)
Group
I. Learning Objectives
By the end of this
workshop, participants will be able to:
• Understand new
FVPSA outcome
reporting
requirements (*)
• Implement
outcome evaluation
in DV programs
incorporating
victim safety,
confidentiality and
best (evaluation)
practices
• Report service
outcomes (*)
New FVPSA
Requirements
II. New FVPSA
Program Mandate for
Outcome Evaluation
• In 2005 the FVPSA Program was reviewed by
the federal Office of Management and
Budget (OMB).
• “Results were not adequately demonstrated.”
• FVPSA Program is now required to have
grantees collect outcome data.
1
So What is Outcome
Evaluation?
• Examining change that has occurred
as a result of a service being provided.
• A change in knowledge, attitude, skills,
behavior, expectation, emotional
status, or life circumstance due to the
service being provided (KABBS).
Outcome Evaluation
• How well did your program work?
• What was the impact of your efforts?
• Did you create the change you set out to
accomplish?
• Are individuals or communities better off
as a result of your program?
• And if so, how do you know?
FVPSA Program
Response
• FVPSA convened a national advisory group
III. FVPSA
’s Data
FVPSA’s
Collection Work Group
Current FVPSA Reports
•
•
•
•
•
Problems with
Current Reports:
Long list
Inconsistency
Lack of definitions
Lack of outcomes
What’s missing?
What they Need:
• Standard data with
clear definitions
• Ability to
adequately
describe work
accomplished thru
FVPSA $, report to
Congress, and to
the public
• Big picture
• Wanted the new requirement to be useful
to programs, and not to be too burdensome
• Advisory group consisted of coalition
directors, national resource centers, state
FVPSA administrators, local program
directors, and evaluation specialists.
Necessary vs.
“Nice to Know”
Key Elements:
• Total DV program
budget & amount
funded by FVPSA
• Unduplicated count
of people served
• Unmet request for
shelter
• Demographics
• Prevention efforts
• Hotline/crisis line
calls
• Community
education/training
events & attendees
• Outcomes
2
Prior DV Evaluation Efforts
• The National Resource Center on Domestic
Violence (NRCDV) had been facilitating
discussions among state coalition
directors, women of color activists, and
others to think critically about our work
• As a result, in 1998 NRCDV initiated the
“Documenting Our Work” project (DOW)
DOW Products
• Outcome evaluation surveys for local
programs to evaluate:
– Hotlines
– Counseling
– Support groups
– Support services and advocacy
– Shelter services
• Pilot-tested in programs across four
states and found to be useful
Why “Document Our Work?”
• To develop consensus on definitions,
goals, and outcomes of our work
• Individual funders are increasingly
requiring outcome evaluation
• Can use to strengthen and to inform
program practice, policy and research
• Can use to encourage accountability to
survivors and to their children
FVPSA Advisory Group
Consensus
• They identified two outcomes that:
– Are appropriate given the varied nature
of survivors’ contact with programs
– Have been empirically shown by
research to lead to long-term outcomes
of increased safety and well-being
FVPSA’s Two Required
Outcomes (*)
IV. FVPSA Required
Outcomes (*)
• As a result of
contact with the
domestic violence
program, 65% or
more of domestic
violence survivors
will have more:
– Strategies for
enhancing their
safety (*)
– Knowledge of
available
community
resources (*)
3
Additional Work Group
Recommendations
• Provide training and technical
assistance to states and programs
• Provide actual tools and databases
• Pilot the forms and databases across
several states to see if people find
them manageable and useful
Conclusions from Pilot
• Overall, survivors
found the forms
easy to fill out
• Programs found
the information
useful
• Overall, staff
found the process
relatively simple
• Some programs
want fewer
questions
DOW Forms Have in Common:
• Completed voluntarily • Overall satisfaction
with services
by survivors (*)
•
Outcomes
of the
• Basic demographics
service,
including
• Checklist of services
two new FVPSA
women may have
outcomes (*) :
wanted and what they
– I know more ways to
received
plan for my safety
• Respect and support
– I know more about
received
community resources
Changes Made Based
on Pilot
• Created “cheat
• Created a “menu” of
sheets” staff can
questions that programs can
use to remind them
use to create their own
how to gather the
surveys (see sample forms)
information
(“Inviting Clients to
• Databases have been created
Complete Program
in Access and Excel for
Evaluation Forms”)
those programs using the
entire DOW forms (sign-up
• Continuing to
sheet)
translate the forms
into languages
other than English
Data Collection:
Getting Started
V. Strategies for
Collecting the New
FVPSA Outcomes
• Getting staff buy- • Deciding:
– What questions to
in
• Deciding who on
staff will do what
ask
– How often to collect
data
– When to collect
– From whom
• Treating survivors
respectfully
4
Staff Buy-in
The Problem:
• Staff often don’t
understand why they
• Staff are already
have to collect the
overworked and
information they do,
tired of paperwork
or what happens to it
that feels
meaningless
• Staff often don’t see
the tabulated
information they DO
collect
Deciding Who on Staff
Will Do What
• In your packet is a form entitled:
“Creating a Plan with Staff for
Collecting Outcome Evaluation Data”
What Will be Used?
• FVPSA recommends using the sample
survey forms available in your packet
• If not, incorporate the two required
outcome questions into forms already
being used by your program (*)
• Important we have consistent
information to share with FVPSA
Administrators
Getting Staff Buy-in
• Share the findings
• Involve them in
with them often
understanding how the
information can be used
• Discuss with them
by the program
how to make program
changes based on
• Explain the new
the findings (i.e.,
requirement and have
Continuous Quality
them participate in
developing a protocol for Improvement, or
CQI)
gathering the
information
Data Collection Protocol
• Forms should be handy and visible to the
Staff who will hand them out to survivors
• Staff should understand when and how to
ask survivors to participate
• Supervision of this process, especially in
the beginning, is important
When Will Data be
Collected?
(#2 & 3)
• Do not collect data when survivors are in
crisis
• Allow enough time for change to occur
– You can’t expect change to occur, for
example, after a woman attends only one
support group
• But collect often enough that you don’t miss
those survivors who receive short-term
services
5
How Often Will Data
be Collected?
(#2 & 3)
• Depends on service:
– Close to exit for shelter residents
– Every 3-6 weeks for support groups and
counseling
– Support services is the most difficult to
determine because you often don’t know
when you’ve “finished.” Allow enough
time for change to occur (at least 2
contacts with an Advocate, at minimum)
From Whom Will Data
be Collected?
(#3 & 4)
• The good news: NOT EVERYONE (*)
• Important to SAMPLE clients (*)
Sampling Strategies
• The key to sampling is
that you must make
sure that the people
you include are as
much like (representative of) the whole
group of people who
receive your services
as possible:
(#4)
– Survivors from all
ages, races and
cultural groups, sexual
orientations, religious
preferences, and
abilities must be
included.
– Dissatisfied as well as
satisfied clients need
to be included.
How Often Throughout the Year
Will Data be Collected?
(#2)
• There are a number of options:
– The first (or second, or third…) week of
every month or quarter
– The first (or second, or third…) month of
every quarter
– All year long
• Whatever you pick, stay consistent
Sampling
(#3)
• Sampling is an
• It is used all the
accepted way of
time to gather
collecting information
information about
the American
from a part of a
group to represent
public (polls,
the views or
census, etc).
experiences of the
group as a whole.
Sample Size
(#2, 3 & 4)
• The number of survivors you collect information
from is not fixed, and depends in part on how big
your program is:
- If you serve hundreds every year, then
collecting information from 20-25% may be
enough, as long as the selection process is
consistent and unbiased.
- In general, the larger the number of survivors
you serve, the smaller the percentage you will
need. If you have 1,000 clients, sampling 10%
or 15% may be enough. If you have 50 clients,
sampling half of them would be better.
6
Sampling Recommendations
(#2, 3 & 4)
• Shelter residents
– Try to get all
residents to
complete
– Residents would
NOT complete
support services
forms
• Support Services &
Advocacy
– After at least 2
contacts with
advocate
– But as late in the
process as possible
• Support group /
Counseling
– Every 3-6 weeks
• Only if the survivor is not in crisis
• Stress that participation is voluntary
• Stress that you use feedback to improve
services
• Stress the forms are brief and they can
skip any questions they want
• Stress how their anonymity is protected
Protecting Survivor
Anonymity (#6)
Protecting Survivor
Anonymity (#6)
• This is CRITICAL
• Survivors need to know you are serious and
have taken steps to ensure anonymity
• Provide a locked box or sealed envelope for
them to return surveys
– If a small program, stress you only open the
box or envelope monthly or quarterly
Accessibility Concerns
Inviting Survivors to
Participate
• Provide either a pencil or a black or blue
pen for client to use to complete survey
• Provide a private space for survey
completion
• NEVER have service provider take the
completed survey back from client
• Verbally explain these things to survivors
(#4)
• The forms are available in English and Spanish,
with other languages being added (Hmong,
Korean, Vietnamese, Mandarin Chinese,
Traditional Chinese, Russian, Polish, Haitian
Creole, Kurdish and Bosnian)
• Discuss with staff how to include women who are
not able to complete written surveys (either due
to illiteracy, disability, or language)
VI. Reporting the Data (*)
• Surveys can be completed verbally, but NOT by
the staff member who delivered the service
7
The Two Outcomes to
Report to OES (*)
• As a result of
contact with the
domestic violence
program, 65% or
more of domestic
violence survivors
will have strategies
for enhancing their
safety.
• As a result of
contact with the
domestic violence
program, 65% or
more of survivors
will have knowledge
of available
community
resources.
The Survey Items
that Measure the
Two Outcomes (*)
• I know more ways
to plan for my
safety:
– Yes or No
What Else Should We Ask?
• At a minimum, you just • Can pick and choose
need to ask the 2
from FVPSA’s menu,
required outcomes (*)
use the sample
forms available, or
• FVPSA recommends
create your own
adding at least a few
other questions
important to your
program
– Yes or No
If Adding Items:
Try to keep the
survey short and
simple,
but do include
questions important
to your agency /
community:
• Don’t just ask about
what you currently
offer; also ask about
other services survivors
might need
• Getting staff input is
helpful, increases buy-in
• Getting input from a
survivor advisory board
is invaluable too!
OES Program Report:
Table 19 (Oct ’08)
The Surveys are In –
Now What?
• These outcomes
• Entering the data
are NOT meant to
– Identify more than
be used by FVPSA
one staff to do this
Administrators
• Demonstration
(OES) to make
– OES Program
funding decisions
Report DV
Assistance Program
(*)
• I know more about
community
resources:
FVPSA Outcomes
Table 19
# Yes
Responses
# Surveys Resource
Completed outcome
# Yes
Responses
Safety
outcome
Shelter
Support Services &
Advocacy
Support Groups
Counseling
Total
8
Sample Databases
• Access and Excel
databases are
available for the
DOW forms
• Instructions for
these databases
are also available
• Sign up if you’d like to
receive electronic
copies of these
databases and/or
participate in potential
follow-up teleconferences
• NOTE: These optional
databases do not
replace OES Program
Report requirement!
Manual, forms, instructions,
& a longer DVD outcome training
are available for
no charge at:
http://pubs.pcadv.net/FVPSA_Outcome/
At the login screen, type:
User name: outcomes
Password: outcomes
Kathy Moore, MSW
Domestic Violence
Research & Evaluation Consultant
Thank You!
[email protected]
510.529.1514 (cell)
We wish you the best of luck
and hope this information is
helpful to you and your program
9