How to Assess the Effectiveness of
Your Language Access Program
Robin Ghertner
May 23, 2017
National Center for State Courts
WHY ASSESS YOUR PROGRAM?
Why do we assess any program?
Attract or
Maintain
Funding
Identify
Gaps in
Services
Improve
Outcomes
Increase
Efficiency
Compliance
Assess
Fidelity to
Model
Stakeholder
Involvement
Why Assess Language Access?
• Compliance with federal, state and local
mandates
• Attract funding or protect against cuts
• Ensure services are delivered effectively
PROGRAM MATURITY
Is this a good model of program
maturity…
Performance
measures
Developing
program
New
program
Process
evaluation
Mature
program
Impact
evaluation
…or is this?
Creating
Adapting
Developing
Becoming
established
Role of Assessment
Creating
Adapting
Developing
Becoming
established
Language Access Logic Model
•
•
•
•
•
•
Inputs
Activities
Outputs
Funds
Volunteers
Staff
Contractors
Clients
Attorneys
• Translation
• Interpretation
• Documents
translated
• Clients served
• Cases
interpreted
• Languages
served
• Courts served
• Days served
Immediate
Outcomes
• Clients
understand
communicati
on
• Clients and
Court able to
successfully
communicate
Long Term
Outcomes
• Fair and
equitable
treatment
Language Access as
Part of Larger Logic Model
Inputs
Activities
Immediate
Outcomes
Outputs
Inputs
•
•
•
•
•
•
Funds
Volunteers
Staff
Contractors
Clients
Attorneys
Activities
• Translation
• Interpretation
Outputs
• Documents
translated
• Clients served
• Cases
interpreted
• Languages
served
• Courts served
• Days served
Long Term
Outcomes
Immediate
Outcomes
• Clients
understand
communication
• Clients and
Court able to
successfully
communicate
Long Term
Outcomes
• Fair and
equitable
treatment
Components of Assessment
Performance
Measurement
Program
Evaluation
Process
evaluation
Impact
evaluation
Performance Measurement vs
Program Evaluation
Performance Measurement
Program Evaluation
Ongoing
Monitoring and reporting
Typically focus on outputs
Compare measures against a
standard
• Addresses “how are we doing”
• Not designed to address
causality
• “One time”
• Assess how well program
working
• Typically focus on processes or
outcomes
• Answer specific questions
• Addresses “does this work” or
“how do we make it better”
• Typically more rigorous – and
expensive
•
•
•
•
Role of Assessment
Performance measurement
Creating
Process evaluation
Adapting
Impact
evaluation
Developing
Becoming
established
OK! Let’s Do This!
• Erica is a the new
language access
coordinator for a county
court.
• She wants to set herself
up for success and
understand how her
program is working.
• Let’s walk her through
one approach.
PERFORMANCE MEASUREMENT
How to Measure your Program
1. Select appropriate measures
2. Collect data on your measures
3. Set standards
4. Use measures to inform decisions
Step 1. Select Measures
What do you want to measure?
• INPUTS. Easiest to measure but less useful
• OUTPUTS. Direct measure but may not mean
goal is reached
• OUTCOMES. Gets at goals but hardest to
measure
Depth
How much
service is
provided
Breadth
Range of ways
service is
provided
Category
Depth
Breadth
Depth and Breadth
•
•
Input
•
Number of staff,
contractors, and
•
volunteers engaged
Number of staff
hours used
•
Languages spoken by
staff, contractors, and
volunteers
•
•
•
•
Output
•
•
Number of clients
served (relative to?)
•
Number of hours
interpreted
Number of
documents
translated
Amount of financial
expenditures
Pay differentials for bilingual
employees
Use of
translation/interpretation
technologies.
Number of relationships with
stakeholders
Number of languages
translated/interpreted
Different ways services
provided (e.g. in-person
interpretation versus
telephonic or videoremote interpretation
versus translation)
•
•
Outcome
•
•
Outcomes for LEP clients
Number of client complaints
received
LEP client satisfaction
Equitable treatment
Erica’s Measures
1. Dollars spent per case.
2. Proportion of cases needing interpretation
that receive it.
3. Proportion of cases needing interpretation, in
each language.
4. Client satisfaction with services.
5. Number of complaints received per number
of LEP cases
Step 2. Collect Data
• How do you measure?
– Existing program records
– New tracking tools
– Surveys/questionnaire
• How often do you measure?
– Balance frequency with burden
Erica’s Data Collection
Measure
Collection
Frequency
Dollars spent per case
Budget data
Weekly
Number of cases
interpreted
New system to log cases
Weekly
Number of cases in
each language
New system to log cases
Weekly
Client satisfaction
Survey questionnaire
At end of each case
Complaints received
Complaint hotline and
receptionist tracking form
Weekly
Step 3. Define Standards
• What are your benchmarks?
– Inadequate, Below Expectations, Poor
– Adequate, Meets Expectations, Good
– Proficient, Exceeds Expectations, Excellent
• Use historical data to understand measures
• Consult with stakeholders
• Validate the standards!
Erica’s Standards
• Spends 2 months collecting regular data.
• Looks at trends in data to understand current
performance.
• Shares trend data with her manager, other county
courts, and local advocacy organization.
• Sets initial levels for “Meets Expectations” and
“Exceeds Expectations”
• Monitors measures for a few months to validate
those measures.
Erica’s Standards
Measure
Collection Method
Frequency
Standard
Dollars spent per Budget data
case
Weekly
$200 – meets
$150 – exceeds
Proportion of
cases
interpreted
New system to log
cases
Weekly
75% - meets
90% - exceeds
Proportion of
cases in each
language
New system to log
cases
Weekly
75% - meets
90% - exceeds
Client
satisfaction
Survey questionnaire
At end of each case
80% satisfied – meets
80% very satisfied exceeds
Complaints
received
Complaint hotline and
receptionist tracking
form
Weekly
25% - meets
10% - exceeds
Step 4. Use measures to inform decisions
• Trends over time
• Comparison across
sites
• Comparison to
benchmark
• Do measures
reflect
accurately?
• Why?
• What does it
mean?
Analyze Data
Interpret
results
Measure
adjustments
Program
adjustments
• How can we
improve?
Erica uses her measures
16
14
Complaints Received
12
10
Site A
8
Site B
Meets expectations
Site C
6
4
Exceeds expectations
2
0
March
April
May
June
July
August September
Be careful with your interpretation!
• Did Site C really improve that much?
– Maybe measurement improved
– Maybe they saw they were being assessed on the
measure and put all their emphasis there
– Maybe county applied more resources there
– Maybe volunteers chipped in
– Maybe ESL classes started
– Maybe the LEP population changed
• Or maybe it did actually improve!
– What did they do differently?
PROGRAM EVALUATION
Types of Program Evaluations
Process Evaluation
• Operating as
intended
• Identifying
improvements or
efficiencies
• Stakeholder
involvement
Impact Evaluation
• Effect of program
out outcomes (in
reference to
counterfactual)
Process Evaluation
• Often in media res – While program is operating
• Objectives:
– Does our program run how we expect it to?
– What are stakeholder perspectives, and are they
involved as they should be?
– Are there improvements we can make?
• Methods: qualitative and quantitative.
– Focus groups, interviews
– Surveys
– Program data
Impact Evaluation
• Often at a point when program is established
and need to assess effectiveness
• Objectives:
– Did our program meet its outcome goals?
– Relative to some baseline, how did our program
impact recipients?
• Methods: quantitative.
– Surveys and program data
– Statistical analysis
Impact Evaluation
• Degrees of rigor!
– Randomized control trials
– Quasi-experimental designs
– Regression modeling
• Often done by outside evaluators
– Independent and objective
Example Impact Evaluation Designs
• Pre/Post
• Pre/Post with Control or Comparison Group
• Interrupted Time Series
• Discontinuity Design
Are you ready to evaluate?
• What is your program maturity?
• Do you have the internal capacity or funds?
• Do you know how you will you use the
results?
• Are you committed?
What type of evaluation should you
do?
• WHAT do you want to know?
• WHO needs to know it?
• HOW much resources do you have?
• WHEN do you need to have answers?
• Do you need EXTERNAL VALIDATION and
INDEPENDENCE?
Where to begin?
• Identify what your evaluation questions are
• Identify broad approach (process vs impact)
• Identify evaluator
• Get stakeholder buy-in
• Plan for how you will disseminate and use results
Questions?
Robin Ghertner
[email protected]
917-613-8442