Data Tools for Evaluating,
Presenting and Measuring
Rapid Re-Housing Performance
Jamie Taylor, Ph.D.
Mark Johnston
The Cloudburst Group
Learning Objectives
I.
Rapid Re-housing Evaluation Approach –
Using HIMS data and matched comparison
groups
II.
RRH Data Dashboard - Shared measurement
tool to support collective approach to datadriven monitoring and evaluation
III.
Establishing Local “HomeStat” cross-sector
workgroup structures; presenting and analyzing data
for continuous quality improvement of RRH service
system
HMIS data = Gold Mine


A great deal of information is
currently available for targeting
local RRH resources and system
improvement decisions in HMIS
“gold mines”.
System-building skills are needed
Gold-mining Team!
to illuminate HMIS data and to
guide decision-maker questions
and solutions  Mining HMIS data is necessary to
determine which homelessness
programs are most effective in
preventing and ending homelessness?
RRH Evaluation - Necessary for Social
Investment into Rapid Re-housing
Investors seek rigorous evaluation methods / mathcing
RRH Evaluation: What would have
happened without RRH?
Matched groups help answer the counterfactual, what would
have happened without a given intervention. Comparison
groups serve as an estimate for what would have happened
without RRH.
With matched groups, the
difference in outcomes
between treated (RRH)
and control (non-RRH)
groups can estimate
true RRH impact.
Answering the counterfactual is critical.
One reason: Very low-income families move often

Among extremely low-income families with
children, 43 percent moved into their current
homes within past two years. (State of Nation’s Housing
2014)

40% of all renters with incomes below poverty had
moved in the past year. Households with higher
income levels do not move as often as very low
income households. (American Housing Survey 2013).
How do we separate the effects RRH assistance has
on housing movement into homelessness from usual
housing patterns for very low income households?
RRH Evaluation Requires Seeing
Process and Impact Outcomes

How do we assess what is happening
inside local RRH systems
− by
the aggregate and
− by individual RRH program effects?
Build RRH Data Dashboards
for Data-driven Planning
1) HMIS data is made visible
2) Allows for stakeholder assessment of current RRH
services
3) Drives the development of system-level goals
4) Helps determine what strategies and improvements
are necessary to achieve RRH system aims
5) Focuses decision-makers on RRH resources that
effectively prevent and end homelessness
IOWA HomeStat
Rapid Re-Housing Report
Last Update: February 20, 2015
HomeStat Presenters:
Iowa Homeless Service System Map
Homeless Service System (Rapid Crisis Response)
State of Iowa 2013 - 2014
Iowa RRH Providers and RRH Households
Map of RRH Providers
RRH Program Enrollment 2013 & 2014
Two Year Total RRH households:
Iowa RRH Households Enrolled
329 by Quarter
265
120
11
145
141
32
177
207
Aggregate RRH enrollments in
2014 a 26% increase over 2013.
2014
What might account for the
fluctuation of overall
enrollments in each quarter?
% of All 2013 - 2014 RRH Households
Enrolled in RRH by Provider
35.6
21.9
4.1 3.4 4.1 2.8 5.5 3.7 3.7 4.9
8.4
1.5 0.5
What might be an overall goal
for increasing RRH enrollment in
2016?
What enrollment goals are set
by individual providers?
RRH Household Characteristics 2013 & 2014
Single Parents accounted for
almost 1/4th of all RRH
households.
• 41% of all RRH households were Black
• 50% of all RRH households were White
Of all RRH households with a
disabling condition, 46% had a
mental health condition, 20%
had a physically disabling
condition and 24% had both
mental health and physical
disabilities.
Why is it important to know how
many TANF families receive RRH
assistance?
How do frontline providers
define “disabling condition” ?
RRH Return to Homelessness 2013 & 2014
The majority of RRH returned
households had returned
within 6 months (71%).
12% of 2014 households had
not yet exited RRH.
What is expected goal for %
return to homelessness?
Percent of Return to Homelessness
RRH Households by Provider
48%
34%
5%
2%
What might be affecting the
variation seen in Return to
Homelessness rates across
providers?
How are these Return results
influenced by entry dates into
RRH?
Rate of Return to Homelessness by Program
Rate of Return to Homelessness by
Program in 2013 & 2014
Rapid Re-Housing
PSH
Transitional Housing
Emergency Shelter
10%
Return to homelessness data
tracked by program entry.
All entries occurred in 2013
or 2014.
HUD System Performance
Measures directly impacted
by RRH effectiveness:
1. Length of time persons
remain homeless;
2. Extent to which persons who
exit homelessness to permanent
housing destinations return to
homelessness
9%
10%
19%
Are RRH and TH serving
similar populations? What is
average cost of TH vs. RRH
households?
How can targeting of RRH vs.
TH resources be evaluated?
RRH System Assessment - Core Components
Data Quality Issue: Length of time in
homelessness
Number Days
Homeless before
RRH
0 Days
HMIS data
showing % of all
RRH households
76%
Length of time in homelessness
data could not be used in RRH
analysis.
It is suspected that most
households with 0 days came
from RRH outreach to
households at imminent risk of
homelessness.
< 30 Days
7%
30 – 89 Days
4%
90 – 179 Days
2%
6 Mo. – 1 Yr.
4%
Why is this data element critical
to analysis of RRH impact?
1 Yr. – 2 Yrs.
4%
> 2 Yrs
2%
How can data quality for “Length
of time in homelessness” be
improved?
Data Quality Issue: Monthly Income
Monthly Household income
Monthly income at entry could
not be used in RRH analysis.
$2,500 / month
In chart, all blue dots above
$2,500 level represent
households with income
>$2,500, a serious data issue.
Why is the income data element
so critical to analysis of RRH
impact?
How can data quality for
household monthly income at
entry be improved?
How can overall RRH data
quality checks be instituted for
frontline staff?
RRH Comparative Impact Analysis
RRH Impact Evaluation:
Iowa’s 2011 – 2012 HMIS data was
analyzed using a statistical matching
method*. The risk of return to
homelessness for RRH households was
compared to similar households who did
not receive RRH.
Return to Homelessness
Results
11.7%
7.1%
RRH Households
Non-RRH Households
*Propensity score match creates balanced comparison groups, ensuring household HMIS
characteristics are similar in both groups to effectively estimate treatment effects of RRH.
All households enrolled in RRH
2011 / 2012 were compared to
similar households who did not
receive RRH from same years.
Return to homelessness analyzed 10/2014
What would have happened
to RRH households if they
had not received RRH?
Rapid Re-housing significantly
decreases the likelihood of a
return to homelessness.
Why do Social Investors require
rigorous matching evaluation
results?
How can we use this evidence
for expanded RRH funding?
Improvement Strategies - RRH Funding
ESG
5%
20%
Iowa
RRH
Funding
Sources
5%
2013-2014
10%
0
10%
CDBG
TANF
These funders and percentages
are hypothetical for Iowa. Actual
RRH funding sources are
currently being calculated for
each RRH program.
TBRA-HOME
50%
Foundations
State RRH
City RRH
Does RRH Evaluation Impact
Results lead to need for
additional RRH funding?
What overall goal can be set for
increasing RRH funding sources?
What RRH funding goals can be
set by individual funders?
How can funding goals be
monitored on RRH data
dashboard?
Data Dashboard supports
Collective Impact Approach
Source: F. Hanleybrown, J. Karmer (2012) Channeling Change: Making
Collective Impact Work. Stanford Social Innovation Review
Collective Impact requires using data
to take ownership / make decisions

RRH Evaluation and Program Monitoring
Data Dashboards help drive system change decisions to:
−
House more people
−
Improve outcomes for people served
−
Maximize use of existing funding
−
Attract additional funding with performance outcomes
(Federal, State, local)
−
Unify local stakeholders around neutral evidence
Collective Impact National Example:
HUDStat
HUDStat encompasses a coordinated, integrated leadership team
meeting quarterly to analyze data on shared performance goals;
assessing progress and making cross-system course corrections to
achieve shared aims.
Collective Impact Local Example:
Iowa’s HomeStat
Steps to establish local HomeStat:
1.
Form cross-sector decision-making group (RRH
Providers, Funders, HMIS, Governments, NonProfits,
homeless)
2.
Identify shared agenda, set shared goals to accomplish
3.
Use data to analyze and prepare agenda, create data
dashboard (i.e. Iowa)
4.
HomeStat group meets regularly to present, listen,
discuss and decide issues
5.
Leave with clear direction on next steps
Collective Impact: Use Data
Dashboard to Set RRH Goals





NEED to formulate clear and compelling goals for RRH
with cross-sector leadership / decision-making group
HomeStat Data dashboard process links RRH program
outcomes to governance and action for RRH
improvement goals
HUD System Performance Goal: To decrease annual
return to homelessness rates
Analyze and compare local or state system outcomes to
national return to homeless rates
Compare costs of local RRH to local TH projects
Sample Goal: Expand RRH services to
additional 500 households/year
With HomeStat, Stakeholders can decide whether to:





Identify specific areas needing more information
Assign follow-up tasks, i.e. assessing need to improve
HMIS data quality
Assign staff to identify other possible sources of RRH
funding. i.e. TANF, HOME, etc
Identify topic(s) to be taken up at next session, i.e. assess
targeting of TH and compare TH to RRH clients
Determine why some RRH programs have much higher
return to homelessness rates that others
RRH Return to Homelessness 2013 & 2014
The majority of RRH returned
households had returned
within 6 months (71%).
12% of 2014 households had
not yet exited RRH.
What is expected goal for %
return to homelessness?
Percent of Return to Homelessness
RRH Households by Provider
48%
34%
5%
2%
What might be affecting the
variation seen in Return to
Homelessness rates across
providers?
How are these Return results
influenced by entry dates into
RRH?
Collective Impact Decision-making
What HomeStat Is…a culture of evidence, building
capacities to respond to data, not just report to HUD

Collective knowledge framing to address issues early-on

Forum to shhare local successes and best practices

Established relentless focus on data quality and problemsolving
What HomeStat is NOT.…

A “show-and-tell” that avoids the challenging issue

Focused ONLY on what is not working

A process that lasts only one or two sessions
WouldData Dashboards and HomeStat
improve your system planning?
Thank you! Contact us for additional
guidance on Local RRH Evaluation
Jamie and Mark
Cloudburst Consulting Group
Contact Jamie Taylor
[email protected]
Phone # 860-716-7392