Designing and Implementing a Data Quality
Assurance Program for your CoC’s HMIS
Michael Lindsay, ICF International
Natalie Matthews, Abt Associates Inc.
Learning Objectives
1. Understand all components of a Data Quality
Assurance Program and how this work fits into the
overall efforts of the CoC
2. Discuss the roles that CoCs, HMIS Leads, and
agencies play in implementing a Data Quality
Assurance Program
3. Learn from the challenges faced by other
communities implementing a Data Quality
Assurance Program
2
Session Overview/Agenda
• Purpose and Intent of a Data Quality Program
• Review each of the 4 components of a DQ Program
• Discuss roles, responsibilities and potential next
steps for your community
3
Definition of Data Quality
• Data quality refers to the reliability and
comprehensiveness of your community’s data (as
collected in HMIS)
– Do you have sufficient data to accurately reflect the
demographics, needs and outcomes of persons experiencing
homelessness?
• Components of data quality include
–
–
–
–
Timeliness
Completeness
Accuracy
Consistency
4
Current Requirements for DQ
• Section 4.2.2 of the HMIS Technical Standards (2004)
“PPI collected by a CHO must be relevant to the purpose
for which it is to be used. To the extent necessary for
those purposes, PPI should be accurate, complete and
timely.”
5
Current Reporting on DQ
6
Proposed and Forthcoming Guidance
• Section 580.37 of the HMIS Proposed Rule (2011)
“..HMIS Leads must set data quality benchmarks for
CHOs, including bed coverage rates and service-volume
coverage rates.”
7
Data Quality Plans
• In anticipation of the HMIS Final Rule, and in
response to NOFA scoring criteria for the CoC
Program, many CoCs have created data quality plans
• Plans often consist of
– Baseline expectations for completeness, timeliness
– Monitoring protocols for reviewing accuracy
8
So why a DQ Program?
9
Elements of Data Quality Program
1.
2.
3.
4.
CoC HMIS Data Quality Plan
Enforceable agreements
Monitoring and reporting
Compliance processes
10
Connection to CoC’s Overall Efforts
Data Quality Program
1. CoC commitment to
improving DQ
2. CoC’s HMIS DQ Plan &
Enforceable Agreements
3. Monitoring &
Reporting
Compliance Processes
4. CoC, Agency and HMIS
Leadership Efforts
5. HMIS Lead’s
administration of HMIS
11
Preparing for the DQ Program
12
Identifying Your Baseline
• Important to take stock of where you are now
– Do you know how many of the homeless assistance and
homelessness prevention projects in your CoC, are actively
participating in HMIS? Baseline for bed coverage
– Have you recently run data completeness reports for your
full HMIS implementation? Baseline data completeness
– When CoC leaders, project staff and HMIS Lead staff
review reports, does the data seem accurate? Baseline for
accuracy
13
Step 1: Ensure CoC’s Commitment
• Important to clarify up front what the expectations
are for the data quality program
– CoC will need to review and approve the DQ Plan
– CoC should also be heavily involved in determining
expectations for monitoring and compliance
• This work cannot and should not fall just on the
shoulders of the HMIS Lead Agency
14
Key Considerations in Step 1
• How will the CoC enforce expectations for data
quality?
• Will these expectations extend to all homeless
assistance and homeless prevention programs in the
community?
• How frequently will the CoC leadership review data
quality reports and data analysis?
15
Step 2: Data Quality Plan & Enforceable
Agreements
• DQ Plan should be focused on
– Defining data quality expectations, by data element and by
program type
•
•
•
•
Completeness
Timeliness
Accuracy
Consistency
– Outlining how data quality will be monitored
• Who will monitor and when
• Who will results be reported to
16
Step 2: Data Quality Plan & Enforceable
Agreements
• Enforceable agreements are critical
• Need to be completed by all agencies participating in
HMIS
• Should provide guidance on what the consequences
are for failure to meet the standards in the DQ Plan
• Identify the process for notification of failure to meet
a standard
• Lay out the responsibilities of BOTH the HMIS
participating agency and the HMIS Lead and CoC
17
Key Considerations in Step 2
• Are the expectations and responsibilities reasonable?
• Have they been discussed in a public forum, to allow
for feedback and to generate buy-in from the CoC?
• How far back do you need to go in terms of data
quality improvements? Are you looking at “old”
data? How does poor data quality impact your
reporting efforts?
18
Step 3: Monitoring, Reporting & Compliance
Processes
• Once the HMIS Data Quality Plan has been reviewed
and approved by the CoC and agreements are in
place, it’s time to get out there and implement
• Will need to train/communicate to agencies and
users first, to ensure that all users understand the
expectations
• Encourage the CoC to allow for a grace period
• Transparency with results is key
19
Key Considerations in Step 3
• Can the HMIS Lead monitor each agency for HMIS
data quality compliance on an at least annual basis?
• Does their monitoring process integrate all 4
elements of data quality?
–
–
–
–
Completeness
Accuracy
Timeliness
Consistency
• How will monitoring results be shared with
the agency? With the CoC?
20
Step 4: CoC, Agency & HMIS Leadership Efforts
• Important to celebrate successes and to allow room
for growth
• Make the connection between the HMIS DQ efforts
and other CoC lead efforts
– Impact of improved data quality on the accuracy of System
Performance Measures and other local data analysis
– Impact of improved data quality on the ability to generate
a By-Name or Prioritization List, to use HMIS for
coordinated entry, etc.
21
Key Considerations in Step 4
• Is everyone at the CoC, agency and HMIS Lead level
clear about the role that they play in ensuring data
quality?
• How has this been communicated?
• How has data quality been integrated into CoC,
agency and HMIS meetings?
• What are the motivations/barriers for getting people
on board? Is special outreach or help needed to
work with agencies that do not get HUD funding?
22
Step 5: HMIS Lead’s Administration of HMIS
• HMIS Lead should complete the monitoring on data
quality
• Will need to run regular data quality reports for
agencies to track progress beyond the monitoring
visit
• HMIS Lead is at the center of this work and needs to
make these connections to CoC efforts with the
community
23
Key Considerations in Step 5
• Is the HMIS Lead regularly communicating about
progress and barriers with the data quality program?
• Has this work become an ongoing effort and is it
integrated into the regular operations of the CoC,
agencies and HMIS Lead?
24
What’s Next?
• Don’t wait!! The quality of your data now will impact
your upcoming SPM reports
• Map out your baseline
• Discuss these steps with your CoC
• Review sample HMIS Data Quality Plans (they’re on
the web!)
• Talk to other CoCs about how they’ve done this sort
of work
• Spend time thinking through monitoring and
compliance
25
Resources and Guidance
HUD Data Quality Toolkit (2009)
• https://www.hudexchange.info/resources/document
s/huddataqualitytoolkit.pdf
HMIS Proposed Rule (2011)
• https://www.hudexchange.info/resources/document
s/HEARTH_HMISRequirementsProposedRule.pdf
HMIS Technical Standards (2004)
• https://www.hudexchange.info/resources/document
s/HEARTH_HMISRequirementsProposedRule.pdf
26
Additional Questions
• Natalie Matthews, [email protected]
• Mike Lindsay, [email protected]
27