Where the Rubber Hits the Road:
Tools and Strategies for Using
Child Outcomes Data for
Program Improvement
Christina Kasprzak
ECTA/ECO/DaSy
Lauren Barton
ECO/DaSy
Ruth Chvojicek
WI Statewide Part B Indicator 7 Child Outcomes Coordinator
March 17, 2014
Encore Webinar: Improving Data, Improving Outcomes Conference
Purposes
• To describe national resources
for promoting data quality
and supporting
program improvement
• To share Wisconsin 619 experience
and strategies to promote data quality
and program improvement
2
Available Resources
Question
Resource
Can I trust the quality of my data?
Where are there red flags?
Pattern Checking Table
See the “using data” page under Outcomes Measurement
on the ECTA website:
http://www.ectacenter.org/eco/pages/usingdata.asp
3
Pattern Checking for Data Quality
Strategies for using data
analysis to improve the
quality of state data by
looking for patterns that
indicate potential issues
for further investigation.
http://www.ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.pdf
Questions to Ask
• Do the data make sense?
– Am I surprised? Do I believe the data?
Believe some of the data? All of the data?
• If the data are reasonable (or when they
become reasonable), what might they tell
us?
5
Available Resources
Question
Resource
Can I trust the quality of my data?
Where are there red flags?
Pattern Checking Table
What are your internal systems
that support quality data?
Local Contributing Factors Tool
6
Local Contributing Factors Tool
Provides ideas about
questions a local team
would consider in
identifying factors
impacting performance.
Introductory video and document at:
http://www.ectacenter.org/eco/pages/usingdata.asp
Available Resources
Question
Resource
Can I trust the quality of my data?
Where are there red flags?
Pattern Checking Table
What are your internal systems that
support quality data?
Local Contributing Factors Tool
What strategies might support
improvement in different outcomes
for Part C?
Relationship of Quality Practices to
Child and Family Outcome
Measurement Results
8
Relationship of Quality Practices to Child
and Family Outcome Measurement Results
Designed to assist states in
identifying ways to improve
results for children and
families through
implementation of quality
practices.
http://www.ectacenter.org/~pdfs/QualityPracticesOutcomes_2012-04-17.pdf
9
Available Resources
Question
Resource
Can I trust the quality of my data?
Where are there red flags?
Pattern Checking Table
What are your internal systems that
support quality data?
Local Contributing Factors Tool
What strategies might support
improvement in different outcomes for
Part C?
Relationship of Quality Practices to
Child and Family Outcome
Measurement Results
How can I use my data to analyze and
improve program practices?
Analyzing Child Outcomes Data for
Program Improvement: A Guidance
Table
10
Analyzing Child Outcomes Data for
Program Improvement
• Quick reference tool
• Consider key issues,
questions, and
approaches for
analyzing and
interpreting child
outcomes data.
http://www.ectacenter.org/eco/assets/pdfs/AnalyzingChildOutcomesData-GuidanceTable.pdf
Describes Steps in Process of Using
Data for Program Improvement
•
•
•
•
•
Defining analysis questions
Clarifying expectations
Analyzing data
Testing inferences
Data-based program improvement
planning
12
Guidance Table
13
14
USING DATA FOR STATE &
LOCAL IMPROVEMENT
WISCONSIN’S PART B
Ruth Chvojicek – WI Statewide Part B Indicator 7 Child
Outcomes Coordinator
WISCONSIN T/TA SYSTEM
Department of
Public
Instruction
Indicator
6/7/12
Coordinators
CESA Program
Support
Teacher
State 11-12 Entry Rating Distribution
35.0%
30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
1
2
3
4
5
6
7
Outcome 1
3.6%
10.8%
9.3%
13.8%
21.4%
20.9%
20.2%
Outcome 2
4.7%
14.2%
16.3%
19.7%
27.8%
14.2%
3.1%
Outcome 3
3.3%
7.5%
8.5%
11.5%
17.3%
21.2%
30.7%
EXAMPLES FROM 2011-2012
WISCONSIN DATA REVIEW PROGRESSION
2011-2012
2012-2013
• Pilot – 20 LEAs
• Delivery via technology
• Pattern common discussion points informed 12-13
statewide PD
• Offered group data reviews within each CESA region and
several large LEA’s
• Focus on enhancing data quality (accuracy of rating)
• Individual District Data Reports
• Process for development of reports
• District action steps: Assessment process, rating practice +
SAMPLE GRAPHS – ENTRY OR EXIT RATING
DISTRIBUTION
SAMPLE – ENTRY RATING BY ELIGIBILITY
Outcome One
Other Health Impairment
Significant Developmental Delay
Speech or Language Impairment
1
50.0%
30.0%
0.0%
2
0.0%
30.0%
0.0%
3
50.0%
40.0%
8.3%
4
0.0%
0.0%
4.2%
5
0.0%
0.0%
16.7%
6
0.0%
0.0%
16.7%
Outcome Two
Other Health Impairment
Significant Developmental Delay
Speech or Language Impairment
1
50.0%
40.0%
0.0%
2
0.0%
20.0%
12.5%
3
50.0%
30.0%
20.8%
4
0.0%
10.0%
20.8%
5
0.0%
0.0%
41.7%
6
0.0%
0.0%
4.2%
Outcome Three
Other Health Impairment
Significant Developmental Delay
Speech or Language Impairment
1
50.0%
20.0%
0.0%
2
0.0%
30.0%
0.0%
3
50.0%
0.0%
8.3%
4
0.0%
10.0%
4.2%
5
0.0%
20.0%
12.5%
6
0.0%
20.0%
8.3%
7
0.0%
0.0%
54.2%
7
0.0%
0.0%
66.7%
SAMPLE – ENTRY RATING COMPARISON
CESA #
Outcome Two
1
2
3
4
5
6
7
1
28
4
1
1
2
24
57
10
5
5
4
Outcome Three
3
4
7
3
26
25
33
39
6
42
6
27
3
4
5
5
26
28
75
86
25
1
6
2
14
40
32
77
50
9
7
1
13
18
31
96
55
31
EXAMPLE RACE/ETHNICITY
Outcome 1
Asian
Black
Hispanic
American Indian Alaskan
Hawaiian Other Pacific Islander
Two or More Races
White
1 2 3 4 5 6 7
4.3% 18.4% 9.9% 9.2% 23.4% 15.6% 19.1%
6.4% 15.3% 11.3% 16.7% 20.1% 15.2% 15.2%
4.9% 11.9% 12.1% 13.5% 21.5% 21.7% 14.5%
2.7% 10.0% 15.5% 19.1% 24.5% 17.3% 10.9%
0.0% 31.6% 21.1% 10.5% 31.6% 0.0% 5.3%
4.5% 7.2% 11.7% 18.9% 26.1% 18.9% 12.6%
2.8% 9.5% 8.1% 13.2% 21.4% 22.2% 22.8%
Outcome 1 State 11-12 Entry
25.0%
20.0%
15.0%
State Black
State Hispanic
State White
10.0%
5.0%
0.0%
1
2
3
4
5
6
7
Outcome 1 CESA 1
25.0%
20.0%
15.0%
CESA 1 Black
CESA 1 Hispanic
10.0%
CESA 1 White
5.0%
0.0%
1
2
3
4
5
6
7
Outcome 1 CESA 2
25.0%
20.0%
15.0%
CESA 2 Black
CESA 2 Hispanic
10.0%
CESA 2 White
5.0%
0.0%
1
2
3
4
5
6
7
Outcome 1 District M 11-12 Entry
45.0%
40.0%
35.0%
30.0%
25.0%
District M Black
20.0%
District M Hispanic
15.0%
District M White
10.0%
5.0%
0.0%
1
2
3
4
5
6
7
Outcome 1 District B 11-12 Entry
50.0%
45.0%
40.0%
35.0%
30.0%
District B Black
25.0%
District B Hispanic
20.0%
District B White
15.0%
10.0%
5.0%
0.0%
1
2
3
4
5
6
7
Outcome One 2011-2012/2012-2013
Entry Rating Distribution Comparison
30
25
20
15
10
5
0
1
2
3
4
5
6
7
2011-2012
3.6
10.8
9.3
13.8
21.4
20.9
20.2
2012-2013
3.9
11
12.9
19
25.6
18.6
9.1
Outcome Two 2011-2012/2012-2013
Entry Rating Distribution Comparison
30
25
20
15
10
5
0
1
2
3
4
5
6
7
2011-2012
4.7
14.2
16.3
19.7
27.8
14.2
3.1
2012-2013
4.7
15.4
20
25.6
25
7.5
1.7
Outcome Three 2011-2012/2012-2013 Entry Rating
Distribution Comparison
35
30
25
20
15
10
5
0
1
2
3
4
5
6
7
2011-2012
3.3
7.5
8.5
11.5
17.3
21.2
30.7
2012-2013
3
8.5
11.1
15.1
24.7
21.6
15.9
BUT … OUTCOME ONE EXIT RATING
Outcome 1 Exit Rating Comparison
50.0%
45.0%
40.0%
35.0%
30.0%
25.0%
20.0%
15.0%
10.0%
5.0%
0.0%
1
2
3
4
5
6
7
11-12 Outcome 1 Exit Rating
1.0%
3.2%
3.9%
7.0%
15.8%
25.3%
43.8%
12-13 Outcome 1 Exit Rating
0.7%
1.9%
3.7%
6.2%
14.6%
28.5%
44.4%
OUTCOME THREE
Outcome 3 Exit Rating
Comparison
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
1
2
3
4
5
6
7
11-12 Outcome 3 Exit Rating
1.3%
2.5%
3.0%
3.9%
9.0%
23.8%
56.4%
12-13 Outcome 3 Exit Rating
0.6%
1.4%
2.4%
4.3%
9.6%
23.7%
58.1%
WISCONSIN DATA REVIEW PROGRESSION
2011-2012
• Pilot – 20 LEAs
• Via technology
• Pattern common discussion points informed 12-13 statewide PD
2012-2013
• Offered group data reviews within each CESA region and several large LEA’s
• Focus on enhancing data quality (accuracy of rating)
• Individual District Data Reports
• Process for development of reports
• District action steps: Assessment process, rating practice +
2013-2014
• Continue group data reviews all CESA regions
• Continue working on data quality – plus…
• Piloting conversation around settings and Indicator 7 progress
• Statewide Stakeholder Workgroup – Discussion using Analyzing Child Outcomes
Data tool
Next Steps?
• Try out using these resources
• Send feedback to ECO Center about the
new Analysis tool
• What are your ‘take aways’ and next
steps related to analyzing your data for
data quality and/or program
improvement? (notes for State Team
time)
35
Find more resources at:
http://www.ectacenter.org
36
Questions? Comments?
• #6 to un-mute
37