Broad Data Analysis
Child Outcome Data
Broad Analysis: Child Outcomes
 Does our state’s data look different than the national data?
 Are our state child outcomes trends stable over time?
 Is the data trending upwards?
 Is the data trending downwards?
 Is our state performing more poorly in some outcomes than
others?
 Are the child outcomes similar across programs?
 What about data quality? Can we be confident in our data?
Child Outcomes: State vs. National
Part C Early Intervention
National and State Percentages for Summary Statement 1:
Substantially Increased Rate of Growth
100
80
Percent of children
67
73
72
79
73
80
60
40
20
0
Social relationships
Note: National data based on 33 states with highest-quality data
Knowledge and skills
National
Virginia
Actions to meet needs
Child Outcomes: State vs. National
Part C Early Intervention
National and State Percentages for Summary Statement 2:
Exited within Age Expectations
100
Percent of children
80
60
60
67
52
57
59
56
40
20
0
Social relationships
Knowledge and skills
National
Note: National data based on 33 states with highest-quality data
Virginia
Actions to meet needs
National Vs. State
Meaningful Differences
Child Outcomes: National vs. State
FFY11 and State FFY12
90%
80%
70%
60%
50%
National FFY11
Virginia FFY11
40%
Virginia FFY12
30%
20%
10%
0%
SS1 - SE
SS1 - KS
SS1 - TA
SS2 - SE
SS2 - KS
SS2 - TA
Data Quality – Trends
Virginia Trends
Virginia Trends
Outcome 3:
Use of Appropriate Behaviors
to Meet Needs
Outcome 2:
Acquisition and Use of
Knowledge and Skills
Virginia Trends
Child Outcome Results FFY11/FFY12
Meaningful Differences Calculation
Child Outcomes: National vs. State
FFY11 and State FFY12
90%
80%
70%
60%
50%
National FFY11
Virginia FFY11
40%
Virginia FFY12
30%
20%
10%
0%
SS1 - SE
SS1 - KS
SS1 - TA
SS2 - SE
SS2 - KS
SS2 - TA
State (n=2509)
NN (n=7)*
KK (n=37)*
HH (n=46)*
A (n=22)*
R (n=84)
MM (n=27)
DD (n=125)*
LL (n=52)*
JJ (n=61)*
C (n=32)
II (n=107)*
S (n=102)*
V (n=90)*
EE (n=14)
FF (n=146)*
W (n=124)*
U (n=164)*
X (n=129)*
J (n=35)
L (n=70)
GG (n=113)
B (n=100)
O (n=181)
I (n=122)
D (n=39)
N (n=64)
M (n=114)
E (n=32)
K (n=79)
Y (n=556)*
BB (n=46)
Q (n=133)*
G (n=125)*
AA (n=40)*
F (n=48)*
H (n=28)
CC (n=33)*
Z (n=40)*
P (n=135)*
T (n=24)*
Child Outcomes: Local vs. State
FFY 2012: Positive Social Relationships, Greater than Expected Growth
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
State (n=2509)
AA (n=40)*
EE (n=14)
DD (n=125)*
X (n=129)*
KK (n=37)*
LL (n=52)*
Z (n=40)*
GG (n=113)*
Q (n=133)*
I (n=122)*
II (n=107)*
U (n=164)*
R (n=84)
L (n=70)
Y (n=556)
M (n=114)
JJ (n=61)
P (n=135)
F (n=48)
S (n=102)
B (n=100)
D (n=39)
HH (n=46)
V (n=90)
A (n=22)
O (n=181)
NN (n=7)
G (n=125)*
FF (n=146)*
E (n=32)
K (n=79)*
C (n=32)
W (n=124)*
BB (n=46)*
N (n=64)*
MM (n=27)*
J (n=35)*
CC (n=33)*
T (n=24)*
H (n=28)*
Child Outcomes: Local vs. State
FFY 2012: Positive Social Relationships, Exited within Age Expectations
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
State (n=3297)
NN (n=7)*
KK (n=37)*
JJ (n=61)*
A (n=22)
LL (n=52)*
HH (n=46)*
W (n=124)*
DD (n=125)*
MM (n=27)
X (n=129)*
R (n=84)*
II (n=107)*
FF (n=146)
C (n=32)
Q (n=133)
S (n=102)
L (n=70)
GG (n=113)
BB (n=46)
I (n=122)
U (n=164)
V (n=90)
B (n=100)
D (n=39)
Z (n=40)
M (n=114)
G (n=125)
O (n=181)
E (n=32)
J (n=35)
H (n=28)
Y (n=556)*
CC (n=33)*
K (n=79)*
N (n=64)*
AA (n=40)*
P (n=135)*
T (n=24)
F (n=48)*
EE (n=14)*
Child Outcomes: Local vs. State
FFY 2012: Knowledge & Skills, Greater than Expected Growth
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Child Outcomes: Local vs. State
FFY 2012: Knowledge and Skills, Exited within Age Expectations
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
State (n=3555)
DD (n=125)*
JJ (n=61)*
X (n=129)*
MM (n=27)
KK (n=37)*
W (n=124)*
HH (n=46)*
LL (n=52)*
C (n=32)
II (n=107)*
NN (n=7)
A (n=22)
V (n=90)
S (n=102)
B (n=100)
R (n=84)
L (n=70)
FF (n=146)
GG (n=113)
D (n=39)
U (n=164)
CC (n=33)
I (n=122)
M (n=114)
G (n=125)
O (n=181)
Z (n=40)
K (n=79)*
F (n=48)
Y (n=556)*
H (n=28)
Q (n=133)*
N (n=64)*
BB (n=46)*
J (n=35)*
P (n=135)*
AA (n=40)*
T (n=24)*
E (n=32)*
EE (n=14)*
Child Outcomes: Local vs. State
FFY 2012: Actions to Meet Needs, Greater than Expected Growth
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Child Outcomes: Local vs. State
FFY 2012: Actions to Meet Needs, Exited within Age Expectations
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Data Quality Elements
 Completeness of data
 number of children reported for the outcome/number who exited
 Virginia’s results: average= 65%; range for Local Systems = 17% - 100%
 Expected Patterns for Progress Categories
Category a
0
>10%
Category e
<5%
>65%
 Virginia’s state date is within these parameters for all three outcomes
 Child Outcomes State Trends Over Time
 As noted on previous slides, Virginia’s results do not show wide variations
which would trigger concerns about data quality
Broad Data Analysis
Family Outcome Data
Broad Analysis: Family Outcomes
 Does our state’s data look different than the national data?
 Are our state family outcomes trends stable over time?
 Is the data trending upwards?
 Is the data trending downwards?
 Is our state performing more poorly in some outcomes than
others?
 Are the family outcomes similar across programs?
 What about data quality? Can we be confident in our data?
Indicator C4: Family Outcomes
NCSEAM States using Rasch Scoring vs. Virginia
100
90
80
86
78
77
75
85
75
Percent of Familes
70
60
50
40
30
20
10
0
Know their rights
Effectively communicate
children's needs
States using NCSEAM with Rasch Scoring (FFY 2011, n=14)
Help their child
develop and learn
Virginia FFY 2012
Family Outcomes:
State Trends over Time
Indicator C4: Family Outcomes
NCSEAM States using Rasch Scoring vs. Virginia
100
90
80
86
78
77
75
85
75
Percent of Familes
70
60
50
40
30
20
10
0
Know their rights
Effectively communicate
children's needs
States using NCSEAM with Rasch Scoring (FFY 2011, n=14)
Help their child
develop and learn
Virginia FFY 2012
Family Outcomes: Local vs. State 2012-2013
4A: EI has helped the family know their rights
Family Outcomes: Local vs. State 2012-2013
4B: EI has helped the family communicate their children’s
needs
Family Outcomes: Local vs. State 2012-2013
4C: EI has helped the family help their child develop and learn
Data Quality
 The data analysis for Virginia’ survey is
quite extensive, using rigorous data
analysis standards.
 Virginia’s response rate, like other states
that use mailing as the means to
disseminate the family survey, is lower
than states who use other mechanisms.
Broad Infrastructure Analysis
Governance
Monitoring and
Accountability
Fiscal
Broad
Infrastructure
Analysis
Technical
Assistance
Quality
Standards
Data
Professional
Development
Analysis Mechanisms
 Use of federal monitoring tools and procedures
 Communication with Local Systems through:




Ongoing TA
Regional Meetings
Trainings
System Manager Meetings
 Monitoring Results
 Record Reviews
 QMRs
 Individualized TA
 Local System Contract Deliverables
Infrastructure Analysis:
Information Sources
 Local self-reporting
 Stakeholder input
 LSM surveys
 Training/meeting evaluations
 Monitoring/QMR
 Dispute Resolutions
 Observations through TA and other
interactions
Broad Infrastructure Analysis
Governance - Mixed
 Strengths
 Code of Virginia establishes infrastructure
 VICC
 State Interagency Agreement; interagency partnerships
 DBHDS local contract with Local Lead Agencies specifies LS
infrastructure requirements
 Weaknesses
 Variable job roles and skill sets of LSMs
 Variable strength and specificity of LLA/provider contracts
 Variable support from LLA leadership
Broad Infrastructure Analysis
Fiscal - Mixed
 Strengths
 Additional state funding
 Medicaid EI Services program
 Trainings on fiscal issues
 Strong working relationship with fiscal office at the state lead agency
(DBHDS)
 Weaknesses
 Variable fiscal skill set of LSMs
 Code of Virginia does not require fiscal commitment at local level
 Variety of Local Lead Agencies
 Inconsistent reporting of fiscal data at local level
 Lack of fiscal data in state data system
 Perception/reality that there is not enough money
Broad Infrastructure Analysis
 Quality Standards – Not sure
 Practice Manual articulates expected practices, but these
are not labeled quality practices
 Monitoring and Accountability – Strength
 Local self-monitoring and supervision for continuous
improvement – variable
 Professional Development – Strength
 Technical Assistance – Strength
 Data – Weakness
Closing Thoughts
Based on this broad review,
did anything strike you as an
area of focus for our
Systemic Improvement Plan?