Supporting States in Building a
Child Outcomes Measurement
System
Prepared for the ECO Advisers Meeting
February 2011
Two Frameworks
• Child Outcomes Measurement
System
• Family Experiences and Outcomes
Measurement System
Early Childhood Outcomes Center
2
Child Outcomes Measurement
System
• The set of components a state
needs to have in place to make full
use of child outcomes data.
• NOT just a data system or a data
collection method.
Early Childhood Outcomes Center
3
Purpose of the Framework
• Provide a common language for ECO and other
TA providers to use in discussing COMSs with
states.
• Provide a organizing structure of categorizing
resources and state examples related to
implementation of a COMS.
• Serve as the organizing structure for the state
self assessment
Early Childhood Outcomes Center
4
Framework and Self-Assessment
• FRAMEWORK
– Set of components and quality indicators
– Provides the structure for the self-assessment
• SELF-ASSESSMENT
– Scale that provides criteria for levels of
implementation within each quality indicator
– Rating assigned based on level of
implementation within each indicator
Early Childhood Outcomes Center
5
Process for Framework
Development
• Built off what we had learned
from ECO work with states
• Literature review
• Repeated discussion and
review internally and with 7
Partner States
Early Childhood Outcomes Center
6
Framework Partner States
State
Part C
California
619
X
Colorado
X
X
Delaware
X
X
Maine
X
X
Minnesota
X
X
New York
X
Ohio
X
Early Childhood Outcomes Center
X
7
Purpose
Data
Collection
and Transmission
Analysis
Reporting
Cross-system Coordination
Evaluation
Early Childhood Outcomes Center 8
Using Data
Quality Indicator
• Provides additional detail as to what
constitutes quality implementation of the
component.
• 18 quality indicators across the 7
components
Early Childhood Outcomes Center
9
Quality Indicators
Components
Elements
Purpose
1.
State has articulated purpose(s) of COMS.
Data Collection and Transmission
Purpose
2.
Data collection procedures are carried out efficiently and effectively.
3.
Providers, supervisors, and others involved in data collection have the
required knowledge, skills, and commitment.
Data
Collection and
Transmission
4.
State's method for entering, transmitting, and storing data is effective and
efficient.
Analysis
State identifies accountability and program improvement questions related to
child outcomes.
Local programs identify accountability and program improvement questions
related to child outcomes.
Analysis
7.
State agency analyzes data in a timely manner.
8.
Local programs analyze data in a timely manner.
9.
State agency ensures completeness and accuracy of data.
Reporting
Using Data
Evaluation
Cross-System
Coordination
Early Childhood Outcomes Center
5.
6.
Reporting
10. State agency interprets, reports, and communicates information related to
child outcomes.
11. Local programs interpret, report, and communicate information related to
child outcomes.
Using Data
12. State agency makes regular use of information on child outcomes to improve
programs.
13. Local programs makes regular use of information on child outcomes to
improve programs.
Evaluation
14. State evaluates its COMS regularly.
Cross-system Coordination
a.
b.
c.
d.
e.
f.
g.
h.
State has…
State has…
State agency..
Representative..
State agency…
State ……
State provides…
State has..
15. Part C and 619 coordinate child outcomes measurement.
16. Child outcomes measurement is integrated across early childhood (EC)
programs statewide.
17. Child outcomes measurement is aligned with state’s early learning
guidelines/standards.
18. State has a longitudinal data system to link child outcomes data from EC
program participation to K–12 data.
10
3 Quality Indicators for Data
Collection and Transmission
• Data collection procedures are carried out
efficiently and effectively.
• Providers, supervisors, and others
involved in data collection have the
required knowledge, skills, and
commitment.
• State's method for entering, transmitting,
and storing data is effective and efficient.
Early Childhood Outcomes Center
11
The Self Assessment
Early Childhood Outcomes Center
12
Purpose of the Self Assessment
• Provide guidance to states on
what constitutes a high quality
child outcomes measurement
system.
• Assist states in setting priorities for improving
their COMS
• Provide information to assist states in
advocating for resources for systems
development
Early Childhood Outcomes Center
13
• Each QI has multiple
elements.
• Evidence for the extent of
implementation for each
element is provided.
• Each element is rated as
• NY = Not Yet or Don’t know
• IP = In Process, or
• IF = Fully Implemented
• The QI is given a rating
based on the ratings of the
elements.
Early Childhood Outcomes Center
14
Back-up for Each Element
• Live link from the element
• Describes the element
• Describes what “fully
implemented looks like”
• Provides examples of what
“In process” might look like
• Provides examples of how
states are addressing the
element
• Provides additional
resources related to the
element
Early Childhood Outcomes Center
15
The Scale for the Quality Indicators
Quality
Indicator
Score
Implementation of Elements
All elements are fully implemented
7
Nearly all elements are fully implemented and the rest
are in process
6
Most of the elements are fully implemented and the rest
are in process.
5
At least one element is fully implemented and the rest are
in process
4
All of the elements are in process
3
Some of the elements are in process
2
None of the elements are yet in process
1
Early Childhood Outcomes Center
16
Early Childhood Outcomes Center
17
Web-based Scoring?
• Information can be
entered online
• Will be stored for state
by ECO
• Score from the QI
page will automatically
transfer to the overall
score page
• Online scoring will
show multiple marking
points (i.e., change
over time)
Early Childhood Outcomes Center
18
Technology Support
• Online capability
– Store the evidence (text)
– Transfer QI rating to the score sheet and generate
bars
– Also, allows ECO to use the aggregate state data to
track where states are overall in building COMS
• Downloadable version with many of the same
features (Except ECO access to data)
Early Childhood Outcomes Center
19
Working with the Self
Assessment
Early Childhood Outcomes Center
20
Possible Process
1. State identifies the highest priority QI(s).
2. Stakeholders convened
– Priority QIs reviewed and rated
– Plan developed to address elements not yet
fully implemented.
– Progress reviewed with stakeholders at
regular intervals.
3. State identifies next set of QI(s), etc.
Early Childhood Outcomes Center
21
Our questions for you
1.
General impressions?
– Content: Comprehensive? Appropriate?
– Format/layout: User friendly? Understandable?
– Length?
– Scoring: Reasonable guideline?
2. Investment of resources
– Continue development?
•
Pick new partners?
– Pursue the web version? Heavily? Lightly?
– Pursue the downloadable Word version?
– How important is it that ECO track where states are in building
their systems?
Early Childhood Outcomes Center
22
For more information
• For updates to the framework and the selfassessment and resources to support the
quality indicators:
www.the-eco-center.org
Early Childhood Outcomes Center
23