Identifying and Addressing Early
Childhood Outcomes Data
Quality to Accurately Measure
Improvement
Cornelia Taylor, DaSy/ECTA
Katrina Martin, DaSy, ECTA
Tony Ruggiero, DaSy
DEC Conference
Louisville, KY, October 2016
Session Objectives
• Provide overview of Child Outcomes
• Facilitate small group discussion on implementation
around recommended practices using the Herman
Scenario
• Provide overview of ENHANCE study and key results
2
Child Outcomes
• States are required to report on infants and toddlers with
Individualized Family Service Plans (IFSPs) and
preschool children with Individualized Education Plans
(IEPs) on three global outcomes:
1. Positive social-emotional skills (including social
relationships);
2. Acquisition and use of knowledge and skills
(including early language/communication [and early
literacy]); and
3. Use of appropriate behaviors to meet their needs.
3
Why Gather Child Outcomes Data?
•
Identify program strengths and weaknesses for
improving program services and delivery
•
Inform stakeholders, including families, about the
effectiveness of the program
•
Provide information for policymakers to justify future
funding of programs
•
Meet federal government requirements
The Two COS Questions
• To what extent does this child show age-appropriate
functioning, across a variety of settings and
situations, on this outcome (Rating: 1-7)
• Has the child shown any new skills or behaviors
related to [this outcome] since the last outcomes
summary? (Yes-No)
5
The COS Process
• Completed by a team including the family
• Information about child functioning across settings
and situations is considered.
• A level of functioning relative to age-expectations is
assigned to functional behaviors.
• Rating on 7-point scale is assigned for each of 3 child
outcomes describing the degree to which functional
behavior in the outcome is age expected.
7 Point Scale
Overall Age-Appropriate
• Child shows functioning expected for his or her age in all or almost all
Completely means:
7
everyday situations that are part of the child’s life. Functioning is
considered appropriate for his or her age.
• No one has any concerns about the child’s functioning in this outcome
area.
• Child’s functioning generally is considered appropriate for his or her age
6
but there are some significant concerns about the child’s functioning
in this outcome area. These concerns are substantial enough to suggest
monitoring or possible additional support.
• Although age-appropriate, the child’s functioning may border on not
keeping pace with age expectations.
• Child shows functioning expected for his or her age some of the time
Somewhat means:
5
and/or in some settings and situations. Child’s functioning is a mix of
age-appropriate and not age-appropriate behaviors and skills.
• Child’s functioning might be described as like that of a slightly younger
child*.
4
• Child shows occasional age-appropriate functioning across settings and
situations. More functioning is not age-appropriate than age-appropriate.
Overall Not Age-Appropriate
• Child does not yet show functioning expected of a child of his or her age
in any situation.
Nearly means:
3
• Child uses immediate foundational skills, most or all of the time,
across settings and situations. Immediate foundational skills are the
skills upon which to build age-appropriate functioning.
• Functioning might be described as like that of a younger child*.
• Child occasionally uses immediate foundational skills across settings
2
and situations. More functioning reflects skills that are not immediate
foundational than are immediate foundational.
• Child does not yet show functioning expected of a child his or her age in
any situation.
• Child’s functioning does not yet include immediate foundational skills
Not yet means:
1
upon which to build age-appropriate functioning.
• Child functioning reflects skills that developmentally come before
immediate foundational skills.
• Child’s functioning might be described as like that of a much younger
child*.
7
8
Age Anchoring
Age-Expected Skills
Immediate Foundational
Skills
Foundational Skills
9
Division of Early Childhood (DEC)
Recommended Practices for Assessment
•
Involves multiple sources of information
information from a child’s family and other significant
individuals in the child’s life
•
Includes multiple measures
observations, interviews, and direct assessments
appropriate for the child’s age and level of development,
sensory, physical, communication, cultural, linguistic,
social, and emotional characteristics
2014 DEC Recommended Practices in Early Intervention and Early Childhood Special Education
The COS process uses:
Multiple Methods
Multiple Sources of Information
•
Curriculum-based assessment
• Family
•
Norm-referenced assessment
• Interventionists
•
Developmental screening tool
• Teachers
•
Observation across settings and
situations
• Service providers
Parent report
• Child care providers
•
• Physicians
• Other people familiar with the child
The COS process produces a synthesis of information.
It is not an assessment.
Activity
Herman Family Scenario
• In small groups:
– Read scenario that illustrates an early intervention team’s
assessment practices
– Review and reflect on the early intervention team’s
assessment practices
• In large group:
– Report out reflections on the early intervention team’s
assessment practices
12
ENHANCE Studies
• 4 Studies (2009-2013)
– Provider survey
– Team decision-making
study (videos)
– Child assessments study
– State data study
ENHANCE
• Validity of the Data From the Child Outcomes
Summary Process: Findings From the ENHANCE
Project
• Research project funded in 2009 by the U.S.
Department of Education, Institute for Educational
Sciences
• Series of studies designed to address whether the
COS process produces meaningful and accurate data
about child outcomes for accountability and program
improvement.
14
Overview of Key Findings
On the basis of evidence collected across four studies, the authors concluded that
when implemented as intended, the COS process produces ratings that are valid for
accountability and program improvement purposes.
15
Key Findings
• Providers understood the types of behaviors
included in each of the three child outcomes.
• Providers could accurately apply their knowledge of
child development and the COS rating criteria.
• The COS process could be incorporated into existing
practice without negative consequences.
• With a few exceptions, children who were rated
higher on the COS also scored higher on the
assessment tools.
16
Key Findings (cont.)
• COS ratings were related to the child’s functional
abilities and type of disability.
• Children who entered EI and ECSE with higher COS
ratings tended to exit the programs with higher COS
ratings.
• Most states had stable percentages of children
making greater than expected growth or exiting at
age expectations over time as measured by the COS.
• Some of the programs studied were not
implementing the COS process as intended.
17
Key Findings (cont.)
• Some of the programs studied were not
implementing the COS process as intended.
– Amount of training providers received varied considerably.
– Team discussions were very brief with an average of 10
minutes; more than half were 9 minutes or less.
– Some teams did not discuss the child’s functioning in the
outcome area in sufficient breadth or depth.
– Providers tended to rate their colleagues’ understanding of
the outcomes and COS concepts lower than their own.
18
Stay In Touch!
• Email
– [email protected]
– [email protected]
– [email protected]
• DaSy website: http://dasycenter.org/
• ECTA website: http://ectacenter.org
19
Resources
• COSF Form –
http://ectacenter.org/eco/assets/pdfs/COSF_with_su
pporting_evidence.pdf
• COS online modules http://dasycenter.org/child-outcomessummary-cos-process-module-collectingusing-data-to-improve-programs/
20
Resources
• ENHANCE Brief http://ectacenter.org/eco/assets/pdfs/ENHANCEbrief
_03-02-16Final.pdf
• ENHANCE Survey http://ectacenter.org/eco/assets/pdfs/snENH_Outco
mesSrvyGen2012_FINAL_outcomesconf10-23-12.pdf
21
The contents of this tool and guidance were developed under grants from
the U.S. Department of Education, #H326P120002 and #H373Z120002.
However, those contents do not necessarily represent the policy of the U.S.
Department of Education, and you should not assume endorsement by the
Federal Government. Project Officers: Meredith Miceli, Richelle Davis, and
Julia Martin Eile.
22