Using the ECO Child
Outcomes Summary Tool
Kathy Hebbeler
ECO Center at SRI International
Prepared for the State TA Call, October 20, 2005
Early Childhood Outcomes Center
Important Information


We will not be going over the
material on slides 4 to 37 on the
call. It is for your information and
will be discussed at the OSEP Early
Childhood Meeting post-season on
outcomes in December.
State options are indicated with a
.
Early Childhood Outcomes Center
2
Go to Slide 38 
Early Childhood Outcomes Center
3
ECO Framework for Developing
an Outcomes System
1. Identify purposes
2. Identify outcomes
3. Specify evidence statements
4. Identify measurement approaches
(including how to transform the
data)
5. Implement procedures for data
transmission
And other steps…..
Early Childhood Outcomes Center
4
Decision: Purpose of the
outcomes system




Produce data for OSEP
Provide evidence of effectiveness
of program (state legislature, ICC,
etc.)
Provide information for program
improvement
Etc.
Early Childhood Outcomes Center
5
Decision: Identify child
outcomes areas

Given: OSEP child outcome areas




Positive social emotional skills (including
positive social relationships)
Acquisition and use of knowledge and skills
(including early language/communication)
Use of appropriate behaviors to meet their
needs
State decision: Other outcome areas?
Sub-outcomes?
Early Childhood Outcomes Center
6
Decision: Identify family
outcomes areas

Given: OSEP family indicators


Percent of families participating in Part C who
report that EI services have helped the family
 Know their rights
 Effectively communicate their children’s needs
 Help their children develop and learn
State decision: Other outcome
areas?
Early Childhood Outcomes Center
7
Child outcome areas: What
states are doing



Adopting the 3 as is
Adding sub-indicators under the
outcomes
?
Early Childhood Outcomes Center
8
Family outcome areas: What
states are doing


Add to/expand the OSEP
indicators
Add to/expand the ECO family
outcomes


E.g., quality of life
?
Early Childhood Outcomes Center
9
Evidence Statements:
OSEP only or More?
Early Childhood Outcomes Center
Decision: What kind of evidence
statements does state want to be
able to produce?
Evidence Statement - a
statement that incorporates a
statistic and provides evidence
as to whether not an outcome
has been achieved
Early Childhood Outcomes Center
11
Decision: What kind of evidence
statements to produce?

Given = OSEP
indicators/measures (evidence
statements):
a. % of infants and toddlers/preschoolers
who reach or maintain functioning at a
level comparable to same-age peers
b. % of infants and toddlers/preschoolers
who improve functioning but are not
in “a”
c. % of infants and toddlers/preschool
children who did not improve
functioning
Early Childhood Outcomes Center
12
OSEP Indicators and
Measurement Categories
5 year old
level
Group a:
maintained or
reached typical
2 year
old level
Entry
Early Childhood Outcomes Center
Exit
13
Indicators and Measurement
Categories
5 year old
level
Group b: made
progress but…
2 year
old level
Entry
Early Childhood Outcomes Center
Exit
14
OSEP Indicators and
Measurement Categories
5 year old
level
Group c: did
not make
progress
2 year
old level
Entry
Early Childhood Outcomes Center
Exit
15
OSEP Indicators and
Measurement Categories
Group a:
maintained or
reached typical
5 year old
level
Group b: made
progress but..
Group c: did
not make
progress
2 year
old level
Entry
Early Childhood Outcomes Center
Exit
16
Decision: Does the state want to
be able to produce any other
evidence statements?


Note: this does not necessarily mean
collecting more data. It could mean
recording or analyzing the same data
in a different way.
If the state wants outcome data for
anything other than to meet the
federal reporting requirements, the
answer to this question is almost
certainly “Yes.”
Early Childhood Outcomes Center
17
Decision: Does the state want to
be able to produce any other
evidence statements?

The investment in building the
outcomes system is in collecting
good assessment data. Rolling
up or analyzing the data
differently adds almost nothing to
the cost.
Early Childhood Outcomes Center
18
Good news, bad news
+ OSEP evidence statements are
relatively straightforward
- OSEP evidence statements are of
limited usefulness if state wants
data for demonstrating
effectiveness or for program
improvement
Early Childhood Outcomes Center
19
Who is in OSEP Group a?
5 year old
level
Group a:
maintained or
reached typical
= children who
“closed the gap”
2 year
old level
Entry
Early Childhood Outcomes Center
Exit
20
Who is in OSEP Group b?
5 year old
level
Group b: made
progress but…
= children who
made progress
toward “closing
the gap”
2 year
old level
Entry
Early Childhood Outcomes Center
Exit
21
OSEP Indicators
More Powerful Alternative
a. % of infants and toddlers who
reach or maintain functioning at a
level comparable to same-age peers
a1. % of children who made sufficient
progress to maintain functioning at a
level comparable to same age peers
a2. % of children who made sufficient
progress to achieve functioning at a
level comparable to same age peers
b. % of infants and toddlers who
improve functioning but are not in a
b1. % of children who mover nearer
to functioning comparable to same age
peers but did not achieve it
b2. % of children who made progress
but not sufficient progress to move
nearer to functioning comparable to
same-age peers
c. % of infants and toddlers
children who did not improve
functioning
c. % of infants and toddlers children
who did not improve functioning
a2+ b1
Changing developmental
trajectories


The strongest evidence for the
effectiveness of EI/ECSE is the
percentage of children whose
developmental trajectories
changed after they received
services
“Gap closers” = a2 + b1
Early Childhood Outcomes Center
23
Illustration: Getting more
powerful evidence (OSEP only)
cat
 OSEP
Category
2007
%
2008
%
2009
%
a
45
46
44
b
53
52
55
c
2
2
1
Illustration: Getting more powerful
evidence (same data, more
information)
2007
Category
%
18
a1
2008
%
17
2009
%
15
29
29
a2
27
b1
45
b2
8
6
5
c
2
2
1
72
46
74
50
79
Measurement:
Good assessment procedures are
the foundation for the child outcome
system
Early Childhood Outcomes Center
What is assessment?

“Assessment is a generic term
that refers to the process of
gathering information for
decision-making.”
(McLean, 2004)
Early Childhood Outcomes Center
27
What is assessment?

“Early childhood assessment is
flexible, collaborative decisionmaking process in which teams of
parents and professionals repeatedly
revise their judgments and reach
consensus about the changing
developmental, educational, medical,
and mental health services needs of
young children and their families.”
Bagnato and Neisworth, 1991
Quoted in DEC Recommended Practices, 2005
Early Childhood Outcomes Center
28
DEC Recommended Practices for
Assessment


Involves multiple sources (families,
professional team members, service
providers, caregivers, etc.)
Involves multiple measures
(observations, criterion-curriculumbased instruments, interviews,
curriculum-compatible norm-referenced
scales, informed clinical opinion, work
samples)
Early Childhood Outcomes Center
29
State decisions related to
assessment

What assessment information to
include in the outcome system?
 Single measure statewide
 List of approved measures
 Whatever measures programs are
using
Early Childhood Outcomes Center
30
State decisions related to
assessment

What assessment information to
include in the outcome system?
 Collect new assessment information
 Use existing assessment information
 States now gathering information
to learn what measures are being
used in the state
Early Childhood Outcomes Center
31
State decisions related to building on
existing assessment information

Options:
 Build upon data from the eligibility
determination process for outcomes
 Build upon data from ongoing
progress monitoring for outcomes
 Combination
Early Childhood Outcomes Center
32
Building on Eligibility Measures

States use standardized, norm-referenced
tests as part of eligibility
+ Easy to anchor to typical development
– Not necessarily reflective of young children’s
skills and behaviors in regular contexts
– Challenges in aligning domains to functional
outcomes
– Challenges with cultural relevance for diverse
populations

States add other information sources
(informed clinical opinion) for eligibility
determination
Early Childhood Outcomes Center
33
Building on Ongoing Progress
Monitoring

States use curriculum based/ criterion
referenced measures for ongoing progress
monitoring
+ Based on observation in everyday contexts by those
who know children well
+ Many c-b measures are a closer match than normreferenced measures for the 3 functional outcomes
– Some have age expectations but for others a
challenge in establishing age anchors
– Training is needed to insure measures is
administered reliably, especially if have large
numbers of “assessors” with varying levels of
experience and expertise
Early Childhood Outcomes Center
34
State decisions related to
assessment

What kind of measures to include?
 Norm-referenced
 Curriculum-based
 Combination
Early Childhood Outcomes Center
35
State decision related to
assessment

Parent input is critical



Consistent with best practice
Impossible to determine how child is doing
across a variety of settings without
information from caregivers
How to incorporate parent input?
 As part of the item scoring within a
measure
 As a separate source of information
 Both
Early Childhood Outcomes Center
36
State decisions related to
assessment

Frequency of data for outcome
system?


Given for OSEP: (Near) entry and
(near) exit
Anything in between?
 every 6 months
 annually
 at a fixed point (e.g., every March)
Early Childhood Outcomes Center
37
Transforming the
assessment data
Early Childhood Outcomes Center
Need to transform the data



Under any assessment option, state
needs a way to transform the
assessment information into the
indicators/evidence statements.
No assessment provides information
directly on the 3 outcomes areas
No assessment provides information
directly on the 3 categories (a,b,c) in
the OSEP indicators
Early Childhood Outcomes Center
39
Need to transform the data


…if assessment process involves
multiple sources of information
…if more than one assessment is
being used in the state
Early Childhood Outcomes Center
40
Input
Time 1 Scores
Time 2 Scores
Cognitive
Cognitive
Communication
Communication
Social
Social
Adaptive
Adaptive
Motor
Motor
≠
Desired
Output
OSEP
Indicators
Social
% a, b, c
Knowledge &…
% a, b, c
Meet Needs
(for 10,000 children!)
Early Childhood Outcomes Center
% a, b, c
41
Decisions related to transforming the
assessment data



What is the process by which the
data gets transformed?
What is the rubric used to “roll up
the data?
Where does the transformation
occur?
 State
 Local program
Early Childhood Outcomes Center
42
What is the rubric?




Depends on the kind of evidence
statements desired
If OSEP only, 2-point scale is
sufficient
If more is desired, need a scale
with more points
ECO has drafted a rubric with a
7-point scale
Early Childhood Outcomes Center
43
If state only wants 3 OSEP
indicators

Transform data for each child into 2
categories at entry:
Q1. [Based on assessment and other information…]
Are child’s skills and behavior related to taking
action to meet needs at the level expected for his
or her age?
1. yes
2. no
Early Childhood Outcomes Center
44
If state only wants 3 OSEP
indicators

And 2 items, 2 categories each, at exit:
Q2a. [Based on assessment and other information…] Are
child’s skills and behavior related to taking action to
meeting his or her needs at the level expected for his or
her age?
1. yes
2. no
Q2b. [Based on assessment and other information…] Has
the child acquired new skills and behaviors related to
taking action to meet needs since entering the program?
1. yes
2. no
Early Childhood Outcomes Center
45
If state only wants 3 OSEP
indicators


Using the answers to those 3
questions, the state can classify
every child as a, b, or c.
However, transforming the data
differently allows the state to
do a lot more with the data.
Early Childhood Outcomes Center
46
If states want stronger evidence
statements: ECO Child Outcomes
Summary Form (under development)




Rubric that summarizes child status on
each outcome on a 7 point scale
NOT an assessment; rather a summary
of information on child functioning
Has gone through several revisions and
is still being revised
In an ideal world, would be extensively
researched before being released
Early Childhood Outcomes Center
47
A way to think about how children
are doing with regard to each
outcome
Movement away from
age-expected
Age-expected
skills and behavior
Movement toward
age-expected
6
5
4
3
Not Yet
7
Emerging
Positive Social Relationships
Somewhat
To what extent does this child show behaviors and skills
appropriate for his or her age across a variety of settings
and situations?
(Circle one number)
Completely
Child Outcomes Summary Form
2
1
Acquiring and Using Knowledge and Skills
7
6
5
4
3
2
__ No
1
6
5
4
3
2
To select an answer, think about the child’s functioning in these and closely related areas:

Taking care of basic needs (e.g., showing hunger, dressing, feeding, toileting, etc.)

Contributing to own health and safety (e.g., follows rules, assists with hand washing, avoids
inedible objects) (if older than 24 months.)

Getting from place to place (mobility) and using tools (e.g., forks, pencils, strings attached to
objects)
As indicated by assessments and based on observations from individuals in close contact with the child
(If these questions have been
answered previously):
Has the child shown any
new skills or behaviors
related to acquiring and
using knowledge and
skills since the last
outcomes summary?
__ Yes
__ No
Not Yet
7
Emerging
Taking Appropriate Action to Meet Needs
Somewhat
Completely
To select an answer, think about the child’s functioning in these and closely related areas:

Thinking, reasoning, remembering, and problem solving

Understanding symbols

Understanding the physical and social worlds
As indicated by assessments and based on observations from individuals in close contact with the child
To what extent does this child show behaviors and skills
appropriate for his or her age across a variety of settings
and situations?
(Circle one number)
Has the child shown any
new skills or behaviors
related to positive social
relationships since the
last outcomes summary?
__ Yes
Not Yet
Emerging
Somewhat
To what extent does this child show behaviors and skills
appropriate for his or her age across a variety of settings
and situations?
(Circle one number)
Completely
To select an answer, think about the child’s functioning in these and closely related areas:

Relating with adults

Relating with other children

Following rules related to groups or interacting with others (if older than 18 months.)
As indicated by assessments and based on observations from individuals in close contact with the child
(If these questions have been
answered previously):
1
(If these questions have been
answered previously):
Has the child shown any
new skills or behaviors
related to taking
appropriate action to
meet his/her needs since
the last outcomes
summary?
___ Yes
___ No
Definitions for Outcome Ratings
Completely
means:
7
Child shows behaviors and skills expected for his or her age in all
or almost all everyday situations that are part of the child’s
life.

6
Behavior and skills are considered typical for his or her age.
Between Completely and Somewhat
Child shows behavior and skills expected for his or her age some
of the time across situations.
Somewhat
means:

5


4
Emerging
means:
3
2
Behavior and skills are a mix of age appropriate and not
appropriate.
Behavior and skills might be described as more like those of a
slightly younger child.
Some behaviors or conditions might interfering with the
child’s capability to achieve age-expected behavior and skills.
Between Somewhat and Emerging
Child does not yet show behaviors and skills expected of a child of
his or her age in any situation. Child’s behaviors and skills include
immediate foundational skills upon which to build age expected
skills.

Behaviors and skills might be described as more like those of a
younger child.

Some behaviors or conditions might be interfering with the
child’s capability to achieve age-expected behavior and skills.
Between Emerging and Not Yet
Child does not yet show behaviors and skills expected of a child
his or her age in any situation. Child’s skills and behaviors also do
not yet include any immediate foundational skills upon which
to build age expected skills.
Not yet
means:
1

Child’s ways of forming and maintaining social relationships
might be described as more like those of a much younger
child.

Some behaviors or conditions might be seriously interfering
with the child’s capability to achieve age-expected behaviors
and skills.
Cover Sheet for Child Outcomes Summary
Child
Information
Name:
____________________________________________________________
Date of Birth ______/_____/______
Mon
Rating
Day
ID number ______________
Year
__ Joint __ Individual
process
People
involved in
summary
decision
Family
report
Name
Role
___ Included in group process
___ Collected separately
___ Incorporated into assessment(s)
Assessments:_______________________________________
_______________________________________
___ Not included
Comments:
Purpose of the Summary Form



Form serves as a way to reduce
complex information to a common scale
so the data can be aggregated
The form is not intended to provide
information to guide interventions
It does not capture the rich diversity of
children’s strengths and needs. It does
just the opposite: it is a technique for
data reduction.
Early Childhood Outcomes Center
52
Using assessment data in rating


ECO “cross-walking” common
assessments to the outcomes to
provide a consistent source of
information on which items/score
areas apply to which outcome.
ECO is NOT endorsing any
assessment instrument
Early Childhood Outcomes Center
53
Crosswalk: High Scope (COR) InfantToddler
Outcome 1: Positive Social
Relationships
II. Social Relations:
E. Forming an attachment to a
primary caregiver
F. Relating to unfamiliar adults
G. Relating to another child
H. Expressing emotion
I. Responding to feelings of
others
J. Playing with others
V. Communication and
Language
T. Participation in give-andtake communication
Outcome 2: Knowledge and
Skills
III. Creative Representation
K. Pretending
L. Exploring building and art
materials
M. Responding to and
identifying pictures and
photographs
V. Communication and
Language
R. Listening and responding
S. Communicating interest
nonverbally
U. Speaking
V. Exploring
W. Showing interest in stories,
rhymes, songs
VI. Exploration and Early
Logic
X. Exploring objects
Y. Exploring categories
Z. Developing number
understanding
AA. Exploring space
BB. Exploring time
Outcome 3: Action to Meet
Needs
I. Sense of Self
A. Expressing initiative
B. Distinguishing self from
others
C. Solving problems
D. Developing self-help skills
IV. Movement
N. Moving parts of body
O. Moving whole body
P. Moving with objects
Q. Moving to music
V. Communication and
Language
S. Communicating interest
nonverbally
Documentation


ECO is recommending that states
also institute a process for
documenting the basis for the
summary rating
Documentation could be reviewed
to insure ratings are being derived
properly (e.g., identify the need
for additional training)
Early Childhood Outcomes Center
55
Child Name:_____________________________
Date:______________________________
Evidence Supporting Summary Rating
Positive social relationships
 Formal assessment(s)  List name(s) and results:
Name of Assessment
Date
Summary of Relevant Results
Acquiring and using knowledge and skills
 Formal assessment(s)  List name(s) and results:
Name of Assessment
Date
Summary of Relevant Results
Taking appropriate action to meet his/her needs
 Formal assessment(s)  List name(s) and results:
Name of Assessment
Date
Summary of Relevant Results
How data could be transformed
Options:
 By a team (e.g., at IFSP meetings)
 By an individual (e.g., therapist)
 By computer based on item level data
scores
or
Options 1 and 2 will require training
All 3 require guidance, directions, rules as
to how the transformation gets
performed

What is typical development for each
outcome?
Early Childhood Outcomes Center
57
Identifying who made progress


Progress = moving up a point on
the scale in a subsequent rating,
e.g., 3 to 4
Progress= staying at the same
rating but having a “yes” for the
progress question (right column)
Early Childhood Outcomes Center
58
Use of the Summary Form


The Summary Form is free.
States can modify it, however, ECO
will be researching and supporting
(developing guidance materials,
training, etc.) one version. If the state
makes significant modifications, neither
the ECO research nor materials will
apply.
Early Childhood Outcomes Center
59
ECO Next Steps for Child
Outcomes Work


Developing additional crosswalks
mapping assessments to the three
outcomes
Working with publishers, test
developers and state staff to
develop guidance for how to
transform assessment results
(“scoring” algorithm)
Early Childhood Outcomes Center
60
ECO Next Steps for Child
Outcomes Work


Researching the Summary Form
(reliability, extent of training
required, etc.)
Possibly/probably revising the
tool based on pilot work and
additional research.
Early Childhood Outcomes Center
61
ECO Next Steps for Child
Outcomes Work

Additional information including
new versions of the Summary
Form will be posted on the web
site as they become available.
www.the-eco-center.org
Early Childhood Outcomes Center
62