Delaware Birth to Three Early
Intervention System Evaluation:
Child Outcomes
July 15, 2004
Conference Call Series: Measuring Child Outcomes
“Examples of using changes in growth rates to measure child outcomes in
Part C”
Rosanne Griff-Cabelli, Delaware Part C
[email protected]
Delaware Birth to Three
Child Outcomes Evaluation
Primary purposes:
1. To document changes in the children served by Part C
Programs
2. To collect data, systematically so that programs can
communicate their impact on children and families to
various audiences.
Impetus for Delaware’s Child Outcome Evaluation
• The State Budget Office, as part of the Interagency Management
Resource Committee (IRMC), requested child change data from
federal and state funded programs.
• Part C’s evaluation was linked through the IRMC to early
childhood programs, such as Head Start and Part B 619.
• The IRMC Annual Report includes Part C evaluation results,
distributed to Department Secretaries and Legislative Joint Finance
Committee.
• Part C formed an Ongoing Program Evaluation Committee (OPEC)
to coordinate, review and update components of child outcome
evaluation.
• Results are shared annually at ICC and Part C regional staff
meetings.
Outcome Areas: Focus and Scope
Outcome Measures:
• Describe changes in eligible children’s development and
functioning over time, while active in Part C programs by:
Developmental
Domains
• Motor
• Cognitive
Play and
Functional
Changes
Outcome Areas: Focus and Scope
Demographic Variables:
• Describe socio-economic factors of family
• Describe demographic information such as gender,
race/ethnicity, county of residence, age child started program,
primary language spoken
Design/Method
Population to Sample:
• Children selected are Part C eligible, have their first MDA
completed, and not older than two years and six months
• Random sample of every 15th child in New Castle County and
every 10th in Kent and Sussex Counties
• Children will have at least two points of evaluation,
approximately one year apart
• No service coordinator will have more than three children
actively involved in Child Change at any given point
Design/Method
Measurement Instruments:
• Bayley Scales of Infant Development II, administered as part of
MDA
• Fewell Play Assessment Scale (PAS) with adaptations for children
with more severe disabilities
• Demographic Information Form, administered by service
coordinators
• HOME [Home Observation for Measurement of the Environment]
Inventory – Birth to Three, administered by service coordinators
Design/Method
• All assessments are administered by Part C program staff.
• The PAS is a valuable addition to the battery of assessments
conducted with children.
• The PAS can be implemented by Part C program personnel but
additional resources are needed for annual training.
• A contract is in place with the University of Delaware, Center for
Disabilities Studies for data analysis and reports.
Delaware Part C Continuous Improvement
Monitoring and Evaluation Timeline
Evaluation Strategy
Family Focus Group
Family Survey
Service Coordinator
Focus Group
Provider Focus Group
Provider Survey
Child Outcomes
2001-02
Fall
Spring
2002-03
Spring
Spring
2003-04
Spring
2004-05
2005-06
Fall
2006-07
Fall
Fall
Spring
Spring
Spring
Spring
Fall
Spring
Ongoing
Analysis Using Proportional Change Index (PCI)
• Delaware’s Part C Child Change Evaluation is analyzed through a
contract with the University of Delaware, Center for Disabilities
Studies. The statistical procedure used is PCI.
• The PCI is a method devised by Mark Wolery of the University of
Kentucky that “compares children’s rate of development at
pretesting to their rate of development during intervention.”1
• The following link is to a paper on measuring developmental
change. It refers to the PCI and its strengths and drawbacks.
http://www.aare.edu.au/94pap/konzd94260.txt
1Mark
Wolery, Exceptional Children, Volume 50, Number 2, page 168, 1983
Analysis Using Proportional Change Index (PCI)
Design/Method
Developmental Gain / Pretest Developmental Age
Time in Intervention / Pretest Chronological Age
Divide second rate of development by first rate of development to
calculate PCI.
– Children who continue to develop at the same rate during intervention as
they did prior to intervention will have a PCI of 1.0
– Children whose rates of development are slower during intervention will
receive a PCI of less than 1.0
– Children whose rates of development accelerated during intervention will
receive a PCI greater than 1.0
Analysis Using Proportional Change Index (PCI)
Design/Method
Developmental Gain / Pretest Developmental Age
Time in Intervention / Pretest Chronological Age
Divide second rate of development by first rate of development to
calculate PCI.
– Children who continue to develop at the same rate during intervention as
they did prior to intervention will have a PCI of 1.0
– Children whose rates of development are slower during intervention will
receive a PCI of less than 1.0
– Children whose rates of development accelerated during intervention will
receive a PCI greater than 1.0
Analysis Using Proportional Change Index (PCI)
Strengths of the PCI
• It is not a measure solely of the actual number of months gained in intervention,
but takes into account the number of months in intervention and the child’s rate
of development at pretesting.
• Comparisons can be made between children at various levels of delay,
chronological age, and developmental age.
Drawbacks of the PCI
• Assumes children’s rates of development at pretesting would be stable in the
absence of intervention.
• Larger sample and three measures for each child would allow use of Growth
Curve Analysis.
Major Findings (sample in child change as of
January 2004)
The Children and Families:
 43 children tracked
 22 Northern CDW
 21 Southern CDW
 23 male; 20 female
 Disabilities:
 8 VLBW
 11 developmental delay
 24 VLBW and developmental delay
 Ethnicity
 15 African American, 18 Caucasian,
6 Hispanic, 4 bi-racial
Major Findings (as of January 2004)
The Children and Families:
 Family SES
 15 families below poverty level
 20 families above poverty level
 8 unknown
 Education Level of Mothers
 2 less than 8th grade
 4 less than high school
 7 high school graduate/GED
 15 some college or technical school
 9 college degree
 2 graduate degree
 4 unknown
Major Findings (as of January 2004)
Play Skills Development:
32 children with two PAS scores; 11 with three scores
 Time between PASs (M=12.9 months)
 Play skill changes (M=11.2 months; this is a .87 rate of
development)
 Expected rate of development (M=.72 or 8.64 months in
12 months time)
 1.21 times the rate of development expected for this group
of children
 There was no significant difference in rates of
development for girls or boys or children living in poverty
and those not living in poverty
Major Findings (as of January 2004)
Mental Skills Development:
32 children with 2 Bayley Mental scores; 11 with three
scores
 Time between Bayleys (M=12.3 months)
 Mental score changes (M=10.6 months; this is a .86
rate of development)
 Expected rate of development (M=.67 or 8.0 months in
12 months time)
 1.28 times the rate of development expected for this
group of children
 There remains no significant difference in rates of
cognitive development for girls and boys or children living
in poverty and those not living in poverty
Major Findings (as of January 2004)
Motor Skills Development:
32 children with 2 Bayley Motor scores; 11 with three scores
 Time between Bayleys (M=12.3)
 Motor score changes (M=10.3 months; this is a .84 rate of
development)
 Expected rate of development (M=.69 or 8.3 months in 12
months time)
 1.22 times the rate of development expected for this
group of children
 No differences in rates of development for children living
in poverty and those not; no differences between boys and
girls
Major Findings (as of January 2004)
Additional Findings:
 Of the 43 children, 36 had rates of development greater
than 1.0 for at least a one year period of time in
both motor and mental skills
Seven children had rates of development equal or greater
to 1.50 for at least a one year period of time in at least one
domain of development
Seven children had rates of development below 1.0 for at
least a one year period of time
After at least one year of services, only two children had
rates of development lower than that which they entered the
program (usually attributed to a progressive or degenerative
disorder)
Lessons Learned
• Early intervention seems to be having a positive effect on children’s play,
and mental and motor skill development across demographic groups.
• Child Change assessments are conducted systematically and program staff
are able to implement the Bayley assessments as part of MDA annual
timelines.
• Child Change results are reported in context of Birth to Three Evaluation
Plan.
• Recommendations
– Report the trend of children’s outcomes in three year intervals so that
a long-term trend can be determined; the cohort should always be
between 40-60 children.
– Efforts are in place to increase the number of males in the evaluation
to 55-60% (currently at 54%).
– Continued efforts need to be in place to encourage families to
participate in the evaluation since a sampling method is used.