Overview of the Early
College High School
Initiative Evaluation
Susan Cole
Mengli Song
Andrea Berger
American Institutes for Research
Presentation at the SREE 2010 Conference
March 4, 2010
About the ECHSI
 Started in 2002 by the Bill & Melinda Gates
Foundation
o Improve postsecondary access and success
o Provide opportunity to earn up to 2 years of
college credit
 In fall 2009, over 200 Early College Schools
(ECSs) open across the nation
About the ECHSI
 Five Core Principles define an Early College
School (ECS)
o Target population
o Partnership with a college or university
o Integrated academic plan
o Student supports
o Advocate for supportive policies
Evaluation Research Questions
1. What are the structural, organizational, and
instructional characteristics of ECSs?
2. What are the intermediate and long-term
outcomes for students attending ECSs,
especially for students traditionally
underserved by the postsecondary system?
Data Sources and
Analytic Methods
 Qualitative data
o Site visits (6 to 20 ECSs annually)
 Quantitative data
o School survey (entire ECS population annually)
o Student survey (35 schools and 2,102 students
in 2007-08)
 Analytic Methods
o School survey- descriptive statistics and
regression
o Student survey- hierarchical linear modeling
RQ1: Characteristics of ECSs
 157 ECSs in 2007-08 across 21 states and
DC
 65% have a 2-year public college partner
 53% are located on a college campus
 76% have admissions criteria
• Fewer than 100 students per grade on
average
Source: 2007-08 ECHSI school survey
RQ1: Who do ECSs serve?
 Minority- 67%
 Free or Reduced-Price Lunch Eligible- 59%
 Limited English Proficiency- 10%
 1st Generation College Going- 46%
Sources: 2007-08 ECHSI school survey; 2007-08 ECHSI student survey
RQ1: What are characteristics of
college classes in ECSs?
 91% of ECSs have at least some students
in college classes
 61% of students have taken at least one
college class
o Half of these college classes are in core academic
areas
o 66% of these college classes are taken on a
college campus
Source: 2007-08 ECHSI school survey; 2007-08 ECHSI student survey
RQ2: How are ECS students doing?
 On average, ECSs had average daily
attendance (ADA) rates over 94%.
 74% of ECS students were proficient in
ELA; 67% were proficient in math.
o ECSs outperformed districts in both ELA
and math proficiency rates by 7%.
Sources: 2007-08 ECHSI school survey; publicly available school data
RQ2: How are ECS students doing?
 Grade-to-grade progression rates
o 85% on average for 9th-to-10th-grade
o 87% for the 12th-to-graduation or grade 13
 Cumulative Promotion Index (CPI)
o For the 12 ECSs with data, the average CPI
was 66%
o Exceeded districts’ CPI by an average of
14%
Sources: 2007-08 ECHSI school survey; publicly available school data
RQ2: How are ECS students doing?
 Graduates earned about a semester to a
year of college credit while enrolled in the
ECS.
o ECS survey: about 8 college classes
o ECS transcripts: about 10 college
classes
Sources: 2007-08 ECHSI school survey; 2004-05 to 2007-08 student
transcript data
RQ2: How are ECS students doing?
 Most ECS students enrolled in college after
graduation.
 The ECS average for college enrollment is
equivalent to or exceeds national averages.
College Enrollment: ECS & National Average
ECS
(2006-07)
Postsecondary
Enrollment
4-year IHE
2-year IHE
National
(2003-04)
88%
72%
45%
43%
44%
28%
Sources: 2007–08 ECHSI school survey; NCES, 2007
RQ2: How are ECS students doing?
 Minor differences between subgroups on
various outcomes
 1st generation college-going students most
consistent gap
o Lower high school and college GPAs
o Lower educational aspirations
o Lower satisfaction with the ECS
Sources: 2007-08 ECHSI student survey; 2007-08 ECHSI school survey;
publicly available school data
RQ2 How are ECS students doing?
 Students at ECSs located on a college
campus had higher outcomes than ECSs at
other locations on:
o ADA (95% and 93%)
o 9th- to 10th-grade progression rates
(89% and 81%)
o Achievement proficiency rates (relative to
their district)
• ELA- 14% above and 1% below
• Math- 16% above and 1% below
Sources: 2007-08 ECHSI student survey; 2007-08 ECHSI school survey;
publicly available school data
Summary of findings
What do we know about ECS students?
 Students are largely from populations
underrepresented in postsecondary institutions.
 ECS students are outperforming districts on state
assessments.
 Students are accumulating college credit.
 On many outcomes, students in ECSs located on
college campuses are doing better than student
attending ECSs not located on college campuses.
Sources: 2007-08 ECHSI student survey; 2007-08 ECHSI school survey;
ECHSI Impact Study: Research Questions
1. Do ECS students have better outcomes than
they would have had at other high schools?
2. Do the effects of ECSs on student outcomes
differ for different types of schools?
3. Do the effects of ECSs differ for students with
different background characteristics?
ECHSI Impact Study: Design and Sample
 Overall Study Design: 3-year multisite RCT
 Sample:
– Sites: Up to 21 ECSs that use lottery-based
admission in at least one year between 2003-04
and 2007-08
– Students (up to 5 cohorts per school):




Treatment: offered space in ECS through lottery
Comparison: not offered space in ECS through lottery
Extant data sample: about 4,600 treatment and 6,700
comparison students across 21 sites
Student survey sample: about 1,800 students
ECHSI Impact Study: Outcome Measures
 High School Outcomes:
– High school persistence and graduation
– ACT/SAT test taking and performance
– College credit accrual while in high school
 College Outcomes:
– Highest educational attainment
– Degrees and certificates earned
ECHSI Impact Study: Data Collection
 Extant Data Collection:
– Data: lottery records, student and school
characteristics and student outcomes
– Sources: ECSs, districts, and/or subcontractors
– March 2010 through November 2011
 Student Survey:
– Designed to capture additional student
outcome information for all sites
– The only source for student outcomes for some
sites
– June to November 2011
ECHSI Impact Study: Analytic Methods for RQ1
RQ1 (Overall Impact):
 Intent-to-treat analyses
 Multilevel models with sites as random effects
(students nested within sites)
– HGLM for binary outcomes
– Multilevel survival analyses
– HGLM for multinomial outcomes
ECHSI Impact Study: Analytic Methods for RQs2 &3
RQ 2 (Differential Impact on different types of
schools):
 Assessed by incorporating measures of ECS
characteristics as site-level predictors into the
main impact model
RQ 3 (Differential Impact on different types of
students):
 Assessed by incorporating ECS-by-studentcharacteristic interactions as student-level
predictors into the main impact model