School Improvement Network
Impact Assessment:
Higher Engagement Schools
versus Lower Engagement Schools
Steven H. Shaha, PhD, DBA
Professor, Center for Public Policy and Administration
Independent Evaluator
December 2012
Overarching Research Question:
Does engagement
in PD 360 and Observation 360,
tools within the Educator Effectiveness System,
significantly affect
student success
and
school-wide metrics?
Sample Description
• High Video Utilizers
– 39 States
– 211 Districts
– 734 Schools
• Metrics:
• Educator
Engagement
• Student Success
• School Impacts
Study of Educator Engagement
• 32 data elements collected or computed
through PD 360 and Observation 360
• Contrasted higher engagement schools
versus lower engagement schools
– Improvement in percentages of students who tested
advanced or proficient in math and reading
– Classified into four quartiles
– Analyses of highest and lowest quartiles only
Metrics for Differentiating Advantages for
Higher Engagement Organizations:
•
•
•
•
Focus Objectives Set Up
Observations Performed
Percent Registered Users
Percent of Users in Communities
•
•
•
•
•
Minutes Viewed
Forums Viewed
Programs Viewed
Segments Viewed
Links Viewed
•
•
•
•
•
•
•
Follow-up Questions Answered
Reflection Questions Answered
Focus Objectives Set Up
Forums Posted
Downloaded Files
Uploaded Files
Participation in Communities
Leadership,
Implementation
and Accountability
Educator
Participation
Educator
Engagement
These are the 15 metrics for which higher engagement schools were
significantly higher than their lower engagement counterparts
Sample of Differentiating Metrics of
Utilization and Engagement
Passive participation (e.g. video viewing alone)
is LESS influential than Active engagement
Links Viewed
Observations Performed
70.3% advantage (p<.001)
Minutes Viewed
63.8% advantage (p<.001)
Follow-up Questions Answered
4.3% advantage (p<.001)
39.0% advantage (p<.001)
Sample of Differentiating Metrics of
Utilization versus Engagement
Downloaded Files
Passive participation (e.g. video viewing alone)
is LESS influential than Active engagement
Forums Viewed
30.5% advantage (p<.001)
Uploaded Files
79.5% advantage (p<.001)
Forums Posted
68.6% advantage (p<.001)
47.3% advantage (p<.001)
Who Cares?
Who cares if educators used it
more?
Did it make a difference
for kids and schools?
Study of Student Success
Student Success:
• Performance on standardized tests
– Percent either proficient or advanced in the
following subjects:
• Reading
• Math
Improved
Student Performance
4.9% gain for lower engagement schools (p<.01)
Improved
Student Performance
Closed the Gap:
267% advantage in gains for
higher engagement schools
(p<.001)
Nearly 4 times the impact
4.9% gain for lower engagement schools
18.0% gain for higher engagement schools
(p<.01)
(p<.001)
Improved
Student Performance
0.5% gain for lower engagement schools (p=ns)
Actually Important Gains:
For every 200 students, 1 more performed at proficient or advanced level
than in the previous year
Improved
Student Performance
Surpassed the Gap:
3,520% advantage in gains for
higher engagement schools
(p<.001)
36 times greater impact
0.5% gain for lower engagement schools (p=ns)
18.9% gain for higher engagement schools (p<.001)
Metrics of School Impact
• Performance on key indicators from
Internet (when publicly available) and
structured telephone interviews:
– Dropout Rates
– Student Discipline Rates
– Teacher Retention Rates
– College-Bound Rates
Improved
Dropout Rates
4.9% improvement for lower engagement schools
(p<.01)
For every 100 students, 5 fewer dropped out than in the previous year
Figures reflect rounding, projections reflect correct math.
Executive Summary
Higher engagement schools
began statistically equal,
then significantly
outperformed their
counterparts (p<.01)
Improved
Dropout Rates
309.1% advantage in
gains for higher
engagement schools
(p<.001)
4.9% improvement for lower engagement schools (p<.01)
20.0% improvement for higher engagement schools (p<.001)
For every 100 students, 20 fewer dropped out than in the previous year.
Improved
Student Discipline Rates
7.4% fewer disciplinary incidents for lower engagement schools (p<.01)
Y-Axis is inverted to reflect improvement as intuitively upward trend.
Figures reflect rounding, projections reflect correct math.
Improved
Higher engagement
Student Discipline Rates
schools significantly
Executive Summary
outperformed their
counterparts (p<.01)
351% advantage in gains for
higher engagement schools
(p<.001)
4 ½ times the impact
7.4% fewer disciplinary incidents for lower engagement schools (p<.01)
33.2% fewer disciplinary incidents for higher engagement schools (p<.001)
For every 100 students, 33 fewer problem students than in the previous year.
Improved
Teacher Retention Rates
1.7% more teachers stayed for lower engagement schools (p<.01)
Figures reflect rounding, projections reflect correct math.
Improved
Higher engagement
Teacher Retention Rates
schools significantly
Executive Summary
outperformed their
counterparts (p<.01)
65.9% advantage in
gains for higher
engagement schools
(p<.001)
Nearly twice the impact
1.7% more teachers stayed for lower engagement schools (p<.01)
2.8% more teachers stayed for higher engagement schools (p<.01)
For every 100 teachers, nearly 3 fewer left than in the previous year.
Figures reflect rounding, projections reflect correct math.
Improved
College-Bound Rates
Percentage of students
schools report as being
college-bound.
No decrease or gain for lower engagement schools (ns)
Figures reflect rounding, projections reflect correct math.
Executive Summary
Higher engagement
schools began statistically
equal, then significantly
outperformed their
counterparts (p<.01)
Improved
College-Bound Rates
Incalculable advantage in
gains for higher
engagement schools (p<.001)
No decrease or gain for lower engagement schools (ns)
9.6% improvement for higher engagement schools (p<.001)
For every 100 students, 10 more were college-bound than in the previous year.
Figures reflect rounding, projections reflect correct math.
Summary of School Impacts
• Dropout Rates
– Approx. 15 fewer dropouts per 100 students than lower engagement
school counterparts
• Student Discipline Rates
– Approx. 33 fewer students “in the office” per 100 students than for
lower engagement school counterparts
• Teacher Retention Rates
– Approx. 3 fewer teachers leaving per 100 teachers, which is 1 fewer
than for lower engagement school counterparts
• College-Bound Rates
– Approx. 10 more college-bound students per 100 students than for
lower engagement school counterparts
A Model for Educational Success
Leadership,
Implementation
and Accountability
Student Success
Educator
Participation
School Impacts
Educator
Engagement