Do You See What I See?
The Impact of School Accountability on
Perceptions of the School Environment
Emilyn Ruble
Whitesell
IESP Summer
Seminar
1
June 2014
Introduction
 Many districts are recognizing parent, teacher, and student perceptions
as important dimensions of school quality (e.g., NYC, Chicago, LA)
 Stakeholder consensus may also be an important aspect of school quality
 May reflect other positive school characteristics (proxy?)
 May help facilitate school improvement efforts
 This is one of few studies to compare the perceptions of stakeholders
within schools
 Between-group consensus
 Within-group consensus
2
Outline of presentation
1.
2.
3.
4.
5.
6.
7.
Background and motivation
Data and sample
Consensus measures and descriptive statistics
Relationship between consensus and passing rates
Impact estimate: effect of accountability on consensus
Mechanism (?)
Conclusion and next steps
3
What do we know about consensus?
 Why might consensus matter for schools?
 May contribute to success in school reforms (e.g., distributed leadership, CSRs)
 Associated with stakeholder satisfaction and performance (Griffith, 2000; O’Day, 1984)
 What school factors are associated with consensus?
 Consensus is lower in schools with recent principal changes (Griffith, 1999)
 Student and parent consensus (within groups) is higher in schools with homogenous
student bodies and lower student turnover (Griffith, 2000 )
 Most research focuses on principal/teacher perceptions of school leadership:
 Consensus is positively associated with more effective leadership, stronger academic
climates, higher teacher morale, lower teacher turnover, and higher student achievement
(Atwater & Yammarino, 1992; Atwater et al., 1998; Covay Minor et al., 2014; Goldring et
4
al., 2012; Urick, 2012)
Focus on views of school expectations and discipline
 Multiple dimensions of school climate, but high expectations and discipline
are particularly important
 Long line of research on the importance of discipline and expectations,
relating these domains to student achievement
 “Academic press,” effective schools research, and a tradeoff with community?:
(e.g., Coleman, Hoffer, & Kilgore, 1982; Purkey & Smith, 1983; Rosenholtz, 1985; Shouse, 1996;
Phillips, 1997)
 “No excuses” charter schools, with zero-tolerance discipline and high expectations:
(e.g., Carter, 2000; Whitman, 2003; Thernstrom & Thernstrom, 2008; Wilson, 2008, Angrist et al.,
2010; Abdulkadiroglu et al., 2011; Angrist, Pathak, & Walters, 2011; Fryer, 2011; Dobbie & Fryer,
2011, 2013)
5
Conceptual framework
 Prior research finds school accountability increases student achievement
(e.g., Car noy & Loeb, 2002; Hanushek & Raymond, 2005; Figlio & Rouse, 2006;
Chiang, 2009; Winters, Trivitt, & Greene, 2010; Rockoff & Tur ner, 2010;
Winters & Cowen, 2012)
 Consensus is a potential mechanism through which accountability affects
test scores
 How might accountability affect perceptions and consensus?
 Information / stigma
 Changes in school practices
6
New York City data, 2007-2012
 NYC Learning Environment Surveys
 Responses from parents, teachers, and students (only in grades 6 -12)
 High coverage and nearly one million respondents per year
 NYC School Progress Reports
 Letter grades (A-F) assigned to schools
 Grades based on student progress (60%), student performance (25%), and the school
environment (15%)
 New York State School Report Cards
 School factors: size, grade span, teacher turnover, student attendance
 Average teacher characteristics: experience, qualifications
 Average student characteristics: performance, race, poverty, educational needs
7
Including schools with at least 10 survey responses in
the relevant stakeholder group
Count of schools by N of survey responses, 2007-2012
1800
1600
1400
1200
1000
800
600
400
200
0
Total schools by school type and N of survey responses, 2012
700
600
500
400
300
200
100
2010
2011
Total Schools
>=10 Parent responses
>=10 Teacher responses
>=10 Student responses
2012
Total Schools
>=10 Parent
responses
>=10 Teacher
responses
Elem
Middle
K8
High
2009
Elem
Middle
K8
High
2008
Elem
Middle
K8
High
2007
Elem
Middle
K8
High
0
>=10
Student
responses
8
Significant overlap in questions across stakeholder groups
Domain
School safety
Teacher
I am safe at my school.
Parent
My child is safe at school.
Bullying
Students in my school are
often threatened or bullied.
Students threaten or bully
other students.
Discipline
Order and discipline are
maintained at my school.
Adults at my school are often
disrespectful to students.
Discipline is enforced fairly at
my child’s school.
School staff are disrespectful to Teachers in my school treat students with
students.
respect.
My child’s school is clean.
My school is kept clean.
The school has high
 I need to work hard to get good
expectations for my child.
grades at my school.
 My teachers expect me to continue
my education after high school.
Teacher respect for
students
Cleanliness
Expectations
The school is kept clean.
 My school sets high
standards for student
learning.
 My school sets high
standards for student
work in their classes.
Student
 I am safe in my classes.
 I am safe in the hallways, bathrooms,
and locker rooms.
 I am safe on school property outside
my school building.
Students threaten or bully other students
at school.
Discipline in my school is fair.
Others: Gang activity, resources for conflict resolution, course / activity offerings, goal setting, academic counseling, parent communication
9
Average stakeholder perceptions, 2007
Average stakeholder perception across all schools, 2007
4
3.5
3
2.5
2
1.5
1
0.5
0
Safety
Gangs
Bullying
Discipline
Parent
Teacher
Respect
Student
Cleanliness
Offerings
Expectations
10
Perceptions of school expectations over time
Average perceptions of school expectations, across all schools
4.0
100%
3.8
95%
3.6
90%
3.4
85%
3.2
80%
3.0
75%
2.8
70%
2.6
65%
2.4
60%
2.2
55%
2.0
50%
2007
2008
Parents Average Rating
Parents % Favorable
2009
2010
Teachers Average Rating
Teachers % Favorable
2011
2012
Students Average Rating
Students % Favorable
11
Perceptions of school discipline over time
Average perceptions of school discipline, across all schools
4.0
100%
3.8
95%
3.6
90%
3.4
85%
3.2
80%
3.0
75%
2.8
70%
2.6
65%
2.4
60%
2.2
55%
2.0
50%
2007
2008
2009
2010
2011
Parents Average Rating
Teachers Average Rating
Students Average Rating
Parents % Favorable
Teachers % Favorable
Students % Favorable
2012
12
Measuring between-group consensus
 Survey responses range from 1-4, with 4 reflecting more favorable views
 Difference between average perceptions of different stakeholder groups,
within schools – “mismatch” variable
 Parent-teacher difference
 Parent-student difference
 Teacher-student difference
 In regressions, use absolute value of difference, so a larger value indicates
less consensus (greater mismatch)
 Calculated separately for each domain and also averaged across domains
13
Between-group consensus varies by domain
Average difference in average stakeholder views, within schools, 2007
0.8
0.6
0.4
0.2
0
Safety
Gangs
Bullying
Discipline
Respect
Cleanliness
Offerings
Expetations
-0.2
-0.4
-0.6
Parent-Teacher
Parent-Student
Teacher-Student
Note: Consensus values are only included if a school has at least 10 respondents from the relevant stakeholder group.
14
Measuring within-group consensus
 Responses range from 1-4, with 4 reflecting more favorable views
 Standard deviation of stakeholder perceptions within schools (variation)
 Teacher variation
 Parent variation
 Student variation
 Larger values reflect greater variation (less consensus)
 Calculated separately for each domain and also averaged across domains to
create an index measure
15
Relatively high variation in student perceptions
Average standard deviation in stakeholder views, within schools, 2007
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Safety
Gangs
Bullying
Discipline
Parent
Teacher
Respect
Student
Cleanliness
Note: Consensus values are only included if a school has at least 10 respondents from the relevant stakeholder group.
Offerings
Expectations
16
Relationship between within-group variation and
average group perceptions, 2007
Discipline
3
2
1
1
2
3
Average stakeholder responses
4
4
Expectations
0
.5
1
Standard deviation in stakeholder responses
Teacher
Parent
1.5
1
Standard deviation in stakeholder responses
1.5
Student
Teacher
Note: Consensus values are only included if a school has at least 10 respondents from the relevant stakeholder group.
Parent
Student
17
.6
.4
0
0
.2
.2
.4
.6
Proficiency rate - ELA
.8
.8
1
1
Relationship between within-group variation and
standardized test passing rates
.2
.4
.6
.8
1
Standard deviation in stakeholder responses (index)
Teacher
Parent
Student
1.2
.2
.4
.6
.8
1
Standard deviation in stakeholder responses (index)
Teacher
Note: Consensus values are only included if a school has at least 10 respondents from the relevant stakeholder group.
Parent
1.2
Student
18
.6
.4
0
0
.2
.2
.4
.6
Proficiency rate - ELA
.8
.8
1
1
Relationship between between-group differences and
standardized test passing rates
0
.2
.4
.6
.8
Difference in average stakeholder responese (index)
Parent-Teacher
Parent-Student
Teacher-Student
1
0
.2
.4
.6
.8
Difference in average stakeholder responses (index)
Parent-Teacher
Parent-Student
Note: Consensus values are only included if a school has at least 10 respondents from the relevant stakeholder group.
1
Teacher-Student
19
Estimating the relationship between
consensus and exam passing rates
(1)PASSINGst = β0 + CONSENSUSst’ β1 + Sst’β2 + Tst’β3 + Xst’ β4 + αs + φt + εst
 PASSING is a passing rate (math or ELA) on standardized tests
 CONSENSUS is a vector of consensus measures for school s in year t
 Within-group standard deviation
 Between-group difference
 S – vector of school characteristics: teacher turnover rate, student attendance rate, school size
 T – vector of teacher characteristics: % with fewer than 3 years experience, % with Master’s degrees
 X – vector of student characteristics: poverty, race, LEP
 School (αs) and year (φt ) effects
20
Relationship between consensus about school
expectations and ELA passing rate
Outcome: ELA passing rate
Within-group SD
Parent
(1)
(2)
-0.078***
(0.015)
Teacher
-0.018*
(0.009)
Student
(3)
(4)
(5)
(6)
Parent-Student
0.002
(0.012)
Teacher-Student
8,234
0.943
4,634
0.952
4,627
0.953
(9)
-0.076***
(0.019)
0.004
(0.012)
-0.031
(0.020)
-0.009
(0.007)
8,250
0.943
(8)
-0.073***
(0.018)
-0.000
(0.011)
-0.044** -0.033*
(0.019)
(0.019)
Between-group difference
Parent-Teacher
Observations
R-squared
(7)
4,629
0.952
8,232
0.943
-0.015**
(0.008)
4,632
0.952
0.001
(0.010)
0.008
(0.013)
-0.017*
(0.010)
4,627
0.952
-0.002
(0.011)
0.016
(0.013)
-0.013
(0.010)
4,627
0.953
Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1
Note: All models include school and year effects. Models also control for student demographics, student attendance rate, teacher experience and education.
21
Relationship between consensus about school
discipline and ELA passing rate
Outcome: ELA passing rate
Within-group SD
Parent
(1)
(2)
(3)
(4)
-0.017
(0.021)
-0.050***
(0.016)
-0.001
(0.010)
-0.008
(0.021)
-0.041***
(0.014)
Teacher
-0.011
(0.007)
Student
Between-group difference
Parent-Teacher
(5)
(6)
-0.019***
(0.004)
-0.015*
(0.008)
Teacher-Student
8,250
0.943
8,234
0.943
4,634
0.952
4,627
0.952
(8)
(9)
-0.064***
(0.017)
0.004
(0.010)
0.017
(0.023)
Parent-Student
Observations
R-squared
(7)
4,629
0.952
8,232
0.943
-0.007
(0.005)
4,632
0.952
-0.013*** -0.013***
(0.005)
(0.005)
-0.011
-0.022**
(0.008)
(0.009)
-0.002
0.000
(0.006)
(0.006)
4,627
4,627
0.952
0.953
Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1
Note: All models include school and year effects. Models also control for student demographics, student attendance rate, teacher experience and education.
22
Effect of accountability on consensus
(2) CONSENSUSst = β0 + β1 ACCTst-1+ Sst β2 + Tst β3 + Xstβ4 + αs + ϴst + st
 CONSENSUS is a consensus measure for school s in year t
 ACCT is an accountability measure for school s in year t-1
 Graded
 Letter grade
 Grade increase / grade decrease
 School effects
 School-specific time trends in lieu of year effects
23!!!aA
Effect of accountability on standard deviation in
stakeholder responses about expectations, 2007-2012
Outcome: SD in responses
(1)
Graded last year
-0.057***
(0.002)
Last year – A
Last year – B
Last year – C
Last year – D
Last year – F
Parent
(2)
Lag grade decrease
8973
0.665
(4)
-0.003
(0.004)
-0.062***
(0.003)
-0.059***
(0.003)
-0.047***
(0.003)
-0.053***
(0.005)
-0.044***
(0.007)
Lag grade increase
Observations
Adj. R-squared
(3)
8973
0.667
Teacher
(5)
(6)
(7)
0.070***
(0.004)
-0.004
(0.005)
-0.005
(0.005)
-0.001
(0.005)
0.001
(0.008)
-0.001
(0.012)
0.004*
(0.003)
0.005*
(0.003)
8973
0.634
8304
0.620
8304
0.620
Student
(8)
(9)
0.074***
(0.004)
0.070***
(0.004)
0.065***
(0.005)
0.063***
(0.007)
0.066***
(0.010)
0.008**
(0.004)
0.002
(0.004)
8304
0.620
5092
0.572
5092
0.573
Note: All models include school effects and school-specific time trends. Models also include student demographics (poverty, race), student attendance rate,
school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from relevant stakeholder groups.
0.005
(0.004)
-0.006
(0.004)
5092
0.524
24
Effect of accountability on standard deviation in
stakeholder responses about discipline, 2007-2012
Outcome: SD in responses
(1)
Graded last year
-0.024***
(0.003)
Last year – A
Last year – B
Last year – C
Last year – D
Last year – F
Parent
(2)
Lag grade decrease
8973
0.593
(4)
-0.011**
(0.005)
-0.026***
(0.003)
-0.024***
(0.003)
-0.019***
(0.004)
-0.030***
(0.005)
-0.019**
(0.008)
Lag grade increase
Observations
Adj. R-squared
(3)
8973
0.593
Teacher
(5)
(6)
(7)
0.033***
(0.003)
-0.013**
(0.006)
-0.014**
(0.006)
-0.012*
(0.006)
0.003
(0.010)
0.005
(0.015)
0.003
(0.003)
-0.003
(0.003)
8973
0.587
8304
0.612
8304
0.612
Student
(8)
(9)
0.033***
(0.004)
0.034***
(0.004)
0.035***
(0.004)
0.028***
(0.006)
0.029***
(0.009)
0.012**
(0.005)
-0.000
(0.005)
8301
0.612
5092
0.616
5092
0.616
Note: All models include school effects and school-specific time trends. Models also include student demographics (poverty, race), student attendance rate,
school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from relevant stakeholder groups.
0.001
(0.003)
-0.005
(0.003)
5092
0.604
25
Effect of accountability on difference in average
stakeholder responses about expectations, 2007-2012
Outcome: Difference (AV)
(1)
Graded last year
-0.026***
(0.005)
Last year – A
Last year – B
Last year – C
Last year – D
Last year – F
Parent-Teacher
(2)
Lag grade decrease
8293
0.487
(4)
-0.124***
(0.006)
-0.017***
(0.006)
-0.025***
(0.006)
-0.025***
(0.007)
-0.056***
(0.010)
-0.045***
(0.016)
Lag grade increase
Observations
Adj. R-squared
(3)
8293
0.488
Parent-Student
(5)
(6)
-0.121***
(0.007)
-0.131***
(0.006)
-0.110***
(0.007)
-0.133***
(0.010)
-0.117***
(0.016)
0.008
(0.005)
-0.010*
(0.005)
8293
0.485
5052
0.576
5052
0.578
0.009*
(0.005)
0.006
(0.006)
5052
0.507
Teacher-Student
(7)
(8)
-0.139***
(0.009)
-0.119***
(0.011)
-0.139***
(0.010)
-0.141***
(0.011)
-0.179***
(0.017)
-0.205***
(0.025)
4638
0.589
4638
0.592
Note: All models include school effects and school-specific time trends. Models also include student demographics (poverty, race), student attendance rate,
school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from relevant stakeholder groups.
(9)
0.023***
(0.008)
-0.002
(0.009)
4638
0.557
26
Effect of accountability on difference in average
stakeholder responses about discipline, 2007-2012
Outcome: Difference (AV)
(1)
Graded last year
-0.104***
(0.010)
Last year – A
Last year – B
Last year – C
Last year – D
Last year – F
Parent-Teacher
(2)
Lag grade decrease
8293
0.612
(4)
0.044***
(0.008)
-0.080***
(0.012)
-0.100***
(0.011)
-0.106***
(0.013)
-0.174***
(0.019)
-0.194***
(0.030)
Lag grade increase
Observations
Adj. R-squared
(3)
8293
0.614
Parent-Student
(5)
(6)
(7)
-0.001
(0.013)
0.055***
(0.010)
0.048***
(0.009)
0.027**
(0.011)
0.031**
(0.016)
0.027
(0.024)
0.018*
(0.009)
0.006
(0.010)
8293
0.604
5052
0.592
5052
0.592
Teacher-Student
(8)
(9)
0.011
(0.016)
-0.014
(0.014)
0.025
(0.016)
-0.009
(0.024)
-0.071**
(0.036)
-0.002
(0.008)
-0.003
(0.008)
5052
0.588
4638
0.497
4638
0.499
Note: All models include school effects and school-specific time trends. Models also include student demographics (poverty, race), student attendance rate,
school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from relevant stakeholder groups.
-0.009
(0.011)
-0.017
(0.012)
4638
0.497
27
Translating the effect: percent change and effect size
Outcome: Expectations
Outcome: Discipline
Native units
% change
Effect size
Native units % change
Effect size
Parent SD
-0.057***
-8.77%
-0.384
-0.024***
-3.52%
-0.175
Teacher SD
-0.003
-
-
-0.011**
-1.54%
-0.055
Student SD
0.070***
10.69%
0.672
0.033***
3.93%
0.321
Parent-Teacher diff
-0.026***
-10.65%
-0.137
-0.104***
-20.43%
-0.258
Parent-Student diff
-0.124***
-54.49%
-0.642
0.044***
9.12%
0.184
Teacher-Student diff
-0.139***
-38.66%
-0.502
-0.001
-
-
28
How does accountability affect consensus?
 To increase consensus, accountability must affect perceptions of different
stakeholders differently (in direction or magnitude)
 Why might this happen – difference in salience of information / school changes?
 Parent views may be more responsive to information, as they have less direct contact with the
school than teachers or students
 Teacher and student views may be more responsive to school-level changes that result from
accountability pressure
 The elasticity of stakeholder views with respect to accountability information
might vary for different school conditions
 Views about “academic” factors may be more responsive accountability information if
stakeholders think grades reflect academic quality
29
Effect of accountability on average perceptions
(3) PERCEPTIONst = β0 + β1 ACCTst-1+ Sst β2 + Tst β3 + Xstβ4 + αs + ϴst + st
 PERCEPTION is an average perception measure (parents, teachers, or students) for school s in year t
 ACCT is an accountability measure for school s in year t-1
 Graded
 Letter grade
 Grade increase / grade decrease
 School effects
 School-specific time trends in lieu of year effects
30!!!aA
Effect of accountability on average stakeholder
perceptions about school expectations, 2007-2012
Outcome: Avg perception
Graded last year
(1)
0.090***
(0.010)
Last year – A
Parent
(2)
(4)
0.105***
(0.015)
0.095***
(0.011)
0.097***
(0.010)
0.074***
(0.012)
0.089***
(0.018)
0.058**
(0.026)
Last year – B
Last year – C
Last year – D
Last year – F
Lag grade increase
Lag grade decrease
Observations
Adj. R-squared
(3)
4627
0.806
4627
0.807
Teacher
(5)
(6)
(7)
-0.052***
(0.007)
0.078***
(0.017)
0.098***
(0.016)
0.117***
(0.018)
0.165***
(0.027)
0.195***
(0.052)
-0.002
(0.006)
0.003
(0.007)
4627
0.790
4627
0.780
4627
0.782
Student
(8)
(9)
-0.057***
(0.008)
-0.053***
(0.007)
-0.048***
(0.008)
-0.046***
(0.010)
-0.048***
(0.017)
-0.027**
(0.011)
0.003
(0.013)
4627
0.772
4627
0.808
4627
0.808
Note: All models include school effects and school-specific time trends. Models also include student demographics (poverty, race), student attendance rate,
school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from all stakeholder groups.
0.002
(0.004)
0.005
(0.005)
4627
0.797
31
Effect of accountability on average stakeholder
perceptions about school discipline, 2007-2012
Outcome: Avg perception
Graded last year
(1)
0.087***
(0.011)
Last year – A
Parent
(2)
(4)
0.185***
(0.026)
0.096***
(0.012)
0.088***
(0.012)
0.072***
(0.014)
0.090***
(0.019)
0.055**
(0.027)
Last year – B
Last year – C
Last year – D
Last year – F
Lag grade increase
Lag grade decrease
Observations
Adj. R-squared
(3)
4627
0.755
4627
0.756
Teacher
(5)
(6)
(7)
0.048***
(0.012)
0.138***
(0.029)
0.180***
(0.028)
0.192***
(0.031)
0.285***
(0.046)
0.350***
(0.092)
-0.006
(0.007)
-0.000
(0.008)
4627
0.741
4627
0.764
4627
0.767
Student
(8)
(9)
0.042***
(0.013)
0.048***
(0.012)
0.050***
(0.014)
0.067***
(0.019)
0.041
(0.037)
-0.047***
(0.018)
-0.011
(0.021)
4627
0.755
4627
0.797
4627
0.797
Note: All models include school effects and school-specific time trends. Models also include student demographics (poverty, race), student attendance rate,
school enrollment, and teacher characteristics (education and experience). Schools only included if they have at least 10 responses from all stakeholder groups.
-0.005
(0.009)
0.004
(0.010)
4627
0.794
32
Discussion and policy implications
 Stakeholder consensus is an overlooked but potentially important aspect
of school quality
 In NYC, consensus measures are related to other positive school attributes
 Favorable views of the environment
 Math and ELA passing rates
 The introduction of the letter grade school accountability system in NYC
seems to have increased consensus
 Contributes to ongoing debates about measuring school quality and the
impact of accountability systems
33
What I’m thinking about next…
 Is consensus always good? Maybe not...
 Interact consensus with average perceptions
 Control for perceptions when estimating the relationship between consensus and outcomes
 Continue to explore how accountability affects consensus – individual models
 Restrict to individuals who remain in school; lagged perceptions or individual FE
 Consider teacher exit and principal turnover – as outcomes or controls
 General modeling:
 What models do I believe? Worth looking at the impact of grades and grade changes, or just grading?
 Estimate the impact of an F – regression discontinuity
 Compare responses to letter grade and AYP information
 Can I distinguish between information/stigma and changes in school practices?
 Hypothesis: parents respond to information, while teachers and students are more likely to respond to
changes in variables to measure school changes
 Additional measures: resources (expenditures, pupil-teacher ratios); teacher perceptions of specific
practices (e.g., data use, professional development, leadership support )
34
Thanks!
[email protected]
The research reported here was supported by the Institute of Education Sciences,
U.S. Department of Education, through Grant R305B080019 to New York
University. The opinions expressed are those of the authors and do not represent
views of the Institute or the U.S. Department of Education.

AECI`S MANAGEMENT APPROACH AND STATUS INTERACTION

The Evolving Scholarly Record: Framing the Landscape