TECHNICAL APPENDICES
Changing How High Schools Serve Black and
Latino Young Men:
A Report on NYC’s Expanded Success Initiative
Adriana Villavicencio
Sarah Klevan
David Kang
May 2015
CONTENTS
Appendix A: Matching Process for Comparison Schools........................................................................ 1
Appendix B: Analytic Method for Focus Group Transcripts.................................................................... 4
Appendix C: Fidelity Rubric ........................................................................................................................ 5
Appendix D: Intensity Rubric...................................................................................................................... 6
Appendix E: Focus Group Protocol for Principals and ESI Liaisons at ESI Schools ............................ 7
Appendix F: “Supplemental Form” for Principals and ESI Liaisons ....................................................... 9
Appendix G: Focus Group Protocol for Teachers at ESI Schools ........................................................ 10
Appendix H: Focus Group Protocol for ESI Students ............................................................................ 12
Appendix I: Interview Protocol for Principals at Comparison Schools................................................. 14
Appendix J: Focus Group Protocol for Teachers at Comparison Schools .......................................... 16
Appendix K: Comparison Rubric ............................................................................................................. 18
Appendix L: Survey Constructs and Items.............................................................................................. 20
Appendix M: Survey Construct Analysis ................................................................................................. 22
Appendix N: Survey and Academic Impacts ........................................................................................... 23
Appendix O: Student Discipline Outcomes ............................................................................................. 31
1
APPENDIX A: MATCHING PROCESS FOR COMPARISON
SCHOOLS
This appendix describes the matching process used to identify an appropriate set of comparison schools for use in our
report evaluating Year 2 of the Expanded Success Initiative, Changing How High Schools Serve Black and Latino Young
Men. As described in Chapter 2 of the report, selecting schools similar to ESI schools to serve as a comparison group
allows us to differentiate the effect of ESI from the effect of other factors that are simultaneously influencing schools
district-wide.
Even though the method we used for our academic analysis, Comparative Interrupted Time Series (CITS), controls
for student characteristics and pre-treatment differences in outcomes between ESI and comparison schools, better
matches mean that the comparison schools provide a more convincing counterfactual for ESI schools, since schools
with similar student populations with similar academic performance are likely to be affected by similar policies. They
are also likely to respond to any future policy shocks in similar ways (Somers et al., 2013).
For each ESI school, we selected two comparison schools that were statistically most similar in terms of the
demographic characteristics and prior academic performance of incoming 9th grade students. Since CITS analysis uses
prior years of academic data to control for trajectories in academic performance, schools were also matched by the
trajectory of academic outcomes for four prior cohorts of 9th grade students.
We used all 80 comparison schools in our academic impact analysis. We attempted to recruit 40 comparison schools
to take the ESI survey. Twenty-three schools agreed to administer the survey. Thirteen of the 23 schools had also
taken the Year 1 survey.
Below, we describe the pool of schools from which we selected comparison schools, describe the characteristics by
which schools were matched, and specify the statistical model we used to conduct the match. We then assess the
quality of the match by comparing characteristics of students in the 40 ESI and 80 comparison schools.
Selection Rounds
Due to data availability, the two comparison schools were selected at different times, using similar procedures. We
first selected 40 schools that would be used as a comparison group for our academic impact study (based on
administrative records), and would also act as the comparison group of schools in the survey analysis. In the second
round, we selected an additional 40 schools for the academic impacts study. The survey schools were matched using
data up to school year 2010-2011, while the second group of schools used data from the 2011-2012, the school year
immediately before ESI began. The procedure described below is the process we used for the second round. The
process for the first round was similar.
Comparison Pool Criteria (Second Round)
Schools in Districts 75 (Special Education) and 84 (Charter schools) were eliminated as potential matches, as were the
City’s nine specialized high schools (to which students are admitted on the basis of a single standardized test). Because
we used CITS analysis, which requires a historical baseline, we restricted the pool to schools that enrolled 9th grade
students in each of the previous five school years.
The pool was then restricted to schools meeting one of the following requirements:



Schools that were invited to and applied for ESI, but were not selected;
Schools that were invited to ESI but did not apply, with 15 or more Black and Latino male students in 20092010, 2010-2011, and 2011-2012; or
Schools that had more than 25 Black and Latino male students and more than 25 percent Black and Latino
male students in the 2011-2012 school year.
This left us with a pool of 328 potential matches.1 We matched schools with similar admissions requirements and,
where possible, grade configurations.
2
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
We measured the similarity of ESI and potential comparison schools along the following characteristics:2














Number of Black and Latino male 9th grade students enrolled in 2011-2012;
Number of Black 9th grade students enrolled in 2012;
Number of Latino 9th graders enrolled in 2012;
Number of male 9th graders enrolled in 2012;
Number of Black and Latino male 9th grade students enrolled in 2012 who received free/reduced price lunch
or attended a universal-feed middle school;
Average attendance rate for Black and Latino male 9th grade students enrolled in 2012;
Average 8th grade NYS math scale score for Black and Latino male 9th grade students enrolled in 2012;
Average 8th grade NYS ELA scale score for Black and Latino male 9th grade students enrolled in 2012;
Number of Black and Latino male 9th grade students enrolled in 2012 not born in the United States;
Number of Black and Latino male 9th grade students enrolled in 2012 classified as English Language Learners
in 8th or 9th grade;
Estimated slope of Black and Latino male 9th grade students’ total credits earned in the prior five years;
Estimated intercept of Black and Latino male 9th grade students’ total credits earned in the prior five years;
Estimated slope of all students’ 9th grade credits earned in the five years prior to implementation; and
Estimated intercept of all students’ 9th total credits earned in the five years prior to implementation.
Matching Method
Comparison schools were selected by minimizing a weighted Euclidian distance (over the fourteen matching
dimensions) between ESI and potential comparison school. Since most high schools in NYC have limited unscreened
admissions policies, limited unscreened ESI schools were matched “without replacement.” Schools with other
admissions policies were initially matched “with replacement” (comparison schools could potentially be matched with
more than one ESI school). Duplicates were resolved by selecting the combination of matches that reduced total
Euclidian distance.
Match Results
To ensure that students in the comparison schools were, in fact, suitable for representing the counterfactual outcomes
for students in ESI schools, we conducted simple tests for baseline equivalency. We calculated effect size differences
for 9th and 10th grade Black and Latino male students in ESI schools and potential comparison schools before the
introduction of ESI. We considered effect sizes of 0.25 or smaller to be acceptable (the smaller the effect size
difference, the more the two groups are considered similar3). See Tables A-1 and A-2 on the next page for results of
our matching tests.
3
Table A-1: Baseline Equivalence for Outcomes Prior to ESI for 9th Grade
Black and Latino Male Students, 2011-2012
Comparison
Schools
ESI Schools
On-Track (%)b
Academic GPA (Weighted)
Chronic Absentee (%)c
Academic Credits Attempted
Academic Credits Earned
Attendance (%)
Students per School
Number of Schools
Effect Size
Differencea
57.3
69.2
50.4
68.5
0.14
0.05
24.8
10.9
8.4
90.0
67.9
40
29.4
10.9
8.2
88.5
64.3
80
-0.10
0.02
0.05
0.09
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: a Effect size difference was calculated using "Hedges's G," defined as the difference in means divided by the
pooled standard deviation. Values of 0.25 or greater are considered not to be equivalent. b On-track for 9th grade is
defined as passing one Regents exam and earning at least 10 total credits. c Chronic absentees are students absent
for 10 percent or more of school days.
Table A-2: Baseline Equivalence for Outcomes Prior to ESI for 10th Grade
Black and Latino Male Students, 2011-2012
ESI Schools
On-Track (%)b
Academic GPA (Weighted)
Chronic Absentee (%)c
Academic Credits Attempted
Academic Credits Earned
Attendance (%)
Students per School
Number of Schools
Comparison
Schools
Effect Size
Differencea
53.4
64.7
49.4
63.9
0.08
0.04
32.6
10.5
7.7
86.6
67.9
40
35.8
10.4
7.6
85.2
64.3
80
-0.07
0.00
0.02
0.07
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: a Effect size difference was calculated using "Hedges's G," defined as the difference in means divided by the
pooled standard deviation. Values of 0.25 or greater are considered not to be equivalent. b On-track for 10th grade is
defined as passing two Regents exams and earning at least 20 total credits. c Chronic absentees are students absent
for 10 percent or more of school days.
4
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX B: ANALYTIC METHOD FOR FOCUS GROUP
TRANSCRIPTS
1. Reflection memos. Each researcher completed a reflection memo for each focus group he or she
conducted, to summarize educators’ and students’ responses to our questions.
2. Analytic memos. After all of the focus groups were completed, the lead researchers developed a
list of 11 analytic themes (e.g., sustainability, college and career) to analyze further, based on
questions from the interview and focus group protocols. Each researcher then wrote analytic
memos for one or two of those themes by reviewing reflection memos from all 40 schools and
tallying subthemes related to the broader topic.
3. Transcript coding. The lead researchers developed a list of codes with which to analyze the
transcript data, based on 1) protocol questions, 2) the ESI theory of action, and 3) the subthemes
identified by the analytic memos. The initial list was revised once and then used to code all 120
interviews and focus groups.
4. Coding memos. After all transcripts were coded, each researcher wrote a coding memo, noting
and recommending codes to be further analyzed based on frequency (e.g., college and career) or
salience to the ESI theory of action (e.g., DOE support, sustainability), or both (e.g., race,
gender, relationships).
5. Thematic outlines. Based on the coding memos, the team collectively decided on a list of codes to
further analyze. Each researcher then conducted a closer reading of the transcript data related to
a particular code and wrote a thematic outline about that code (e.g., college and career
supports). Researchers also wrote brief memos which summarized the highlights of each
thematic outline. Two lead researchers combined these thematic outlines and memos into an
annotated outline, upon which this report is based.
5
APPENDIX C: FIDELITY RUBRIC
Category
Data Sources
3: High Fidelity
2: Achieving
Fidelity
Programs represent
only two of the
design area
1: Low Fidelity
Evidence
Representation
of Three Design
Areas
Supplemental form
Programs represent
each design area at
least once
Evidence of CRE
Training
Supplemental form
Principal transcript: Does
your school or partner
working with your school
provide any training to your
staff on culturally relevant
education?
Teacher transcript: Does your
school or a partner working
with your school provide any
training to teachers on
culturally relevant education?
Supplemental form
Both the principal and
teachers reported that
staff participated in PD
related to educating
Black and Latino
students or male
students
The principal
reported that the
school provided PD
related to educating
Black and Latino
students or male
students, but the
teachers were
unaware of these
opportunities
Neither the principal
nor the teachers
reported any PD
related to educating
Black and Latino
students or male
students
Two programs target
only males
One program targets
only males
No programs target
only males
Programs that Target
Only Males:
1.
2.
Early College/
Career Supports
Supplemental form
School offers two or
more early
college/career
supports
School offers one
early college/ career
support
School offers no early
college/career
supports
Attendance at
Liaison Meetings
DOE data
School attended fivesix liaison meetings
School attended
three-four liaison
meeting
School attended two
or fewer liaison
meetings
Early College/ Career
Supports:
1.
2.
3.
Number of meetings
attended:
Population
Served
Programs represent
only one of the
design areas
List programs and
design areas here:
1.
2.
3.
Principal: Y /N
Protocol Page #:
Teacher: Y/N
Protocol page #:
Score
6
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX D: INTENSITY RUBRIC
Category
Data Source
2: Achieving
Intensity
3: High Intensity
1: Low Intensity
Evidence
Number of
programs
Supplemental
form
4+
3
1-2
Audience
Supplemental
form
3 programs serve
9th and/or 10th
grade male
students
1-2 programs serve
9th and/or 10th grade
male students
No programs serve
9th and/or 10th
grade male
students
Frequency of
Student
Programming
Supplemental
form
3 programs meet
daily, weekly, or
semi-weekly
1-2 programs meet
daily, weekly, or
semi-weekly
No programs meet
daily, weekly, or
semi-weekly
List programs here and
frequency:
1.
2.
3.
Duration (start
and end)
Supplemental
form
3 programs ran for
6-7 consecutive
months throughout
the school year
1-2 programs ran
for 6-7 consecutive
months throughout
the school year
No programs ran
for 6-7 consecutive
months throughout
the school year
List programs here and
duration:
1.
2.
3.
4.
5.
List programs here:
1.
2.
3.
4.
List programs that serve 9th
and 10th grade males:
1.
2.
3.
Score
7
APPENDIX E: FOCUS GROUP PROTOCOL FOR PRINCIPALS AND
ESI LIAISONS AT ESI SCHOOLS
Statement to Subjects
The Research Alliance for New York City Schools at New York University is conducting an evaluation of the Expanded Success
Initiative (ESI). The objective of this research is to learn which services are implemented under ESI, the challenges that schools
confront as they implement ESI services, and the supports schools need to implement ESI services more effectively.
As part of this study, we are conducting focus groups with principals and ESI design team leaders in ESI schools. Focus groups will
take approximately 45 minutes and will be audio recorded. The Research Alliance will use the information participants provide for
research purposes only. We will keep confidential the identities of those who participate in the research and will not attribute any
comments to specific individuals or schools.
Your participation in this study is voluntary. You may request at any time that we stop the focus group or turn off the audio
recording. You may also request at any time that we not include the focus group in the study.
Do you have any questions?
Background and Start-up
To get started, I’d like to ask you about your role in this school and your participation specifically with ESI.
1. First, how long have you been a principal/staff member in your school?
2. Please describe your role with ESI in your school. What are your main responsibilities?
Overall Impressions
Later in the interview, I’m going to be asking you about different ESI programs, but before that, I have some broader questions about
ESI.
3. To get started, briefly tell me about your overall impressions of ESI in your school so far.
4. Have there been any substantial changes in your school since ESI started?
 If so, can you give me an example?
 Probe for: changes to curriculum or instruction, relationships between teachers/students or
between students, student behavior, student attitudes towards learning, teacher expectations, etc.
Race and Gender/College (will be used in comparison protocols)
As you know, the Expanded Success Initiative is designed to better prepare Black and Latino boys for college and careers.
5. How does the staff at your school discuss approaches to educating Black and Latino male students?
6. Are there particular challenges that this population of students faces at your school? (In-school challenges? Outof-school challenges? Challenges to serving this population?)
7. Does your school or an external partner working with your school provide any training to your staff on
Culturally Relevant Education? Has your staff participated? Follow up for how many, how often, description of training,
etc.
8. Finally, are there any college- or career-related programs or services in your school targeted at 9th and 10th grade
students? Are there any college- or career-related programs or services in your school for juniors and seniors?
Probe for: how many college/career programs they have, if they are ESI programs or not. If they don’t have any college/career
programs/services for 9th/10th grade students already established, probe for what (if any) efforts are underway to establish such
programs.
8
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
ESI Programs
9. What is the single most successful or promising ESI program in your school (this year)?
 How do you know it’s successful? Is there evidence of its success? Probe for: data sources, measurable
outcomes, other types of outcomes, etc.
 What factors have contributed to this program’s success?
10. Which ESI program would you say is the least effective (this year)?
 Why do you think this program is less effective than other ESI programs? Probe for: challenges,
obstacles, etc.
 What advice would you give to another school trying to implement this program in their school?
Challenges and Sustainability
11. Have you encountered any challenges around implementing ESI in your school? If so, can you give me one or
two examples?
 How did you or your staff address these challenges?
12. Which programs will your school continue next year?
 How are you and your staff making this decision?
 What will you need in place in order to implement these programs next year? Probe for: expertise,
resources, partners.
13. How do you plan to sustain ESI programming beyond the funding period?
Partnerships and Support
14. Are there any external partners that have played a key role in ESI at your school?
 How have they helped your staff implement ESI in your school?
15. What role, if any, has the DOE team played in ESI at your school? Probe for school’s use on resources, such
as the wiki, the data snapshots, the PDs, principal/point person meetings, etc.
 What types of support would help your school implement ESI more effectively?
Closing
Earlier in the interview, I asked you about your overall impressions of ESI….
16. Has talking about specific aspects of this school prompted any other thoughts about ESI?
17. In closing, is there anything else that I should have asked you about ESI or that you’d like to share to help us
understand the implementation of ESI in your school?
[Turn off the tape recorder]
Is there anything else that you want to say with the tape recorder turned off?
9
APPENDIX F: “SUPPLEMENTAL FORM” FOR PRINCIPALS AND
ESI LIAISONS
School Name: _____________________________________
ESI Program 1: __________________________________________________________
Target Population
Number
Adult-Student Ratio
Frequency/Duration
Start Date/End
Date
External Partners
Partner’s Role
ESI Program 2: __________________________________________________________
Target Population
Number
Adult-Student Ratio
Frequency/Duration
Start Date/End Date
External Partners
Partner’s Role
(Schools were asked to list up to 8 programs)
10
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX G: FOCUS GROUP PROTOCOL FOR TEACHERS AT
ESI SCHOOLS
Statement to Subjects
The Research Alliance for New York City Schools at New York University is conducting an evaluation of the Expanded Success
Initiative (ESI). The objective of this research is to learn which services are implemented under ESI, the challenges that schools
confront as they implement ESI services, and the supports schools need to implement ESI services more effectively.
As part of this study, we are conducting focus groups with 10th grade teachers in ESI schools. Focus groups will take approximately
45 minutes and will be audio recorded. The Research Alliance will use the information participants provide for research purposes
only. We will keep confidential the identities of those who participate in the research and will not attribute any comments to specific
individuals or schools.
Your participation in this study is voluntary. You may request at any time that we stop the focus group or turn off the audio
recording. You may also request at any time that we not include the focus group in the study.
Do you have any questions?
To get started, I’d like to ask you about your role in this school and your general impressions of ESI.
Background and Overall Impressions of ESI
1. How long have you been a teacher at this school and what do you teach?
2. What are your overall impressions of ESI in your school so far?
Race and Gender/College (will be used in comparison protocols)
As you likely know, the Expanded Success Initiative is designed to better prepare Black and Latino boys for college and careers.
3. How does the staff at your school discuss approaches to educating ESI’s target population? Has this changed
at all since last school year?
4. Are there particular challenges that this population of students faces at your school? (In-school challenges?
Out-of-school challenges? Challenges to serving this population?)
5. Does your school or a partner working with your school provide any training to teachers on culturally
relevant education? Have you participated in this training? Probe for description of training.
6. Finally, are there any college- or career-related programs or services in your school targeted at 9th and 10th
grade students? Are there any college- or career-related programs or services for juniors and seniors? Probe
for: how many college/career programs they have, if they are ESI programs or not. If they don’t have any college/career
programs/services for 9th/10th grade students already established, probe for what (if any) efforts are underway to establish
such programs.
Changes Due to ESI
7. Have there been any substantial changes in your school since ESI started? If so, can you give me an example?
Probe for: institutional changes, student behavior, student attitudes towards learning, teacher expectations,
etc.
8. Have you noticed any changes with respect to the nature of teacher-student relationships in your school?
Probe for specific examples.
9. Have you noticed any changes with respect to the nature of relationships between students in your school?
Probe for specific examples.
11
ESI Programming & Resources
I’m going to shift gears now and ask you some questions about ESI programming and resources.
10. [Low priority] What do you think is most successful or promising ESI program in your school?
 How do you know it’s successful? Is there evidence of it success? Probe for: data sources, measurable
outcomes, other types of outcomes, etc.
 What factors have contributed to this program’s success?
11. Please describe any ESI-related professional development that you’ve participated in.
 Probe as appropriate for PD-related challenges
 Probe: have there been any PD opportunities that you were unable to attend?
There are a few closing questions I’d like to ask before ending the interview.
Closing
12. In closing, is there anything else that I should have asked you about ESI or that you’d like to share to help us
understand the implementation of ESI in your school?
 Are there any additional challenges that you or other teachers have faced in the implementation of ESI
programming that we have not already discussed?
[Turn off the tape recorder]
Is there anything else that you want to say with the tape recorder turned off?
12
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX H: FOCUS GROUP PROTOCOL FOR ESI STUDENTS
Statement to Subjects
The Research Alliance for New York City Schools at New York University is conducting an evaluation of the Expanded Success
Initiative (ESI). The purpose of this research is to learn about programs implemented under ESI, the challenges of implementing ESI,
and the ways ESI could be improved.
As part of this study, we are conducting focus groups with students in ESI schools. Focus groups will take approximately 45 minutes
and will be audio recorded. The Research Alliance will use the information you provide for research purposes only. We will keep the
identities of interviewees confidential and not attribute any comments to any specific individuals or schools.
Your participation in this study is voluntary. You may request at any time that we stop the interview or turn off the audio recording.
You may also request at any time that we not include the focus group in the study.
Do you have any questions?
We’ll begin with some questions about your overall impressions of your school.
Successes and Challenges in School
1. What are your overall impressions of your school?
 What’s your favorite thing about your school?
 What about your school would you like to see change? What would you like to see more of?
2. What are some of the things that are going really well for you at school?
3. Is there anything that gets in the way of you doing well in school? Is there anything that gets in the way of
other students doing well at this school? Probe: Academic challenges? Attendance? Social Challenges? Challenges
with peers? Teachers?
Ok, now I’d like to ask you a few questions about your relationships with your teachers and with other students in your school.
Relationships
4. How would you describe your interactions with your teachers?
 Who is your favorite teacher? What makes that teacher your favorite?
5. Are there any other adults at your school, or that you know through school, that you have a close
relationship with? Probe: mentors, non-teaching school personnel (i.e., sports coaches, school security workers).
6. How would you describe the interactions between students in this school? Probe for specific examples.
Your school is part of a group of schools that were selected to participate in ESI, the Expanded Success Initiative.
Expanded Success Initiative and School Supports
7. What do you know about ESI?
 What is the purpose of ESI?
8. Are you involved with any ESI programs?
 Are you involved with any programs specifically for boys? Probe for descriptions of programs.
 How often do you attend?
 Is the program a part of a class?
 How’s it going so far?
13
9. As you might know, ESI includes funds programs specifically for Black and Latino male students. Why do
you think there might be funding for these specific students?
10. Aside from the programs you already talked about [if appropriate], what are some of the programs or
activities your school has in place to help you succeed? Probe for details about specific supports, programs, classes.
11. Are there programs that your school offers that you do not participate in? Probe for reasons why student does not
participate, if applicable.
12. What additional things could your school do to help you succeed during high school and after high school?
At this point, I want to shift gears and discuss your plans for after high school.
College, Career, and Future Goals
13. What are your goals for after high school?
14. In what ways is your school helping you to achieve these goals?
15. [Skip if short on time] Are there college or career supports that your school offers that you do not
participate in? Probe for reasons why student does not participate, if applicable. Probe for supports that students would
like their school to offer.
16. What would you say teachers and counselors communicate to you about going to college?
17. When talking about college with your classmates, what topics or concerns do you discuss?
Closing
18. Is there anything else that you’d like to share about your experiences in school before we end the focus
group?
[Turn off the tape recorder]
Is there anything else that you want to say with the tape recorder turned off?
14
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX I: INTERVIEW PROTOCOL FOR PRINCIPALS AT
COMPARISON SCHOOLS
Statement to Subjects
The Research Alliance for New York City Schools at New York University is conducting an evaluation of the Expanded Success
Initiative (ESI), which is designed to improve postsecondary readiness for Black and Latino young men. The objective of your
inclusion in the study is so that we can learn about 1) services and supports at your school specifically targeted towards Black and
Latino males, 2) the challenges that your school confronts in the process of implementing such services, and 3) the supports needed to
implement these services effectively.
As part of this study, we are interviewing principals and 10th grade teachers. Interviews will take approximately 45 minutes and will
be audio recorded. The questions will focus on programming that your school offers which targets the success of Black and Latino
boys.
The Research Alliance will use the information participants provide for research purposes only. We will keep confidential the
identities of those who participate in the research and will not attribute any comments to any specific individuals or schools. Your
participation in this study is voluntary. You may request at any time that we stop the interview or turn off the audio recording. You
may also request at any time that we not include the interview in the study.
Do you have any questions?
Principal Background
1. First, how long have you been a principal in your school?
Overall Impressions
2. Later in the interview, I’m going to be asking you specific questions about different programs targeted at
Black and Latino boys, but to get started, please tell me a little about your impressions of how well your
school serves this population of students.
3. How does the staff at your school discuss approaches to educating Black and Latino male students?
4. Are there particular challenges that this population of students faces at your school?
 In-school challenges? Out-of-school challenges?
5. What challenges has your school encountered in efforts to serve this population?
 Nature of challenges?
 How have these challenges been addressed?
 What supports would be needed to address these challenges?
6. Does your school provide any professional development to your staff that is related to educating boys of
color? Has your staff participated? Probe for how many, how often, basic description of PD, etc.
7. What is the nature of teacher-student relationships at your school?
8. What is the nature of relationships between students at your school?
Next, I’m going to ask a few questions about specific programs you may have in your school.
Targeted programming:
9. Are there any college- or career-related programs or services in your school for juniors and seniors? If yes,
probe for basic descriptions and any quantifiable info (e.g., number of students served, frequency/duration, etc.).
15
10. Are there any college- or career-related programs or services in your school targeted at 9th and 10th grade
students? If yes, probe for basic descriptions and any quantifiable info (e.g., number of students served,
frequency/duration, etc.).
11. Are there any programs or services in your school that are specifically targeted at Black and Latino boys? If
yes, probe for basic descriptions and any quantifiable info (e.g., number of students served, frequency/duration, etc.).
Wrap up
12. In closing, is there anything else that I should have asked you about programming at your school or that
you’d like to share to help us understand how to effectively serve Black and Latino boys?
[Turn off the tape recorder]
Is there anything else that you want to say with the tape recorder turned off?
16
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX J: FOCUS GROUP PROTOCOL FOR TEACHERS AT
COMPARISON SCHOOLS
Statement to Subjects
The Research Alliance for New York City Schools at New York University is conducting an evaluation of the Expanded Success
Initiative (ESI), which is designed to improve postsecondary for Black and Latino young men. The objective of your inclusion in the
study is so that we can learn about 1) services and supports at your school specifically targeted towards Black and Latino males, 2) the
challenges that your school confronts in the process of implementing such services, and 3) the supports needed to implement these
services effectively.
As part of this study, we are interviewing principals and 10th grade teachers. Interviews will take approximately 45 minutes and will
be audio recorded. The questions will focus on programming that your school offers which targets the success of Black and Latino
boys.
The Research Alliance will use the information participants provide for research purposes only. We will keep confidential the
identities of those who participate in the research and will not attribute any comments to any specific individuals or schools. Your
participation in this study is voluntary. You may request at any time that we stop the interview or turn off the audio recording. You
may also request at any time that we not include the interview in the study.
Do you have any questions?
Background
1. How long have you been teaching in this school? What grades and subjects do you teach?
Overall Impressions
2. Later in the interview, I’m going to be asking you specific questions about different programs targeted at
Black and Latino boys, but to get started, please tell me a little about your impressions of how well your
school serves this population of students.
3. How does the staff at your school discuss approaches to educating Black and Latino boys?
4. Are there particular challenges that this population of students faces at your school?
 In-school challenges? Out-of-school challenges?
5. What challenges has your school encountered in efforts to serve this population?
 Nature of challenges?
 How have these challenges been addressed?
 What supports would be needed to address these challenges?
6. Does your school provide any professional development that is related to educating boys of color? Have you
participated? Probe for how many participated, how often, basic description of PD, etc.
7. What is the nature of teacher-student relationships in your school?
8. What is the nature of relationships between students in your school?
Targeted Programming
9. Are there any college- or career-related programs or services in your school for juniors and seniors? If yes,
probe for basic descriptions and any quantifiable info (e.g., number of students served, frequency/duration, etc.).
10. Are there any college- or career-related programs or services in your school targeted at 9th and 10th grade
students? If yes, probe for basic descriptions and any quantifiable info (e.g., number of students served,
frequency/duration, etc.).
17
11. Are there any programs or services in your school that are specifically targeted at serving Black and Latino
boys? If yes, probe for basic descriptions and any quantifiable info (e.g., number of students served, frequency/duration,
etc.).
Wrap up
12. In closing, is there anything else that I should have asked you about programming at your school or that
you’d like to share to help us understand how to effectively serve Black and Latino boys?
[Turn off the tape recorder]
Is there anything else that you want to say with the tape recorder turned off?
18
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX K: COMPARISON RUBRIC
SCHOOL NAME/BN #:
REVIEWER:
Measure
Level of color-cognizance
Definition of Cognizance
- Participants express an
understanding of the importance
of racial and ethnic differences
because their profound impact
on individual, groups,
communities, and societies
accompanied by
acknowledgement of
widespread racism and
discrimination
Source of information
- Protocol question: How does
the staff at your school discuss
approaches to educating Black
and Latino male students?
3
Both the principal and at
least one teacher
express color-cognizant
views.
2
Either the principal or at
least one teacher
expresses colorcognizant views.
1
Neither the principal nor
any teachers express
color-cognizant views.
Scores and Sources
Score:
- Supplemental form for ESI
schools
-Protocol question for comp.
schools: Does your school
provide any professional
development to your staff that
is related to educating boys of
color?
Both the principal and
teachers reported that
staff participated in PD
related to educating
Black and Latino
students or male
students.
The principal reported
that the school provided
PD related to educating
Black and Latino
students or male
students, but the
teachers were unaware
of these opportunities.
Neither the principal nor
the teachers reported
any PD related to
educating Black and
Latino students or male
students.
Principal: Y/N
Transcript Page #:
Teacher: Y/N
Transcript Page #:
- Participants consider
connections between student
background and school
practices
Some interviewees may make
remarks that demonstrate both
color cognizance and color
blindness. However, if an
interviewee demonstrates color
cognizance even if they also
demonstrate color blindness,
they should receive credit for the
purposes of this rubric.
Professional development
related to educating Black and
Latino students or male students
Score:
19
Measure
College- or career-related
programs or services targeted at
11th and 12th grade students
Source of information
- Supplemental form for ESI
schools
- Protocol question: Are there
any college- or career-related
programs or services in your
school for 11th and 12th
graders?
3
The school provides 3 or
more college- or careerrelated programs or
services targeted at 11th
and 12th grade students.
2
The school provides 1-2
college- or career-related
programs or services
targeted at 11th and 12th
grade students.
1
The school does not
provide college- or
career-related programs
or services targeted at
11th and 12th grade
students.
Scores and Sources
Program 1:
Source/Page #:
Program 2:
Source/Page #:
Program 3:
Source/Page #:
College- or career-related
programs or services targeted at
9th and 10th grade students
Programs or services targeted at
serving Black and Latino male
students
- Supplemental form for ESI
schools
- Protocol question for comp.
schools: Are there any collegeor career-related programs or
services in your school
targeted at 9th and 10th grade
students?
The school provides 3 or
more college- or careerrelated programs or
services targeted at 9th
and 10th grade students.
- Supplemental form for ESI
schools
- Protocol question for comp.
schools: Are there any
programs or services in your
school that are specifically
targeted at serving Black and
Latino boys?
The school provides 2 or
more programs targeted
at serving Black and
Latino male students.
The school provides 1-2
college- or career-related
programs or services
targeted at 9th and 10th
grade students.
The school does not
provide college- or
career-related programs
or services targeted at
9th and 10th grade
students.
Score:
Program 1:
Source/Page #:
Program 2:
Source/Page #:
Program 3:
Source/Page #:
The school provides 1
program targeted at
serving Black and Latino
male students.
The school does not
provide any programs
targeted at serving Black
and Latino male
students.
Score:
Program 1:
Source/Page #:
Program 2:
Source/Page #:
Score:
TOTAL SCORE:
Additional Comments: List any programs mentioned in principal or teacher transcript, along with the exact page number that do not have a clear
target audience.
20
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX L: SURVEY CONSTRUCTS AND ITEMS
Construct
Item
Scale
Academic Domain
Academic Self-Concept
I am confident in my academic abilities.
1- Not at all true
I do well in school.
2- A little true
I learn new concepts quickly.
3- Really true
I am successful in school.
4- Very true
I am confident in my ability to succeed in school.
Critical Thinking
I can easily express my thoughts on a problem.
1- Not at all true
I usually use multiple source of information before making a decision.
2- A little true
I compare ideas when thinking about a topic.
3- Really true
I keep my mind open to different ideas when planning to make a decision.
4- Very true
I am able to tell the best way of handling a problem.
Youth Development Domain
Fairness of School Rules
In my
the punishment for breaking school rules is the same no matter who you are.
school...
if a rule is broken, students know what kind of punishment will follow.
1- Not at all true
2- A little true
all students are treated fairly when they break school rules.
3- Really true
the rules are strictly enforced.
4- Very true
the rules are fair.
everyone knows the rules for student conduct.
Racial, Gender, and Cultural Differences
In my
I have been disrespected or mistreated by an adult because of my race,
school…
ethnicity, or nationality.
I have been disrespected or mistreated by an adult at this school
because of my gender.
there is a lot of tension between different races, ethnicities, and nationalities.
1- Not at all true
2- A little true
3- Really true
4- Very true
21
Construct
Item
Scale
School Culture Domain
Conversations about College
In your
school,
how much
do you
talk with
adults
about…
whether you should go to college
1- Not at all
how you can increase your chances of getting into a good college?
2- A little
which high school courses you should be taking to prepare for college?
3- Some
what activities you can do outside of school to help you prepare for college?
4- A lot
what financial aid might be available to help you pay for college?
Conversations about Careers
In your
school,
how much
do you
talk with
adults
about…
your special interests and talents.
1- Not at all
what you can be doing now to make sure you reach your life goals?
2- A little
what career you would like when you get older?
3- Some
4- A lot
22
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
APPENDIX M: SURVEY CONSTRUCT ANALYSIS
Cronbach’s alpha, which captures the degree to which responses to survey items are correlated with each other, is a
common indicator of reliability. Values for Cronbach’s alpha range from zero to one, with values close to one
indicating high reliability. According to widely used survey research standards, a value of 0.70 or higher indicates
sufficient correlation among the items in a measure to be considered reliable. Table M-1 shows that all six survey
constructs we analyzed met this benchmark.
Table M-1: Standardized Cronbach’s Alphas for ESI Year 2 10th Grade Survey
Construct
Academic Self Concept
Standardized
Critical Thinking
Standardized
Conversations about College
Standardized
Conversation about Careers
Standardized
Fair Treatment
Standardized
Racial, Gender, Cultural Climate
Standardized
Alpha
0.86
0.75
0.89
0.84
0.84
0.77
Source: Research Alliance calculations.
Intraclass correlation is a direct measure of the proportion of overall variation in the measures across the schools we
surveyed compared to the variation within the schools. Values close to zero indicate that there was very little
difference in responses across schools and that most of the difference occurred within schools. The intraclass
correlations for the measures in Table M-2 were all below 0.10, indicating that they had limited capacity to
differentiate among schools. This implies that when we use these measures to look at differences between ESI and
non-ESI schools, we should expect to see only modest differences.
Table M-2: Intra-Class Correlations (ICC) for ESI Year 2 10th Grade Survey
Construct
Academic Self Concept
Critical Thinking
Conversations about College
Conversations about Careers
Fair Treatment
Racial, Gender, Cultural Climate
Source: Research Alliance calculations.
ICC
0.02
0.03
0.06
0.05
0.07
0.04
23
APPENDIX N: SURVEY AND ACADEMIC IMPACTS
Student Background Characteristics
9th Grade Students
Tables N-1 and N-2 describe the 9th grade students in Year 2 of our evaluation of the Expanded Success Initiative (the
2013-14 school year). We removed students from the sample who were repeating the 9th grade in 2013-14 (these
students were considered to be in the 10th grade for program evaluation purposes). Table N-1 describes all students in
ESI and comparison schools, while N-2 only includes Black and Latino males.
Table N-1: Background Characteristics of 9th Grade Students in ESI Schools,
Comparison Schools, and Other High Schools, 2013-2014
ESI
Schools
Comparison
Schools
Other NYC High
Schools
Gender (%)
Female
Race (%)
Asian
Black
Latino
White
Other
43.0
49.1
50.5
4.5
47.5
43.2
3.1
1.7
7.4
38.0
46.9
5.7
2.0
19.7
25.7
36.6
15.8
2.2
-0.2
-0.2
-0.2
-0.3
0.1
0.1
83.3
10.6
18.9
82.1
12.2
19.4
73.1
10.7
17.1
28.6
128
40
31.3
149
80
25.8
248
229
Academicsa
ELA Scaled Score
Math Scaled Score
Poverty Status (%)b
English Learner (%)
Special Education (%)
Overage (%)c
Students per School
Schools
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: Sample includes only students in the 9th grade for the first time in fall 2013. aTest scores are from the 8th grade NYS
Exams and reported in Z-score units, standardized with district-wide means and standard deviations. bReceives free lunch,
reduced price lunch, or is enrolled in a universal-feed school as of 9th grade. c15 or older as of 12/31 of the fall semester of their
9th grade year.
24
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
Table N-2: Background Characteristics of Black and Latino Male 9th Grade
Students in ESI Schools, Comparison Schools, and Other High Schools, 20132014
ESI Schools
Race (%)
Black
Latino
Comparison
Schools
Other NYC High
Schools
52.7
47.3
44.4
55.6
40.0
60.0
-0.3
-0.3
-0.4
-0.4
-0.3
-0.3
83.9
11.1
23.7
83.9
11.1
26.4
79.2
12.7
25.6
31.5
64
40
36.2
63
80
34.8
83
225
Academicsa
ELA Scaled Score
Math Scaled Score
Poverty Status (%)b
English Learner (%)
Special Education (%)
Overage (%)c
Students per School
Schools
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: Sample includes only students in the 9th grade for the first time in fall 2013. aTest scores are from the 8th grade NYS
Exams and reported in Z-score units, standardized with district-wide means and standard deviations. bReceives free lunch,
reduced price lunch, or is enrolled in a universal-feed school as of 9th grade. c15 or older as of 12/31 of the fall semester of their
9th grade year.
10th Grade Students
Tables N-3 and N-4 on the next page describe the students who were first-time 9th graders in Year 1 of ESI, and who
were scheduled to be 10th graders in Year 2 of ESI. (The sample includes students who may have been retained in 9th
grade in the 2013-2014 school year).
25
Table N-3: Background Characteristics of 10th Grade Students in ESI Schools,
Comparison Schools, and Other High Schools, 2013-2014
ESI Schools
Comparison
Schools
Other NYC High
Schools
Gender (%)
Female
43.2
49.3
50.5
4.5
47.3
44.0
2.9
1.3
7.3
38.6
47.1
5.5
1.5
19.9
25.7
36.6
15.9
1.9
-0.2
-0.2
84.0
10.5
18.9
28.3
124
40
-0.2
-0.3
82.8
12.2
19.4
30.7
146
80
0.1
0.1
73.4
10.7
17.1
25.5
169
339
Race (%)
Asian
Black
Latino
White
Other
Academicsa
ELA Scaled Score
Math Scaled Score
Poverty Status (%)b
English Learner (%)
Special Education (%)
Overage (%)c
Students per School
Schools
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: Sample includes only students who were first time 9th graders in 2012 who remained in ESI, comparison, or other
district high schools in 2013-2014. aTest scores are from the 8th grade NYS Exams and reported in Z-score units,
standardized with district-wide means and standard deviations. bReceives free lunch, reduced price lunch, or is enrolled in a
universal-feed school as of 9th grade. c15 or older as of 12/31 of the fall semester of 9th grade.
Table N-4: Background Characteristics of Black and Latino Male 10th Grade
Students in ESI Schools, Comparison Schools, and Other High Schools, 2013-2014
ESI Schools
Comparison
Schools
Other NYC High
Schools
Race (%)
Black
Latino
52.2
47.8
44.4
55.6
40.1
59.9
-0.3
-0.3
-0.4
-0.4
-0.3
-0.3
84.4
11.2
23.7
84.3
12.3
26.4
79.5
12.6
25.7
31.3
62
40
35.1
62
80
34.5
60
315
Academicsa
ELA Scaled Score
Math Scaled Score
Poverty Status (%)b
English Learner (%)
Special Education (%)
Overage (%)c
Students per School
Schools
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: Sample includes only students who were first time 9th graders in 2012 who remained in ESI, comparison, or other district high
schools in 2013-2014. aTest scores are from the 8th grade NYS Exams and reported in Z-score units, standardized with district-wide
means and standard deviations. bReceives free lunch, reduced price lunch, or is enrolled in a universal-feed school as of 9th grade. c15
or older as of 12/31 of the fall semester of 9th grade.
26
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
Survey Analysis
Our main report included selected results from our analysis of the Year 2 ESI Survey. The following tables include
further survey analysis details. While the report presents the estimated difference between ESI and Comparison
schools as effect sizes, these tables show the estimated difference between ESI and Comparison schools in raw score
terms. Responses were on a 1-4 scale. See Chapter 4 of the full report for effect sizes.
Table N-5: ESI Survey Results for Black and Latino Male Students in 9th and 10th Grade,
2013-2014 (Raw Scores)
9th Grade
Academic SelfConcept
Critical Thinking
Conversations About
Career
Conversations About
College
Gender/Race Climate
Sense of Fair
Treatment
10th Grade
Academic SelfConcept
Critical Thinking
Conversations About
Career
Conversations About
College
Gender/Race Climate
Sense of Fair
Treatment
ESI (vs.
Comparison
Schools)
Black (vs.
Latino)
Poverty
Status
English
Learner
Special
Education
Overage
0.03
0.06
0.07*
0.06
0.05
-0.14
-0.05
0.04
-0.14*
-0.11*
-0.05
0
0.04
0.11*
-0.02
0.16
0.03
0.08
0.08
0.03
0.13*
-0.15*
-0.01
-0.14
0.23*
-0.11
-0.06
-0.06
0.09
-0.06
0.11
-0.07
-0.13
-0.02
-0.03
0.06
0.06
0.06
0.02
0
-0.1
0.01
-0.07
-0.02
-0.05
-0.05
-0.07
0.07
0.15*
0.13*
-0.05
0.08
0.13*
0.08
0.14
-0.1
0.12*
-0.02
-0.02
-0.17
-0.04
-0.15*
-0.02
0.09
0.12*
0.01
0.07
-0.01
-0.15
0
0.02
0.1*
Source: Research Alliance calculations based on ESI Survey and data obtained from the NYC Department of Education.
Notes: Entries in each column are differences in survey responses (scale 1-4) controlling for all other columns. Regression model also includes
previous year attendance rate, 8th grade math/ELA Scores and self-reported family education level.
27
Academic Impact Analysis
Our academic impact study uses a Comparative Interrupted Time Series method, which isolates the effect of ESI from
other system- or school-wide events by conducting a series of predictions and comparisons. First, we used student
demographics and prior performance data, along with four years of academic outcomes for prior cohorts of students
in ESI schools, to predict the outcomes for ESI students. Second, we made the same prediction for comparison school
students. Then, we measured how the difference between actual student performance for ESI students differed from
their predicted performance after the introduction of ESI. We also measured how much comparison student
performance differed from their predicted performance in the same year, in order to detect system-wide changes that
might have been happening simultaneously to the introduction of ESI. Finally, we compared the ESI schools’ change
to the comparison school change to determine if there were any program impacts statistically different from zero. We
also controlled for individual student background characteristics of students, such as their prior performance and
prior attendance.
The tables in this section illustrate results obtained through this method. The first column in each table shows
projected outcomes for ESI students, based on demographic characteristics, prior performance, and four years of data
from students enrolled in ESI schools before the initiative began. The second column indicates the difference between
how ESI students performed and how we predicted they would perform. The third column shows the difference
between how comparison school students performed and how we predicted they would perform. And the final
column compares the ESI change to the comparison school change to estimate academic impacts.
Table N-6 includes academic impact outcomes for Black and Latino males in 9th and 10th grades at ESI schools.
Table N-6: Academic Impact of ESI for Black and Latino Male 9th and 10th Grade Students,
2013-2014
Change ESI Schools
Change Comparison
Schools
Impact
56.12
69.39
8.55
89.95
11.37
-0.09
0.78
0.82*
-1.07
0.72
0.34
1.53*
0.84*
-0.01
0.99*
-0.43
-0.76
-0.02
-1.06
-0.27
62.05
-0.64
-0.83
0.19
50.23
63.87
7.69
87.07
10.18
-0.79
-0.63
0.09
-0.95
0.07
-0.52
1.19*
0.17
-0.23
0.28
-0.27
-1.82*
-0.08
-0.72
-0.20
59.72
2,678
40
67
-1.56
2,678
40
67
-1.27
4,856
80
61
-0.28
Expected
9th Grade
On-Track for Regents Diploma (%)
Academic GPA (weighted by Credits)
Academic Credits Earned
Attendance (%)
Total Credits Earned
Passed at Least One Regents by End of
Grade (%)
th
10 Grade
On-Track for Regents Diploma (%)
Academic GPA (weighted by Credits)
Academic Credits Earned
Attendance (%)
Total Credits Earned
Passed at Least Two Regents by End of
Grade (%)
Students
Schools
Students per School
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: 9th grade sample includes first-time 9th graders. 10th grade sample includes students who were first-time 9th graders in 2012-2013 and
remained enrolled in an ESI or comparison school. On-Track in 9th grade is defined as passing one Regents and accrued 10 credits. On-Track in
10th grade is defined as passing two Regents and accruing 20 credits. * Denotes statistical significance from zero at the 0.05 level.
28
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
All Student and Subgroup Analysis
We conducted subgroup analyses to better understand our results for Black and Latino Male students, and to gauge
possible effects that ESI programming has on different student populations. For comparison purposes, the following
tables present academic impact results for all 9th and 10th grade students at ESI schools (Table N-7), Black and Latino
female 9th and 10th grade students at ESI schools (Table N-8) and for Asian and White (both male and female) 9th and
10th grade students at ESI schools (Table N-9).
Table N-7: Academic Impact of ESI for 9th and 10th Grade Students, 2013-2014
Change - ESI
Schools
Change Comparison
Schools
Impact
57.78
71.24
8.76
-1.94
-0.16
0.67
0.49
1.97*
0.81*
-2.42
-2.13
-0.13
89.92
11.59
63.29
-1.46*
0.49
-1.84
-0.29
1*
-0.7
-1.17
-0.51
-1.14
Academic GPA (weighted by Credits)
53.83
65.58
-2.49
-0.96
-0.04
1.11*
-2.45
-2.07*
Academic Credits Earned
Attendance (%)
8.05
87.18
0.04
-1.01*
0.17
-0.26
-0.13
-0.75
Total Credits Earned
10.52
0
0.26
-0.27
Passed at Least Two Regents by End of Grade (%)
61.37
-2.71
-0.56
-2.15
Students
5,194
5,194
11,432
Schools
40
40
80
Students per School
130
130
143
Expected
9th Grade
On-Track for Regents Diploma (%)
Academic GPA (weighted by Credits)
Academic Credits Earned
Attendance (%)
Total Credits Earned
Passed at Least One Regents by End of Grade (%)
10th Grade
On-Track for Regents Diploma (%)
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: 9th grade sample includes first-time 9th graders. 10th grade sample includes students who were first-time 9th graders in 2012-2013 and
remained enrolled in an ESI or comparison school. On-Track in 9th grade is defined as passing one Regents and accrued 10 credits. On-Track in
10th grade is defined as passing two Regents and accruing 20 credits. * Denotes statistical significance from zero at the 0.05 level.
29
Table N-8: Academic Impact of ESI for Black and Latino Female 9th and 10th Grade Students, 2013-2014
Change - ESI
Schools
Change Comparison
Schools
Impact
59.09
72.88
8.96
89.67
11.78
-4
-1.04
0.43
-2.15*
0.07
0.42
2.28*
0.73*
-0.57
0.93*
-4.42
-3.32
-0.31
-1.58
-0.86
64.19
-2.8
-0.32
-2.48
57.04
67.56
8.49
87.51
10.98
-4.42
-1.15
-0.02
-1.21
-0.13
0.4
1.04
0.24
-0.2
0.31
-4.83
-2.2
-0.27
-1.01
-0.44
62.42
-3.95
-0.16
-3.78
2,032
39
52
2,032
39
52
4,856
80
61
Expected
9th Grade
On-Track for Regents Diploma (%)
Academic GPA (weighted by Credits)
Academic Credits Earned
Attendance (%)
Total Credits Earned
Passed at Least One Regents by End of
Grade (%)
th
10 Grade
On-Track for Regents Diploma (%)
Academic GPA (weighted by Credits)
Academic Credits Earned
Attendance (%)
Total Credits Earned
Passed at Least Two Regents by End of
Grade (%)
Students
Schools
Students per School
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: 9th grade sample includes first-time 9th graders. 10th grade sample includes students who were first-time 9th graders in 2012-2013 and remained
enrolled in an ESI or comparison school. On-Track in 9th grade is defined as passing one Regents and accrued 10 credits. On-Track in 10th grade is
defined as passing two Regents and accruing 20 credits. * Denotes statistical significance from zero at the 0.05 level.
30
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
Table N-9: Academic Impact of ESI for White and Asian 9th and 10th Grade Students, 20132014
Change - ESI
Schools
Change Comparison
Schools
Impact
63.31
74.77
9.35
92.85
12.33
-1.58
-0.66
1.1
-0.74
1.12
-0.43
2.21*
1*
0.09
1.15*
-1.15
-2.87
0.11
-0.83
-0.03
67.15
-1.77
-1.61
-0.15
53.80
62.39
7.52
85.92
10.98
-3.38
-0.76
-0.16
-0.74
-0.13
-2.99
0.1
0.04
-0.5
0.31
-0.39
-0.87
-0.19
-0.24
-0.44
63.69
-2.56
-1.65
-0.91
484
37
13
484
37
13
1,720
78
22
Expected
th
9 Grade
On-Track for Regents Diploma (%)
Academic GPA (weighted by Credits)
Academic Credits Earned
Attendance (%)
Total Credits Earned
Passed at Least One Regents by End of
Grade (%)
10th Grade
On-Track for Regents Diploma (%)
Academic GPA (weighted by Credits)
Academic Credits Earned
Attendance (%)
Total Credits Earned
Passed at Least Two Regents by End of
Grade (%)
Students
Schools
Students per School
Source: Research Alliance calculations based on data obtained from the NYC Department of Education.
Notes: 9th grade sample includes first-time 9th graders. 10th grade sample includes students who were first-time 9th graders in 2012-2013 and
remained enrolled in an ESI or comparison school. On-Track in 9th grade is defined as passing one Regents and accrued 10 credits. On-Track in
10th grade is defined as passing two Regents and accruing 20 credits. * Denotes statistical significance from zero at the 0.05 level.
31
APPENDIX O: STUDENT DISCIPLINE OUTCOMES
Student Discipline Policy
In the 2012-2013 school year, the DOE instituted changes in the disciplinary code as it relates to suspensions. This
resulted in a significant drop in the total number of suspensions, 4 which we were able to document in ESI and
comparison schools (see below). Continuing public focus on disparities in the application of school disciplinary policy
suggest that further changes are possible. The CITS method used in our analysis will allow us to take these systemwide changes into account.
Student Discipline Impact Results
As reported in Chapter 4, suspension rates (per student) in ESI schools dropped for Black and Latino male 9th grade
students, particularly for suspensions associated with “disruptive” infractions.5 We did not find similar impacts for 10th
grade students. Tables O-1 shows the full set of results from our analyses of suspension rates among Black and Latino
male students.
Table O-1: ESI’s Impact on Suspensions for Black and Latino Male Students in
9th and 10th Grade, 2013-2014
Expected
Change ESI
Change Comparison
Impact
0.25
0.28
0.04
0.13
0.09
0.02
-0.11*
-0.13*
-0.04*
-0.07*
-0.01
-0.01
-0.05*
-0.06*
-0.04*
0.00
-0.01
0.00
-0.06
-0.07
0.00
-0.07*
0.00
-0.01
0.20
0.22
0.01
0.01
0.09
0.09
-0.01
-0.01
-0.01
0.00
-0.01
0.00
0.01
0.01
-0.01*
-0.02*
0.02
0.00
-0.02
-0.02
0.00
0.01
-0.03
0.00
9th Grade
Suspensions per student
Infractions per student
Disorderly Infractions
Disruptive Infractions
Aggressive Infractions
Violent Infractions
10th Grade
Suspensions per student
Infractions per student
Disorderly Infractions
Disruptive Infractions
Aggressive Infractions
Violent Infractions
Source: Research Alliance calculations based on data from the NYC Department of Education.
Notes: 9th grade sample only includes first-time 9th graders. 10th grade sample includes students who were in the 9th grade for the
first time in 2012-13. A single suspension can be associated with multiple infractions.
32
CHANGING HOW HIGH SCHOOLS SERVE BLACK AND LATINO YOUNG MEN
References
Cooper, H., and Hedges, L., eds. (1994). The Handbook
of Research Synthesis. New York, NY: The Russell
Sage Foundation.
Somers, M.A., Zhu, P., Jacob, R. and Bloom, H. (2013).
The Validity and Precision of the Comparative
Interrupted Time Series Design and the Difference-inDifference Design in Educational Evaluation. New
York, New York: MDRC.
See page 14 of the IES What Works Clearinghouse
handbook:
ies.ed.gov/ncee/wwc/pdf/reference_resources/
wwc_procedures_v2_1_standards_handbook.pdf
4
See ny.chalkbeat.org/2013/10/31/suspensionsfall-but-disparities-for-some-studentsremain/#.VT28tiFViko
5 Full description of the NYC Department of
Education discipline code can be found at
http://schools.nyc.gov/RulesPolicies/Discipline
Code/default.htm
3
Notes
There were 190 potential matches in the original
selection round. The difference is likely due to
more schools having five years of pre-ESI data.
2 With the exception of slopes and intercepts, all of
these matching variables were standardized based
on the mean value found in the full school-level
ESI and matching pool sample.
1
Find this report online at http://steinhardt.nyu.edu/research_alliance/publications/esi_year2
285 Mercer Street, 3rd Floor | New York, New York 10003-9502
212 992 7697 | 212 995 4910 fax
[email protected] | www.steinhardt.nyu.edu/research_alliance
The Research Alliance for
New York City Schools conducts
rigorous studies on topics that
matter to the city’s public schools.
We strive to advance equity and
excellence in education by
providing nonpartisan evidence
about policies and practices that
promote students’ development
and academic success.