Drilling Down in SWIS
Data
Using a Problem-Solving
Model to Address Discipline
Disproportionality
Jennifer Rollenhagen, Eben McKnight, &
Susannah Everett
NEPBIS Leadership Forum 2017
Acknowledgements: Educational and
Community Supports
ECS is a research unit within the
University of Oregon’s College of
Education. ECS focuses on the
development and implementation of
practices that result in positive,
durable, and scientifically
substantiated change in the lives of
individuals. ECS engages in federal
and state funded projects focused on
PBIS to support research, teaching,
dissemination, and technical
assistance.
Thank you Bert Eliason & Katie
Conley!
Session Intentions
Goal
Use a problem-solving model focused on the use
of data to address discipline disproportionality
Resources
• Data Guide for School Teams (pbis.org, 2014)
• SWIS Ethnicity Report (pbisapps.org)
• Four-step problem-solving model
• Data Sources
Organizing and
• Common Metrics
Analyzing the Data • Drill Down Process
Putting It Into
Practice
• Simulated School Example
Intentions vs. Realities
We truly believe that…
• Schools want good outcomes for all
their students
• By and large, the profession is not
purposely leaving anyone behind
• But unfortunately, the outcomes are
not equal for everyone
Disproportionality:
Disproportionality in discipline referral data is
the overrepresentation or
underrepresentation of a subgroup of
students based on their race or gender in
comparison to the total student population.
PBIS.org (September 2014)
Does Disproportionality Really Exist?
Is it everywhere?
Who does it affect?
What can we do about it?
Disproportionality in School Discipline
(Losen et al., 2015)
http://civilrightsproject.ucla.edu/resources/projects/center-for-civil-rights-remedies/school-to-prisonfolder/federal-reports/are-we-closing-the-school-discipline-gap
How do we measure this?
• What is necessary?
• What else is recommended?
• How do we get this information?
U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot
(School Discipline) March 21, 2014
• Black students (16.4%) are 3 x more likely to be suspended or expelled than white
students (4.6%).
• Black boys and girls have higher suspension rates than any of their peers. Twenty
percent (20%) of black boys and more than 12% of black girls receive an out-ofschool suspension.
U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot
(School Discipline) March 21, 2014
U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot
(School Discipline) March 21, 2014
U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot
(School Discipline) March 21, 2014
Percent of Districts in Each State that would be
Significantly Disproportionate if Any Subgroup Had a
Risk Ratio Greater than Two Times the National Median
100%
90%
80%
70%
60%
50%
47%
40%
30%
20%
10%
Nat’l.
AK
AL
AR
AZ
CA
CO*
CT
DC
DE
FL
GA
HI
IA
ID
IL
IN
KS
KY
LA
MA
MD
ME
MI
MN
MO
MS
MT
NC
ND
NE
NH
NJ
NM
NV
NY
OH
OK
OR
PA
RI
SC
SD
TN
TX
UT
VA
VT*
WA
WI
WV
WY
0%
States?
State
Districts
Above
Threshold
Districts in the
State
% of Districts in State that
would be
Significantly Disproportionate
CT
122
197
61.9%
DE
19
43
44.2%
MA
257
418
61.5%
ME
59
259
22.8%
NH
62
178
34.8%
NJ
410
691
59.3%
NY
491
1008
48.7%
PA
338
738
45.8%
RI
33
57
57.9%
Nat’l.
8,148
17,371
46.9%
The Data Guide Identified:
1. Data Needed for Investigating Disproportionality
2. Four-step Problem Solving Model
3. Metrics to Use to Monitor Disproportionality
4. Bias in Decision Making – Explicit vs Implicit
5. Vulnerable Decision Points
From the Data Guide
Problem Solving Model
Why is it
happening?
Is there a
problem?
Is the plan
working?
1. Problem
Identification
2. Problem
Analysis
4. Plan
Evaluation
3. Plan
Implementation
What should
be done?
Step 1: Problem Identification
Use valid & reliable metrics.
Is there a
problem?
1. Problem
Identification
Quantify the difference
between current outcomes
and goals.
This is the performance gap!
Step 1: Problem Identification
For disproportionality
Is there a
problem?
1. Problem
Identification
• Quantify outcomes across
racial/ethnic subgroups
• Compare differences
• Specified group vs. Comparator group
• Subgroup vs. White subgroup
• Subgroup vs. All Other students
• Native vs. All Non-Native students
Multiple metrics are recommended!
IDEA Data Center, 2014
Risk Ratios
• A metric commonly used to measure
disproportionality
• Represents the likelihood of discipline referrals
for one group in relation to a comparison group
• Calculated by using the Risk Index of one group
divided by the Risk Index of the comparison
group
• Example threshold: If at or above 1.5, there is a
risk for one group to be over represented for
discipline referrals compared to another group
Automatically
calculates the
risk ratio to
compare one
group with all
other groups
SWIS Ethnicity Reports -
overrepresentation
underrepresentation
Automatically
calculates the
proportionality
between
subgroup’s
percent of
population and
subgroup’s
percent of
students with
referrals.
SWIS Ethnicity Reports -
overrepresentation
Automatically
calculates the
proportionality
between
subgroup’s
percent of
underrepresentation population and
subgroup’s
percent of total
referrals.
Step 1: Problem Identification
Review
1. Select multiple metrics to use
• Relative measure – Use Risk Indices to calculate Risk Ratio
• Absolute measure – ODRs per 100 Students per Day
• Composite measurements – What is composition of the problem?
2. Calculate metrics and compare to reasonable goals
–
–
–
–
Previous years from same school
Local, state or national norms
Logical criteria – “4/5ths Rule”
0.80 - 1.25
State’s reasonable threshold for “significant disproportionality”
3. Monitor metrics throughout the year
• Monthly or quarterly
• Be careful of small “Ns”
• Be careful of risk indices
Step 2: Problem Analysis
What is
happening and
why?
Is there a
problem?
1. Problem
Identification
2. Problem
Analysis
Step 2: Problem Analysis
Purpose: Identify context and underlying cause(s)
Focus:
Systems & practices that can be changed
Evaluate: Tier 1 (universal systems)
• Check fidelity of PBIS implementation
o TFI - CR Companion
Disparities other than discipline
•
•
•
•
Academic Placement
Attendance
School climate
Graduation
Discipline data for patterns of bias
 Explicit bias
 Implicit bias
What is implicit bias?
• Unconscious, automatic
 Generally not an indication of our beliefs and
values
 We all have it (even those affected by it)
• Based on stereotypes
• More likely to influence:
– Snap decisions
– Decisions that are ambiguous
Step 2: Problem Analysis
What is a Vulnerable Decision Point?
A VDP is a specific decision that is more
vulnerable to effects of implicit bias.
Two parts:
1. Elements of the situation
• External or environmental context
2. The person’s decision state
• Internal state or personal context
Step 2: Problem Analysis
Two Systems for Decision Making
System 1: Fast Decisions
– Automatic, snap judgments
– Intuitive, unconscious
System 2: Slow Decisions
– Deliberate decisions
– Allows for conscious attention
Kahneman, 2011
Step 2: Problem Analysis
VDPs from national ODR data
Subjective problem behavior
– Defiance, Disrespect, Disruption
Vague discipline system
– Major vs. Minor
Non-classroom areas
– Hallways
– Playgrounds
Afternoons
Step 2: Problem Analysis
What can we do about these VDPs?
Teach two-step Neutralizing Routine for Staff:
When you see problem behavior, stop and ask:
1. Is this possibly a VDP?
– Consider the context of the situation
– Consider your personal decision state
2. If so, use an agreed-upon alternative response
• Slow down, consider options
Step 2: Problem Analysis
Neutralizing Routines for Reducing Effects of
Implicit Bias
Self-assessment
“Is this a vulnerable
decision point?”
Setting event
Lack of
positive
interactions
with student
Fatigue
Antecedent
Loud
complaints
about work
(subjective
behavior)
Behavior
Consequence
Send
student to
office
(ODR)
Student
leaves class
(Escape social
interaction)
Alternative
Response
“See me after class.”
Step 2: Problem Analysis
Defining Problems with Precision
– Who is involved?
– What are the problem behaviors?
– Where is it happening?
– When is it happening?
– Why are these things happening?
• Perceived function of problem behavior
Step 2: Problem Analysis
When we believe we have a problem related to
disproportionality….
Identified Subgroup
Location
Assess
• PBIS implementation fidelity
o TFI-CR Companion
o Performance gaps
Time of Day
Problem Behavior
Motivation
•
•
•
•
Achievement gap
Academic placement
Attendance
School climate
Many seventh grade Black students are receiving referrals from the
classroom in the afternoon for inappropriate language. Referrals are
perceived to be task avoidance and getting adult attention.
SWIS Drill
When we have a problem with
disproportionate discipline…
Precise Problem
Statement
Step 2: Problem Analysis
SWIS Drill Down
Subgroup: African American Students
Who?
When?
What?
Where?
Why?
3rd grade
11:30 - Noon
Physical Aggression
Playground
Peer Attention
4th grade
8:00 AM– 9:30 AM
M-Defiance
Classroom
Avoid task
Inappropriate Language
Classroom
Hallway
Avoid task
Obtain Peer
Attention
7th grade
After 12:00 PM
Step 2: Problem Analysis
Precise Problem Statements
 African American students in the 3rd grade are receiving
referrals for physical aggression during noon recess. Referrals
seem to be related to gaining peer attention.
 African American students in the 4th grade are more likely to
receive referrals for minor defiance in the classroom during
the morning instructional block. Referrals seem to be related
to task avoidance.
 African American students in the 7th grade are receiving
afternoon referrals in the classroom and hallways for
inappropriate language. Referrals are related to avoiding tasks
and gaining peer attention.
Drill down
Step 3: Plan Implementation
What is
happening and
why?
Is there a
problem?
1. Problem
Identification
2. Problem
Analysis
3. Plan
Implementation
What should
be done?
Step 3: Plan Implementation
• Information from Step 2 is used to select tasks
and strategies to address the problem.
• An action plan is created to ensure adequate
implementation of the tasks and strategies.
• Action plans show everyone –
WHO will do WHAT by WHEN.
• Action plans that are published –
Help create accountability
Step 3: Plan Implementation
Examples
All issues
• Calculate and share disproportionality data regularly
Inadequate PBIS implementation
• Implement core features of PBIS to establish a foundation of
support
• Consider using TFI Culturally Responsive Companion
Misunderstandings regarding school-wide expectations
• Enhance culturally responsive PBIS with input from the
students/families
Academic achievement gap
• Implement effective academic instruction
Step 4: Plan Evaluation
What is
happening and
why?
Is there a
problem?
Is the plan
working?
1. Problem
Identification
2. Problem
Analysis
4. Plan
Evaluation
3. Plan
Implementation
What should
be done?
Step 4: Plan Evaluation
Evaluation
1. Regularly assess
• Progress and fidelity of plan implementation
• Fidelity of PBIS implementation
2. Calculate metrics from Step 1
3. Compare to the goal determined in Step 1
4. Share results with relevant stakeholders
5. Plan for what is next
Step 4: Plan Evaluation
Evaluation Time Frame:
Identify time periods for regularly evaluating and analyzing
disproportionality data.
Caution: Disproportionality metrics may not be sensitive to
rapid change.
– Consider monthly assessment of plan implementation &
quarterly assessment of disproportionality metrics.
– Avoid relying on risk indices as they will increase
throughout the year.
– Use multiple measures to ensure that you are tracking
the correct thing.
Problem Solving Model
Why is it
happening?
Is there a
problem?
Is the plan
working?
1. Problem
Identification
2. Problem
Analysis
4. Plan
Evaluation
3. Plan
Implementation
What should
be done?
PBIS.org - National Resource Center
pbis.org
is the national
resource center
for materials and
information
related to the
implementation
of PBIS with
fidelity.
 School
PBIS.org - National Resource Center
pbis.org
is the national
resource center
for materials and
information
related to the
implementation
of PBIS with
fidelity.
 School
 Equity & PBIS
5-point Multicomponent Approach to Reduce Disproportionality in Schools
1. Collect, Use, and Report Disaggregated Discipline Data
2. Effective Instruction
3. School-wide PBIS
4. Policies with Accountability for Disciplinary Equity
5. Teach Neutralizing Routines for Vulnerable Decision Points
Recommendations for Addressing
Discipline Disproportionality
• 5-Point Intervention Approach to Enhance Equity in
School Discipline
1. Collect, use, and report disaggregated discipline data
2. Implement a behavior framework that is preventative,
multi-tiered, and culturally responsive
3. Use engaging academic instruction to reduce the
opportunity (achievement) gap
4. Develop policies with accountability for disciplinary
equity
5. Teach strategies to neutra
Point #2: Implement a Behavior Framework
• Why start with a foundation of SWPBIS?
• Proactive, instructional approach may prevent
unwanted behavior and exposure to biased responses
to unwanted behavior
• Increasing positive student-teacher interactions may
enhance relationships to prevent challenges
• More objective referral and discipline procedures may
reduce subjectivity and influence of cultural bias
• Professional development may provide teachers with
more instructional responses
Effects of PBIS on Discipline
• Vincent et al., 2011: Statistically significant lower blackwhite ODR disproportionality in 72 schools implementing
SWPBIS than in 81 schools not implementing SWPBIS
• Vincent et al., 2009: Decreases in ODRs seen across
racial/ethnic groups in 69 schools implementing SWPBIS
• Scott, 2001: Larger decreases in suspensions for Black
students when SWPBIS implemented
• McIntosh et al., 2014: Sustained decrease in suspensions
over eight years of SWPBIS implementation in an
Indigenous school