Harvesting Data for
Problem Solving
REBECCA PIERMATTEI
Wayne Hickman
Christina Jordan
PBIS Maryland Coaches Meeting
April 30, 2014 & May 5, 2014
WHAT CAN WE LEARN
FROM DATA?
 What we are doing well and what challenges
we face
 Which students, teachers, systems need more
support
 Whether our support is successful
In a Nutshell
 Know what data you need and how to access the
data.
 Make sure your data is reliable
 Review and share data on a regular basis.
 Keep it simple and reasonable.
 Use your data to guide your decision making.
 Use your data to evaluate individual progress as well
as program and school-wide success.
BEFORE YOU START…
Resource Mapping
Resource MappingInventory of Current Practices
• What are the practices in place at each
tier of the triangle?
• Are they evidence-based practices?
• How are you measuring effectiveness of
practices (data)?
• Who are the service delivery
teams/personnel (e.g., graduation coach,
PALS teacher, Math Coach)
Triangle
Activity:
Tier 3
Practices, Initiatives,
Programs for a FEW
Applying the
Three-Tiered
Logic to
Your School
Tier 2
Practices, Initiatives,
Programs for SOME
Tier 1
Practices, Initiatives,
Programs for ALL
7
BEFORE YOU START…
 Resource Mapping
 Early Warning Indicators
Early Warning Indicators
On-Track
On-Track Indicators
At-Risk for
Off Track
Off-Track
Highly OffTrack
Extremely
Off-Track
Course
Performance in
Core Subjects
GPA
Credits
FCAT/
Concordance
Scores
Attendance
Meeting all
2.5 or more
graduation
requirements
Cs or better in all
areas
Meeting credit
graduation
requirement for
grad plan year
Level 3 or Above
or concordant
scores within the
same school year
4% or less
absences per
quarter or
semester
Lacking 1
graduation
requirement
2.0 to 2.49
Behind
1 Credits
Level 2 on
FCAT
5% or more
absences per
quarter or
semester
Lacking 2
graduation
requirements
Failing 1-3
classes
Less than 2.0
Behind 3 credits
Not passed both
sections of 10th
grade FCAT or
retakes
No concordant
scores
10% absences
per quarter or
semester
Lacking 2 or
Less than or
more graduation equal to 1.5
requirements
Currently failing
3 or more classes
Behind 4 or more Not passed 10th
credits
grade FCAT or
retakes
No concordant
scores
15% or more
absences per
quarter or
semester
Meeting no
Less than or
graduation
equal to 1.0
requirements
2-3 Years Behind
Not meeting
cohort
graduation plan
Not passed 10th
grade FCAT or
retakes
No concordant
scores
20% or more
absences per
quarter or
semester
Office
Discipline
Referrals
Additional
Factors
3 or less Level I Disengagement
and/or minor
No extra curricular
referrals
involvement
Substance Abuse
High Mobility
Mental health
4 or less Level I issues
and/or minor
Free/Reduced
referrals
lunch
Level II ODRs Foster/group home
per semester
Transient/Homeles
5 or more Level s
I and/or Level II Parent
ODRs per
unemployment
semester
Student
employment
Changes in
5 or more Level behavior/
appearance
II ODRs for
fighting/
More recent
profanity/
traumatic event
disruption per
Missed guidance
semester
appointments
No show for
Established
pattern of severe yearbook picture
behavior
Level II & III
ODRs
Part A: SCHOOL-WIDE DATA
 Identify what academic and behavioral data you need
 ACADEMIC: Homework completion, GPA, Credit Accrual,
Benchmark Assessments
 BEHAVIORAL: ODRs (Big 5), attendance,
suspension/expulsion, minor incidents, nursing and counseling
logs
 Identify the sources of that data
 Identify the person responsible for getting and presenting the
data
 Identify how often and it what manner it will be shared with
the team, faculty, and administration
Part B: WHAT DO WE DO
WITH THE DATA?
 Celebrate success!
 Identify problems with PRECISION using the 5 W’s:
 What, Where, When, Who, Why
 Determine whether your data indicates a need for schoolwide practice or small group/individual response?
 Develop Solution Options
 Create Problem Solving Action Plan
 Evaluate Solution
HELPFUL TOOLS
 Data Decision Rules
 Problem Solving Action Plan
IF:
Focus On:
> 35% of students receive 1 or more referrals
Average referrals per student > 2.5
Schoolwide
Systems
> 35% of referrals come from non-classroom settings
> 15% of students who receive a referral are referred from
non-classroom settings
Non-Classroom
Systems
>50% of referrals are from classroom settings
>40% of referrals come from less than 10% of the classrooms
Classroom
Systems
At least 10-15 students have 5 or more referrals
Targeted Group
Interventions
<10 students with 10 or more office referrals
<10 students continue w/referrals after targeted intervention
Small number of students destabilizing overall functioning
Individual
Student Systems
EXAMPLE
 PRECISION PROBLEM STATEMENT:
Many students from all grade levels are engaging in
disruption, inappropriate language, and harassment in the cafeteria
and hallway during lunch, and the behavior is maintained by peer
attention.
Identify 5 W’s
Identify which system you will target
Problem Solving Action Plan
Precise Problem
Statement
Solution Actions
Who?
When?
Goal, Timeline, Rule
& Updates
Many students from all
grade levels are engaging
in disruption,
inappropriate language
and harassment in
cafeteria and hallway
during lunch, and the
behavior is maintained by
peer attention
Prevention: Teach behavioral
expectations in cafeteria
Teachers will take class to
cafeteria; Cafeteria staff will
teach the expectations
Rotating schedule on
November 15
Goal: Reduce cafeteria
ODR’s by 50% per month
(Currently 24 per month
average)
Changes begin on
Monday
Timeline: Review Data &
Update Monthly
A smaller number of
students engage in
skipping and
noncompliance/defiance
in classes, (mostly in
rooms 13, 14 and 18), and
these behaviors appear to
be maintained by escape.
Recognition: Establish “Friday
Five”: Extra 5 min of lunch on
Friday for five good days
Maintain current lunch schedule,
but shift classes to balance
numbers
Principal to adjust schedule and
send to staff
School Counselor and Principal
will create chart & staff extra
recess
Principal to give
announcement on
intercom on Monday
Corrective Consequence- Active
supervision and continued early
consequence (minor/major
ODR’s)
Hall and Cafeteria Supervisors
Ongoing
Data Collection – Maintain ODR
record & supervisor weekly
report
SWIS data entry person &
Principal shares report with
supervisors
Weekly
Extinction: Encourage all
students to work for “Friday
Five”… make reward for
problem behavior less likely
TIER 2/3: Sorting Students into
Interventions
BEFORE YOU START…
 Resource Mapping
 Early Warning Indicators
 Decision Rules for Access to Interventions
Decision Rules for Access to Advanced Tiers
 Specific to each intervention
 Identify objective variables/criteria as well as the
minimum/maximum for each
 EX: GPA between 1.0 and 1.5
 You may need to scale up or down depending on your capacity
Part C: Sorting Students into
Interventions
 1. How will students be identified for this intervention? What
are the data decision rules for access – criteria used, min and
max for each variable?
 2. What data do we need?
 3. Where will we get the data?
 4. Who will be responsible for collecting the data?
HELPFUL TOOLS
 EARLY WARNING INDICATORS
 RESOURCE MAPPING
 DECISION RULES FOR ACCESS TO
ADVANCED TIERS
Part D: Progress Monitoring and
Program Evaluation
1. How will student progress be measured/monitored? Who is
responsible for measuring/monitoring progress?
2. What indicators/benchmarks will show that a student is
responding to the intervention? Not responding?
3. What indicators/benchmarks will show that a student is ready
to exit the program?
4.
How will overall program success be measured/monitored?
HELPFUL TOOL
Intervention Tracking Tool
*Illinois PBIS Network
Tier 2/Tier 3 Intervention Tracking Tool
School Name: _____________________________ __
Interventions
Check-in Check-out
(CICO)
Social/Academic
Instructional Groups
# Students # Students
# Students # Students
Participating Responding Participating Responding
Total School Population as of October 1:________
Individualized CheckIn/Check-Out, Groups &
Mentoring
# Students # Students
Participating Responding
Brief FBA/BIP
(Functional Behavior
Assessment/Behavior
Intervention Planning)
# Students # Students
Participating Responding
Complex FBA/BIP
Wraparound Support
# Students # Students
Participating Responding
# Students # Students
Participating Responding
July
August
September
October
November
December
January
February
March
April
May
June
Data-based Decision-rules for defining “response to intervention”: Please list below your data-based decision-rule/s to determine youth ‘response
for each of the six levels of intervention. Ex. Students received 80% or better on Daily Progress Report for 4 consecutive weeks.
Responding to Check-in Check-out (CICO):
Responding to Social/Academic Instructional Groups:
Responding to Individualized CICO, Groups & Mentoring:
Responding to Brief FBA/BIP:
Responding to a Complex FBA/BIP:
Responding to Wraparound Support:
OH NO!
What do you do if your data shows
your program is not working?
FIDELITY MEASURES
 Determine whether you are implementing
with fidelity
(Are you doing it the way you are supposed to be doing it?)
 Consider BOQ, SET, PBIS-TIC
 Consider intervention specific fidelity
measures
IMPORTANT
CONSIDERATION
DATA MUST BE RELIABLE
 EXAMPLE: Behavior Referrals
 Define your behaviors
 Agreement re: major/minor
 Standardized procedures for gathering referrals
 All staff trained in how to complete and submit referrals
In a Nutshell
 Know what data you need and how to access the
data.
 Make sure the data is reliable
 Review and share data on a regular basis.
 Keep it simple and reasonable.
 Use your data to guide your decision making.
 Use your data to evaluate individual progress as well
as program and school-wide success.
Acknowledgements
 MDS3 is funded by a grant from the USDOE.
 Federal Grant CFDA# Q184Y100015
 Sheppard Pratt Health System:
 Rebecca Piermattei, M.S. [email protected]
 Wayne Hickman, Ed.D. [email protected]
 Christina Jordan, M.Ed. [email protected]
 Maryland State Department of Education
 Johns Hopkins University