INTER-RATER
AGREEMENT
IN KANSAS
Summer Principals Academy
July 22-24, 2014
Abilene, KS
2
What is Inter-Rater Agreement?
The degree 2 raters
using the same scale
give the same rating
in identical situations.
• Consistent
• Consistent
understanding process
• Consistent
• Consistent
agreement
credibility
3
Where is Inter-rater Agreement Useful??
Inter-Rater Agreement is useful in all
areas of administrative practice
including:
• Kansas Educator Evaluations
• Kansas Accreditation
• Personnel Decisions
• Family Engagement Programs
• Budget Management
4
What are the Goals of Inter-rater Agreement?
• The Inter-Rater Agreement Statistical Goal is 75% -
agreement with no rating more than one level apart.
• The Goal is to have an administrative Common Frame of
Reference for your district such as:
• Agreement with no rating more than one level apart.
• Consistent understanding
• Consistent agreement
• Consistent process
• Consistent credibility
5
How is Inter-rater
Agreement Helpful??
• What is a Common
Understanding?
• How does Inter-rater
Agreement help
administrators??
• Facilitates Quality
• Provide Consistent Messaging
• Impacts Increased Student
•
•
•
•
•
•
Growth
Enhances Trust
Facilitates Quality Feedback
Legally Defensible
Assists Accurate Data Collection
Allows for Increase Cooperation
over Time
Passes Public Scrutiny
6
District-wide Agreement
about External Criteria
• Agreed Upon External Criteria for:
• Observations
• Artifacts
• Units of Study
• Processes
• Program Development
• Time Limits
• Walk-Throughs
• Protocols
7
Pragmatics/Considerations
Potential Landmines
• Discuss with an Elbow
Partner Possible Issues that
may cause Rater Error
• Leniency: Benefit of the
•
•
•
•
Doubt
Central Tendency: Safety in
the Middle
Comparing One with Another:
Stick to the Rubric
Relying on Gut too Much:
Stick to the Rubric
Evaluating Absent Evidence:
Stick to What is Observed
8
How to Train Administrators
in using Inter-rater Agreement
• Inter-Rater Agreement Training – Things to Consider:
• Synchronous and Asynchronous
• Experience of the Participants
• How to Ensure Fidelity
• Use of Video Examples
• Use of Case Studies
• Bias Exists
• Single versus Multiple Observations
• Professional Judgment
• Degree of Familiarity
9
Getting Started in YOUR District
• Getting Your Bearings
• Developing a Guidebook
• Developing a Training Protocol
10
I. Getting Your Bearings
Pilot with Administrative Team
Pilot Approach: Start with Safe Example:
1.
•
•
Use a video from YouTube
Assess Specific Area of Teaching
•
•
•
Student Engagement
Teacher Activity
Classroom Environment
Review current issue/goal/vision/needs
Review current practices/instrumentation
2.
3.
•
•
•
Rubrics
Protocol
Training
11
II. Developing a Guidebook
• Paper and Electronic: Synchronous and Asynchronous
• EXCELLENT Administrative Council Endeavor –
Ultimately to be used during Professional Learning
Times with the Staffs
• Pieces and Parts
• Terms/Definitions: Frame of Reference/Common Understanding
• “Demonstrates”
• “Frequently”
• “Usually”
12
II. Developing a Guidebook
• Pieces and Parts (continued)
• Examples and Rubrics
• Areas to Observe
• Establishes/maintains a culture of safety, respect
and rapport
• Knows how to make content accessible to all
learners
• Plans standards-based instruction and formative
assessments
• Uses a variety of instructional strategies to meet
needs
• Deepens understanding through critical thinking,
problem-solving and student reflection
• Analyzes student performance and provides
feedback
• Collaborates to support student learning
13
II. Developing a Guidebook
• Pieces and Parts (continued)
• Exemplars: Video and Artifacts
• Master-Coded Videos: Leads to high level of
agreement
• Example Artifacts
• Classroom Rules
• Lesson Plans
• Parent Note Home
• Lesson Plan for Adults: Andragogy
14
III. Developing a Training Protocol
• Step-by-Step Procedure
1.
Initial Training: Goals/Terms
• Practice Interpreting Rubrics
• Differentiate between Different Performance Levels with Examples
2.
Calibration of the Players
• Video and Artifacts
3.
4.
5.
6.
7.
Review of the System
Adjustments (If needed)
Implementation
Evaluate/Reflect/Adjust
Re-Calibrate…
15
Take Aways – To Do List
• Start Slowly
• Start with
“Engagement” with
a Small Group
• Get Agreement
• Gain Confidence
Over Time
• Write and Refine in
Teams
16
Final Thoughts
• Start with the End in Mind
• Develop Common
Understandings
• Use Groups to Ensure
Buy-In
• Build Measurable Steps to
Reach Each Goal
17
Contact Information
Teacher Licensure and Accreditation Team Director
Scott Myers
785-296-8010
[email protected]