Los Angeles Unified School District
Local District E
Instructional Support Services
Standards-Based Education
Performance Assignment
Training Session for Analytic Scoring
2003-04
PERFORMANCE ASSIGNMENT
ELEMENTARY AND MIDDLE SCHOOL
LOGISTICS
Data Results
Changes
Testing Schedule
Rater Reliability
PERFORMANCE ASSIGNMENT
DATA RESULTS
In grades 2-9, scores increased 2% (total
students tested = 276,319)
40% scored proficient or above in English
(107,497)
42% scored proficient or above in Spanish
(3,181)
29% improved their performance level
(53,608)
PERFORMANCE ASSIGNMENT
DATA RESULTS
Responses from Student and Teacher
Questionnaire validate the correlation
between students’ opportunity to learn and
their Performance Assignment scores.
Positive relationship between higher
frequency of quality reading and writing
experiences and higher scores.
Students who have less practice with
reading and writing score lower.
PERFORMANCE ASSIGNMENT
CHANGES FOR 2003-04
Elementary Grades
Performance Assessment aligns with OC
Third periodic assessment
Middle School
Third periodic assessment
SBACs & Literacy Coaches work together
Analytic scoring replaces holistic scoring
PERFORMANCE ASSIGNMENT
TESTING SCHEDULE
Elementary
Third Periodic Assessment
Middle School
Third Periodic Assessment
Single Track:
Three Track: A
B&C
Four Track: A & B
C&D
February 23-27
March 29 - April 3
January 26-30
March 1-5
January 26-30
PERFORMANCE ASSIGNMENT
RATER RELIABILITY
Target Percentage
2001-02
2002-03
2003-04
50-60%
55-65%
60-70%
PERFORMANCE ASSIGNMENT
RATER RELIABILITY – SCHOOLS
70-79% RATER AGREEMENT
Allesandro Coop. Learning Magnet (grade 4)
Alta Loma (grade 2)
Arlington Heights (grade 4)
Atwater (grade 3)
Buchanan (grade 3)
Bushnell Way (grade 4)
Clifford (grade 4)
PERFORMANCE ASSIGNMENT
RATER RELIABILITY – SCHOOLS
70-79% RATER AGREEMENT
Dayton Heights (grade 2)
Fletcher Drive (grades 2 and 3)
Garvanza (grade 3)
Grant (grade 4)
Hobart (grade 3)
Ivanhoe (grade 4)
Micheltorena (grade 4)
PERFORMANCE ASSIGNMENT
RATER RELIABILITY – SCHOOLS
70-79% RATER AGREEMENT
Monte Vista (grade 5)
Pio Pico (grade 3)
Ramona (grades 2 and 3)
San Pascual (grade 3)
Selma (grade 4)
Wilshire Crest (grade 3)
Wilton Place (grade 3)
PERFORMANCE ASSIGNMENT
RATER RELIABILITY – SCHOOLS
80-89% RATER AGREEMENT
Atwater (grade 2)
Franklin (grade 3)
Gardner (grade 5)
Grant (grade 3)
Selma (grade 3)
Wilton Place (grade 5)
PERFORMANCE ASSIGNMENT
RATER RELIABILITY – SCHOOLS
90-100% RATER AGREEMENT
Aragon (grade 2)
Arroyo Seco Museum Sci. Magnet (grade 5)
Cheremoya (grade 4)
Dorris Place (grade 4)
Eagle Rock G/HA Elem. Magnet (grade 3)
PERFORMANCE ASSIGNMENT
RATER RELIABILITY – SCHOOLS
90-100% RATER AGREEMENT
Micheltorena (grade 3)
Pio Pico (grade 2)
Queen Anne (grades 4 and 5)
Selma (grade 2)
Rater Reliability Data from: Use of Periodic
Assessments to Promote Literacy Achievement
PERFORMANCE ASSIGNMENT
RATER RELIABILITY
SECONDARY SCHOOLS
Eagle Rock (gd. 8)
Pio Pico (gd. 7)
Irving (gd. 8)
Burbank (gd. 6)
Burroughs (gd. 8)
LeConte (gd. 6)
King (gd. 6)
Mount Vernon (gd. 7/8)
74%
58%
57%
54%
51%
51%
50%
47%
PERFORMANCE
ASSIGNMENT
Analytic Scoring
for Elementary
SCORING
Holistic Scoring
Procedure which yields a
single score based on an
overall impression of
students performance
SCORING
Analytic Scoring
Procedure which yields multiple
scores based on two or more
separate scales for evaluating
performance on selected
dimensions or traits; each trait
receives a separate score.
SCORING
looks at the entire
writing piece
captures an overall
picture of student’s
writing
a form of
assessment
that looks at
writing
analyzes separate a
spects of student
writing
writing is read throug
h for each score
scores are given for
both use rubric
each analytic categor
s
y
gives one score
both can be
uses multiple scale
indicators fo
s
Provides more detaile
r teacher’
a single scale
info for improving i
s
next d
nstruc- tion and commu
steps
is often mor
nicating
with st
e efficient
udents
SCORING
Analytic Scoring for Unit 3
Applications (genre)
=
Writing Strategies (traits) =
45%
45%
WOEL Writing and Oral English Language
(Conventions)
=
10%
PREPARING FOR A SCORING TRAINING SESSION
Scoring Handbook (Part I)
Major Responsibilities:
Determine who will attend
Schedule training session
Find appropriate location
Prepare training packets
PREPARING FOR A SCORING TRAINING SESSION
Scoring Handbook (Part I)
Materials
Training packet (grade level)
Prompt and rubric
Set of anchor and training papers
Scoring Handbook
Assessment form for recording
student scores (Literacy Coach)
PREPARING FOR A SCORING TRAINING SESSION
Scoring Handbook (Part I)
Materials
Training Paper Score Sheet
Rater Agreement Summary
Sheet
Calculator
Training Papers Key
PREPARING FOR A SCORING TRAINING SESSION
Scoring Handbook (Part I)
Time
1.5 to 2.5 hours
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Open Court Writing
Unit 1
Scoring Materials
Grade 3
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 1
Read the task (prompt) carefully…
Think about the stories you have read in the
Friendship Unit and what it is like to be a friend.
Write a story about your best friend and tell
about a special time or memorable event that
you have shared as friends. Be sure to use
concrete sensory details to describe where you
were. Describe what was special to you and
your friend and tell why this special time was
something you will always remember.
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 2
Review the rubric for Applications…
Working with a partner, carefully
read aloud the description of
each score point in the rubric.
Start with level 3/Proficient.
5 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 3
Read and discuss anchor paper
focusing only on Applications
Anchor papers represent the lowest
performance level
In packet, there are anchor papers
for 2, 3, and 4
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 3
Focusing only on Applications…
Read Anchor 3 paper aloud. Briefly discuss
the qualities of the paper, referring to each
component of the rubric. Describe why it
received the score it did. Repeat for Achors
2 and 4. Discuss distinctions in levels
referring to the rubric as needed.
10 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 4
Repeat the same process from Steps 2
and 3 focusing only on the Writing
Strategies category.
Step 2 – review the rubric
Step 3 – read and discuss anchor papers
10 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 5
Repeat the same process from
Steps 2 and 3 focusing only on the
WOEL (Conventions) category.
Step 2 – review the rubric
Step 3 – read and discuss anchor papers
5 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6
Score training papers
Training papers already have
assigned scores by expert raters
Goal is to obtain at least 70%
agreement
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
A. Score Training Paper A
Read Training Paper A. Compare each
response to the set of anchor papers
and rubric criteria for each category.
Use the Training Paper Score Sheet
(at the bottom of your packet) to
record scores.
5 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
A. Score Training Paper A
You should have 3 scores for paper A.
Using the Rater Agreement Summary
Sheet, record the frequency of each
score (Applications, Writing
Strategies, Conventions). In each
category, circle the highest number.
5 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
A. Score Training Paper A
Overall Score = 2
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
B. Score Training Paper B
(use same procedures as before)
Read Training Paper B. Compare each
response to the set of anchor papers
and rubric criteria for each category.
Use the Training Paper Score Sheet
(at the bottom of your packet) to
record scores.
5 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
B. Score Training Paper B
(use same procedures as before)
Read Training Paper B. Compare each
response to the set of anchor papers
and rubric criteria for each category.
Use the Training Paper Score Sheet
(at the bottom of your packet) to
record scores.
5 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
B. Score Training Paper B
Overall Score = 3
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
C. Score Training Paper C & D
Score training papers C and D
following the same procedures
used to score training papers A
and B.
5 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
C. Score Training Paper C & D
Training Paper C – Score=3
Training Paper D – Score=2
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 6: Score Training Papers
D. Check rater agreement
Using the Rater Agreement
Summary Sheet, and the sample
page WS-4 in your Scoring
Handbook, follow the directions to
calculate rater agreement.
10 minutes
CONDUCTING A SCORING TRAINING SESSION
Scoring Handbook (Part II)
Step 7: Student Writing Samples
Score student writing samples
and record on the Classroom
Assessment Summary Form
provided by the Literacy Coach.