Session 2: The “R” word…
Tresha Dutton, Ph.D.
Outcomes Assessment Coordinator
Anne Marie Karlberg, Ph.D.
Director of Assessment and Institutional Research
Friday October 12, 2012
LDC 211
1
Check off all items from the list that apply to
you.
2
Overview
1. What is a rubric?
2. Do you need a rubric?
3. How to create a rubric
4. Activity
5. Analytic rubrics
6. Holistic rubrics
What is a rubric?
4
A rubric is a tool that…
• articulates consistent criteria for evaluating student
work
• divides an outcome into its component parts
• provides a detailed description of acceptable or
unacceptable performance
5
Example: Science lab outcome
Levels of performance
Criteria
Proficient
Acceptable
Limited
Thorough
observations
All
observations
are clear and
detailed
Most
Observations
observations are are absent or
clear and
vague
detailed
Reasonable
predictions
All predictions
are reasonable
Most predictions Predictions
are reasonable
are absent or
irrelevant
Appropriate
conclusion
Conclusion is
consistent with
observations
Conclusion is
consistent with
most
observations
Conclusion is
absent or
inconsistent
with
observations
6
Example: Solo audition outcome
NA
Poor
Fair
Good
Very
Good
Vocal
tone
0
1
2
3
4
5
Vocal
technique
0
2
4
6
8
10
Rhythm
0
1
2
3
4
5
Diction
0
1
2
3
4
5
Musicality
0
3
6
9
12
15
Excellent
7
Example: Oral comm. outcome
Mastery
•usually makes eye contact
•volume is always appropriate
•enthusiasm present throughout presentation
•summary is completely accurate
Proficiency
•usually makes eye contact
•volume is usually appropriate
•enthusiasm is present in most of presentation
•only one or two errors in summary
Developing
•sometimes makes eye contact
•volume is sometimes appropriate
•occasional enthusiasm in presentation
•some errors in summary
Inadequate
•never or rarely makes eye contact
•volume is inappropriate
•rarely shows enthusiasm in presentation
•many errors in summary
8
Do you need a rubric?
If you checked off 3 or more items…
you should consider one!
9
Why use a rubric? Rubrics…
•
•
•
•
•
•
•
•
•
help clarify vague, fuzzy outcomes.
clarify expectations.
help students self-improve.
can inspire better performance.
make scoring more accurate, unbiased, consistent.
make scoring easier and faster.
improve feedback to students.
reduce arguments with students.
improve feedback to faculty/staff.
How to create a rubric
1.
2.
3.
4.
5.
6.
7.
Assignment details
Checklist (optional)
Criteria
Rating scale (levels of performance)
Descriptions of the criteria (optional)
Weight (optional)
Comments (optional)
Parts of a rubric
1.Assignment
details
2. Checklist
(optional)
3. Criteria
(indicators)
4. Rating scale
(level of
performance)
5.Descriptions
of criteria
(optional)
6. Weight
(optional)
7.Comments
(optional)
1.Assignment
details: List the
learning
outcome,
assignment title
and description
(if too long,
abbreviate
description and
list reference
[e.g., “see
syllabus”]).
2. Checklist
(optional): List
the minimum
criteria that are
non-negotiable
(e.g., My paper
is at least 500
words. I spellchecked and
proof-read my
paper.)
3. Criteria
(indicators):
List the most
important
indicators (or the
component parts)
that will be used
to evaluate the
outcome.
• Use 1-4 words.
• Avoid
descriptions of
the quality of
the indicator
(e.g., say
“Organization”
, rather than
“Good
organization”).
• 8 maximum.
4. Rating scale
describes
how well or
poorly the
task has been
performed.
• Use terms
that are
descriptive
and clear.
• Aim for 4
levels (3-5
okay, may
start with 3).
• Keep the
distance
between
levels equal
in value.
• List the
highest level
on the far
left.
5.Descriptions of
criteria:
Describe what
constitutes each
level of
performance for
each criterion
identified.
• Fill in highest
first, then
lowest, then
middle.
• Identify at the
least a
description of
the highest level
of performance.
• Lowest level:
describe the
most common
ways students
fail to meet
the highest
level of
expectations.
• Some rubrics
build on the
previous
performance
level.
• When a
student’s
performance
has elements
from 2
different
columns,
circle/check
elements
from each
column that
apply.
6.Weight
(optional):
Determine
the relative
weight of
each
criterion
based on its
level of
importance.
7.Comments
(optional):
(optional):
Add
Add
personalized
personalized
comments,
comments,
eitherspecific
specific
either
each
totoeach
criterionoror
criterion
overallatatthe
the
overall
bottomofofthe
the
bottom
rubric.
rubric.
How to create
a rubric
1.Assignment
details: List
the learning
outcome,
assignment
title and
description.
2.Checklist
(optional):
List the
criteria that
are nonnegotiable.
3.Criteria
(indicators):
In 1-4 words,
list the most
important
criteria that
will be used to
evaluate the
outcome.
4.Rating scale:
Determine the
number of levels
of performance
(typically 4, start
with 3, 5 max).
List the highest
level on the far
left.
5.Descriptions of
criteria: Write a
brief description
for each level of
each criterion
identified or,
minimally, the
highest level.
Start by
describing the
highest then the
lowest levels of
performance.
Avoid using
positive or
negative
comments.
6.Weight
(optional):
Determine the
relative weight of
each criterion.
7.Comments
(optional): Add
personalized
comments.
Activity: That’s the way the cookie
crumbles
In groups of 5…
1. Rate each cookie on a scale of 1-10. Write 1
sentence explaining why you give it that rating.
2. Rate each cookie on a scale of 1-10 based on the
following criteria: chocolate chips, texture, color,
and taste.
3. Rate each cookie based on the checklist rubric.
4. Rate each cookie based on the rating scale rubric.
5. Rate each cookie based on the descriptive rubric.
6. Rate each cookie based on the descriptive
weighted rubric.
7. Rate each cookie based on the holistic rubric.
21
Activity discussion
We evaluated the cookie in different ways:
1. Rating form: Given no criteria
2. Structured observation rubric: Given only criteria
3. Checklist rubric
4. Rating scale rubric: given criteria and scale
5. Descriptive rubric: given criteria with descriptions
and scale
6. Descriptive rubric with weighting
7. Holistic rubric
22
Rating form
Rating out of 10
Cookie 1
Cookie 2
Cookie 3
Analytic rubrics
…identify and assess individual criterion of a finished
product.
Better suited for…
• providing feedback about relative strengths and
weaknesses of different components.
• providing detailed feedback.
• assessing complicated skills or performance.
• students to self-assess their understanding or
performance.
24
Types of analytic rubrics
1. Structured observation rubric
2. Checklist rubric
3. Rating scale rubric
4. Descriptive rubric
25
1. Structured observation rubric
Feedback
Criteria
Cookie 1
Cookie 2
Cookie 3
Chocolate chips Lots of chips.
Low quality milk Only 3 chips.
chocolate was
used.
Texture
Cookie crumbled Moist
Too hard
Color
Golden brown
Burnt
Brown
Taste
Delicious
Too salty
Too greasy.
1. Structured observation rubric
A rubric without a rating scale. The criteria do not identify
the highest level of performance. Provides only qualitative
feedback.
Better suited for…
• doing qualitative and formative assessments.
• assisting faculty to articulate criteria for rating scale or
descriptive rubrics.
2. Checklist rubric
Criteria
Chocolate chips: The highest quality
dark chocolate chips are used and there at
least a couple of chips in every bite.
Cookie 1
Yes
No
Cookie 2
Yes
No
Cookie 3
Yes
No
Texture is chewy.
Yes
No
Yes
No
Yes
No
Color is golden brown.
Yes
No
Yes
No
Yes
No
Taste: The cookie has a home-baked
flavor.
Rating out of 10
Yes
No
Yes
No
Yes
No
8.5 / 10
9 / 10
8 / 10
2. Checklist rubric
Use checklist rubrics to…
Examples…
Record observed
performance/task (useful for
self or peer assessment). Good
indicator of “can do–can't do.

Keep track of progress over
time (e.g., pre/post test). Useful
for formative evaluation.


May use to indicate minimum
requirements (non-negotiable)
of a product, which are not
included in the rubric. Isn’t
about quality—it just lists the
required elements (“present—
absent”).


Can name 5 sports.
 Can ask a question about what others do in
their free time.
Can respond to simple yes/no questions.
Can respond with a couple of words to
simple questions.
 Can communicate with peers in informal
conversations.
My paper is typed and double-spaced.
My paper is at least 500 words.
 I spell-checked and proof-read my paper.
 I submitted a copy of the rubric with my
assignment.
 I included a list of resources.
3. Rating scale rubric
Criteria for
delicious
Delicious Tasty Edible
Chocolate chips:
4
3
2
The highest
quality dark
chocolate chips are
used and there are
a couple of chips
in every bite.
Texture is chewy.
4
3
2
Color is golden
brown.
Taste: Has a
home-baked
flavor.
Inedible Comments
Excellent
1
premium
chocolate chips!
Subtotal
3
Could use more
chocolate flavor.
4
3
2
1
Moist, breaks
apart
easily with some
crumbs.
Too dark.
4
3
2
1
A little too salty.
1
3
3
2
Total Score: 11 / 16
Percent: 69%
3. Rating scale rubric
A checklist with a rating scale that contains only the
description of the highest level of performance.
Better suited for…
• quick and easy rubric creation.
• first step in rubric creation.
• minor assignments.
4. Descriptive rubric
Level of Performance
Subtotal
3
Criteria
Delicious (4)
Tasty (3)
Edible (2)
Inedible (1)
Comments
Chocolate  uses highest  uses highest  uses standard
 uses standard
Excellent
chips
quality dark
quality milk
commercial dark
commercial
premium chips!
(quality /
chocolate
chocolate
chocolate
milk chocolate
number)  has chips in  has chips in
 has chips in 50%
 has too few or
Could use more
every bite
about 75% of
of bites
too many chips chocolate
bites
flavor.
Texture
 is chewy and  is chewy in
 is either crispy /
 is too hard or
Moist, breaks
3
moist, but
middle, crisp
crunchy or
 is uncooked
apart
not greasy.
on edges
 is 50% uncooked
easily with
some
crumbs.
Color
 is golden
 is either dark
 is either dark
 is burned or
Perfect.
4
brown
from
brown from
 is too light
overcooking
overcooking or
or
 is light from
 is light from
being raw
undercooking
Taste
 has a rich
 has a quality
 is tasteless
 has a storeA little too
3
chocolate,
store-bought
bought flavor; salty.
buttery, and
taste
stale, chalky;
home-baked
preservative
flavor.
after taste
Total Score
13/16
Percent
81%
4. Descriptive rubric
Replaces the checkboxes or circles of rating scale rubrics
with brief description of the performances that merit each
possible rating. Explicitly documents standards for student
performance.
Better suited for…
• providing students with clear, detailed feedback on
strengths and weaknesses.
• increasing consistency among faculty.
5. Descriptive rubric with weighting
Level of Performance
Criteria
Delicious (4)
Tasty (3)
Chocolate  uses highest  uses highest
chips
quality dark
quality milk
(quality /
choc.
chocolate
number)
 has chips in
 has chips in
every bite
Texture
Color


is chewy
and moist,
but not
greasy.
is golden
brown



Taste

has a rich
chocolate,
buttery, and
home-baked
flavor.

Edible (2)

about 75% of
bites
is chewy in
middle, crisp
on edges

is either dark
from
overcooking
is light from
undercooking

has a quality
store-bought
taste




uses
standard
commercial
dark choc.
has chips in
50% of bites
is crispy /
crunchy or
is 50%
uncooked
is either
dark brown
from
overcooking
is light from
being raw
is tasteless
Inedible (1)







uses standard
commercial
milk
chocolate
has too few /
many chips
is too hard or
is uncooked
is burned or
is too light
has a storebought
flavor; stale,
chalky;
preservative
after taste
Comments
SubScore Weight total
Excellent
premium
chips! Needs
more
chocolate
flavor.
Moist, breaks
easily with
some
crumbs.
Perfect.
3
X 4
12
3
X 2
6
4
X 1
4
A little too
salty.
3
X 4
12
Total Score 34/44
Percent 77%
3. Rating scale rubric with weighting
Criteria for
Subdelicious
Delicious Tasty Edible Inedible Comments
total
Excellent premium
Chocolate chips:
4
3
2
1
3
chocolate chips!
The highest
Could use more
quality dark
chocolate flavor.
chocolate chips are
used and there are
a couple of chips
in every bite.
Moist, breaks apart
Texture is chewy.
4
3
2
1
3
Color is golden
brown.
Taste: Has a
home-baked
flavor.
2
1.5
1
.5
easily with some
crumbs.
Too dark.
2
1.5
1
.5
A little too salty.
1
1.5
Total Score: 8.5 / 12
Percent: 71%
Holistic rubrics
Assess student work as a whole.
Better suited for…
• providing quick snapshots of achievement (where a gross
judgment is desired).
• providing feedback when a single set of criteria is
adequate to define quality.
Not suited for providing detailed feedback to students on
specific ways to improve.
36
Holistic rubric: Example
Delicious
Tasty
Edible
Inedible
High
quality
dark;
several
chips in
every bite;
chewy;
golden
brown;
home-baked
taste
High quality
milk; chips in
about 75% of
bites;
chewy in
middle, crisp
on edges;
either light
from
overcooking or
light from
being 25%
raw; quality
store-bought
taste
Standard
commercial dark;
chips in 50% of
bites; texture
either
crispy/crunchy or
50% uncooked;
either dark
brown from
overcooking or
light from
undercooking;
tasteless
Standard
commercial milk;
too few or too
many chips;
texture resembles a
dog biscuit;
burned;
store-bought
flavor, stale, hard,
chalky,
preservative after
taste
37
Dos
• Limit the rubric to 1 page.
• Always discuss the criteria in the rubric with students before
they begin the assignment.
• Have students staple rubric to completed assignment when
they hand it in (no rubric, no grade—encourages students to
read rubrics).
• Include personalized comments on assignment or rubric.
• When possible, involve students in creating or modifying
rubrics (results in better teaching and learning).
• Try to create rubrics that can be used for multiple
assignments.
38
Homework
1. Create a rubric for at least one course outcome. Rubric
templates are located on the assessment website
 Whatcom’s homepage
 About WCC
 Assessment and Accreditation
 Course outcomes \ Rubrics
2. Pilot the rubric.
3. Come to the next session (Fri. Nov. 2) with the rubric
and your reflections on how things went.
39