Outcomes Assessment
Using Capstone Experiences
Joseph A. Shaeiwitz
Chemical Engineering Department
West Virginia University
P.O. Box 6102
Morgantown, WV 26506-6102
304-293-2111 ext. 2410
[email protected]
http://www.che.cemr.wvu.edu/ugrad/outcomes/
Assessment Survey
 Please
complete the preworkshop assessment survey
Outline
 Introduction/Motivation
 How
to include as many
outcomes as possible in a
capstone experience
 Desired attributes of capstone
experiences
Outline (cont’d)
 Development
of rubrics to assess
capstone experiences
 Simultaneous assessment and
grading
 Using oral presentations for
assessment of capstone
experiences
 Summary
Outline
 Introduction/Motivation
 How
to include as many
outcomes as possible in a
capstone experience
 Desired attributes of capstone
experiences
Motivation
Self-assessment measures such as
surveys are insufficient, assessment
by the faculty is needed
 All engineering programs already have
capstone experiences
 Students are expected to apply
previously learned knowledge in
capstone experiences

education process
Feedback Model
one
class
one course
entering
college
one
class
one course
graduate
alumnus
Outline
 Introduction/Motivation
 How
to include as many
outcomes as possible in a
capstone experience
 Desired attributes of capstone
experiences
Required Outcomes
Apply math, sci,  Communication
engr
 Broad education
 Experiments
 Life-long learning
 Design
 Contemp. Issues
 Teams
 Modern tools
 Solve engr problems
 Ethics

Exercise 1
 Suggest
a capstone experience
that includes as many required
outcomes as is possible
 Focus on what needs to be added
to commonly used experiences to
include additional outcomes
Some Ideas





Comprehensive individual/group design
project
Includes all technical aspects
Include material not taught in class
Can include safety, life-cycle analysis,
pollution prevention, environmental
impact statement
Requires use of specialized, sophisticated
software
Outline
 Introduction/Motivation
 How
to include as many
outcomes as possible in a
capstone experience
 Desired attributes of capstone
experiences
Attributes of Capstone
Experiences
 What
attributes do you expect
to see in your capstone
experiences?
 What skills and/or knowledge
do you expect to be
demonstrated in capstone
experiences?
Exercise 2
 List
as many attributes as
possible for the capstone
experience described in Exercise 1
 Focus on what you would be
evaluating when grading the final
project report
Some Ideas
Writing – grammar, punctuation,
format, good technical explanations,
logical development
 Oral – use of visual aids,
organization, delivery mechanics
 Technical – apply math, science,
engineering; design of equipment;
optimization based on economics

Outline
 Development
of rubrics to assess
capstone experiences
 Simultaneous assessment and
grading
 Using oral presentations for
assessment of capstone
experiences
 Summary
Development of Rubrics
 What
attributes do you expect
to see in your capstone
experiences?
 What skills and/or knowledge
do you expect to be
demonstrated in capstone
experiences?
Development of Rubrics
(cont’d)
 How
would you describe
excellence for each attribute?
 What is not acceptable for
each attribute?
 Fill in intermediate
descriptions
Four-Point Scale for
Rubrics
4
– exceeds expectations
3
– meets expectations
2
– below expectations
1
– not acceptable
Use of Rubrics
 Oral
Reports
 Written
Reports
 Technical
Content
Advantages of Method
 Can
involve many faculty
members
 Standardization of evaluation
 Is done while grading
 Can be adapted to guideline
for grading
Exercise 3
 For
the attributes listed in
Exercise 2, develop a rubric for
evaluation of these attributes
Attribute
Report Mechanics
Organization
1-Not acceptable
inappropriate content of some content in
most sections of report inappropriate
section of report
Complete Story Told
no story told, very
incomplete
Aesthetics
unacceptable – e.g.,
tables and figures
cannot be
read/understood, fonts
difficult to read
so many format errors
as to make report
useless
Format
2-Below
expectations
Spelling
any spelling errors
Grammar and Punctuation
too many grammar and
punctuation errors
Length
far too long or too
short
3-Meets
expectations
4-Exceeds
expectations
content appropriate to unique organization
all section of report
enhances readability
and/or
understandability of
report
aspects of complete complete story told
additional material
story missing
enhances quality of
report
some portions are
text, tables, figures
text, tables, figures
sloppy and difficulty readable and
so clear and
to read
understandable
understandable as
to enhance report
impact
some format errors format followed
unique format
aspects that
enhance report
impact
only spelling errors no spelling errors
are different
spellings for same
pronunciation
grammar and
only a very few
no grammar or
punctuation errors
grammar or
punctuation errors
punctuation errors
too long or too
appropriate length
short
Score
Attribute
1-Not acceptable
2-Below
expectations
3-Meets
expectations
4-Exceeds
expectations
Effective use of Visual Aids
(VA)
Clarity and readability
not clear or readable
difficulty reading
clear and readable
Use of space on VA
VA unreadable because
too crowded
appropriate amount
of information on VA
Lettering readable
Color, over- or under-use (if
used)
font unreadable
colors too hard to
distinguish, colors do
not project well
slides full of text
so much information
per VA or so much
missing information to
make VA useless
too little or too
much information of
VA
font too small
poor choice and use
of colors
superior clarity and
readability
VAs very well laid
out
Wording concise
Appropriate amount of
information per VA
Presentation Organization
Logical order of topics
Appropriate use of time: Not
too long /short
Complete "story" told
Introduction: Problem stated
font readable
primary/easily
distinguishable colors
use of color
enhances clarity of
presentation
slides too wordy
slides appropriate
too much
appropriate level of
information per VA, information per slide
missing information
such as size of total
pie
totally disjointed, no
organization
some items
presented out of
order
organization as per
guidelines
far too long or far too
short
story missing, no story
told
problem not stated
somewhat too long
or too short
story incomplete
appropriate length
problem poorly
stated
problem clearly stated
superior
organization
enhances
communication
complete story told
problem clearly
stated, good
perspective on
problem shown
Score
Attribute
1-Not acceptable
2-Below
expectations
3-Meets
expectations
4-Exceeds
expectations
Design of equipment,
Analysis of performance of
existing equipment,
Understand
interrelationship between
equipment in process
Design of individual
equipment
major errors in individual
equipment design
some errors in
equipment design
equipment designed
correctly
Understand interrelationship
between equipment on
flowsheet
no understanding of
equipment
interrelationship
clear understanding
of equipment
interrelationship
Constraints/limitations of
individual equipment and
flowsheet understood
constraints/
limitations not understood
minimum
understanding of
equipment
interrelationship
not all constraints/
limitations understood
Response to questions
indicates understanding of
ChE principles
Significance of conclusions
understood
Apply chemistry, math,
physics, engineering
science
Apply engineering science
response to questions
demonstrates lack of
understanding
lack of understanding
response to questions
shows gaps in
understanding
gaps in understanding
response to
questions shows
clear understanding
clear understanding
unique aspects of
equipment design
enhance result
exploitation of
equipment
interrelationship to
enhance result
exploitation of
constraints/
limitations to enhance
result
response to questions
shows superior
understanding
superior
understanding
Response to questions
indicates ability to apply
these principles
inability to apply principles weak application of
principles
response to questions
response to questions
demonstrates lack of
shows gaps in ability
ability to apply these
to apply these
principles
principles
constraints/
limitations clearly
understood
good application of
principles
response to
questions shows
clear ability to apply
these principles
superior application
of principles
response to questions
shows superior ability
to apply these
principles
Score
Scoring Sheet for all Design Projects
Technical Content
Design of equipment, analysis of performance of existing
equipment, understand interrelationship between equipment in
process
Apply chemistry, math, physics, engineering science
Resolve complex problem into components
Apply economic, physical constraints and optimization methods to
obtain solution
Computer usage
Application of safety principles
Other economic, global, societal, and legal considerations
Oral Presentation
Effective use of visual aids
Presentation organization
Effective use of software to prepare visual aids
Group synergism/dynamics
Presentation mechanics
Response to questions
Written Report
Report mechanics
Content
Effective use of software to prepare written report
Outline
 Development
of rubrics to assess
capstone experiences
 Simultaneous assessment and
grading
 Using oral presentations for
assessment of capstone
experiences
 Summary
Primary Trait Analysis
Rubric-like grading guide
 Converts “unstated criteria” (e.g., “it
feels like a B,” to “highly explicit
criteria.”
 Converts from norm-referenced to
criterion-referenced grading
 B. E. Walvoord and V. J. Anderson,
Effective Grading. A Tool for Learning
and Assessment, Jossey-Bass, 1998.

Our Experiences
Five-point scale is difficult to develop
 We use four-point scale shown earlier
 Approximate grades

• A = several 4, all others 3
• B = all 3
• C = average about 2.5 or so
• D = average around 2
• F = average well below 2
Large-Group Design Project
Students work under direction of
student chief engineer
 Faculty members play roles – one is
client, other is student company vice
president
 Evaluations like in company
 Rubric developed for grading

Participation
(35%)
F
D
 No attendance  Missed 5 or
more group
 Did not
meetings
complete
assignment
 Turned in
and displayed
assignments
poor attitude
sporadically
 No
 Displayed a
participation
poor attitude
in group
during group
meetings
activities
Technical
Quality (40%)
F
D
C
B
A
 Missed 3 or
 Attended all
 Attended all group
more group
group meetings
meetings
meetings
 Completed all  Completes all
assignments
assignments with an
 Turned in late
assignments  Worked well
excellent attitude
with the group  Performed extra tasks
 Was not task
oriented
during meetings
that produced new
during group
insight within the
meetings
group
C
B
A
 No
 Poor
 Average
 Good
 Excellent
understanding
understanding
understanding
understanding
understanding &
& synthesis of
& synthesis of
& synthesis of
& synthesis of
synthesis of research
research
research
research
research
 Ideas exceeded
 The ideas
 The ideas
 The ideas
 The ideas were
expectations and were
were not
were not
were
expressed
expressed clearly
expressed
expressed
expressed
clearly
 No errors in design
clearly
clearly
clearly
calculations
 No errors in
 Major errors  Many minor  Few errors in
design
throughout
errors in
design
calculations
design
calculations
calculations
calculations
Communication
(25%)
F
D
C
B
A
Outline
 Development
of rubrics to assess
capstone experiences
 Simultaneous assessment and
grading
 Using oral presentations for
assessment of capstone
experiences
 Summary
Feedback from Q & A
Session
 Have
presentation followed by
question and answer session
 Students get immediate feedback
 Ask questions and follow-up
questions
 Look for patterns – common
errors or common good ideas
Types of Questions Asked
 Can
they explain what was done
and why?
 Quiz students on related or
peripheral concepts
 Error is observed – convert
session into tutorial
 How has student applied what
was learned?
Information Seeking
 Level
of understanding and ability
to communicate it
 Was “right” answer obtained for
wrong reason or accidentally?
 Is “big picture” understood?
 How was solution obtained?
Example – weak student
 Student
chooses reactor
temperature of 325°C
 Question: Why was this reactor
temperature chosen?
 Good: Because it was economic
optimum
 Bad: I just chose it
Example – better student
Student chooses reactor temperature
of 325°C and shows graph illustrating
that it is economic optimum
 Question: Why is this optimum? Why
are economics less favorable above
and below this temperature?
 Good: Provides explanation
 Bad: I just tried a range of
temperatures and this is what I found.

Example – very good student
Student chooses reactor temperature
of 325°C, shows graph illustrating that
it is economic optimum, and explains
why
 Question: Did you investigate other
variables such as pressure, conversion?
 Good: Yes, and explains result
 Bad: No, I just investigated
temperature.

Example – excellent student
 Student
chooses reactor
temperature of 325°C, shows
graph illustrating that it is
economic optimum, explains why,
and does same for other decision
variables
 Question: It becomes more
difficult to find good questions
Exercise 4
 Ask
me questions about this
presentation aimed at revealing
my understanding of assessment,
ABET EC 2000, rubric
development, etc.
Outline
 Development
of rubrics to assess
capstone experiences
 Simultaneous assessment and
grading
 Using oral presentations for
assessment of capstone
experiences
 Summary
Summary
 Assessment
using capstone
experiences can complement
surveys – assessment by the
faculty
 It is possible to include many
outcomes in capstone experiences
 Identify desired attributes of
capstone experiences
Summary (cont’d)
 Can
develop rubrics to assess
capstone experiences
 Can use primary trait analysis for
simultaneous assessment and
grading
 Can use oral presentations as part
of assessment of capstone
experiences
Web Site
http://www.che.cemr.wvu.edu/
ugrad/outcomes/
Assessment Survey
 Please
complete the postworkshop assessment survey