Direct Assessment Techniques
of SLO in Student Services
0
MICHAEL C. SACHS, JD, CCEP
ASSISTANT VICE PRESIDENT FOR
STUDENT AFFAIRS
EAST STROUDSBURG UNIVERSITY
PRESENTED AT THE MSCHE
2014 CONFERENCE
© Michael C. Sachs 2014
Program Learning Outcomes
1
 To be able to recognize direct vs. indirect
 To be able to identify types of direct assessment
 Provide examples that can be implemented on
campus
© Michael C. Sachs 2014
MSCHE
2
The characteristics of good evidence of student
learning include considerations of direct and indirect
methods for gathering evidence of student learning.
(Student Learning Assessment: Options and Resources, Second
Edition, 2007)
© Michael C. Sachs 2014
Direct Assessment
3
In direct assessment students display knowledge or
skills as the result of an assessment measure
(presentation, test, etc). Direct measures of student
learning require students to display their knowledge
and skills as they respond to the instrument itself.
Objective tests, essays, presentations, and classroom
assignments all meet this criterion.
(Palomba, C.A., & Banta, T.W., 1999)
© Michael C. Sachs 2014
Indirect Assessment
4
Indirect assessment learning is inferred instead of
being supported by direct evidence (i.e., usage data,
satisfaction surveys). Students reflect on learning
rather than demonstrate it.
(Palomba, C.A., & Banta, T.W., 1999)
© Michael C. Sachs 2014
Indirect Assessment in Student Services
5
 Student Services has often relied on indirect assessment
for reporting success such as:







Satisfaction surveys (NSSE, CCSSE, etc.)
Evaluations
Attendance rates
Usage data
Quality measures
Focus groups
Indirect assessment is very useful when determining if
a student likes or enjoys an event, activity, or program,
but not in conveying if they learned something
© Michael C. Sachs 2014
Example: SLO Direct / Indirect
6
 Which is direct / indirect?
1.
Attendance
(Indirect)
1.
Build Community
(Direct or Indirect)
1.
Have students write three things they learned from the
program
(Direct)
You must have predetermined goals

How do we know they learned something and that
they learned what we want them to learn?
© Michael C. Sachs 2014
Alcohol Education Program
7
1 Shot = 1 Glass of Wine = 1 Beer
 SLO Goal: Consuming several shots can be far
more intoxicating due to the alcohol content than
having the same quantity of beer or wine.
© Michael C. Sachs 2014
Alcohol Education
8
 Indirect:
 Did you learn something from this presentation?
 Direct:
 What did you learn from the presentation?
List three items (+/- with predetermined answers)
 Fill in the blank
 Create a tool to measure the learning for small group discussions
(have the recorder turn in their notes)

© Michael C. Sachs 2014
Unwanted Outcome
9
 Direct Assessment Outcome:
“Cool, I can get drunk faster and gain less weight
by doing shots than drinking beer.”
 Indirect assessment would not likely have brought
this answer forward
© Michael C. Sachs 2014
Qualitative & Quantitative vs. Direct &
Indirect
10
 Don’t confuse qualitative & quantitative with direct
& indirect
© Michael C. Sachs 2014
Qualitative & Quantitative vs. Direct & Indirect
11
 Quantitative
Quantitative assessment method uses structured,
predetermined response options that can be
summarized into meaningful numbers and analyzed
statistically.
(Suskie, L., 2004)
© Michael C. Sachs 2014
Qualitative & Quantitative vs. Direct & Indirect
12
 Qualitative:
Qualitative assessment methodology involves asking
participants broad, general questions, collecting the
detailed views of the participants in the form of words or
images, and analyzing the information for descriptions
and themes
(Creswell, J., 2007)
Data that does not lend itself to quantitative methods but
rather to interpretive criteria.
(Carnegie Mellon Eberly Center Teaching Excellence & Educational
Innovation)
© Michael C. Sachs 2014
Sample Direct Assessment Techniques
 Pre/Post test
13
 Portfolios
 Direct Observation
 Projects
 Video Observation
 Capstones
 Completion Accuracy
 Goal Completion
 Reflection Papers
 Training Others
 Performance
 Essays
Observation
 Demonstrations
 Interviews (not focus
groups)
 Competitions
© Michael C. Sachs 2014
 Quick Checks during
program
 Work Groups/ Table Top
Exercises
 Juried Evaluators
Key to Direct Assessment
14
 A direct assessment technique is only as good as the
tool used to measure it:

A post test with no predetermined acceptable answers
(remember the alcohol example?)
 An indirect assessment technique could have direct
assessment embedded into it:


Evaluation with questions about student learning embedded
into the evaluation
A focus group with specific questions asked to individual
participants about what they learned
© Michael C. Sachs 2014
Rubrics
15
 Before we get into the examples, a few words on
rubrics

It is essential to create a tool such as a rubric that will measure
the anticipated learning goals and quality of the answers
© Michael C. Sachs 2014
What is a Rubric?
16
 A set of criteria specifying the characteristics of an
outcome and the levels of achievement for each
characteristic
(J. Levy, 2012)
 A rubric is commonly defined as a tool that
articulates the expectations for an assignment by
listing criteria, and for each criteria, describing levels
of quality
(Andrade, 2000; Arter & Chappuis, 2007; Stiggins, 2001)
© Michael C. Sachs 2014
Rubrics Continued
17
 Rubrics contain three essential features:
1.
Criteria students are to attend to in completing the
assignment
2.
Markers of quality (typically rating scales)
3.
Scoring
(University of California, Berkley Center for Teaching and
Learning)
© Michael C. Sachs 2014
Beginning 1
(Lower 50%)
Developing 2
(50%-79%)
Accomplished 3
(80%-89%)
Outstanding 4
(Top 10%)
Quality &
Organization
Engaging
Presentation
Skills
Resource
Knowledge
Sample Rubric – Admissions Tour
Guides - Direct Observation
© Michael C. Sachs 2014
18
Now For Examples!
19
 The following goals are examples of direct
assessment processes in various areas of Student
Services
 Caveats
It is essential to create a tool that measures your SLO
goals
 Not all examples will be feasible on all campuses
 This presentation does not take into account campus
resources (both human and financial)
 Each institution will need to determine their specific SLO
outcomes

© Michael C. Sachs 2014
Residence Life: RA Training
20
 Program / SLO:
 Behind Closed Doors – Role Play Training

Students will be able to apply training to real life scenarios
 Tool:
 Direct observation by professional staff
© Michael C. Sachs 2014
© Michael C. Sachs 2014
21
Campus Police
22
 Program / SLO:
 Campus Police Active Shooter Response
 Tool:
 Table Top Exercise
 Program / SLO:
 Arrest / Stop Interview with Student
 Tool:
 Post Interview with Conduct Officer or Police
© Michael C. Sachs 2014
Yes/No + Notes
Lock & Barricade Doors
Turn Off Lights
Close Blinds
Turn Off Radios & Computer
Monitors
Keep Occupants Calm, Quiet and Out
of Sight
Take Adequate Cover
Silence all Cell Phones
Place Signs in Exterior Windows for
Injured Persons
Sample Rubric – Campus Police
© Michael C. Sachs 2014
23
Conduct Office
24
 Program / SLO:
 Reflection Paper with Topical Goals
 Tool:
 Rubric with Stated Goals for paper content
© Michael C. Sachs 2014
Beginning
Intermediate
Exemplary
Self Disclosure
Paper Connected to
Conduct Violation
Understand
Connection to
Community
Standards
Sample Rubric – Student Conduct:
Reflection Paper
© Michael C. Sachs 2014
25
Food Services
26
 Program / SLO:
 Students will better understand food quality, preparation,
waste management, etc.
 Tool - Post Test:
 With Specific Answers or fill in the blank
© Michael C. Sachs 2014
Enter an ‘X’ in the correct box for your answer.
What items are not
baked on premises
or are brought in
from local bakeries?
Dinner Rolls
Wraps
Hamburger/
Hotdog Buns
Hoagie/ Sub
Rolls
Bagels
Donuts
Enter an ‘X’ in the correct box for your answer.
What percentage of
meat on average is
delivered frozen?
++
5%
20%
60%
80%
Enter an ‘X’ in the correct box for your answer.
Most
vegetables/fruit are
freshly prepared,
what are the two
exceptions?
Peas
Carrots
Beets
Green Beans
Cabbage
Cling
Peaches
Sample Rubric 27– Food Services
© Michael C. Sachs 2014
Sandwich
Bread
Theater Club
28
 Program / SLO:
 Students will produce an entertaining and semi-professional
theatrical production
 Tool - Video or Direct Observation
 Of the Performer or the Audience
© Michael C. Sachs 2014
Criteria
4
3
2
1
VOICE
Voice was loud and
clear; words were
easily understood
Student spoke
clearly but it was
difficult to
understand some
of the script;
could’ve been
louder.
Voice and
language was not
very clear;
could’ve been
much louder.
Could not
understand what
was being said due
to unclear and low
speech.
Needed more
audience
awareness and
connection.
No audience
awareness or
connection at all.
AUDIENCE
MEMORIZATIO
N/
IMPROVISATIO
N
(When
applicable)
OVERALL
Audience felt like
part of the show.
Was aware and
well-connected to
the audience.
Script was fully
memorized; student
improvised in place
of lines.
Script was almost
fully memorizedsome improv used
to make up for
missed lines.
Script was
partially
memorized;
student did not
attempt
improvisation.
Script was not at all
memorized; no
improvisation used.
Committed,
cooperated &
concentratedWOW!
Semicommitted,
concentrated &
cooperativeGREAT!
Almost
committed,
cooperative &
concentratedNOT TOO
BAD…
No commitment,
cooperation or
concentration
MORE
REHEARSAL!
Theatre Club
© Michael C. Sachs 2014
29
TOTAL
Career Development
30
 SLO:
 Students who attend the resume workshop will be able to
create a quality basic resume
 Scoring rubric with criteria
© Michael C. Sachs 2014
Skill
Outstanding
Good
Average
Unsatisfactory
PRESENTATION
/
FORMAT
§ Typed or computer
generated
§ Balanced margins
with eye appeal
§ Format highlights
strengths and
information
§ Appropriate fonts and
point size used with
variety
§ Typed or computer
generated
§ Balanced margins
§ Format identifies
strengths and
information
§ Appropriate fonts and
point size used
§ Typed or computer
generated
§ Somewhat
balanced margins
§ Format identifies
strengths and
information
§ No variation in fonts
and/or point size
§ Typed or computer
generated
§ Unbalanced
margins
§ Format detracts
from strengths and
information
§ Fonts distract from
readability
10
8
7
6
§ All action phrases
used to describe
duties and skills
§ Information
demonstrates ability
to perform the job
§ Professional
terminology used
when describing skills
§ 1-2 duties/skills lack
action phrases
§ Information
demonstrates ability
to perform the job
§ Some professional
terminology used
when describing skills
§ 3-4 duties/skills
lack action phrases
§ Some information
demonstrates ability
to perform the job
15
12
11
10
§ 3-4 spelling errors
§ 3-4 grammar errors
§ 5-6 spelling errors
§ 5-6 grammar errors
6
4
Ranking Points
JOB-SPECIFIC /
VOLUNTEER
INFORMATION
Ranking Points
SPELLING &
GRAMMAR
§ No spelling errors
§ No grammar errors
Ranking Points
10
§ 1-2 spelling errors
§ 1-2 grammar errors
8
Total
§ 5-6 duties/skills
lack action phrases
§ Information does
not clearly
demonstrate ability
to perform the job
TOTAL
SCORE:
Sample Rubric – Career Development
© Michael C. Sachs 2014
31
Student Government
 Program / SLO
 Student Government Officers will learn leadership Skills
Portfolio
of their year of work
Programs & Events: Successes and Challenges
Reflection Paper
Train the Trainer
Public Speaking
Registrar
33
 Program / SLO:
 Workshop on completing graduation application: students
who attend registration workshop will do better than those
who did not attend
 Tool
 Comparison of attendee results vs. non attendees
© Michael C. Sachs 2014
Athletics
34
 Program/SLO:
 Athletic Team Sport: Sportsmanship
© Michael C. Sachs 2014
Conduct
Criteria Scale
2
4
3
Ideal
Acceptable
Tolerable
1
0
Unacceptable
Absent
Behavior towards
•
Officials
•
Opponents
•
Host school
•
Host students
Play / Participation
•
Rules
•
Spirit of event
•
On the field
Respectful
Polite
Gracious
Positive interaction
Consistently
neither rude nor
polite.
Lacking politeness
Attitude is not
respectful.
Tantrums
Disrespectful
Fighting
Swearing
Unacceptable
behavior in all
possible areas.
Honorable
Playing under
control.
Fully engaged in
respectful play.
Solid good play.
Abides by the
rules.
Play that follows
rule but selfish or
lacking true spirit.
Unable to follow
rules.
Unwillingness to
grow as athletes
Team Work
•
Unity
•
Organization
•
Cooperation
Cooperative.
United.
Respectful to
team- mates.
Good leadership.
Working together
but some
problems with
communication.
Sometimes
working together
or disagreeing
with own team.
Cheating
Roughness
Out of control.
Inappropriate
taking
advantage.
Disjointed play.
Inter-fighting
Disorganized
Lacking
leadership.
No teamwork
whatsoever
displayed.
Average Score
Sample Rubric – Athletics
© Michael C. Sachs 2014
35
Make it Fun
36
 Discrimination workshop / program Pre-Post Test
Cross Word Puzzle determining knowledge after
presentation
© Michael C. Sachs 2014
To be added leave blank
37
 Slide to be added
© Michael C. Sachs 2014
Integration into Assessment Plans - ESU
38
 Decide who will participate
 Get Buy in
 Set your SLO goals for Direct Assessment
 ESU 50% or more must be direct assessment of SLO in
Student Services reported to Univ. Assessment Committee
(UAC)
 Small programs 1-2 per year, large 5+ per year
 Individual create their own assessment plans and criteria
 Operational Assessment is separate
 Yearly mapping submitted to UAC departmental
Assessment plan reviewed by SS Assessment
Committee
© Michael C. Sachs 2014
39
© Michael C. Sachs 2014
Questions?
Michael C. Sachs, JD, CCEP
40
ASSISTANT VICE PRESIDENT FOR STUDENT
AFFAIRS
EAST STROUDSBURG UNIVERSITY OF
PENNSYLVANIA
[email protected]
© Michael C. Sachs 2014