Providing Direct Evidence of
Student Learning in Student Affairs
Michael N. Christakis, Ph.D.
Assistant Vice President for Student Success
University at Albany
2012 Ohio Student Affairs Assessment Conference s The Ohio State University s July 20, 2012
(Appropriately…) Learning Outcomes
• By attending this presentation, participants will be
able to:
– Identify modes of building staff capacity (and gaining staff
buy-in) around a learning outcomes framework (and a
clearly articulated learning outcomes vernacular);
– Developing specific, measurable learning outcomes;
– Identify methods that provide direct measures of student
learning.
This Presentation Will…
What we are going to talk about…
• Very quickly, learning outcomes…what, why,
how.
• Our experience (at UAlbany) with learning
outcomes: the good, the bad, and everything
in between.
• Our framework – when it finally clicked for us.
• Our actual learning outcomes and some
examples of how we’re measuring them.
Learning Outcomes
(the quick and dirty version)
• Learning outcomes examine cognitive skills
that students (or other stakeholders) develop
through department interactions.
• Statements indicate what participants will
know, think or be able to do as a result of an
event, activity, program, etc.
• Need to be specific and measurable!
Learning Outcomes
(the quick and dirty version)
Attendance
• Measures attendance.
Needs
Assessment
• Identify needs.
Satisfaction
• Measures satisfaction.
Learning
Outcomes
• Identify and demonstrate educational attainment
and experiences, individual development, etc.
Learning Outcomes
(the quick and dirty version)
Creating specific
and
measurable learning outcomes!
Learning Outcomes
(the quick and dirty version)
Audience/Who
• Who does the outcome pertain to?
Behavior/What
• What do you expect the audience to know/be
able to do?
Condition/How
• Under what conditions or circumstances will the
learning occur?
UAlbany's "Student Learning Outcomes Areas"
(Summer 2009)
Students will be able to…
1) Make healthy life choices that reflect a balance of intellectual,
emotional physical, professional, social, and spiritual well-being.
2) Respect the diversity of our community as well as the individuality
of each member.
3) Engage in the life of the university and other communities as
participants and leaders.
4) Develop goals and identify resources needed for their attainment.
5) Communicate effectively, appropriately address and resolve
challenges, and manage conflict.
"Student Learning Outcomes Areas"
Summer
2009
Summer
2010
January 2011
•We asked…
•We asked…again…
•We stopped asking…
"Student Learning Outcomes Areas"
• The challenges:
– Lack of support;
– Lack of direction;
– Lack of resources (education);
– Lack of understanding.
• The opportunities:
– Evidence of learning outside
the classroom;
– Moving beyond satisfaction.
"Student Learning Outcomes Areas"
• The solution (January 2011):
– Three two-hour “workshops”
• Learning Outcomes Overview
• Measuring Learning Outcomes
• Final outcomes and methods
– Well-defined expectations
• Every unit participates
• At least two learning outcomes each year
"Student Learning Outcomes Areas"
Learning Outcomes “Boot Camp”?
• Session #1: Writing Learning
Outcomes.
– Workshop-style format where
outcomes are actually written
during the initial two-hour session.
– Outcomes are submitted (to Vice
President’s Office) for
comment/revision.
– “Final” outcomes are due in time
for session #2.
Learning Outcomes “Boot Camp”?
• Session #2: Measuring Learning
Outcomes.
– Workshop-style format where
(direct) measurement methods are
identified and instruments are
drafted during the two-hour session.
– Assessment method/instrument is
submitted (to Vice President’s Office)
for comment/revision.
– “Final” instrument is due in time for
session #3.
Direct vs. Indirect Methods
• Direct Methods - Any process employed to
gather data which requires students to display
their knowledge, behavior, or thought
processes.
• Indirect Methods - Any process employed to
gather data which asks students to reflect
upon their knowledge, behaviors, or thought
processes.
Direct vs. Indirect Methods
INDIRECT
Please rate your level of agreement
with the following….
• I know of resources on campus to
consult if I have questions about
which courses to register for in the
fall.
a)
b)
c)
d)
Strongly agree
Moderately agree
Moderately disagree
Strongly disagree
DIRECT
• Where on campus would you
go or who would you consult
with if you had questions
about which courses to
register for the fall?
– Open text field
Learning Outcomes “Boot Camp”?
• Session #3: Bringing it together!
– Show and tell with colleagues
(Assessment Council).
– Outcomes related to Division-level
framework.
– Final learning outcomes submitted
to Vice President’s Office and
proposed assessment included in
unit’s assessment plan.
Yay! Learning Outcomes!
Not so fast…
How do they relate?
To what do they relate?
Why relate?
Our New Dilemma
• Sought to purposefully identify learning experiences that UAlbany students
engage in through their daily interactions with programs, activities and
services throughout Student Success.
Spring 2009
Summer
2009
• Adopted five broad “learning outcome statements” and urged units across
the Division to adopt these statements or develop unit-specific statements
that remain consistent with the spirit of the five Divisional statements.
• Learning Outcomes “Boot Camp.”
Winter 2011
Our New Dilemma
Spring 2011
Summer
2011
Fall 2011
• Limited progress was made towards realizing learning
as illustrated in the five learning outcome statements.
• Major revision to UAlbany’s learning outcomes
framework.
• Introduced the “Student Learning Project” Divisionwide.
Learning Outcomes Framework:
Identifying, Identifying, Identifying...
• Identifying learning domains (CAS, Learning
Reconsidered, etc.).
• Identifying (specific) learning dimensions.
• Identifying (even more specific) learning
outcomes: the Student Learning Project.
Campus Models
UAlbany's (NEW!) "Framework"
• New Student Learning Outcomes
Framework – composed of three learning
domains – articulates the types of learning
experiences we value most in Student
Success.
• Student Success’ Student Learning Project
aims at developing unit level learning
outcomes in each of the Division’s three
learning domains each academic year,
beginning in 2011 and concluding in 2014.
• Learning outcomes should be both
attainable and measurable.
UAlbany's (NEW!) "Framework"
Division of Student Success
Unit/Program-Level
UAlbany's (NEW!) "Framework"
• Learning Domain – “Learning domains” are broad titles which characterize
experiential learning experiences students will have outside of the classroom
while enrolled at UAlbany. Domains can consist of one or more learning
dimensions.
• Learning Dimensions – “Learning dimensions” identify relevant and desirable
student learning and development outcomes as prescribed by the CAS
Frameworks for Assessing Learning and Development Outcomes (FALDOS).
• Learning Outcomes – “Learning outcomes,” developed at the unit-level, are
program-specific and encourage the achievement of one or more learning
dimensions. Learning outcomes must provide evidence, through direct
measurement, of student learning. Learning outcomes are identified, realized
and measured as part of the Division’s Student Learning Project.
UAlbany's (NEW!) "Framework"
• Balanced Life Choices – Students will achieve
balance between educational, personal, physical,
professional, interpersonal and spiritual pursuits in
order to become well rounded individuals.
• Cultural Awareness – Students will demonstrate an
awareness of and have an appreciation for the
differences and similarities between cultures.
• University Citizenship – Students will be informed
and engaged University citizens, while contributing
one’s talents to make a difference on campus and in
the surrounding community.
UAlbany's (NEW!) "Framework"
• The “Student Learning Project” attempts to:
– Identify particular learning experiences;
– Operationalize those experiences through an
assessment instrument which provides a direct
measure of student learning;
– Articulate to key stakeholders (e.g. students,
faculty, etc.) the efficacy of students learning
experiences outside of the classroom.
UAlbany's (NEW!) "Framework"
• “The Student Learning Project is a three year
long enterprise that focuses our Divisional
assessment efforts on student learning…”
• “What we do not have…is good evidence –
through direct measures – of students’
learning as a result of…our services…programs
and activities.”
• Student Learning Project (Pocket Guide), 2011
UAlbany's (NEW!) "Framework"
Division of Student Success
Unit/Program-Level
The Student Learning Project at Work
• Multicultural Student
Success
• Orientation
• Explore UAlbany
• Other Examples
The Student Learning Project
The Learning Outcome:
Multicultural Student Success
Learning
Outcome
Intervention
• After attending the Diversity and Inclusion
presentation at Freshman Orientation,
freshman will be able to express the
differences between diversity and inclusion.
• Diversity and Inclusion presentation at
Freshman Orientation.
The Assessment Method :
Multicultural Student Success
• “Quiz-like” question included on Orientation
evaluation at the conclusion of two-day
orientation session.
– “The presence of a variety of difference is defined as
_____________________; the act of allowing or
welcoming all to participate is defined as
_____________________.”
a)
b)
c)
d)
diversity; inclusion
inclusion; diversity
cultural competency; kindness
respect; communication
The Finding(s) :
Multicultural Student Success
• 96% of freshmen were able to correctly
identify the terms “diversity” and “inclusion”
after attending the diversity and inclusion
presentation at summer Orientation.
• 603 out of 630 correctly identified the terms
“diversity" and “inclusion.”
The Learning Outcome:
Orientation Assistant Training
Learning
Outcome
Intervention
• As a result of attending the University
Auxiliary Services Presentation during the
three week Leadership Training Program,
student staff will be able to accurately identify
and summarize meal plan information.
• Presentation by UAS staff to Orientation
Assistants during training.
The Assessment Method :
Orientation Assistant Training
• Pre/Post Test administered to Orientation
Assistants before and after the meal plan
presentation.
– “Please list the meal plans available to students
and briefly describe the main elements of each.”
• Administered BEFORE training.
• Administered AFTER training.
• Results were compared.
The Finding(s):
Orientation Assistant Training
• 72% of Orientation Assistants were able to
accurately explain Meal Plan Options as a
result of attending UAS training.
• 60% increase in the number of Orientation
Assistants who were able to accurately explain
meal plan options following training
(33.3% before; 72.2% after).
The Learning Outcome:
Explore UAlbany
Learning
Outcome
• By attending Explore UAlbany,
students will be able to describe
the importance of connecting with
faculty.
Intervention
• Explore UAlbany during extended
fall orientation (“welcome week”).
The Assessment Method :
Explore UAlbany
• Open-ended question asking students to
articulate the importance of connecting with
faculty.
– “Please describe one way in which you can
connect with Faculty on campus.”
• Open-ended response.
The Finding(s) :
Explore UAlbany
• 87% of students were able to identify one way
to connect with faculty by attending Explore
UAlbany.
• 137 out of 158 were able to identify one way
to connect with faculty, including (top three):
– Office Hours
– Food for Thought Lunch Program
– Participate and ask questions in class
Other Examples
UNIT
LEARNING OUTCOME
DIRECT
MEASUR
E
Conflict Resolution
& Civic
Responsibility
Students who apply to be released from Disciplinary Rubric
Probation will be able to explain how their
experience with the conduct system has impacted
their future goals.
Disability Resource
Center
Students with Disabilities, who utilize individual
appointments, will be able to make a semester
calendar listing all their academic obligations and
use it to better plan their time and meet deadlines.
Personal Safety &
Off-Campus Affairs
As a result of reading e-mail messages, publications Survey
delivered to their apartment, and information on
(open
the “Living Off Campus website, students will be
ended)
able to identify two personal safety tips for being an
off-campus student.
Rubric
Other Examples
UNIT
LEARNING OUTCOME
DIRECT
MEASURE
Residential Life
As a result of participating in one-on-one staff
meetings, student staff members will be able
to articulate their strengths.
Survey
(open
ended)
Student Involvement &
Leadership
After attending the “Introduction to
Leadership” session, students will be able to
successfully identify 3 or more leadership
theories.
Survey
(quiz)
University Health Center Students who visit the Health Center for an
upper respiratory illness will be able to
identify symptoms to consider in determining
if they can self-manage their illness or if it is
appropriate to contact a provider for care.
Survey
(quiz)
Next Steps
• Replicated the January 2011 exercise in
January 2012 (slightly abridged) and again in
January 2013.
• Two new learning outcomes developed for
each unit.
– 28 learning outcomes in 2011
– 28 learning outcomes in 2012
– 28 learning outcomes in 2013 (anticipated)
Next Steps
• 84 specific, measurable learning outcomes!
• Evidence of “learning.”
Next Steps
• Increasing use of “direct measures” of student
learning across the Division.
• Deeper examination of aspects to students
learning/experience while at UAlbany.
– Example: Identifying how to succeed academically
and actually doing those things.
• Provide a Phase II Findings Report by June 2013.
Lessons Learned
• Never underestimate
leadership.
• Strive to build capacity
sooner versus later.
• Be flexible!
• Good things take time.
• If at first you don't
succeed...
Lessons Learned
• It's not always about the
outcome, sometimes it's
about the effort!
• It's usually about the
outcome, but always
about the measure!
• Don't reinvent the wheel;
there is lots of great
information and valuable
models out there already.
Contact
Michael N. Christakis, Ph.D.
Assistant Vice President for Student Success
University at Albany, SUNY
1400 Washington Avenue | Albany, NY 12222
email: [email protected]
phone: 518.956.8140
| twitter: @MikeChristakis
web: www.albany.edu/studentsuccess/assessment