Enhanced Assessment for Enhanced
Achievement: Linking Assessment Tools
to First-Year Seminar Learning
Outcomes
25th Annual Conference on The First-Year Experience
February 27, 2006
Dr. Robert Krueger ([email protected])
Dr. Kristin Bransford ([email protected])
Concordia University, St. Paul
Today’s Objectives
The participant will:
 Become familiar with the process CSP used to
establish learning outcomes for FYS
 Review assessment tools used to collect data
for the FYS
 Develop a model to link learning outcomes
and assessment tools
What is FYS?



The First-Year Seminar (FYS) provides a
topics-based curriculum with common
learning outcomes.
The faculty of FYS advise all traditional
incoming first-year students.
The FYS faculty guide students through the
transition to major advisors (mentors) in the
spring semester of their first year at CSP.
Development of FYS




Mentoring Circles
Mentoring Pilot
Pre and post student surveys from Fall 2004
New Student Seminar
Used data to develop our First-Year Seminar
for Fall 2005
Goals for FYS



Advising
Building Community through Mentoring
Enhance Academic Skills
Structure of FYS






12 classes of 15 students each
1 credit (two class periods per week)
Faculty Instructors paid for extra workload
Topics-based course
Graded ABCDF
Peer Advisor
FYS Titles












American Idol or Idle American?
Authentic Happiness - Myth or Reality
Explore Your Individual and Social Identity within the University
The Flowering of the Mind
Humor in the Market Place of Ideas
I’ll Never Use This in the Real World . . .
Intersecting Voices: Culture at the Crossroads
Money: Could you be the next millionaire?
1968: There’s Never Been a Year Like It!
Peeking into Peking: The Dragon Awakes
Push Me / Pull You: Exploring Where to Get & How to Give Motivation
What’s it all about? College I mean!
June Conference



Started with the broad goals of FYS
Broke into two groups to think about learning
outcomes after a morning of discussion and
ideas from an outside organization
Came back together and we virtually had the
same four ideas
Learning Outcome - Explore
To explore the academic community and
develop new relationships with friends,
faculty, staff, and peers
Specified Achievements
1.
2.
To encourage intellectual pursuits within the
academic setting provided through the First-Year
Seminar course.
To become familiar with the Concordia community
including convocations, performing groups, and
other campus activities.
Learning Outcome - Experience
To experience the academic world, specifically
with regard to three categories
Specified Achievements
1.
2.
3.
To become familiar with Concordia’s campus
and surrounding community.
To pursue personal and intellectual inquiry.
To form new relationships with the instructor
and peer advisor.
Learning Outcome - Evaluate
To evaluate sources of information, seeking
answers to personal and intellectual
questions
Specified Achievements
1.
2.
3.
To effectively frame a research question and search
for information on a research topic.
To evaluate resources for appropriate supporting
material.
To develop the concept of critical thinking in
various disciplines.
Learning Outcome - Express
To express ideas by clearly articulating
thoughts in written and oral forms of
communication.
Rubric Design


A rubric was developed for each specified
achievement of the general learning outcome
and entered into a software program called
eLumen.
FYS faculty assess students on each of the
specified achievements and enter the results
into eLumen.
FYS Syllabi




There were 12 different topics-courses
offered.
Abstracts were developed by the twelve
individual instructors.
The learning outcomes provided the
framework for each course.
Students chose top five topics of interest.
Assessment Tools



Faculty assessing the students relative to the
four learning outcomes (eLumen).
Student Surveys: Perceptions about their
experience in the course, the campus
community, and confidence in academic skills.
Faculty Surveys: Perceptions about the FYS
process, their roles, and building student
connections and skills.
Power of Software Package



We can review a student’s individual strengths
and areas of needed improvement.
We have the opportunity to examine the entire
first-year class.
We can analyze a student’s progress over their
tenure at our institution.
Student Survey Examples


Pre and Post Surveys were given
Questions included:



FYS learning outcomes
Levels of confidence in academic skills
Satisfaction with the institution and course
Faculty Surveys

Surveys were distributed at mid-semester and
at the end of the term to evaluate:



Faculty confidence in their roles
Engagement in the program
Effectiveness of the program
Fall 2005 FYS Faculty
Engagement



Faculty Feedback: Is FYS a successful
addition to Concordia’s program for entering
students?
On a scale of 1 (Strongly Disagree) to 5
(Strongly Agree), FYS faculty reported 4.22.
Faculty Retention: eleven out of twelve
returning to program, one retiring.
Learning Outcomes
What are your FYS learning outcomes?
Learning Outcomes should be:
 Specific
 Measurable
If the general learning outcome is too
broad, develop specified achievements
related to the general outcome.
Specified Achievement for Explore
To encourage intellectual pursuits
within the academic setting provided
through the First-Year Seminar
course.
Rubric for Specified Achievement
0
1
No demonstrated achievement
Below Expectations: Missing four or more classes
or activities. Is seldom engaged during specific class
discussions and activities.
2 Meets Expectations: Missing fewer than four classes
or activities. Demonstrates adequate engagement
during specific class discussions and activities.
3 Exceeds Expectations: Missing fewer than two
classes or activities. Demonstrates full engagement
during specific class discussions and activities.
Problem



Combined two characteristics in one
specified achievement
What is most important? Engagement or
Attendance
Do your rubrics really match the
objectives of the learning outcomes?
Specified Achievement for Experience
To become familiar with Concordia's
campus and surrounding community.
Rubric for Specified Achievements
0
1
No demonstrated achievement
Below Expectations: Infrequently participates in
FYS activities including service-learning or other
community opportunities.
2 Meets Expectations: Participates regularly in FYS
activities including service-learning or other
community opportunities.
3 Exceeds Expectations: Takes a leadership role in
FYS activities including service-learning or other
community opportunities.
Problem




Measurable: How do we quantify leadership
role and infrequently or regularly participates?
FYS courses included a wide variety of
experiences that allowed for different
interpretation of this rubric.
To use the data to evaluate the program, we
need consistent measures.
Words were not quantifiable so could be
interrupted too broadly
Reflecting on your learning outcomes
1.
2.
Are your FYS learning outcomes specific
and measurable?
How is your institution measuring the
learning outcomes?
Mentoring


One of the primary goals in the creation of the
First-Year Seminar program was to build
mentoring relationships between faculty and
students.
We include this objective under the learning
outcome: Experience the Academic World
Learning Outcome: Experience the
Academic
World
Specified Achievement:
To form new relationships with the
instructor and peer advisor.
(Please ignore our mistake of including the instructor
and peer advisor in the same specified achievement.)
Rubric for Specified Achievement
0
1
2
3
No demonstrated achievement
Below Expectations: Makes and keeps few or no
appointments with instructors and peer advisor.
Meets Expectations: Makes and keeps most
appointments with instructors and peer advisor.
Exceeds Expectations: Makes and keeps all
appointments with instructors and peer advisor.
Faculty Impressions
To form new relationships with the
instructor and peer advisor.

Faculty eLumen Assessment:



Exceeds Expectation: 59.3%
Meets Expectation: 25.9%
Faculty survey: Reported 3.6 out of a 5 point scale
(5 = strongly agree) when asked if they felt
successful in their role as a mentor in the FYS.
Student Impressions


Student Survey Question: I am interested in
developing a mentoring relationship with
some of my professors.
Student Response:


Pre-test: 56% agree and 35% somewhat agree
Post-test: 53% agree and 38% somewhat agree
Student Impressions


Student Survey Question: First-Year Seminar
helped me to understand the value of
mentoring relationships with my professors.
Student Response on Post-test:


Agree: 26 %
Somewhat Agree: 38%
Student Impressions


Student Survey Question: I feel like I had a
mentoring relationship with a professor this
semester.
Student Response on Post-test


Agree: 40%
Somewhat Agree: 39%
Mentoring Question?




For this learning outcome, we want to
measure if we are “Building Community”
What did the student survey data tell us?
They want mentoring relationships.
How do we help them build these
relationships and how do we assess whether
we helped them in this process?
Experience the Academic World



This learning outcome is broad and hard to measure.
A possible achievement is student needs to take
responsibility for forming new relationships with the
instructor.
When developing specified achievements:
 What characteristics would you want students to
demonstrate?
 What activity would you assess?
Possible Revision - Rubric





No Achievement: Did not participate.
Beginning:
Developing:
Accomplished:
Exemplary:
Possible Revision - Activity


Activity: Student completes the FYS
Registration Worksheet prior to meeting with
the advisor.
Faculty will prepare worksheet at the spring
FYS Conference.
Possible Revision - Achievement


New Specified Achievement: Student
prepared for registration meeting with
advisor.
This new specified achievement



reinforces a consistent registration process for all
first-year students.
ensures a consistent measure.
models an important behavior for mentoring
relationships.
What to do with the results



Elimination of a learning outcome or
specified achievement
Reinforces to the faculty the importance of
assessment activities
A new outcome or achievement may be
required
Wrap-up comments


Our institutional goal of assessment: facilitate
a process in which faculty talk about what
their students should know.
Regardless of the mistakes that were made,
we have learned immensely about our course,
the students involved, and the faculty who
invested in it.