The GLOSSARI Project:
Assessing Learning Outcomes
of Study Abroad
Richard C. Sutton
University System of Georgia Board of Regents
U.S. Department of Education Title VI Centers Conference
Washington, DC
26 September 2006
Accountability in Higher Education
Evaluating resource allocation (e.g., cost/credit hr)
Evaluating instructional processes (e.g., technology
utilization)
Evaluating learning outcomes
Most appropriate from the teaching-and-learning
paradigm
Emphasis on value-added
Treats study abroad as integral to academic
affairs, not “enrichment”
Significance of Study Abroad in
Higher Education
Nearly 200,000 U.S.
students currently go
abroad each year
About 8% of all UG
degree recipients have
part of their education
abroad
Students can earn up to
one-fourth of their
academic degree from
overseas study
Study abroad
participation among all
U.S. students
increased 20% since
2001.
Georgia participation
will triple within 10
years (1997-2007)
New federal funding
initiatives to increase
study abroad
Growth of USG Study Abroad Participation
7000
6000
5000
4000
3000
2000
1000
0
1997- 1998- 1999- 2000- 2001- 2002- 2003- 2004- 2005- 200698
99
00
01
02
03
04
05
06
07
15% average annual growth rate
Assessment Efforts in Study Abroad
Strong research efforts to assess
second language acquisition learning
outcomes from study abroad
Some good studies on documenting
gains in intercultural sensitivity and
personal growth
Limited attention paid to knowledge
and skills acquired abroad
GLOSSARI
GEORGIA
LEARNING
OUTCOMES OF
STUDENTS
STUDYING
ABROAD
RESEARCH
INITIATIVE
Primary Objectives of GLOSSARI
• Identify cognitive learning outcomes attributable to
•
•
diverse study abroad experiences
for students at a wide variety of public institutions
• Identify impact on academic performance indicators
• Identify impact of study abroad on core liberal arts
aspirations (critical thinking, leadership, adaptability,
etc.)
• Identify program characteristics that optimize learning
outcomes, so as to guide program development
• Identify student characteristics that predict
•
•
likely participants
successful participants
• Refine, replicate, and disseminate methods for
assessing the impact of study abroad on student
learning outcomes.
New Federal Grant to Accelerate the
GLOSSARI Research Project

GLOSSARI project began in 2001 with modest
internal funding

U.S. Department of Education: International
Research & Studies Program Grant for 2006-2009

GLOSSARI research team headed by Prof. Don
Rubin (University of Georgia)

Six components examine discipline-specific and
cross-disciplinary learning outcomes
The Six Phases of GLOSSARI
Stage I: Learning Outcomes of SA Participants & NonParticipants
a) self-reported
b) skills and knowledge testing
Stage II: Pre- and Post-participation Learning Outcomes
a) with current survey
b) correlated with other testing instruments
Stage III: Teaching the same course content abroad & at
home
Stage IV: Academic performance measures among SA
participants and non-participants
Stage V: Program design features that make a difference
Stage VI: Impact 2- to 5-year post-graduation
What we’ve learned so far….
 Better navigational
skills
 Improved academic
performance upon
return
 Much higher
persistence and
graduation rates
Learning Outcomes Factor 1
Knowledge of Verbal Resources
Knows how to adjust to new situations
Knows when to take risks
Knows how to talk one’s way out of difficult
situations
Knows how to lead discussions
Comparison between study-abroad
participants and nonparticipants on
knowledge of verbal resources
(not statistically significant; p>.05)
degree of self reported
knowledge
4
3.5
3
3.99
3.91
2.5
2
1.5
1
0.5
0
particpants
nonparticipants
Learning Outcomes Factor 2
Sensitivity to Cultural Context
Knows how to interact in different culture
Knows the importance of withholding
judgment
Sensitive to language and culture
differences
Sensitive to one’s own reactions to others
Comparison between study-abroad
participants and nonparticipants on
sensitivity to cultural context
degree of self reported
knowledge
(p<.001)
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
4.38
4.05
participants
nonparticipants
Leaning Outcomes Factor 3
Knowledge of Self as Cultural Being
Understands how settings affect one’s own
style
Understands how one’s self is viewed by
others
Knows how to define the term “culture”
Comparison between study-abroad
participants and nonparticipants on
knowledge of the self as a cultural being
degree of self reported
knowledge
(p<.001)
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
4.17
participants
3.93
nonparticipants
Learning Outcomes Factor 4
Functional Knowledge of Cultural Practices
Knows how to compare and contrast culture
Knows norms and taboos
Knows how to locate information in a
newspaper
Knows how to buy toothpaste and can
opener
Knows how to give coherent directions
Learning Outcomes Factor 4
Functional Knowledge of Cultural Practices
(continued)
Knows different ways to express ideas
Knows what’s funny
Knows how to take a train or a bus in a
foreign country
Knows how to use a public telephone
Can locate safe clubs or bars
Knows how to pacify an angry person
Comparison between study-abroad
participants and nonparticipants on
functional knowledge of cultural practices
(p<.001)
degree of self reported
knowledge
4
3.5
3.83
3
2.86
2.5
2
1.5
1
0.5
0
participants
nonparticipants
Learning Outcomes Factor 5
Knowledge of World Geography
Can name six countries in Africa
Can name four South American capitals
Can name four rivers in Europe
Comparison between study-abroad
participants and nonparticipants on
knowledge on world geography
(p<.001)
degree of self reported
knowledge
4
3.5
3.69
3
3.26
2.5
2
1.5
1
0.5
0
participants
nonparticipants
Learning Outcomes Factor 6
Knowledge of Global Interdependence
Understands how freedoms in U.S. compare with
other nations
Can explain presence of U.S.troops abroad
Can explain aspects of U.S. foreign policy
Understands how foreign markets might affect
one’s own career
Understands how foreign manufacturing affects
pricing in U.S.
Comparison between study-abroad participants and
nonparticipants on knowledge of global
interdependence (p<.005)
degree of self reported
knowledge
4
3.5
3
2.5
3.7
3.46
2
1.5
1
0.5
0
particpants
nonparticipants
Correlations Among Factors
Factor 1: Verbal adeptness
Factor 2: Sensitivity to cultural context
Factor 3:Knowledge of self as cultural being
Factor 4: Knowledge of cultural practice
Factor 5: Knowledge of world geography
Factor 6: Knowledge of global interdependence
4
6
2
1
5
4
1.000
6
.423
1.000
2
.419
.269
1.000
1
.298
.314
.358
1.000
5
.377
.464
.157
.204
1.000
3
.308
.213
.446
.207
.124
3
1.000
Persistence Toward Graduation at
Test Sample of Three USG State Universities
100%
92%
80%
60%
Studied Abroad
40%
20%
22%
Didn't
0%
Graduated and/or Stayed in
School
Blue column includes 2000-2001 study abroad students from three universities who had graduated or were still enrolled in Spring 2003.
Red column represents average graduation rate at the three institutions.
Improved Graduation Rates…
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
85.1%
53.0%
90.8%
57.6%
45.6%
Studied Abroad
30.6%
Didn't
University of Minnesota
study of 1999 entering
freshman class.
Four-year
rates
Five-year
rates
Six-year
rates
…especially among lower-aptitude
students (ACT/SAT + HSGPA index)…
90%
77.2%
80%
81.0%
70%
60%
50%
47.1%
40.5%
40.9%
40%
30%
20%
Studied Abroad
Didn't
17.7%
University of Minnesota
study of 1999 entering
freshman class (cohorts
with low admissions index;
AAR = 100-119)
10%
0%
Four-year rates
Five-year rates
Six-year rates
…and among students of color
90%
84.8%
76.1%
80%
70%
60%
50%
40%
30%
20%
41.3%
40.3%
44.0%
Studied Abroad
Didn't
21.0%
10%
University of Minnesota
University of Minnesota
study of 1999 entering
study of 1999 entering
freshman class. freshman class.
0%
Four-year Five-year Six-year
rates
rates
rates
Next Steps, Questions, & Challenges:
Refine Cognitive Learning Outcome Measures
in Student Survey and Expand Test
Populations
Analyze Academic Performance Measures
Identify and Test Critical Thinking and Other
Skills Assessments
Content Learning Abroad vs. At-Home
Invite New State/Institutional Partnerships to
Replicate Assessments
For Further Information:
On-going reports on this project will be
posted periodically on our website:
www.usg.edu/oie
Or contact us at:
Office of International Education
University System of Georgia Board of Regents
270 Washington Street SW
Atlanta, GA 30334
Tel: 404/651-2950