OCS Assessment: The Challenges of
Learning from Experience
Carol Trosset
Institutional Research and Assessment
OCS Workshop, November 2017
Institutional Research and Assessment
1
Findings from Last Year
Carleton’s Goals for Off-Campus Study
Programs
with this
Goal
Reported
by
Seniors
Reported
by
Alumni
#1 – Use intercultural encounters, field study, and their own
experiences, to expand their curiosity and ways of knowing.
72%
35%
36%
#2 – Acquire understanding of an unfamiliar environment.
72%
5%
4%
#3 – Make connections between the new environment and
their own.
31%
11%
40%
#4 – Develop discipline-related and interdisciplinary skills.
90%
26%
13%
#5 – Recognize and think about connections between local,
national, and global challenges, and pursue solutions to these
challenges.
17%
9%
9%
0%
11%
6%
14%
28%
6%
#6 – Off-campus study informs the student’s further
education.
#7 – Expand vision of life after Carleton and explore a broader
range of life and career opportunities.
Institutional Research and Assessment
2
Goals – Carleton, AACU, ACM
Carleton
AACU
ACM
Empathy; Communication;
Curiosity; Openness
Navigation;
Communication;
Pose thoughtful questions
#2 – Acquire understanding of an
unfamiliar environment.
Learn about cultural diversity and
this culture’s worldview
Close observation;
Grasp of complexity
#3 – Make connections between
the new environment and their
own.
Cultural self-awareness;
Perspective taking
Awareness of differences;
Balance own view and
emic views
#4 – Develop disciplinary and
interdisciplinary skills.
Knowledge application
Use academic work to
understand surroundings
#5 – Recognize connections
between local, national, and
global challenges.
Global systems;
Global self-awareness;
Personal and social responsibility
#1 – Expand their curiosity and
ways of knowing.
#6 – Inform the student’s further
education.
#7 – Expand the student’s vision
of life after Carleton.
Institutional Research and Assessment
3
ACM essay assignment, 2014-15
• Programs in Botswana, Costa Rica, Florence, India, Jordan, Tanzania
• 80 early essays, 85 late essays
Paraphrase of the “Neighborhood Walk” prompt:
• Spend 30 minutes or more observing this area. What are your initial
impressions? What seems familiar, or strange? What puzzles you? How
does your current level of language ability affect your understanding of
the neighborhood? How does your prior academic work help you
understand what you are observing? What skills help you navigate? Talk
with at least one local person about how they see this area. Write a page
or two describing your observations and reflections.
Institutional Research and Assessment
4
Navigation and Communication
Students get better at this – finding their way around, interacting
with local people in the local language, and feeling more confident
while doing so.
Institutional Research and Assessment
5
Posing Thoughtful Questions
Most students do not pose any questions. Early essays included more questions than late ones.
• “Each house in the neighborhood has very high walls, barb wire on the walls that surround the house, and
an outside door that has a complicated lock system. It strikes me that the community here is very cautious
and very insecure. My host mother says that the neighborhood is very safe but people have these
structures as precaution. I am hoping to discover why there is a lack of trust in my neighborhood even
though it is considered very safe.”
• “I wonder if the decoration on this bridge signifies that it is more significant to Firenze, or was more
significant historically, or was perhaps the first bridge to be rebuilt (and thus had more time and money
invested in it).”
Institutional Research and Assessment
6
Close Observation
Many students seem quite observant, but most make no attempt to use their observations to
understand the local culture.
• “On the outskirts of the city, al-Bayader seems to be the product of rapid development; the
community seems to be an odd mixture of new and old. For example, a new building with
storefronts opened just down the street from our host family’s apartment, but at the same
time a shepherd every day leads his flock to the empty plot of land across the street to graze.
Also, 8th circle seems to be one of the busier circles in the city, but somebody seems to have
taken it upon themselves to cultivate some of the weird empty plots of land right next to it.”
Institutional Research and Assessment
7
Grasp of the Host Culture as Complex
Most references to complexity mention the presence of more than one ethnic group. A few
mention features of the physical environment.
• “Places like Ubungo have a wonderful hodgepodge of Tanzanians of different backgrounds, and they offer
the opportunity to get to know people of different mindsets and histories. Going to Ubungo, I was
reminded of a fact that I’ve slowly been internalising all semester: all Tanzanians are distinct individuals,
and just because one person says something about Tanzanian culture, does not mean that it’s true for
everyone else.”
• “[The market] is also acutely responsive
to its environment, with the temporary
and mobile nature of frames allowing the
market to shift locations, popping up
around dala dala junctures or popular
student hubs.”
• “I notice the differences in each of the
different neighborhoods. … There are
both tons of tourists, as well as many
Florentine families wandering around.”
Institutional Research and Assessment
8
Awareness of Cultural Differences
Many mentions of differences are superficial and focus on the infrastructure.
• “I can imagine the ride from the airport to my house when I go back to the United States to
feel strange. The roads will be something I will have to readjust to.”
Other students comfort themselves by finding similarities to their home.
• “No matter what country you are in, it is possible to find something that is familiar. You will
almost always find someone who is looking for the same thing as you in life, or has the same
interests or beliefs. All of humanity is connected somehow by one of these things. I hope
that while I am here I can discover more similarities between the beliefs and interests of the
people of Costa Rica and mine.”
• “It kind of strikes me to be impractical,
though, that most of the vendors that sell
souvenirs all sell basically the same
items. I don’t understand how hundreds
of people selling the same thing really
can be very effective, but it must work
reasonably well. I suppose it’s similar to
going to Times Square in New York City
with ‘I love New York’ stuff everywhere.”
Institutional Research and Assessment
9
Balances Own and Emic Points of View
Most seem unaware that the local people might understand things differently. Many students
only mention their own viewpoint. Others attribute their own views to the local people.
• “I immediately noticed that trash was everywhere: on the ground, in bags, in the ditches near the road,
and slowly burning on the side of the hill. However, I observed that Tanzanians have other methods of
cleaning. When we arrived at the market, I saw a man washing a dala-dala with soap and water. A few
minutes later, I noticed a woman sweeping the street (amidst the speeding dala-dalas, bujajis, and other
vehicles) and school children running around in clean and neat uniforms. I think that by cleaning
something, one inherently has some amount of pride for the object. This leads me to believe that
Tanzanians take pride in different things than Americans typically do. So, while Tanzanians may not worry
about littering to the extent that Americans do, they take pride in their education and the cleanliness of
their vehicles and roadways.”
• “In American it is normal that people arrive on
time for their work and work with a lot of stress.
While in Monteverde, my boss told me that we
meet around 8:00 but in reality that means we’ll
meet around 9:30 or maybe few days later…
Schedules and times are very different here. I
was angry with that style of work at the
beginning. Later, I realized that it is the way of
life in this community is different and there are
different values in the communities: In terms of
work, they are a lot slower and the people are
very calm and do not like stress.”
Institutional Research and Assessment
10
Applies Academic Learning to Surroundings
Most present some superficial information about the location.
Hardly any mention anything learned in coursework.
• “I recognize the symbol of the Rucellai family who paid for this façade (something
I learned in Professor Mariotti’s class).”
• "My Economics class has helped me to understand some of the economic system
in Jordan."
• “I noticed that in the area we were in,
most of the clothes and accessories
seemed to be designed for men. Also, the
majority of the stall and shop vendors
were men, and I could see women in the
back either cooking or cleaning. It
reminded me of a passage in the book
Africans and Americans where the author
describes the difference between men’s
work and women’s work in Tanzanian
culture. This was definitely noticeable in
the market, and I often saw groups of
men trying to sell items and groups of
women cooking beans or cornmeal.”
Institutional Research and Assessment
11
So, what are they thinking about?
Most tell a story about their walk:
• “W.S. Bakers was our next stop, and it was not on the map that we were given, so we had to
ask around for directions. Several people did not understand what we were saying, and some
had not heard of W.S. Bakers. Eventually we found a girl who did not know where it was, but
who called a helpline to try to find it. We walked with her while she talked to the person on
the phone (who honestly was not very helpful), and eventually one of us spotted the bakery.
The bakery was really small inside, with only enough chairs for the four of us.”
and reflect on how comfortable they have become:
• “No longer do road crossings seem like an impossible task. Now I understand the traffic flow
and don’t think much before walking into a sea of traffic. I understand that if I see an
opening in traffic I must take it now.”
• “I stop at a jeans store near my house and I have a conversation with the owner, a
conversation I probably could not have had just a few months before. I keep walking, smiling
to myself because I can successfully ride the bus or get places walking without a constant
fear of going in the wrong direction.”
Institutional Research and Assessment
12
As they become more comfortable,
they show less interest in their surroundings.
• “Taking the assigned walk left little impression on me. I remember the stress of
walking with the group, and my original impressions (namely that I believe, and
still believe, the piazza at Piti Palace to be rather ugly), none of which had
changed.”
• “We buy biscuits and a muffin [to feed a stray dog]. Those drinking garam chaha
stop to smile and stare at the strange group of foreign girls feeding a dog. I laugh
and smile, knowing that we are certainly making a name for ourselves but not
necessarily committing any cultural faux-pas. It’s nice to know this though, as well
as to have the confidence to be a little more silly while here. Much of the time
early on, I tried so hard to blend in to adapt to Indian lifestyle, and now I feel
more comfortable deciding between some aspects of Indian and American
culture.”
• “Honestly speaking — we were slightly bored on this walk. We stopped at
Starbucks and grabbed a drink (like true Americans) and continued on our walk.
The scene was so normal for us by this point we were lost in conversation with
each other.”
Institutional Research and Assessment
13
Many seem to equate navigation skills with
cultural adaptation.
• “I often don’t give a second thought to what I see, hear and smell. I paid more attention to
people during the first walk but felt more connected to people during the second walk. It
was much easier to ask for directions because I am now more comfortable with my
surroundings while in Pune. People seemed more receptive during this walk than during the
first. I think this is because we are more confident in where we were headed and didn’t have
to try so hard to take in our surroundings.”
• “Other than knowing where I am now, what’s different than the first neighborhood walk is
that I now feel more confident being here. I can walk into a random bar, order a cappuccino
‘porta via,’ and not stop to feel awkward or out of place. I can mosey through Piazza’s and
parks, and not worry about avoiding eye contact with everyone as to not attract unwanted
Italian conversation, and have to admit to being from somewhere else.”
• “This whole new world…is now a place where I feel comfortable… Comparing [here to home]
at the moment is difficult, because the ability to cross a street without being honked at or
rushed by speeding vehicles, the stereotypical Tico times, the one handed hand clap, and the
kiss on the check when meeting someone seems normal. All of the barriers that existed
during the transition period are now part of my everyday life.”
Institutional Research and Assessment
14
My conclusions
• Many students are not naturally inclined to do the kind of
observation and thinking that we want them to do out of
class. They don’t learn these things just by going abroad.
• Even if they are capable of this kind of thinking, it is not a
widespread “habit of mind.”
• Those who are observant don’t yet know how to think
analytically about their observations.
• Many will only focus attention outside themselves if they
have to.
• Many set the “cultural adaptation” bar very low.
• These are probably common qualities, which means we need
to teach the desired skills very intentionally.
Institutional Research and Assessment
15
Discussion with ACM OCS Directors
• Students are typically more interested in their own experience of a place
than they are in the place itself.
• Students regularly compartmentalize – “these are my academic classes,
and this is my life (‘down time’) outside of class.” Most don’t see the
experiential part of OCS as academic.
• Just telling students to “reflect” usually results in them thinking about
themselves.
• Students don’t already know how to learn experientially.
• Faculty who know a location well are not necessarily effective at teaching
the students how to learn from their own experiences.
Institutional Research and Assessment
16
Observational sciences as a partial exception
• In field biology, geology, and cultural anthropology, the academic work is
about the things you are observing. Some other disciplines could be
taught that way (?), but often aren’t.
• Example: Jay Levi collected essays in Guatemala, asking students to
interpret some aspect of the local culture. I applied the ACM rubric:
Institutional Research and Assessment
17
When teaching an OCS program,
how do you make sure that your students:
•Think of their out-of-class “down time” as part of
their learning experience?
•Focus on trying to understand their surroundings?
•Grapple with differences that are not superficial?
•Avoid confusing a basic ability to function with real
understanding of another culture?
•Apply classroom learning to everyday experiences?
Institutional Research and Assessment
18