The EXBODIED MIND: Motion in Communication and Cognition Research
RWTH A
Aachen
h U
University,
i
it 8
8-9
9A
Aprilil 2011
Gesture,
Conceptualization, and
Distributed Cognition
Robert F. Williams
Appleton Wisconsin
Appleton,
Approach
Distributed Cognition
Cognitive Semantics
Studying Human Cognition
Classical View
Studying Human Cognition
Classical View
SHIFT
Situated View
Cognitive Functional Systems:
Everyday Examples
• Tracking attendance
• Determining order of service
• Counting objects
• Telling time
Functional System
?
!
?
C
Coordination
di ti
C
Conceptualization
t li ti
Questions
• How do people accomplish cognitive activities?
– coordination of representational media => computation
• How do they construct relevant meanings?
– conceptualization
• How do they guide the meaning of others?
– instructional discourse (multimodal)
• How do they reason collaboratively?
–g
group discourse ((multimodal))
Methods
• Data collection
– Ethnographic study (cognitive ethnography)
– Quasi-experimental situations
• Analysis
– Distributed cognition: how activities are accomplished
– Cognitive semantics: conceptual structures/processes;
guiding conceptualization
=> role of bodily actions (gestures, manipulations, enactments, etc.)
Example: Counting Objects
“one,
o e, two,
t o, three…”
t ee
How
many?
?
Coordinating
action
…
#
Functional Systems for Counting
one, two, three…
one, two, three…
six, seven, eight…
A, B, C…
Functional Systems for Counting
MOVING OBJECTS
SEQUENTIAL TOUCHING
(POINTING / LOOKING)
“six” “four”
“one”
“two”
“three”
“two”
…
USING FINGER PROXIES
[b] [c]
[a]
[b] [c]
[a]
Coordination:
Use of the body to coordinate spoken representations with material objects
Functional Systems for Counting
MOVING OBJECTS
SEQUENTIAL TOUCHING
(POINTING / LOOKING)
To be counted
Already counted
S
“one”
“two”
To be counted
“six” “four”
“three”
“two”
G?
TR
…
Already counted
USING FINGER PROXIES
Already counted
Already counted
[b] [c]
[b] [c]
[a]
[a]
To be counted
S
TR
To be counted
Conceptualization:
Image-schematic structure (SOURCE-PATH-GOAL, PROXIMITY, CONTAINER)
Material/spatial anchoring of conceptual categories
Instructional Discourse:
Counting on the Clock
Selecting Episodes for Analysis
Activity
Focus
Duration
8:56
Presentation
Review dividing a circle into halves and fourths on felt board
0:34
Presentation
Equate one fourth to one quarter by analogy to money
0:30
Presentation
Divide the clock face into quarters
0:33
P
Presentation
t ti
R d a ti
Read
time as ““quarter
t past”
t”
0 41
0:41
Group practice
Read a quarter past eight with prompting
0:15
Group practice
Read a quarter past ten and a quarter past three
0:15
Count on the clock face to read the time as “_ fifteen
fifteen”
0:26
Presentation
Group practice
Read five fifteen and a quarter past five
0:17
Group practice
Read a quarter past seven and seven fifteen
0:13
Individual p
practice
Read eight
g fifteen and a q
quarter p
past eight
g
0:29
Individual practice
Read a quarter past two and two fifteen
0:28
Presentation
Write two fifteen as ‘2:15’
0:14
Individual practice
Read a quarter past four and four fifteen; write 4:15
0:46
Individual practice
Read six fifteen and a quarter past six; write 6:15
1:23
Individual practice
Read ten fifteen and a quarter past ten; write 10:15
1:52
Transcribing
Transcribing
Transcribing
Analyzing
• Diagramming conceptual inputs and operations
step-by-step in the unfolding discourse
• A
Analyzing
l i th
the roles
l off b
bodily
dil movements,
t th
their
i
relation to speech, coupling with environment,
etc.
etc
Prompting for a New Conceptualization
11 12 1
2
10
9
3
8
4
7
now another way that we sa:y it
6
5
Activating a Cognitive Model
11 12 1
“fi
“five,
tten, fift
fifteen…””
2
10
9
G
3
8
S
4
7
…
Counting by Fives
is we count by fi:ves
6
5
Mapping #1
11 12 1
#1
“fi
“five,
tten, fift
fifteen…””
2
10
9
G
3
8
S
4
7
…
11 12 1
2
10
9
3
8
4
7
6
5
when we move this,
6
5
Mappings #2 and #3
11 12 1
“fi
“five,
tten, fift
fifteen…””
2
10
9
G
3
8
S
#2
…
7
#3
11
S
12 1
10
9
2
3
8
4
7
4
6
5
G?
from number to number=
6
5
Mapping #4 (and #5)
days
...
11 12 1
“fi
“five,
tten, fift
fifteen…””
G
2
10
9
S
3
8
#4
...
4
7
…
6
...
5
...
hours
minutes
...
...
(#5)
Time Measurement
11
S
12 1
10
2
3
4
5 min
9
8
7
6
5
G?
there’s five minutes between each number
Transition
days
...
11 12 1
“fi
“five,
tten, fift
fifteen…””
G
2
10
9
S
3
8
...
4
7
…
11
6
S
12 1
10
...
5
2
3
4
5 min
9
8
7
6
5
G?
so if we were going to count by fives it would be:
...
hours
minutes
...
...
Counting on the Clock
days
...
11 12 1
“five, ten, fifteen…”
9
G
S
11
11
2
9
3
8
4
6
(0.5) fi:ve
5
6
...
5
S
“five”
five
10
7
4
7
12 5 1
...
3
8
…
S
2
10
...
10
12 5 1 “ten”
5
2
3
8
4
6
(0.6) te:n
hours
minutes
...
...
S
9
7
...
5
11
...
12 5 1
5
10
9
8
2
5
3 “fifteen”
4
7
6
(0.4) fiftee:n=
5
G
Conceptual Integration Network
days
...
11 12 1
“fi
“five,
tten, fift
fifteen…””
G
S
2
10
9
3
8
4
7
…
...
6
...
5
Counting by Fives
...
hours
minutes
...
...
Time Measurement
S
11
“five”
125 1
2
10
9
3
8
4
7
6
5
G?
Counting on the Clock
Group Reasoning
Explaining the phases of the moon
Group Reasoning
Reasoning about the seasons
Methodological Issues
• Capturing phenomena
– Ethnography provides “real” data & warrants
access, time, equipment, when/what to record, missed data
– Quasi
Quasi-experimental
experimental situations provide control & capture
less natural, lack ethnographic warrants for interpretations
Methodological Issues
• Transcribing (distilling)
– Coding gesture highlights types & patterns
isolates gestures, may mask functions
– Annotated images highlight gesture functions
no support for categorical analysis
Methodological Issues
• Interpreting / analyzing
– Distributed cognition & cognitive semantic analysis
can’t be automated (requires expert analyst), hard to generalize
– Quantitative / statistical analysis
bleaches out situated aspects of cognition & communication
What can be done?
• For now
– Multiple avenues: match approach to question
• For the future: new digital tools
– Transcripts with embedded videos & metadata?
– Links to analyses?
– Searchable databases or gesture corpora?
• And then: …?
Selected References
Fauconnier, G. & Turner, M. (2002). The Way We Think: Conceptual Blending and the
Mind’s Hidden Complexities. New York, NY: Basic Books.
Hutchins E
Hutchins,
E. (1995)
(1995). Cognition in the Wild
Wild. Cambridge,
Cambridge MA: MIT Press.
Press
Hutchins, E. (2005). Material anchors for conceptual blends. Journal of Pragmatics,
37(10): 1555-1577.
Williams, R. F. (2006). Using cognitive ethnography to study instruction. In S. A. Barab,
K. E. Hay, and D. T. Hickey (Eds.), Proceedings of the 7th International Conference
of the Learning Sciences (vol. 2, pp. 838-844). International Society of the Learning
Sciences / Erlbaum.
Williams, R. F. ((2007).
) Counting
g and conceptual blending.
g 10th International Cognitive
g
Linguistics Conference, Krakow, July 15-20.
Williams, R. F. (2008a). Gesture as a conceptual mapping tool. In A. Cienki & C. Müller
(Eds.), Metaphor and Gesture [Gesture Studies 3] (pp. 55-92). Amsterdam: John
Benjamins.
Williams, R. F. (2008b). Situating cognition through conceptual integration. 9th
Conference on Conceptual Structure, Discourse, and Language, Case Western
Reserve University, Cleveland, OH, October 18-20
Williams R.
Williams,
R F.
F (2010).
(2010) Gesture in everyday scientific reasoning and explanation
explanation. 4th
Conference of the International Society for Gesture Studies, Europa University
Viadrina, Frankfurt/Oder, July 25-30.
Thank You
Natural Media & Engineering
Human Technology Centre (HumTec)
RWTH Aachen University
Interdisciplinary Center for Clinical Research
University Hospital Aachen
RWTH Aachen University