“And the words I uttered myself, and which
must nearly always have gone with an effort
of the intelligence, were often to me as the
buzzing of an insect. And this is perhaps one
of the reasons I was so untalkative, I mean
this trouble I had in understanding not only
what others said to me, but also what I said
to them. It is true that in the end, by dint of
patience, we made ourselves understood, but
understood with regard to what I ask of you,
and to what purpose?”
(Samuel Beckett Trilogy, p.50)
Interactive Misalignment:
The Role of Repair in Semantic Co-ordination.
Pat Healey,
Greg Mills, James King, Nik Swoboda, Ichiro Umata,
Interaction, Media and Communication Research Group,
Department of Computer Science,
QMUL
Outline:
1. How do we communicate? How do co-ordinated languages
evolve?
…despite differences in interpretation…
2. Group-specific sub-languages.
Verbal Dialogue: Spatial Reference in the Maze Task
Graphical Dialogue: The Music Drawing ʻPictionaryʼ Task
3. What does interaction contribute?
Interfering with graphical dialogue
Interfering with text-based dialogue
4. Repair-driven co-ordination
'Naïve' Code Model of Communication:
Message
Message
NOISE
ENCODING
DECODING
Signal
SOURCE
RECIEVER
Problem:
Successful communication requires identical codes &
processes at source and receiver.
– assumption of linguistic homogeneity
“ [it is a] patent truth that no two speakers of the same language
ever speak exactly the same dialect of that language. “
(Fodor & Lepore, 1992, p.10)
• ordinary communication depends on ignoring differences in
interpretation (Garfinkel, Putnam)
Overt Repair and Clarification
• ʻDisfluenciesʼ / Self-repairs:
“well, . I mean this . uh Mallet said Mallet was
uh said something about uh you know he felt it”
– (Brennan & Schober 2001, Clark & Foxtree, 2002)
• Clarification Questions / NTRIʼs
“what?” “eh?” “who?” “Oscar what?”
–
(Purver et. al. 2003)
• Paraphrase / Reformulation
“oh you mean…” “yes no I mean…”
–
(Schegloff, 1987,1992)
Approx 1 overt repair event every 3 turns.
Covert Repair / Clarification
• Embedded correction (Jefferson)
Mary: “I have been invited to their marriage
next week. I can’t wait.”
Susan: “I’d love to go to their wedding, but
am not sure I can make it.”
Mary: “If you can make it we’ll go to the
wedding together.”
• negative ‘contrast’ evidence more important for acquisition than
positive evidence (Saxton, 1997; 2005)
Clarification and Repair is:
• Ubiquitous
• Structurally complex (i.e. not just noise)
• Defining characteristic of dialogue
How do we co-ordinate understanding
despite differences in interpretation?
Consider two tasks:
The Maze Task
• Recurrent problem of describing target locations
• Target alternates and configuration changes on each trial
Player A
Player B
Maze Task: Description Types
Figurative (Figural / Path): sensitive to particular configuration
A: right on the right hand side there are four boxes,
B: mmhum
A: then there are two shapes and then there's another four linked boxes,
B: yes
A: right it's the second from the bottom.
B: mmm, ummm, take the bottom left hand corner,
A: yes
B: up one box
A: yes
B: right one box
A: yes
B: up one box
Maze Task Description Types
Abstract: (Line / Co-ordinate): abstracts underlying grid structure
A: ummm, fourth row down and the second from the right, [12]
B: okay it's the second row down and second in from the left, [13]
B: er: two two, [3]
A: six: six three, [4]
B: four three, [5]
(Kappa = 0.76, N =455, k= 2)
The Music Drawing Task:
Exclusively graphical interaction via virtual whiteboard
• Pairs seated in separate rooms
– 30 sec piano piece each - SAME or DIFFERENT?
• Draw picture of target: no letters or numbers
Room A
Room B
Music Drawing Task: Drawing Types
Figurative:
– Ad hoc associations: faces, figures, objects or situations
Abstract:
– Graph-like representation of domain structure e.g., pitch,
intensity, rhythm
Composite:
– Mixture of Abstract and Figurative
(Kappa = 0.9, N =287, k= 2)
Sequence of ʻFigurativeʼ Trials:
Sequence of ʻAbstractʼ Trials:
How do we co-ordinate on ʻAbstractʼ or
ʻFigurativeʼ?
1. AUTONOMOUS PROBLEM SOLVING:
• Select scheme that is optimal for the task.
– co-ordination develops through aggregate individual
experience (e.g., Clark, Lewis…)
2. COLLABORATIVE CO-ORDINATION
• Use interaction to establish a co-ordinated 'sub-language'.
– ʻlanguage gameʼ metaphor
Experimental Test:
Phase1: Community Development
Subject 1
Subject 6
Round 1 =
Round 2 =
Round 3 =
Round 4 =
Subject 2
Subject 5
Subject 3
Subject 4
Both Tasks Phase 1:
1. Different partner on each round
2. Common ʻinteraction historyʼ accumulates
3. Manipulations of group and dyad structure are hidden
•
Music task: 10 ʻcommunitiesʼ of 6 people
– 4 rounds of 12 trials
– reliable increase in speed and accuracy
•
Maze Task: 4 ʻcommunitiesʼ of 8 people
– 5 rounds of 20 trials
– reliable increase in number of items completed
Phase 2: Experimental Manipulation
Within Group
=
Between Group
=
Music Drawing Results for Phase 2:
0.7
Abstract
Figurative
0.6
Composite
Proportion
0.5
0.4
0.3
0.2
0.1
0
Within Community
Between Community
Chi2(2) =19.0, p=0.00
Music drawing Results for Phase 2:
Maze Task Results for Phase 2
0.8
0.7
0.6
0.5
Abstract
Figurative
0.4
0.3
0.2
0.1
0
Within-Group
Cross-Group
Maze Task Results for Phase 2
Crossing between sub-groups:
• Changes choice of description types
– (cross-group vs. within-Group: Chi2(3) = 129.6, p=0.00)
• Provokes twice as many clarifications
– (cross group 37%, within group 16%)
• Cross-group pairs are not distinguishable from ʻnaïveʼ pairs on
trial 1
– Cross-group vs. Naïve: Chi2(2) = 3.34, p=0.19.
What is the mechanism of co-ordination?
Not Precedence / Co-ordination Equilibria (Lewis 1969, Clark
1996)
• choose description type, agree to use it
– systematic shifts in choice of description type over time
– systematic shifts in response to trouble
– alternatives are not arbitrary equivalents
Not Alignment / Matching (Pickering and Garrod, 2005)
• Within and Cross-group pairs have same level of alignment
• People converge on the least ‘primed’ initial description type
Not Explicit Negotiation
• Maze task: rare, often violated, and most common after coordination achieved.
• Music Task: no meta-language
– bootstrapping problem: explicit negotiation presupposes a coordinated meta-language
ʻAbstractʼ vs. ʻFigurativeʼ Semantic Models
Co-ordination models are semantically neutral but abstract-figurative
contrast is not
ʻAbstractʼ Semantic Models capture regularities across items:
1. Systematicity: support direct comparison within and between
items
• can infer relationships between different descriptions/drawings
2. Proto-compositionality: meaningful elements (w.r.t. the domain)
• (relatively) consistently individuated ontology
Why are the ʻabstractʼ representations more unstable?
Co-ordination of ʻAbstractʼ models?
•
multiple possible ontological schemes:
– boxes, squares, links, points, rows, columns, diagonals…
– pitch, melody, rhythm, tempo, intensity, ….
•
need to narrow down possible differences
•
models are not directly manifest in particular items
(you canʼt point at them)
How does direct interaction help?
Hypothesis: by exploiting the potential for juxtaposing and
contrasting contributions.
Graphical Interaction Mechanisms?
Mechanisms:
•localisation
•alignment.
Manipulation 1: Block Localisation
Room A
Screen
Room B
Screen
Annotation of
otherʼs drawing
blocked.
Manipulation 2: Block Alignment
Room A
Screen
Room B
Screen
Transpose
- BLOCKING
+ BLOCKING
+ TRANSPOSITION
- TRANSNPOSITON
Experimental Design:
Subject A’s View
Subject B’s View
Subject A’s View
Subject B’s View
Effects of Interference with Interaction:
Figurative
0.70
Abstract
Proportion of Drawings
0.60
0.50
0.40
0.30
0.20
0.10
0.00
+B +T
+T-B
-T + B
Level of Interaction
-T -B
(Blocking: Chi2(3) = 96.70, p =0.00, Transposition: Chi2(3) = 81.61, p =0.00)
Interim Conclusions
• Changes to interaction (blocking & transpostion) cause change in
type of drawing
– independently of target items and of individual abilities
• Participantsʼ ability to manipulate each otherʼs representations
• not editing / annotation / revision per se
• ʻLocalisationʼ and ʻalignmentʼ exploit the potential for juxtaposition
and contrast.
What about conversation?
Repair and clarification also exploit juxtaposition and contrast
Laura:
 Jan:
Laura:
Can I have some toast please?\\
Some?
Toast.
Billy
 Peggy
Billy
aahh I see, kinda sick here and a bit pissed
drunk pissed?
angry pissed
Single Window
Dual Window
Single Window vs. Split Window
Distribution of Description Types:
• Multinomial Regression Analysis, Single vs. Split:
Chi2(3) = 247.6, p=0.000
Total Number of Turns:
Single Window: 4510
Split Window: 5244
Number of Spatial Descriptions:
Single Window: 4510
Split Window: 5244
Interim Conclusion
• Changes to interaction (turn integration) cause change in
description type
– independently of target items and of individual abilities
• Repair and clarification rely on the potential for juxtaposition and
contrast.
– not reformulation / revision per se
Conclusions:
1. Semantic co-ordination develops through the successive detection
and resolution of misunderstandings.
2. The key mechanisms are juxtaposition and contrast of contributions
– i.e., joint manipulation of external representations
3. Mechanisms are modality independent
– significant parallels between verbal and graphical (and
gestural?) dialogue mechanisms
• grounding, accommodation, turn-taking, ʻrepairʼ
4. Expressive power of ʻlanguageʼ depends on the specific interaction
mechanisms available to people
– not aggregate experience / expertise