Revisit L&W 1967. Reliability of Narrative Analysis
(Labov & Waletzky, 1967)
Natural Language and Dialogue Systems Lab
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
What is a narrative?
 Referential and evaluative
 A sequence of clauses in the correct temporal order
 Contains at least one temporal juncture
 I interpret this to mean some ‘causality’
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
A complete narrative contains:





Orientation
Complication
Evaluation
Resolution
Coda
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
From “The Security Breach”
by Kim Wolterman
Action
Evaluation
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
Orientation
Coda
UC SANTA CRUZ
Aim: Can people do this reliably, with high agreement?
 Is the model Good? Robust? Comprehensive?
 Are there categories of stuff in narrative that this model is
missing?
 Is it possible to reliably distinguish between the categories
that are there?
 Reliability: need for both theoretical and empirical,
scientific reasons
 My proposal: forget about ‘complication’, ‘temporal
junction’ and ‘displacement sets’
 Focus on whether we can distinguish orientation,
evaluation, actions and coda.
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Aim: Can people do this reliably, with high agreement?
 Machine Learning: Categories clearly separated
 Need some level of agreement at least about 70%
(Kappa of .7)
Expert or Xbar
Coder
Action
Action
33
Evaluation
Orientation
Coda
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
Evaluation
Orientation
Coda
55
34
22
UC SANTA CRUZ
Aim: Can people do this reliably, with high agreement?
 Machine Learning: Categories clearly separated
 Need agreement
 What would you do if this is what you got?
Expert or Xbar
Coder
Action
Action
33
Evaluation
Evaluation
Orientation
Coda
55
4
22
Orientation
Coda
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
34
22
22
UC SANTA CRUZ
How Kappa is calculated & methods for 
 Define a codebook, annotation manual
 Collapse categories that are confusable
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Aim: Can people do this reliably, with high agreement?
 My proposal: forget about ‘complication’, ‘temporal
junction’ and ‘displacement sets’
 Focus on whether we can distinguish orientation,
evaluation, actions and coda.
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Stephanie Lukin and Georg Albrecht
Natural Language and Dialogue Systems Lab
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
From “The Security Breach”
by Kim Wolterman
Action
Evaluation
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
Orientation
Coda
UC SANTA CRUZ
From “Canyon Living: Riding Horses”
by Jennifer Doherty Brown
Action
Evaluation
Orientation
Coda
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Action or Orientation?
 “We started by gathering the horses [which took a little
under half an hour] since Zip did not want to be corralled and
saddled”
 “Once the horses, saddled and bridled, loaded up in the
trailer, we were off to the pasture east to get some strays
before moving cattle closer to the house [(a mere five mile
ride)].”
 “[By the end of several hours], I had only a sore knee to keep
me company (and we did grill steaks and fresh zucchini for
lunch).”
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Evaluation or Orientation?
 “As I intimated in the last post, my trip home from
Canada was rather exciting.”
 “Canada international travel is somewhat interesting
as you have to clear customs there before heading
into the United States.”
 “We figured since we were arriving in St. Louis
before the guys”
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Action or Evaluation?
 “I found an old favorite bay horse named Tivio to be
my perfect companion.”
Evaluation or Coda?
 “I'd say more riding is in my future. I loved the time
out in nature and with my husband.”
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Action or Evaluation?
 “First clue that something was awry is that the gal told
Ted he had checked one bag, [which he had not].”
 “I am assuming that was because [we used his credit
card, so she just attached it to his ticket].”
Evaluation
or Coda?
 “But alas that was not all the AA agent had screwed up.”
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Some differences
 Action or Evaluation? (section contained in [])
 “First clue that something was awry is that the gal
told Ted he had checked one bag, [which he had
not].”
 “I am assuming that was because [we used his credit
card, so she just attached it to his ticket].”
 Evaluation or Coda?
 “But alas that was not all the AA agent had screwed
up.”
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Towards annotating narrative structures reliably
Natural Language and Dialogue Systems Lab
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
The Narrative Clause
 The "basic unit" of narrative
 A clause that is temporally ordered
 Contains a verb, the "narrative head”
 WHAT KIND OF VERB?
 STATIVE VS. ACTIVE?
 MENTAL STATE VERBS
 Has an "unordered displacement set”
 Labov 1997 defines this much better
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
The Narrative Clause
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Livia Polanyi
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Orientation
 Formally: The group of free clauses that precede the first
narrative clause.
 The beginning part of a narrative that "sets the scene" to help
orient the audience.
 Used to describe who/where/when/why.
 Ex: "One afternoon, after finishing lunch, my friends and I were
walking to our car when suddenly...".
 Thorne: Orientation (which orients the listener with respect to
person, place, time, and situation) is formally defined as preceding
the first action clause (p. 27), as in “It was a dark and stormy
night, or “Last Friday,” or “remember that guy from back home,
well last night when we were at the restaurant, he …” (p. 27).
However, orientations can be scattered throughout the story.
Orientation is not a crucial feature of a story. Temporal action
and evaluation are crucial aspects.
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Evaluation







SEEMS LIKE IT COULD OCCUR ANYWHERE
Highlights the significance of the result.
May include the story's "moral" or the narrator's perspective.
SUBJECTIVITY? SENTIMENT?
Thorne: 5. Evaluation is crucial to a story because it tells the attitude of the teller, or the point of
the story; stories are told for a reason (p. 28), to “establish some point of personal interest” (p. 29).
Evaluations can be very brief. For example, at the end of a near-death story, the teller says “You’d
have been dead.” This tells the listener that the experience was really dangerous; anyone could have
died. (p. 30). Or the teller might say “It was cool man, I survived”, which conveys a very different
attitude on the part of the teller. “The evaluation .. reveals the attitude of the narrator toward the
story by emphasizing the relative importance of some narrative units relative to others.” (p. 32) the
evaluation might serve the function of self-aggrandizement (as in “You’d have been dead, I survived,
man!” (p. 34) At some point L & W say that evaluations suspend the action (sort of like stepping
out of the action to express a point of view about the meaning of the action or experience to the
teller or to other characters in the story).
6. Stories vary in the degree to which evaluations are stated directly as the teller’s attitude or point
of view ; “external” evaluations state the teller’s point of view directly (I thought it sucked”. Some
evaluations are “internal” in that they are expressed through 3 rd parties in the story (“he said it
sucked”). evaluations are stated directly or more internalized. (pp. 34-35). page 35 shows a chart of
evaluations that vary from internal to external.
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
CODA
•
•
Occasionally present after the Resolution.
Returns the narrative perspective to the present.
o
o
•
o
Deixis: Shifting narrative perspective to present and
referencing the story as occurring in the past.
Reference to the present state of some aspect of
narrative.
Reference the current effect of the narrative.
All codas are separated from the Resolution by a
temporal junction.
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Some other NLP work doing narrative annotation
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
Extra slides
Natural Language and Dialogue Systems Lab
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ
This one claims that different types of stories have different event types
NATURAL LANGUAGE AND DIALOGUE SYSTEMS LAB
UC SANTA CRUZ