Lessons Learned from Evaluation of
Summarization Systems:
Nightmares and Pleasant Surprises
Kathleen McKeown
Department of Computer Science
Columbia University
Major contributers: Ani Nenkova, Becky Passonneau
1
2
Questions
 What kinds of evaluation are possible?
 What are the pitfalls?
 Are evaluation metrics fair?
 Is real research progress possible?
 What are the benefits?
 Should we evaluate our systems?
3
What is the feel of the evaluation?
 Is it competitive?
 Does it foster a feeling of community?
 Are the guidelines clearly established
ahead of time?
 Are the metrics fair? Do they measure
what you want to measure?
4
5
The night Max wore his wolf suit and made mischief of one kind
6
and another
and another
7
His mother called him “WILD THING” and he said “I’LL EAT YOU UP!” so
he was sent to bed without eating anything.
8
DARPA GALE: Global Autonomous
Language Environment
 Three large teams: BBN, IBM, SRI
 SRI: UC Berkeley, U Washington, UCSD, Columbia, NYU,
UMASS, NCRI, Systran, Fair Isaacs, Ohio State
 Generate responses to open-ended questions
 17 templates: definitions, biographies, events, relationships, reactions,
etc.
 Using English, Chinese, and Arabic text and speech, blogs
to news
 Find all instances when a fact is mentioned (redundancy)
9
GALE Evaluation
 Can systems do at least 50% as well as a human?
 If not, the GALE program will not continue
 The team that does worst may be cut
 Independent evaluator: BAE
 Has never done text evaluation before
 Has experience with task based evaluation
 Gold Standard
 System responses graded by two judges
 Relevant facts added to the pool
 Granularity of scoring: nuggets
 Metrics
 Variants of precision/recall weighted
 Document citations
 Redundancy
10
Year 1: Sample Q&A
LIST FACTS ABOUT [The Trial of Saddam Hussein]
 The judge , however, that all people should have heard voices, the order
of a court to solve technical problems. (Chi)
 His account of events surrounding the torture and execution of more than
140 men and teenage boys from the Dujail , appeared to do little to
advance the prosecution's goal of establishing Saddam 's "command
responsibility" for the deaths.
 A trial without Saddam could be an embarrassment for the U.S.
government, which has worked hard to help create a tribunal that would be
perceived by Iraqis as independent and fair.
 As the trial got under way, a former secret police officer testified that he
had not received any orders from Saddam during the investigations that
followed an assassination attempt against him in Dujail in 1982 .
11
Year 1: Results
 F-value (Beta of 1)
 Machine average:
 Human average:
0.230
0.353
 Machine to Human average:
0.678
12
DUC – Document Understanding
Conference
 Established and funded by DARPA TIDES
 Run by independent evaluator NIST
 Open to summarization community
 Annual evaluations on common datasets
 2001-present
 Tasks






Single document summarization
Headline summarization
Multi-document summarization
Multi-lingual summarization
Focused summarization
Update summarization
13
DUC is changing direction again
 DARPA GALE effort cutting back
participation in DUC
 Considering co-locating with TREC QA
 Considering new data sources and
tasks
14
DUC Evaluation
 Gold Standard
 Human summaries written by NIST
 From 2 to 9 summaries per input set
 Multiple metrics
 Manual




Coverage (early years)
Pyramids (later years)
Responsiveness (later years)
Quality questions
 Automatic

Rouge (-1, -2, -skipbigrams, LCS, BE)
 Granularity
 Manual: sub-sentential elements
 Automatic: sentences
15
TREC definition pilot
 Long answer to request for a definition
 As a pilot, less emphasis on results
 Part of TREC QA
16
Evaluation Methods
 Pool system responses and break into nuggets
 A judge scores nuggets as vital, OK or invalid
 Measure information precision and recall
 Can a judge reliably determine which facts
belong in a definition?
17
Considerations Across Evaluations
 Independent evaluator
 Not always as knowledgeable as researchers
 Impartial determination of approach
 Extensive collection of resources
 Determination of task
 Appealing to a broad cross-section of community
 Changes over time





DUC 2001-2002 Single and multi-document
DUC 2003: headlines, multi-document
DUC 2004: headlines, multilingual and multi-document, focused
DUC 2005: focused summarization
DUC 2006: focused and a new task, up for discussion
 How long do participants have to prepare?
 When is a task dropped?
 Scoring of text at the sub-sentential level
18
Task-based Evaluation
 Use the summarization system as
browser to do another task
 Newsblaster: write a report given a broad
prompt
 DARPA utility evaluation: given a request for
information, use question answering to write
report
19
Task Evaluation
 Hypothesis: multi-document summaries
enable users to find information efficiently
 Task: fact-gathering given topic and questions
 Resembles intelligence analyst task
20
User Study: Objectives
 Does multi-document summarization help?
 Do summaries help the user find information needed to




perform a report writing task?
Do users use information from summaries in gathering
their facts?
Do summaries increase user satisfaction with the online
news system?
Do users create better quality reports with summaries?
How do full multi-document summaries compare with
minimal 1-sentence summaries such as Google News?
21
User Study: Design
 Compared 4 parallel news browsing systems
 Level 1: Source documents only
 Level 2: One sentence multi-document summaries (e.g.,
Google News) linked to documents
 Level 3: Newsblaster multi-document summaries linked
to documents
 Level 4: Human written multi-document summaries linked
to documents
 All groups write reports given four scenarios
 A task similar to analysts
 Can only use Newsblaster for research
 Time-restricted
22
User Study: Execution
 4 scenarios
 4 event clusters each
 2 directly relevant, 2 peripherally relevant
 Average 10 documents/cluster
 45 participants
 Balance between liberal arts, engineering
 138 reports
 Exit survey
 Multiple-choice and open-ended questions
 Usage tracking
 Each click logged, on or off-site
23
“Geneva” Prompt
 The conflict between Israel and the
Palestinians has been difficult for government
negotiators to settle. Most recently,
implementation of the “road map for peace”,
a diplomatic effort sponsored by ……
 Who participated in the negotiations that produced the
Geneva Accord?
 Apart from direct participants, who supported the Geneva
Accord preparations and how?
 What has the response been to the Geneva Accord by
the Palestinians?
24
Measuring Effectiveness
 Score report content and compare
across summary conditions
 Compare user satisfaction per summary
condition
 Comparing where subjects took report
content from
25
Newsblaster
26
User Satisfaction
 More effective than a web search with
Newsblaster
 Not true with documents only or single-sentence
summaries
 Easier to complete the task with summaries
than with documents only
 Enough time with summaries than documents
only
 Summaries helped most
 5% single sentence summaries
 24% Newsblaster summaries
 43% human summaries
27
User Study: Conclusions
 Summaries measurably improve a news
browser’s effectiveness for research
 Users are more satisfied with Newsblaster
summaries are better than single-sentence
summaries like those of Google News
 Users want search
 Not included in evaluation
28
Potential Problems
29
That very night in Max’s room a forest grew
30
And grew
31
And grew until the ceiling hung with vines and
the walls became the world all around
32
And an ocean tumbled by with a private boat for Max
and he sailed all through the night and day
33
And he sailed in and out of weeks and almost over a year
to where the wild things are
34
And when he came to where the wild things are they roared their
terrible roars and gnashed their terrible teeth
35
Comparing Text Against Text
 Which human summary makes a good gold
standard? Many summaries are good
 At what granularity is the comparison
made?
 When can we say that two pieces of text
match?
36
Measuring variation
 Types of variation between humans
Translation  same content
 different wording
Applications
Summarization  different content??
 different wording
Generation
 different content??
 different wording
37
Human variation: content
words (Ani Nenkova)
• Summaries differ in vocabulary
 Differences cannot be
explained by paraphrase
•7 translations
 20 documents
•7 summaries
 20 document sets
• Faster vocabulary growth
in summarization
38
Variation impacts evaluation
 Comparing content is hard
 All kinds of judgment calls
 Paraphrases
 VP vs. NP


Ministers have been exchanged
Reciprocal ministerial visits
 Length and constituent type


Robotics assists doctors in the medical operating theater
Surgeons started using robotic assistants
39
Nightmare: only one gold standard
 System may have chosen an equally good sentence
but not in the one gold standard
 Pinochet arrested in London on Oct 16 at a Spanish judge’s
request for atrocities against Spaniards in Chile.
 Former Chilean dictator Augusto Pinochet has been
arrested in London at the request of the Spanish government
 In DUC 2001 (one gold standard), human model had
significant impact on scores (McKeown et al)
 Five human summaries needed to avoid changes in
rank (Nenkova and Passonneau)
 DUC2003 data
 3 topic sets, 1 highest scoring and 2 lowest scoring
 10 model summaries
40
How many summaries are
enough?
41
Scoring
 Two main approaches used in DUC
 ROUGE (Lin and Hovy)
 Pyramids (Nenkova and Passonneau)
 Problems:
 Are the results stable?
 How difficult is it to do the scoring?
42
ROUGE: Recall-Oriented
Understudy for Gisting Evaluation
Rouge – Ngram co-occurrence
metrics measuring content overlap
Counts of n-gram overlaps between
candidate and model summaries
Total n-grams in summary model
43
ROUGE
 Experimentation with different units of comparison:
unigrams, bigrams, longest common substring, skipbigams, basic elements
 Automatic and thus easy to apply
 Important to consider confidence intervals when
determining differences between systems
 Scores falling within same interval not significantly different
 Rouge scores place systems into large groups: can be hard to
definitively say one is better than another
 Sometimes results unintuitive:
 Multilingual scores as high as English scores
 Use in speech summarization shows no discrimination
 Good for training regardless of intervals: can see trends
44
Pyramids




Uses multiple human summaries
Information is ranked by its importance
Allows for multiple good summaries
A pyramid is created from the human
summaries
 Elements of the pyramid are content units
 System summaries are scored by comparison
with the pyramid
45
Content units: better study of
variation than sentences
 Semantic units
 Link different surface realizations with
the same meaning
 Emerge from the comparison of several
texts
46
Content unit example
S1 Pinochet arrested in London on Oct 16
at a Spanish judge’s request for
atrocities against Spaniards in Chile.
S2 Former Chilean dictator Augusto
Pinochet has been arrested in London at
the request of the Spanish government.
S3 Britain caused international controversy
and Chilean turmoil by arresting former
Chilean dictator Pinochet in London.
47
SCU: A cable car caught fire
(Weight = 4)
A. The cause of the fire was unknown.
B. A cable car caught fire just after entering a
mountainside tunnel in an alpine resort in Kaprun,
Austria on the morning of November 11, 2000.
C. A cable car pulling skiers and snowboarders to the
Kitzsteinhorn resort, located 60 miles south of
Salzburg in the Austrian Alps, caught fire inside a
mountain tunnel, killing approximately 170 people.
D. On November 10, 2000, a cable car filled to capacity
caught on fire, trapping 180 passengers inside the
Kitzsteinhorn mountain, located in the town of
Kaprun, 50 miles south of Salzburg in the central
Austrian Alps.
48
SCU: The cause of the fire is
unknown (Weight = 1)
A. The cause of the fire was unknown.
B. A cable car caught fire just after entering a
mountainside tunnel in an alpine resort in Kaprun,
Austria on the morning of November 11, 2000.
C. A cable car pulling skiers and snowboarders to the
Kitzsteinhorn resort, located 60 miles south of
Salzburg in the Austrian Alps, caught fire inside a
mountain tunnel, killing approximately 170 people.
D. On November 10, 2000, a cable car filled to capacity
caught on fire, trapping 180 passengers inside the
Kitzsteinhorn mountain, located in the town of
Kaprun, 50 miles south of Salzburg in the central
Austrian Alps.
49
Idealized representation
 Tiers of differentially
W=3
weighted SCUs
 Top: few SCUs, high
weight
 Bottom: many
SCUs, low weight
W=2
W=1
50
Comparison of Scoring Methods
in DUC05
 Analysis of scores for the 20 pyramid sets
 Columbia prepared pyramids
 Participants scored systems against pyramids
 Comparisons between Pyramid
(original,modified), responsiveness, and
Rouge-SU4
 Pyramids score computed from multiple humans
 Responsiveness is just one human’s judgment
 Rouge-SU4 equivalent to Rouge-2
51
Creation of pyramids
 Done at Columbia for each of 20 out of 50 sets
 Primary annotator, secondary checker
 Held round-table discussions of problematic
constructions that occurred in this data set
 Comma separated lists

Extractive reserves have been formed for managed harvesting of
timber, rubber, Brazil nuts, and medical plants without
deforestation.
 General vs. specific

Eastern Europe vs. Hungary, Poland, Lithuania, and Turkey
52
Characteristics of the Responses
 Proportion of SCUs of Weight 1 is large
 44% (D324) to 81% (D695)
 Mean SCU weight: 1.9
Agreement among human responders is
quite low
53
# of SCUs
at each
weight
SCU Weights
54
Preview of Results
 Manual metrics
 Large differences between humans and machines
 No single system the clear winner
 But a top group identified by all metrics
 Significant differences
 Different predictions from manual and automatic metrics
 Correlations between metrics
 Some correlation but one cannot be substituted for another
 This is good
55
Human performance/Best sys
Pyramid
Modified
B: 0.5472
B: 0.4814
A: 0.4969
A: 0.4617
~~~~~~~~~~~~~~~~~
14: 0.2587
10: 0.2052
Resp
A: 4.895
B: 4.526
4: 2.85
ROUGE-SU4
A: 0.1722
B: 0.1552
15: 0.139
Best system ~50% of human performance on manual metrics
Best system ~80% of human performance on ROUGE
56
Pyramid
original
14: 0.2587
17: 0.2492
15: 0.2423
10: 0.2379
4: 0.2321
7: 0.2297
16: 0.2265
6: 0.2197
32: 0.2145
21: 0.2127
12: 0.2126
11: 0.2116
26: 0.2106
19: 0.2072
28: 0.2048
13: 0.1983
3: 0.1949
1: 0.1747
Modified
10: 0.2052
17: 0.1972
14: 0.1908
7: 0.1852
15: 0.1808
4: 0.177
16: 0.1722
11: 0.1703
6: 0.1671
12: 0.1664
19: 0.1636
21: 0.1613
32: 0.1601
26: 0.1464
3: 0.145
28: 0.1427
13: 0.1424
25: 0.1406
Resp
4: 2.85
14: 2.8
10: 2.65
15: 2.6
17: 2.55
11: 2.5
28: 2.45
21: 2.45
6: 2.4
24: 2.4
19: 2.4
6: 2.4
27: 2.35
12: 2.35
7: 2.3
25: 2.2
32: 2.15
3: 2.1
Rouge-SU4
15: 0.139
4: 0.134
17: 0.1346
19: 0.1275
11: 0.1259
10: 0.1278
6: 0.1239
7: 0.1213
14: 0.1264
25: 0.1188
21: 0.1183
16: 0.1218
24: 0.118
12: 0.116
3: 0.1198
28: 0.1203
27: 0.110
13: 0.1097
57
Pyramid
original
14: 0.2587
17: 0.2492
15: 0.2423
10: 0.2379
4: 0.2321
7: 0.2297
16: 0.2265
6: 0.2197
32: 0.2145
21: 0.2127
12: 0.2126
11: 0.2116
26: 0.2106
19: 0.2072
28: 0.2048
13: 0.1983
3: 0.1949
1: 0.1747
Modified
10: 0.2052
17: 0.1972
14: 0.1908
7: 0.1852
15: 0.1808
4: 0.177
16: 0.1722
11: 0.1703
6: 0.1671
12: 0.1664
19: 0.1636
21: 0.1613
32: 0.1601
26: 0.1464
3: 0.145
28: 0.1427
13: 0.1424
25: 0.1406
Resp
4: 2.85
14: 2.8
10: 2.65
15: 2.6
17: 2.55
11: 2.5
28: 2.45
21: 2.45
6: 2.4
24: 2.4
19: 2.4
6: 2.4
27: 2.35
12: 2.35
7: 2.3
25: 2.2
32: 2.15
3: 2.1
Rouge-SU4
15: 0.139
4: 0.134
17: 0.1346
19: 0.1275
11: 0.1259
10: 0.1278
6: 0.1239
7: 0.1213
14: 0.1264
25: 0.1188
21: 0.1183
16: 0.1218
24: 0.118
12: 0.116
3: 0.1198
28: 0.1203
27: 0.110
13: 0.1097
58
Pyramid
original
14: 0.2587
17: 0.2492
15: 0.2423
10: 0.2379
4: 0.2321
7: 0.2297
16: 0.2265
6: 0.2197
32: 0.2145
21: 0.2127
12: 0.2126
11: 0.2116
26: 0.2106
19: 0.2072
28: 0.2048
13: 0.1983
3: 0.1949
1: 0.1747
Modified
10: 0.2052
17: 0.1972
14: 0.1908
7: 0.1852
15: 0.1808
4: 0.177
16: 0.1722
11: 0.1703
6: 0.1671
12: 0.1664
19: 0.1636
21: 0.1613
32: 0.1601
26: 0.1464
3: 0.145
28: 0.1427
13: 0.1424
25: 0.1406
Resp
4: 2.85
14: 2.8
10: 2.65
15: 2.6
17: 2.55
11: 2.5
28: 2.45
21: 2.45
6: 2.4
24: 2.4
19: 2.4
6: 2.4
27: 2.35
12: 2.35
7: 2.3
25: 2.2
32: 2.15
3: 2.1
Rouge-SU4
15: 0.139
4: 0.134
17: 0.1346
19: 0.1275
11: 0.1259
10: 0.1278
6: 0.1239
7: 0.1213
14: 0.1264
25: 0.1188
21: 0.1183
16: 0.1218
24: 0.118
12: 0.116
3: 0.1198
28: 0.1203
27: 0.110
13: 0.1097
59
Pyramid
original
14: 0.2587
17: 0.2492
15: 0.2423
10: 0.2379
4: 0.2321
7: 0.2297
16: 0.2265
6: 0.2197
32: 0.2145
21: 0.2127
12: 0.2126
11: 0.2116
26: 0.2106
19: 0.2072
28: 0.2048
13: 0.1983
3: 0.1949
1: 0.1747
Modified
10: 0.2052
17: 0.1972
14: 0.1908
7: 0.1852
15: 0.1808
4: 0.177
16: 0.1722
11: 0.1703
6: 0.1671
12: 0.1664
19: 0.1636
21: 0.1613
32: 0.1601
26: 0.1464
3: 0.145
28: 0.1427
13: 0.1424
25: 0.1406
Resp
4: 2.85
14: 2.8
10: 2.65
15: 2.6
17: 2.55
11: 2.5
28: 2.45
21: 2.45
6: 2.4
24: 2.4
19: 2.4
6: 2.4
27: 2.35
12: 2.35
7: 2.3
25: 2.2
32: 2.15
3: 2.1
Rouge-SU4
15: 0.139
4: 0.134
17: 0.1346
19: 0.1275
11: 0.1259
10: 0.1278
6: 0.1239
7: 0.1213
14: 0.1264
25: 0.1188
21: 0.1183
16: 0.1218
24: 0.118
12: 0.116
3: 0.1198
28: 0.1203
27: 0.110
13: 0.1097
60
Significant Differences
 Manual metrics
 Few differences between systems
Pyramid: 23 is worse
 Responsive: 23 and 31 are worse

 Both humans better than all systems
 Automatic (Rouge-SU4)
 More differences between systems
 One human indistinguishable from 5 systems
61
Correlations: Pearson’s, 25
systems
Pyr-orig
Pyr-mod
Resp-1
Resp-2
R-2
Pyr-mod
Resp-1 Resp2
R-2
R-SU4
0.96
0.77
0.86
0.84
0.80
0.81
0.90
0.90
0.86
0.83
0.92
0.92
0.88
0.87
0.98
62
Correlations: Pearson’s, 25
systems
Pyr-orig
Pyr-mod
Resp-1
Resp-2
R-2
Pyr-mod
Resp-1 Resp2
R-2
R-SU4
0.96
0.77
0.86
0.84
0.80
0.81
0.90
0.90
0.86
0.83
0.92
0.92
0.88
0.87
0.98
Questionable that responsiveness could be a gold standard
63
Pyramid and responsiveness
Pyr-orig
Pyr-mod
Resp-1
Resp-2
R-2
Pyr-mod
Resp-1 Resp2
R-2
R-SU4
0.96
0.77
0.86
0.84
0.80
0.81
0.90
0.90
0.86
0.83
0.92
0.92
0.88
0.87
0.98
High correlation, but the metrics are not mutually substitutable
64
Pyramid and Rouge
Pyr-orig
Pyr-mod
Resp-1
Resp-2
R-2
Pyr-mod
Resp-1 Resp2
R-2
R-SU4
0.96
0.77
0.86
0.84
0.80
0.81
0.90
0.90
0.86
0.83
0.92
0.92
0.88
0.87
0.98
High correlation, but the metrics are not mutually substitutable
65
Correlations
 Original and modified can substitute for each
other
 High correlation between manual and
automatic, but automatic not yet a substitute
 Similar patterns between pyramid and
responsiveness
66
Nightmare
 Scoring metric that is not stable used to
decide funding
 Insignificant differences between
systems determine funding
67
Is Task Evaluation Nightmare
Free?
 Impact of user interface issues
 Can have more impact than the summary
 Controlling for proper mix of subjects
 Quantity of subjects and time to carry
out is large
68
Till Max said “Be still!” and tamed them with the magic trick
69
Of staring into their yellow eyes without blinking once
And they were frightened and called him the most wild thing of all
70
And made him king of all wild things
71
“And now,” cried Max “Let the wild rumpus start!”
72
73
74
75
Are we having fun yet?
Benefits of evaluation
 Emergence of evaluation methods
 ROUGE
 Pyramids
 Nuggetteer
 Research into characteristics of metrics
 Analyses of sub-sentential units
 Paraphrase as a research issue
76
Available Data
 DUC data sets
 4 years of summary/document set pairs

Multidocument summarization training data not available
beforehand
 4 years of scoring patterns
 Led to analysis of human summaries
 Pyramids




Pyramids and peers for 40 topics (DUC04, DUC05)
Many more from Nenkova and Passonneau
Training data for paraphrase
Training data for abstraction -> see systems moving
away from pure sentence extraction
77
Wrapping up
78
Lessons Learned
 Evaluation environment is important
 Find a task with broad appeal
 Use independent evaluator
 At least a committee
 Use multiple gold standards
 Compare text at the content unit level
 Evaluate the metrics
 Look at significant differences
79
Is Evaluation Worth It?
 DUC: creation of a community
 From ~15 participants year 1 -> 30 participants year 5
 No longer impacts funding
 Enables research into evaluation
 At start, no idea how to evaluate summaries
 But, results do not tell us everything
80
And he sailed back over a year, in and out of weeks and through a
day
81
And into the night of his very own room where he found his
supper waiting for him .. And it was still warm.
82