Text Summarization
Intro to NLP - ETHZ - 13/05/2013
Summary
●
Introduction to text summarization
●
Techniques for content selection
○
Rhetorical parsing
○
Unsupervised techniques
○
Supervised techniques
●
Other topics related to summarization:
○
Redundancy removal
○
Information ordering
○
Query-focused
○
Sentence compression
●
Evaluation
J & M, chapter 23
Introduction
Automatic Text Summarization
"Distilling the most important information
from a text to produce an abridged version
for a particular task and user."
(Mani and Maybury, 1999)
Text Summarization
Very common in real life:
● Document outlines.
● Abstracts of scientific articles.
● Headlines of news articles.
● Web search result snippets.
● Meeting action items.
● Email/discussion thread summaries.
● Compressed sentences.
● Answers to complex questions.
Web search snippets
Web search snippets
Scientific abstracts
News headlines, outline
Answers to complex questions
Automatic Text Summarization
Main types:
● Extractive/Abstractive
● Single/multi-document.
● Length: Headline vs. highlights vs. longer text.
● Focus: Generic vs. Query-focused vs. Opinion-focused.
● Update / chronology generation.
● etc.
Extracts vs. Abstracts
Abraham Lincoln's Gettysburg Address (after
the battle of Gettysburg)
Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in
liberty, and dedicated to the proposition that all men are created equal.
Now we are engaged in a great civil war, testing whether that nation, or any nation, so conceived and
so dedicated, can long endure. We are met on a great battle-field of that war. We have come to
dedicate a portion of that field, as a final resting place for those who here gave their lives that that
nation might live. It is altogether fitting and proper that we should do this.
But, in a larger sense, we can not dedicate, we can not consecrate, we can not hallow this ground.
The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to
add or detract. The world will little note, nor long remember what we say here, but it can never forget
what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they
who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great
task remaining before us—that from these honored dead we take increased devotion to that cause for
which they gave the last full measure of devotion—that we here highly resolve that these dead shall
not have died in vain—that this nation, under God, shall have a new birth of freedom—and that
government of the people, by the people, for the people, shall not perish from the earth.
Extracts vs. Abstracts
Abraham Lincoln's Gettysburg Address (after
the battle of Gettysburg)
Sentence extraction
Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in
liberty, and dedicated to the proposition that all men are created equal.
Now we are engaged in a great civil war, testing whether that nation, or any nation, so conceived and
so dedicated, can long endure. We are met on a great battle-field of that war. We have come to
dedicate a portion of that field, as a final resting place for those who here gave their lives that that
nation might live. It is altogether fitting and proper that we should do this.
But, in a larger sense, we can not dedicate, we can not consecrate, we can not hallow this ground.
The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to
add or detract. The world will little note, nor long remember what we say here, but it can never forget
what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they
who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great
task remaining before us—that from these honored dead we take increased devotion to that cause for
which they gave the last full measure of devotion—that we here highly resolve that these dead shall
not have died in vain—that this nation, under God, shall have a new birth of freedom—and that
government of the people, by the people, for the people, shall not perish from the earth.
Extracts vs. Abstracts
Abraham Lincoln's Gettysburg Address (after
the battle of Gettysburg)
Sentence extraction
Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in
liberty, and dedicated to the proposition that all men are created equal.
Now we are engaged in a great civil war, testing whether that nation, or any nation, so conceived and
so dedicated, can long endure. We are met on a great battle-field of that war. We have come to
dedicate a portion of that field, as a final resting place for those who here gave their lives that that
nation might live. It is altogether fitting and proper that we should do this.
But, in a larger sense, we can not dedicate, we can not consecrate, we can not hallow this ground.
The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to
add or detract. The world will little note, nor long remember what we say here, but it can never forget
what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they
Sentence compression
who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great
task remaining before us—that from these honored dead we take increased devotion to that cause for
which they gave the last full measure of devotion—that we here highly resolve that these dead shall
not have died in vain—that this nation, under God, shall have a new birth of freedom—and that
government of the people, by the people, for the people, shall not perish from the earth.
Extracts vs. Abstracts
Abraham Lincoln's Gettysburg Address (after
the battle of Gettysburg)
This speech by Abraham Lincoln commemorates soldiers who laid down their lives in the Battle of
Gettysburg.
It reminds the troops that it is the future of freedom in America that they are fighting for.
Single- vs. Multi-document
Single-document:
● Summarization of one, isolated document.
○ Abstracts of scientific articles.
○ News headline/highlights generation.
○ Meeting notes summarization.
○ Fiction summarization.
Multi-document:
● Summarization of a set of documents.
○
○
○
News-cluster summarization.
Answers to complex questions.
Combined UIs for full search-results page.
Length
Headline: short, concise. Usually hard to do by
sentence extraction alone:
Spain's indignados return to the streets amid fears of crackdown.
Highlights: one or several sentences summarizing the
main events. No need for discourse coherence
between them, but they must be self-contained.
Protesters plan four-day campaign to mark the anniversary of
Madrid's occupy movement.
Generic summary: at least one paragraph describing
the most important points in the news.
Spain's indignados plan four-day campaign to mark the
anniversary of Madrid's occupy movement. Several thousand
peopl were taking part. There were similar demonstrations in
Barcelona and other cities around the country.
Length
Headline:
New York State considers banning teens from
tanning.
Highlights:
●
●
The debate continues over whether or not
NY should ban tanning for teens under 18.
Several health groups began urging NY
lawmakers to approve the ban earlier this
week.
Scientific data-focused summary:
Studies have found that ultraviolet radiation from indoor
tanning beds increases a person's risk of developing
melanoma, a deadly form of skin cancer, by 75 percent.
Another report found that indoor tanning is common
among young adults, with the highest rates of indoor
tanning among white women aged 18-21 years and 2225 years.
About half of people, ages 18-29, reported at least one
sunburn in the past year.
Update Summarization
Generic summary:
Chimpanzee Santino achieved international fame in
2009 for his habit of gathering stones and
manufacturing concrete projectiles to throw at zoo
visitors.
The researchers now conclude that Santino deliberately
engaged in deceptive concealment of the stones, and
that this was a new, innovative behavior on his part.
Update summary:
A new study shows that Chimpanzee Santino 's
innovativeness when he plans his stone-throwing is
greater than researchers have previously observed.
He not only gathers stones and manufactures
projectiles in advance; he also finds innovative ways of
fooling the visitors. Before 2010, Santino had never put
stones under hay piles or behind logs.
The study, which was carried out at Lund University,
has been published in PLoS One.
Summarization Architecture
Three main components:
1. Content selection: what to say? Usually sentences or
clauses are taken as the basic unit to select from the
original documents.
2. Information ordering: in which order and with what
structure?
3. Sentence resolution: how to say it? Make sure that the
final sentence is grammatical, coherent and cohesive.
Summarization Architecture
Summarization based on
Rhetorical parsing
Summarization based on
Rhetorical Parsing
Rhetorical Structure Theory (RST, Mann and Thompson,
1987) is a model of text organization based on 23 rhetorical
relations.
Most relations hold between two text spans:
- The nucleus, which is central to the writer's purpose.
- The satellite, generally only interpretable with respect to
the nucleus.
Summarization based on
Rhetorical Parsing
Evidence: the stellite presents evidence for the proposition
expressed in the nucleus.
[N Kevin must be here]. [S His car is parked outside].
Elaboration: the satellite gives further information about the
content of the nucleus.
[N The company wouldn't elaborate],
[S citing competitive reasons]
Summarization based on
Rhetorical Parsing
Attribution: the satellite gives the source of attribution for
the proposition stated in the nucleus.
[S Analysts estimated] [N that sales declined this quarter,
too].
Contrast: two or more nuclei are contrasted across a
particular dimension.
[N One was in a bad temper], [N but the other was happy.]
Summarization based on
Rhetorical Parsing
List: a series of nuclei are given without contrast or
comparison.
[N Billy Bones was the mate;]
[N Long John, he was quartermaster].
Concession: the satellite is apparently inconsistent with
the nucleus but also stated by the author.
[S Although he wasn't happy], [N he smiled.]
Summarization based on
Rhetorical Parsing
Backgroud: the satellite gives context for interpreting the
nucleus.
[S T is the pointer to the root of a binary tree.]
[N Initialize T].
Condition: Conditioning situation for the nucleus.
[S If you come home early], [N we can go to the cinema.]
Summarization based on
Rhetorical Parsing
Summarization based on
Rhetorical Parsing
RST-based summarization:
●
Parse the input text with a discourse parser, e.g. http://www.isi.
edu/~marcu/discourse/AnnotationSoftware.html
●
●
Use the intuition that nuclei are more important than
satellites.
Example from the previous tree:
2 > 8 > 3 > 1,4,5,7 > 6
(2) Mars experiences frigid weather conditions.
Summarization based on
Rhetorical Parsing
RST-based summarization:
●
Parse the input text with a discourse parser, e.g. http://www.isi.
edu/~marcu/discourse/AnnotationSoftware.html
●
●
Use the intuition that nuclei are more important than
satellites.
Example from the previous tree:
2 > 8 > 3 > 1,4,5,7 > 6
(2) Mars experiences frigid weather conditions.
(8) Most martian weather involves blowing dust or carbon dioxide.
Summarization based on
Rhetorical Parsing
RST-based summarization:
●
Parse the input text with a discourse parser, e.g. http://www.isi.
edu/~marcu/discourse/AnnotationSoftware.html
●
●
Use the intuition that nuclei are more important than
satellites.
Example from the previous tree:
2 > 8 > 3 > 1,4,5,7 > 6
(2) Mars experiences frigid weather conditions.
(3) Surface temperature typically average about -60 degrees Celsuis at the
equator and can dip to -123 degrees C near the poles.
(8) Most martian weather involves blowing dust or carbon dioxide.
Unsupervised content
selection
Unsupervised content selection
A simple intuition is that sentences that contain the most
central words in a document (or collection) will be more
important.
A common strategy for content selection is:
●
●
Build a model of important words from a document or collection.
Select the sentences with most important words.
Unsupervised content selection
Common strategies to measure word importance:
●
tf-idf weighting: weight(wi) = tfi * idfi
●
Log-likelihood ratio: ratio between the likelihood of seeing a word in the
document w.r.t. in a background corpus.
LDA and topic models.
●
Once the most important words (and their weights) have
been selected, we have a vector representing the central
content in the document.
The centrality score of a sentence can be measured as the
cosine between the sentence frequency vector.
Supervised content
selection
Supervised summarization
Sometimes training data is available (e.g. hand-picked sentences as headlines
or news highlights). In this case, a supervised classifier can be trained. Some
settings:
●
Binary classification: classify sentences (or clauses) depending on
whether they were selected or not for the summary.
●
Sentence ranking: select fragments of a sentence to rate, e.g. unigrams
or bigrams. Score a sentence based on fragment-overlap with manual
summaries. Intuitively, a sentence with more bigrams in common with a
reference summary should be ranked higher by the classifier.
Supervised summarization - Features
Other topics
Redundancy removal
In multiple-document summarization, there is the possibility of selecting similar
content from different documents.
The purpose of redundancy removal is to avoid this from happening.
A simple, common method of redundancy removal is Maximal Marginal
Relevance (MMR, Carbonell and Goldstein, 1998). When ranking sentences by
importance:
MMR(si) = lambda * (weight(si) - (1 - lambda) maxsj selected sim(si, sj))
Hence, sentences that are similar to already-selected sentences will be
penalized.
Information ordering
Strategies for sentence ordering:
●
Chronological: requires all sentences with dates, produces summaries
with low coherence.
●
Lexical cohesion: sentences with high overlap of related words (see
lecture on word similarity/relatedness) should appear close to each other.
●
Centering theory: identify the entity of interest in the text, make sure it is
introduced in sentence position. Use linguistic knowledge of transitions
between sentences (e.g. entity going from subject to object position in
consecutive sentences).
Query-focused summarization
In query-focused summarization, the summary is written in response of a user
query.
The main difference of query-focused summaries with respect to generic
summaries is the inclusion of a query-dependent relevance score used when
performing content selection.
A combination of the centrality scores already described and this relevance
score will be used in ranking.
This also fits nicely the supervised settings, where we have different features
and a training set.
Sentence compression
Sentence compression consists of removing unimportant parts of sentences.
It usually starts from the parse tree (or dependency tree) of the sentence, and
selectively removes constituents.
Sentence compression - Example
"Greek radical left leader Alexis Tsipras says he will not join or support a probailout coalition government, saying he cannot agree to what he terms a
mistake."
Sentence compression - Example
"Greek radical left leader Alexis Tsipras says he will not join or support a probailout coalition government, saying he cannot agree to what he terms a
mistake."
Sentence compression - Example
"Greek radical left leader Alexis Tsipras says he will not join or support a probailout coalition government, saying he cannot agree to what he terms a
mistake."
"... Alexis Tsipras says he will not join or support a ... coalition government ..."
Sentence compression
Two approaches:
●
●
Hand-crafted rules (this is simple enough for the course project!):
○
Remove appositives
○
Remove attribution clauses ("XXX said")
○
Remove PPs without named entities
○
Remove initial adverbials ("For example", "On the other hand", ...)
Machine learning:
○
Use a corpus of compressed sentences.
○
Learn the likelihood that a given word can be removed some particular
modifier in the dependency parse.
○
Generate the compression that is smaller than the target length and
keeps only the most important constituents.
Intrinsic evaluation
of text summarization
Manual evaluations
A simple option is to rate the summaries according to some dimentions, e.g.
●
Grammaticality
●
Coherence
●
Informativeness
●
Task dependent metrics, e.g. relevance to user's query
Manual evaluations - Pyramid
Pyramid evaluation of summarization:
●
Generate N summaries manually.
●
Identify Summary Content Units (SCU): semantic units mentioned in the
manual summaries.
●
Weight each SCU with the number of manual summaries mentioning them.
●
Manually identify which SCUs appear in each automatic summary.
Manual evaluations - Pyramid
Example with three manual summaries:
Greek politicians are debating a new coalition government. Failure might lead to Greek exit from
the euro zone, which would not be fatal blow, ECB policy-maker says. EU sees euro zone
contraction in 2012.
Greek politicians are in talks to agree on a new governement. Mr. Tsipras, leader of left Siriza
party, said he will not join a coalition government, and this may lead to new elections.
Greek politicians are in talks to agree on a new governement. Mr. Tsipras said he does not
agree.
Manual evaluations - Pyramid
Example with three manual summaries:
Greek politicians are debating a new coalition government. Failure might lead to Greek exit from
the euro zone, which would not be fatal blow, ECB policy-maker says. EU sees euro zone
contraction in 2012.
Greek politicians are in talks to agree on a new governement. Mr. Tsipras, leader of left Siriza
party, said he will not join a coalition government, and this may lead to new elections.
Greek politicians are in talks to agree on a new governement. Mr. Tsipras said he does not
agree.
SCU1 = Greek politicians in talks about the new government (weight = 3)
Manual evaluations - Pyramid
Example with three manual summaries:
Greek politicians are debating a new coalition government. Failure might lead to Greek exit from
the euro zone, which would not be fatal blow, ECB policy-maker says. EU sees euro zone
contraction in 2012.
Greek politicians are in talks to agree on a new governement. Mr. Tsipras, leader of left Siriza
party, said he will not join a coalition government, and this may lead lead to new elections.
Greek politicians are in talks to agree on a new governement. Mr. Tsipras said he does not
agree.
SCU1 = Greek politicians in talks about the new government (weight = 3)
SCU2 = Mr. Tsipras does not agree (weight = 2)
Manual evaluations - Pyramid
Example with three manual summaries:
Greek politicians are debating a new coalition government. Failure might lead to Greek exit from
the euro zone, which would not be fatal blow, ECB policy-maker says. EU sees euro zone
contraction in 2012.
Greek politicians are in talks to agree on a new governement. Mr. Tsipras, leader of left Siriza
party, said he will not join a coalition government, and this may lead lead to new elections.
Greek politicians are in talks to agree on a new governement. Mr. Tsipras said he does not
agree.
SCU1 = Greek politicians in talks about the new government (weight = 3)
SCU2 = Mr. Tsipras does not agree (weight = 2)
Other SCUs with weight 1.
ROUGE
ROUGE measures n-gram recall between an automatic summary and one or
several manual summaries. For example, ROUGE-2 calculates, using bigrams:
sum_{all references} sum_{all bigrams} count_match(bigram)
-------------------------------------------------------------------------------------sum_{all references} sum_{all bigrams} count(bigram)
ROUGE_L measures the longest common subsequences.
ROUGE-S and ROUGE-SU measures skip-bigrams (two words in the same
order but with possible other words in between)
Recap
●
Introduction to text summarization
●
Techniques for content selection
○
Rhetorical parsing
○
Unsupervised techniques
○
Supervised techniques
●
Other topics related to summarization:
○
Redundancy removal
○
Information ordering
○
Query-focused
○
Sentence compression
●
Evaluation
J & M, chapter 23
Project
●
Make sure to enter the ROUGE scores in the spreadsheet.
●
You can prepare one or two slides for next class (May 27). We will discuss
the results and what you did, and will try to draw conclusions on what
worked and what did not work.
●
Any project-related questions?