Annotating Topics of Opinions
Veselin Stoyanov
Claire Cardie
Talk Overview

Fine-grained sentiment analysis



Definitions
Examples
Opinion topic annotation




Definitions
Issues
Approach and Corpus
IA agreement
May 29, 2008
LREC 2008, Marakech, Morocco
Background

Sentiment Analysis:
Extraction and representation of attitudes, evaluations,
opinions, and sentiment in text.

Fine-grained Sentiment Analysis:
At the level of individual expressions of opinions.
May 29, 2008
LREC 2008, Marakech, Morocco
Fine-grained vs. Coarse-grained
Sentiment Analysis

Coarse-grained


May 29, 2008
Sentiment classification
Useful in the product
review domain
Review 1
Positive
Review 2
Negative

Fine-grained


Individual expressions of
opinions
Multiple opinions per
document (even sentence)
The
[SThe
Australian
Australian
press
press]
has has
launched
launched
a bitter
a bitter
attack
on
attack
Italy on
after
[TItaly]
seeing
after
their
seeing
beloved
their
Socceroos
beloved
eliminated
[TSocceroos]
on aeliminated
controversial
on alate
controversial
penalty. Italian
late
coach
[Tpenalty].
Lippi has
[S+Tbeen
Italian
blasted
coach for
Lippi]
his has
favorable
also been
comments
blasted fortoward
his favorable
the penalty.
comments toward [Tthe
penalty].
Lippi is preparing his side for the upcoming clash
with
LippiUkraine.
is preparing
He hailed
his side
10-man
for the
Italy's
upcoming clash
determination
with Ukraine. to
[SHe]
beathailed
Australia
10-man
and reiterated
[TItaly]'s that
the
determination
penalty was to
rightly
beat given.
Australia and reiterated
that the [Tpenalty] was rightly given.
LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
The Australian press has launched a bitter attack
on Italy.
May 29, 2008
LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
The Australian press has launched a bitter attack
on Italy.
Definitions





differ, but five main components:
Opinion trigger (opinion
words)
Source (opinion holder)
Polarity – positive/negative
Strength
Topic (target)
May 29, 2008
LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
The Australian press has launched a bitter attack
on Italy.
Definitions





differ, but five main components:
Opinion trigger (opinion
words)
Source (opinion holder)
Polarity – positive/negative
Strength
Topic (target)
May 29, 2008
launched a bitter attack
LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
[SThe Australian press] has launched a bitter attack
on Italy.
Definitions





differ, but five main components:
Opinion trigger (opinion
words)
Source (opinion holder)
Polarity – positive/negative
Strength
Topic (target)
May 29, 2008
launched a bitter attack
The Australian press
LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
[SThe Australian press] has launched a bitter attack
on Italy.
Definitions





differ, but five main components:
Opinion trigger (opinion
words)
Source (opinion holder)
Polarity – positive/negative
Strength
Topic (target)
May 29, 2008
launched a bitter attack
The Australian press
negative
LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
[SThe Australian press] has launched a bitter attack
on Italy.
Definitions





differ, but five main components:
Opinion trigger (opinion
words)
Source (opinion holder)
Polarity – positive/negative
Strength
Topic (target)
May 29, 2008
launched a bitter attack
The Australian press
negative
high
LREC 2008, Marakech, Morocco
Fine-grained opinions: Example
[SThe Australian press] has launched a bitter attack
on [TItaly]
Definitions





differ, but five main components:
Opinion trigger (opinion
words)
Source (opinion holder)
Polarity – positive/negative
Strength
Topic (target)
May 29, 2008
launched a bitter attack
The Australian press
negative
high
Italy
LREC 2008, Marakech, Morocco
Fine-grained opinions

Five components

Source (opinion holder)


Opinion trigger (opinion words)


As above
Strength


e.g. [Yu and Hatzivassiloglou, 2003] [Riloff and Wiebe, 2003]
Polarity – positive/negative


e.g. [Bethard et al., 2004] [Choi et al., 2005] [Kim and Hovy, 2006]
e.g. [Wilson et al. 2004]
Topic (target)

May 29, 2008
????
LREC 2008, Marakech, Morocco
Annotating Topics of Fine-grained
Opinions




Definitions
Issues
Approach and Corpus
IA agreement
May 29, 2008
LREC 2008, Marakech, Morocco
Examples
(1)[OH John] likes Marseille for its weather and cultural diversity.
(2)[OH Al] thinks that the government should tax gas more in
order to curb CO2 emissions.
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(1)[OH John] likes Marseille for its weather and cultural diversity.
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(1)[OH John] likes Marseille for its weather and cultural diversity.
Topic: city of Marseille

Topic - the real-world object, event or abstract entity
that is the subject of the opinion as intended by the
opinion holder
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(1)[OH John] likes [TOPIC SPAN Marseille] for its weather and cultural
diversity.
Topic: city of Marseille

Topic - the real-world object, event or abstract entity
that is the subject of the opinion as intended by the
opinion holder

Topic span - the closest, minimal span of text that
mentions the topic
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(1)[OH John] likes [TARGET+TOPIC SPAN Marseille] for its weather and
cultural diversity.
Topic: city of Marseille

Topic - the real-world object, event or abstract entity
that is the subject of the opinion as intended by the
opinion holder

Topic span - the closest, minimal span of text that
mentions the topic

Target span - the span of text that covers the syntactic
surface form comprising the contents of the opinion
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that the government should tax gas
more in order to curb CO2 emissions.
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that [TARGET SPAN the government should
tax gas more in order to curb CO2 emissions].
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the
government] should [TOPIC SPAN? tax gas] more in
order to [TOPIC SPAN? curb [TOPIC SPAN? CO2
emissions]]].
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that [TARGET SPAN the government should
tax gas more in order to curb CO2 emissions].
Context:
(3) Although he doesn’t like government imposed taxes, he
thinks that a fuel tax is the only effective solution.
May 29, 2008
LREC 2008, Marakech, Morocco
Definitions
(2)[OH Al] thinks that [TARGET SPAN the government should
[TOPIC SPAN tax gas] more in order to curb CO2
emissions].
Context:
(3) Although he doesn’t like government imposed taxes, he
thinks that a fuel tax is the only effective solution.
May 29, 2008
LREC 2008, Marakech, Morocco
Related Work

Product reviews
E.g. Kobayashi et al. (2004), Yi et al. (2003), Popescu and
Etzioni (2005), Hu and Liu (2004
 Limit “topics” to mentions of product names, components, and
their attributes
 Lexicon look-up
 Focused on methods for lexicon acquisition
MPQA corpus (Wiebe, Wilson, Cardie, 2004)
 Fine-grained opinions
 Topic annotation deemed too difficult
 Target span annotation is underway



Kim & Hovy (2006)


Target span extraction using semantic frames
Limited evaluation
May 29, 2008
LREC 2008, Marakech, Morocco
Issues in Opinion Topic
Identification

Multiple potential topics mentioned within a single target span
(2)[OH Al] thinks that [TARGET SPAN [TOPIC SPAN? the government] should [TOPIC
SPAN? tax gas] more in order to [TOPIC SPAN? curb [TOPIC SPAN? CO2
emissions]]].

Requires context
Topic of an opinion is the entity that comprises the main
information goal of the opinion based on the discourse
context.
May 29, 2008
LREC 2008, Marakech, Morocco
Issues in Opinion Topic
Identification

Opinion topics are not always explicitly
mentioned
(4) [OH John] believes the violation of Palestinian
human rights is one of the main factors.
Topic: ISRAELI-PALESTINIAN CONFLICT
(5) [OH I] disagree entirely!
May 29, 2008
LREC 2008, Marakech, Morocco
A Coreference Approach

Hypothesize that the notion of topic
coreference will facilitate identification of
opinion topics
Two opinions are topic-coreferent if they share
the same opinion topic.

Easier than specifying the topic of each
opinion in isolation
May 29, 2008
LREC 2008, Marakech, Morocco
Opinion Topic Corpus
Build on the MPQA corpus:




(www.cs.pitt.edu/mpqa)
535 Documents manually annotated for finegrained opinions
No opinion topic annotation
Our goal: Add the opinion topic information on
top of the existing opinion annotations
Created and used a GUI
May 29, 2008
LREC 2008, Marakech, Morocco
Annotation Process
List of opinions
to be processed
May 29, 2008
Set of current
clusters
LREC 2008, Marakech, Morocco
Document text
Annotation Process
May 29, 2008
LREC 2008, Marakech, Morocco
Annotation Process
fuel tax
May 29, 2008
LREC 2008, Marakech, Morocco
Interannotator Agreement

Annotator 1


Annotator 2


150 of the 535 MPQA documents
20 of these 150
IAG measures from noun phrase coreference resolution
B3
a
CEAF
all opinions
.64
.55
.69
sentiment-bearing
opinions
.72
.73
.80
strong opinions
.74
.77
.82
May 29, 2008
LREC 2008, Marakech, Morocco
Interannotator Agreement

Annotator 1


Annotator 2


150 of the 535 MPQA documents
20 of these 150
IAG measures from noun phrase coreference resolution
B3
a
CEAF
all opinions
.64
.55
.69
sentiment-bearing
opinions
.72
.73
.80
strong opinions
.74
.77
.82
May 29, 2008
LREC 2008, Marakech, Morocco
Baselines

all-in-one


1 opinion per cluster


assigns all opinions to the same cluster
assigns each opinion to its own cluster
same paragraph

opinions in the same paragraph are assigned to
the same cluster
May 29, 2008
LREC 2008, Marakech, Morocco
Results


Baselines
B3
a
CEAF
all-in-one
.37
-.10
.30
1 opinion per cluster
.29
.22
.27
same paragraph
.55
.31
.50
vs. Interannotator agreement
all opinions
.64
.55
.69
sentiment-bearing
.72
.73
.80
strong opinions
.74
.77
.82
May 29, 2008
LREC 2008, Marakech, Morocco
Thank you
Questions?
Annotation instructions + more information available at:
www.cs.cornell.edu/~ves
May 29, 2008
LREC 2008, Marakech, Morocco
Example
The Australian press has launched a bitter attack
on Italy after seeing their beloved Socceroos
eliminated on a controversial late penalty. Italian
coach Lippi has been blasted for his favorable
comments toward the penalty.
Lippi is preparing his side for the upcoming clash
with Ukraine. He hailed 10-man Italy's
determination to beat Australia and reiterated that
the penalty was rightly given.
May 29, 2008
LREC 2008, Marakech, Morocco
Example – fine-grained opinions
[SThe Australian press] has launched a bitter
attack on [TItaly] after seeing [Stheir] beloved
[TSocceroos] eliminated on a controversial late
[Tpenalty]. [S+TItalian coach Lippi] has also been
blasted for his favorable comments toward [Tthe
penalty].
Lippi is preparing his side for the upcoming clash
with Ukraine. [SHe] hailed 10-man [TItaly]'s
determination to beat Australia and reiterated that
[Tthe penalty] was rightly given.
May 29, 2008
LREC 2008, Marakech, Morocco
Motivation

Sentiment analysis: Useful as stand-alone
application




Product reviews
Tracking opinions in the press
Flame detection, etc.
Opinion information can benefit many NLP
applications

Multi-Perspective Question Answering
[Stoyanov, Cardie, Litman and Wiebe. AAAI WS 2004] and
[Stoyanov, Cardie and Wiebe. HLT-EMNLP 2005]


Opinion-Oriented Information Retrieval
Clustering, etc.
May 29, 2008
LREC 2008, Marakech, Morocco
Annotation Process
May 29, 2008
LREC 2008, Marakech, Morocco
Annotation Process
May 29, 2008
LREC 2008, Marakech, Morocco
`
May 29, 2008
LREC 2008, Marakech, Morocco