Penn
Hierarchical Sense Distinctions
Martha Palmer
University of Pennsylvania
with Olga Babko-Malaya, Nianwen Xue,
and Ben Snyder
July 25, 2004
Senseval3 Workshop – ACL-04
1
Outline





Penn
Granularity of sense distinctions
Propbanks
Hierarchical sense distinctions
Lessons learned
Moving forward
2
WordNet – Princeton
(Miller 1985, Fellbaum 1998)
Penn
On-line lexical reference (dictionary)
 Nouns, verbs, adjectives, and adverbs grouped
into synonym sets
 Other relations include hypernyms (ISA),
antonyms, meronyms
 Typical top nodes - 5 out of 25
 (act, action, activity)
 (animal, fauna)
 (artifact)
 (attribute, property)
 (body, corpus)
3
WordNet – Princeton
Penn
(Miller 1985, Fellbaum 1998)
 Limitations as a computational lexicon
 Contains little syntactic information
 Comlex has syntax but no sense distinctions
 No explicit lists of predicate arguments
 Sense distinctions very fine-grained,
 Definitions often vague
 Causes problems with creating training data for
supervised Machine Learning – SENSEVAL2




29 Verbs > 16 senses (including call)
Inter-annotator Agreement ITA 73%,
Automatic Word Sense Disambiguation, WSD 60.2%
Slow annotation speed – 60 tokens per hour
Dang & Palmer, SIGLEX02
4
WordNet – call, 28 senses
Penn
1. name, call -- (assign a specified, proper name to;
"They named their son David"; …)
-> LABEL
2. call, telephone, call up, phone, ring -- (get or try to get into
communication (with someone) by telephone;
"I tried to call you all night"; …)
->TELECOMMUNICATE
3. call -- (ascribe a quality to or give a name of a common
noun that reflects a quality;
"He called me a bastard"; …)
-> LABEL
4. call, send for -- (order, request, or command to come;
"She was called into the director's office"; "Call the police!")
-> ORDER
5
WordNet: - call, 28 senses
WN2 , WN13,WN28
WN3
WN19
Penn
WN15 WN26
WN4 WN 7 WN8 WN9
WN1 WN22
WN20
WN25
WN18 WN27
WN5 WN 16
WN6
WN23
WN12
WN17 , WN 11
WN10, WN14, WN21, WN24
6
WordNet: - call, 28 senses,
Senseval2 groups (engineering!)
WN5, WN16,WN12
Loud cry
WN3
WN19
WN1 WN22
Label
WN15 WN26
Bird or animal cry
WN4 WN 7 WN8 WN9
Request
WN20
WN18 WN27
Challenge
WN2 WN 13
Phone/radioWN28
WN17 , WN 11
Penn
WN6
WN25
Call a loan/bond
WN23
Visit
WN10, WN14, WN21, WN24,
Bid
7
Grouping improved scores:
ITA 82%, MaxEnt WSD 69%
Penn
 Call: 31% of errors due to confusion between senses
within same group 1:
 name, call -- (assign a specified, proper name to; They named
their son David)
 call -- (ascribe a quality to or give a name of a common noun
that reflects a quality; He called me a bastard)
 call -- (consider or regard as being;I would not call her beautiful)
 75% with training and testing on grouped senses vs.
 43% with training and testing on fine-grained senses
Palmer, Dang, Fellbaum,, submitted, NLE
8
Proposition Bank:
From Sentences to Propositions (Predicates!)
Powell met Zhu Rongji
Penn
battle
wrestle
join
debate
Powell and Zhu Rongji met
Powell met with Zhu Rongji
Powell and Zhu Rongji had
a meeting
consult
Proposition: meet(Powell, Zhu Rongji)
meet(Somebody1, Somebody2)
...
When Powell met Zhu Rongji on Thursday they discussed the return of the spy plane.
meet(Powell, Zhu)
discuss([Powell, Zhu], return(X, plane))
9
A TreeBanked phrase
1M words WSJ – Penn TreeBank II
Penn
A GM-Jaguar pact would give the
U.S. car maker an eventual 30% stake
in the British company.
S
NP
a GM-Jaguar
would
pact
VP
VP
give
NP
NP
the US car
an eventual
maker
30% stake
NP
PP-LOC
in
NP
the British
company
10
The same phrase, PropBanked
Same data – released, March’04
Penn
A GM-Jaguar pact would give the U.S.
car maker an eventual 30% stake in the
British company.
Arg0
a GM-Jaguar
pact
would give
Arg2
Arg1
an eventual 30% stake in the
British company
the US car
maker
give(GM-J pact, US car maker, 30% stake)
11
Frames File example: give
< 4000 Frames for PropBank
Penn
Roles:
Arg0: giver
Arg1: thing given
Arg2: entity given to
Example:
double object
The executives gave the chefs a standing ovation.
Arg0:
The executives
REL:
gave
Arg2:
the chefs
Arg1:
a standing ovation
12
Frames File example: give
Penn
Roles:
Arg0: giver
Arg1: thing given
Arg2: entity given to
Example:
double object
The executives gave the chefs a standing ovation.
Arg0: Agent
The executives
REL:
gave
Arg2: Recipient the chefs
Arg1: Theme
a standing ovation
VerbNet – based on Levin classes
13
Word Senses in PropBank
Penn
 Orders to ignore word sense not feasible for 700+
verbs
 Mary left the room
 Mary left her daughter-in-law her pearls in her will
Frameset leave.01 "move away from":
Arg0: entity leaving
Arg1: place left
Frameset leave.02 "give":
Arg0: giver
Arg1: thing given
Arg2: beneficiary
How do these relate to traditional word senses in WordNet?
14
WordNet: - call, 28 senses, groups
WN5, WN16,WN12
Loud cry
WN3
WN19
WN1 WN22
Label
WN15 WN26
Bird or animal cry
WN4 WN 7 WN8 WN9
Request
WN20
WN18 WN27
Challenge
WN2 WN 13
Phone/radioWN28
WN17 , WN 11
Penn
WN6
WN25
Call a loan/bond
WN23
Visit
WN10, WN14, WN21, WN24,
Bid
15
Overlap with PropBank Framesets
WN5, WN16,WN12
Loud cry
WN3
WN19
WN1 WN22
Label
WN15 WN26
Bird or animal cry
WN4 WN 7 WN8 WN9
Request
WN20
WN18 WN27
Challenge
WN2 WN 13
Phone/radioWN28
WN17 , WN 11
Penn
WN6
WN25
Call a loan/bond
WN23
Visit
WN10, WN14, WN21, WN24,
Bid
16
Overlap between Senseval2
Groups and Framesets – 95%
Penn
Frameset2
Frameset1
WN1 WN2
WN3 WN4
WN6 WN7 WN8
WN11 WN12 WN13
WN19
WN5 WN 9 WN10
WN 14
WN20
develop
17
Sense Hierarchy
(Palmer, et al, SNLU04 - NAACL04)
Penn
 PropBank Framesets – ITA >90%
coarse grained distinctions
20 Senseval2 verbs w/ > 1 Frameset
Maxent WSD system, 73.5% baseline, 90% accuracy
 Sense Groups (Senseval-2) - ITA 82%
Intermediate level
(includes Levin classes) – 69%
 WordNet – ITA 71%
fine grained distinctions, 60.2%
18
Summary of English/Chinese TreeBanks,
Penn
PropBanks
Monolingual
Corpora English
Text
1M WSJ
Treebank
90’s
PropBank
March ‘04
Parallel
Corpora
Text
Treebank
Oct, 04
PropBank
Dec’04
Chinese
Treebank
Chinese 500K
English 400K
Chinese 500K
English 100K
Chinese 250K
English 100K
19
Lessons Learned
Penn
 Desiderata
Balanced corpora with high quality annotation
 Issues
Annotation process requires speed and accuracy
What is “semantic” accuracy? ITA?
 Compromises that enable speed and ITA
PropBank requires frames files, Arg descriptions
Sense tagging requires coarser granularity
Are they useful? Are they useful enough?
20
Penn
PropBank I I
Event variables;
sense tags;
nominal reference;
discourse connectives
{Also,} [Arg0substantially lower Dutch corporate tax rates]
helped [Arg1[Arg0 the company] keep [Arg1 its tax outlay]
[Arg3-PRD flat] [ArgM-ADV relative to earnings growth]].
ID#
REL
Arg0
Arg1
h23
help
tax rates
help2,5 tax rate1
k16
keep
the company its tax outlay
keep1 company1
Arg3PRD
ArgM-ADV
flat
relative to
earnings…
the company
keep its tax
outlay flat
21