Generalized Inference with
Multiple Semantic Role Labeling Systems
Peter Koomen, Vasin Punyakanok, Dan Roth, (Scott) Wen-tau Yih
Department of Computer Science
University of Illinois at Urbana-Champaign
Page 1
Outline

System Architecture





Pruning
Argument Identification
Argument Classification
Inference
[main difference from other systems]
Inference with Multiple Systems

The same approach used by the SRL to assure a coherent
output is used with input produced by multiple systems.
Page 2
System Architecture

Identify argument candidates


Pruning
Argument Identifier


Binary classification
Classify argument candidates

Argument Classifier


Multi-class classification
Inference



Use the estimated probability distribution given
by the argument classifier, and
Expressive structural and linguistic constraints.
Infer the optimal global output – modeled as a
constrained optimization problem
Page 3
Pruning [Xue&Palmer 2004]
Devel
Prec
Rec
F1
Gold
30.19
96.57
46.00
Charniak
26.61
85.47
40.59

Significant errors due to PP attachment

Consider PP as attached to both NP and VP
Page 4
Modified Pruning
Devel
Prec
Rec
F1
Gold
30.19
96.57
46.00
Charniak
26.61
85.47
40.59
Charniak
Modified heuristic
23.31
87.59
36.83
Page 5
Argument Identification


Argument identifier is trained with a phrase-based
classifier.
Learning Algorithm – SNoW

A sparse network of linear classifiers


Weight update: a regularized variation of the Winnow multiplicative
update rule
When probability estimation is needed, we use softmax
Page 6
Argument Identification (Features)


Parse tree structure from Collins & Charniak’s parsers
Clauses, chunks and POS tags are from UPC
processors
Page 7
Argument Classification


Similar to argument identification, using SNoW as a
multi-class classifier
Classes also include NULL
Page 8
Inference


Occasionally, the output of the argument classifier
violates some constraints.
The inference procedure [Punyakanok et al., 2004]


Input: the probability estimation (by the argument classifier), and
structural and linguistic constraints
Output: the best legitimate global predictions

Formulated as an optimization problem and solved via Integer
Linear Programming.

Allows incorporating expressive (non-sequential) constraints on
the variables (the arguments types).
Page 9
Integer Linear Programming Inference

For each argument ai


Set up a Boolean variable: ai,t indicating if ai is classified as t
Goal is to maximize


 i score(ai = t ) ai,t
Subject to the (linear) constraints


Any Boolean constraints can be encoded this way.
If score(ai = t ) = P(ai = t ), the objective is find the
assignment that maximizes the expected number of
arguments that are correct and satisfies the constraints
Page 10
Constraints

No overlapping or embedding arguments
ai,aj overlap or embed: ai,NULL + aj,NULL  1
Page 11
Constraints

Constraints







No overlapping or embedding arguments
No duplicate argument classes for A0-A5
Exactly one V argument per predicate
If there is a C-V, there must be V-A1-C-V pattern
If there is an R-arg, there must be arg somewhere
If there is a C-arg, there must be arg somewhere before
Each predicate can take only core arguments that appear in its
frame file.

More specifically, we check for only the minimum and maximum ids
Page 12
Results
Dev
WSJ
Brown
Prec
Rec
F1
Collins
73.89
70.11
71.95
Charniak
75.40
74.13
74.76
Collins
77.09
72.00
74.46
Charniak
78.10
76.15
77.11
Collins
68.03
63.34
65.60
Charniak
67.15
63.57
65.31
Page 13
Inference with Multiple Systems

The performance of SRL heavily depends on the
very first stage – pruning [IJCAI 2005]
 which

is derived directly from the full parse trees
Joint Inference allows improvement over
semantic role labeling classifiers
 Combine
different SRL systems through joint inference
 Systems are derived using different full parse trees
Page 14
Inference with Multiple Systems

Multiple Systems
 Train
and test with Collins’ parse outputs
 Train with Charniak’ best parse outputs

Test with 5-best Charniak’ parse outputs
Page 15
Naïve Joint Inference
..., traders say, unable to cool the selling panic in both stocks and futures.
traders
a1
Null A0
0.2
the selling panic in both stocks and futures
a4
A1
A2
0.4 0.2 0.2
traders
b1
Null A0
0.3
Null A0
A1
A2
0.3
0.7
0
0
the selling panic in both stocks and futures
b2
b3
A1
A2
0.3 0.2 0.2
Null A0
0.1
A1
A2
0.2 0.4 0.3
Null A0
0.1
A1
A2
0.3 0.2 0.4
Page 16
Joint Inference – Phantom Candidates
a1
a4
b1
Null
A0
A1
A2
0.55
0.2
0.15
0.1
a2
a3
b2
b3
b4
Default Priors
Page 17
Results of Joint Inference
F1
Col
Char
Char-2
Char-3
Char-4
Char-5
Combined
Devel
77.35
WSJ
79.44
Brown
67.75
60
70
80
90
Page 18
Results of Joint Inference
Recall
Devel
74.83
WSJ
76.78
Brown
62.93
60
65
70
75
Col
Char
Char-2
Char-3
Char-4
Char-5
Combined
80
Page 19
Results of Joint Inference
Precision
Devel
80.05
WSJ
82.28
Brown
73.38
60
70
80
Col
Char
Char-2
Char-3
Char-4
Char-5
Combined
90
Page 20
Results of Different Combination
F1
Devel
Combined
Col+Char1
Char1-5
Best Single
WSJ
Brown
60
70
80
90
Page 21
Conclusion

The ILP inference can naturally be extended to reason
over multiple SRL systems.
Page 22
Thank You
Page 23