A coherence model based on syntactic patterns
by Annie Louis and Ani Nenkova
M.Sc. Seminar: Discourse Coherence Theories and Modeling
Nikolina Koleva
Saarland University
Department of Computational Linguistics
June 10, 2013
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
1 / 32
Overview
1
Motivation
2
Coherence models based on syntax
Evidence for syntactic coherence
Representing syntax
Local co-occurrence model
Global model
3
Evaluation
Prediction on reports
Prediction on academic articles
4
Conclusion
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
2 / 32
Motivation
Factors contributing to coherence
1
attentional structure (items under discussion)
2
organization of discourse segments
3
intentional structure (purpose of the discourse)
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
3 / 32
Motivation
Factors contributing to coherence
1
2
3
"
organization of discourse segments
" content approaches
intentional structure: purpose of the discourse
% not much work
attentional structure: items under discussion
entity approaches
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
3 / 32
Motivation
Every discourse has a purpose
• explaining a concept
• narrating an event
• critiquing an idea
• ...
each sentence in a text has a communicative goal
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
4 / 32
Motivation
Example
1
An aqueduct is a water supply or navigable channel constructed
to convey water.
2
In modern engineering, the term is used for any system of pipes,
canals, tunnels, and other structures used for this purpose.
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
5 / 32
Motivation
Example
1
An aqueduct is a water supply or navigable channel constructed
to convey water.
2
In modern engineering, the term is used for any system of pipes,
canals, tunnels, and other structures used for this purpose.
1
Cytokine receptors are receptors that bind cytokines.
2
In recent years, the cytokine receptors have come to demand
more attention because their deficiency has now been directly
linked to certain debilitating immunodeficiency states.
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
5 / 32
Motivation
Example
1
An aqueduct is a water supply or navigable channel
constructed to convey water.
2
In modern engineering, the termis used for any system of
pipes, canals, tunnels, and other structures used for this
purpose.
1
Cytokine receptors are receptors that bind cytokines.
2
In recent years, the cytokine receptors have come to
demand more attention because their deficiency has now
been directly linked to certain debilitating
immunodeficiency states.
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
5 / 32
Motivation
Example
1
An aqueduct is a water supply or navigable channel
constructed to convey water.
2
In modern engineering, the term is used for any system of
pipes, canals, tunnels, and other structures used for this
purpose.
1
Cytokine receptors are receptors that bind cytokines.
2
In recent years, the cytokine receptors have come to demand
more attention because their deficiency has now been directly
linked to certain debilitating immunodeficiency states.
unique syntactic structure of definitions, questions etc.
syntax as proxy for the communicative goal
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
5 / 32
Coherence models based on syntax
Coherence model based on syntax
Underlying assumptions:
1
Sentences with similar syntax are likely to have the same
communicative goal.
2
Regularities in intentional structure will be manifested in
syntactic regularities between adjacent sentences.
supported by recent related work
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
6 / 32
Coherence models based on syntax
Evidence for syntactic coherence
Pilot study for the validation of assumption No: 2
• Material: gold standard parse trees from the Penn Treebank
• Unit of analysis: two adjacent sentences, a pair (S1 , S2 )
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
7 / 32
Coherence models based on syntax
Evidence for syntactic coherence
Pilot study for the validation of assumption No: 2
• Material: gold standard parse trees from the Penn Treebank
• Unit of analysis: two adjacent sentences, a pair (S1 , S2 )
Steps:
1
enumerate all productions = 197 unique productions
• productions with frequency < 25 are removed
2
for all ordered pairs (p1 , p2 ) compute
• c (p1 , p2 ) ,c (p1 , ¬p2 ), c (¬p1 , p2 ) and c (¬p1 , ¬p2 )
c (p1 , p2 ): # of sentence pairs where p1 ∈ S1 and p2 ∈ S2
3
perform chi-square test to
• prove significance of the count c (p1 , p2 )
• check independence of the occurrences of p1 and p2
where, p1: production 1 and p2: production 2
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
7 / 32
Coherence models based on syntax
Evidence for syntactic coherence
Outcome of the study
• small fraction of repetitions (5%)
p1: VP → VBD SBAR
p2: VP → VBD SBAR
1
S1: Documents filed with the Securities and Exchange Commission on the
pending spinoff [[disclosed]VBD [that Cray Research Inc. will withdraw the
almost $ 100 million in financing it is providing the new firm if Mr. Cray leaves
or if the product-design project, he heads, is scrapped]SBAR ]VP .
2
S2: The documents also [[said]VBD [that although the 64-year-old Mr. Cray has
been working on the project for more than six years , the Cray-3 machine is at
least another year away from a fully operational prototype]SBAR ]VP .
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
8 / 32
Coherence models based on syntax
Evidence for syntactic coherence
Outcome of the study
• finance domain-specific
p1: NP → NP NP-ADV
p2: QP → CD CD
1
S1: The two concerns said they entered into a definitive merger agreement
under which Ratners will begin a tender offer for all of Weisfield’s common
shares for [$57.50 each]NP .
2
S2: Also on the takeover front, Jaguar’s ADRs rose 1/4 to 13 7/8 on turnover of
[4.4 million]QP .
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
9 / 32
Coherence models based on syntax
Evidence for syntactic coherence
Outcome of the study
• neither repetitions nor domain dependent
p1: VP → VB VP
p2: NP-SBJ → NNP NNP
1
S1: "The refund pool may not [be held hostage through another round of
appeals]VP , " Judge Curry said.
2
S2: [Commonwealth Edison]NP −SBJ said it is already appealing the underlying
commission order and is considering appealing Judge Curry’s order.
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
10 / 32
Coherence models based on syntax
Evidence for syntactic coherence
Outcome of the study
• neither repetitions nor domain dependent
p1: VP → VB VP
p2: NP-SBJ → NNP NNP
1
S1: "The refund pool may not [be held hostage through another round of
appeals]VP , " Judge Curry said.
2
S2: [Commonwealth Edison]NP −SBJ said it is already appealing the underlying
commission order and is considering appealing Judge Curry’s order.
• S1 present hypothesis or speculation
• S2 introduces an entity (PERS, ORG) that gives explanation or opinion
on the statement
• intentional structure: SPECULATE , ENDORSE
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
10 / 32
Coherence models based on syntax
Evidence for syntactic coherence
Outcome of the study
• neither repetitions nor domain dependent
p1: NP-LOC → NNP
p2: S-TPC-1 → NP-SBJ VP
1
S1: "It has to be considered as an additional risk for the investor," said Gary P.
Smaby of Smaby Group Inc., [Minneapolis]NP −LOC .
2
S2: ["Cray Computer will be a concept stock,"]S −TPC −1 he said.
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
11 / 32
Coherence models based on syntax
Evidence for syntactic coherence
Outcome of the study
• neither repetitions nor domain dependent
p1: NP-LOC → NNP
p2: S-TPC-1 → NP-SBJ VP
1
S1: "It has to be considered as an additional risk for the investor," said Gary P.
Smaby of Smaby Group Inc., [Minneapolis]NP −LOC .
2
S2: ["Cray Computer will be a concept stock,"]S −TPC −1 he said.
• S1 introduces location name associated with an entity
• S2 contains quote from that entity
• intentional structure: INTRODUCE X , STATEMENT BY X
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
11 / 32
Coherence models based on syntax
Representing syntax
Representing syntax
1
productions
• sentence as set of grammatical productions (LHS →RHS)
• RHS could be very long and thus rather specific
• available information only about nodes of the same constituent
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
12 / 32
Coherence models based on syntax
Representing syntax
Representing syntax
1
productions
• sentence as set of grammatical productions (LHS →RHS)
• RHS could be very long and thus rather specific
• available information only about nodes of the same constituent
2
d-sequence
• cut the parse tree at level d
• sentence as sequence of leaf nodes (of the cut tree)
• for each node in the sequence augmented the tag of the left most child
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
12 / 32
Coherence models based on syntax
Representing syntax
d-sequence example
• depth-2 sequence: " S:dt , " NP:nnp VP:vbd .
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
13 / 32
Coherence models based on syntax
Representing syntax
d-sequence example
Please, write down the depth-3 sequence.
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
13 / 32
Coherence models based on syntax
Representing syntax
d-sequence example
Please, write down the depth-3 sequence.
• depth-3 sequence: " NP:dt VP:vbz , " NNP NNP VBD .
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
13 / 32
Coherence models based on syntax
Local co-occurrence model
Local co-occurrence model idea
to test assumption No 2:
Regularities in intentional structure will be manifested
in syntactic regularities between adjacent sentences.
Steps:
1
estimate probabilities of pairs of syntactic items (from the training set)
2
use these probabilities to compute the coherence of a new text
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
14 / 32
Coherence models based on syntax
Local co-occurrence model
Local co-occurrence model implementation
• n: number of sentences
y
• Sx the y th item of the x th sentence
• δC : smoothing constant
• |V |: size of the vocabulary
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
15 / 32
Coherence models based on syntax
Local co-occurrence model
Local co-occurrence model example
s2:
s1:
1
S → NP VP
2
NP → DT N
3
VP → VBD NP
4
NP → DT N
Nikolina Koleva (CoLi Saarland)
1
S → NP VP
2
NP → DT N
3
VP → VBD PP
4
PP → P NP
5
NP → DT N
Syntactic Approach to Modeling Coherence
June 10, 2013
16 / 32
Coherence models based on syntax
Local co-occurrence model
Local co-occurrence model example
s2:
s1:
1
S → NP VP
2
NP → DT N
3
VP → VBD NP
4
NP → DT N
1
S → NP VP
2
NP → DT N
3
VP → VBD PP
4
PP → P NP
5
NP → DT N
• [ p (S → NP VP | S → NP VP ) + p (S → NP VP | NP → DT N ) + p (S → NP VP | VP →
VBD NP) + p (S → NP VP | NP → DT N ) ] * [ p (NP → DT N | S → NP VP ) + p (NP →
DT N | NP → DT N ) + p (NP → DT N | VP → VBD NP) + p (NP → DT N | NP → DT N )
] * ...
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
16 / 32
Coherence models based on syntax
Local co-occurrence model
Global coherence model idea
to test assumption No 1:
Sentences with similar syntax are likely to have the
same communicative goal.
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
17 / 32
Coherence models based on syntax
Local co-occurrence model
Clusters from abstracts of journal articles
Cluster a: VP → VBZ ADJP ; ADJP → JJ PP
1
This method [is [capable of sequence-specific detection of DNA with
high accuracy]ADJP ]VP
2
The same [is [true for synthetic polyamines such as
polyallylamine]ADJP ]VP
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
18 / 32
Coherence models based on syntax
Local co-occurrence model
Clusters from abstracts of journal articles
Cluster a: VP → VBZ ADJP ; ADJP → JJ PP
1
This method [is [capable of sequence-specific detection of DNA with
high accuracy]ADJP ]VP
2
The same [is [true for synthetic polyamines such as
polyallylamine]ADJP ]VP
Cluster b: VP → MD VP ; VP → VB VP
1
Our results for the difference in reactivity [can [be linked to
experimental observations]VP ]VP
2
These phenomena taken together [can [be considered as the signature
of the gelation process]VP ]VP
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
18 / 32
Coherence models based on syntax
Local co-occurrence model
Clusters from abstracts of journal articles
Cluster a: VP → VBZ ADJP ; ADJP → JJ PP
1
This method [is [capable of sequence-specific detection of DNA with
high accuracy]ADJP ]VP
2
The same [is [true for synthetic polyamines such as
polyallylamine]ADJP ]VP
captures descriptive sentences
Cluster b: VP → MD VP ; VP → VB VP
1
Our results for the difference in reactivity [can [be linked to
experimental observations]VP ]VP
2
These phenomena taken together [can [be considered as the signature
of the gelation process]VP ]VP
captures speculative sentences
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
18 / 32
Coherence models based on syntax
Global model
Global coherence model idea
and to capture
the common patterns in the intentional structure for
the domain (by using HMM)
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
19 / 32
Coherence models based on syntax
Global model
Global coherence model idea
and to capture
the common patterns in the intentional structure for
the domain (by using HMM)
Steps:
1
cluster sentences of different documents by syntactic similarity
features :
• productions: frequency of production in a parse tree
• d-sequence: n-grams of size one to four
2
estimate emission and transition probabilities (from the training set)
3
use these probabilities to compute the coherence of a new text
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
19 / 32
Coherence models based on syntax
Global model
Global coherence model
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
20 / 32
Coherence models based on syntax
Global model
Global coherence model implementation
• n: number of sentences, St : the t th sentence
• ht : the t th state
• d (ht ): # of docs whose sentences appear in ht
• d (ht , ht −1 ) # of docs where subsequent sentences in subsequent states
• δM : smoothing constant, C: # of clusters
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
21 / 32
Evaluation
Evaluating syntactic coherence
1
prediction on reports
• use pairs of articles: (original article, random permutation)
• testing on identifying the original article
• compare with content and entity approaches
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
22 / 32
Evaluation
Evaluating syntactic coherence
1
prediction on reports
• use pairs of articles: (original article, random permutation)
• testing on identifying the original article
• compare with content and entity approaches
2
prediction on academic articles
• original vs. permuted sections
• conference vs. workshop papers
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
22 / 32
Evaluation
Prediction on reports
Prediction on reports
Model
baseline
Prod
d-seq
PoS
Prod
d-seq
Egrid
Content
Airplane Accidents
Earthquake
Parameters
Accuracy
Parameters
50.0
Local co-occurrence model
72.8
depth MVP + 2
71.8
depth MVP + 1
61.3
HMM-syntax
clus. 37
74.6
clus. 5
depth MVP + 8, clus. 8
82.2
depth MVP + 9, clus. 45
Other approaches
history 1
67.6
history 1
clus.48
71.4
clus. 23
Accuracy
50.0
55.0
65.1
42.6
93.8
86.5
82.2
84.5
the articles of each corpus have the same intentional structure
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
23 / 32
Evaluation
Prediction on reports
Combining predictions of different models
Accuracy
Airplane Accidents
Earthquake
Model
Content + Egrid
Content + HMM-prod
Content + HMM-d-seq
Egrid + HMM-prod
Egrid + HMM-d-seq
Egrid + Content + HMM-prod
Egrid + Content + HMM-d-seq
Egrid + Content + HMM-prod + HMM-d-seq
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
76.8
74.2
82.1
79.6
84.2
79.5
84.1
83.6
90.7
95.3
90.3
93.9
91.1
95.0
92.3
95.7
June 10, 2013
24 / 32
Evaluation
Prediction on reports
Combining predictions of different models
Accuracy
Airplane Accidents
Earthquake
Model
Content + Egrid
Content + HMM-prod
Content + HMM-d-seq
Egrid + HMM-prod
Egrid + HMM-d-seq
Egrid + Content + HMM-prod
Egrid + Content + HMM-d-seq
Egrid + Content + HMM-prod + HMM-d-seq
76.8
74.2
82.1
79.6
84.2
79.5
84.1
83.6
90.7
95.3
90.3
93.9
91.1
95.0
92.3
95.7
syntax supplements content and entity grid methods
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
24 / 32
Evaluation
Prediction on academic articles
Corpora of academic articles
1
ART Corpus
• 225 Chemistry journal articles
• manually annotated for intentional structure
2
ACL Anthology Network (AAN)
• 500 ACL-NAACL conference articles
• 500 ACL-sponsored workshop articles
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
25 / 32
Evaluation
Prediction on academic articles
Detected clusters vs. manually annotated zones
• manually annotated zones in ART
Motivation
Background
3 Hypothesis
4 Objective
5 ...
1
2
• compare to detected clusters
• compute c (Ci , Zj ): # of sentences that are annotated as Zj and are
contained in Ci
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
26 / 32
Evaluation
Prediction on academic articles
Original vs. permuted sections
Table : Accuracy in %
Data
Section
Test Pairs
ART Corpus
Abstract
Introduction
Abstract
Introduction
Rel. work
1633
1640
8815
9966
10,000
ACL
Nikolina Koleva (CoLi Saarland)
Local-prod
Local-d-seq
HMM-prod
HMM-d-seq
Oracle zones
57.0
44.5
44.0
54.5
54.6
52.9
54.6
47.2
53.0
54.4
64.1
58.1
58.2
64.4
57.3
55.0
64.6
63.7
74.0
67.3
80.8
94.0
Syntactic Approach to Modeling Coherence
June 10, 2013
27 / 32
Evaluation
Prediction on academic articles
Distinguish conference and workshop articles
• conference articles: complete work
information presentation differ in abstracts and introductions
• workshop articles: preliminary studies
• used features
• indicating perplexity of the local and global models
• fine-grained taken from the local model
• most significant 30 pairs
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
28 / 32
Evaluation
Prediction on academic articles
Distinguish conference and workshop articles
• conference articles: complete work
information presentation differ in abstracts and introductions
• workshop articles: preliminary studies
• used features
• indicating perplexity of the local and global models
• fine-grained taken from the local model
• most significant 30 pairs
Table : Accuracy above confidence level
Conf
Abstract
Introduction
Rel. work
>= 0.5
59.3
50.3
55.4
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
28 / 32
Conclusion
Summary
Syntactic patterns are reliable clues for intentional structure
detection
• Possible syntactic representations
•
• productions
• d-sequence
•
Local coherence model: exploring pairs of adjacent sentences
•
Global coherence model: clustering sentences based on syntactic
similarity
•
High accuracy on distinguishing coherent and incoherent news
articles
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
29 / 32
Conclusion
Thank you for your attention!
Any questions?
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
30 / 32
Conclusion
Discussion
• Would this approach work for languages with free word order?
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
31 / 32
Conclusion
References
Annie Louis and Ani Nenkova.
A coherence model based on syntactic patterns.
In Proceedings of the 2012 Joint Conference on Empirical Methods in
Natural Language Processing and Computational Natural Language
Learning, EMNLP-CoNLL ’12, pages 1157–1168, Stroudsburg, PA, USA,
2012. Association for Computational Linguistics.
URL http:
//dl.acm.org/citation.cfm?id=2390948.2391078.
Manfred Stede.
Discourse Processing.
Synthesis Lectures on Human Language Technologies. Morgan &
Claypool Publishers, 2011.
Nikolina Koleva (CoLi Saarland)
Syntactic Approach to Modeling Coherence
June 10, 2013
32 / 32