Introduction
Background
System Architecture
Evaluation
Conclusion
Question Generation with Minimal Recursion
Semantics
Xuchen Yao1 and Yi Zhang2
1 European
Masters in Language and Communication Technologies
University of Groningen & Saarland University
2 Saarland University
German Research Center for Artificial Intelligence
Handleiding
18 June, QG/QGSTEC/2010
logobestanden
versie 2.0, augustus 2007
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Outline
Introduction
Template/Syntax/Semantics-based Approaches
Why Semantics-based?
Background
MRS/ERG/PET/LKB
System Architecture
Overview
MRS Transformation for Simple Sentences
MRS Decomposition for Complex Sentences
Language Independence and Domain Adaptability
Evaluation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Approaches
• Template-based (Mostow and Chen (2009))
• What did <character> <verb>?
• Syntax-based (Wyse and Piwek (2009), Heilman and Smith
(2009))
•
•
•
•
•
John plays football. (S NP (VP (V NP)))
John plays what? (S NP (VP (V WHNP)))
John does play what? (S NP (VP (Aux-V V WHNP)))
Does John play what? (S Aux-V NP (VP (V WHNP)))
What does John play? (S WHNP Aux-V NP (VP (V)))
• Semantics-based
• play(John, football)
• play(John, what)
• play(who, football)
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Approaches
• Template-based (Mostow and Chen (2009))
• What did <character> <verb>?
• Syntax-based (Wyse and Piwek (2009), Heilman and Smith
(2009))
•
•
•
•
•
John plays football. (S NP (VP (V NP)))
John plays what? (S NP (VP (V WHNP)))
John does play what? (S NP (VP (Aux-V V WHNP)))
Does John play what? (S Aux-V NP (VP (V WHNP)))
What does John play? (S WHNP Aux-V NP (VP (V)))
• Semantics-based
• play(John, football)
• play(John, what)
• play(who, football)
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Outline
Introduction
Template/Syntax/Semantics-based Approaches
Why Semantics-based?
Background
MRS/ERG/PET/LKB
System Architecture
Overview
MRS Transformation for Simple Sentences
MRS Decomposition for Complex Sentences
Language Independence and Domain Adaptability
Evaluation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
References
Why Semantics-based?
• Something different than template/syntax-based.
• More intuitive?
• More language independent (universal)?
• Make use of the generation function of the English Resource
Grammar
• Sag, I. A. & Flickinger, D. Generating Questions with Deep
Reversible Grammars. In Proceedings of the First Workshop on
the Question Generation Shared Task and Evaluation
Challenge. 2008.
• Deeper is better?
Introduction
Background
System Architecture
Evaluation
Conclusion
References
Why Semantics-based?
• Something different than template/syntax-based.
• More intuitive?
• More language independent (universal)?
• Make use of the generation function of the English Resource
Grammar
• Sag, I. A. & Flickinger, D. Generating Questions with Deep
Reversible Grammars. In Proceedings of the First Workshop on
the Question Generation Shared Task and Evaluation
Challenge. 2008.
• Deeper is better?
Introduction
Background
System Architecture
Evaluation
Conclusion
Outline
Introduction
Template/Syntax/Semantics-based Approaches
Why Semantics-based?
Background
MRS/ERG/PET/LKB
System Architecture
Overview
MRS Transformation for Simple Sentences
MRS Decomposition for Complex Sentences
Language Independence and Domain Adaptability
Evaluation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
References
DELPH-IN (MRS/ERG/PET/LKB)
Deep Linguistic Processing with HPSG: http://www.delph-in.net/
John likes Mary.
like(John, Mary)
Parsing
with PET
Generation
with LKB
John likes Mary.
Minimal Recursion Semantics
INDEX: e2
RELS: <
[ PROPER_Q_REL<0:4>
[ NAMED_REL<0:4>
LBL: h3
LBL: h7
ARG0: x6
ARG0: x6
RSTR: h5
(PERS: 3 NUM: SG)
BODY: h4 ]
CARG: "John" ]
[ _like_v_1_rel<5:10>
LBL: h8
ARG0: e2 [ e SF: PROP TENSE: PRES ]
ARG1: x6
ARG2: x9
[ PROPER_Q_REL<11:17> [ NAMED_REL<11:17>
LBL: h13
LBL: h10
ARG0: x9
ARG0: x9
(PERS: 3 NUM: SG)
RSTR: h12
CARG: "Mary" ]
BODY: h11 ]
>
English Resource
HCONS: < h5 qeq h7 h12 qeq h13 >
Grammar
Introduction
Background
System Architecture
Evaluation
Conclusion
References
Details
(THEORY)MRS: Minimal Recursion Semantics
a meta-level language for describing semantic structures in some
underlying object language.
(GRAMMAR)ERG: English Resource Grammar
a general-purpose broad-coverage grammar implementation under
the HPSG framework.
(TOOL)LKB: Linguistic Knowledge Builder
a grammar development environment for grammars in typed
feature structures and unification-based formalisms.
(TOOL)PET: a platform for experimentation with efficient
HPSG processing techniques
a two-stage parsing model with HPSG rules and PCFG models,
balancing between precise linguistic interpretation and robust
probabilistic coverage.
Introduction
Background
System Architecture
Evaluation
Conclusion
References
Details
(THEORY)MRS: Minimal Recursion Semantics
a meta-level language for describing semantic structures in some
underlying object language.
(GRAMMAR)ERG: English Resource Grammar
a general-purpose broad-coverage grammar implementation under
the HPSG framework.
(TOOL)LKB: Linguistic Knowledge Builder
a grammar development environment for grammars in typed
feature structures and unification-based formalisms.
(TOOL)PET: a platform for experimentation with efficient
HPSG processing techniques
a two-stage parsing model with HPSG rules and PCFG models,
balancing between precise linguistic interpretation and robust
probabilistic coverage.
Introduction
Background
System Architecture
Evaluation
Conclusion
Outline
Introduction
Template/Syntax/Semantics-based Approaches
Why Semantics-based?
Background
MRS/ERG/PET/LKB
System Architecture
Overview
MRS Transformation for Simple Sentences
MRS Decomposition for Complex Sentences
Language Independence and Domain Adaptability
Evaluation
References
Introduction
Background
System Architecture
Evaluation
MrsQG (Task B)
http://code.google.com/p/mrsqg/
Conclusion
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Term Extraction
Plain text
4
1
Term
extraction
Generation
with LKB
Coordination Decomposer
2
NEday
NEperson
FSC
NEusPresident
NEcapital
construction
3
5
Stanford
Named Entity Recognizer MRS
MRS
Decomposition
●a regular expression NE tagger Transformation
MRS
●an Ontology NE tagger
XML
6
Apposition Decomposer
●
Subclause Decomposer
7
NEfirstName
NElocation
NEmonth
NEyear
Subordinate Decomposer
FSC
XML
NEprovince
Output
NEstate
selection
NElocation
8
Output
to
Jackson
was born on Why
August
29, 1958 in Gary,
Indiana.
Decomposer
Parsing
with PET
NEperson
MRS
XML
console/XML
NEdate
NElocation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Term Extraction
Plain text
4
1
Term
extraction
5
Stanford
Named Entity Recognizer MRS
MRS
Decomposition
●a regular expression NE tagger Transformation
MRS
●an Ontology NE tagger
XML
6
Apposition Decomposer
●
Coordination Decomposer
2
Subclause Decomposer
FSC
who
construction
3
Generation
with LKB
where
7
when
Output
which location
selection
which day
Subordinate Decomposer
FSC
XML
8
Jackson was born onWhy
August
29, 1958 in Gary,
Indiana.
Output
to
Decomposer
Parsing
with PET
NEperson
MRS
XML
console/XML
NEdate
NElocation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Outline
Introduction
Template/Syntax/Semantics-based Approaches
Why Semantics-based?
Background
MRS/ERG/PET/LKB
System Architecture
Overview
MRS Transformation for Simple Sentences
MRS Decomposition for Complex Sentences
Language Independence and Domain Adaptability
Evaluation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
MRS Transformation
Plain text
4
1
5
MRS Decomposition
Apposition Decomposer
Term
extraction
MRS
XML
Generation
with LKB
7
Subclause Decomposer
FSC
construction
3
6
Coordination Decomposer
2
MRS
Transformation
Subordinate Decomposer
FSC
XML
Parsing
with PET
Output
selection
8
MRS
XML
Why Decomposer
Output to
console/XML
References
Introduction
Background
System Architecture
Evaluation
Conclusion
References
WHO
proper_q(x)
named(x,"John")
which_q(x)
proper_q(y)
named(y,"Mary") like(e,x,y)
person(x)
proper_q(y)
named(y,"Mary") like(e,x,y)
Figure: “John likes Mary” → “Who likes Mary?”
Introduction
Background
System Architecture
Evaluation
Conclusion
References
WHERE
proper_q(x)
named(x,"Broadway")
which_q(x)
proper_q(y)
named(y,"Mary")
sing(e,y),
on_p(x)
place(x)
proper_q(y)
named(y,"Mary")
sing(e,y),
loc_nonsp(x)
Figure: “Mary sings on Broadway.” → “Where does Mary sing?”
Introduction
Background
System Architecture
Evaluation
Conclusion
References
WHEN
def_implict_q(x)
numbered_hour(x,"10")
which_q(x)
proper_q(y)
named(y,"Mary")
time(x)
sing(e,y),
at_p_temp(x)
proper_q(y)
named(y,"Mary")
Figure: “Mary sings at 10.” → “When does Mary sing?”
sing(e,y),
loc_nonsp(x)
Introduction
Background
System Architecture
Evaluation
Conclusion
References
WHY
proper_q(x)
named(x,"Mary")
which_q(x)
proper_q(y)
named(y,"John")
reason(x)
fight(e,y),
for_p(x)
proper_q(y)
named(y,"John")
Figure: “John fights for Mary.” → “Why does John fight?”
fight(e,y),
for_p(x)
Introduction
Background
System Architecture
Evaluation
Conclusion
Outline
Introduction
Template/Syntax/Semantics-based Approaches
Why Semantics-based?
Background
MRS/ERG/PET/LKB
System Architecture
Overview
MRS Transformation for Simple Sentences
MRS Decomposition for Complex Sentences
Language Independence and Domain Adaptability
Evaluation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
MRS Decomposition
Complex Sentences -> Simple Sentences
Plain text
4
1
5
MRS Decomposition
Apposition Decomposer
Term
extraction
MRS
XML
Generation
with LKB
7
Subclause Decomposer
FSC
construction
3
6
Coordination Decomposer
2
MRS
Transformation
Subordinate Decomposer
FSC
XML
Parsing
with PET
Output
selection
8
MRS
XML
Why Decomposer
Output to
console/XML
References
Introduction
Background
System Architecture
Evaluation
Conclusion
References
Subclause Decomposer
identifies the verb, extracts its arguments and reconstructs MRS
proper_q(x)
named(x,"Bart")
proper_q(x)
the_q(y)
the_q(z)
dog(z)
named(x,"Bart")
be(e1,x,y)
the_q(y)
cat(y)
be(e1,x,y)
cat(y),chase(e2,y,z)
Figure: “Bart is the cat that chases the dog.” → “Bart is the cat.”
Introduction
Background
System Architecture
Evaluation
Conclusion
References
Subclause Decomposer
identifies the verb, extracts its arguments and reconstructs MRS
the_q(y)
proper_q(x)
named(x,"Bart")
cat(y)
the_q(y)
the_q(z)
dog(z)
be(e1,x,y)
the_q(z)
dog(z)
chase(e2,y,z)
cat(y),chase(e2,y,z)
Figure: “Bart is the cat that chases the dog.” → “The cat chases the
dog.”
Introduction
Background
System Architecture
Evaluation
Conclusion
MRS Decomposition
Complex Sentences -> Simple Sentences
Plain text
4
1
5
MRS Decomposition
Apposition Decomposer
Term
extraction
MRS
XML
Generation
with LKB
7
Subclause Decomposer
FSC
construction
3
6
Coordination Decomposer
2
MRS
Transformation
Subordinate Decomposer
FSC
XML
Parsing
with PET
Output
selection
8
MRS
XML
Why Decomposer
Output to
console/XML
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Outline
Introduction
Template/Syntax/Semantics-based Approaches
Why Semantics-based?
Background
MRS/ERG/PET/LKB
System Architecture
Overview
MRS Transformation for Simple Sentences
MRS Decomposition for Complex Sentences
Language Independence and Domain Adaptability
Evaluation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Language Independence
MrsQG aims to stay language-neutral based on a semantics
transformation of sentences.
In Principle
It needs little modification to adapt to other languages.
In Practice
It is difficult to guarantee absolute language independence.
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Domain Adaptability
Plain text
4
1
Term
extraction
Coordination Decomposer
PET Parser:
● re-train with an HPSG treebank.
7
2
Subclause Decomposer
FSC
construction
3
HPSG grammars:
● Hand-written
Subordinate Decomposer
● Generalize well
● Steady performance
FSC
XML
Parsing
with PET
5
MRS
Needs to re-train or modify:
MRS Decomposition
●Stanford
Transformation
Named Entity Recognizer
●a regular expression NE tagger
MRS
XML
●an
6
Apposition
Decomposer
Ontology
NE tagger
MRS
XML
Why Decomposer
Generation
with LKB
Output
selection
8
Output to
console/XML
References
Introduction
Background
System Architecture
Evaluation
Conclusion
References
QGSTEC2010
The Question Generation Shared Task and Evaluation Challenge (QGSTEC) 2010
Task B: QG from Sentences.
Participants are given one complete sentence from which their
system must generate questions.
1. Relevance. Questions should be relevant to the input
sentence.
2. Question type. Questions should be of the specified target
question type.
3. Syntactic correctness and fluency. The syntactic
correctness is rated to ensure systems can generate sensible
output.
4. Ambiguity. The question should make sense when asked
more or less out of the blue.
5. Variety. Pairs of questions in answer to a single input are
evaluated on how different they are from each other.
Introduction
Background
System Architecture
Evaluation
Conclusion
References
Examples
TEXT: Alexander Graham Bell, who had risen to prominence
through his invention of the telephone, took a great
interest in recording sounds, even suggesting to
Edison that they might collaborate.
WHO: Who took a great interest in recording sounds?
WHO: Who is Alexander Graham Bell?
WHAT: A great interest in what did Alexander Graham Bell
take?
WHAT: What did Alexander Graham Bell take a great
interest in?
WHY: Why Alexander Graham Bell took a great interest in
cording sounds?
WHY: Why do they collaborate?
Introduction
Background
System Architecture
Evaluation
Conclusion
Outline
Introduction
Template/Syntax/Semantics-based Approaches
Why Semantics-based?
Background
MRS/ERG/PET/LKB
System Architecture
Overview
MRS Transformation for Simple Sentences
MRS Decomposition for Complex Sentences
Language Independence and Domain Adaptability
Evaluation
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Conclusion
• semantics-based (easy in theory, difficult in practice)
• multi-linguality
• cross-domain
• deep grammar (worry less, wait more)
• generation <-> grammaticality
• heavy machinery
References
Introduction
Background
System Architecture
Evaluation
Conclusion
Conclusion
• semantics-based (easy in theory, difficult in practice)
• multi-linguality
• cross-domain
• deep grammar (worry less, wait more)
• generation <-> grammaticality
• heavy machinery
References
Introduction
Background
System Architecture
Evaluation
Conclusion
References
References
M. Heilman and N. A. Smith. Question Generation via
Overgenerating Transformations and Ranking. Technical report,
Language Technologies Institute, Carnegie Mellon University
Technical Report CMU-LTI-09-013, 2009.
Jack Mostow and Wei Chen. Generating instruction automatically
for the reading strategy of self-questioning. In Proceeding of the
2009 conference on Artificial Intelligence in Education,
Amsterdam, The Netherlands, 2009. ISBN 978-1-60750-028-5.
Brendan Wyse and Paul Piwek. Generating Questions from
OpenLearn study units. In Proceedings of the 2nd Workshop on
Question Generation In Craig, S.D. & Dicheva, S. (Eds.) (2009)
AIED 2009: 14th International Conference on Artificial
Intelligence in Education: Workshops Proceedings, 2009.