Experiences with Planning for Natural
Language Generation
Alexander Koller1,2 and Ron Petrick1
1 School
2 Department
of Informatics, University of Edinburgh
of Computational Linguistics, Saarland University
Scheduling and Planning Applications woRKshop (SPARK-08), Sydney, Australia
15 September 2008
Motivation
•
Natural Language Generation (NLG) is a major subfield of natural
language processing, concerned with computing natural language
sentences or texts that convey a piece of information to the user
•
Can be viewed as a problem involving actions, beliefs, and goals
A speaker tries to change the mental state of the
hearer by applying actions that correspond to the
utterance of words or sentences
⇒ Problems in NLG result in computationally challenging domains (often
more challenging than in other areas of natural language processing)
⇒ Obvious parallels to planning
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
2
NLG and planning
•
The link between NLG and planning has a long tradition
– Appelt (1985)
– Young & Moore (1994)
– ...
•
More recently, there has been a resurgence of interest in applying
modern planning techniques to NLG
–
–
–
–
•
Koller & Stone (2007)
Steedman & Petrick (2007)
Benotti (2008)
...
Recent approaches use planning as a modelling tool, with the hope of
improved efficiency over traditional NLG methods
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
3
This work
1. Describes two recent planning domains that arise from NLG
– Sentence generation
– Generating Instructions in Virtual Environments (GIVE)
2. Shares our experiences in these domains using off-the-shelf planners
that perform well in traditional planning domains
– FF (Hoffmann & Nebel, 2001)
– SGPLAN (Hsu et al., 2006)
– Java implementation of GraphPlan (Blum & Furst, 1997)
⇒ Empirical results using off-the-shelf planners are mixed
⇒ Domains are freely available; we believe these problems offer
interesting challenges for the planning community
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
4
I. Sentence generation
•
Sentence generation is the problem of computing, from a grammar
and semantic representation, a sentence that expresses this meaning
1. Sentence planning: the semantic representation is enriched
with more information, e.g., referring expressions
2. Surface realization: the enriched representation is translated
into a natural language sentence using the grammar
•
The process of determining referring expressions often interacts with
the realization step; beneficial to perform both steps together
•
E.g., SPUD system (Stone et al., 2003)
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
5
Example: “The white rabbit sleeps.”
•
Tree Adjoining Grammar (Joshi & Schabes, 1997)
N:self
S:self
NP:self
white
NP:subj ↓
VP:self
the
N:self *
N:self
V:self
rabbit
sleeps
{sleep(self,subj)}
{rabbit(self)}
{white(self)}
Elementary trees contribute semantic content.
Trees are instantiated by substituting individuals for semantic roles.
•
Knowledge base
–
–
–
–
a and b are rabbits
a is white
b is brown
e is an event in which a sleeps
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
6
Example: “The white rabbit sleeps.” (2)
N:self
S:self
NP:self
white
NP:subj ↓
the
VP:self
N:self *
N:self
V:self
rabbit
sleeps
{sleep(self,subj)}
{rabbit(self)}
{white(self)}
+ Knowledge base
A derivation to express {sleep(e, a)}:
S:e
NP:a ↓
S:e
VP:e
V:e
sleeps
NP:a
the
N:a
the
white
N:a
white
VP:e
NP:a
N:a
V:e
rabbit
sleeps
N:a *
rabbit
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
7
Sentence generation as planning
•
The sentence generation problem can be translated into a planning
problem (Koller & Stone, 2007)
•
Actions add elementary trees to a derivation
•
A plan must satisfy the syntactic and semantic linguistic requirements
[add-sleeps(root, a), add-rabbit(subj(root), a), add-white(subj(root), a)]
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
8
II. GIVE Challenge
•
“Generating Instructions in Virtual Environments” (Koller et al., 2007)
•
A challenge for the NLG community
Build an NLG system capable of producing natural
language instructions to guide a human user in
performing some task in a virtual environment
•
A theory-neutral task that tests all components of an NLG system
•
Virtual 3D client; evaluation possible over the Internet
•
First GIVE evaluation will take place in late 2008
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
9
Virtual 3D GIVE client
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
10
Planning in instruction giving
•
A GIVE problem is very similar to a Gridworld problem
– Discretised tiles of equal size
– User can turn in 90o steps in either direction and move forward
– Additional need to press buttons in the right order, reason about
more objects, and navigate more complicated room shapes
•
PDDL description of available actions
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
11
Example GIVE map
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
12
Challenges for NLG
•
Plan summarisation
“Turn left; turn left; walk forward”
⇒ “Turn around and walk through the door”
•
Plan elaboration
“Press the green button on the wall to your right”
⇒ “Walk to the centre of the room; turn right; now press the green
button in front of you”
•
...
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
13
Challenges for planning
•
Strict run-time requirements
– Planning must happen in (near) real time
– User can interact with the world while the system is deliberating
•
System must monitor a user’s actions and compare them against the
generated instruction set
– Mental state of the user is not known
– User’s actions may not match the generated instructions exactly
but still meet the intended goal ⇒ identify “equivalent” plans
– User can communicate intentions through action and inaction
•
Plans are often non-trivial, e.g., 108 steps in sample domain
[ turn-left(north,west),
move(pos 5 2,pos 4 2,west),
manipulate-b1-off-on(pos 5 2), . . . ]
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
14
Experiments and results
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
15
Experiment 1: Sentence generation
•
Compute a plan representing a sentence of the form
“Mary likes the Adj1 . . . Adjn rabbit.”
•
Knowledge base for each problem instance
– m rabbits
– n properties are required to distinguish the target reference from
other rabbits, where n ≤ m
•
The n properties are realised as n different adjectives, in any order
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
16
Experiment 1: Results
1000000
100000
SGPLAN total
FF total
FF grounding
SGPLAN parsing
GraphPlan (Java)
10000
1000
100
10
1
2,1 3,1 4,1
5,1
6,1
7,1
8,1
9,1
10,1
10,10
Runtime (ms) vs. (m,n)
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
17
Experiment 2: Minimal GIVE worlds
u1
u2
(n = 2)
h
l1
l2
•
Simplified GIVE world consisting of a 2n by h grid and buttons in
alternating positions
•
Player starts in position (1,1)
•
All buttons must be pressed to successfully complete the game
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
18
Experiment 2: Results
100
FF total
FF grounding
SGPLAN total
SGPLAN parsing
90
80
70
60
50
40
30
20
10
0
1
10
20
30
40
Runtime (sec) vs. n (h = 20)
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
19
Experiment 3: GIVE + extra positions
u1
u2
(n = 2)
h
l1
•
l2
w
We vary the structure of the GIVE world by adding another w by h
empty “junk” positions to the right of the minimal world
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
20
Experiment 3: Results
25
FF total
FF grounding
SGPLAN total
SGPLAN parsing
20
15
10
5
0
1
10
20
30
40
50
60
70
Runtime (sec) vs. w (h = 20, n = 5)
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
21
Observations
•
Modern planners are fairly good at the sentence generation task
– FF generates non-trivial 14-word sentences in about 2 seconds
– Special purpose NLG algorithms can generate referring
expressions in milliseconds (Areces et al., 2008)
– Larger, but realistically-sized, instances are still problematic for
the planners we tested
•
GIVE domain examples are more problematic
– Focus is on generating relatively short plans in large universes
– Planning times over a couple seconds negatively affect the
overall response time of the NLG system
– Restricting bottleneck seems to be the initial grounding stage
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
22
Conclusions
•
NLG problems offer a real-world testbed for planning under
uncertainty, resource management, execution monitoring, plan
presentation, reasoning about knowledge and sensing, . . .
•
Mixed results with off-the-shelf planners in two NLG planning domains
•
FF and SGPLAN are good at controlling search in our domains but
the total planning time is dominated by grounding time; large problem
instances remain a challenge
•
It’s not all bad news for planning! The GIVE challenge wouldn’t have
been possible 10 years ago...
•
Future work: experiments with other planners; other grammars . . .
•
Domain descriptions and code available online
http://code.google.com/p/crisp-nlg/
http://www.give-challenge.org/
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
23
References
Appelt, D. 1985. Planning English Sentences. Cambridge, UK: Cambridge
University Press.
Areces, C.; Koller, A.; and Striegnitz, K. 2008. Referring expressions as formulas
of description logic. In Proceedings of the 5th International Natural Language
Generation Conference.
Benotti, L. 2008. Accommodation through tacit sensing. In LONDIAL 2008 - The
12th SEMDIAL Workshop on the Semantics and Pragmatics of Dialogue.
Blum, A., and Furst, M. 1997. Fast planning through graph analysis. Artificial
Intelligence, volume 90.
Hoffmann, J., and Nebel, B. 2001. The FF planning system: Fast plan generation
through heuristic search. Journal of Artificial Intelligence Research, volume 14.
Hsu, C. W.; Wah, B. W.; Huang, R.; and Chen, Y. X. 2006. New features in
SGPlan for handling soft constraints and goal preferences in PDDL 3.0. 5th
International Planning Competition, ICAPS-2006.
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
24
References...(2)
Joshi, A., and Schabes, Y. 1997. Tree-Adjoining Grammars. In Rozenberg, G.,
and Salomaa, A., eds., Handbook of Formal Languages. Berlin: Springer-Verlag.
Koller, A.; Moore, J.; di Eugenio, B.; Lester, J.; Stoia, L.; Byron, D.; Oberlander, J.;
and Striegnitz, K. 2007. Shared task proposal: Instruction giving in virtual worlds.
In White, M., and Dale, R., eds., Working group reports of the Workshop on
Shared Tasks and Comparative Evaluation in Natural Language Generation.
Koller, A., and Stone, M. 2007. Sentence generation as planning. In Proceedings
of the 45th ACL.
Steedman, M., and Petrick, R. P. A. 2007. Planning dialog actions. In Proceedings
of the 8th SIGdial Workshop on Discourse and Dialogue.
Stone, M.; Doran, C.; Webber, B.; Bleam, T.; and Palmer, M. 2003. Microplanning
with communicative intentions: The SPUD system. Computational Intelligence,
volume 19(4).
Young, R. M., and Moore, J. D. 1994. DPOCL: a principled approach to discourse
planning. In Proceedings of the 7th International Workshop on Natural Language
Generation.
A. Koller & R. Petrick / Scheduling and Planning Applications woRKshop (SPARK-08) / 2008-09-15
25