Overlay – a non-monotonic
operation for discourse processing
Jan Alexandersson
(Tilman Becker, Markus Löckelt, Norbert Pfleger)
German Research Center for Artificial Intelligence – DFKI GmbH
Stuhlsatzenhausweg 3, Geb. 43.8 - 66123 Saarbrücken
phone: (+44) 681 302 5347
Email: [email protected]
http://www.dfki.de/~janal
© Löckelt, Becker, Pfleger, Alexandersson; DFKI
Edilog 2002 Workshop
What is this Talk about
• Dialogue systems must deal with utterances relating
elliptically to previous dialogue
System: „at what time would you like to start recording?“
User: „half past five“
User: „I would like to start recording at half past five“
• ... or utterances only partially related to the previous
discourse
User: „what films are on TV tonight?“
System: [Showing a list of films] „Here is a list of films“
User: ``Thats a boring program. I‘d rather go to the movies´´
User: ``Thats a boring program. I‘d rather go to the movies tonight´´
© Alexandersson; DFKI
Nancy 18.10.2002
What is this Talk about?
• There are similar elliptical phenomena like one-anaphora
``...an expensive large portrait of the queen by Wainright hanging in the library´´
 ``and one of the princess over the mantel´´
 ``and an expensive large portrait of the princess hanging over the
mantel´´
© Alexandersson; DFKI
Nancy 18.10.2002
Overview
•
•
•
•
•
•
•
SmartKom
Domain Model
Overlay
Discourse Modelling
Action Planning
Processing Partial Utterances
Conclusion
© Alexandersson; DFKI
Nancy 18.10.2002
What is the challenge?
• General tasks for a dialogue manager
– Enrichment:
• Given incomplete information, enrich it from other
sources in order to proceed.
– Validation:
• Validate hypotheses against the current discourse state
© Alexandersson; DFKI
Nancy 18.10.2002
Our Approach
• One general operation – Overlay
– Domain representation:
• (Typed) Feature Structures
– Non-monotonic:
• Always succeeds
– Computes a score:
• Reflecting the structural consistency of the two
arguments of overlay
© Alexandersson; DFKI
Nancy 18.10.2002
SmartKom: Task-Oriented Dialogue System
Application
Layer
Mobile:
Car and
Pedestrian
Navigation
MM
Dialogue
BackBone
Public:
Cinema,
Phone,
Fax,
Mail,
Biometrics
Home:
SmartKom-Mobile
Consumer Electronics
EPG
SmartKom-Public
SmartKom-Home/Office
© Alexandersson; DFKI
Nancy 18.10.2002
Module Overview on the SmartKom Control GUI
© Alexandersson; DFKI
Nancy 18.10.2002
The True Story
© Alexandersson; DFKI
Nancy 18.10.2002
The DFKI Dialogue Back-Bone
Analysers
•
•
•
•
•
Speech
Gesture
Modality
Fusion
Discourse
Modelling
DiM
Action
Planning
AP
Speech
Graphics
Generators
Gesture
Presentation
Manager
External
Services
Communication pools
Main data flow
Context information
© Alexandersson; DFKI
Nancy 18.10.2002
Domain Model
• Used for communication in the back-bone
• Frame-based ontology; representation as Typed Feature Structures in M3L (XML)
CinemaReservation theater: MovieTheater address: Address
seats: SeatStructure…
movie: Movie
name: String
director: Person firstName: String
lastName: String…
cast: PersonList
yearOfProduction: PositiveInteger…
reservationNumber: PositiveInteger
• Application objects composed of subobjects
• Slots: Feature paths meaningful for the dialogue (entities that can be talked
about / referenced to); e.g. movie:director:lastName in a CinemaReservation
object
• Slots can recursively contain other slots
© Alexandersson; DFKI
Nancy 18.10.2002
Discourse Modeling
• Lattice of intention hypothesis sequences
from Modality Fusion
• Enrichment and Validation
Intention
lattice
– compare and enrich with a selected
number of discourse states:
• fill in consistent information
• compute a score
– for each hypothesis - background pair:
Covering:
Background:
• Overlay (covering, background)
• Overlay gives a score representing how
well the covering fits the background.
• Scores of overlay, speech interpretation etc.
give overall score
• One augmented hypothesis sequence with
Selected
augmented
hypothesis
sequence
best score is passed on to Action Planning
© Alexandersson; DFKI
Nancy 18.10.2002
Example for Overlay with TFS
User: What films are on TV tonight?
System: [presents list of films]
User: That‘s a boring program, I‘d rather go to the movies.
How do we inherit “tonight” ?
© Alexandersson; DFKI
Nancy 18.10.2002
The Domain Model
Entertainm ent 
beginTime : Time 
...

title : String

Broadcast

channel : Channel 
...

A named TV
program at
some time on
some channel
© Alexandersson; DFKI
A named
entertainment at
some time
Performanc e

cinema : Cinema 
...

A named Movie
at some time at
some cinema
Nancy 18.10.2002
Unification Simulation
Entertainm ent 
beginTime : Time 
...

title : String

Broadcast

channel : Channel 
...

Performanc e

cinema : Cinema 
...

Films on TV tonight
Go to the movies
Broadcast


Time

beginTime
:

" tonight " 
channel : any

...



© Alexandersson; DFKI
Fail – type clash
Performanc e

beginTime : Time
cinema : any

...

Nancy 18.10.2002
Overlay Simulation
Entertainm ent 
beginTime : Time 
...

title : String

Broadcast

channel : Channel 
...

Films on TV tonight
Broadcast


Time

beginTime
:

" tonight " 
channel : any

...



© Alexandersson; DFKI
Assimilation
Performanc e

cinema : Cinema 
Performanc e

...


 Background
Time


beginTime : " tonight" 
Go
... to the movies 
Performanc e


Time

beginTime
:
Performanc e


" tonight " 
beginTime : Time 
cinema : any

cinema : any




...
 Covering ...
Nancy 18.10.2002
``Formal´´ Definition Overlay
• Let
T
– co be covering
– bg be background
• Step 1:
– Assimilate(co,bg)
• Step 2:
bg
– Overlay(co,assimilate(co,bg))
• If co and bg are frames: recursion
• If co is empty: use bg
co
• If bg is empty: use co
• If conflict: use co
© Alexandersson; DFKI
Nancy 18.10.2002
Domain Models with Multiple Inheritance
• Assimilate(co,bg)
T
– Compute the set of minimal upper
bounds (MUB)
– Specialize the MUBs
– Unify the specialized MUBs
MUB
MUB
co
bg
• Overlay remains untouched
© Alexandersson; DFKI
Nancy 18.10.2002
Processing Partial
User Contributions
• Our Dialogues contain a lot of partial contributions
– ``Three´´
– ``Three o‘clock´´
– ``At three o‘clock´´
– ``At three [] o‘clock´´
– ``Later´´
– ``Information []´´
– ...
• Requirement
– Advanced discourse modelling and action planning
– Bidirection communication DiM - AP
© Alexandersson; DFKI
Nancy 18.10.2002
Discourse Modelling - Three-Tiered
Context Representation
Domain layer
Discourse layer
DO1
DO2
DO3
Modality layer
DomainObject2
DomainObject1
LO1
show
LO2
list
...
...
GO1

DO10
DO11
DO12
LO3
LO4
LO5
DO9
...
heidelberg
System: Here [pointing gesture] I show
you a list of films running in Heidelberg.
© Alexandersson; DFKI
LO6
reserve ticket first
User: Reserve a ticket for
the first!
Nancy 18.10.2002
Action Planner – Regression Planning
•
AP processes plan operators
–
•
Uniform description of mixed initiative with user or applications based on
Games and Moves. Example:
•
User: request – System: response
•
System: request – Application: response
Processing steps
1.
Constructs a plan from plan operators
–
2.
Interprets the plan
–
•
A Plan specifies a partial order of games
Execute the games
Expectations are published during execution of the plan
–
Guide the interpretation of input
© Alexandersson; DFKI
Nancy 18.10.2002
Action Planning - Regression planning
provides(C)
needs()
provides(A)
needs(C)
provides(X)
needs(A,B)
Goal
provides()
needs(X,Y)
provides(B)
needs(D)
successor
states
provides(D)
needs(E)
Current State
provides(?)
needs(?)
© Alexandersson; DFKI
provides(F,G)
needs(H)
provides(Y)
needs(D)
condition(D=x)
For Example: Slot D=y
Nancy 18.10.2002
AP - Example Plan: VCR
VCR_getChannel
VCR_getStartTime
Goal: VCR_record
VCR_checkMedium
VCR_rewindMedium
© Alexandersson; DFKI
VCR_getEndTime
Nancy 18.10.2002
Action Planning
• Initiation of sub-dialogues to gather necessary information
• Order proposed by the system, but the user may not follow
the trail
• Several cases of user responses to requests:
– Providing a matching answer,
– Supplying information not (yet) asked for,
– Changing already established information
– Acting uncooperatively or changing goals
© Alexandersson; DFKI
Nancy 18.10.2002
Expectations
• When AP initiates a sub-dialogue, it expects a certain answer
• An Expectation structure is published to help the analysis modules
– Expected Slots: corresponding to the request
– Possible Slots: plausible in context
– Filled Slots: already provided
– The currently active Application
• Allows for a fine-grained distinction between types of possible
utterances
• Discourse Modelling can employ different rankings for each category
© Alexandersson; DFKI
Nancy 18.10.2002
Integration of Partial
• Partial Utterances are analyzed as subobjects
– validation and enrichment include the integration into the
preceding context (to achieve a coherent discourse)
– we consider three types of partial utterances:
1. The system has the initiative and the user responds elliptically to a
system request
2. An elliptical user utterance does not correspond to a request but
can be interpreted in context
3. Misinterpretation or non-cooperative user behaviour
© Alexandersson; DFKI
Nancy 18.10.2002
Integration of Partial - Possible Situations
User Initiative
System
Initiative
expected,
Expected slot
NA
unify,
plan continues
Possible slot
Filled slot
Other
© Alexandersson; DFKI
plausible,
plausible,
unify,
unify,
replanning
replanning
plausible,
plausible,
overlay,
overlay,
replanning
replanning
User contribution
DiM processing
AP processing
implausible – recover strategies
Nancy 18.10.2002
DiM - Processing User Responses
• User provides expected slots
– Bridging:
• a new application object is created (of same type as the current
application object)
• for each subobject in the hypotheses its type is compared to the
expected slots
– matching subobjects are integrated into the new application object
– remaining subobjects are interpreted as plausible partial utterances
(next slide)
– Overlay:
• The extended application object is overlayed over the application object
currently in focus
© Alexandersson; DFKI
Nancy 18.10.2002
Processing Plausible Partial Utterances
• User provides possible slots
– processed like expected slots (but get a score penalty)
• User changes filled slots
– search for a discourse object DO of the same type in the discourse
context
– create application object AO1 of the same type as the application
object AO2 within which DO was mentioned
– overlay AO1 over AO2
– gets a score penalty based on position in discourse history
• If integration fails, all subobjects are passed on
unchanged (integration fails)
© Alexandersson; DFKI
Nancy 18.10.2002
Example: Expected and Possible Slots
• (1) User: I want to record a film.
(2) System: When should I start recording?
(3) User: 1:30pm on channel two.
• Analysis of (1) contains the setting of a goal VCR_record (empty
application object of type VCR introduced)
• AP triggers presentation (2) and publishes an expectation containing:
– VCR as the type of the current application object
– VCR_startTime as the (only) expected slot
– some possible slots (e.g., VCR_endTime and VCR_channel)
– and maybe some filled slots (e.g. VCR_mediumPresent)
• Analysis of (3) contains two subobjects
– the expected slot is filled by “1:30pm”
– but also one possible slot is filled by “channel two”
© Alexandersson; DFKI
Nancy 18.10.2002
Example: Filled Slots
(1) User: What is currently running on TV?
(2) System: The following [Smartakus points at a list of
programmes] programmes are currently running.
(3) User: And tonight?
© Alexandersson; DFKI
Nancy 18.10.2002
Example: Filled Slots
Domain layer
Discourse layer
Modality layer
hypothesis



broadcast








beginTime time 


" now"






domainObject 
avMedium





avMedium









...
 

...



time

"tonight"


DO1
DO2
DO3
DO4
LO1
LO2
LO3
LO4
show program currently
© Alexandersson; DFKI
hypothesis



broadcast







 
beginTime time

" tonight"  



 
domainObje
ct



avMedium





avMedium









...
 

...



tonight
Nancy 18.10.2002
Conclusion
• Uniform, flexible and robust mechanism for interpreting
possible partial utterances
• Pragmatical classification of partial utterances from the
point of view of the action planner
• Modality independent
• Approach integrates seamlessly with other aspects of
discourse processing, e.g. anaphora resolution
• Implemented and running in, e.g., a large multimodal
dialogue system
© Alexandersson; DFKI
Nancy 18.10.2002
Papers
• Jan Alexandersson and Tilman Becker. Overlay as the Basic Operation for
Discourse Processing in a Multimodal Dialogue System. In: Proceedings of
the IJCAI Workshop ``Knowledge and Reasoning in Practical Dialogue
Systems,´´ Seattle, 2001.
• Norbert Pfleger, Jan Alexandersson, and Tilman Becker. Scoring Functions
for Overlay and their Application in Discourse Processing. In Proceedings of
``KONVENS 2002,´´ Saarbrücken, Germany, 2002.
• Markus Loeckelt, Tilman Becker, Norbert Pfleger and Jan Alexandersson
Making Sense of Partial. In: Bos, Foster & Matheson (eds): ``Proceedings of
the sixth workshop on the semantics and pragmatics of dialogue (EDILOG
2002),´´ Edinburgh, UK, Pages 101-107.
• Jan Alexandersson and Tilman Becker. The Formal Foundations Underlying
Overlay. Submitted to IWCS5
© Alexandersson; DFKI
Nancy 18.10.2002
Thank
you
very much
your
attention!
Merci
infiniment
de for
votre
attention!
Babelfish.altavista.com 
© Alexandersson; DFKI
Nancy 18.10.2002