Distributed Multimodal
Synchronization Protocol (DMSP)
Chris Cross
IETF 65
March 20, 2006
With Contribution from Gerald McCobb and Les Wilson
March 20, 2006
© 2005 IBM Corporation
DMSP Background
• Result of 4 year IBM R&D effort
• ID to IETF July 8, 2005 by IBM & Motorola
• Reason for contribution
– A standard is needed for mobile device access to network based
services for executing multimodal web applications
– Other protocols may have overlap but do not address all
multimodal interaction requirements
– Other IETF IDs and RFCs:
• Media Server Control Protocol (MSCP)
• LRDP: The Lightweight Remote Display Protocol (Remote UI BoF)
• Media Resource Control Protocol Version 2 (MRCPv2)
• Widex
March 20, 2006
© 2005 IBM Corporation
DMSP Architecture
•
There are 4 DMSP building blocks:
1.
2.
3.
4.
Modalities
Model-View-Controller (MVC) design pattern
View Independent Model
Event-based modality synchronization
March 20, 2006
© 2005 IBM Corporation
DMSP Architecture Building Blocks
1.
Modalities are Views in
the MVC Pattern
•
•
•
GUI, Speech, Pen
Individual browsers for
each modality
Compound browsers for
multiple modalities
Compound Browser
March 20, 2006
© 2005 IBM Corporation
DMSP Architecture Building Blocks
2. Model-View-Controller (MVC) design pattern
•
•
•
Multimodal system can be modeled
in terms of the MVC pattern
Each modality can be decomposed
and implemented in its own MVC
pattern
A modality can implement a view
independent model and controller
locally or use one in the network
(e.g., an IM)
March 20, 2006
© 2005 IBM Corporation
DMSP Architecture Building Blocks
3. View Independent Model
•
•
•
•
•
Enables a centralized model
Modality interaction updates view and
model
Local event filters reflect “important”
events to view independent model
A modality listens to view independent
model for only the events it cares about
Compound clients, centralized control
(IM) as well as distributed modalities all
enabled with a single protocol
March 20, 2006
© 2005 IBM Corporation
DMSP Architecture Building Blocks
4. Event-based synchronization
•
•
•
Compound Client: All
modalities rendered in client
Interactions in one modality
reflected in others thru event
based changes to one or
more model
GUI DOM serves as View
Independent model
March 20, 2006
© 2005 IBM Corporation
DMSP Architecture Building Blocks
4. Event-based synchronization (CONT’D)
•
•
•
•
Distributed Modality: A modality is handled in
the infrastructure
Requires the DMSP for distributing modality
Event based synchronization via View
Independent Model gives a modality
independent distribution mechanism
Enables multiple topographies
–
Compound Client w/ Distributed Modality
–
IM
March 20, 2006
Distributed Modality
© 2005 IBM Corporation
DMSP Design
•
There are 4 abstract interfaces
1.
2.
3.
4.
•
•
•
Command
Response
Event
Signal
Each interface defines a set of methods and
related data structures exchanged between
user agents
Specified as a set of messages
XML and Binary message encodings
March 20, 2006
© 2005 IBM Corporation
DMSP Message Types
1. Signals
•
One-way asynchronous messages used to
negotiate internal processing states
•
•
•
Initialization (SIG_INIT)
VXML Start (SIG_VXML_START)
Close (SIG_CLOSE)
March 20, 2006
© 2005 IBM Corporation
DMSP Message Types
2. Command and control messages
•
•
•
•
•
•
•
•
•
•
Add and remove event listener
(CMD_ADD/REMOVE_EVT_LISTENER)
Can dispatch (CMD_CAN_DISPATCH)
Dispatch event (CMD_DISPATCH_EVT)
Load URL (CMD_LOAD_URL)
Load Source (CMD_LOAD_SRC)
Get and Set Focus (CMD_GET/SET_FOCUS)
Get and Set Fields (CMD_GET/SET_FIELDS)
Cancel (CMD_CANCEL)
Execute Form (CMD_EXEC_FORM)
Get and Set Cookies (CMD_GET/SET_COOKIES)
March 20, 2006
© 2005 IBM Corporation
DMSP Message Types
3. Responses
•
Response messages to commands
•
•
•
•
OK (RESP_OK)
Boolean (RESP_BOOL)
String (RESP_STRING)
Fields (RESP_FIELDS)
•
•
Contains 1 or more Field data structures
Error (RESP_ERROR)
March 20, 2006
© 2005 IBM Corporation
DMSP Message Types
4. Events
•
•
•
Asynchronous notifications between user agents
with a common data structure
Events correlated with event listeners
DOM events
•
•
HTML 4 events
•
•
•
DOMActivate, DOMFocusIn, and DOMFocusOut
Click, Mouse, Key, submit, reset, etc
Error and abort
VXML Done (e.g., VoiceXML form complete)
March 20, 2006
© 2005 IBM Corporation
DMSP Message Types
Events (CONT’D)
4.
•
Form Data
•
•
Recognition Results
•
•
Play back of audio or TTS prompts has started or stopped
Start and stop play back mark
•
•
One or more Result EX data structures with raw utterance, score,
grammar, and semantics
Start and stop play back
•
•
One or more Result data structures with raw utterance, score,
and one or more Field data structures
Recognition Results EX
•
•
One or more Field data structures (GUI or Voice)
TTS encounters a mark in the play text
Custom (i.e., application-defined)
March 20, 2006
© 2005 IBM Corporation
DMSP Conclusion
• A protocol dedicated to distributed multimodal interaction
• Based on the Model-View-Controller design pattern
• Enables both IM and Client based View Independent
Model topographies
• Asynchronous signals and events
• Command-response messages
• Can be generalized for other modalities besides GUI and
Voice
• Supports application specific result protocols (e.g.
EMMA) through extension TBD
• Interested in getting more participation
March 20, 2006
© 2005 IBM Corporation