User Interface Engineering Team
CLIPS-IMAG Laboratory, University of Grenoble
Mixed Reality:
A Model of Mixed Interaction
Céline Coutrix, Laurence Nigay
Introduction: Mixed Systems
• Merging
Physical and
Digital
Worlds
QuickTime™ and a
Photo - JPEG decompressor
are needed to see this picture.
• In many
application
fields
1
Introduction: Designers Needs
A framework for :
Understanding and capitalizing
on existing Mixed Systems
Descriptive power
Creating designs
Generative power
Assessing multiple design
alternatives and increasing
usability
Comparative power
2
Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
3
Outline
• Mixed Interaction Model
• Mixed object
• Mixed interaction
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
4
Example
QuickTime™ et un
décompresseur codec YUV420
sont requis pour visionner cette image.
5
Mixed Interaction Model: Mixed Object
mixed object
tool
task object
6
Mixed Interaction Model: Mixed Object
physical object
physical
properties
mixed object
physical
properties
?
digital
properties
digital object
digital
properties
• Interaction modality m = (d, l)
• speech = (microphone, natural language)
7
Mixed Interaction Model: Mixed Object
• Linking modality
mixed object
input
device
acquired
physical
data
input
language
physical
properties
digital
properties
output
device
generated
physical
data
output
language
8
Mixed Interaction Model: Mixed Object
QuickTime™ and a
Photo - JPEG decompressor
are needed to see this picture.
mixed drawing
camera
image
computer
vision
physical
properties
shapes
projector
image
selection
9
Mixed Interaction Model: Mixed Object
QuickTime™ et un
décompresseur codec YUV420
sont requis pour visionner cette image.
a music bottle
physical
properties
screen
table
loud
speakers
lighting
colored
sequence
pitch and
volume
analysis
audio
sequence
identity
music
part
10
Mixed Interaction Model: Mixed Object
mixed object
physical
properties
tool
Linking
modalities
digital
properties
task object
11
Mixed Interaction Model: Mixed Interaction
tool
elementary
task
action
response
reaction
task object
feedback
12
Mixed Interaction Model: Mixed Interaction
•
Interaction modality m = (d, l)
13
Mixed Interaction Model: Mixed Interaction
mixed tool
input
device
physical
properties
output
device
acquired
physical
data
input
language
generated
physical
data
output
language
digital
properties
interaction interaction
output
input
language language
task object
14
Mixed Interaction Model: Mixed Interaction
QuickTime™ and a
Photo - JPEG decompressor
are needed to see this picture.
QuickTime™ and a
Photo - JPEG decompressor
are needed to see this picture.
15
Mixed Interaction Model: Mixed Interaction
mixed tool
camera
paper
button
image
computer
vision
button
pressed
word,
place
on drawing
mixed tool
translation
drawing
16
Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
17
Descriptive Power
• Schemes for classification and better
understanding of mixed systems
18
Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
19
RAZZLE: Case Study
20
RAZZLE: Case Study
mixed piece
location
sensor
physical
properties
numeric
values
geometric
transforma°
orienta°
sensor
numeric
values
geometric
transforma°
head
mounted
display
image
3D
projection
Identifier
Location
Orientation
21
Generative Power
mixed tool
?
?
?
?
?
22
RAZZLE: Gestural Modality
QuickTime™ et un
décompresseur DV - PAL
sont requis pour visionner cette image.
23
RAZZLE: Gestural Modality
mixed tool
camera
physical
properties
image
computer
vision
recognized
gestures
translation
24
RAZZLE: Tactile Modality
QuickTime™ et un
décompresseur DV - PAL
sont requis pour visionner cette image.
25
RAZZLE: Tactile Modality
mixed tool
touchpad
physical
properties
data
input
language
click
translation
26
Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
27
Comparative Power
• Criteria within the model for assessing
multiple design alternatives
• Observability
• Spatial continuity
28
Comparative Power: Observability
mixed tool
input
device
physical
properties
output
device
acquired
physical
data
input
language
generated
physical
data
output
language
digital
properties
interaction interaction
output
input
language language
task object
input
device
physical
properties
output
device
acquired
physical
data
input
language
generated
physical
data
output
language
digital
properties
29
Comparative power: Observability
mixed tool
camera
physical
properties
image
input
language
recognized
gestures
translation
30
Comparative Power: Observability
mixed tool
touchpad
physical
properties
data
input
language
click
translation
31
Comparative Power: Spatial Continuity
mixed tool
input
device
physical
properties
output
device
acquired
physical
data
input
language
generated
physical
data
output
language
digital
properties
interaction interaction
output
input
language language
task object
input
device
physical
properties
output
device
acquired
physical
data
input
language
generated
physical
data
output
language
digital
properties
32
Comparative Power: Spatial Continuity
mixed tool
camera
image
computer
vision
physical
properties
recognized
gestures
translation
mixed piece
location
sensor
physical
properties
orienta°
sensor
head
mounted
display
numeric
values
geometric
transforma°
numeric
values
geometric
transforma°
image
3D
projection
Identifier
Location
Orientation
33
Comparative Power: Spatial Continuity
mixed tool
touchpad
data
input
language
physical
properties
click
translation
mixed piece
location
sensor
physical
properties
orienta°
sensor
head
mounted
display
numeric
values
geometric
transforma°
numeric
values
geometric
transforma°
image
3D
projection
Identifier
Location
Orientation
34
Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
35
Conclusion
• Definition of a Mixed Interaction Model which
• Capitalizes existing approaches to describe
mixed systems
• Allows designers to create new interaction
modalities
• Allows them to choose between multiple design
alternatives
36
Future Work
• Improving comparative power by identifying more
criteria
37
Future Work
• Providing a development environment
based on ICARE
38
User Interface Engineering Team
CLIPS-IMAG Laboratory, University of Grenoble
Mixed Reality:
A Model of Mixed Interaction
Céline Coutrix, Laurence Nigay