Multimodal research at
UIAH
Kristiina Jokinen
MediaLab
University of Art and Design Helsinki
[email protected]
15-Nov-2002
MUMIN Workshop
1
University of Art and Design




Education and research in the fields of design,
new media, audiovisual communication, art
education, arts
Largest art school in Scandinavia
Nearly 1600 students, 15% from abroad
Lume, the Finnish centre for media research
and development, was opened in 2000
•
Departments of film and television, design for theatre,
film and television, and new media
15-Nov-2002
MUMIN Workshop
2
Media Lab


Formed in 1993
Explore, discover and comprehend the new digital
technology
•
•
•


Impact in society
Possibilities for communication, interaction and expression
Challenges to new media and information design
2-year masters programme
•
•
MA in New Media (full time)
New Media program for professionals (flexible study method)
20 full-time, 15 professionals, and approx. 30 minor
subject students annually to the MA programmes
15-Nov-2002
MUMIN Workshop
3
Adaptive Systems for Complex
Interaction

Research on natural interaction between humans
and computers
•
•
•

Apply machine-learning techniques to dialogue
processing
•

Interaction strategies, cooperative response planning
Various input modalities
Concepts, models
Compare and test applicability of the techniques
Support Design-for-all principles in designing
intelligent interfaces
15-Nov-2002
MUMIN Workshop
4
Human-computer interaction


Computer as a tool
•
•
Passive and transparent
Supports the human goals, human control
Computer as an agent
•
•
•
•
•
Models of beliefs, desires, intentions (BDI)
Intelligent software mediating between the human and
an application
Cooperation, negotiation
Complex interaction
Multimodal communication
15-Nov-2002
MUMIN Workshop
5
Projects



USIX-Interact: Natural Interaction and Adaptive
Methods http://www.mlab.uiah.fi/interact/
DUMAS: Dynamic User Modelling for Adaptive
Speech Interfaces http://www.sics.se/dumas/
MUMMI: Multi-Modal Museum Interfaces (Study
project together with Marjo Mäenpää and Antti
Raike, Design for All, Virtual Art Exhibition)
http://mlab.uiah.fi/mummi/
15-Nov-2002
MUMIN Workshop
6
Natural interaction
Language that suits to computers
 Language that humans use to
communicate
=> Language that humans and computers
use when interacting with each other
- Different ? How?
- How it emerges from interaction?

15-Nov-2002
MUMIN Workshop
7
Interact: key aspects for adaptivity





Conversational ability
• Dialogue modelling and natural communication
Learning systems
• Various methods and techniques
• Various interface techniques (speech, text, map)
Language technology
• Finnish and multilingual
Agent-based architecture
• Jaspis development platform
http://www.mlab.uiah.fi/interact/
15-Nov-2002
MUMIN Workshop
8
Interact Partners




University of Art and
Design, Media Lab
University of Helsinki,
Language Technology
University of Tampere,
TAUCHI unit
Helsinki University of
Technology, Neural
Networks Research
Centre
15-Nov-2002






Fujitsu Invia oyj
Tecnomen oyj
Lingsoft oy
The Arla Institute
Finnish Association
for the Deaf
Finnish Technology
Agency
MUMIN Workshop
9
DUMAS -
Dynamic Universal Mobility for
Adaptive Speech interfaces

EU 5th framework R&D project
•
•
•
•
•
•
•
•

Swedish Institute of Computer
Science
UIAH, Media Lab
University of Tampere, TAUCHIunit
UMIST, Manchester
ETEX, Frankfurt
Conexor oy, Helsinki
Timehouse oy, Helsinki
KTH, Stockholm
http://www.sics.se/dumas
15-Nov-2002
MUMIN Workshop
10
DUMAS Objectives
Interactive
•
•
•
•
•
•
email application
Dynamic – various capabilities
Universal – various situations and formats
Mobility – various mobile applications for
Adaptive – learning systems
Speech – spoken and text input
Interfaces – intelligent interaction
15-Nov-2002
MUMIN Workshop
11
Goals


Development of speech-based applications
•
•
Main application: AthosMail
•
•
•

multilingual: Finnish, Swedish, English
adapts to the user’s needs and habits
Experiments at the end of the project:
Athos-radiostation
Athos-text-TV
UIAH responsibility: user modelling
components
http://www.sics.se/dumas/
Challenges for User Modelling

Learn from the user-computer interaction those
aspects that are important in making the use flexible
and enjoyable:
•
•
•


cognitive load
speaking habits
dialogue strategies
User Model is involved in almost all decision making
from speech recognition to dialogue management to
speech synthesis
Classification and learning methods e.g.
•
neural networks, Bayes-nets, reinforcement learning
http://www.sics.se/dumas/
Learning via Interaction

Situation: language is activity between rational agents (cf.
Allwood)
•



Contact + perception + understanding + reaction
Task: achieve a communicative goal
•
•
maximise mutual comprehensibility
minimise ambiguity
Constraints:
•
•
language is possessed by a group of agents => cooperation
limited resources => adaptation to new situations
Reinforcement learning
•
•
agent takes an action a, finds itself in a state s, and receives a
reward r
The task is to find a policy that maximizes the agent's reward in an
environment
15-Nov-2002
MUMIN Workshop
14
Adaptive multimodal interfaces

What and when to adapt?
User-centred parameters:

Environmental parameters:

• Habits and preferences
• Attitudes and intentions
• E.g. speech recognizer accuracy
15-Nov-2002
MUMIN Workshop
15
Action paths for an average user
A
Greet
Read
Dictate
0.949
1.395
Dictate
1.407
Farewell
0.979
Listen
End
1.682
0.858
0.059
1.267
B
Farewell
1.000
15-Nov-2002
Move to folder
0.842
Farewell
0.840
...
Send
Dictate
Prompt for
action
0.947
Send
1.371
Dictate
1.550
Listen
0.939
Cancel
0.943
End
0.143
Farewell
1.050
Greet
MUMIN Workshop
16
List messages
0.821
MUMMI: Multimodality and
Museum Interfaces



Study project together with Marjo Mäenpää, Antti Raike
Cooperation with the Finnish National Gallery: Marjatta
Levanto, Riikka Haapalainen
New ways of relating the arts that are both visually
interesting and accessible in terms of contents:
• Virtual art exhibition, interactive guiding of the user
•
•

through the exhibition
Text, speech, signing avatar
Design for all
Accessibility to the virtual visitors on museum web sites
15-Nov-2002
MUMIN Workshop
17
Design for All





Places the user in the centre (user-centred design)
Cognitive factors (perception, memory, learning,
problem-solving, etc.) as they come into play during
interactions with things
Usefulness: what is relevant
•
do the functions, information, etc., match what the user actually
needs?
Usability: ease-of-use
•
a simple concept, but not always easy or intuitive to implement
New ways to interact with computers?
15-Nov-2002
MUMIN Workshop
18
Other Multimodal Projects at MediaLab

QuiQui’s Giant Bounce
(Kukakumma Muumaassa)
• Perttu Hämäläinen, Johanna Höysniemi
• http://www.kukakumma.net/
• use your body to play
• interaction with your body
• child-centred design
15-Nov-2002
MUMIN Workshop
19
Other Multimodal Projects at MediaLab

Cinemasense (Elokuvantaju)
• Antti Raike
• http://elokuvantaju.uiah.fi/
• web portal for film production (learning
•
•
material)
organise cinematic concepts in the student’s
mind
especially sign language
15-Nov-2002
MUMIN Workshop
20
Other Multimodal Projects at MediaLab

Experimental, affective interfaces
• Jukka Ylitalo, Heidi Tikka
• http://mlab.uiah.fi/eia/
• interactive media, media and art
15-Nov-2002
MUMIN Workshop
21
Burning Issues




Conversational interfaces
•
Dialogue processing: turn taking, feedback, repairs, nonverbal elements
Architectures
• Learning in agent-based architectures
• How to plug-&-play?
Processing techniques
• Cognitive models of language understanding
• Machine learning (supervised vs. unsupervised)
Design for all
• Usability: for whom, why, what
• Evaluation
15-Nov-2002
MUMIN Workshop
22
References









Cinemasense http://elokuvantaju.uiah.fi/
DUMAS http://www.sics.se/dumas
Experimental interfaces http://mlab.uiah.fi/eia/
Interact http://www.mlab.uiah.fi/interact/
MUMMI http://mlab.uiah.fi//mummi/
QuiQui http://www.kukakumma.net/
Jokinen et al. (2002). Adaptive Dialogue Systems – Interaction with
Interact. Proceedings of the 3rd SIGDial Workshop, Philadelphia, US.
Jokinen, K., J. Rissanen, H. Keränen, and K. Kanto (2002). Learning
interaction patterns for adaptive user interfaces. The 7th ERCIM UI4All
Workshop, October, Paris, France.
Jokinen, K. and A. Raike (2002). Multimodality – the latest technology
and visions and demands for the future. Multimodality IT-seminar,
Castberggård, Denmark.
15-Nov-2002
MUMIN Workshop
23