MUMIS
User Group Workshop
P. Wittenburg
Max-Planck-Institut für Psycholinguistik
Nijmegen
Agenda
• 11.00
•
•
•
•
•
•
11.45
12.30
13.15
14.00
15.00
15.30
Introduction
ASR Aspects
Peter Wittenburg
Helmer Strik
NLP/IE Aspects
Thierry Declerck
Demo of Prototype
Lunchbreak
Potential of MUMIS Technology
Typical Questions
Other Points of Interest
End
Peter Wittenburg
Discussion
Discussion
Discussion
Objectives
MUMIS is about
• Technology development to automatically index
lengthy multimedia recordings
Find and mark interesting scenes
• Technology development to exploit indexed
multimedia archives
Search for interesting scenes and play them via Internet
• Test Domain: Soccer Games / UEFA Tournament 2000
Off-line Tasks
Indexing by
• Automatic Speech Recognition (Radio/TV Broadcasts)
Transfer the speech signals into texts automatically
• Natural Language Processing
Take all texts (newspapers, speech transcript, tickers, ...)
and determine interesting scenes
• Create a database with formal annotations
• Use video processing to adjust time marks
Off-line Tasks
Newspaper
Newspaper
Newspaper
Newspaper
Text
Text
Text
Texts
3 Languages
RadioCommenting
Commenting
Radio
Radio Commenting
Audio Commenting
(TV, Radio)
3 Languages
Languages
33 Languages
3 Languages
Newspaper
Newspaper
Newspaper
Close
Captions
Text
Text
Text
3 Languages
indexed video recording
Freekick
Goal
Pass
Defense
17 min
18 min
24 min
28min
Foul
Freekick
Kohler
Basler
Dribbling
Matthäus
Basler
25 m
Wörns
Bierhoff
25 m
60 m
Event
End-Signal
Offside
Goal-Kick
Stop-ofAttack
Kick-Off
Chest Ball
Wall Pass
Dribbling
Corner
Own-Goal
Throw-In
Nice Ball
Foul
Freekick
Yellow Card
Halftime
Signal
Hands Ball
Head Ball
Top-Post-Ball
Pass
Post Ball
Save
Red Card
Bicycle Kick
Sliding
Penalty
Goal
Shot-on-Goal
Keeper-Save
Injury
Change
ID
ID
ID
ID
Time Player1 Player2 Location Destination Cause Consequence Count Ref
Time
Ref
Time Player1
Location
Ref
Time Player1 Player2 Location Destination
Ref
ID Time Player1
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
Time
Time
Time
Time
Time
Time
Time
Time
Time
Time
Time
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Time
Time
Time
Time
Time
Time
Time
Time
Time
Time
Time
Time
Time
Time
Time
Player2
Player2 Location Destination
Player2 Location Destination
Location
Player2 Location Destination
Location
Player2 Location Destination
Player2 Location Destination
Player2 Location
Player2 Location Destination
Player2
Cause Consequence
Ref
Ref
Consequence
Ref
Consequence
Ref
Consequence
Ref
Cause Consequence
Ref
Cause
Score Ref
Cause Consequence
Ref
Consequence
Ref
Cause Consequence
Ref
Cause Consequence
Ref
Cause
Ref
Ref
ID Time
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
ID
Location
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Player1
Location
Cause Consequence
Ref
Player2 Location Destination
Consequence
Ref
Location
Cause
Ref
Player2 Location Destination
Consequence
Ref
Location
Cause
Ref
Player2 Location
Cause
Ref
Player2
Cause Consequence
Ref
Location Destination
Consequence
Ref
Player2 Location
Consequence
Ref
Player2
Cause Consequence
Ref
Player2
Cause
Score Ref
Player2 Location
Cause Consequence
Ref
Player2 Location
Cause
Ref
Player2 Location
Cause Consequence
Ref
Player2
Cause
Ref
Event
Table
On-line Tasks
Searching and Displaying
• Search for interesting events with formal questions
Give me all goals from Overmars shot with his head in 1. Half.
Event=Goal; Scorer=Overmars; Cause=Head; Time<=45
• Indicate hits by thumbnails & let user select scene
• Play scene via the Internet & allow scrolling
Of course: slow motion, fast play, start/stop, etc
On-line Tasks
Knowledge Guided
User Interface
&
Search Engine
Freekick
Goal
Pass
Defense
17 min
18 min
24 min
28min
Foul
Freekick
Kohler
Basler
München - Porto
1996
Matthäus
Basler
Wörns
Bierhoff
25 m
München - Ajax
1998
Dribbling
25 m
Deutschland - Brasilien
1998
Play
Movie
Fragment
of that Game
60 m
On-line SW Architecture
first interaction with Web-server
download of applet
contact between applet and
distributed Java server via
RMI
contact via media server via RMI
and media streaming
annotations from DB or file
server
JMF serves for merging of
streams on client
Abstract Corpus Model
this kernel is operational
(distributed and local)
On-line HW Architecture
Media Server
RAID
100Mbps
MM Stations
1Gbps
FC Switch
Tape
Library
3.5 TB
GB
Switch
Gb-Switch
Media Server
Internet
2nd location
Mirror site
Tape
Library
25 TB
• efficient & reliable storage management
(near-line capacity, media change, 2. Location)
• high storage capacity (n TB, 1 h MPEG1 = 1 GB)
• powerful media servers / powerful network
Notebook
Gain
Current Procedure
MUMIS Procedure
Manual Video Annotation
Integration Central DB
Automatic Video Annotation
and DB Integration
Query via PC
Query via PC
Results on PC
Contact Video Archive
Get Video Tapes
Search on Tape on VCR
Results on PC
And
Select & Play
Segment & Play
• What gets lost? Is it necessary?
• Potential: direct Internet Service, less dependencies
Formal Aspects
• Start:
• Duration:
• EC-Funds:
1.7.2000
30 Months
2.4 Mio Euro
Consortium
•
•
•
•
•
•
CTIT
TSI
DFKI
MPI
DCS
ESTEAM
• VDA
University of Twente, Enschede, NL NLP/IE
University of Nijmegen, Nijmegen, NL
Saarbrücken, D
Nijmegen, NL
University of Sheffield, UK
Gothenburg, SE (location Athens, GR)
Hilversum, NL
ASR
NLP/IE
Online Part
NLP/IE
Translation
Software
Dissemination
Acknowledgements
• UEFA
• DFB, FA, KNVB
• EBU, WDR, NOS