Dance Movement Patterns Recognition
(Part I)
Israel Solà Cerdán
EEMCS - HMI
1
Acknowledgments
• This project would not has been possible without the
collaboration of our project directors:
– Anton Nijholt
– Dennis Reidsma
– Jordi Carrabina
• Thanks for the received aid of:
– Herwin van Welbergen
– Ronald Poppe
• Thanks to our family, friends, girlfriends (MLG) and
partners.
EEMCS - HMI
2 / 40
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
3 / 40
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
4 / 40
39
Presentation
• Joint project
• The Virtual Dancer
• Obtaining a greater interaction
• Recognizing dance movement patterns
EEMCS - HMI
5 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
6 / 40
39
Goals
1. Starting up The Virtual Dancer application (computer vision part
and artificial intelligent part)
2. Analyzing the application and developing improvements
strategies
3. Recognition of simple dance movements
4. Recognition of dance movement patterns
5. Automatic generation of dance movement search patterns
6. Finding patterns without a reference
EEMCS - HMI
7 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
8 / 40
39
Organization of the Exposition
•
Virtual Dancer application
•
The computer vision system “ParleVision”
•
The work environment with the setup application and the used material
•
The study carried out at the beginning of the project
•
The implemented modifications in the computer vision system “ParleVision”
•
Dance Movement Patterns Recognition (Part II)
–
–
–
–
–
HMM
Recognizing Simple/Complex Movements
Auto Generation of Complex Patterns Graphs
Test Bench
Conclusions
EEMCS - HMI
9 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
10 / 40
39
The Virtual Dancer
EEMCS - HMI
11 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
12 / 40
39
Parlevision
EEMCS - HMI
13 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
14 / 40
39
Material
•
Hardware
– 2 computers with the following properties
• Pentium 4 2.8GHz
• 512 MB of RAM
• Microsoft Windows XP with the SP 2
– 1 camera
• Philips ToUcam XS
•
Software
– ParleVision
– Virtual Dancer
•
Others
– Human Dancer
– A black background
– t-shirts with different colors
EEMCS - HMI
15 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
16 / 40
39
Setup
3
1
2
5
Port 7521
LAN
4
EEMCS - HMI
6
1
Computer Vision PC
2
Artificial Intelligent PC
3
Camera
4
User
5
Background
6
LAN
17 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
18 / 40
39
Interaction Dimensions
TEMPORAL
(When occurs the fact)
GOAL OF THE
INTERACTION
(What we want to do)
ACTIONS
(What to do to get these goals)
CAUSE OF THE ACTION
(Why are we doing this action?)
WAY TO DO THIS ACTION
(How we want to do it)
EEMCS - HMI
19 / 40
39
Interaction Dimensions
• Temporal dimension
–
–
–
–
Before the song
During the song
After the song
Between songs
TEMPORAL
(When occurs the fact)
GOAL OF THE
INTERACTION
(What we want to do)
ACTIONS
(What to do to get these
goals)
CAUSE OF THE ACTION
(Why are we doing this action?)
WAY TO DO THIS ACTION
(How we want to do it)
EEMCS - HMI
20 / 40
39
Interaction Dimensions
• Goal of the interaction
–
–
–
To attract the attention
To entertain
To leave a good memory
TEMPORAL
(When occurs the fact)
GOAL OF THE
INTERACTION
(What we want to do)
ACTIONS
(What to do to get these
goals)
CAUSE OF THE ACTION
(Why are we doing this action?)
WAY TO DO THIS ACTION
(How we want to do it)
EEMCS - HMI
21 / 40
39
Interaction Dimensions
• Actions
– To attract the attention
•
•
•
To say hello
To invite the user
To be seen
– To entertain
•
•
•
•
•
To start the dance
To encourage the user
To do a choreography / chorus
To enrich the dance
To explain some step
– To leave a good memory
• To congratulate
• To say goodbye
TEMPORAL
(When occurs the fact)
GOAL OF THE
INTERACTION
(What we want to do)
ACTIONS
(What to do to get these
goals)
CAUSE OF THE ACTION
(Why are we doing this action?)
WAY TO DO THIS ACTION
(How we want to do it)
EEMCS - HMI
22 / 40
39
Interaction Dimensions
• Cause of the action
– Reactive
– Deliberative
TEMPORAL
(When occurs the fact)
GOAL OF THE
INTERACTION
(What we want to do)
ACTIONS
(What to do to get these
goals)
CAUSE OF THE ACTION
(Why are we doing this action?)
WAY TO DO THIS ACTION
(How we want to do it)
EEMCS - HMI
23 / 40
39
Interaction Dimensions
• Way to do this action
– Verbal
– Non Verbal
TEMPORAL
(When occurs the fact)
GOAL OF THE
INTERACTION
(What we want to do)
ACTIONS
(What to do to get these
goals)
CAUSE OF THE ACTION
(Why are we doing this action?)
WAY TO DO THIS ACTION
(How we want to do it)
EEMCS - HMI
24 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
25 / 40
39
Interaction Tree
Interaction
Before the song
To attract the
attention
During the song
To entertain
To say hello
To entertain
To start the dance
After the song
To attract the
attention
To do a Choreography
/ Chorus
Between songs
To leave a good
memory
To be seen
To entertain
To congratulate
To encourage the
user
Reactive /
Deliberative
Reactive /
Deliberative
Reactive /
Deliberative
Reactive /
Deliberative
Reactive /
Deliberative
Reactive /
Deliberative
Verbal / Non
Verbal
Verbal / Non
Verbal
Verbal / Non
Verbal
Verbal / Non
Verbal
Verbal / Non
Verbal
Verbal / Non
Verbal
To invite to dance
To encourage the
user
To enrich the
dance
To say good bye
To explain some
step
Reactive /
Deliberative
Reactive /
Deliberative
Reactive /
Deliberative
Reactive /
Deliberative
Reactive /
Deliberative
Verbal / Non
Verbal
Verbal / Non
Verbal
Verbal / Non
Verbal
Verbal / Non
Verbal
Verbal / Non
Verbal
To encourage the
user
To be seen
Reactive /
Deliberative
Reactive /
Deliberative
Verbal / Non
Verbal
Verbal / Non
Verbal
EEMCS - HMI
26 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
27 / 40
39
Interaction State Machine
No
To be seen
(2)
CV detection
Waiting a User
(1)
To invite to
dance (4)
Yes
Pad & CV detection
To say hello
(3)
User says Yes
User says No
To start the
dance (6)
To encourage
the user (5)
User says No
User says Yes
To encourage
the user (13)
Random
To start the song
To congratulate
(12)
To finish the
song
To do a
choreography
(8)
Chorus detected
To enrich the
dance (9)
Random
Dance time (7)
To detect passive movements
To explain
some step (14)
To say good
bye (15)
EEMCS - HMI
To encourage
the user (10)
User go out
Absent-minded
user
To be seen
(11)
28 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
29 / 40
39
Variables, Animations & Transitions
• What animations we could use in each state
• New animations
• Conditions required for changing the state
• Available Variables
• New variables
EEMCS - HMI
30 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
31 / 40
39
Possible Improvements
•
Animations
– To say hello moving the hand, looking at the screen center
– To bow to the user
– To clap to the user
– To say hello putting the head up
•
Outputs
– That our dancer could speak
•
Inputs
– To recognize users’ speech
– To recognize head’s movements
•
Pattern Recognition
– Simple movements’ recognition
– Patterns recognition
– Creation of patterns automatically
– Search of Patterns
EEMCS - HMI
32 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
33 / 40
39
New Added Features
EEMCS - HMI
34 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
35 / 40
39
New Created Pipelines
• Save Video Pipeline
• From Video Pipeline
EEMCS - HMI
36 / 40
39
New Created Pipelines
• Pipeline with Radial Histogram
EEMCS - HMI
37 / 40
39
Contents
•
Introduction
–
–
–
•
•
•
The Virtual Dancer
Parlevision
Work Environment
–
–
•
Material
Setup
Previous Study
–
–
–
–
–
•
Presentation
Goals
Organization of the Exposition
Interaction Dimensions
Interaction Tree
Interaction State Machine
Variables, Animations & Transitions
Possible Improvements
The Built System (CV)
–
–
–
New Added Features
New Created Pipelines
New Sent Variables
EEMCS - HMI
38 / 40
39
New Sent Variables
•
•
•
•
•
•
•
•
cumulativeHorActivityNormalized
cumulativeVertActivityNormalized
lHand
rHand
cumulativeRActivityNormalized
r
x
y
EEMCS - HMI
39 / 40
39
Dance Movement Patterns Recognition
(Part II) …
Bedankt voor uw aandacht !
EEMCS - HMI
40 / 40
39