ICMC | SMC 2014 - ATHENS
GREECE
SEPT/2014
!
T O P O S T O O L K I T F O R P U R E D ATA
exploring the spatial features of dance gestures for interactive musical applications
LUIZ NAVEDA
S TA T E U N I V E R S I T Y O F M I N A S
GERAIS (BRAZIL)
I V A N I S A N TA N A
FEDERAL UNIVERSITY OF
BAHIA (BRAZIL)
CONTEXT
accelerometers
MOTION
CAPTURE
SYSTEMS
MOTION
CAPTURE
D ATA
<100 USD
infrared
3D image
<300 USD
infrared
3D marker
>5000 USD
PROBLEMS
• How to extract realtime high-level information from
motion capture data of music and dance performances?
• Imposed Cartesian coordinates ➜ meaningful feature
descriptors for dance and music?
• Complex full body datasets ➜ simple and accessible
motion descriptors
• “Ad-hoc” programming routines ➜ fast prototyping and
mapping
R E P R E S E N TAT I O N O F M U S I C A N D
DANCE GESTURE
Becking curves
1928
Truslit
1938
R E P R E S E N TAT I O N O F M U S I C A N D
DANCE GESTURE
Becking curves
1928
Truslit
1938
C O G N I T I O N O F S PA C E A N D G E S T U R E
Paillard (1991)
Morphokinetic:
Deals with shape or form regardless of spatial
location
Topokinetic:
Deals with spatial location and spatial
intentionality regardless of shape
C O G N I T I O N O F S PA C E A N D G E S T U R E
Paillard (1991)
Morphokinetic:
Deals with shape or form regardless of
spatial location
Leman & Naveda (2010)
Topokinetic:
Deals with spatial location and spatial
intentionality regardless of shape
Naveda & Leman (2010)
Naveda & Leman (2010)
TGA
T O P O L O G I C A L G E S T U R E A N A LY S I S
Topokinetic:
Deals with spatial location and spatial
intentionality regardless of shape
Naveda & Santana (2010)
WTGA
W I N D O W E D T G A A N A LY S I S
• Topologies are defined by previous and next movement trajectories
XYZ
-0.5 sec +0.5 sec
-0.5 sec
Short-Term Memory
Preceding gesture
+0.5 sec
Intentionality
Upcoming gesture
TOPOS ALGORITHM
Projection onto a point cloud (ID)
Code:
First inlet: gemlist
new sample
...
gemhead
6
Second inlet:
x y z floats
pack f f f
5
point cloud
4
Clo
c
k
topos ID XYZ 100 10
3 2 1
6
xyz
trajectories
[topos]
Transformations
Topos-Feature
extraction
Relationship
Sampling period
3D data
features
Geom-Feature
number of points
Symbolic link extraction
Transformation
topos ID xyz1 100 20
topos.quality
“ T O P O S ” F E AT U R E S
TOPOS LIBRARY
D ATA F L O W A N D
MANAGEMENT
V I S U A L I Z AT I O N
synapse
skeleton
F E AT U R E
DESCRIPTION
R E L AT I O N S H I P S
osceleton
Point clouds
Point clouds
points and point clouds
Points (2,4)
points
TOPOS LIBRARY
• INPUTS: KINECT
z
ories
(OSCELETON, SYNAPSE),
MOCAPTOOLBOX* OR
R E C O R D E D D ATA
• 11 ABSTRACTIONS/OBJECTS
• E N T I R E LY W R I T T E N I N P U R E
Topos-Feature
[topos]
D
ATA
extraction
Transformations
Relationship
features
Geom-Feature
extraction
mation
eature
action
topos ID xyz1 100 20
topos.features ID
topos.quality
topos.geo.2 ID xyz1 xyz2
topos.geo.4 ID xyz1 xyz2 xyz3 xyz4
tion of
nships
topos.rel.regions ID IDtopos1 IDtopos2 0.3
topos.rel.points ID xyz1 xyz2 0.3
topos.rel.pointRegion ID xyz1 IDtopos 0.3
Tools
topos.skeleton 1
topos.rec
topos.rec.slave
P L AY / R E C
V I S U A L I Z AT I O N
A N A LY S I S
F E AT U R E R E T R I E V A L
SYNTHESIS
gemhead
Distances from lhand to the
point cloud of rhand are used
as magnitudes of oscilators
in addirive synthesis
freeze $1
topos rhand1 1/r_hand/pos 30 50
topos.rel.pointRegion hands l_hand/pos rhand1 0.3
route velocity distanceFrom
t b f
random 24
d(n)
pack f f
+ 60
mtl/additive~
d
n points
dac~
list-math / 3
C A S E : D A R M S TA D 5 8 ' ( 2 0 1 3 - 1 4 )
AUDIO, VIDEO, CONTROL
BRAZIL
MEXICO
S PA I N
Receiving 2 skeletons
Sending 1 skeleton
CASE: MIRADAS DO CAOS II (2014)
kinect
blob detection
(pix_freenect)
Topos Library
F I N A L C O N S I D E R AT I O N S
• Contributions
• Minimal set of tools for high-level
descriptors
• Provide access to a number of features and
gestural relationships
• Problems
• Contextual features X strict synchronisation
• Dance traditions X expertise in controlling
sound
Thanks!
Library, tutorials and documentation
www.naveda.info