EyesWeb XMI
Multimodal data recording,
playing and analysis
M. Mancini, Università di Genova (Italy)
[email protected]
Goal
1. record multimodal data:
• video (rgb or silhouette)
• audio
• sensor
2. play multimodal data
3. analyze multimodal data in real-time
• silhouette (if available)
• Contraction Index
• Quantity of Motion
• sensor
• energy
• smoothness
1. Recording
K-th audio buffer
N-th video frame
media file
writer
csv file
writer
N
tsv file
writer
M-th sensor frame
current time
clock
current time
2. Player
media file
reader
N-th video frame
N
csv file
reader
current time
tsv file
reader
M-th sensor frame
3. Analysis: Expressive Features
Global measures depending on full body movement (e.g.,
body orientation, overall motion direction).
Measures from psychological research, e.g., Boone &
Cunningham’s amount of upward movement.
Cues from Rudolf Laban’s Theory of Effort, e.g.,
directness, impulsiveness.
Cues derived from analogies with audio analysis, e.g.,
Inter Onset Intervals, frequency analysis.
Kinematic measures such as velocity, acceleration,
average and peak velocity and acceleration.
Silhouette Motion Images
• SMIs (Silhouette Motion Images) carry information on
variations of a blob (usually the silhouette of a user) in
the last few frames.
SMIs and Motion Index
• SMIs are different with respect to MHIs, since they do not
include the last silhouette, i.e., the current posture.
• Thus, SMIs carry information about the movement
detected by the video-camera in the last n frames.
• The SMI area can be therefore considered as a measure of
the detected amount of motion.
• The SMI area, normalized by the silhouette area, is called
Motion Index (or Quantity of Motion).
Motion_Index[t] = Area(SMI[t, n]) / Area(Silhouette[t])
• Note that this is an approximated measure: e.g., movement
against the video-camera is not detected.
Weighted Motion Index
• Motion Index can also be computed by differently
weighting pixels in the input blob.
• So, it is possible to compute a Motion Index where
pixels near to the centre of the blob weight more
than pixels near to the contour (i.e., something
more similar to the physical concept of
momentum).
• Or it is possible to compute a Motion Index where
pixels near to the contour weight more than pixels
near to the centre of the blob (i.e., a more
perceptual measure where limbs have a stronger
impact on perception of movement).
Contraction Index
• Contraction Index is a measure of how the user’s
body uses the space surrounding it.
• The simplest way to compute it is the ratio
between the area of the blob and the area
of the bounding rectangle.
CI = Area(Blob) / Area(Bounding rectangle)
Silhouette Features Extraction
media file
reader
N-th silhouette frame
blob
extractor
QoM
extractor
current time
CI
extractor
csv file
writer
Sensor Features Extraction
tsv file
reader
jerk (1/fluidity)
acceleration
differentiate
energy
integrate
current time
csv file
writer