Social Signal Processing
Understanding Nonverbal Behavior
in Human-‐‑‒Human Interactions
A.Vinciarelli
University of Glasgow and Idiap Research Institute
http://www.dcs.gla.ac.uk/~∼vincia
e-‐‑‒mail: [email protected]
1
Outline
•“Socialism” and Big Data
•Social Signal Processing
•The Conflict Example
•Conclusions
2
2
Outline
•“Socialism” and Big Data
•Social Signal Processing
•The Conflict Example
•Conclusions
3
3
“Socialism” and Big Data
90
67.5
45
22.5
0
2004
2005
2006
2007
2008
2009
2010
2011
2012
Events showing the word “social” in their title (from
dbworld, a multimedia retrieval mailing list)
4
4
“Socialism” and Big Data
90
67.5
45
22.5
0
2004
2005
2006
2007
2008
2009
2010
2011
2012
Events showing the word “social” in their title (from
dbworld, a multimedia retrieval mailing list)
4
4
Outline
•“Socialism” and Big Data
•Social Signal Processing
•The Conflict Example
•Conclusions
5
5
Social Signals
Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging
Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009
6
6
Social Signals
Height
Mutual
Gaze
Vocal
Behavior
Forward
Posture
Forward
Posture
Nonverbal
Behavioral
Cues
Interpersonal
Distance
Gestures
Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging
Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009
6
6
Social Signals
Height
Mutual
Gaze
Vocal
Behavior
Forward
Posture
Forward
Posture
Nonverbal
Behavioral
Cues
Interpersonal
Distance
Social
Signals
Gestures
Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging
Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009
6
6
Outline
•“Socialism” and Big Data
•Social Signal Processing
•The Conflict Example
•Conclusions
7
7
Conflict
“[Conflict is a] mode of interaction [where]
the attainment of the goal by one party
precludes its attainment by the others.”
C.M. Judd, “Cognitive Effects of Attitude Conflict Resolution ”, Journal of
Conflict Resolution, 22(3):483-498, 1978.
8
8
Data
Samples
1430 clips from Canal9
Samples Length
30 seconds (11h:55m)
Subjects
138 (5 moderators)
Subjects per Sample
At least 2
Assessors
551 (10 per sample via MTurk)
Questionnaire
15 items
Total Items
214,500
Vinciarelli, Kim,Valente and Salamin, “Automatic Detection of Conflicts in
SpokenConversations”, Proc. of IEEE International Conference on Audio,
Speech and Signal Processing, pp. 1-4, 2012.
9
9
Measuring Conflict
• The atmosphere is relaxed
• People wait for their turn before speaking
• One or more people talk fast
• One or more people fidget
• People argue
• One or more people raise their voice
• One or more people shake their heads and nod
• People show mutual respect
• People interrupt one another
• One or more people gesture with their hands
• One or more people are aggressive
• The ambience is tense
• One or more people compete to talk
• People are actively engaged
• One or more people frown
10
10
Measuring Conflict
• The atmosphere is relaxed
• People wait for their turn before speaking
• One or more people talk fast
• One or more people fidget
• People argue
• One or more people raise their voice
• One or more people shake their heads and nod
• People show mutual respect
• People interrupt one another
• One or more people gesture with their hands
• One or more people are aggressive
• The ambience is tense
• One or more people compete to talk
• People are actively engaged
• One or more people frown
+
+
11
11
Measuring Conflict
• The atmosphere is relaxed
• People wait for their turn before speaking
• One or more people talk fast
• One or more people fidget
• People argue
• One or more people raise their voice
• One or more people shake their heads and nod
• People show mutual respect
• People interrupt one another
• One or more people gesture with their hands
• One or more people are aggressive
• The ambience is tense
• One or more people compete to talk
• People are actively engaged
• One or more people frown
+
+
11
11
Measuring Conflict
• The atmosphere is relaxed
• People wait for their turn before speaking
• One or more people talk fast
• One or more people fidget
• People argue
• One or more people raise their voice
• One or more people shake their heads and nod
• People show mutual respect
• People interrupt one another
• One or more people gesture with their hands
• One or more people are aggressive
• The ambience is tense
• One or more people compete to talk
• People are actively engaged
• One or more people frown
+
+
11
11
Measuring Conflict
Kim et al.,“Predicting the Conflict Level in Television Political Debates: an
Approach Based on Crowdsourcing, Nonverbal Communication and Gaussian
Processes”, Proc. of ACM MM, pp. 793-796, 2012
12
12
Speech Analysis
•
Clip-based statistics (pitch and intensity)
13
13
Speech Analysis
•
•
Clip-based statistics (pitch and intensity)
Turn-based statistics (pitch and intensity)
14
14
Speech Analysis
•
•
•
Clip-based statistics (pitch and intensity)
Turn-based statistics (pitch and intensity)
Speaker activity statistics
15
15
Speech Analysis
•
•
•
•
Clip-based statistics (prosody)
Turn-based statistics (prosody)
Speaker activity statistics (time budget)
Overlapping speech statistics (prosody and length)
16
16
Results
Correlation Actual / Predicted Conflict Level
0.85
Manual
0.8
0.80
0.78
0.75 0.76
0.74
0.81
0.77
0.80
0.77
0.81
Automatic
w.o.s
0.77
Automatic
0.74
0.73
0.7
0.71
0.71
GPR ARD
SVR LIN
0.69
0.65
BLR
GPR RBF
SVR RBF
17
17
Outline
•“Socialism” and Big Data
•Social Signal Processing
•The Conflict Example
•Conclusions
18
18
Conclusions
19
19
Conclusions
•
Nonverbal behaviour provides machine detectable
evidence of social and psychological phenomena
19
19
Conclusions
•
Nonverbal behaviour provides machine detectable
evidence of social and psychological phenomena
•
Big Data is an opportunity: SSP approaches are
data driven and can benefit from large corpora
19
19
Conclusions
•
Nonverbal behaviour provides machine detectable
evidence of social and psychological phenomena
•
Big Data is an opportunity: SSP approaches are
data driven and can benefit from large corpora
•
Big Data is a challenge: Psychologically oriented
methodologies need to be adapted
19
19
Conclusions
•
Nonverbal behaviour provides machine detectable
evidence of social and psychological phenomena
•
Big Data is an opportunity: SSP approaches are
data driven and can benefit from large corpora
•
Big Data is a challenge: Psychologically oriented
methodologies need to be adapted
http://www.sspnet.eu
19
19
Thank You!
20