Cinema Data Mining: The Smell of Fear
Jörg Wicker Nicolas Krauter Bettina Derstorff
Christof Stönner Efstratios Bourtsoukidis Thomas Klüpfel
Jonathan Williams Stefan Kramer
Johannes Gutenberg University Mainz
Max-Planck-Institute for Chemistry
August 13, 2015
Johannes Gutenberg University Mainz
The Smell of Fear – Emotional Response Analysis
Human emotional response analysis well
studied on many modalities (EEG, skin
resistance,...)
Little known about exhalation of
Volatile Organic Compounds (VOCs) in
relation to emotional response
Do we communicate via exhaling
VOCs?
Which VOCs do we exhale given
certain emotional stimuli?
Jörg Wicker– August 13, 2015
2
Johannes Gutenberg University Mainz
The Smell of Fear – Emotional Response Analysis
Human emotional response analysis well
studied on many modalities (EEG, skin
resistance,...)
Little known about exhalation of
Volatile Organic Compounds (VOCs) in
relation to emotional response
Do we communicate via exhaling
VOCs?
Which VOCs do we exhale given
certain emotional stimuli?
Jörg Wicker– August 13, 2015
2
Johannes Gutenberg University Mainz
VOCs Data – Experimental Setup
Cinema – Screening Room
Jörg Wicker– August 13, 2015
3
Johannes Gutenberg University Mainz
VOCs Data – Experimental Setup
Cinema – Screening Room
Ventilation System
air flow
air flow
Outside
Jörg Wicker– August 13, 2015
3
Johannes Gutenberg University Mainz
VOCs Data – Experimental Setup
Cinema – Screening Room
Ventilation System
air flow
air flow
air flow
Outside
Mass Spectrometer
Jörg Wicker– August 13, 2015
3
Johannes Gutenberg University Mainz
VOCs Data – CO2
CO2 (ppm)
3,000
2,000
1,000
2013-12-18
2013-12-23
2013-12-28
2014-01-02
2014-01-07
2014-01-12
Time
CO2 (ppm)
CO2 (ppm)
1,000
1,000
800
600
500
00:00
04:00
08:00
12:00
2014-12-26
Jörg Wicker– August 13, 2015
16:00
20:00
00:00
400
12:30
13:15
14:00
14:45
15:30
16:15
17:00
Hunger Games – 2014-12-26
4
Johannes Gutenberg University Mainz
VOCs Data – CO2
CO2 (ppm)
3,000
2,000
1,000
2013-12-18
2013-12-23
2013-12-28
2014-01-02
2014-01-07
2014-01-12
Time
CO2 (ppm)
CO2 (ppm)
1,000
1,000
800
600
500
00:00
04:00
08:00
12:00
2014-12-26
Jörg Wicker– August 13, 2015
16:00
20:00
00:00
400
12:30
13:15
14:00
14:45
15:30
16:15
17:00
Hunger Games – 2014-12-26
4
Johannes Gutenberg University Mainz
VOCs Data – CO2
CO2 (ppm)
3,000
2,000
1,000
2013-12-18
2013-12-23
2013-12-28
2014-01-02
2014-01-07
2014-01-12
Time
CO2 (ppm)
CO2 (ppm)
1,000
1,000
800
600
500
00:00
04:00
08:00
12:00
2014-12-26
Jörg Wicker– August 13, 2015
16:00
20:00
00:00
400
12:30
13:15
14:00
14:45
15:30
16:15
17:00
Hunger Games – 2014-12-26
4
Johannes Gutenberg University Mainz
VOCs Data – CO2
12/26
12/27
1,500
12/28
CO2 concentration (ppm)
12/29
1,000
500
12:45
13:15
13:45
14:15
14:45
15:15
15:45
16:15
16:45
Time
Jörg Wicker– August 13, 2015
5
Johannes Gutenberg University Mainz
VOCs Data – CO2
12/26
12/27
1,500
12/28
CO2 concentration (ppm)
12/29
1,000
500
12:45
13:15
13:45
14:15
14:45
15:15
15:45
16:15
16:45
Time
Jörg Wicker– August 13, 2015
5
Johannes Gutenberg University Mainz
VOCs Data – CO2
12/26
12/27
1,500
12/28
CO2 concentration (ppm)
12/29
1,000
500
12:45
13:15
13:45
14:15
14:45
15:15
15:45
16:15
16:45
Time
Jörg Wicker– August 13, 2015
5
Johannes Gutenberg University Mainz
VOCs Data – CO2
12/26
12/27
1,500
12/28
CO2 concentration (ppm)
12/29
1,000
500
12:45
13:15
13:45
14:15
14:45
15:15
15:45
16:15
16:45
Time
Jörg Wicker– August 13, 2015
5
Johannes Gutenberg University Mainz
VOCs Data – CO2
12/26
12/27
1,500
12/28
CO2 concentration (ppm)
12/29
1,000
500
12:45
13:15
13:45
14:15
14:45
15:15
15:45
16:15
16:45
Time
Jörg Wicker– August 13, 2015
5
Johannes Gutenberg University Mainz
Scene Annotations
15 movies in 104 screenings
6 movies selected for analysis
63 screenings (46 of them usable)
But: No standard scene annotations
available
Use combination of previously used
labels and movie genres
Genre, Content
Multiple annotators
Jörg Wicker– August 13, 2015
6
Johannes Gutenberg University Mainz
Analysis – Time Series Prediction
romance
action
...
Acetone
Isoprene
...
Jörg Wicker– August 13, 2015
7
Johannes Gutenberg University Mainz
Analysis – Time Series Prediction
romance
action
...
Acetone
target
forward
Isoprene
...
t = -00:30
...
t = -05:00
Jörg Wicker– August 13, 2015
t=0
7
Johannes Gutenberg University Mainz
Analysis – Time Series Prediction
romance
Acetone
target
action
backward /
Isoprene
abductive
reasoning
...
...
Acetone
target
forward
Isoprene
...
t = -00:30
...
t = -05:00
Jörg Wicker– August 13, 2015
t=0
t = +00:30
...
t = +05:00
7
Johannes Gutenberg University Mainz
Analysis – Time Series Prediction
forward-backward
romance
target
action
backward /
Acetone
Isoprene
abductive
reasoning
...
...
Acetone
target
forward
Isoprene
...
t = -00:30
...
t = -05:00
t=0
t = +00:30
...
t = +05:00
forward-backward
Jörg Wicker– August 13, 2015
7
Johannes Gutenberg University Mainz
Analysis – Causality
CO2
Acetone
target
backward /
abductive
reasoning
Isoprene
...
t=0
Jörg Wicker– August 13, 2015
t = +00:30
...
t = +05:00
8
Johannes Gutenberg University Mainz
Analysis – Causality
CO2
Acetone
target
backward /
abductive
reasoning
Isoprene
ROC Curve
...
t=0
Jörg Wicker– August 13, 2015
t = +00:30
...
t = +05:00
8
Johannes Gutenberg University Mainz
Analysis – Causality
CO2
7
Acetone
target
backward /
abductive
reasoning
3
Isoprene
ROC Curve
...
t=0
Jörg Wicker– August 13, 2015
t = +00:30
...
t = +05:00
8
Johannes Gutenberg University Mainz
Analysis – Causality
CO2
Acetone
target
backward /
abductive
reasoning
7
7
Isoprene
ROC Curve
...
t=0
Jörg Wicker– August 13, 2015
t = +00:30
...
t = +05:00
8
Johannes Gutenberg University Mainz
Analysis – Causality
CO2
Acetone
target
backward /
abductive
reasoning
3
Isoprene
7
ROC Curve
...
t=0
Jörg Wicker– August 13, 2015
t = +00:30
...
t = +05:00
8
Johannes Gutenberg University Mainz
Results
30% holdout
Leave-one-Movie-out Validation
Several findings for causal hints, for example
blood (violence) – Ammonia
blood (violence) – Acetone
comedy – Formaldehyde
New experiments with more annotators
Good AUCs for suspense, injury, hidden threat, mystery, comedy
suspense – Isoprene
romance – Isoprene
injury – Siloxanes
Jörg Wicker– August 13, 2015
9
Johannes Gutenberg University Mainz
Conclusion
Data set is available at GitHub: https://github.com/joergwicker/smelloffear
Currently identifying molecules potentially related to human signals
Jörg Wicker– August 13, 2015
10
Johannes Gutenberg University Mainz
Conclusion
Data set is available at GitHub: https://github.com/joergwicker/smelloffear
Currently identifying molecules potentially related to human signals
New measurements done in 02/15 (not yet part of the data set)
Jörg Wicker– August 13, 2015
10
Johannes Gutenberg University Mainz
Conclusion
Data set is available at GitHub: https://github.com/joergwicker/smelloffear
Currently identifying molecules potentially related to human signals
New measurements done in 02/15 (not yet part of the data set)
New measurements planned for 12/15
Jörg Wicker– August 13, 2015
10
Johannes Gutenberg University Mainz
Conclusion
Data set is available at GitHub: https://github.com/joergwicker/smelloffear
Currently identifying molecules potentially related to human signals
New measurements done in 02/15 (not yet part of the data set)
New measurements planned for 12/15
New experimental setups planned
Jörg Wicker– August 13, 2015
10
Johannes Gutenberg University Mainz
Conclusion
Data set is available at GitHub: https://github.com/joergwicker/smelloffear
Currently identifying molecules potentially related to human signals
New measurements done in 02/15 (not yet part of the data set)
New measurements planned for 12/15
New experimental setups planned
Unique combination
Atmospheric Chemistry
Breath analysis
Emotional response analysis
Movie analysis
Data Mining
Jörg Wicker– August 13, 2015
10
Johannes Gutenberg University Mainz
Thank you for your attention! – Questions?
[email protected]
http://joerg-wicker.org
Jörg Wicker– August 13, 2015
11