Supplementary Material: The Social Perceptual Salience Effect 1
Running head: SUPPLEMENTARY MATERIAL: THE SOCIAL PERCEPTUAL SALIENCE EFFECT
Supplementary Material: The Social Perceptual Salience Effect
Martin P. Inderbitzin1 , Alberto Betella1 , Antonio Lanatà2 , Enzo P.
Scilingo2 , Ulysses Bernardet1 , Paul F. M. J. Verschure1,3
1
Laboratory for Synthetic, Perceptive and Emotive Systems, Technology Department,
Universitat Pompeu Fabra, Barcelona, Spain.
2
Interdepartmental Research Center E. Piaggio, Faculty of Engineering, University of
Pisa, Italy.
3
Catalan Institute for Research and Advanced Studies (ICREA), Barcelona, Spain.
March 7, 2012
Author Note
Correspondence concerning this paper should be addressed to:
Paul F.M.J.Verschure, Synthetic, Perceptive, Emotive, Cognitive Systems Group
Universitat Pompeu Fabra, Roc Boronat 138, 08018 Barcelona, Spain.
E-mail: [email protected], Phone: (0034) 93 5421372
Supplementary Material: The Social Perceptual Salience Effect 2
Supplementary Material: The Social Perceptual Salience Effect
Preprocessing of Physiological Data
The raw data was collected from biosensors and pre-processed. Each signal was
segmented according to the time duration of the stimulating epochs. The methods for
data analysis of the physiological signals are based on previously published algorithms.
Detailed descriptions can be found in (Valenza, Lanata, & Scilingo, 2011).
Heart Rate Variability (HRV)
Electrocardiogram (ECG) was pre-filtered through a Moving Average Filter (MAF) in
order to extract and subtract the baseline. Since HRV refers to the change over time of
the Heart Rate (HR). We adapted an automatic algorithm to detect the Q-, R- and S-wave
forms of the ECG signal (QRS complex) (Pan & Tompkins, 1985).
The time interval between two successive QRS complexes is defined as the R-wave
to R-wave (RR) interval (tR−R ). Thereafter the heart rate (HR) is defined as:
HR =
60
tR−R
(1)
Because the HR is a time series sequence of non-uniform RR intervals, we re-sampled
the signal using the algorithm of Berger et al. (2007)
Respiration (RSP)
We identified the baseline and removed movement artifacts. Additionally we filtered
the signal using a tenth order low-pass finite impulse response filter (FIR) with a cut-off
frequency of 1 Hz approximated by Butterworth polynomial.
Supplementary Material: The Social Perceptual Salience Effect 3
Electrodermal Response (EDR)
The EDR signal was filtered using a 2.5 Hz low-pass FIR filter. Because it has been
shown that the energy of the tonic component is in the frequency band from 0 to 0.05 Hz
and the energy of the phasic component in the band from 0.05 to 1-2 Hz (Ishchenko &
Shev’ev, 1989), we applied a twelve level decomposition wavelet filter in order to identify
two main response components in these bands. Approximation at level 1 of the filter was
the tonic component and subsequent details were the phasic component.
Standard Feature Set Identification
We calculated all the features for each neutral as well as for each stimulation
session. We used 43 standard features and 8 features extracted using non-linear
dynamic methods which are described in the next section. The standard feature set was
derived from the following components of the signal: time series, statistics, frequency
domain and geometric analysis.
Heart Rate Variability (HRV) HRV features were decomposed in features describing both
the time and frequency domain. Defining the beat-to-beat time window (NN), we
calculated the mean of the NN (MNN) and the standard deviation of the NN (SDNN).
Additionally the root mean square of successive differences of intervals (RMSSD) and the
number of successive differences of intervals which differ by more than 50 ms (pNN50 %)
was calculated.
The triangular index, that refers to the morphological changes of the HRV, was
derived from the histogram of RR intervals by performing a triangular interpolation over
the NN windows (TINN).
We based all features extracted in the frequency domain on the Power Spectral
Density (PSD) of the HRV. Three main spectral components were distinguished in a
Supplementary Material: The Social Perceptual Salience Effect 4
spectrum calculated from short-term recordings: Very Low Frequency (VLF), Low
Frequency (LF), and High Frequency (HF) components. We additionally calculated the
LF/HF Ratio which should give information about the Sympatho-Vagal balance (Camm et
al., 1996).
Respiration (RSP) By defining a time window W the respiration rate (RSPR) was
calculated as the frequency corresponding to the maximum spectral magnitude. We
identified the maximum (MAXRSP) and the minimum (MINRSP) value of breathing
amplitude and their difference (DMMRSP) to characterize the differences between
inspiratory and expiratory phase (range or greatest breath).
Electrodermal Response (EDR) We applied the same standard methods as used for the
RSP signal processing to identify both the tonic and the phasic EDR the central
frequency, mean and standard deviation of the amplitude. Additionally we calculated the
maximum peak and the relative latency from the beginning of the interaction phase,
frequency (rate) and magnitude (max) of the maximum component of the phasic EDR.
Non-Linear Dynamic Methods for Feature Extraction
The here documented non-linear dynamic methods for feature extraction are based
on the study of Valenza et al. (2011).
We based our analysis on the so-called embedding procedure. Embedding of a
time series xt = (x1 , x2 , ..., xN ) is realized by creating a set of vectors Xi such that
Xi = [xi , xi+4 , xi+24 , ..., xi+(m−1)4 ]
where 4 is the delay in number of samples and m is the number of samples of the array
Xi . In order that the vector Xi represents the values that reveal the topological
(2)
Supplementary Material: The Social Perceptual Salience Effect 5
relationship between subsequent points in the time series, we must define the dimension
of m and Xi and the delay ∆. We can represent the temporal evolution of the system by
projecting the vectors Xi onto a trajectory through a multidimensional space, often
referred to as phase space or state space. The Recurrence Plot (RP) visualizes all times
at which a state of the dynamical system recurs (Marwan, Carmen Romano, Thiel, &
Kurths, 2007). Higher dimensional phase spaces can be visualized by projecting them
into two or three dimensional sub-spaces (Eckmann, Kamphorst, & Ruelle, 1987). When
a state at time i recurs also at time j, the element (i, j) of a squared matrix N xN is set to
1, 0 otherwise. This representation is called recurrence plot (RP). We can mathematically
express such an RP as
Ri,j = Θ (
i
− ||xi − xj ||)
where xi Rm , i, j = 1, ...., N ; N is the number of considered states xi , εi is a threshold
distance, ||.|| a norm and Θ (.) the Heaviside function which is defined as:
1,
if z ≥ 0
(3)
H (z) =
0,
if z < 0
We chose the optimal value of εi (Schinkel, Dimigen, & Marwan, 2008) as following:
i
= 0.1 ∗ AP D
(4)
where AP D is averaged phase space diameter of data xi .
To quantify the number and duration of recurrences of a dynamical system
presented by its state space trajectory the Recurrence Quantification Analysis (RQA) can
be applied (Zbilut & Webber Jr, 2006). In this study we calculated the following features:
Recurrence Rate (RR) is the percentage of recurrence points in an RP and it
corresponds to the correlation sum:
RR =
N
1 X
Ri,j
2
N
i,j=1
(5)
Supplementary Material: The Social Perceptual Salience Effect 6
where N is the number of points on the phase space trajectory.
The determinism (DET ) is defined as the percentage of recurrence points which
form diagonal lines:
N
P
lP (l)
l=lmin
DET = P
N
(6)
Ri,j
i,j=1
where P (l) is the histogram of the lengths l of the diagonal lines.
Laminarity (LAM ) is the percentage of recurrence points which form vertical lines:
N
P
LAM =
υP (υ)
υ=υmin
N
P
(7)
υP (υ)
υ=1
where P (υ) is the histogram of the lengths υ of the diagonal lines.
Trapping Time T T is the average length of the vertical lines:
N
P
TT =
υP (υ)
υ=υm in
N
P
(8)
P (υ)
υ=υm in
Ratio (RAT I O) is the ratio between DET and RR:
RAT I O =
DET
RR
(9)
Averaged diagonal line length (L) is the average length of the diagonal lines:
N
P
lP (l)
l=lmin
L= P
N
(10)
P (l)
l=lmin
Entropy (EN T R) is the Shannon entropy of the probability distribution of the
diagonal line lengths p(l):
N
X
EN T R = − p(l) lnp(l)
l=lmin
(11)
Supplementary Material: The Social Perceptual Salience Effect 7
Longest diagonal line (Lmax ) The length of the longest diagonal line:
Lmax = max ({li ; i = 1, ..., Nl })
(12)
where Nl is the number of diagonal lines in the recurrence plot.
It has been shown that Approximate Entropy (ApEn) can be used to measure the
complexity or irregularity of the signal (Fusheng, Bo, & Qingyu, 2000; Richman &
Moorman, 2000). Small values of ApEn indicate a more regular signal, lager values a
high irregular one.
To compute the ApEN first a set of length m vectors uj is formed:
uj = (RRj , RRj+1 , ..., RRj+m−1 ),
(13)
where j = 1, 2, ..., N − m + 1, m is the embedding dimension, and N is the number of
measured RR intervals. The maximum absolute difference between the corresponding
elements defines the distance between these vectors:
d(uj , uk ) =
max
n=0,...,m−1
{|RRj+n − RRk+n |}
(14)
For each uj the relative number of vectors uk for which d(uj , uk ) ≤ r is calculated. r
is the tolerance value. The index is denoted with C mj (r) and can be written in the form:
Cjm (r) =
nbr of {uk |d(uj , uk ) ≤ r}
∀k
N − m+1
(15)
m
Due to the normalization, the value of C m
j (r) is smaller or equal to 1. The value of C j (r)
is at least 1/(N − m + 1) since uj is also included in the count. The averaged natural
logarithm of each C jm (r) yields to:
Φm (r) =
N −m+1
X
1
ln Cjm (r).
N − m + 1 j=1
(16)
The approximate entropy finally can be calculated as:
ApEn(m, r, N ) = Φm (r) − Φm+1 (r)
(17)
Supplementary Material: The Social Perceptual Salience Effect 8
Three parameters are influencing the value of ApEn: the length m of the vectors uj , the
tolerance r, and the data length N. In this work we have chosen m = 2. The length N of
the data also affects ApEn. As N increases the ApEn approaches its asymptotic value.
The tolerance r has a strong effect on ApEn and should be selected as a fraction of the
Standard Deviation of all Normal-to-Normal (SDNN) intervals, i.e. the standard deviation
of the intervals between successive normal QRS complexes. A common selection for r is
r = 0.2 · SDN N , which is also used here.
Feature reduction strategy
We obtained a high-dimensional feature space, that we reduced by applying the
Principal Component Analysis (PCA) method (Jolliffe, 2002). We implemented this
approach by means of the Singular Value Decomposition (SVD). Each training set vector
can be approximated by taking only the first few k, where, k ≤ r, Principal Components.
This mathematical method is based on the linear transformation of the different variables
in principal components which could be assembled in clusters.
Classification
For classification a Nearest Mean Classifier (NMC)) is used. This classifier uses the
similarity between patterns to decide on a good classification. The question is how to
define similarity. NMC defines the features of a class as a vector and represents the class
with the mean of the elements of this vector. Thus, any unlabeled vector of features will be
classified as the class with the nearest mean value. Template matching uses a template
for defining class labels, and tries to find the most similar template for classification.
The classification task was evaluated using the confusion matrix. The generic
element rij of the confusion matrix indicates how frequently a pattern belonging to the
stimulus class i was classified as belonging to the response class j. The matrix has to be
read by columns. We used 80% of the feature dataset for training and the remaining 20%
Supplementary Material: The Social Perceptual Salience Effect 9
for the testing phase. In order to obtain unbiased classification results, we performed
40-fold cross-validation steps. This procedure allowed us to consider the distribution of
the classification results as Gaussian. The classification is described by the mean and
standard deviations among the 40 confusion matrices (See Table 1).
Supplementary Material: The Social Perceptual Salience Effect 10
Questionnaire
We would like to evaluate in more detail how you perceived the interaction with the
real person or the avatar at DIFFERENT distances. ’Very close’ was the interaction
distance of < 0.5 meter. ’Close’ the distance of 1.2 meters. Please mark with a cross your
personal experience. Thanks.
Virtual Interaction
I perceived the close interaction with the VIRTUAL avatar as:
Pleasant
1
Negative
2
1
3
2
4
3
5
4
6
5
7
6
Unpleasant
7
Positive
I perceived the very close interaction with the VIRTUAL avatar as:
Pleasant
1
Negative
2
1
3
2
4
3
5
4
6
5
7
6
Unpleasant
7
Positive
Physical Interaction
I perceived the close interaction with the PHYSICAL person as:
Pleasant
1
Negative
2
1
3
2
4
3
5
4
6
5
7
6
Unpleasant
7
Positive
I perceived the very close interaction with the PHYSICAL person as:
Pleasant
Negative
1
2
1
3
2
4
3
5
4
6
5
7
6
Unpleasant
7
Positive
Supplementary Material: The Social Perceptual Salience Effect 11
References
Berger, R., Akselrod, S., Gordon, D., & Cohen, R. (2007). An efficient algorithm for
spectral analysis of heart rate variability. Biomedical Engineering, IEEE
Transactions on(9), 900–904.
Camm, A., Malik, M., Bigger, J., Breithardt, G., Cerutti, S., Cohen, R., et al. (1996). Heart
rate variability: standards of measurement, physiological interpretation, and clinical
use. Circulation, 93(5), 1043–1065.
Eckmann, J., Kamphorst, S., & Ruelle, D. (1987). Recurrence plots of dynamical systems.
EPL (Europhysics Letters), 4, 973.
Fusheng, Y., Bo, H., & Qingyu, T. (2000). Approximate Entropy and its application in
biosignal analysis. Nonlinear biomedical signal processing, 72.
Ishchenko, A., & Shev’ev, P. (1989). Automated complex for multiparameter analysis of
the galvanic skin response signal. Biomedical Engineering, 23(3), 113–117.
Jolliffe, I. (2002). Principal component analysis. Wiley Online Library.
Marwan, N., Carmen Romano, M., Thiel, M., & Kurths, J. (2007). Recurrence plots for the
analysis of complex systems. Physics Reports, 438(5-6), 237–329.
Pan, J., & Tompkins, W. (1985). A real-time QRS detection algorithm. IEEE Transactions
on Biomedical Engineering, 230–236.
Richman, J., & Moorman, J. (2000). Physiological time-series analysis using approximate
entropy and sample entropy. American Journal of Physiology- Heart and Circulatory
Physiology , 278(6), H2039.
Schinkel, S., Dimigen, O., & Marwan, N. (2008). Selection of recurrence threshold for
signal detection. The European Physical Journal-Special Topics, 164(1), 45–53.
Valenza, G., Lanata, A., & Scilingo, E. P. (2011). The Role of Nonlinear Dynamics in
Affective Valence and Arousal Recognition. IEEE Transaction on Affective
Computing, 1–14.
Supplementary Material: The Social Perceptual Salience Effect 12
Zbilut, J., & Webber Jr, C. (2006). Recurrence quantification analysis. Wiley Online
Library.
Supplementary Material: The Social Perceptual Salience Effect 13
Table 1
NMC classifier - Physiological signal classification based on a 20 component feature set.
The rows are "response class" while the columns are "stimulus class". The table must be
read column-wise.
NMC
Physical Intimate
Virtual Intimate
Physical Intimate
88.23 (21.5)
23.53 (25.3)
Virtual Intimate
11.76 (21.5)
76.47 (25.3)
Physical Neutral
Physical
98.75 Neutral
(7.9)
Physical
0.0 (0.0)
Intimate
Physical Intimate
1.25 (7.9)
100.0 (0.0)
Virtual Neutral
Virtual
79.15 Neutral
(25.0)
Virtual
12.50 Intimate
(21.9)
Virtual Intimate
20.84 (25.0)
87.50 (21.9)
Virtual Neutral
Virtual
60.00 Neutral
(35.7)
Physical
48.33 (33.4)
Neutral
40.00 (35.7)
51.66 (33.4)
Physical Neutral