Patrick Suppes 90th Birthday Symposium
Language and the Brain
15 years of the Suppes
Brain Lab
Marcos Perreau Guimaraes
A quick Tour
Suppes Brain Lab – Marcos Perreau Guimaraes
Suppes Brain Lab – Marcos Perreau Guimaraes
EEG Recording
Early Years
•
•
•
•
•
•
•
•
•
•
•
•
Patrick Suppes, Zhong-Lin Lu, and Bing Han. Brain-wave recognition of words. Proceedings of
the National Academy of Sciences,94, 1997, pp. 14965-14969.
Patrick Suppes, Bing Han, and Zhong-Lin Lu. Brain-wave recognition of sentences. Proceedings of the National
Academy of Sciences, 95, 1998, pp. 15861-15866.
Patrick Suppes, Bing Han, Julie Epelboim, and Zhong-Lin Lu. Invariance between subjects of brain-wave
representations of language.Proceedings of the National Academy of Sciences USA, 96, 1999, pp. 14658-14663.
Patrick Suppes, Bing Han, Julie Epelboim, and Zhong-Lin Lu. Invariance of brain-wave representations of simple
visual images and their names. Proceedings of the National Academy of Sciences USA, 96, 1999, pp. 14658-14663.
Patrick Suppes and Bing Han. Brain-wave representation of words by superposition of a few sine waves. Proceedings
of the National Academy of Sciences, 97, 2000, pp. 8738-8743.
Dik Kin Wong, Marcos Perreau Guimaraes, E. Timothy Uy, and Patrick Suppes. Classification of individual trials
based on the best independent component of EEG-recorded sentences. Neurocomputing, 61, 2004, pp. 479-484.
Dik Kin Wong, Marcos Perreau Guimaraes, E. Timothy Uy, and Patrick Suppes. Tikhonov-based regularization of a
global optimum approach of one-layer neural networks with fixed transfer function by convex optimization. M. Zhao
and Z. Shi (Eds.), Proceedings of the 2005 IEEE International Conference on Neural Networks and Brain, 3. Beijing:
IEEE Press, 2005, pp. 1564-1567.
Dik Kin Wong, Marcos Perreau Guimaraes, E. Timothy Uy, Logan Grosenick, and Patrick Suppes. Multichannel
classifications of single EEG trials with independent component analysis. J. Wang, et al. (Eds.), Advances in Neural
Networks-ISNN 2006. Berlin: Springer, 150, 2006, pp. 354-359.
Dik Kin Wong, E. Timothy Uy, Marcos Perreau Guimaraes, W. Yang, and Patrick Suppes. Interpretation of perceptron
weights as constructed time series for EEG classification. Neurocomputing, 70, 2006, pp. 373-383.
Marcos Perreau Guimaraes, Dik Kin Wong, E. Timothy Uy, Logan Grosenick, and Patrick Suppes. Single-trial
classification of MEG recordings. IEEE Transactions on Biomedical Engineering, 54, 2007, pp. 436-443.
Patrick Suppes and J. Acacio de Barros. Quantum mechanics and the brain. Quantum Interaction: Papers from the
AAAI Spring Symposium, Technical Report SS-07-08. Menlo Park, CA: AAAI Press, 2007, pp. 75-82.
Dik Kin Wong, Logan Grosenick, E. Timothy Uy, Marcos Perreau Guimaraes, Claudio G. Carvalhaes, Peter Desain,
and Patrick Suppes. Quantifying inter-subject agreement in brain-imaging analyses. NeuroImage, 39, 2008, pp. 10511063.
Source Decomposition and Classification
Independant Component Analysis
x5
x6
x7 x8
x4
x10
x3
x2
x1
x9
• Linear Discriminant Analysis
• Regression
 Ridge
 Lasso
 Elastic net
x11
s1
s2
x12
s3
x13
Maximize independance of
Estimated Sources
Support Vector Machine
Suppes Brain Lab – Marcos Perreau Guimaraes
Best Source and Classification
Some results
60% for 9 words : first, second, third, yes, no, right, left, here, there
97% for pairs first vs fecond and second vs third
Marcos Perreau Guimaraes, Dik Kin Wong, E. Timothy Uy, Logan Grosenick,
and Patrick Suppes. Single-trial classification of MEG recordings. IEEE
Transactions on Biomedical Engineering, 54, 2007, pp. 436-443.
First
Second
Thirds
Partial orders of similarity differences invariant
between EEG-recorded brain and perceptual
representations of language.
Patrick Suppes, Marcos Perreau-Guimaraes, and Dik Kin Wong.
Neural Computation. 21, 2009, pp.3228-3269.
World:
Language
Objects
EEG
Structure
rules
metric
similarities
Structure
rules
metric
similarities
Brain
Structure
rules
metric
similarities
Suppes Brain Lab – Marcos Perreau Guimaraes
Partial Orders and Similarity Trees
 Partial Orders of imilarity Differences
Threshold
c
a
d
a
b
c
b
d
10
Suppes Brain Lab – Marcos Perreau Guimaraes
Sentences Experiment
III+IV
II Aud
II Vis
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11)
12)
13)
14)
15)
16)
17)
18)
19)
20)
21)
22)
23)
24)
# trials
1020
2040
4590
The capital of italy is paris
London is not the capital of poland
The largest city of france is not berlin
Warsaw is not the largest city of russia
Moscow is east of warsaw
Rome is north of london
Paris is not west of berlin
Rome is not south of moscow
The capital of germany is warsaw
Moscow is not the capital of russia
The largest city of italy is not rome
London is not the largest city of france
Paris is east of berlin
Moscow is north of paris
Warsaw is not west of london
Berlin is not south of rome
The capital of italy is not berlin
Warsaw is the capital of france
The largest city of germany is berlin
London is the largest city of russia
Moscow is not east of paris
Rome is not north of warsaw
London is west of moscow
Paris is south of rome
LDC
LDC regularized LIBSVM linear LIBSVM radial
32.10%
36.00%
34.00%
30.20%
10.60%
11.70%
14.50%
14.90%
15.00%
15.40%
18.60%
18.10%
Exp I Vis LDCr france 1 paris 1 london 1 berlin 1 warsaw 1 moscow 1 france 2 paris 2 london 2 berlin 2 warsaw 2 moscow 2 North South East West poland russia germany
france 1
14.2 1.5 3.1 0.1
3.5
3.4 3.5 5.7 3.9 4.5
8.2
7.5 6.6 7.3 5 4.3 2.9 8.5 6.2
paris 1
1.3 20.3 10.6 2.7
8.6 11.5
3 3.2 2.9 4.2
4.1
2.3 6.1 3.7 3.2 3.2 1.2 3.8
4
london 1
1 9.4 19.1 2.9 10.5 10.5 2.8 3.3 2.9 4.1
3.9
3.1 3.6 6.1 3.4 3.6 1.8 3.4 4.5
berlin 1
1.2 10 6.2 8.7
9.9 11.8 3.3 4.3 2.6 3.9
4.4
3.7 5.9 6.1 3.2 3.5 2 3.3 5.9
warsaw 1
2 9 10.5 2.7
17
9.5 2.8 3 3.8 4.2
3.8
3.1 6 5.6 3.1 3.1 1.5 4 5.2
moscow 1
1.4 9.4 9.7 3.9
9.2 17.5 2.3 3.9 3.4 4.3
4.1
2.6 5.1 5 3.4 3.7 1.7 3.9 5.6
france 2
0.6 1.6 2.9
0
1.3
3.4 15.2 9.7
5 10.4
8.3
5.1 4.5 7.5 5.4 4.1 5.1 7.6 2.3
paris 2
1.4 1.5 1.2 0.2
1.5
3.8 9.6 15 4.8 8.4
9.9
8.6 3.8 6.9 5.9 3.9 5.5 6.5 1.8
london 2
1.3 1.4 2.6 0.4
2.4
7.3 6.2 6.4 11.4 9.6
4.9
4.3 8.2 7.6 4.9 4.9 2.8 8.8 4.6
berlin 2
1.7 1.5 2.2 0.1
1.9
4.3 6.2 6.7 5.1 17
7.3
4.9 6.8 6.8 6.4 5.1 4 9.7 2.4
warsaw 2
1.5 1.6 1.9 0.2
1.7
4 6.2 10.7 4.6 9.1 12.6 12.6 4.2 6.4 5.3 4 4.4 6.7 2.2
moscow 2
1.4 1.5 2.5 0.1
1.7
4.3
6 10
3 6.9 16.7 12.9 3.6 6.3 5.5 5.7 3 6.4 2.6
North
1.7 3.7 4.1 0.5
4.5
4.5 3.4 4.5 6.9
8
5.4
3.4 15.3 6.6 6 7.3 2.3 5.1 6.9
South
1.4 1.7 4.2 0.5
3.2
4.5 4.5 3.8 3.7 6.8
3.2
3 6.9 19.2 8.8 10.8 1.9 3 8.9
East
2.1 2.1
3 0.3
3
4.1 5.2 5.6 3.7 10.5
4.5
4.8 7.5 8.6 13.7 9.1 2.5 5.4 4.3
West
2.3 2.2 3.5 0.3
2.5
2.6 6.1 4.5 3.9 7.7
5.1
4.6 8 11 9.6 12.9 2 4.1 7.2
poland
2.2 0.5 2.1
0
0.9
3 8.8 13 4.3 8.6
7.3
4.4 5.2 8.9 5.4 2.8 10.2 10.2 2.2
russia
1.6 2.2 1.9 0.1
1.6
4.3 8.2 6.6 5.4 14.7
5.9
3.4 5.8 4.5 5.8 5.1 5 15.2 2.4
germany
2.7 4.2 4.6 0.7
3.3
6.1 2.6 3.7 3.8 5.7
2.8
3.3 8.5 9.1 4.8 6.5 2.3 2.4 22.8
11
Suppes Brain Lab – Marcos Perreau Guimaraes
Initial Consonants
EEG 1998
Miller and
Nicely 1955
Intersection
Suppes Brain Lab – Marcos Perreau Guimaraes
Pat’s next “small step” :
Dual Recording of Brain Activity
in Couple Therapy
• Record Sound + Video + Dual
EEG
• We have already more than 50h of
recording
• 6 couples from two counselors
• English and Vietnamese
Transcription
• Manual transcription with
~5ms accuracy for the onset of
words
• ~100h to transcribe 1h
• Software Development
acoustic_score="92.1339"
confidence="0.543" >but</arc>
acoustic_score="122.588"
confidence="0.543" >but</arc>
• Automatic Transcription
• Hard problem
o 3 speakers
o Conversational
o Disfluent
• Software Development
Coding of Emotions
• 4 to 19 emotions
• Verbal and non verbal
• Levels of
• Intensity
• Confidence
• Direction
• Insight
• ~30h to code 1h
Analysis of the Speech
01 1 1: MichelleFinal U MargotFinal
Emotions
for “you”
1 1
68
Pitch
1st formant
2nd formant
66
64
Classification rate
62
60
58
56
54
52
50
0
100
200
300
400
500
ms
600
700
800
900
Length of time segment after onset of “you”
in milliseconds
1000
• Control for words
• Non linear classification
(SVM) of Sadness versus
Anxiety using Frequency
features (Pitch, Formants),
Energy in auditory bands and
Dynamics of the speech
envelope
Analysis of the EEG
• Male EEG when The Female
express emotions
• 4 Emotions : Joy, Sadness,
Anxiety and Anger.
• Scalp, face and jaw muscle
artifacts
• Too little data yet but
promising first results
Thank You and Happy
Birthday Pat