Spatio-Temporal Models for
Mental Processes from fMRI
Raghu Machiraju
Firdaus Janoos, Fellow, Harvard Medical
Istavan (Pisti) Morocz, Instuctor, Harvard
Medical
Premise
Understanding the mind not only requires a
comprehension of the workings of low–level
neural networks but also demands a
detailed map of the brain’s functional
architecture and a description of the large–
scale connections between populations of
neurons and insights into how relations
between these simpler networks give rise to
higher–level thought
Goals
• Understanding the representation of mental
processes in functional neuroimaging
– Distributed interactions
– Space and time !
• Comparing processes across subjects
• Neurophysiologic interpretability
Outline
• fMRI Analysis
• Representations
• Spatio-temporal Models
• Conclusion
What Is fMRI ?
• fMRI is a non-invasive tool
for studying brain activity
• Spatio-temporal data (4D)
• Spatial resolution – mm
• Temporal resolutions – secs
• Functional specialization
• Classical neuroscience
• Functional integration
• Functional and effective
The fMRI Signal
• The BOLD Effect
– Measure of cerebral metabolism
• Task related
• Default-state networks
• Confounds/Nuisance
– Random – thermal + quantum mechanical
– Structured component
• Distortions, physiological, motion, reconstruction
The BOLD Effect
Measure of “oxygenated
blood” in the brain
– Volume of
deoxyhemoglobin
– T2* weighted EPI
sequences
The
exact coupling between neuronal activity and the
BOLD signal unknown
Linked primarily to metabolic activity at synapses
Depends on rCBF, rBVO2, rCMRO2
The hemodynamic response function is highly variable
fMRI Noise
• Acquisition
• Reconstruction
• Magnetic field
• Inhomogeneities
• Instability
• Physiologic functions
• Aliased onto signal
• Head motion
• Correlated with the task
• Registration / Correction
Classical Pipeline
fMRI Analysis
• Functional Localization
• Static Activity Maps
• GLM, PCA, ICA, PLS,
• Functional Integration
• Functional Connectivity
• CCA, ICA, PCA, DBN
• Effective Connectivity
• SEM, DCM, DBN
Typical DCM
Benefits
• fMRI provides information about the activity of large
neural assemblies
– Static pictures of the foci of activity and the
interconnections
• Mental processes arise from dynamic relationships
between the neural substrates
– Spatially distributed, temporally transient and occur at
multiple scales of space and time.
• Time resolved analysis
– Ordering of information processing
Cascadic
Recruitment
State-of-the-Art
Janoos et al.,
EuroVis2009
Need Decoding !
• VOXEL-wise Representations Limited
• Dynamic Processes
• Distributed Representations Needed
– Beyond functional localization
• Where vs. how
– Distributed activity and functional interactions
• Pattern Classifiers
• Atoms of Thought for Cracking Neural Code 
Haxby, 2001
Mitchell, 2008
Challenges
• Very controlled experiments with copious training
• General results have not always been positive
• Applications to arbitrary settings ?
• Temporal nature of mental processes
• Neurophysiologic interpretability
• Multi-subject analysis
Inspiration
Lehmann, 1994
Preliminary Results
visuo–spatial working memory
2 Patients
Functional Networks
Functional Connectivity
Estimation
Gaussian smoothing
HAC until
≈0.25N
Cluster-wise Correlation Estimation and Shrinkage
Voxel-wise Correlation Estimation
Functional Distance
Zt – activation patterns
f - transportation
Cost Metric
Functional Distance
t1
t2
t3
Algorithm
Mental Arithmetic
• Involves basic manipulation of number and
quantities
• Magnitude based system – bilateral IPS
• Verbal based system – left AG
• Attentional system – ps Parietal Lobule
• Other systems – SMA, primary visual cortex,
liPFC, insula, etc
Paradigm
Clustering in Functional Space
10
Brain State Label
5
0
10
5
0
0s
4s
8s
0s
4s
8s
Spatial Maps
10
10 same as 8
9
8
8 auditory
cortices
7
6 , judgment
6
5
5 Frontal,
parietal lobes
4
3
3 visual size
estimation
2
1
1 Visual Cortex
0
0s
4s
8s
+5.0
0
-5.0
Critique
• No neurophysiologic model
– Point estimates
– Hemodynamic uncertainty
– Temporal structure
• Functional distance - an optimization
problem
– No metric structure
– Expensive !
Functional Distance
Cost Metric
Cost Metric
Distortion minimizing
Feature Space
Φ
Orthogonal Bases
Graph Partitioning
Normalized graph
Laplacian of F
Feature Space Φ
Feature Selection
Y
Resampling with Replacement
Functional Network Estimation
Basis Vector φ(l,m) Computation
Bootstrap Distribution of Correlations ρ (l,m)
Feature Selection
Retain φ(l,m) if Pr[ρ (l,m) ≥ τΦ] ≥ 0.75
Φ
R
times
State-Space Model
α,π
xt
xt+1
xt+2
.. .
xt+L
μ k Σk
K
zt
zt+1
zt+2
.. .
zt+L
γ
yt
yt+1
yt+2
Janoos et al., MICCAI 2010
.. .
yt+L
h
T
Σε
μ γ σγ
(Reduced) State-Space Model
α, π
μ k Σk
K
xt
xt+1
xt+2
…
xt+L-1
γ
yt
y t+1
y t+2
…
h
y t+L
T
Σε
μγ σγ
Model Size Selection
• Typically strike a balance between model
complexity and model fit
• Information theoretic or Bayesian criteria
– Notion of model complexity
• Cross-validation
– IID Assumption
Maximally Predictive Criteria
• Multiple spatio-temporal patterns in fMRI
– Neurophysiological
• task related vs. default networks
– Extraneous
• Breathing, pulsatile, scanner drift
• Select a model that is maximally
predictive with respect to task
– Predictability of optimal state-sequence from
stimulus, s
Dyscalculia
Difficulty in learning arithmetic that cannot
be explained by mental retardation,
inappropriate schooling, or poor social
environment
• Core conceptual deficit dealing with
numbers
• Very common : 3-6% of school-age
children
• Heterogeneous
Paradigm
Results
Self – same subject
Cross – train on one subject and predict on another
Comparing Models
Φ1
st
st+1
u t+2
xt
x t+1
x t+2
…
x t+L-1
zt
z t+1
z t+2
…
z t+L-1
…
st+L-1
λW
W
μ k Σk
K
yt
y t+1
y t+2
…
xt
xt+1
xt+2
…
xt+L-1
xt
xt+1
xt+2
…
xt+L-1
μ h Σh
h
Σε
y t+L-1
T
Subject 1
Φ2
st
st+1
u t+2
xt
x t+1
x t+2
…
…
st+L-1
λW
W
x t+L-1
μ k Σk
zt
z t+1
z t+2
…
K
z t+L-1
μ h Σh
h
yt
y t+1
y t+2
…
Σε
y t+L-1
1
2
⁞
41
42
T
Subject 2
.
.
.
fMRI Data
Φ42
st
st+1
u t+2
xt
x t+1
x t+2
…
x t+L-1
zt
z t+1
z t+2
…
z t+L-1
…
st+L-1
λW
W
μ k Σk
yt
y t+1
y t+2
…
y t+L-1
T
Subject 42
xt
K
μ h Σh
h
Σε
xt+1
xt+2
…
xt+L-1
1
10.00
8.94
⁞
8.50
5.40
2
8.94
10.00
⁞
1.54
0.29
…
…
…
…
…
41
8.50
1.54
⁞
10.00
3.95
42
5.40
0.29
⁞
3.95
10.00
MDS Plot
MDS Plot
Drawbacks
• Approximations in the model
– Elimination of the activity pattern layer
– Spatially unvarying hemodynamics
• Unsupervised approach
– No explicit link to the experiment
– May not necessarily learn relevant patterns
Semi-supervised Approach
• Loose dependency between stimulus and
signal
– Not preclude discovery of un-modeled effects
– Stabilize estimation
• Generalizable to unconstrained designs
• Functionally well-defined representation
The Model
st
xt
st+1
xt+1
u t+2
xt+2
…
…
st+L-1
λW
W
xt+L-1
μ k Σk
zt
zt+1
zt+2
…
K
zt+L-1
μ h Σh
h
yt
y t+1
y t+2
…
y t+L-1
T
Janoos et al., IPMI 2011
Janoos et al., NeuroImage 2011
Σε
EM Algorithm
Mean Field Approximation
Estimation
fMRI Data
Φ
Hyperparameter
Selection
Y
Feature-Space Transformation
Error
Rate
K, λW
Feature-space basis
y
Hyper
parameters
Model Estimation
E-step
Compute q(n)(x,z) from p(y,z,x|θ(n))
Until convergence
M-step
(n+1)
Estimate θ
: L(q(n), θ(n+1)) > L(q(n), θ(n))
s
Stimulus
Parameters
State Sequence Estimation
E-step
Compute q(n)(z) from p(z| y,x(n),θ)
Until convergence
M-step
x(n+1) = argmax L(q(n), x)
x
θ
Dyslexia
Selective
inability
to
build
a
visual
representation of a word, used in subsequent
language processing, in the absence of general
visual impairment or speech disorders
• Affects 5-10% of the population
• Spelling, phonological processing, word
retrieval
• Disorder of the visual word form system
• Multiple varieties
– Occipital, temporal, frontal, cerebellum
Paradigm
Comparative Results
0.7
FS:Φ
0.6
0.5
FS:PCA-NONE
0.4
FS:PCA-PH
0.3
FS:PCA-FULL
0.2
0.1
Error for PH
with FS:PCAPH
0
SVM
SSM:FULL
SSM:PH
SSM:NONE
Overall Results
Spatial Maps
1
2
3
Hemodynamic Responses
Motor Cortex
Intra Parietal Sulcus
MDS Plots
MDS Plots
Control Male
Control Female
Dyslexic Male
Dyslexic Female
Dyscalculic Male
Dyscalculic Female
MDS Plots (2)
Phase 1
Phase 1: Product Size
Phase 2
Phase 2: Problem Difficulty
Conclusion
• Process model for fMRI
– Spatial patterns and the temporal structure
– Identification of internal mental processes
• Neurophysiologically plausible
– Test for the effects of experimental variables
– Parameter interpretation
• Comparison of mental processes
– Abstract representation of patterns
Thank You