How does the brain represent and retrieve word meanings?
Jeremy R. Manning & Michael J. Kahana
Princeton University, University of Pennsylvania
Methods
Overview
We studied how the brain represents, organizes, and
retrieves conceptual (semantic) information.
b.
3
4
NOSE
ELF
BOWL
DRUM
LEG
BENCH
BREAD
LAMP
“truck”
“bowl”
“lamp”
“bread”
Electrode
Spectra
7
16
0.16
P
SH
EL
L
TR
UC
K
W
AV
E
SH
EE
SE
A
G
AT
GO
T
DO
CA
0.6
0.5
SHEEP
0.4
SHELL
TRUCK
WAVE
5
15
6
7
8
9
10
11
12
13
14
15
NOSE ELF ...
16
17
18
19
...
3+5+2 = ?
wi =
d.
Content reinstatement
5
0.6
0.5
10
15
0.3
0
10
15
Feature number
5
20
Context reinstatement
0.0
-0.2
-0.4
0.1
0.5
15
0.3
0.1
5
0.5
0.7
0.16
0.14
Similarity
Similarity
0.18
0.4
0.3
2
0.5
0.3
0.1
0
Lag
0.6
0.4
0.2
0.12
4
4
2
0
Lag
2
4
4
2
0
Lag
0.13
0.14
Semantic similarity
0
10
15
Feature number
Neural clustering
0.8
0.12
Figure 8. Neural vs. semantic similarity. Neural
activity is recorded just prior to each recall. The
dots indicate the means of 30 equally sized bins.
3
0.6
0.2
0.11
10
0
10
15
Feature number
0.2
0.6
5
2
4
Figure 6. Predicted neural similarity as a function of lag according to three models. a. In each simulation, a single
neuron is activated during each experimental event, i. Once activated, a neuron’s activity decays gradually
according to equations (1) and (2). b. Autocorrelated noise. Each experimental event activates a random
neuron, irrespective of which item is being presented or recalled. c,d. Each neuron is activated by a single item
or distractor during study. c. Content reinstatement. During recall of the jth presented item we set fi = wj.
d. Context reinstatement. During recall of the jth presented item we set fi = fj.
0.1
TRUCK
SHELL
0.0
−0.1
0.3
−0.3
0.2
−0.4
0.1
−0.5
0.0
2
Semantic similarity
Semantic similarity
−0.4
−0.2
0.0
0.2
Dimension 1
0.4
0.5
Figure 5. Clustering during free recall. a. Semantic clustering. The
semantic clustering score measures a participant’s tendancy to
successively recall semantically related words. b. Temporal
clustering. The temporal clustering score measures a participant’s
tendancy to successively recall words that appeared at nearby
positions on the studied list.
0
-1
We also found that the correlation between neural and semantic
similarity predicted semantic clustering. The prefrontal cortex, lateral
temporal cortex, and hippocampus exhibited this effect, indicating that
these regions are involved with organizing the memories of the words.
Semantic similarity
-2
0.45
0.5
0.55
0.6
Semantic clustering
0.65
Figure 9. Neural vs. semantic clustering. Each dot
represents a single participant. Neural clustering
indicates the t-value from the correlation
between neural and semantic similarity. (Data
are from PFC.)
The magnitude of the neural signature of contextual reinstatement
predicted participants’ tendencies to temporally cluster their recalls.
participant i’s
neural activity
C
selected semantic
features
E
**
*
#
0
0
Lag
0.4
semantic
D
C
E
G
E
G
B
A
F
Correlation
t-value
Probability
Neural similarity
0.05
0
Lag
context
**
or
avi
0
0.1
1.0
The correlation between neural and semantic similarity was preserved
in prefrontal, occipital, and lateral temporal cortex, indicating that
these regions are involved with representing the meanings of words.
h
be
0
0.15
Temporal clustering score
0.6
We also identified components of brain activity that evolved gradually
during study. These “contextual features” were reinstated just prior to
recall.
1
r = 0.42
p = 0.007
4
“...leg...bench....uh...
lamp...bread...”
DOG
H
d
c.
0.3
“...nose...drum...bread...
umm...road...”
WAVE
CAT
cap
ture
b.
“...oar...boat....uh...
glove...ball...”
−0.2
G
a.
“...whale...kite...fudge...
umm...road...”
SEA
0.2
We identified components of brain activity that varied with the
meanings of words during study. We examined these same “semantic
features” just prior to recall.
0.4
Neural similarity
0.6
5
Time (event number)
10
4
0.7
Time (event number)
0.8
3
Neural similarity
c.
Autocorrelated noise
5
2
1.0
Conclusions
0.6
Neural similarity
1 + β2 [( fi -1 · wi ) 2 − 1] − β ( fi -1 · w i )
1
Distractor
features
Neural similarity
fi =
Item
features
Semantic clustering score
0.3
0.7
SEA
3
Study distance
0.4
0.9
0.8
GOAT
SHEEP
GOAT
0.5
1.0
Dimension 2
Neural similarity
DOG
not similar
a.
−4
Frequency
similar
Time (event number)
2
Figure 4. Feature selection. a. Contextual feature selection. We compute the neural similarity between each pair of studied words, for
each neural component. Correlations between neural and temporal similarity imply that the given neural component evolves gradually
during study. b. Semantic feature selection. We compute the neural and semantic similarity between each pair of studied words. (A
seperate neural similarity matrix must be constructed for each neural component.) Correlations between neural and semantic similarity
imply that the given neural component represents the meanings of the studied words.
0.1
Neural similarity
1
0.8
= ρ i fi -1 + β w i
2
15
CAT
0.18
Figure 1. Representing and retrieving conceptual memories. a. Distributed memory hypothesis.
Concepts are broken down into constituent semantic features; each feature is represented by a
distributed pattern of brain activity. b. Contextual reinstatement hypothesis. Context drifts gradually
over time and is associated with each experienced event.
4
14
0.5
1
5
13
Figure 3. Analysis. a. Participants study and freely recalls lists of 15 or 20 common nouns. b. For each electrode we compute mean power
contained in 50 frequencies (2 - 99 Hz) during each study and recall event. c. We apply PCA and select features that either vary with the meanings
of studied words or vary gradually during the study interval. We examine the selected features during the recall interval.
b.
0.08
b.
12
−149
0.10
(2) ρ i =
11
Feature
selection
a.
}
i
10
PCA
Courtesy Michael Jacobs
c.
Figure 2. Our recording setup. Patients are implanted
with subdural and depth electrodes. Experiments are
administered on a bedside laptop computer.
b.
(1) f
2
}
Whereas previous studies have attempted to decode
stimulus identities from neural patterns, our goal was to
infer how stored representations of stimuli are organized
in participants’ memories.
Results
1
Normalized power
Normalized PCA
scores
We examined recordings from neurosurgical patients as
they studied and freely recalled lists of words.
a.
Recall
Study
a.
H
D
G
F
selected contextual
features
neural basis of
semantic clustering
A
D
G
I
G
neural basis of
temporal clustering
C
G
G
behaviorally relevant
neural activity captured
by analyses
Talia Mannin g
Courtesy
(Hippocampus)
0.5
0.6
0.7
0.8
Temporal clustering score
Figure 7. Neural signature of contextual reinstatement. a. Observed neural similarity between the feature vector
corresponding to recall of a word from serial position i and study of a word from serial position i+lag.
b. Probability of recalling an item from serial position i+lag immediately following an item from serial postion i,
conditional on the availability of an item in that list position for recall. c. Participants exhibiting greater context
reinstatement also exhibited more pronounced contiguity effects.
All
FR
PFC
Temp
MTL
Hippo
Occ
Par
Figure 10. Regions that predict semantic clustering.
Each bar indicates the correlation between neural
and semantic clustering from the indicated region
of interest.
Figure 11. Interpreting components of neural activity. The box represents the full pattern of brain activity exhibited
by participant i during the experiment. Our feature selection framework identifies neural patterns in segments C, E,
G, and H. Neural patterns in segment A, D, F, and G contain information about the order in which participants will
recall the words.
For reprints contact [email protected]