A model of episodic memory function via corticohippocampal interactions and its implications for
the understanding of consciousness
Lokendra Shastri
International Computer Science Institute
Berkeley, CA
 Lokendra Shastri
ICSI, Berkeley
Episodic Memory
• Memories of events and situations in our daily lives
– Who did what to whom where when and how
• Memories of specific events acquired by
• watching a newscast
• reading a newspaper
• participating in a dialog
• …
• Memories of overt thoughts and reflections
• Prospective memories (+)
Lokendra Shastri
ICSI, Berkeley
Episodic Memory
• Episodic memories enable us to recall our
conscious experiences and place them within
specific spatio-temporal contexts
Lokendra Shastri
ICSI, Berkeley
Episodic Memory
Contrasts with other forms of “memories”
• semantic knowledge
– entities, categories
– semantic facts (abstractions, statistical summaries)
– “Common sense” knowledge (causal knowledge)
• perceptual and motor skills (procedural knowledge)
– riding a bicycle
– playing the piano
• priming
Lokendra Shastri
ICSI, Berkeley
Episodic and Semantic Memory: Some distinctions
Acquired rapidly
One-shot
Acquired from
multiple instances
Specific events and
situations (instances)
Statistical summaries and
generalizations acquired
from several experiences
Situated in specific
spatio-temporal
context
include source
information
Typically, lack specific
spatio-temporal context
and source information
Lokendra Shastri
ICSI, Berkeley
Episodic and Semantic Memory: Some distinctions
• Both episodic and semantic memories are responsive to
partial cues (content addressable, support pattern completion)
• But episodic memory has much higher specificity and
sensitivity to mis-mismatches (support pattern separation)
– episodic memory can store and differentiate several
similar memories
– response to a matching cue is clearly distinct from
response to a near miss
– but support reminding
Lokendra Shastri
ICSI, Berkeley
The hippocampal system plays a critical role in
episodic memory function
• lesion studies
• imaging studies
• Neuropathology of
Alzheimer’s disease
 Lokendra Shastri
ICSI, Berkeley
Impact of bilateral hippocampal lesion
• Inability to form new episodic memories (anterograde
amnesia)
• Loss of episodic memories spanning several decades
prior to the insult (retrograde amnesia)
• Pre-morbid semantic memories spared
• Some new semantic memories are acquired, but with
significant difficulty
• Procedural (e.g., sensorimotor) memory spared
• Priming and familiarity-based object recognition intact
• No direct impact on the ability to understand and
produce language (but, lack of content and context…)
 Lokendra Shastri
ICSI, Berkeley
An overview of cortico-hippocampal interactions
HS
?
+
presentation of an
event to HS
HF
response of HS
presentation of a
query to HS
EC
DG, CA3, CA2
CA1 and Sub
?
+
response of HS
to query
High-level cortical circuits
(HLCCs)
Lokendra Shastri
Subcortical regions including
the amygdala, septum, and the brain stem
?significance of ongoing experience?
ICSI, Berkeley
Hippocampal System Architecture
HS
PHC
Higher
order
sensory
and
polysensory
association
areas
including
PFC
8 mil 17,000
DG
14
CA3
15 mil
2.7 mil
CA2
Sub
PER
EC
4.5 mil
CA1
15 mil
HLCCs
Subcortical regions including
amygdala, septum, and the brain stem
Lokendra Shastri
ICSI, Berkeley
Hippocampal System
• Different types of inhibitory interneurons
• Specific sorts of local circuits formed by
principal cells and inhibitory interneurons
• Occurrence of rhythmic activity in gamma
and theta bands
• Synaptic plasticity along all key pathways
Lokendra Shastri
ICSI, Berkeley
Cortico-hippocampal interactions
• How are events and situations (episodes) expressed
as transient patterns of activity in high-level
cortical circuits?
• How is an episode encoded as a memory trace in
the HS?
(feature vector, conjunctive code, relational
structure, auto-associative memory, …)
• How is this trace laid down rapidly?
 Lokendra Shastri
ICSI, Berkeley
SMRITI* – a model of episodic memory
•
Examined the architecture, local circuitry, and
plasticity of the HS
•
Analyzed the functional requirements of encoding
episodes
•
Identified a neural circuit that satisfies the
requirements of encoding episodes
•
Demonstrated that this circuit can be learned rapidly
and automatically by the HS in response to cortical
activity
•
Analyzed the model to explain and predict properties of
episodic memory
*System
Lokendra Shastri
for memorizing relational instances from transient impulses
ICSI, Berkeley
Events and Situations Are Relational Instances
“John gave Mary a book
in the library on Tuesday”
Mary
giver:
recipient:
give-object:
location:
temporal-loc:
library
give
John
a-book
Lokendra Shastri
John
Mary
a-book
library
Tuesday
Tuesday
ICSI, Berkeley
Events and Situations Are Relational Instances
“John gave Mary a book
in the library on Tuesday”
Mary
giver:
recipient:
give-object:
location:
temporal-loc:
library
give
John
a-book
Lokendra Shastri
John
Mary
a-book
library
Tuesday
Tuesday
ICSI, Berkeley
Events and Situations Are Relational Instances
“John gave Mary a book
in the library on Tuesday”
Mary
giver:
recipient:
give-object:
location:
temporal-loc:
library
give
John
a-book
Lokendra Shastri
John
Mary
a-book
library
Tuesday
Tuesday
ICSI, Berkeley
Events and Situations Are Relational Instances
“John gave Mary a book
in the library on Tuesday”
giver:
recipient:
give-object:
location:
temporal-loc:
Mary
John
Lokendra Shastri
library
give
e-27
a-book
John
Mary
a-book
library
Tuesday
Tuesday
ICSI, Berkeley
Events and Situations Are Relational Instances
“John gave Mary a book
in the library on Tuesday”
recipient
Mary
library
giver
location
John
give-object
Lokendra Shastri
e-27
a-book
temporal-loc
Tuesday
ICSI, Berkeley
The twin requirements:
• Responsiveness to partial cues
• High specificity (pattern separation)
=>
An event’s episodic memory trace must be
sensitive to binding mismatches –
Memorized event: (<r1=a>, <r2=b>)
Partial Cue:
(<r1=a>)
Erroneous Cue: (<r1=a>, <r2=c>)
Lokendra Shastri
ICSI, Berkeley
Episodic memory trace
An event’s episodic memory trace must contain
functional elements for
• memorizing role-entity bindings and detecting a match
between the memorized bindings and those specified in
a cue (binding-match detectors)
• detecting a mismatch between the memorized bindings
and those specified in a cue (binding-error detectors)
• detecting a match between a cue and the memorized
event based on the responses of the above detectors
(relational-match indicators)
• reinstating bindings of the memorized event in HLCCs
in response to a matching cue (binding-reinstators)
Lokendra Shastri
ICSI, Berkeley
Retrieval: reinstatement of bindings
• An event’s episodic memory trace becomes active if the
cue (partially) matches the memorized event
• The bindings encoded by the activated memory trace get
reinstated within high-level cortical circuits (HLCCs)
• These HLCCs encode semantic knowledge about
entities, categories, and actions, and interact with
perceptual and motor schemas
• When activated with the appropriate bindings, the
HLCCs reconstruct the retrieved event by a process akin
to a mental simulation
Lokendra Shastri
ICSI, Berkeley
How are episodes expressed as a pattern of activity
in HLCCs?
Lokendra Shastri
ICSI, Berkeley
Relational Schemas in HLCCs: Focal-Clusters
Buy
Lokendra Shastri
+
-
?
buyer
buy-obj
ICSI, Berkeley
Focal-cluster of a Relational Schema
perceptual schemas
associated with buy
focal-clusters of other
relational schemas
causally related to buy
Buy
+
-
motor schemas
associated with buy
Lokendra Shastri
?
buyer
buy-obj
episodic
memories of
buy events
lexical knowledge
associated with buy
ICSI, Berkeley
Focal-clusters
Cells in the buy focal-cluster become active when
•perceiving a buy event
•remembering a buy event
•understanding a sentence about a buy event
•experiencing a buy event (buying)
A focal-cluster is like a “supra-mirror” cluster
Lokendra Shastri
ICSI, Berkeley
Focal-cluster of an Entity
focal-clusters of other
entities and categories
semantically related
to Tom
Perceptual schemas
Associated with Tom
Tom
+
motor schemas
associated with Tom
Lokendra Shastri
?
episodic memories
where Tom is one
of the role-fillers
focal-clusters of lexical knowledge
associated with Tom
ICSI, Berkeley
Focal-cluster of an Entity
focal-clusters of other
entities and categories
semantically related
to me
Perceptual schemas
Associated with me
I / me
motor schemas
associated with me
Lokendra Shastri
+
?
episodic memories
where I am one
of the role-fillers
focal-clusters of lexical knowledge
associated with me
ICSI, Berkeley
“John fell in the hallway”
Fall
+ -- ?
fall-pat
fall-loc
+:Fall
fall-loc
Hallway
+
fall-pat
?
+:Hallway
+
?
+:John
John
Lokendra Shastri
ICSI, Berkeley
“John slipped in the hallway” => “John fell in the hallway”
Slip
+
-
?
+:Fall
slip-pat slip-loc
fall-loc
fall-pat
mediator
+
?
r1
r2
+:slip
slip-loc
slip-pat
Fall
+
-
?
fall-pat fall-loc
+:Hallway
+:John
Hallway
+
Lokendra Shastri
?
John
+
?
ICSI, Berkeley
Massively Parallel Inference
• Assuming -band activity underlies dynamic bindings
 inferring “John may be hurt” from “John fell” takes only
~ 200 msecs.
• Time required to perform inference is independent of the size of
the causal model
• Anytime property
Lokendra Shastri
ICSI, Berkeley
Predictions: constraints on reflexive inference
• -band activity (~ 25-60Hz) underlies dynamic binding
• jitter in synchronous firing ~  3 msec
 only ~ (40 ÷ 6) distinct phases can occur within
ongoing rhythmic activity (cf. 7  2)
cross-talk among entities increases as the number of
phases approaches and exceeds 7
Lokendra Shastri
ICSI, Berkeley
Predictions: constraints on “working memory”
underlying reflexive reasoning
1. A large number of relational instances (facts) can be
co-active, and numerous “rules” can fire in parallel,
but
2. Only a small number of distinct entities can serve as
role-fillers in this activity (7  2)
3. Only a small number of instances of the same
predicate can be co-active at the same time
2 & 3 specify limits on working memory underlying RR
Lokendra Shastri
ICSI, Berkeley
“John fell in the hallway. Tom had cleaned it.
He got hurt.”
Lokendra Shastri
ICSI, Berkeley
Lokendra Shastri
ICSI, Berkeley
Lokendra Shastri
ICSI, Berkeley
The activation trace of +:slip and +:trip
S1
Lokendra Shastri
S2
S3
ICSI, Berkeley
A Metaphor for Reasoning
• An episode of reflexive reasoning is a transient propagation of
rhythmic activity
• Each entity involved in this reasoning episode is a phase in
this rhythmic activity
• Bindings are synchronous firings of cell clusters
• Rules are interconnections between cell-clusters that support
context-sensitive propagation of activity
• Unification corresponds to merging of phases
• A stable inference - explanation/answer - corresponds to
reverberatory activity around closed loops
– The results of the above become conscious
Lokendra Shastri
ICSI, Berkeley
-
+
?
buyer
buy-obj
Buy
HS
Fido
+
?
Tom
+
?
High-level cortical circuits
Lokendra Shastri
ICSI, Berkeley
Activity-based encoding of a relational instance
-
+
?
buyer
buy-obj
Buy
HS
Fido
+
?
Tom
+
?
High-level cortical circuits
Lokendra Shastri
ICSI, Berkeley
“ Tom bought Fido”
+:buy
buy-obj
buyer
+:Fido
+:Tom
Lokendra Shastri
ICSI, Berkeley
from
to
+:Tom
from
?:Buy
from
buyer
from
?:Tom
from
?:Fido
from
b-obj
from
to
+:buy
+:Tom
?Buy
buyer
?:Tom
?:Fido
b-obj
+:Buy
from
to
+:Fido
+:Fido
HLCC
EC
DG
CA3
CA2
CA1
Sub
Linking
cells
Lokendra Shastri
ICSI, Berkeley
from
to
+:Tom
from
?:Buy
from
buyer
from
?:Tom
from
?:Fido
from
b-obj
from
to
+:buy
+:Tom
?Buy
buyer
buyer=Tom
?:Tom
?:Fido
b-obj=Fido
b-obj
+:Buy
from
to
+:Fido
+:Fido
HLCC
EC
DG
Linking
cells
Bindingdetector
cells
Lokendra Shastri
CA3
CA2
CA1
Sub
ICSI, Berkeley
from
to
+:Tom
from
?:Buy
from
buyer
from
?:Tom
from
?:Fido
from
b-obj
from
to
+:buy
+:Tom
?Buy
(buyer=Tom |E)
buyer
buyer=Tom
?:Tom
?:Fido
b-obj=Fido
b-obj
(b-obj=Fido |E)
+:Buy
from
to
+:Fido
+:Fido
HLCC
EC
DG
CA3
Linking
cells
Bindingdetector
cells
Binding-errordetector
circuits
Lokendra Shastri
CA2
CA1
Sub
ICSI, Berkeley
from
to
+:Tom
from
?:Buy
from
buyer
from
?:Tom
+:Tom
?Buy
(buyer=Tom |E)
buyer
buyer=Tom
?:Tom
bei (E)
from
?:Fido
from
b-obj
from
to
+:buy
?:Fido
b-obj=Fido
b-obj
(b-obj=Fido |E)
+:Buy
from
to
+:Fido
+:Fido
HLCC
EC
DG
Linking
cells
Bindingdetector
cells
Lokendra Shastri
CA3
CA2
CA1
Sub
Binding-error- Bindingerrordetector
integrator
circuits
circuits
ICSI, Berkeley
from
to
+:Tom
from
?:Buy
from
buyer
from
?:Tom
+:Tom
?Buy
(buyer=Tom |E)
buyer
buyer=Tom
?:Tom
bei (E)
remind(E)
from
?:Fido
from
b-obj
from
to
+:buy
?:Fido
b-obj=Fido
b-obj
(b-obj=Fido |E)
+:Buy
from
to
+:Fido
+:Fido
HLCC
EC
DG
Linking
cells
Bindingdetector
cells
Lokendra Shastri
CA3
CA2
Binding-error- Bindingerrordetector
integrator
circuits
circuits
CA1
Sub
Relationalmatchindicator
circuits
ICSI, Berkeley
from
to
+:Tom
from
?:Buy
from
buyer
from
?:Tom
+:Tom
?Buy
(buyer=Tom |E)
buyer
buyer =Tom|E
buyer=Tom
?:Tom
bei (E)
remind(E)
from
?:Fido
from
b-obj
from
to
+:buy
?:Fido
b-obj=Fido
b-obj
(b-obj=Fido |E)
+:Buy
from
to
+:Fido
+:Fido
HLCC
EC
DG
Linking
cells
Bindingdetector
cells
Lokendra Shastri
b-obj of E=Fido|E
CA3
CA2
Binding-error- Bindingerrordetector
integrator
circuits
circuits
CA1
Relationalmatchindicator
circuits
Sub
Bindingreinstator
cells
ICSI, Berkeley
from
to
+:Tom
from
?:Buy
from
buyer
from
?:Tom
+:Tom
?Buy
(buyer=Tom |E)
buyer
buyer =Tom|E
buyer=Tom
?:Tom
bei (E)
remind(E)
from
?:Fido
from
b-obj
from
to
+:buy
?:Fido
b-obj=Fido
b-obj
(b-obj=Fido |E)
+:Buy
from
to
+:Fido
+:Fido
HLCC
EC
DG
Linking
cells
Bindingdetector
cells
Lokendra Shastri
b-obj of E=Fido|E
CA3
CA2
Binding-error- Bindingerrordetector
integrator
circuits
circuits
CA1
Relationalmatchindicator
circuits
Sub
Bindingreinstator
cells
ICSI, Berkeley
Long-term Potentiation (LTP)
Long-term increase in synaptic strength resulting from
the pairing of pre-synaptic activity with post-synaptic
depolarization
• rapidly induced, long lasting, synapse specific
• LTP of a synapse requires the synchronous arrival of
activity at multiple synapses of the post-synaptic cell
(co-operativity)
– ideal for transforming a transient, activity-based
expression of a relation between multiple items into a
persistent, synaptic-efficacy-based structural encoding
of the relation
Lokendra Shastri
ICSI, Berkeley
Memorization of F2: [r6 = f16][r8 = f19]
Existing fact memorized in episodic memory:
F1: [r6 = f10] [r8 = f2]
Notation: [role1 = filler1][role2 = filler2]
e.g., Existing fact:
Novel fact:
Lokendra Shastri
“John invited Mary”
“Susan invited Tom”
ICSI, Berkeley
Some assumptions underlying quantitative analysis
• Conceptual space
– 16,000 relations (chuck, throw, toss, lob …)
– 50,000 entities (serving as role-fillers)
• Cell and synapse count in various HS regions are
based on anatomical data, where available
• Plausible physiological parameters governing LTP
Lokendra Shastri
ICSI, Berkeley
Functional Characteristics of Episodic Memory
• Rapid, one-shot acquisition (~ 500 millisecond)
• high capacity (~ 50,000 episodes )
• responsiveness to partial cues (pattern completion)
• high specificity (pattern separation)
– response to a matching cue clearly distinct from response
to a near miss
– but support reminding
• robust against mild cell loss
Lokendra Shastri
ICSI, Berkeley
Neural mass recruited for each memory trace
“John gave Mary a book in the library on Tuesday”
Statistic
DG
[bind]
CA3
[bed]
CA2
[bei]
CA1
[remind]
Sub
[reinstate]
E<recruit>
(s)
195.0
(14.0)
16.0
(4.0)
13.2
(3.6)
51.4
(7.2)
56.0
(7.5)
•Probability of failure to encode a memory trace < 10
-18
•Sparse encoding, yet
•Highly redundant
•Copies of each functional unit are physically distributed
Lokendra Shastri
ICSI, Berkeley
Lokendra Shastri
ICSI, Berkeley
Episodic Memory
• Operates seamlessly with semantic memory and
procedural knowledge
• Serves as a basis for explanatory and predictive
inferences
episodic memory of having seen Tom walking a puppy
in the morning enables one to
•subsequently “explain” Tom buying dog food
•respond “probably yes” to Does Tom own a pet?
Lokendra Shastri
ICSI, Berkeley
Buy
-
+
?
buyer
buy-obj
<Tom bought
Fido at the pet
store on Friday>
<…>
<…>
<…>
HLCCs
Own
+ -
?
HS
owner own-obj
Fido
+
+e +v
Lokendra Shastri
?
?v ?e
Dog
+
?
Tom
ICSI, Berkeley
Buy
-
+
?
buyer
buy-obj
<Tom bought
Fido at the pet
store on Friday>
<…>
<…>
<…>
HLCCs
Own
+ -
?
HS
owner own-obj
Fido
+
+e +v
Lokendra Shastri
?
?v ?e
Dog
+
?
Tom
ICSI, Berkeley
Buy
-
+
?
buyer
buy-obj
<Tom bought
Fido at the pet
store on Friday>
<…>
<…>
<…>
HLCCs
Own
+ -
?
HS
owner own-obj
Fido
+
+e +v
Lokendra Shastri
?
?v ?e
Dog
+
?
Tom
ICSI, Berkeley
Buy
-
+
?
buyer
buy-obj
<Tom bought
Fido at the pet
store on Friday>
<…>
<…>
<…>
HLCCs
Own
+ -
?
HS
owner own-obj
Fido
+
+e +v
Lokendra Shastri
?
?v ?e
Dog
+
?
Tom
ICSI, Berkeley
Buy
-
+
?
buyer
buy-obj
<Tom bought
Fido at the pet
store on Friday>
<…>
<…>
<…>
HLCCs
Own
+ -
?
HS
owner own-obj
Fido
+
+e +v
Lokendra Shastri
?
?v ?e
Dog
+
?
Tom
ICSI, Berkeley
Buy
-
+
?
buyer
buy-obj
<Tom bought
Fido at the pet
store on Friday>
<…>
<…>
<…>
HLCCs
Own
+ -
?
HS
owner own-obj
Fido
+
+e +v
Lokendra Shastri
?
?v ?e
Dog
+
?
Tom
ICSI, Berkeley
Buy
-
+
?
buyer
buy-obj
<Tom bought
Fido at the pet
store on Friday>
<…>
<…>
<…>
HLCCs
Own
+ -
?
HS
owner own-obj
Fido
+
+e +v
Lokendra Shastri
?
?v ?e
Dog
+
?
Tom
ICSI, Berkeley
Predictions and explanations
•
Each HS sub-region serves a specific functional role in
supporting episodic memory
 cell-loss in distinct sub-regions leads to distinct forms of
memory deficits
e.g., false alarms, forgetting, selective inability to
respond to wh-queries
•
Patients with severe insult to CA1, but with intact EC, DG
and CA3 may detect novelty, in spite of being amnesic
•
Damage to DG is likely to make the “receptive field” of CA3
pyramidal cells more promiscuous
•
Greater DG activity is likely to lead to reduced CA3 and
CA2 activity (especially, during retrieval)
Lokendra Shastri
ICSI, Berkeley
Predictions and explanations
• An episodic memory trace can encode only a limited
number of bindings
 Events are construed in terms of specific action
schemas such as push, pull, and shove, rather
than in terms of generic schemas such as apply
force.
Lokendra Shastri
ICSI, Berkeley
The role of the HS in memory
• Memories of specific relational instances (events and
situations) must remain coded in the HS
• Consolidation of episodic memories involves
– consolidation of episodic memory trace in the HS
– fine-tuning of cortical representations (priors of
event types and conditional probabilities in
semantic structures)
– formation of semantic representations that are
generalizations and summarizations of experience
– formation of simpler representations of salient
events (e.g., sans source and spatiotemporal context)
Lokendra Shastri
ICSI, Berkeley
HS contribution to working memory
• HS is always online and rapidly records an event (~500msec)
– intact HS will (almost) always contribute to WM
• role of the HS becomes more critical as the “relational
complexity” of the event retained in WM increases
• during the delay period, recurrent flow of activity along the
PFC  HS loop would be crucial for holding a relational item
in WM
– PFC: serves as a “clamp”; HS provides the memory
encoding
Lokendra Shastri
ICSI, Berkeley
What necessitates HS involvement in the
memorization of an item?
• Item corresponds to a relational instance
– collection of bindings between roles and entities that fill
these roles in a given instance
• Item is to be memorized as a specific instance and
distinguished from other similar items
• Spatio-temporal context is part of the memory
• Role-fillers are from diverse (arbitrary) domains
• Role-fillers of multiple roles may belong to the same
domain
Lokendra Shastri
ICSI, Berkeley
Active forgetting (during sleep?)
• Activate a prior memory
• Evaluate its significance
• Consolidate memory if it is found to be
significant (pragmatically or emotionally
salient)
• Forget the memory if found to be
insignificant
• (change result from syanptic changes)
Lokendra Shastri
ICSI, Berkeley
Salient aspects of SMRITI
• A functionally adequate and biologically grounded
account of episodic memory function
• One-shot encoding (~ 500 msec) and rapid
retrieval of events and situations (relational
instances)
• Suggests a remarkable match between the form and
function of the HS
• Leads to several testable predictions
Lokendra Shastri
ICSI, Berkeley
Ongoing and future work
• Detailed modeling of interactions between episodic
memory and cortical representations
• semantic memory, sensorimotor schemas
• Acquisition of extended episodes (stories)
• Design of memory modules for “assisted
cognition” devices and autonomous agents
• Modeling other forms of rapid one-shot learning
• e.g., word acquisition
• Evaluation of predictions in collaboration with
psychologists and neurologists
Lokendra Shastri
ICSI, Berkeley
Ongoing and future work
•
Large-scale network simulations
•
Predict memory deficits resulting from
different cell-loss profiles associated with
aging and pathological brain conditions
–
•
Lokendra Shastri
behavioral tests for the very early detection of
Alzheimer’s disease
… finally, perhaps, develop a prosthesis for
hippocampal patients
ICSI, Berkeley
For publications…
www.icsi.berkeley.edu/~shastri
Lokendra Shastri
ICSI, Berkeley