Interactions between Prediction, Perception and Episodic Memory
DISSERTATION
Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy
in the Graduate School of The Ohio State University
By
Adam E. Hasinski
Graduate Program in Psychology
The Ohio State University
2015
Dissertation Committee:
Per B. Sederberg, Ph.D., Advisor
Trisha Van Zandt, Ph.D.
Julie Golomb, Ph.D.
© Copyrighted by
Adam E. Hasinski
2015
Abstract
For a variety of reasons, cognitive scientists tend to divide the study of the brain into
separate domains. Some of the most studied domains are episodic memory, visual
perception, and prediction. However, the underlying processes that give rise to one’s
memories, perceptions, and predictions do not operate independently. Instead each of
these processes influences the others. Previous research has demonstrated that the
memory system is critical for the generation of predictions (Buckner, 2010; Corbit &
Balleine, 2000; Johnson, Meer, & Redish, 2007; Lisman & Redish, 2009). Other
researchers have argued that portions of the memory system are actually a part of the
visual perception system (Baxter, 2009; Bussey & Saksida, 2007). Here we present
research that furthers our understanding of how these processes interact. First we
demonstrate, through a series of behavioral experiments, that predictions influence
memory. Specifically, in high-predictability environments, we find enhanced memory for
items that could have been predicted to recur, but did not. We then present a
computational model of recognition memory that attempts to formally explain this pattern
using a prediction-based learning mechanism. Finally, we present neural evidence that
the information content of the perceptual system is contingent on subsequent memory.
Taken together, these results demonstrate how the perception of present stimuli and the
prediction of future stimuli interact with the memory system.
ii
Dedicated to Dave Qualkinbush, Jim Petitto, Harry Finkbone, and Joe Panaccione,
who taught me how to focus, persevere, and how to get up after being knocked down.
iii
Acknowledgments
I would like to thank my advisor for his help and instruction over the years. I
would also like to thank the committee members for their time and feedback, both during
and before the dissertation process.
I would also like to thank my family, especially my parents, for their endless
support and patience. I did not fully appreciate their devotion until I came to graduate
school.
Finally, I would like to thank the friends I met in Columbus for their
encouragement and camaraderie over the years.
iv
Vita
June 2004 .......................................................Westlake High School
2004-2008 ......................................................Presidential Scholar, Ohio University
2008 ...............................................................B.A. Psychology & History, Ohio
University
2008-2009 .....................................................Graduate Fellow, Department of
Psychology, The Ohio State University
2009-2011 .....................................................Graduate Teaching Associate, Department
of Psychology, The Ohio State University
2010 ...............................................................M.A, Psychology, The Ohio State
University
2011-2015 .....................................................Graduate Research Associate, Department
of Psychology, The Ohio State University
Publications
T.A. Smith, A.E. Hasinski, P.B. Sederberg, “The context repetition effect: Predicted
events are remembered better, even when they don’t happen”. Journal of Experimental
Psychology: General, 142(4):1298, 2013.
v
M.A. Martens, A.E. Hasinski, R.R. Andridge, W.A. Cunningham, “Continuous cognitive
dynamics of the evaluation of trustworthiness in Williams syndrome”. Frontiers in
Psychology, 3, 2012.
J.J. Ratcliff, G.D. Lassiter, V.M. Jager, M.J. Lindberg, J.K. Elek, A.E. Hasinski, “The
hidden consequences of racial salience in videotaped interrogations and confessions”.
Psychology, Public Policy & Law, 16(2):200-218, 2010.
Fields of Study
Major Field: Psychology
vi
Table of Contents
Abstract ............................................................................................................................... ii
Acknowledgments.............................................................................................................. iv
Vita...................................................................................................................................... v
Table of Contents .............................................................................................................. vii
List of Tables ..................................................................................................................... xi
List of Figures ................................................................................................................... xii
Chapter 1: Introduction ...................................................................................................... 1
Context: The Cornerstone of Episodic Memory ............................................................. 2
Memory does not Operate in Isolation ............................................................................ 4
Memory and Perception............................................................................................... 5
Memory and Prediction ............................................................................................... 7
Chapter 2: Repeating an Item’s Temporal Context Enhances its Memorability .............. 10
Overview of Experiments .......................................................................................... 14
Experiment 1: A Predictable Environment with Scenes ............................................... 16
Method ....................................................................................................................... 16
vii
Results ....................................................................................................................... 20
Preliminary Discussion .............................................................................................. 22
Experiment 2: A Predictable Environment with Words................................................ 24
Results ....................................................................................................................... 25
Preliminary Discussion .............................................................................................. 29
Experiment 3: An Unpredictable Environment with Images ........................................ 30
Method ....................................................................................................................... 31
Results ....................................................................................................................... 31
Preliminary Discussion .............................................................................................. 33
Results ....................................................................................................................... 34
Preliminary Discussion .............................................................................................. 35
Discussion ..................................................................................................................... 35
Implications for the Study of Memory ...................................................................... 36
Open Questions and Future Directions ...................................................................... 39
Conclusions ............................................................................................................... 41
Chapter 3: A Formal Account of the Context Repetition Effect ...................................... 43
The Context Repetition Effect ................................................................................... 44
A Formal TD-SR Account of the Context Repetition Effect .................................... 47
TCM-SR: TCM with a TD-SR Learning Rule .............................................................. 49
viii
Simulation and Results .................................................................................................. 55
Hierarchical Bayesian Framework ............................................................................ 59
Results ....................................................................................................................... 64
Discussion ..................................................................................................................... 72
Other Models and the Context Repetition Effect ...................................................... 73
Differences from TD-SR ........................................................................................... 74
Limitations and Future Work .................................................................................... 76
Conclusions ............................................................................................................... 80
Chapter 4: Face-specific Information in the Fusiform Face Area Depends on Memory.. 81
Method .......................................................................................................................... 87
Subjects ...................................................................................................................... 87
Stimuli ....................................................................................................................... 87
Procedure ................................................................................................................... 88
MRI Data Acquisition ............................................................................................... 92
Calculating Face-specific Information ...................................................................... 98
Mixed-effects Regression ........................................................................................ 100
Results ......................................................................................................................... 103
Behavioral Results ................................................................................................... 103
Neuroimaging Results ............................................................................................. 105
ix
Discussion ................................................................................................................... 109
Conclusions ................................................................................................................. 113
Chapter 5: General Discussion........................................................................................ 114
Memory and Prediction ........................................................................................... 114
Memory and Perception........................................................................................... 117
Bridging Domains.................................................................................................... 118
Limitations and Future Directions............................................................................... 118
Recognition vs. Recall ............................................................................................. 118
The Neural Correlates of the Context Repetition Effect ......................................... 120
A Model of Memory, Perception, and Prediction ................................................... 122
The Role of Attention .............................................................................................. 123
Conclusions ................................................................................................................. 124
Bibliography ................................................................................................................... 125
x
List of Tables
Table 1. Free and fixed parameters. .................................................................................. 58
Table 2. Essential equations. ............................................................................................. 63
xi
List of Figures
Figure 1. Structure of encoding lists. Each color represents an example triplet-pair for
one of the four main experimental conditions. The leftmost condition, RCRT, does not
appear in Experiments 3 and 4. First and second triplets appear on upper and lower rows,
respectively. The lag between actual triplet pairs was randomized (see Experiment 1
methods). ........................................................................................................................... 18
Figure 2. Memory performance across conditions. Bars show the mean hit rate for all
target conditions and the false alarm rate for lures. Panels A, B, C, and D correspond to
Experiments 1, 2, 3, and 4, respectively. Error bars indicate standard errors of the mean.
........................................................................................................................................... 28
Figure 3. Context repetition effects. Bars show the effect of contextual repetition: Within
each panel, the left bar indicates the mean difference in hit rate between once-presented
targets whose contexts later repeat and control targets (RCNT1 vs. NCNT1), while the
right bar indicates the mean difference in hit rate between once-presented targets that
follow repeated contexts and control targets (RCNT2 vs. NCNT2). Panels A, B, C, and D
correspond to Experiments 1, 2, 3, and 4, respectively. Error bars indicate standard errors
of the mean. ∗ 𝑝 < 0.05. ................................................................................................... 29
Figure 4. Hierarchical structure. Each model variant was fit to all participants i from
Experiment 1 (High Predictability) and all participants j from Experiment 3 (Low
xii
Predictability). The same hyperpriors were used across both experiments for all
parameters except 𝛾. ......................................................................................................... 62
Figure 5. Model fits: Memory performance across conditions. Dark blue bars show the
mean hit rates across conditions, and false-alarm rates for lures. Light blue bars indicate
mean model behavior for each condition using MAP estimates. Panel rows correspond to
different experiments. ....................................................................................................... 65
Figure 6. Model fits: Context repetition effects. Dark blue bars show the behavioral
context repetition effect. Light blue bars indicate model behavior using MAP estimates.
Within each panel, the left bars indicate the mean difference in hit rate between oncepresented targets whose contexts later repeat and control targets (RCNT1 vs. NCNT1).
The right bars indicate the mean difference in hit rate between once-presented targets that
follow repeated contexts and control targets (RCNT2 vs. NCNT2). Panel rows
correspond to different model variants. Panel columns correspond to different
experiments. ...................................................................................................................... 66
Figure 7. Model fits: Individual differences. Scatterplots show the relationship between
each participant’s context repetition effect and the simulated context repetition effect for
their corresponding sub-model. Best fitting regression lines and 𝑅2 values are included.
Panel rows correspond to different model variants. Panel columns correspond to different
experiments and presentations. ......................................................................................... 67
Figure 8. Model comparison: Variants 1 and 2. The histogram shows the frequency of
within-participant differences in BPIC scores between Variants 1 and 2. Greater values
indicate a stronger preference for Variant 2, with 0 indicating no preference.................. 70
xiii
Figure 9. Model comparison: Variants 2 and 3. The histogram shows the frequency of
within-participant differences in BPIC scores between Variants 2 and 3. Greater values
indicate a stronger preference for Variant 3, with 0 indicating no preference.................. 72
Figure 10. Overview of experimental design. (A) Faces were presented one at a time, for
1.2 seconds, with a randomly varied interstimulus interval. Each target face occurred
twice. (B) Each presentation of a given target face occurred in either the same context
(low encoding variability) or in different contexts (high encoding variability). Same-face
similarity was computed between a first-presentation target (red square) and its identical
second presentation (dark-blue square). Different-face similarities were computed
between the first-presentations and non-identical second presentations with matching
genders and encoding variability (light-blue square)........................................................ 91
Figure 11. Functional and anatomical ROIs. Superposition of every subject’s FFA (red),
OFA (green) and PPA (blue), along with aIT (violet) in MNI standard space. ................ 97
Figure 12. Subsequent memory performance as a function of encoding condition. Bars
show mean proportion of faces recognized during the memory test for faces that occurred
under high encoding variability (dark green) and low encoding variability (light green).
For a reference, the mean proportion of once-presented items that were subsequently
remembered is shown in grey. Error bars reflect standard errors. The solid red line
indicates the mean proportion of lures that were incorrectly judged as having been seen
previously, with dashed red lines indicating standard error. .......................................... 105
Figure 13. Average similarity between face repetitions. Bars show similarity between
first presentation targets and either (dark blue) same or (light blue) different second
xiv
presentations, split by memory performance. (A) FFA shows significant face-specific
information–greater similarity between identical faces than between different faces of the
same gender–only for faces that are subsequently remembered. The difference in facespecific similarity between remembered and forgotten faces is also significant. (B) aIT
shows the same pattern, although the difference in face-specific information is no longer
significant after correcting for multiple comparisons. † 𝑝 < 0.05 uncorrected, ∗ 𝑝 < 0.05
corrected, ∗∗ 𝑝 < 0.01 corrected. ................................................................................... 108
xv
Chapter 1: Introduction
Research into human episodic memory has come a long way since Ebbinghaus
shuffled cards with three-character nonsense syllables, presented them to himself to the
beat of a metronome, and then tested his own memory (Ebbinghaus, 1913). Modern
episodic memory researchers replace shuffled cards with randomized study lists,
presented with precise timing via computer. The content of these lists can be anything:
printed or spoken words, images of faces, places, or even images collected from
participants’ own lives. As memory scientists adjust various factors of these lists, they
look for effects: reliable patterns of behavior from the participants they study. These
patterns provide insight into the mechanisms underlying memory. Some of the most wellknown effects include: primacy, enhanced memory for the first few items on a study list;
recency, enhanced memory for the last few list items; contiguity, the tendency to recall
items that were studied near a previously recalled item; and the von-Restorff effect,
enhanced memory for items that stand out from the rest of the list (Kahana, 2012). To
make sense of the dozens of memory effects and paradigms, memory theorists develop
computational models. These models are explanations of memory phenomena, written in
the language of math to avoid the ambiguities of informal languages. To date, no unified
theory of episodic memory has been proposed that can account for all of these
phenomena. Instead different research groups propose various models to explain handfuls
1
of effects or particular paradigms. While the mechanisms of these models vary widely, a
nearly ubiquitous feature is the use of context.
Context: The Cornerstone of Episodic Memory
Context is perhaps the single most important construct in the study of memory.
Most models of memory posit that effective retrieval from long-term memory depends
upon either associating studied items with context (Dennis & Humphreys, 2001; Gillund
& Shiffrin, 1984; Howard & Kahana, 2002; Polyn, Norman, & Kahana, 2009;
Raaijmakers & Shiffrin, 1981; Sederberg, Howard, & Kahana, 2008) or encoding an
item’s context, along with the item itself, as a memory trace (Hintzman, 1988). Specific
memories can be retrieved by cueing the memory system with a particular context. In the
real world this is presumed to happen spontaneously all the time. For instance, when you
enter a store—a specific context—you may be reminded of a friend you ran into the last
time you were in that store. In the laboratory, this often takes the form of asking
participants to recall items that occurred on specific study lists. Alternatively, the degree
of association between an item and a particular context can be used to judge whether or
not that item previously occurred in that context. As an example, an eyewitness may be
shown a lineup and asked if any of the suspects were at the scene of the witnessed crime.
In the lab, item recognition tests ask participants to judge whether test items occurred on
earlier study lists. Thus, context not only enables one to retrieve information from
memory, but also to identify when and where that information was encoded. Perhaps in
part because context is such an important and enduring idea in memory research, context
has been defined in many different ways.
2
In the most general sense, context refers to the relatively stable aspects of one’s
experience. “Relatively stable” certainly does not mean fixed or constant. In fact many
models of memory define context as something that changes (Estes, 1955; Howard &
Kahana, 2002; Shankar & Howard, 2012). Rather, context is something that is stable
relative to the more fleeting stimuli that make up an episode of experience. The actual
information that comprises context varies from theory to theory. Rather than thinking
about one or another theory as having the “correct” definition of context, it is useful to
see different theories and paradigms as focusing on different aspects of context.
A common definition of context is the spatial environment (e.g. a store or a
psychology lab). Changing the spatial context between study and test phases has been
shown to diminish memory performance (Falkenberg, 1972). For instance, one classic
study had divers learn and recall lists of words underwater and on land. Memory
performance was degraded when participants had to recall words in an environment that
was different from the encoding environment (Godden & Baddeley, 1975). Aspects of
one’s internal state can also be viewed as a type of context. Indeed changing various
aspects of one’s current physiology, including the amount of caffeine consumed
(Mystkowski, Mineka, Vernon, & Zinbarg, 2003) or degree of inebriation (Weingartner,
Adefris, Eich, & Murphy, 1976) has been shown to impede memory performance. Even
one’s mood can be considered a context, with studies again showing better memory for
congruent study-test moods (Ucros, 1989).
One commonality across the above-mentioned research is that context was treated
as discrete. Participants learned items in one stable context, and then were tested in either
3
that same context or a different one. Alternatively, context can be defined as evolving
continuously over time (Estes, 1955; Howard & Kahana, 2002; Shankar & Howard,
2012). Early conceptualizations of a continuously evolving context envisioned it as
fluctuating randomly (Estes, 1955). More recent accounts define context as a recencyweighted summation of temporally recent information (Howard & Kahana, 2002;
Shankar & Howard, 2012). In other words, recent experience serves as the context for
one’s current experience at this particular moment. As this moment passes, it becomes a
part of the context of the next moment. As time progresses and experiences drift farther
away from the current point in time, these experiences decay out of context. The rate of
this decay is often assumed to be constant (Howard & Kahana, 2002; Polyn et al., 2009;
Sederberg et al., 2008). However, recent work integrating theory from cognitive science
with empirical findings from animal learning strongly suggests that this decay may be
nonlinear, such that elements of context decay rapidly at first, but then remain active at
low levels for an extended period of time (Shankar & Howard, 2012).
As a final note, we emphasize a distinction between the actual information that
makes up one’s experience—be it environmental information or internally generated
information—and the memory system’s representation of that information. For brevity, in
this document the term “context” will be used to describe experiential information and its
representation in the brain.
Memory does not Operate in Isolation
Of course, episodic memory is just one of many cognitive processes, all of which
work together to produce human behavior and subjective experience. To completely
4
understand memory, researchers will need theories that describe the interactions between
episodic memory and these other processes. In order to develop integrative theories,
experimental research is needed to explore how manipulations of memory affect other
processes and vice versa. In the present research, we contribute to this endeavor by
exploring how memory interacts with prediction and perception. Specifically, we
investigate the relationship between perceptual representation and the episodic memory
system, as well as how the memory system is influenced by prediction.
We have chosen these three processes—memory, perception, and prediction—
because they represent stimuli and events at different points in time: the past, present, and
hypothetical future, respectively. At encoding, the memory system forms enduring traces
of the information that is being currently processed by the perceptual system. Later, these
traces can be retrieved in order to reconstruct previously perceived information.
Information in the memory system can also be used to generate predictions about what
events or stimuli are likely to be perceived in the near future. Thus, while these three
processes are likely fully interconnected, the relationships between memory and both
prediction and perception may be the most critical. We focus on these relationships in the
present work.
Memory and Perception
Memory and perception are typically viewed as distinct processes that correspond
to different neural substrates, the medial temporal lobe memory system (MTL) and the
ventral visual pathway, respectively. However, this view has been challenged in recent
years (Suzuki & Baxter, 2009). Some have argued that portions of the MTL, the
5
perirhinal cortex in particular, may also be involved in object perception (Baxter, 2009;
Bussey & Saksida, 2007). In essence, the MTL is viewed as an extension of the visual
pathway. Other researchers have contested this interpretation of the existing evidence
(Suzuki, 2009; Suzuki & Baxter, 2009). Additionally, recent neuroimaging work has
shown that some of the same parietal regions are active during both memory and
perceptual tasks (Cabeza et al., 2011). The researchers argue that these parietal regions
may be fulfilling similar attentional roles for both memory and perception. This claim is
consistent with the recent distinction made between external attention, the selection of a
subset the perceivable environment, and internal attention, the selection of certain stored
memories (Chun, Golomb, & Turk-Browne, 2011). Thus, the idea that memory and
perception overlap is not new; however, their relationship is far from understood.
In the present research, we further explore the connection between memory and
perception. We were specifically interested in the perception of, and memory for, human
faces. We placed participants inside an MRI scanner and used fMRI to measure their
brain activity while they viewed a series of unfamiliar faces. After participants exited the
scanner, they completed a surprise recognition test for the faces they saw. We used
multivariate neuroimaging techniques to compare the distributed neural representations
evoked by different and identical faces. Our investigation focused on the ventral visual
processing pathway, an interconnected set of regions responsible for representing what
we see. Our results showed evidence that face-specific information is present in ventral
visual regions. However, this face specific information was only found for faces that were
subsequently remembered during the recognition test. In other words, later memory for a
6
face is correlated with the amount of face-specific information present in the online
neural representation of that face. Although several interpretations of these findings are
possible, our results further establish the link between perception and memory.
Memory and Prediction
A great deal of research has shown that prediction is one of the core functions the
human brain (Bar, 2009). This is not surprising given the advantages offered by
prediction. As generating predictions requires a prior knowledge base, it is not surprising
that the episodic memory system—which encodes, maintains, and retrieves information
about experiences (Davachi, 2006; Eichenbaum, 2000, 2004)—has been also been
implicated as a source of predictions (Buckner, 2010; Johnson et al., 2007; Levy, 1989;
Levy, Hocking, & Wu, 2005; Lisman & Redish, 2009; Schacter, Addis, & Buckner,
2007). It may be that the same reconstructive processes that give rise to memory retrieval
also enable prediction and mental simulation more generally (Maguire & Hassabis, 2011;
Schacter & Addis, 2009).
In the present research, we argue that prediction can also influence memory. This
assertion stems from a framework (Gershman, Moore, Todd, Norman, & Sederberg,
2012) that merges reinforcement learning rules (Dayan, 1993; White, 1995), with models
of episodic memory (Howard & Kahana, 2002; Polyn et al., 2009; Sederberg et al.,
2008). Based on this framework, we developed a computational model that learns via
prediction and is fully capable of reproducing recognition memory behaviors. Because of
this prediction-based learning, the model posits a very particular effect: Once-presented
items will be better remembered if they are predicted to recur. However, this hypothesis
7
cannot be tested using conventional memory paradigms. The reason for this is that most
paradigms never repeat items. Even in cases where repetitions do occur, list
randomization eliminates any signal that would suggest repetitions may be coming. In
other words, nothing about the experimental setting enables the participant to effectively
generate predictions. To test this new model, we needed a new paradigm that enables the
memory system to make use of prediction-based learning.
We developed such a paradigm and conducted a series of behavioral experiments
to test whether prediction-based learning actually occurs when applicable. For some
study list items, a repetition of that item was preceded by a repetition of its original
context. Using context, participants could predict when these item repetitions would
occur. We focused on subsequent memory performance for other items that did not
repeat, even though their contexts did. Across two studies we demonstrated a context
repetition effect: Later repetition of an item’s context enhanced memory for that item. We
used different stimulus types across the two experiments, images and words, to ensure
that the results are not unique to one stimulus category. We did not find this effect in two
additional experiments where we eliminated the ability to predict item repetitions based
on context. These results suggest that the human memory system is capable of learning
based on prediction; however, it may only do so when predictions are perceived as
credible.
We begin in the next chapter by describing our behavioral paradigm and results in
more detail. For clarity, we keep our theoretical explanation pithy while discussing the
empirical findings. In the following chapter, we fully describe our theory. This
8
description is accompanied by a computational implementation, along with qualitative
and quantitative assessments of how well the model fits our behavioral data. We then use
neuroimaging to explore the interplay between memory and perception. Finally, we
present some considerations and future directions for the research presented here.
9
Chapter 2: Repeating an Item’s Temporal Context Enhances its Memorability1
The ability to predict the occurrence of future stimuli and environments confers
obvious advantages in terms survival and well-being. Indeed, this utility may have
secured prediction’s place as one of the fundamental functions of the brain, across
domains and levels of analysis (Bar, 2009). Prediction plays a potential role in everything
from perception (Stokes, Atherton, Patai, & Nobre, 2012) and stimulus–response learning
(Rescorla & Wagner, 1972) to the experience of emotions (Kirkland & Cunningham,
2011) and the navigation of social interactions (Mitchell, 2009). In particular, the medial
temporal lobe (MTL) has been implicated as a system for producing such predictions
(Buckner, 2010; Johnson et al., 2007; Levy, 1989; Levy et al., 2005; Lisman & Redish,
2009; Schacter et al., 2007).
The MTL has long been known to be critical for storing and retrieving episodic
memories (Eichenbaum, 2000, 2004), which it accomplishes by binding neural
representations of stimuli to representations of the context in which they are experienced
(Davachi, 2006). Context can refer to any element of one’s experience, including spatial
information (Mizumori, Ragozzino, Cooper, & Leutgeb, 1999), environmental
configuration (Fanselow, 2000), and internally generated mental information (Kennedy &
1
This chapter presents the research previously published in an article to The Journal of Experimental
Psychology: General (Smith, Hasinski, & Sederberg, 2013). The dissertation author was an author on that
publication.
10
Shapiro, 2004). As mentioned previously, when we refer to context, we are referring
specifically to temporal context: a neural representation of recently experienced past
states (Howard & Kahana, 2002). Previous research has utilized the concept of temporal
context to account for many aspects memory, including human behavioral effects
(Howard & Kahana, 2002; Polyn et al., 2009; Sederberg et al., 2008) and non-human
MTL function (Howard, Fotedar, Datey, & Hasselmo, 2005).
Given that the MTL is critical for the formation and retrieval of memories
(Davachi, 2006; Eichenbaum, 2000, 2004), as well as the use of those memories to
generate predictions (Buckner, 2010; Lisman & Redish, 2009; Schacter et al., 2007), a
logical question is whether or not these predictions can also subsequently influence
memory processes. The answer to this question depends on the learning mechanism used
by the MTL. Several models of episodic memory (Howard & Kahana, 2002; Polyn et al.,
2009; Sederberg et al., 2008) posit that the MTL uses Hebbian learning (Hebb, 1949),
whereby associations are formed between context and the to-be-learned stimulus. Other
memory models similarly learn associations between stimuli that occur close in time, as
well as between those stimuli and a list context (Gillund & Shiffrin, 1984; Raaijmakers &
Shiffrin, 1981). No variant of this simple associative learning involves prediction. If the
memory system utilizes a simple Hebbian or associative learning rule, as these models
suggest, then predictions should not influence memory.
Alternatively, “prediction-based” learning rules also modify associations between
context and predicted stimuli. These predictions are generated using information the
memory system has already learned. Variants of prediction-based learning have been
11
successfully used in models of stimulus-response learning (Rescorla & Wagner, 1972)
and semantic memory (Shankar, Jagadisan, & Howard, 2009). Prediction-based learning
is also widely used in the field of reinforcement learning (Sutton & Barto, 1998) to
develop agents that maximize rewards by optimally navigating from state to state (Dayan,
1993; White, 1995). Although prior theoretical work has proposed applying a predictionbased learning rule to the task of learning episodes (Gershman et al., 2012), to our
knowledge the behavioral consequences of using such rule have not been tested
empirically.
Comparing prediction-based learning to simple Hebbian learning is not
meaningful using typical memory paradigms where encoding lists are carefully
randomized to avoid regularities or where each stimulus is presented only once. When
stimuli are encountered only once, predictions cannot be effectively made. As a result,
prediction-based and Hebbian learning rules would behave the same way: merely binding
each stimulus to context. When sequences of stimuli are repeated, however, predictions
about what stimuli will be experienced next can be made based on prior experience.
These predictions can be used to enhance learning beyond what is possible with
traditional Hebbian learning. Under Hebbian learning, repetitions will lead to
strengthened memory only for the repeated stimuli. In contrast, prediction-based learning
rules can also modify memory based on what is predicted to recur. Thus, comparing
memory models using Hebbian vs. prediction-based learning requires an experimental
design where study lists occasionally repeat sequences, thereby introducing predictability
into the experimental environment.
12
In such an environment, prediction-based learning can deviate from Hebbian
learning in two ways: Firstly, predictions about what the current stimulus should be—
based on prior learning—can curb additional learning for the current stimulus.
Specifically, stimuli that were accurately predicted to recur do not require much
additional strengthening. This is analogous to error-driven rules, used in animal models
of stimulus-response learning (Rescorla & Wagner, 1972), where only the predictionerror is learned. In this way, prediction-based learning can avoid catastrophic strength
growth, which can plague Hebbian algorithms (Goodhill & Barrow, 1994).
Secondly, by learning predictions about what stimulus should appear in the near
future—again based on previous experience—memory can also be strengthened for
stimuli that were predicted to recur but never do. For example, if a sequence of three
stimuli is experienced and later the first two stimuli from that sequence are reexperienced, one may then predict to see the third stimulus again. A prediction-based
learning system could learn this prediction, strengthening memory for the third stimulus
even if it does not actually recur. Simple Hebbian learning will not learn this prediction.
We will define that third stimulus as a “target” because memory strength for it will allow
us to discriminate between Hebbian and prediction-based accounts of episodic learning.
Consistent with prior memory research (Howard & Kahana, 2002; Polyn et al., 2009;
Sederberg et al., 2008), we will define the two stimuli that preceded the target as its
“context.” Concretely, prediction-based learning rules predict that a once-presented
stimulus whose temporal context is later repeated should be better remembered relative to
other once-presented stimuli whose contexts do not repeat. Hebbian learning rules predict
13
no difference in subsequent memory. We will refer to enhanced memory for oncepresented targets whose temporal contexts later repeat as a “context repetition effect.”
In summary, prediction-based learning offers advantages over Hebbian learning.
By making use of prediction, a memory system could both bound the growth of memory
strengths and strengthen memory for items that are predicted to reappear but do not.
Whether the human memory system actually employs prediction-based learning is an
open empirical question. Because Hebbian and prediction-based learning produce
divergent predictions only when sequences of stimuli are repeated, one cannot
discriminate between these frameworks using conventional memory paradigms where
stimulus order is randomized to avoid regularities. Hence, we developed a paradigm with
repeated stimulus sequences designed to elicit a context repetition effect. The presence or
absence of such an effect would be evidence that the brain uses prediction-based or
Hebbian learning, respectively. We conducted a series of experiments to answer this
question.
Overview of Experiments
Here we present four experiments that tested for the existence of prediction-based
learning as indicated by the presence or absence of a context repetition effect.
Specifically, we created a controlled environment that featured repetitions of sequences
of stimuli. In Experiments 1 and 2, the list sequences were designed such that the
recurrence of certain target stimuli could be predicted by the repetition of context stimuli
that preceded these targets’ first presentations. Experiments 3 and 4 omitted this
regularity. We used two different stimulus classes—images for Experiments 1 and 3 and
14
words for Experiments 2 and 4. These classes were used to investigate whether any
effects found could be influenced by stimulus imageability or rehearsability. The
resulting set of four experiments comprise a 2 (environmental predictability: high
vs. low) X 2 (stimulus type: images vs. words) factorial design.
The predictability factor investigates whether the presence or magnitude of a
context repetition effect is contingent on the existence of predictable repetitions in the
current environment. Because prediction-based learning will be most useful in
circumstances where accurate prediction is possible, we posit that the memory system
modulates its generation or use of predictions based on environmental predictability. The
result should be a down-weighting of prediction-based learning in unpredictable
environments. Therefore, we hypothesize that a context repetition effect will occur only
in the high-predictability experiments (1 and 2) and not—or at least to a lesser degree—in
the low-predictability experiments (3 and 4).
The stimulus type factor investigates the generality of a context repetition effect
across stimulus classes that differ in terms of imageability and rehearsability. If context
repetition effects are the result of stimulus vividness or rehearsal (Hasher & Zacks,
1979), then such an effect should only appear in Experiment 1 or 3, respectively. We do
not have strong a priori hypotheses regarding this factor.
15
Experiment 1: A Predictable Environment with Scenes
Method
Participants. Participants were 40 Ohio State introductory psychology students.
Each participant provided informed consent and participated in exchange for partial
credit in the course.
Design. Experiments 1 and 2 utilized an identical, within-subject design. The
experiment was divided into 3 parts: encoding, distraction, and test.
During encoding, participants experienced study lists as streams of stimuli,
presented one at a time. Unbeknownst to participants, these stimuli were arranged as
pairs of triplets. For each triplet, the first two stimuli were defined as the (temporal)
context of the third stimulus: the target. These triplets were paired such that each triplet
had a corresponding triplet that occurred either earlier or later in the same list. The lag
between these corresponding triplets varied randomly from six to 21 stimuli (i.e. two to
seven intervening triplets). Figure 1 provides a visualization of the triplet-pair structure
and experimental conditions.
The different experimental conditions were determined based on differences
between triplets of each pair. Experiments 1 and 2 utilized a 2 (context: same
vs. different) X 2 (target: same vs. different) factorial design. This led to four primary
conditions: Firstly, a repeated-context, repeated-target condition (RCRT) featured
identical context and target stimuli for both triplets within each triplet-pair (i.e. 2
identical triplets for each pair). Critically, this condition creates a high-predictability
environment where the recurrence of targets can be predicted when a context is repeated.
16
Secondly, a repeated-context, novel-target condition (RCNT) features identical contexts
but different targets for both triplets within each triplet-pair (i.e. the first and second
elements of both triplets in each pair are identical). This condition provides an
opportunity to elicit a context repetition effect: Later repetition of a target’s context may
enhance memory for that target even though it is never repeated. Together these two
conditions create an experimental environment where the repetition of a particular
context indicates a 50% chance that the next stimulus will be the target that previously
followed this context.
Thirdly, a novel-context, repeated-target condition (NCRT) features identical
targets but different contexts for both triplets in each triplet-pair (i.e. the third element of
both triplets in each pair are identical). This condition allows us to estimate memory
performance for twice-presented stimuli in the absence of contextual repetitions.
Fourthly, in a novel-context, novel-target condition (NCNT) neither the context stimuli
nor the target stimuli are identical for paired triplets (i.e. no identical elements between
triplets of a given pair). This condition serves as a control, allowing us to estimate
baseline memory performance for once-presented stimuli.
Note that this design produces four nested sub-conditions within the two noveltarget conditions: RCNT1 or NCNT1, the first triplet within each triplet-pair, and RCNT2
or NCNT2, the second triplet within each triplet-pair. To look for effects of contextual
repetition on memory for once-presented targets, we contrasted the memory performance
for each repeated-context novel-target sub-condition with its corresponding novel-context
novel-target sub-condition (RCNT1 vs. NCNT1 and RCNT2 vs. NCNT2, respectively).
17
The first contrast lets us test for a context repetition effect for memory. The second
contrast allows us to distinguish between associative and predictive accounts of this
context repetition effect.
Figure 1. Structure of encoding lists. Each color represents an example triplet-pair for
one of the four main experimental conditions. The leftmost condition, RCRT, does not
appear in Experiments 3 and 4. First and second triplets appear on upper and lower rows,
respectively. The lag between actual triplet pairs was randomized (see Experiment 1
methods).
Materials. We chose images as stimuli because we wanted a stimulus set that
would not lend itself to verbal rehearsal. Specifically, we used a pool of 645 color
photographs of indoor (336) and outdoor (309) scenes that had been used in previous
research (Turk-Browne, Simon, & Sederberg, 2012). Stimuli for each participant were
drawn randomly from this pool. Each study list consisted of 96 presentations, but only 72
unique stimuli, because some presentations were repetitions. We presented five unique
encoding lists to each participant. For each encoding list, we created a corresponding 96item test list. Each test list contained all 24 targets from the corresponding encoding list
along with 24 randomly selected context items and 48 lures drawn at random from the
stimulus pool. We used the Python Experiment Programming Library (Geller, Schleifer,
18
Sederberg, Jacobs, & Kahana, 2007) to present stimuli to participants and record their
responses.
Procedure. Participants first completed the encoding phase, during which they
were exposed to five lists of stimuli. They were not told that memory would be assessed
later, in an effort to capture incidental learning. Each stimulus was presented for 1200
ms, centered on a monitor, with a 600- to 800-ms jittered interstimulus interval. To keep
participants attending to the stimuli, we asked them to complete a task during the
encoding phase. Participants judged each stimulus to be an image of either an indoor or
outdoor scene by pressing one of two keys.
Participants then completed the distraction phase, which consisted of an n-back
task for 5 minutes. The n-back task is divided into a series of blocks. During each block,
participants are shown a stream of numbers, one at a time. During each presentation,
participants judge whether the current number is the same as the number that occurred n
presentations earlier. The value of n varies across blocks, from 1 to 4, with higher
numbers presenting greater challenge.
Finally, they completed a surprise item-recognition test, which assessed memory
for all target stimuli, a subset of the context stimuli, and an equal number of non-studied
lures. Test probes were presented one at a time for 1,200 ms, with a 600- to 800-ms
jittered interstimulus interval. On each test trial, participants judged a probe to be “old” or
“new” by pressing one of two keys. The order of test trials was randomized and, unlike
the encoding phase, did not feature any repetition or regularity.
19
Results
We quantified memory performance in terms of within-participant hit rates and
false-alarm rates: the number of tested stimuli that a participant judged as “old” divided
by the total number of stimuli for which they made judgments. Hit rates were calculated
separately for targets and context stimuli in each condition and sub-condition and false
alarm rates were calculated for lures (see Figure 2). Considering all the stimuli at once, a
2 (context repetition: novel vs. repeated) X 2 (target repetition: novel vs. repeated) X 2
(stimulus type: context vs. target) repeated measures analysis of variance (ANOVA)
revealed main effects of context repetition, F(1, 39) = 45.90, MSE = 0.0091, p < .0001,
and target repetition, F(1, 39) = 12.15, MSE = 0.0119, p = .0012, but no significant main
effect of stimulus type, F(1, 39) = 2.08, MSE = 0.0080, p = .1570. We also found
significant interactions between stimulus type and both context repetition, F(1, 39) =
26.36, MSE = 0.0084, p < .0001, and target repetition, F(1, 39) = 20.32, MSE = 0.0063, p
< .0001. Given that we found significant interactions with stimulus type, we proceeded to
separately analyze target and context stimuli.
Memory for Targets. We first looked for differences in memory performance
between the four main conditions. A 2 (context repetition: novel vs. repeated) X 2 (target
repetition: novel vs. repeated) repeated measures ANOVA on target stimuli demonstrated
a main effect of target repetition, F(1, 39) = 36.38, MSE = 0.0075, p < .0001. Repeated
targets (M = 0.55, SEM = 0.017) were more likely to be remembered than once-presented
targets (M = 0.47, SEM = 0.016). We found no significant main effect of repeating
context, F(1, 39) = 1.75, MSE = 0.0087, p = .1940. Additionally, we found no interaction
20
between context and target repetition (F < 1). In other words, only the repetition of
targets improved memory performance for targets.
We then examined the nested sub-conditions for the once-presented stimuli,
looking for evidence of a context repetition effect via planned contrasts. We found that
once-presented targets were more likely to be recognized if their contexts were later
repeated (RCNT1) relative to other once-presented targets whose contexts were never
2
repeated (NCNT1), t(39) = 2.46, p = .0185, 𝜔𝑝𝑎𝑟𝑡𝑖𝑎𝑙
= 0.06. In contrast, we found no
difference in memory performance for once-presented targets whose contexts had already
been presented (RCNT2) relative to corresponding once-presented targets with novel
contexts (NCNT2), t(39) = 0.11, p = .9 (see Figure 3, panel A).
Memory for Contexts. Again, we first looked for differences in memory
performance between the four main conditions. A 2 (context repetition: novel
vs. repeated) X 2 (target repetition: novel vs. repeated) repeated measures ANOVA on
context stimuli demonstrated a main effect of context repetition. Specifically, context
stimuli were more likely to be remembered F(1, 39) = 71.51, MSE = 0.0087, p < .0001,
when they were repeated (M = 0.59, SEM = 0.018) than if they were presented only once
(M = 0.47, SEM = 0.015). This finding parallels the effect of target repetition on memory
for targets, mentioned above. The ANOVA revealed no significant main effect of target
repetition (F < 1) or the interaction between context and target repetition (F < 1).
Similar to our analysis of the target stimuli, we next examined the nested subconditions, looking for an effect of target repetition on their corresponding context
stimuli. Planned contrasts revealed no effect of later target repetition on memory for
21
once-presented novel context stimuli. This null result was found both for the first triplets
from each triplet-pair (NCRT1 vs. NCNT1), t(39) = 0.1638, p = .80, as well as the
second triplets from each triplet pair (NCRT2 vs. NCNT2), t(39) = 0.
Preliminary Discussion
Experiment 1 tested for evidence of a context repetition effect in a moderately
predictable environment. Specifically, we looked for changes in recognition memory for
once-presented images due to repetition of their temporal contexts at a later point in time.
We found that repeating the context of a once-presented target (RCNT1) increased the
likelihood of subsequently recognizing that target. However, contextual repetition had no
effect on memory for the new target (RCNT2) that followed a context’s second
presentation.
Not surprisingly, we also observed item repetition effects for both context and
target stimuli: Twice-presented images were more likely to be subsequently recognized
than once-presented images. Additionally, we found no evidence of a bidirectional effect
of repetition. That is, repetition of a target did not alter the probability of subsequently
remembering the context that preceded either its first (NCRT1) or second (NCRT2)
presentation.
This counterintuitive pattern of effects cannot be explained by traditional theories
of memory. Strength-based theories of recognition memory (Murdock, 1993) have no
mechanism for improving memory performance for an item that is not repeated. Other
theories, using associative learning mechanisms to explain recognition memory (Dennis
& Humphreys, 2001; Gillund & Shiffrin, 1984; Howard & Kahana, 2002; Shiffrin &
22
Steyvers, 1997), also have problems explaining the full pattern of results. Generally,
these models say that the likelihood of recognizing a target depends on the strength of the
association between that target and context. Some models could predict that repeating a
context would increase that context’s memory trace. This could then enhance memory for
all targets associated with this context, making them seem more familiar at test. This
could explain half of the context repetition effect: enhanced memory for targets whose
contexts are later repeated. However, this mechanism would also predict enhanced
memory for the novel targets that follow repetitions of context. In contrast, the context
repetition effect in Experiment 1 shows enhanced memory only for targets that follow the
first presentation of a twice-presented context. Some associative theories of memory
would also predict a target repetition effect: enhanced memory for context items that
precede twice-presented targets. However, our results do not bear out this prediction.
A theory of memory that includes a prediction-based learning mechanism
naturally explains the full pattern of results. When a context is experienced for the first
time, the memory system has no basis for making a prediction about what target could
appear next. However, the context does become associated with the actual subsequent
stimulus via a Hebbian-like learning process. When this context occurs again at a later
time, these context-target associations enable the memory system to predict that the same
target may also be repeated. If this prediction reinstates the target’s representation then
the target’s associations to context will be further strengthened. Thus, strengthening can
occur even if the target does not actually repeat. In this case, the novel target that follows
the repeated context (RCNT2) would become associated to that context just like any
23
other once-presented target (e.g. NCNT2). If the target actually does repeat as predicted,
the amount of additional Hebbian learning that occurs during the repetition will only be
as large as the prediction error (Rescorla & Wagner, 1972). Thus, the more a target is
strengthened in memory due to prediction, the less additional strengthening will occur as
a result of its actual repetition. Thus, repeating the context of a target could enhance
memory for it almost as much as repeating the target itself.
Experiment 2: A Predictable Environment with Words
Our second experiment had two purposes: Firstly, we attempted to replicate the
pattern of results found in Experiment 1. Secondly, we used a different stimulus class to
ensure that the results of Experiment 1 are not specific to scenes or highly imageable
stimuli more generally. Thus, Experiment 2 featured an identical design, but used words
as stimuli instead of scenes.
Method
Participants. Participants were 31 Ohio State introductory psychology students.
Each participant provided informed consent and participated in exchange for partial
credit in the course.
Design, Materials, and Procedure. We employed the same experimental design
used in Experiment 1. The stimulus pool consisted of 630 medium- to high-frequency
words, which were selected from the University of Southern Florida free-association
database (Nelson, McEvoy, & Schreiber, 2004). For each participant, a random
subsample from this pool was selected to create unique lists, paralleling Experiment 1.
The only difference in procedure from Experiment 1 was a change to the task that
24
participants performed during the encoding phase. In Experiment 2, participants judged
whether each word represented a living or nonliving thing. Although there is no correct
answer for many of the words in the pool, a prior internal pilot study demonstrated that
approximately the same numbers of words were judged to be living and nonliving.
Results
Considering all stimuli at once, a 2 (context repetition: novel vs. repeated) X 2
(target repetition: novel vs. repeated) X 2 (stimulus type: context vs. target) repeated
measures ANOVA indicated a main effect of context repetition, F(1, 30) = 17.16, MSE =
0.0069, p = .0003; along with a marginal main effect of target repetition, F(1, 30) = 4.12,
MSE = 0.0114, p = .0514. However, we found but no main effect of stimulus type (F <
1). As with Experiment 1, we also found a significant interaction between target
repetition and stimulus type, F(1, 30) = 8.79, MSE = 0.0123, p = .0059. However, unlike
Experiment 1, the interaction between context repetition and stimulus type was not
significant, F(1, 30) = 1.31, MSE = 0.0112, p = .20. Due to the significant target
repetition by stimulus type interaction, we next analyzed context and target items
separately, paralleling the analyses of Experiment 1.
Memory for Targets. We first looked for differences in memory performance
between the four main conditions. A 2 (context repetition: novel vs. repeated) X 2 (target
repetition: novel vs. repeated) repeated measures ANOVA on target items revealed a
main effect of target repetition, F(1, 30) = 13.09, MSE = 0.0115, p = .0011. Specifically,
twice-presented targets (M = 0.67, SEM = 0.022) were more likely to be subsequently
remembered than once-presented targets (M = 0.60, SEM = 0.018). We also found a
25
marginal main effect of context repetition, F(1, 30) = 4.08, MSE = 0.0060, p = .0525.
Targets presented with repeated contexts (M = 0.65, SEM = 0.022) were remembered
marginally better than targets presented with novel contexts (M = 0.62, SEM = 0.019).
We did not find a significant interaction between target and context repetition (F < 1).
We then examined the nested sub-conditions for the once-presented stimuli,
looking for evidence of a context repetition effect via planned contrasts. These contrasts
replicate the results from Experiment 1. Once-presented targets whose contexts were later
repeated (RCNT1) were more likely to be subsequently recognized, relative to other
once-presented targets whose contexts were never repeated (NCNT1), t(30) = 2.24, p =
2
.0327, 𝜔𝑝𝑎𝑟𝑡𝑖𝑎𝑙
= 0.06. Also consistent with Experiment 1, we found no difference in
memory performance between once-presented targets that followed a repetition of
context (RCNT2) and control targets (NCNT2), t(30) = -0.63, p < .5 (see Figure 3, panel
B).
Memory for Contexts. Shifting to the context stimuli, we first looked for
differences in memory performance between the four main conditions. A 2 (context
repetition: novel vs. repeated) X 2 (target repetition: novel vs. repeated) repeated
measures ANOVA on context stimuli revealed a main effect of context repetition, F(1,
30) = 8.94, MSE = 0.0120, p = .0055, consistent with Experiment 1. Twice-presented
context stimuli were better remembered (M = 0.67, SEM = 0.023) than once-presented
contexts (M = 0.61, SEM = 0.019). We found no significant main effect of target
26
repetition on memory for context stimuli (F < 1). Similarly we found no significant
interaction between context and target repetition, F(1, 30) = 2.95, MSE = 0.0138,
p = .0964.
Planned contrasts, testing for an effect of target repetition on memory for the
preceding context stimuli, once again showed no effect of target repetition on memory
performance for once-presented contexts. This was true whether a context preceded the
first presentation of a target (NCRT1 vs. NCNT1), t(30) = 0.55, p = .50, or the second
presentation (NCRT2 vs. NCNT2), t(30) = 0.71, p = .40.
27
Figure 2. Memory performance across conditions. Bars show the mean hit rate for all
target conditions and the false alarm rate for lures. Panels A, B, C, and D correspond to
Experiments 1, 2, 3, and 4, respectively. Error bars indicate standard errors of the mean.
28
Figure 3. Context repetition effects. Bars show the effect of contextual repetition: Within
each panel, the left bar indicates the mean difference in hit rate between once-presented
targets whose contexts later repeat and control targets (RCNT1 vs. NCNT1), while the
right bar indicates the mean difference in hit rate between once-presented targets that
follow repeated contexts and control targets (RCNT2 vs. NCNT2). Panels A, B, C, and D
correspond to Experiments 1, 2, 3, and 4, respectively. Error bars indicate standard errors
of the mean. ∗ 𝑝 < 0.05.
Preliminary Discussion
Experiment 2 replicated the main results from Experiment 1 while using a
different stimulus category, words instead of images. In addition to expected item
29
repetition effects for both context and target stimuli, we once again found a context
repetition effect: enhanced recognition memory performance for once-presented items
whose temporal context is later repeated, but no enhancement for once-presented images
that follow a repetition of context. Together with the findings from Experiment 1, these
results provide strong evidence for the context repetition effect and suggest that it may
occur for a wide variety of stimulus categories.
Experiment 3: An Unpredictable Environment with Images
We describe the encoding phases of Experiments 1 and 2 as “predictable
environments” because the participants saw sequences of three stimuli (context and
target) fully repeated at a later time (the RCRT condition) as well as sequences of two
stimuli (context only) repeated (the RCNT condition). Thus, when a context recurred, it
was possible to predict what target would follow, albeit not perfectly. The RCRT
condition may have indicated to participants that this type of prediction had utility in the
experimental environment. In other words the predictable environment may have
prompted participants’ memory systems to generate or make use of these predictions,
which ultimately enhanced memory.
We test this hypothesis in Experiments 3 and 4 by removing the RCRT condition,
thereby reducing the predictability of the experimental environment. Specifically, target
repetitions can no longer be predicted from repetitions of context. We predict that the
reduction in predictability will reduce a memory system’s use of prediction. This should
eliminate, or at least attenuate, the context repetition effect.
30
Method
Participants. Participants were 52 Ohio State introductory psychology students.
Each participant provided informed consent and participated in exchange for partial
credit in the course.
Design, Materials, and Procedure. Experiment 3 was identical to Experiment 1,
except that the RCRT condition was omitted. This reduced the predictability of the
environment by ensuring that participants never saw repeated sequences of more than two
stimuli. The resulting design featured single main condition with three-levels: NCRT,
RCNT, and NCNT. Consistent with the previous studies, the novel target conditions
contained nested sub-conditions: RCNT1, RCNT2, NCNT1, and NCNT2. The materials
and procedure were also identical to Experiment 1, except that the number of triplet pairs
in each condition was increased slightly to keep the duration and stimulus exposure
comparable to Experiment 1. Each participant saw a total of 450 stimulus presentations
(375 unique pictures), across five blocks. As with Experiments 1 and 2, test lists were
constructed to correspond to each encoding list. Each test list featured all 25 target items
and 20 randomly chosen context items from the relevant encoding list, along with 45
lures.
Results
Because the context and target repetition factors no longer create a 2 X 2 factorial
design (as they did in Experiments 1 and 2), the three remaining conditions were merged
into a single factor. The result is a 3 (condition: NCRT, RCNT, and NCNT) X 2
(stimulus type: context vs. target) repeated measures ANOVA. Paralleling Experiments 1
31
and 2, we found a significant main effect of condition, F(2, 102) = 11.08, MSE = 0.0078,
p < .0001, and a significant interaction between condition and stimulus type, F(2, 102) =
42.49, MSE = 0.0072, p < .0001. Also paralleling Experiments 1 and 2, we found no
main effect of stimulus type (F < 1). We then used planned pairwise t tests to test for
differences between the three experimental conditions separately for targets and contexts.
Memory for Targets. Planned comparisons revealed better memory for repeated
targets (NCRT) than once-presented targets with either once-presented contexts (NCNT),
t(51) = 6.45, p < .0001, or twice-presented contexts (RCNT), t(51) = 7.04, p = .0001.
Additionally, whether context was twice-presented (RCNT) or not (NCNT) had no
significant overall effect on memory for once-presented targets, t(51) = 1.06, p = .20.
Thus, consistent with Experiments 1 and 2, target repetition, but not context repetition
was associated with better memory for target items.
We then tested the novel-target sub-conditions for the context repetition effect.
Importantly, we found no such effect for once-presented targets whose contexts were
later repeated (RCNT1 vs. NCNT1), t(51) = -0.42, p = .60, or for once-presented targets
that followed a repetition of context (RCNT2 vs. NCNT2), t(51) = -0.89, p = .30 (see
Figure 3, panel C).
Memory for Contexts. Memory for context stimuli displayed a pattern consistent
with our previous experiments. Specifically, we found an item repetition effect: better
subsequent memory for twice-presented contexts (RCNT) relative to once-presented
contexts that preceded either once-presented targets (NCNT), t(51) = 4.81, p < .0001, or
twice-presented targets (NCRT), t(51) = 5.45, p < .0001.
32
We also found no evidence that repeating a target influenced memory for the
context items that preceded either the target’s first presentation (NCRT1 vs. NCNT1),
t(51) = 0.34, p = .70, second presentation (NCRT2 vs. NCNT2), t(51) = 0.03, p = .90, or
both presentations taken together, t(51) = 0.22, p = .80.
Preliminary Discussion
Many of Experiment 3’s results are similar to findings from Experiment 1.
However, unlike Experiment 1, we found no evidence of a context repetition effect. The
only difference between these experiments was the removal of the RCRT condition for
Experiment 3. This suggests that predictable environments are necessary to produce a
context repetition effect. To replicate this null result, we ran a second experiment without
the RCRT condition.
Experiment 4: An Unpredictable Environment with Words
Method
Participants. Participants were 71 Ohio State introductory psychology students.
Each participant provided informed consent and participated in exchange for partial
credit in the course.
Design, Materials, and Procedure. Experiment 4’s design was identical to the
design used in Experiment 3; however, the stimulus pool was the same set of words used
in Experiment 2. Accordingly, the encoding task was the same living-nonliving judgment
used in Experiment 2.
33
Results
Experiment 4’s results were similar to the results of Experiment 3. The 3
(condition: NCRT, RCNT, and NCNT) X 2 (stimulus type: context vs. target) repeated
measures ANOVA detected a significant main effect of condition, F(2, 140) = 16.63,
MSE = 0.0065, p < .0001, a significant interaction between condition and stimulus type,
F(2, 140) = 43.56, MSE = 0.0070, p < .0001, but no significant main effect of stimulus
type (F <1).
Memory for Targets. Consistent with the previous studies, we found an item
repetition effect for targets: Twice-presented targets (NCRT) were more likely to be
remembered than once-presented targets with novel contexts (NCNT)), t(70) = 7.24, p <
.0001, or repeated contexts (RCNT), t(70) = 8.09, p < .0001. Additionally, subsequent
memory performance did not differ for novel targets with novel vs. repeated contexts,
t(70) = 0.53, p < .50.
We again found no evidence of a context repetition effect: Recognition memory
was not improved for once-presented targets whose context was later repeated (RCNT1
vs. NCNT1), t(70) = -0.50, p = .60, or for once-presented targets that followed a
repetition of context (RCNT2 vs. NCNT2), t(51) = -0.19, p = .80 (see Figure 3, panel D).
34
Memory for Contexts. Once again, repeated contexts were more likely to be
remembered than the novel contexts that preceded either novel targets, t(70) = 5.49, p <
.0001, or repeated targets, t(70) = 5.51, p < .0001. Further, target repetition did not affect
subsequent memory for contexts (NCRT vs. NCNT), t(70) = 0.69, p = .40. Null effects of
target repetition on context were also found when we separately tested first presentations
(NCRT1 vs. NCNT1), t(70) = 1.11, p = .20, and second presentations (RCNT2 vs.
NCNT2), t(70) = 0.05, p = .90.
Preliminary Discussion
The absence of a detectable context repetition effect in Experiments 3 and 4 is
evidence supporting our claim that only predictable environments enable or cue the
memory system to use prediction-based learning. In environments where meaningful
predictions from context are not possible, the repetition of a target’s temporal context will
not improve memory for that target. Experiments 3 and 4 suggest that human memory
processes are sensitive to the utility of predictions in the current environment and that use
of prediction-based learning is modulated by this utility.
Discussion
In two experiments, we found that repeating the temporal contexts of oncepresented stimuli enhanced their memorability relative to once-presented control stimuli
whose contexts were never repeated. However, we found no difference in subsequent
memory for the similarly once-presented stimuli that followed a repetition of context
relative to once-presented control stimuli. We termed this combined pattern of behavior
the context repetition effect. We expected this pattern if the episodic memory system
35
learned using predictions (Gershman et al., 2012) rather than simply learning contextstimulus associations (Howard & Kahana, 2002). In order to elicit the effect, we
constructed encoding lists where sequences of stimuli were occasionally repeated after
some delay. Thus, sometimes the repetition of a target stimulus could be predicted after a
repetition of the contextual stimuli it originally followed.
A very different pattern emerged when we reduced the predictability of the
experimental environment. In two additional experiments, where full context-target
pairings were never repeated, we found no effect of contextual repetition for the oncepresented targets that followed either presentation of a twice-presented context. These
four experiments, taken together, are strong preliminary evidence that the human episodic
memory system employs a type of prediction-based learning when—and perhaps only
when—the current environment provides regularity. That is, the memory system can
encode information that is predicted to appear in the environment, even when that
information does not, in fact, appear. However, this will only occur when these
predictions have a reasonable chance of being accurate. We found this context repetition
effect for multiple stimulus classes: pictures of scenes and words.
Implications for the Study of Memory
The existence of a context repetition effect means some sort of prediction-based
learning mechanism must be incorporated into any theory that intends to be a truly
comprehensive explanation of human memory. Indeed, while many models of memory
that utilize Hebbian learning (Howard & Kahana, 2002; Polyn et al., 2009; Sederberg et
al., 2008) or similar mechanisms (Gillund & Shiffrin, 1984; Raaijmakers & Shiffrin,
36
1981) are sufficiently powerful to account for many episodic memory effects, they would
fail to reproduce the context repetition effect. No model currently offers anything close to
complete account of memory. Why, then, should models of memory undergo revision to
account for this particular effect, especially when it only manifests under peculiar
experimental circumstances involving repetitions of sequences of stimuli (i.e. contexts
and targets)?
Such regularity is intentionally avoided at all cost in typical, tightly-controlled
memory experiments. Thorough randomization procedures are used to ensure that item or
inter-item idiosyncrasies do not give rise to false positives or mask real effects. In fact,
the memory lists used in most paradigms avoid any stimulus repetition whatsoever.
Studies of the spacing effect (Glenberg, 1979) and item repetition effects (Ratcliff,
Hockley, & McKoon, 1985) are exceptions to this norm only because they, by definition,
require repetition of individual items. Even in these cases, however, list randomization is
still paramount, albeit more complicated, and predictable patterns are still avoided.
Consequently, a context repetition effect could never appear in a typical memory
experiment because the repetition of sequences required to elicit the effect would never
occur.
However, repetitions and regularities are ubiquitous outside the controlled
environment of the laboratory. The route one takes to any frequently visited destination
provides a prime example of repetition of sequences in the real world. Even in cases
where events are not guaranteed to occur—such as whether or not one will meet a
neighbor while walking down the street—potential outcomes can often be predicted.
37
These scenarios are analogous to the experimental environment created for the present
research. Given the amount of regularity—and hence predictability—in real-world
experience, the effects of prediction-based learning may be prevalent in everyday life.
Therefore, rather than viewing the context repetition effect as a peculiar phenomenon,
elicited by contrived circumstances, we view it as an important effect that provides
information both about how the MTL system learns and about the types of memory
phenomena that are being ignored using traditional encoding lists.
Fortunately, the present research is part of a growing literature that emphasizes
the importance of regularity, prediction, and the use of naturalistic stimuli. For instance,
spaced presentations of semantically related words lead to enhanced memory for the first
presentation. This occurs because the presentation of the second, related word reminds
the participant of the first word (Tullis, Benjamin, & Ross, 2014). Prediction is also
important for segmenting one’s stream of experience into separate episodic events
(Zacks, Speer, Swallow, Braver, & Reynolds, 2007). The predictability of the
environment also influences neural responses underlying the perception of visual stimuli
(Summerfield, Trittschuh, Monti, Mesulam, & Egner, 2008; Turk-Browne et al., 2012).
Still other researchers use advancements in experience sampling techniques to investigate
memory for stimuli that actually occur in participants’ everyday lives (Milton et al.,
2011; Nielson, Smith, Sreekumar, & Sederberg, submitted). Experience sampling
methods potentially offer the best chance to capture the regularities present in everyday
life.
38
Open Questions and Future Directions
Although the effects of prediction-based learning and environmental regularities
may be commonplace, important boundary conditions may exist. The current work used
two different stimulus classes: images of scenes, which are both highly imageable and
difficult to rehearse, as well as medium to high frequency words, which are not as
imageable but much easier to rehearse. We found the context repetition effect for both
stimulus types. However, different stimuli may not produce these effects. For instance,
images of fractal patterns, which are imageable but relatively difficult to discriminate,
may not produce a context repetition effect. Another question is the degree of
predictability required to elicit a context repetition effect. In Experiments 1 and 2, which
produced context repetition effects, a repetition of context preceded the repetition of a
target 50% of the time. In Experiments 3 and 4, which did not produce a discernable
context repetition effect, a contextual repetition never signaled the repetition of a target
(i.e. 0% of the time). More work is needed to determine the predictability threshold at
which a context repetition effect will appear, as well as how the magnitude of the effect
relates to further increases in predictability (e.g. 75%). Finally, the effects of additional
repetitions of context are currently unknown. How a third repetition of context, with a
different novel target following it (i.e. RCNT3), influences memory for the targets that
followed the first (RCNT1) and second (RCNT2) presentations of context is an open
question.
Additionally, the neurological mechanisms that give rise to the context repetition
effect specifically, and prediction-based learning more generally are not fully understood.
39
The MTL is believed to play a critical role in the generation of predictions from episodic
memory (Buckner, 2010; Johnson et al., 2007; Levy, 1989; Levy et al., 2005; Lisman &
Redish, 2009; Schacter et al., 2007). However, other brain regions also seem to respond
to manipulations of predictability (Summerfield et al., 2008; Turk-Browne et al., 2012).
Much of this work centers on regions within the ventral visual processing stream, and
looks for changes in neural repetition attenuation: a reduction in neural responsiveness to
subsequent presentations of a particular stimulus (Grill-Spector, Henson, & Martin,
2006). For instance, the fusiform face area, a region of the inferior temporal lobe that
preferentially responds to faces (Kanwisher & Yovel, 2006; Kanwisher, McDermott, &
Chun, 1997), shows greater repetition attenuation during periods (i.e. experimental
blocks) when repetition is more likely to occur (Summerfield et al., 2008). Additionally,
the parahippocampal place area, a region that preferentially responds to scenes (Epstein
& Kanwisher, 1998), produces greater repetition attenuation when a repetition of a target
scene follows a repetition of its initial temporal context (Turk-Browne et al., 2012).
Future work will be required to determine where the neural correlates of the context
repetition effect lie. These correlates may be found in, and perhaps only in, the MTL.
Alternatively, the context repetition effect may have neural correlates in other regions,
perhaps depending on the stimuli used. For instance, scenes and words may produce
neural context repetition effects in the parahippocampal place area (Epstein &
Kanwisher, 1998) and word sensitive regions (Dehaene, Le Clec’H, Poline, Le Bihan, &
Cohen, 2002; McCandliss, Cohen, & Dehaene, 2003), respectively. Our results, along
40
with the existing findings of prediction-related effects in different brain areas, strongly
support the claim that prediction is a fundamental function of the brain (Bar, 2009).
Finally, although the prediction-based learning account is based on mechanisms
from reinforcement learning (Dayan, 1993; Gershman et al., 2012; White, 1995), it has
never been implemented in a full recognition memory model. Even up until this point, we
have only presented it as a verbal theory. This enabled us to focus on the empirical
findings, namely the context repetition effect. However, now that we have established
this effect, we must investigate the extent to which prediction-based learning actually can
account for memory enhancements due to contextual repetitions. We accomplish this in
the next chapter where we describe and test a computational implementation. In that
chapter, we will briefly review the experimental design and main findings before fitting
our model to some of the data described in this chapter. It turns out that capturing both
the context repetition effect in high-predictability environments and the null context
repetition effect in low-predictability environments will require a few important
deviations from conventional prediction-based learning rules.
Conclusions
The painstaking efforts memory researchers go through to “properly” randomize
their study lists may actually be causing them to miss important memory phenomena
caused by environmental regularities. Given that the environment outside the laboratory
is full of regularity and periodicity, it is not surprising that the brain, especially the MTL
makes use of it. We have shown that, in predictable environments, memory for a oncepresented stimulus can be improved by repeating the temporal context of that stimulus.
41
To our knowledge, this behavior can only be explained through a prediction-based
learning mechanism. The results presented here are part of a recent push to examine the
role that such regularities have on memory, perception, and the function of the brain more
generally (Bar, 2009).
42
Chapter 3: A Formal Account of the Context Repetition Effect2
As mentioned in the previous chapter, a considerable literature has developed to
explain the relationship between memory and prediction. Researchers studying both
humans and animals have found converging evidence that the medial temporal lobe
system (MTL), a collection of regions known to be critical for storing and retrieving
memories (Davachi, 2006; Eichenbaum, 2000, 2004), is also involved in making
predictions (Buckner, 2010; Johnson et al., 2007; Levy, 1989; Levy et al., 2005; Lisman
& Redish, 2009; Schacter et al., 2007). Sequences of events are critical for the generation
of predictions (Hawkins, George, & Niemasik, 2009), and the MTL seems especially
suited for learning sequential information (Lisman & Redish, 2009; Skaggs,
McNaughton, Wilson, & Barnes, 1996). Although the neocortex may also learn
sequences and use them to generate predictions (Hawkins et al., 2009), some have argued
that the MTL is required for all but the most basic predictions (Corbit & Balleine, 2000).
It seems then that prediction-making may actually be a primary function of the MTL and
a prevalent process in the brain overall (Bar, 2009; Schacter et al., 2007).
To date, most research at the intersection of memory and prediction has focused
on how memories or the memory system can enable and influence predictions. Less
2
This chapter presents research that is being written up in as a manuscript to be submitted to Psychonomic
Bulletin and Review. The authors will be Hasinski and Sederberg.
43
research has investigated how predictions may influence memory. One exception to this
trend is the set of experimental findings from the preceding chapter (Smith et al., 2013),
which demonstrated a context repetition effect. That is, participants were more likely to
subsequently recognize a once-presented item as occurring in a study list if the context of
that item was repeated later in the study phase. However, no boost in memory
performance was found for once-presented items that followed a repeated context. This
result cannot be readily explained by existing models of episodic memory. We present a
new computational model of memory that formally explains and reproduces the context
repetition effect. We first summarize the prior experimental findings, before explaining
our computational model and procedure. We devote to the rest of this article to assessing
model fit, comparing model variants, and discussing implications and future directions.
The Context Repetition Effect
Deviating from typical episodic memory paradigms that strive to avoid
predictable repetitions of stimuli, we designed a paradigm that intentionally repeated
sequences of stimuli (Smith et al., 2013). More precisely, sequences of three stimuli were
occasionally repeated in the same order at later points during the same encoding list. We
denoted the third item in each triplet as a “target” of interest and defined the preceding
two items as the “context” of the target. The term context is appropriate because the first
two items comprise the most recent experience, or temporal context (Howard & Kahana,
2002), of the target. We named this condition the repeated-context repeated-target
condition (RCRT). The experiment also featured a condition where targets were
presented twice, but the two context stimuli that preceded each presentation of a
44
particular target were unique for each target presentation. We referred to this as the
novel-context repeated-target condition (NCRT). Of particular importance was a third
repeated-context novel-target condition (RCNT), where context stimuli were presented
twice, but the target that followed each presentation of context was unique. As a result,
targets in this condition can be broken down into two sub-conditions: RCNT1 and
RCNT2, corresponding to the targets that followed first or second presentations of
particular contexts, respectively. Finally a novel-context novel-target condition (NCNT)
served as a control, its triplets consisting of only once-presented stimuli. NCNT1 and
NCNT2 sub-conditions correspond to the RCNT1 and RCNT2 sub-conditions. The entire
paradigm consisted of five encoding lists, then a short distractor task, and finally five
corresponding test lists. Readers are referred to the previous chapter for more details
about design and procedure.
Looking at the recognition performance across participants, we found two effects
of repetition. Firstly, we found an effect of item repetition: Twice-presented items were
more likely to be subsequently recognized as occurring during study than once-presented
items. This effect is not surprising as the relationship between additional presentations
and memory performance has been demonstrated (Ratcliff et al., 1985). More
importantly, we also found an effect of contextual repetition on memory for the targets
that followed: Participants were more likely to remember targets whose temporal
contexts were repeated at a later point in time (RCNT1) than other once-presented targets
whose contexts were never repeated (NCNT1). Furthermore, this effect was asymmetric:
Once-presented targets that followed the repetition of a context (RCNT2) were no more
45
likely to be subsequently recognized than control targets (NCNT2). These results were
consistent across two experiments that used very different stimulus categories: images of
scenes and words.
However, a very different pattern emerged when we removed the repeatedcontext repeated-target condition from the experimental design. In two additional
experiments, we found no evidence of elevated recognition memory for once-presented
targets whose contexts were later repeated relative to once-presented control targets
(i.e. RCNT1 vs. NCNT1). Memory performance for once-presented targets that followed
a repeated context did not differ from similar once-presented control targets (RCNT2
vs. NCNT2), consistent with the first two studies. In other words, we found a null context
repetition effect. Additionally, memory for twice-presented targets was significantly
better than memory for once-presented targets, replicating the traditional item repetition
effect that was also found in the first two studies.
To explain the context repetition effect, we previously proposed a verbal theory
centered on prediction-based learning. We argue that the episodic memory system, which
is believed to generate predictions (Bar, 2009; Buckner, 2010; Lisman & Redish, 2009),
may also be influenced by these predictions. In particular, when the memory system
predicts that a particular stimulus or event will appear in the near future, this may
reactivate the representation of that stimulus. This reactivation can then strengthen
memory for the stimulus, perhaps nearly as much as if the item had actually been
repeated. Furthermore, we argue that repetition of item sequences is the critical indicator
that predictions and should be made. As a result, predictions should only occur in
46
predictable environments where accurate forecasting is possible. Experiments 1 and 2
qualify as predictable environments because the presence of the RCRT condition lets
participants learn that occasionally sequences of three stimuli will be repeated. Thus,
when a context is repeated in an RCNT2 triplet, a participant may make a prediction—
perhaps nonconsciously—that the RCNT1 target will reappear. The result would be a
reactivation and strengthening of the RCNT1 target in memory, even though it is never
actually repeated. However, in Experiments 3 and 4, the RCRT condition is missing,
preventing participants from ever developing an expectation that three items in a row
would be repeated. Consequently, predictions are not utilized, and the RCNT1 targets
receive no additional learning.
Although this verbal theory sounds reasonable, it is far from a complete
explanation. It is missing necessary details about how the memory system strengthens
items and turns those memory strengths into recognition judgments. Further, we have not
yet put forth evidence that such a theory can account for the data. We next outline a
formalization of this verbal theory. We present a computational model that inherits
principles both from previous memory models and the reinforcement learning literature.
This model specifies all of the components required to reproduce the recognition
behavior found previously (Smith et al., 2013). Following this outline, we demonstrate
that the model can account for item and context repetition effects.
A Formal TD-SR Account of the Context Repetition Effect
Accounting for the context repetition effect described above requires a memory
model to produce higher likelihoods of recognizing once-presented items whose temporal
47
contexts are later repeated (RCNT1) relative to the likelihoods of recognizing oncepresented control items (NCNT1). However, the model must not produce elevated
recognition likelihoods for once-presented items that follow a repetition of context
(RCNT2). This turns out to be a difficult pattern of behavior to replicate. Most models of
memory would require enhanced or additional processing of an item (Gillund & Shiffrin,
1984; Hintzman, 1988; Murdock, 1993) in order to increase its memory strength. This
could come from many factors, such as elevated attention, additional presentations, or
longer presentation duration. However, none of these factors can explain the context
repetition effect because the RCNT1 targets are presented only once and for the same
duration as any other stimulus. Further, the fact that their contexts are not repeated until
later means there is no cue to make these targets more salient to participants. Some
associative models could explain the enhanced memory for RCNT1 targets: Contextual
repetition would strengthen the representation of context items in memory. Strengthened
context items could then increase the likelihood of recognizing any targets that were
associated with them. However, this would also boost memory for RCNT2 targets, which
is inconsistent with the context repetition effect. It seems, then, that a new mechanism is
required.
One plausible mechanism is a learning rule that makes use of prediction, known
in the reinforcement learning literature as the temporal difference-successor
representation (TD-SR) (Dayan, 1993; White, 1995). As a reinforcement learning
algorithm, TD-SR provides a mechanism for an agent to successfully navigate from one
state to another so as to maximize long-term rewards. Previous research has shown that
48
the TD-SR framework parallels the temporal context model (TCM), a computational
model of memory (Howard & Kahana, 2002). In fact, under certain conditions, as when
the environment avoids repetitions, a TD-SR model simplifies to a Hebbian learning
model that is equivalent to TCM (Gershman et al., 2012). However when regularities
occur, such that sequences of states repeat, TD-SR makes use of past experience to
generate predictions which enhance learning. We propose that a TCM-like memory
model with a TD-SR learning rule (Gershman et al., 2012), which we will refer to as
TCM-SR, can qualitatively account for both aspects of the context repetition effect. This
mechanism is consistent with the verbal theory that was put forth previously to explain
the context repetition effect (Smith et al., 2013). However, the current investigation
formally defines a principled mechanism and implements a complete model that can be
quantitatively evaluated. The remainder of this article focuses on explaining the
computational model and then testing it.
TCM-SR: TCM with a TD-SR Learning Rule
TCM-SR inherits its basic architecture, a two-layer associative network, from
TCM (Howard & Kahana, 2002). An item layer, 𝑓, represents the currently experienced
stimulus3 as a vector of features. The context layer, 𝑡, is the hallmark of TCM-like
models and represents the temporal context of the currently experienced stimulus.
Specifically, the context vector represents recent experience—information that was
recently represented in the item layer—such that the more recently something was
experienced, the stronger its representation will be in context. The context vector can be
3
This can include internally generated information; however, for the present purposes, we will restrict the
item layer to representing the features of the current experimental stimulus.
49
thought of as a graded version of a short-term memory store (Gillund & Shiffrin, 1984).
These two layers are linked via an associative matrix, 𝑀, such that each element of the
context vector is connected to every element of the item layer. The strengths or weights
of these connections are adjusted through learning. The weight of each connection
corresponds to an element value within 𝑀. 𝑀 represents long-term memory and is
queried at test to determine an item’s memory strength.
A new stimulus is presented to the model by activating that stimulus’
representation in item layer, 𝑓. The model then learns by increasing the connection
weights between active elements of 𝑓 and active elements of the context vector, 𝑡. In
most variants of TCM, this learning rule is a simple associative or Hebbian learning
process (Hebb, 1949):
𝑀 = 𝑀 + 𝛼𝑓1 𝑡0𝑇 , (1)
where 𝑀 is a matrix of connection weights, 𝑓1 is a column vector representing the new
stimulus features, 𝑡0 is a column vector representing the current temporal context, and T
represents the transpose operation, turning a column vector into a row vector. The 𝛼
parameter governs the rate of learning. The larger 𝛼 is, the greater the increase to the
connection weights (i.e. the more the model will learn). For the purposes of the present
simulations, we fixed 𝛼 to .5. Importantly this learning rule makes no use of prediction
and cannot reproduce the context repetition effect.
In contrast, the TD-SR learning rule from reinforcement learning makes use of
previously acquired information to generate predictions. It then incorporates these
predictions into the learning mechanism,
50
𝑀 = 𝑀 + 𝛼[𝑓1 − 𝑀𝑓0 + 𝛾𝑀𝑓1 ]𝑡0𝑇 , (2)
where 𝑀𝑓0 is a query of long-term memory using the features of the previous item, 𝑓0 .
The query uses what has already been learned—weights in 𝑀—to determine what has
followed the previous item, 𝑓0 , in the past and then predict that those features should have
reappeared during the current item presentation. Similarly, 𝑀𝑓1 is a query to 𝑀 using the
new item features to generate a prediction about what features are expected to appear
next. The 𝛾 parameter takes values on the interval [0, 1] and weights these predictions,
with larger values of 𝛾 producing stronger predictions. Hence 𝛾 will be critical in
explaining the context repetition effect. The term 𝑓1 − 𝑀𝑓0 is equivalent to the prediction
error term in error-driven learning (Rescorla & Wagner, 1972; Sutton & Barto, 1998).
𝑀𝑓0 serves two functions: Firstly, it keeps connection weights from growing without
bound, because if the prediction was perfect then it will cancel out 𝑓1 . Unbounded weight
growth can be problematic for purely Hebbian models (Goodhill & Barrow, 1994).
Secondly, 𝑀𝑓0 lets the model unlearn predictions that are not borne out. While this is a
useful feature for a reinforcement learning algorithm trying to maximize rewards, it is
less than ideal for a memory system trying to retain episodic information.
To adapt TD-SR to episodic memory, we introduce an important change for the
TCM-SR learning rule:
𝑀 = 𝑀 + 𝛼⟨𝑓1 − 𝑀𝑓0 + 𝛾𝑀𝑓1 ⟩+ 𝑡0𝑇 , (3)
where ⟨•⟩+ represents a rectifying operation such that any elements within • that are less
than zero are set to zero. This modification prevents the model from unlearning features
that were predicted but did not recur. Although this deviates from the TD-SR rule used in
51
reinforcement learning (Dayan, 1993), it makes intuitive sense for an episodic memory
system. A memory system that unlearns past events just because they are not repeated
when expected would be problematic.
When a new stimulus is encountered, both item and context layers are updated.
For item layer, 𝑓, the update is straightforward: Vector elements are activated to the
degree that they represent the stimulus, and no activation for the previous stimulus
remains. That is 𝑓 simply changes from 𝑓1 to 𝑓2 . The updating of context layer, 𝑡, is more
involved. First, the input to 𝑡 is computed. The input, 𝑡 𝐼𝑁 , is a weighted combination of
the previous item’s features and any context information previously bound to those
features:
𝑡 𝐼𝑁 = ∥𝛽𝑓1 + (1 − 𝛽)𝑓1 𝑀∥∥, (4)
where 𝑡 𝐼𝑁 is a vector representing the input to context, 𝑓1 is a vector representing the
previous item features, and 𝑓1 𝑀 represents a query to long-term memory to reinstate any
context information that was previously associated with 𝑓1 . The result of this query is
another vector representing reinstated context. This reinstated context vector would be a
vector of zeros if 𝑓1 has never been experienced before. The 𝛽 parameter takes values on
the interval [0, 1] and governs the tradeoff between how much of 𝑓1 vs. 𝑓1 ’s reinstated
context contributes to 𝑡 𝐼𝑁 . Thus smaller values of 𝛽 mean that input to context will be
dominated by the previous item’s reinstated context, provided that item has been bound
to a prior context. Finally, ∥•∥ represents a normalizing operation to ensure that • is unit
length.
52
Once input to 𝑡 has been calculated, context itself evolves according to the
following rule:
𝑡1 = ∥𝜌𝑡0 + 𝑡 𝐼𝑁 ∥, (5)
where 𝑡1 and 𝑡0 are the new and old context vectors, respectively, and 𝜌 is a parameter on
the interval [0, 1] that governs the rate at which context drifts. Larger values of 𝜌 lead to
a context vector that evolves more slowly. In contrast, a value of 𝜌 at or close to 0 would
produce context vectors with little to no representation of less recent experience.
Finally, we set the value of a specific element of 𝑡, which we denote as
representing the context of the learning phase (i.e. a list context), to
𝑡1𝑙𝑖𝑛𝑑 = 𝜆, (6)
where 𝑙 𝑖𝑛𝑑 denotes the element of 𝑡 representing this list context and 𝜆 is another
parameter on the interval [0, 1], that denotes the value of the list context. Thus the
activation of the list context element will be constant throughout a simulation.
Importantly, the normalizing operation in Equations 4 and 5 do not include the list
context element in the normalization process. That is, vectors 𝑡 𝐼𝑁 and 𝑡 each have a
length of one, ignoring the list context element. The list context element will prove to be
critical for recognition.
During the test phase, test probes are presented to the model just like study items.
Context continues to evolve with each test probe. However, we allow the value of 𝛽 to be
different at test than it was at encoding, so that
𝑡 𝐼𝑁 = ∥𝛽𝑅 𝑝 + (1 − 𝛽)𝑝𝑀∥∥, (7)
53
where 𝑝 is the test probe, and 𝛽𝑅 is a free parameter, on the interval [0, 1], that governs
the degree of context reinstatement during retrieval. This allows the model to have
different degrees of context reinstatement at encoding and retrieval. This is important
because successful recognition will depend on the values of the reinstated context vector,
𝑡 𝐼𝑁 .
The presentation of each test probe first leads to a new input to context, according
to equation 7. The probe’s memory strength, 𝑠, is then computed as a function of the
similarity between this new input to context and the previous context:
𝑠 = 𝑡 𝐼𝑁 ⋅ 𝑡0 𝜉 + 𝜈, (8)
where 𝑠 is the resulting memory strength, 𝑡0 denotes the previous context, and ⋅ denotes
the dot product operation. The more similar 𝑡 𝐼𝑁 and 𝑡0 are, the larger the dot product will
be. The 𝜉 parameter takes values on the interval [0, ∞) and multiplicatively scales this
dot product while 𝜈, a parameter on the interval (−∞, ∞), is an additive increment. When
the probe is a lure, the dot product will be 0. Thus 𝜈 represents the minimum amount of
strength probes will have during the test phase, regardless of evidence coming from longterm memory. Both 𝜈 and 𝜉 are critical for fitting the model’s performance to actual
human recognition behavior.
To compare TCM-SR to behavioral data, which consists of old/new recognition
judgments for every test probe, we must convert each probe’s memory strength, 𝑠, to a
likelihood of responding “old.” To do this, we use a signal detection theory (SDT) (Green
& Swets, 1966) likelihood ratio method (Glanzer, Hilford, & Maloney, 2009).
Specifically, the model makes a standard SDT assumption that memory strength values
54
can come from one of two normal distributions: a noise distribution, if a probe is a lure
that was not studied, or a signal distribution, if a probe was a studied target. We compute
the likelihood, 𝑜, that the model judged each probe to be “old” by taking the probability
density of the strength under the signal distribution over the summed probability densities
of the strength under both the signal and lure distributions, such that
𝑜 = 𝑑𝑛 (𝑠, 𝜇𝑆 , 𝜎𝑆 )/(𝑑𝑛 (𝑠, 𝜇𝑆 , 𝜎𝑆 ) + 𝑑𝑛 (𝑠, 𝜇𝑁 , 𝜎𝑁 )), (9)
where 𝑜 is the resulting likelihood of judging a probe as old, 𝑠 is the memory strength, 𝜇𝑆
and 𝜇𝑁 are parameters for the means of the signal and noise distributions, respectively,
and 𝜎𝑆 and 𝜎𝑁 are standard deviation parameters for the signal and noise distributions
,respectively. Finally, 𝑑𝑛 (𝑥, 𝜇, 𝜎) calculates the probability density of a normal
distribution, with mean, 𝜇, and standard deviation, 𝜎, at value 𝑥. We fix 𝜇𝑁 to be 0 and
set 𝜇𝑆 as a free parameter on the interval [0, ∞). We also make an equal variance
assumption, fixing both 𝜎𝑆 and 𝜎𝑁 to 1. Given these simplifications, the likelihood of
judging a test probe as old becomes:
𝑜 = 𝑑𝑛 (𝑠, 𝜇𝑆 , 1)/(𝑑𝑛 (𝑠, 𝜇𝑆 , 1) + 𝑑𝑛 (𝑠, 0,1)). (10)
Combining a TCM framework with a modified TD-SR learning rule and an SDT
decision rule, we have all the components necessary to reproduce recognition memory
results. In the next section, we explain how we fit this model to the data.
Simulation and Results
We chose to fit our model to Experiments 1 and 3 from the previously described
behavioral research (Smith et al., 2013). Both experiments used images as stimuli.
Experiment 1 created a predictable environment that produced a context repetition effect
55
across participants. Experiment 3 removed the condition that created this predictability.
As a consequence, no context repetition effect was found for Experiment 3. TCM-SR
should reproduce both patterns of behavior. We predict that the only significant change to
the fitted parameters between these two experiments will be the value of 𝛾. Specifically,
we predict higher values of 𝛾 for Experiment 1, relative to Experiment 3.
Experiments 1 and 3 had 40 and 52 participants, respectively. All participants
completing five encoding lists and then five corresponding test lists, with a single
distraction phase separating the encoding and test phases. Because stimuli did not occur
on multiple lists for a given participant, we did not have to worry about inter-list
interactions. Therefore, we trained and tested TCM-SR on one list at a time.
To model the encoding phase, we first initialized the model with an empty longterm memory matrix, 𝑀. We then presented each list item to the model, by activating
specific item features in item layer, 𝑓. For simplicity, we chose a unit-length, localist
representation for each item. Specifically, each item representation was a vector of zeros
except for a single element set to 1. Each item representation was orthogonal to every
other item representation. A localist representation scheme reflects the sparse, largely
non-overlapping representations of the hippocampus (Norman & O’Reilly, 2003) and has
been used successfully in similar models of episodic memory (Howard & Kahana, 2002;
Polyn et al., 2009; Sederberg et al., 2008). However, this representation scheme does not
allow us to realistically simulate perceptual processes. For each presentation, the model
learned associations according to Equation 3 and then updated context according to
Equations 4 and 5.
56
The presence of a distraction phase between encoding and test lists means that
elements of context that were active at the end of the encoding phase (i.e. elements
representing the last several items to occur during encoding) should no longer be active at
the start of the test phase. This is a consequence of the continual evolution of context
according to Equation 5. To simulate this drift in the model, we simply evolved context
once using a more extreme drift parameter:
𝑡1 = ∥𝜌𝑑𝑖𝑠𝑡 𝑡0 + 𝑓𝑑𝑖𝑠𝑡 ∥, (11)
where 𝜌𝑑𝑖𝑠𝑡 was set to .01 in order to clear previous context, 𝑡0 , and 𝑓𝑑𝑖𝑠𝑡 is a feature
vector, orthogonal to all items, representing the distraction task.
To model the test phase, we presented each test probe to the model. This included
lure items that were represented in the same localist fashion as studied target items. For
each probe, memory strength was assessed according to Equation 8, and then the
probability of judging the probe as a studied item was calculated according to Equation
10.
We opted to fit our model to each trial rather than to summary statistics. By fitting
to the trial-level data, we need not concern ourselves with determining a sufficient set of
summary statistics (Turner & Sederberg, 2013). Because our memory model returns a
likelihood for each test trial, fitting the model to all trials simply involves the
multiplication of likelihoods:
𝑛
𝑙 = ∏ 𝑟 (𝑜𝑖 ) + (1 − 𝑟)(1 − 𝑜𝑖 ), (12)
𝑖=1
57
where 𝑙 is the likelihood of the model responding as the participant did on all test probes,
𝑟 represents the participant’s actual response to each probe such that 𝑟 is 0 and 1 for
“new” and “old” responses, respectively, and 𝑜 is the model’s likelihood of responding
“old” to each probe.
For computational efficiency, we first log-transform the trial-level likelihoods,
and sum these log-likelihoods across all trials:
𝑙𝑙 = ∑𝑟log 𝑒 (𝑜) + (1 − 𝑟)log 𝑒 (1 − 𝑜), (13)
where 𝑙𝑙 is the resulting log-likelihood value. We fit our model to the data using these
log-likelihood values.
Table 1. Free and fixed parameters.
Parameter
𝛾
𝛽
𝛽𝑅
𝜌
𝜆
𝜉
𝜈
𝜇𝑆
𝜂
𝛼
𝜇𝑁
𝜎𝑆
𝜎𝑁
𝜌𝑑𝑖𝑠𝑡
Purpose
prediction coefficient
context reinstatement
context reinstatement at test
contextual drift
list context activation
probe strength coefficient
baseline memory strength
signal mean
negative scaling
learning rate
noise mean
signal variance
noise variance
probe strength coefficient
58
Domain
[0,1]
[0,1]
[0,1]
[0,1]
[0,1]
[0,∞)
(−∞,∞)
(−∞, ∞)
[0,1]
.5
0
1
1
.01
Equations
2,3,14,15
4
7
5
6
8
8
9,10
15
1,2,3,14,15
9
9
9
11
Hierarchical Bayesian Framework
We fit our model using a hierarchical Bayesian framework. This framework
requires us to specify a prior distribution for each parameter. The benefit of the Bayesian
approach is that it allows us to go beyond merely finding best-fitting parameter sets as
well as posterior distributions.
The hierarchical nature of our framework lets us capture individual differences
across participant, while constraining participant-level parameters via hyperpriors. The
posterior distribution of these hyperparameters can capture across-participant patterns
within the data. We favor this approach for two reasons: Firstly, the context repetition
effect exhibited considerable variation across participants, even within a given
experiment. The estimation of different parameters for each participant may allow our
model to capture these individual differences. However, each participant only
experienced 20 trials of each condition type. Thus, the amount of data per participant is
quite low, and participant-level parameter estimations may be unreliable. Our hyperpriors
serve to constrain participant-level estimations by making use of what is known about the
dataset as a whole. In other words, our hierarchical Bayesian approach offers us the
chance to look at participant-level differences while still using the entire dataset to
produce stable parameter estimates (Busemeyer & Diederich, 2010).
We chose mildly informative priors for each parameter. For parameters that exist
on the interval [0, 1], including: 𝛽, 𝛽𝑅 , 𝜌, and 𝜆, we defined the functional form of their
priors as normal distributions, which we transformed to the interval [0, 1] using an
inverse logit transform. For each of these parameters, we defined 92 participant-level
59
priors, one for each participant. The parameters for those participant-level priors
depended on mean and standard deviation hyperparameters. Mean hyperparameters were
drawn from normally distributed hyperpriors, 𝛽 𝑀 , 𝛽𝑅𝑀 , 𝜌𝑀 , and 𝜆𝑀 , each defined as a
normal distribution, 𝑁(𝜇 = 0, 𝜎 = 1). Standard deviation hyperparameters were drawn
from hyperpriors, 𝛽 𝑆 , 𝛽𝑅𝑆 , 𝜌 𝑆 , and 𝜆𝑆 , each defined as an inverse gamma distribution,
𝐼𝐺(𝛼 = 3, 𝛽 = 1).
Participant-level priors for 𝛾 were defined in exactly the same way, with one
important exception. The hyperparameters for participant-level gamma priors were drawn
from different hyperpriors, depending on experiment. Specifically, mean
hyperparameters were drawn from normally distributed hyperpriors, 𝛾 𝑀1 or 𝛾 𝑀1 , for
participants from Experiments 1 and 3, respectively. Both hyperpriors were defined as
normal distributions, 𝑁(𝜇 = 0, 𝜎 = 2). Standard deviation hyperparameters were drawn
from hyperpriors, 𝛾 𝑆1 or 𝛾 𝑆3 , for participants from Experiments 1 and 3, respectively.
These hyperpriors were defined as inverse gamma distributions, 𝐼𝐺(𝛼 = 3, 𝛽 = 1). Thus,
although the two 𝛾 hyperpriors are identically defined, they are free to change
independently as the model fits both experiments simultaneously.
Two parameters, 𝜉 and 𝜇𝑆 , were defined as truncated normal distributions on the
interval [0, ∞). Once again, we defined participant-level priors. The lower and upper
limits of each participant-level truncated normal prior were fixed at 0 and 𝑖𝑛𝑓,
respectively. The mean and standard deviations of each participant-level prior were
hyperparameters. Mean hyperparameters were drawn from normally distributed
hyperpriors, 𝜉 𝑀 and 𝜇𝑆𝑀 , each defined as a normal distribution, 𝑁(𝜇 = 0, 𝜎 = 1).
60
Standard deviation hyperparameters were drawn from hyperpriors, 𝜉 𝑆 and 𝜇𝑆𝑆 , each
defined as an inverse gamma distribution, 𝐼𝐺(𝛼 = 3, 𝛽 = 1).
Finally, the parameter 𝜈, was defined on the interval (−∞, ∞) and took the form
of a normal distribution. Again, participant-level hyperparameters were drawn from mean
and standard deviation hyperpriors, 𝜈 𝑀 and 𝜈 𝑆 , respectively. The former was defined as a
normal distribution, 𝑁(𝜇 = 0, 𝜎 = 1), while the latter was defined as an inverse gamma
distribution, 𝐼𝐺(𝛼 = 3, 𝛽 = 1).
61
Figure 4. Hierarchical structure. Each model variant was fit to all participants i from
Experiment 1 (High Predictability) and all participants j from Experiment 3 (Low
Predictability). The same hyperpriors were used across both experiments for all
parameters except 𝛾.
62
We used a custom Markov-chain Monte-Carlo (MCMC) algorithm to find best
fitting parameters and compute a joint posterior distribution. The algorithm searches the
parameter spaces of each participant-level sub-model and each hyperprior, testing 50
parameter sets at a time (i.e. 50 chains). For each participant-level sub-model, the MCMC
algorithm proposes 50 new parameter sets. The algorithm then performs a MetropolisHastings step to calculate the likelihood of each parameter set and determine which
parameter sets will be retained (i.e. added to a parameter chain). After doing this process
on each participant’s sub-model, the algorithm performs an analogous procedure on each
hyperprior. The process then begins again with the next iteration. We performed 3,200
iterations, discarding the first 1,500 as a burn-in period.
Table 2. Essential equations.
Equation
Function
Number
𝑀 = 𝑀 + 𝛼⟨𝑓1 − 𝑀𝑓0 + 𝛾𝑀𝑓1 ⟩+ 𝑡0𝑇
learning rule, Variant 1
3
𝑀 = 𝑀 + 𝛼⟨𝑓1 − 𝛾𝑀𝑓0 + 𝛾𝑀𝑓1 ⟩+ 𝑡0𝑇
learning rule, Variant 2
14
𝑀 = 𝑀 + 𝛼⟨𝑓1 − 𝛾𝑀𝑓0 + 𝛾𝑀𝑓1 ⟩𝜂 𝑡0𝑇
learning rule, Variant 3
15
𝑡 𝐼𝑁 = ∥𝛽𝑓1 + (1 − 𝛽)𝑓1 𝑀∥∥
context reinstatement
4
𝑡1 = ∥𝜌𝑡0 + 𝑡 𝐼𝑁 ∥
context evolution
5
𝑡1𝑙𝑖𝑛𝑑 = 𝜆
list context activation
6
𝑡 𝐼𝑁 = ∥𝛽𝑅 𝑝 + (1 − 𝛽)𝑝𝑀∥∥
test context reinstatement
7
𝑠 = 𝑡 𝐼𝑁 ⋅ 𝑡0 𝜉 + 𝜈
probe strength
8
𝑜 = 𝑑𝑛 (𝑠, 𝜇𝑆 , 1)/(𝑑𝑛 (𝑠, 𝜇𝑆 , 1) + 𝑑𝑛 (𝑠, 0,1))
prob. of responding “old”
10
𝑡1 = ∥𝜌𝑑𝑖𝑠𝑡 𝑡0 + 𝑓𝑑𝑖𝑠𝑡 ∥
context clearing
11
𝑙𝑙 = ∑𝑟𝑙𝑜𝑔𝑒 (𝑜) + (1 − 𝑟)𝑙𝑜𝑔𝑒 (1 − 𝑜)
log likelihood function
13
63
Results
Our model fit the behavioral data quite well. Taking the maximum a posteriori
(MAP) parameter set from each participant sub-model, we were able to produce
simulated hit rates that closely tracked mean behavioral performance across all conditions
and in both experiments (see Figure 5, top panels}. Notably, the model produced higher
hit rates for twice-presented targets than once-presented targets. False-alarm rates for
lures were also appropriately lower than hit rates for targets.
Furthermore, the model also qualitatively captured both components of the
context repetition effect for participants in high-predictability Experiment 1 (see Figure 6,
panel A). Specifically, the model produced higher hit rates for once-presented targets
whose contexts were later repeated relative to control targets (RCNT1 - NCNT1);
however, there was no difference in hit rates between once-presented targets that
followed repeated contexts and control targets (RCNT2 - NCNT2). Additionally, only a
slight advantage for RCNT1 targets over NCNT1 targets was produced in Experiment 3
(see Figure 6, panel B). We return to this point in the discussion.
Going beyond mean model performance, we next checked to see if the model
captured individual differences in the context repetition effect. For first presentations
(RCNT1 - NCNT1), the performances of sub-models appeared to track individual
variability in the context repetition effect (see Figure 7, panels A and B}). However this
correlation only reached statistical significance in the low-predictability experiment,
Pearson’s r = 0.37, n = 52, p = .007.
64
Figure 5. Model fits: Memory performance across conditions. Dark blue bars show the
mean hit rates across conditions, and false-alarm rates for lures. Light blue bars indicate
mean model behavior for each condition using MAP estimates. Panel rows correspond to
different experiments.
65
Figure 6. Model fits: Context repetition effects. Dark blue bars show the behavioral
context repetition effect. Light blue bars indicate model behavior using MAP estimates.
Within each panel, the left bars indicate the mean difference in hit rate between oncepresented targets whose contexts later repeat and control targets (RCNT1 vs. NCNT1).
The right bars indicate the mean difference in hit rate between once-presented targets that
follow repeated contexts and control targets (RCNT2 vs. NCNT2). Panel rows
correspond to different model variants. Panel columns correspond to different
experiments.
66
Figure 7. Model fits: Individual differences. Scatterplots show the relationship between
each participant’s context repetition effect and the simulated context repetition effect for
their corresponding sub-model. Best fitting regression lines and 𝑅 2 values are included.
Panel rows correspond to different model variants. Panel columns correspond to different
experiments and presentations.
67
We then simulated a variant of TCM-SR, which utilized parameter 𝛾 in a slightly
different way. Instead of making 𝛾 a coefficient of predictions of future stimuli, as
originally used (see Equation 3), we now also make 𝛾 a coefficient of predictions for the
current stimulus:
𝑀 = 𝑀 + 𝛼⟨𝑓1 − 𝛾𝑀𝑓0 + 𝛾𝑀𝑓1 ⟩+ 𝑡0𝑇 . (14)
Thus, rather than modulating the learning of predictions of future items, the 𝛾 parameter
in Variant 2 modulates the use of all prediction-based learning. Note that this alteration
does not change the number of free parameters in the model. In all other respects Variant
2 was identical to Variant 1, as was the simulation procedure.
Variant 2 performed similarly well in terms of fitting overall hit rates across
conditions (see Figure 5, middle panels). This variant did not quite capture the full
magnitude of the context repetition effect found in the high-predictability experiment (see
Figure 6, panel C). However, it still qualitatively reproduced the effect. Paralleling the
previous simulation, Variant 2 also produced a minor context repetition effect in the lowpredictability experiment (see Figure 6, panel D).
Variant 2 tracked individual differences in the context repetition effect better than
Variant 1 (see Figure 7, panels E and F}). Individual’s context repetition effects (for first
presentations) were significantly correlated with the model’s behavior both in the highpredictability, r = .48, n = 40, p = .002, and low-predictability experiments, r = .34, n =
52, p = .014.
We then performed a model comparison between Variants 1 and 2 using the
Bayesian predictive information criterion (BPIC) (Ando, 2007). BPIC assesses a model’s
68
ability to predict data using the whole posterior distribution, not just a MAP estimate.
BPIC is also ideally suited for evaluating models where the posteriors were estimated
using MCMC. For both of our model variants, we computed BPIC scores for each
participant sub-model. This allowed us to see if each variant best captured the behavior of
some of the participants.
Our results indicate that Variant 2 outperformed Variant 1 on 72 of our 92
participants (see Figure 8. This seems counterintuitive at first, given that Variant 1
seemed to reproduce the context repetition effect slightly better. However, that effect—
much like hit rates for each condition—is a set of summary statistics. Those summary
statistics partially describe the data, but are not sufficient to fully reproduce it (Turner &
Sederberg, 2013). Because of this, we fit our model variants to each participant’s triallevel data. Thus, our model comparison indicates that Variant 2 provides a somewhat
better explanation of the data overall.
69
Figure 8. Model comparison: Variants 1 and 2. The histogram shows the frequency of
within-participant differences in BPIC scores between Variants 1 and 2. Greater values
indicate a stronger preference for Variant 2, with 0 indicating no preference.
We tested one additional variant of TCM-SR, which built upon Variant 2 by
relaxing the rectification operation in the learning rule (see Equation 14). Instead of
simply setting negative features to 0, Variant 3 scaled down these negative features, so
that
𝑀 = 𝑀 + 𝛼⟨𝑓1 − 𝛾𝑀𝑓0 + 𝛾𝑀𝑓1 ⟩𝜂 𝑡0𝑇 . (15)
The 𝜂 parameter takes values on the interval [0, 1] and multiplicatively scales any
negative features before they become associated with context. Hence, the value of 𝜂
70
controls the rate of unlearning. If 𝜂 is 0, then negative features get similarly set to 0 and
the model behaves identically to Variant 2. If 𝜂 is 1, then the model will unlearn at the
same rate as traditional TD-SR models. We chose normally distributed participant-level
priors and hyperpriors that were consistent with our other parameters. Specifically,
participant-level hyperparameters were drawn from mean and standard deviation
hyperpriors, 𝜂𝑀 and 𝜂 𝑆 , respectively. The former was defined as a normal distribution,
𝑁(𝜇 = 0, 𝜎 = 1), while the latter was defined as an inverse gamma distribution, 𝐼𝐺(𝛼 =
3, 𝛽 = 1).
Qualitatively, Variant 3 also captured the mean hit rates for each target condition
and the false-alarm rate for lures (see Figure 5, lower panels). However, it only partially
reproduced the context repetition effect. In the high-predictability experiment, the
likelihood of recognizing once-presented items whose contexts were later repeated was
only slightly greater than the recognition likelihood for control items (see Figure 6, panel
E). Furthermore, a marginal context repetition effect was also found in the lowpredictability experiment (see Figure 6, panel F), consistent with previous variants.
Variant 2’s sub-models did capture some of the individual differences in the context
repetition effect for first presentations (RCNT1 - NCNT1), both in high-predictability, r =
.39, n = 40, p = .012, and low-predictability, r = .35, n = 52, p = .011, experiments (see
Figure 7, panels I and J}). A quantitative comparison between Variants 2 and 3 using
BPIC revealed Variant 2 to be the better fitting model for all 92 participants (see Figure
9). Thus, Variant 3 was qualitatively and quantitatively inferior to Variant 2, a more
parsimonious model.
71
Figure 9. Model comparison: Variants 2 and 3. The histogram shows the frequency of
within-participant differences in BPIC scores between Variants 2 and 3. Greater values
indicate a stronger preference for Variant 3, with 0 indicating no preference.
Discussion
We fit multiple variants of our TCM-SR model to the experimental data presented
in the previous chapter (Smith et al., 2013). All variants were able to reproduce hit rates
for twice presented stimuli and once-presented control stimuli, as well as false alarm rates
for lures. Importantly, two of these model variants were also able to reproduce the
context repetition effect: elevated recognition likelihoods for once-presented items whose
context is later repeated, but not for the similarly once-presented items that followed
72
repetitions of context. Our computational model is consistent with the verbal theory
proposed previously (T. A. Smith et al., 2013), which argued for prediction-based
learning. However, in the process of formally implementing this framework, we made
critical changes to the learning rule and added the necessary storage, retrieval, and
decision mechanisms in order to adequately simulate memory performance.
The importance of prediction in accounting for the context repetition effect is
plainly evident based on the current simulations. Without predictions of the future, the
model would not be able to produce elevated recognition probabilities for once-presented
targets whose contexts later repeat (RCNT1). The prediction coefficient parameter, 𝛾, is
critical to enabling and regulating this behavior. Without the ability to tune this value, the
model would not be able to capture the null context repetition effect of Experiment 3.
Other Models and the Context Repetition Effect
We know of no existing episodic memory model that, in its current form, can
predict the context repetition effect. Formal models of memory rely on a variety of
encoding mechanisms, including—but not limited to—rehearsal buffers (Gillund &
Shiffrin, 1984; Raaijmakers & Shiffrin, 1981), convolution (Murdock, 1993), exemplar
encoding (Hintzman, 1988) and Hebbian learning (Howard & Kahana, 2002). None of
these learning mechanisms involve learning predictions of the future. Without this
component, current models of memory are incapable of explaining the current findings.
That is not to say that these established models could not be adapted, as we have
adapted TCM, to accommodate our new data. Most episodic memory models could be
modified to learn based on prediction. All that is required is that a given model query the
73
contents of its long-term memory storage with a cue—the currently perceived stimulus—
and then learn the output based on whatever learning rule the model uses. As an example,
consider the search for associative memory model (SAM) (Gillund & Shiffrin, 1984),
which learns associations via rehearsal of items in a short-term buffer. If SAM were
modified such that it 1) generated free-recall outputs during encoding and 2) put those
outputs into the buffer for rehearsal, then the model would begin associating recent items
with items predicted by current cues. To put it in Marr’s (Marr, 1982) terminology, TDSR is not the only algorithm that can solve the problem posed by the context repetition
effect; however, it does characterize the computational steps required for a solution.
Another potential way of accounting for the context repetition effect may come
from the study of semantic memory. The predictive temporal context model (pTCM)
(Shankar et al., 2009) updates semantic representations based, in part, on predictions
generated from current representations. Nevertheless, pTCM is a model of semantic
memory and uses predictions to learn semantic relationships between words, which is
quite different from the goal of recognizing specific episodes. Given that semantic
learning generally requires many encounters with a concept, pTCM as currently
implemented is probably ill-equipped to rapidly learn episodes of experience.
Consequently, pTCM would have to undergo modification before it could function as an
episodic store at all, let alone account for the context repetition effect.
Differences from TD-SR
The learning mechanism in TCM-SR does have subtle, yet important, differences
from the typical TD-SR learning rule. Firstly, our second variant uses the prediction
74
parameter 𝛾 as a coefficient for both the prediction of future and current items. TD-SR,
along with our first variant, places 𝛾 only next to the prediction of future states. This
difference changes the interpretation of 𝛾 and has important implications for the model.
For our second variant, 𝛾 is regulating the learning of any prediction. If 𝛾 is set to zero,
then the model simplifies to Hebbian learning (Hebb, 1949). In our first variant, as well
as the standard TD-SR rule, 𝛾 simply weights how much predictions of the future are
learned. When 𝛾 is set to zero, the learning rule reduces to error-driven learning (Rescorla
& Wagner, 1972). Although Variant 2 outperformed Variant 1 overall, both variants
provided reasonably good fits to the data with 𝛾 set above zero.
Additionally, TCM-SR rectifies the vector of features that gets bound to context
(see Equation 3). Any features with negative values are set to zero; thus no unlearning
occurs in the model. This stands in contrast to conventional TD-SR which unlearns
suboptimal transitions as it gains experience with the environment (Dayan, 1993).
Rectification is a logical adjustment given that the presumed goal of episodic memory is
to retain information. Nevertheless, it could be the case that some degree of unlearning is
tolerable, or even desirable, in TCM-SR. However, our third variant replaced the
rectifying function with a negative-scaling parameter. This parameter scaled down the
magnitude of negative feature values instead of setting them to zero. Interestingly, this
added flexibility did not improve performance and in fact led to a worse fitting model. In
the next section we highlight several other modifications that could be incorporated into
TCM-SR.
75
Limitations and Future Work
Although the present incarnation of TCM-SR qualitatively reproduces the context
repetition effect (Smith et al., 2013), the model does not perfectly capture the data. In
particular, all variants produced a slight context repetition effect in the low-predictability
experiment, which is not present in the behavioral data. Furthermore, sub-models do not
perfectly correlate with individual differences in the effect. One especially noticeable
shortcoming is the fact that TCM-SR does not produce appreciable negative context
repetition effects, despite the fact that some participants exhibit this behavior (see Figure
7). Variants 1 and 2 lack a mechanism (i.e. unlearning) to generate such an effect. In
principle, Variant 3 could unlearn via 𝜂, however this behavior is not produced by the
best fitting sub-models. Below we discuss several potential additions or modifications to
TCM-SR that could improve its explanatory power.
The use of a constant list context unit appears to be an odd choice given that that
basis of our model is a continually evolving context vector, based on TCM (Howard &
Kahana, 2002). However, recent work in neurophysiology and cognitive science
(Shankar & Howard, 2012) suggests that context does not decay at a uniform rate (i.e. in
accordance with drift parameter 𝜌) as suggested by TCM. Rather, different elements of
context are proposed to decay nonlinearly. Briefly, newly activated context elements are
thought to decay rapidly at first; however, their rate of decay diminishes as time passes.
This results in less recent information being represented in context at low but relatively
stable levels of activation. In this framework, effectively recognizing probes would hinge
upon the strength of associations between item features and these relatively stable context
76
elements. We made use of a single list-context element as a computationally simple first
approximation of this more elaborate framework. Future work could replace the list
context unit in TCM-SR with a more advanced implementation of context.
One could frame the context repetition effect in terms of enhanced internal and
external attention (Chun et al., 2011). Encoding into memory is a function of available
attention (Chun & Turk-Browne, 2007). Hence, to the extent that participants have been
paying greater attention to the stimuli during a particular RCNT1 sequence, they should
have formed stronger associations between the context and target stimuli. This would
enable their memory system to make stronger predictions during the RCNT2 sequence.
Future research could treat attention as an individual difference, inferring level of
attention from behavioral (e.g. response latencies during an encoding task, where shorter
latencies correspond to greater attention) or physiological (e.g. pupillometry or eye
tracking) measures. Participants’ attention at encoding should correlate positively with
their memory performance across conditions at test. If the measure of attention were
more fine-grained, such that list- or even item-level indices of attention could be reliably
measured, then those indices should similarly correlate with memory for said lists or
items. The learning rate, 𝛼, could conceivably be scaled by the attention index. This
would allow the model to more accurately predict which stimuli get strongly encoded and
control for variance in memory that results from fluctuations in attention.
Internal attention—attention to internally generated information (Chun et al.,
2011), such as predictions—may help account for the context repetition effect. Under this
interpretation, the memory system is always making predictions if possible. When the
77
environment is less predictable (i.e. Experiment 3) then these predictions receive less
attention. In that sense the prediction coefficient 𝛾 could be thought of as representing the
degree of internal attention directed at predictions. If an appropriate index of internal
attention (e.g. a neural or other physiological measure) could be attained, then prediction
coefficient 𝛾 could be modulated by this index. This could potentially explain why oncepresented items that follow a repetition of context in Experiment 1 but not 3 are better
remembered without requiring separate 𝛾 hyperpriors for each experiment. Of course,
this approach would not explain how attention changes across trials or between
experiments, but would merely account for those changes.
One way of mechanistically accounting for trial-level changes to 𝛾 would be to
add a mechanism to the model that tracks the accuracy of the model’s predictions over
time. A recency-weighted running average of prediction accuracy could be used to scale
𝛾 (or potentially even replace 𝛾). In high-predictability environments, the model would
find that the stimuli it predicts have a relatively high probability of actually occurring.
Consequently, 𝛾 would be scaled up and predictions would be strongly encoding into
memory. However when entering an unpredictable environment, predictions will no
longer be borne out. Hence, the running average of prediction accuracy will diminish and
the value of 𝛾 along with it. As a result, predictions will not be strongly encoded into
memory. This type of model would not only readily fit both high- and low-predictability
environments without separate parameters, but would also make predictions about the
memory performance for stimuli that occur during transitions from predictable to
78
unpredictable environments or vice versa. Future empirical work will need to modify our
behavioral paradigm (Smith et al., 2013) in order to test these predictions.
In some cases, modeling the episodic memory system may not be enough to
capture human behavior. Semantic memory is known to influence episodic memory in
many ways, including the order in which recalls are made during free recall (Bousfield,
1953; Bousfield & Sedgewick, 1944), and even whether or not participants will recall
items that were not studied (Deese, 1959; Roediger & McDermott, 1995). Previous
computational models have incorporated semantic memory strengths to better account for
episodic memory performance for word lists (Kimball, Smith, & Kahana, 2007; Polyn et
al., 2009). Similar mechanisms could be incorporated into a future version of our model.
As such mechanisms are most useful when modeling memory for words, they do not
have much relevance for the present simulations.
Finally, future iterations of this model should strive for greater biological
plausibility. One aspect of the current model that is lacking in biological realism is our
representation of the stimuli. The item layer, 𝑓, uses orthogonal localist representations
for each item. That is, each stimulus is represented by a unit vector with a single non-zero
element. This is a reasonable simplification of representation in the hippocampus, which
likely encodes episodes using mostly non-overlapping activation patterns. However,
other brain regions, such as those that actually represent visual stimuli, use overlapping
distributed representations (Norman & O’Reilly, 2003). To completely understand how
the transient perceptions of current experience lead to enduring and accessible memory
traces, we will need to understand how and where the perceptual system represents
79
stimuli and how those representations interact with the memory system. In the next
chapter, we take a step towards this goal.
Conclusions
Our previous research demonstrated that repeating a sequence of events can
improve memory for information that followed the original presentation of that sequence
(Smith et al., 2013). Importantly, this context repetition effect may only occur in
environments featuring sequential regularities that enable accurate forecasting. Given that
real world experience is full of such regularities, the mechanism that gives rise to this
effect may be employed with considerable frequency. In fact, such a mechanism may be a
fundamental component of the memory-prediction system. We have already argued that
prediction-based learning can account for the context repetition effect. Our formal
account of memory, featuring prediction-based learning, qualitatively captures the effect.
We believe this model provides a principled starting point for further explorations into
the interaction between memory and prediction.
80
Chapter 4: Face-specific Information in the Fusiform Face Area Depends on Memory4
As we mentioned in the previous chapter, our computational model of memory
makes no effort to account for how the perceptual system represents stimuli. We view
this omission as a serious limitation given that these perceptual representations ultimately
provide input to the medial temporal lobe memory system (MTL) (Davachi, 2006;
Eichenbaum, 2000, 2004). In fact, some researchers have argued that these two systems
are not distinct. Instead, portions of the MTL can be considered part of the ventral visual
pathway (Baxter, 2009; Bussey & Saksida, 2007), which is critical for the perception of
objects of all kinds (Martin, 2007). Different portions of the ventral pathway are sensitive
to particular categories of stimuli, such as scenes, words, limbs, and faces (Martin, 2007).
However, what these category-selective regions actually represent is a matter of debate.
In this chapter we focus on representations in face-selective regions and their relationship
with the memory system.
How the human brain represents face information is a core question in cognitive
neuroscience, with implications ranging from machine vision (Tistarelli, Bicego, &
Grosso, 2009) to social biases (Van Bavel, Packer, & Cunningham, 2008). Converging
evidence from human neuroimaging (Collins & Olson, 2014; Haxby, Hoffman, &
4
This chapter presents research that is discussed in a manuscript currently being revised for a resubmission
to NeuroImage. The authors are Hasinski and Sederberg.
81
Gobbini, 2000; Kanwisher & Yovel, 2006; Kanwisher et al., 1997) and cellular
recordings of nonhuman primates (Freiwald, Tsao, & Livingstone, 2009; Tsao, Freiwald,
Tootell, & Livingstone, 2006) supports the existence of specialized regions in the primate
brain that preferentially process faces. It has been proposed that these regions form a
face-processing network, with different regions assumed to play unique but potentially
overlapping roles in the processing of face stimuli (Collins & Olson, 2014; Haxby et al.,
2000; Nestor, Plaut, & Behrmann, 2011). However, what exactly each region
represents—and how they come to represent it—is not yet understood.
Several key face-processing regions lie in the ventral visual processing pathway.
The most studied of these, the fusiform face area (FFA) is a portion of the fusiform gyrus
that responds preferentially to face stimuli relative to many other categories, such as
scenes or objects (Kanwisher & Yovel, 2006; Kanwisher et al., 1997). FFA responds to
constituent face features, but it is also sensitive to complete face configurations,
suggesting it may contain holistic representations of faces (Harris & Aguirre, 2010; Liu,
Harris, & Kanwisher, 2009). In contrast, the more posterior occipital face area (OFA)
does not appear sensitive to intact face configurations, suggesting it represents only the
components of faces (Haxby et al., 2000; Liu et al., 2009; Pitcher, Walsh, & Duchaine,
2011; Pitcher, Walsh, Yovel, & Duchaine, 2007). Additionally, the anterior inferior
temporal region (aIT)5 is strongly implicated in both the perception and memory of faces
(Collins & Olson, 2014). Indeed, lesions of aIT produce person-memory deficits
5
We refrain from describing aIT as a “face area” per se because we reserve that phrase for regions that are
typically functionally defined as more responsive to faces than other stimulus categories. Due known issues
in imaging the anterior temporal lobe with fMRI (Devlin et al., 2000; Visser, Jefferies, & Lambon Ralph,
2009), functional localization of a face-sensitive ROI in anterior inferior temporal lobe is less common.
82
(Gainotti & Marra, 2011; Olson, McCoy, Klobusicky, & Ross, 2013; Ross & Olson,
2010) as well as deficits in face discrimination (Busigny et al., 2014; Olson, Ezzyat,
Plotzker, & Chatterjee, 2014). In contrast, the evidence for FFA’s involvement in facememory is mixed, and OFA is believed to be uninvolved (Collins & Olson, 2014).
The precise function of these regions is still an open question. For instance, the
FFA has been viewed as performing face detection (Kanwisher, Tong, & Nakayama,
1998), face identification (Gauthier et al., 2000), or both (Grill-Spector, Knouf, &
Kanwisher, 2004). Taking advantage of advancements in multivariate analysis of fMRI,
recent research has shown that face-sensitive regions contain information that
differentiates individuals (Anzellotti & Caramazza, 2014). An early study presented one
male and one female to participants many times during scanning. Multivariate analyses
revealed that information in the aIT, but not the FFA, was distinct for each face
(Kriegeskorte, Formisano, Sorger, & Goebel, 2007). Although this result could reflect the
presence of gender information, more recent work also found information in faceselective voxels in the FFA and anterior temporal lobe that individuated different faces of
the same gender (Nestor et al., 2011). More recently, the OFA, FFA, and aIT were shown
to contain information identifying individuals even when some aspect of individuals’
faces changed across presentations. For instance, a pattern classifier trained on faces from
several viewpoints was used to effectively discriminate between those faces when tested
on novel viewpoints (Anzellotti, Fairhall, & Caramazza, 2013). Other researchers
extracted individuating information from faces that persisted across changes in facial
expression (Nestor et al., 2011). Indeed, converging neuroimaging and lesion data
83
suggest that the integration of information across these face areas may be essential for
face identification in humans (Anzellotti & Caramazza, 2014; Collins & Olson, 2014;
Haxby et al., 2001; Natu et al., 2009).
One commonality across these studies is that they all present the same faces
multiple times during scanner sessions, usually with additional presentations during a
pre-scanning training phase. Consequently, participants inevitably become familiar with
the stimuli used. Often, this is training or familiarization process is an intentional, and
perhaps critical, element of the experiment. Because familiarization is a constant feature
of these studies, it leads to a question about when and how this differentiating
information arises in face regions. One possibility is that these regions contain
individuating information regardless of familiarity. Face regions may give rise to
information for individual faces (i.e. face-specific information) through the bottom-up
processing of face stimuli and the representation of each face’s unique constellation of
features. Under this hypothesis, patterns in face-sensitive regions should individuate faces
without any prior exposure. Consequently, face-specific information should be present in
both OFA and FFA, which are believed to represent face components (Harris & Aguirre,
2010; Liu et al., 2009). OFA, which appears to more specifically represent lower-level
features (Haxby et al., 2000; Liu et al., 2009; Pitcher et al., 2011; Pitcher et al., 2007),
may show greater face-specific information than FFA. Finally, face-specific information
may or may not be present in aIT, which is typically associated with more abstracted or
view-invariant representations of identities (Anzellotti et al., 2013; Collins & Olson,
2014; Freiwald & Tsao, 2010).
84
An alternate hypothesis is that familiarization of a face is what leads to this
individuating information. According to this latter hypothesis, face regions should only
individuate faces that are familiar to participants. Additionally, the face regions more
strongly associated with face or person memory, FFA (Grill-Spector et al., 2004) and
especially aIT (Collins & Olson, 2014; Gainotti & Marra, 2011; Olson et al., 2013),
should contain this face-specific information, whereas OFA may not contain information
for individual faces.
The current study directly compares these hypotheses. To do this, we forego
extensive training and repeated presentations of faces during scanning. Instead, we use a
single repetition of target faces during scanning, and then test memory for those faces
after scanning. We operationalize familiarity as subsequent memory for these target
faces. That is, remembered faces will retrospectively indicate whether faces were familiar
to a participant at the initial repetition and forgotten faces will have remained unfamiliar
during study. According to the familiarity hypothesis, only faces that are subsequently
remembered should show significant pattern similarity between presentations.
Furthermore, we would expect this similarity to be present in the more downstream
regions: FFA and especially aIT. However, under the alternate hypothesis, faces should
show significant pattern similarity between presentations regardless of subsequent
memory performance, and this similarity should be stronger in more upstream regions:
FFA and especially OFA.
Potential support for the familiarity hypothesis comes from recent research that
found greater pattern similarity between repetitions of words that were subsequently
85
remembered relative to the similarity observed between repetitions of forgotten words
(Xue et al., 2010)6. The researchers further argued that their finding provided evidence
against the encoding variability hypothesis, which states that repeatedly experienced
items are better remembered if the repetitions occur in varied contexts (i.e., with different
preceding items), as opposed to all repetitions occurring in the same context (Bray,
Robbins, & Witcher, 1976; Martin, 1968; Melton, 1970). Although they did not explicitly
manipulate the context in which spaced items were presented, Xue and colleagues
equated greater pattern similarity across repetitions with decreased encoding variability.
As such, when they observed that items with lower pattern similarity exhibited worse
subsequent memory performance, they concluded this contradicted the encoding
variability hypothesis.
Given that the context across spaced repetitions was never the same in the Xue et
al. study, the relationship between neural pattern similarity, memory, and context has yet
to be fully explored. To that end, we added a secondary question to the current
experiment: Does degree of encoding variability influence subsequent memory via
altering the degree of similarity between presentations? We manipulated encoding
variability by controlling the temporal contexts (Howard & Kahana, 2002; Sederberg et
al., 2008; Turk-Browne et al., 2012) in which faces appeared across repetitions. This
manipulation allowed us to test directly whether encoding variability in the traditional
sense is related to subsequent memory. If we found a significant relationship, we would
6
In fact, Xue et al. (2010) did show greater pattern similarity for remembered faces in a separate
experiment. However, the experiment using face stimuli did not feature trial-level analyses, so they could
not assess whether or not representations for individual faces were more similar across repetitions when
subsequently remembered.
86
then test whether the effect is mediated by the degree of pattern similarity in facesensitive regions. However, if we found context to be unrelated to memory behaviorally,
then we would then drop the mediation analysis and focus instead on our primary
question: What is the relationship between subsequent memory and face-specific
information, irrespective of context?
Method
Subjects
Twenty-seven Ohio State community members (11 female, all right-handed, mean
age 22.3 years) participated in the main experiment and a functional localizer task. All
subjects had normal or corrected-to-normal visual acuity, provided informed consent, and
received monetary compensation. The study protocol was approved by the Institutional
Review Board for Human Subjects at the Ohio State University.
Stimuli
Face stimuli consisted of color photographs of nonfamous male and female faces,
facing forward, from the shoulders up, with neutral expression and various hair styles
(See Figure 1). All faces appeared before a white background. A separate set of faces–
along with sets of scenes, objects, and scrambled objects–were also used in the functional
localizer task. Using separate pools of faces for each task prevented any additional
exposures to our target stimuli before the surprise memory test (see below). A Christie
digital projector displayed images at 60 Hz with a resolution of 1280 x 1024 onto a
screen behind the scanner bore. Participants viewed images with a mirror attached to the
head coil.
87
Procedure
Anatomical Scan. After obtaining informed consent and providing instructions,
we began scanning with a high-resolution T1-weighted anatomical scan, lasting 3
minutes and 25 s, was collected from each subject.
Functional Localizer. Participants then completed a functional localizer task
based on previous work (Epstein & Kanwisher, 1998). This task, which consisted of three
4 minute and 24 s runs, enabled us to functionally define ROIs, independently of our
main experiment. Consistent with similar tasks used by others (Epstein & Kanwisher,
1998), each localizer run alternated between blocks of scenes, faces, objects and
scrambled objects. Each block contained 16 stimuli, all from the same category. Each
stimulus was presented for 500 ms, followed by an ISI of 500 ms. Blocks were arranged
in a pseudorandom fashion. A block of each category would appear once, in random
order, followed by 12 s of fixation. This occurred 3 times per run, with the category
blocks arranged in a different order each time. To keep participants engaged during the
localizer, they were instructed to detect immediate repetitions of stimuli. In each block,
two stimuli were randomly chosen to be repeated immediately. Participants indicated
when they detected an immediate repetition via button press. All responses in the scanner
were made using a Current Designs fiber-optic response pad.
Main Task. The main scanner task consisted of four runs, each lasting 8 minutes
and 48 s. Each run consisted of 96 faces, presented one at a time for 1.2 s, with a
randomly jittered inter-stimulus interval (ISI) between 2.3 s and 5.3 s (see Figure 10A).
88
To keep participants engaged in the task, they were instructed to indicate whether each
face was male or female. Participants indicated their response via the response pad.
The main task featured a structure identical to the paradigm discussed in previous
chapters (Smith et al., 2013), although it used different stimuli and timing sequences.
Because the target stimuli in this experiment are different from our behavioral research,
we will explain the design differently. Each target face appeared twice, with between 5
and 20 other faces occurring in between the first and second presentations (see Figure
10B). Consistent with previous research (Smith et al., 2013; Turk-Browne et al., 2012)
and theoretical work on temporal context (Howard & Kahana, 2002; Sederberg et al.,
2008), we denoted the two faces that preceded each target as the “context” in which each
target presentation occurred. Faces making up the high encoding variability (high-EV)
condition were preceded by different context faces on their first and second presentations.
That is, each face occurred in a unique temporal context on both presentations. In
contrast, faces making up the low encoding variability (low-EV) condition were preceded
by the same context faces on their first and second presentations. However, different lowEV face pairs appeared in different contexts.
Across all runs, each participant viewed 16 targets in the high-EV condition and
16 targets in the low-EV condition, with each target being presented twice. Importantly,
these stimuli were completely novel to the participants. They had no training or previous
exposure to them prior to the study and only minimal familiarization–two presentations–
during the scanner task.
89
Participants were not informed of the organization of the stimulus lists, nor were
they informed that some faces were “target” stimuli. They were simply told that they
would see a sequence of faces, that some of them may be repeated, and that they should
indicate the gender of each face as quickly and accurately as possible. Additional filler
faces were interspersed throughout each run, consistent with similar paradigms used
previously (Smith et al., 2013). These faces were either presented once or twice to
obscure the organizational structure of the target stimuli, but they had no bearing on the
hypotheses in question here.
90
Figure 10. Overview of experimental design. (A) Faces were presented one at a time, for
1.2 seconds, with a randomly varied interstimulus interval. Each target face occurred
twice. (B) Each presentation of a given target face occurred in either the same context
(low encoding variability) or in different contexts (high encoding variability). Same-face
similarity was computed between a first-presentation target (red square) and its identical
second presentation (dark-blue square). Different-face similarities were computed
between the first-presentations and non-identical second presentations with matching
genders and encoding variability (light-blue square).
Recognition Memory Task. Following the last structural scan, participants exited
the scanner and completed a recognition memory task on a laptop in an adjacent testing
room. They were shown all of the target and comparison faces for all four conditions (96)
and 20% of the context faces (38), along with an equal number (134) of lures and asked
91
whether or not each face appeared in the study task. Again, faces were presented one by
one for 1.2 s and subjects had the same amount of time to make a decision. Subjects were
given four response options: “sure old,” “old,” “new,” or “sure new”, with each
corresponding to a different keyboard key. Participants indicated their decisions with a
single keypress.
MRI Data Acquisition
MRI Data were acquired with a 3T Siemens Trio scanner and a 12-channel matrix
head coil at the Center for Cognitive and Behavioral Brain Imaging (CCBBI) at the Ohio
State University. For the main encoding task, functional images were obtained with a T2weighted EPI sequence: TR = 2200 ms, TE = 26 ms, field of view = 250 x 220 mm [read
x phase], flip angle = 75°, thickness = 2.7 mm (2.5 x 2.5 x 2.7 mm voxels). 41 oblique
axial slices were collected in interleaved order. 237 volumes were acquired.
For the functional localizer scans, T2-weighted images were obtained with the
following parameters: TR = 3000 ms, TE = 28 ms, field of view = 250 x 220 mm [read x
phase]; flip angle = 80°; thickness = 2.5 mm (2.5 x 2.5 x 2.5 mm voxels). 47 oblique
axial slices were collected in interleaved order. 85 volumes were acquired.
For each subject, we used the high-resolution T1-weighted anatomical scan to
align scans align all functional scans. 160 sagittal slices were collected, with a thickness
of 1 mm (1 x 1 x 1 mm voxels).
Data Processing
Preprocessing. Processing of both functional and anatomical MRI images was
carried out with a combination of AFNI (Cox, 1996) and FSL (Jenkinson, Beckmann,
92
Behrens, Woolrich, & Smith, 2012; Smith et al., 2004). FSL’s brain extraction tool
(Jenkinson, Pechaud, & Smith, 2005) was applied to the T1-weighted structural images to
isolate voxels containing brain tissue.
For functional scans, we began by discarding the first two volumes of each
functional run. Functional runs were preprocessed and analyzed using AFNI (Cox, 1996).
Processing then followed standard AFNI protocols (Cox, 2012) and included despiking
the time series, aligning the runs together and correcting for head motion. The functional
localizer data were then smoothed using a 4 mm FWHM Gaussian kernel. Because of our
primary analyses involved patterns of activation, we did not smooth the main task data.
Finally, all functional data were scaled to have a mean of 100 as part of AFNI’s standard
data analysis (Cox, 2012).
Modeling the BOLD Response. For both functional tasks, we modeled the neural
response to each stimulus by fitting canonical hemodynamic response functions (HRFs),
convolved with a square function equal to the stimulus duration, in a general linear model
(3dDeconvolve in AFNI) (Ward, 2002) for each participant. Each HRF was estimated by
fitting its amplitude (i.e. one regressor per HRF) to the data. The result of normalizing
our data to have a mean of 100 is that the regression coefficients are equivalent to
estimates of percent signal change (Ward, 2002).
For the functional localizer, separate canonical HRF regressors were entered for
every stimulus category (face, scene, object and scrambled object), convolved to a .5 s
square function, representing stimulus duration. The resulting statistical maps were
similar if we used a single 16 s regressor for each category block.
93
For the main task, all hemodynamic responses were convolved to a 1.2 s square
function, representing stimulus duration. We modeled the hemodynamic response to each
target face presentation with its own regressor, using the least-squares all method
(Rissman, Gazzaley, & D’Esposito, 2004). This produced 64 single-trial regressors (2
presentations for each of 16 targets in both the high-EV and low-EV conditions).
Extracting estimated BOLD response for these individual trials allowed us to find
patterns of activation associated with each individual presentation.
Conventional condition regressors were used to model all non-target stimuli. We
estimated a single hemodynamic response for each type of context and filler face. This
resulted in 20 additional regressors (16 for context faces and 4 for filler faces). This was
done to improve the stability of the estimates of the 64 trials of interest. We also included
6 motion parameters, which were estimated during preprocessing. Finally, we included 4
polynomial regressors to remove slow changes in the BOLD signal (i.e. due to drift in the
magnetic field). This procedure resulted in 94 total regressors for the main encoding task.
Note that even though our task is a rapid-presentation, event-related design, the
spacing between our target stimuli is considerable (mean: 76.6 s, SD: 21.7 s). Thus, we
effectively have a slow-presentation, event-related design where our regressors of interest
are concerned, with BOLD responses due to non-target stimuli accounted for with stable
condition regressors.
Participants’ resulting coefficient maps were transformed to their own highresolution T1-weighted anatomicals. Registration between functional and anatomical
maps was performed using FSL’s linear volumetric registration tool (Jenkinson & Smith,
94
2001; Jenkinson, Bannister, Brady, & Smith, 2002). Coefficient maps in anatomical
space were used for all ROI analyses.
Regions of Interest. We used the functional localizer to define regions of interest
(ROIs) for our main analysis. Consistent with previous research (Anzellotti et al., 2013),
we used a faces - scenes contrast to obtain a statistical map of face selectivity. Left and
right fusiform face areas (FFAs) were identified separately for each subject. Contrasts
were clusterized, using 20 voxels as the minimum cluster size. This ensured that all ROIs
had at least 20 voxels in functional space, providing ROIs of sufficient size to extract
reliable signal from our task data.
The process for selecting ROIs involved three steps: First, for each hemisphere,
we focused on a broad anatomical region that coincides with established FFA location
(e.g. the fusiform gyrus). Then we set an initial threshold equivalent to p = .0001. If no
clusters survived this initial threshold we did not define an ROI for that subject and
region. Finally, if a cluster was found, we decreased the p-value until the cluster shrank to
20 voxels, or as close to 20 as possible without breaking up. All ROIs were between 20
and 40 2.5x2.5x2.7mm voxels in size.
ROI were then transformed into each subject’s high-resolution anatomical space.
We then created a mask out of the 200 most significant 1x1x1mm voxels in anatomical
space. This allowed us to reliably identify anatomical landmarks and ensured that ROIs
would align with our task data, which were also transformed to anatomical space. Finally,
we combined–when possible–left and right hemispheres into a single mask for each ROI.
95
This was done because we did not have specific hypotheses regarding hemisphere and we
wanted to keep the number of statistical comparisons to a minimum.
This process enabled us to create OFA and FFA cluster in 26 of 27 participants.
We were also able to create a parahippocampal place area (PPA) (Epstein & Kanwisher,
1998) ROI in 26 of 27 participants by following the same process but reversing the
statistical contrast to be scenes > faces. We use PPA as a control region in the following
analyses to ensure we don’t get face-specific activation in regions not known to be
sensitive to faces.
We were unable to functionally localize an anterior temporal face area in more
than a small fraction of our participants. This was likely due to signal loss in the ventral
anterior temporal lobe, a known issue (Devlin et al., 2000; Visser et al., 2009). As an
alternative, we created an anatomically defined 400-voxel (1 mm^3 voxels) anterior
inferior temporal lobe ROI (aIT) masks for each participant, using the Harvard-Oxford
atlas distributed by FSL (Mark Jenkinson et al., 2012; Smith et al., 2004) as a reference.
We treat this analogue as a substitute for a functionally defined aIT.
The use of participant-specific ROIs obviated the need for transforming any data
to a standard, across-participant space. Therefore, we perform this second transformation
on our ROI masks only for the purpose of aggregating ROIs across participants for
graphical purposes (see Figure 11). This anatomical-to-standard space transformation was
accomplished with FSL nonlinear volumetric registration tool (Andersson, Jenkinson, &
Smith, 2007). ROIs are aligned to the Montreal Neurological Institute (MNI-152) atlas
provided by FSL (Jenkinson et al., 2012; Smith et al., 2004).
96
Figure 11. Functional and anatomical ROIs. Superposition of every subject’s FFA (red),
OFA (green) and PPA (blue), along with aIT (violet) in MNI standard space.
97
Calculating Face-specific Information
To investigate the presence of face-specific information, we used each
participant’s masks to extract a pattern of neural activity at each ROI for each target
presentation. We then used a trial-level variant of correlation analysis (Haxby et al.,
2001; Xue et al., 2010). Specifically, for each subject we correlated activity for each firstpresentation target with activity for every second-presentation target. This produced two
types of correlations: “same-face” correlations between each first presentation and the
identical second presentation, and “different-face” correlations between each first
presentation and all non-identical second presentations. We then applied a Fisher ztransformation to all correlations, converting them to similarity values on the interval
(−∞, ∞).
The same-face similarities reflect the dependency or overlapping information
between the neural representations of first and second presentations of identical faces.
The different-face similarities are important because collectively they quantify the
amount of overlapping information between neural representations of first-presentation
faces to other non-identical faces on average. In other words, different-face similarities
let us estimate the amount of overlapping information that is due to the stimulus category
of faces. We use these different-face similarities to estimate the amount of information in
the same-face similarity values that can be attributed to representing the same exact face
rather than the same stimulus category. Specifically, for each first presentation, we
98
created a vector of face-specific similarity values: the same-face similarity minus each
different-face similarity:
𝐟𝐢 = [𝑠𝑖 − 𝑑𝑖1 , 𝑠𝑖 − 𝑑𝑖2 , . . . , 𝑠𝑖 − 𝑑𝑖𝑛 ], (16)
where 𝑠𝑖 is the scalar same-face similarity between 𝑖 𝑡ℎ first presentation and its identical
second presentation, din is the scalar different-face similarity between the 𝑖 𝑡ℎ first
presentation 𝑛𝑡ℎ different-face second presentation, and 𝐟𝐢 is the resulting vector of facespecific similarity values.
The different-face similarity values are meant to control for any pattern similarity
that is not due to specific faces. Therefore, we avoid using different-face similarities that
may be low due to systematic differences between first and second presentations. Past
research has shown that full multivariate techniques can be used to distinguish
representations of male and female faces (Kriegeskorte et al., 2007). In searching for
face-specific information in face-sensitive regions, we did not want our results to be due
to gender differences between faces. Similarly, we only use different-face similarity
values when the same-face and different-face targets have the same gender. Additionally,
we only use different-face similarity values when the different-face target occur in the
same encoding variability condition as the same-face target. Finally, we only use
different-face similarity values when the subsequent memory performances of the sameface targets and different-face targets match.
The eligible different-face similarities were used to create face-specific
similarities according to Equation 16. We use these resulting values as the dependent
variable in our neural analyses.
99
Mixed-effects Regression
To test the competing hypotheses that face-specific information is either
dependent on subsequent memory or not, we fit our data with a mixed-effects regression
model. We performed a separate analysis for each of our four ROIs. Our predictor
variable of interest is an indicator variable for whether or not each target face was
recognized during the memory test. Additionally, we wanted to rule out the possibility
that any effect, or lack of effect, could actually be due to the spacing between faces or
overall activation within ROIs. We included two nuisance regressors to control for these
issues.
First, we needed to control for the variation in lag between the first and second
presentations. Autocorrelation is a known feature of fMRI time-series data. In general,
when two presentations are closer to one another in time, the neural patterns they evoke
maybe more similar regardless of the actual evoked neural activity. Thus, the lags
between the first and second presentations are a potential source of variance we must
account for. More specifically, because our face-specific similarity values are the
difference between two values–same-face similarity and different-face similarity–we
needed to account for the relationship between same-face lags and different-face lags.
To accomplish this we created a lag-log-ratio term, which we will explain further:
We began by calculating the same-face lags, the spacing between first presentations and
their identical second presentations, and the different-face lags, the spacing between first
presentations and non-identical second presentations. We then log-transformed both types
of lags. Finally we subtracted the different-face log-lags from the same -face log-lags:
100
𝐥𝐫𝐢
𝑙𝑖𝑠
𝑙𝑖𝑠
𝑙𝑖𝑠
= [log 𝑒 ( 𝑑 ), log 𝑒 ( 𝑑 ), . . . , log 𝑒 ( 𝑑 )] , (17)
𝑙𝑖1
𝑙𝑖2
𝑙𝑖𝑛
where 𝑙𝑖𝑠 is the lag between the first and second presentations of the 𝑖 𝑡ℎ face, ldin is the
scalar lag between the 𝑖 𝑡ℎ first presentation and and the 𝑛𝑡ℎ different-face second
presentation, and 𝐥𝐫𝐢 is the resulting vector of lag-log-ratios. Note that these values are the
log of the ratio between same-face lags over different-face lags.
The resulting lag-log-ratios have three useful properties. Firstly, they combine
both same-face and different-face lags into a single value, keeping the complexity of our
inferential models to a minimum. Secondly the ratio preserves the relative difference in
spacing between same-face pairs and different-face pairs. Finally, the log transformation
provides an intuitive and interpretable zero point. That is, when the same-face lag equals
the different-face lag, the resulting lag-log-ratio value is zero. Thus, this lag-ratio variable
quantifies uneven lag between same- and different-face similarity values. As an
additional safeguard against autocorrelation, we discarded pairings where the lag between
the first and second presentation was less than 30 seconds.
We also wanted to ensure that any face-specific pattern similarity was not due to,
or obscured by, changes in overall level of activation from first to second presentations.
This repetition modulation (RM) has already been shown to covary with familiarity
(Eger, Schweinberger, Dolan, & Henson, 2005), so we created a nuisance regressor to
partial out any covariation between subsequent memory and face-specific similarity that
is shared with RM. We followed a similar approach to our calculation of lag-log-ratios
above. First we calculated the mean difference in activation between first and second
101
presentations (RM) for same-face and different-face pairings. We then log-transformed
these RM values. Finally, we took the difference between all same-face log-RMs and all
the corresponding different-face RMs:
𝐫
𝐦𝐢
𝑠
= [log 𝑒 (
𝑚𝑖
𝑑
𝑚𝑖1
𝑠
), log 𝑒 (
𝑚𝑖
𝑑
𝑚𝑖2
𝑠
), . . . , log 𝑒 (
𝑚𝑖
𝑑
𝑚𝑖𝑛
)] , (18)
𝑠
d
where 𝑚𝑖 is the mean RM between the first and second presentations of the 𝑖 𝑡ℎ face, min
is the scalar mean RM between the 𝑖 𝑡ℎ first presentation and the 𝑛𝑡ℎ different-face second
𝐫
presentation, and 𝐦𝐢 is the resulting vector of RM-log-ratios. These values quantify the
relative difference in RM between same- and different-face pairings, such that the values
would be zero when the relative difference was zero.
We included both lag-log-ratio and RM-log-ratio as nuisance regressors in our
statistical model. However, we had no hypotheses concerning these variables. Therefore,
we avoid making inferences about them, and do not report statistics or significance in this
paper.
Finally, our model included three random effects terms: Firstly, individual brains
may differ in the degree to which they exhibit face-specific similarity. Therefore, we
treated participant as a random effect. This allowed us to account for any variation in
face-specific similarity that may be due to the particular participants in our sample.
Additionally, some faces may tend to evoke more distinct neural responses than others.
To control for any differences in face-specific similarity due to stimulus distinctiveness,
we treated each same-face target and each different-face target as random effects.
102
Determining degrees of freedom for mixed effects regression models is not
straightforward. Instead, parametric statistical inference requires an estimation of the
degrees of freedom (Kenward & Roger, 1997). To circumvent this estimation, we
performed non-parametric permutation tests to ascertain the p-values for each of our
comparisons of interest (Ernst, 2004). For each ROI analysis, we performed 10,000
permutations of the dependent data by randomly flipping the sign of the similarity
difference. We then calculated two-tailed p-values based on the null distribution of tvalues for each regression term of interest. Finally, to correct for multiple comparisons
we used a Benjamini-Hochberg correction (Genovese, Lazar, & Nichols, 2002) to keep
the false discovery rate at 𝛼 = 0.05 across all tests. For significant results, we provide the
raw p-values, followed by corrected p-values in parentheses.
Results
Behavioral Results
Participants responded correctly on 93.1% of all trials of the gender categorization
task during encoding. We defined incorrect responses as those where the participant
provided the wrong gender or failed to make a response within the 1.2 s the face was on
the screen. Only face pairs where the participant responded correctly during both
encoding presentations were included in subsequent neural similarity analyses. This
ensured that differences in similarity were not due to a lack of attention to the faces.
The mean reaction time for completed trials was 647.1 ms (SD=158.1 ms). We
looked for evidence that the first presentation of a face would facilitate the gender
103
judgement response for the second presentation of the same face. We did not find
evidence of repetition priming (mean=4.03 ms, SD=41.48 ms), t(26)=0.49, p=0.62.
Turning to the recognition test, we calculated hit rates and false-alarm rates for
each subject. The mean hit rate for all target items was 51.6% (SD=18.8%), with a
corresponding mean false-alarm rate of 35.9% (SD=14.3%). As expected, given the use
of unfamiliar faces, performance on the memory task was poor, yet, on average,
participants performed above chance. For a comparison, the mean hit rate for non-target,
once-presented faces was 41.6% (SD=13.8%). As would be expected, a paired samples ttest revealed a significant boost in memory performance for twice-presented items,
t(26)=2.36, p=0.026.
Figure 12 shows the memory performance for target trials split by condition. We
find a mean hit rate of 50.9% (SD=21.6%) for faces presented twice in the same context
(low-EV), and a mean hit rate of 52.3% (SD=19.8%) for those presented in different
contexts (high-EV). A paired-samples t-test did not detect a significant difference
between conditions, t(26)=-0.32, p=0.75. Thus, at least in this experiment, variability in
encoding context does not appear to be related to memory performance. Because
encoding variability is unrelated to memory performance, we forego investigating
whether this (null) relationship is mediated by neural similarity. We collapse across
encoding variability conditions for the remainder of our analyses, focusing instead on a
potential relationship between similarity and subsequent memory.
104
Figure 12. Subsequent memory performance as a function of encoding condition. Bars
show mean proportion of faces recognized during the memory test for faces that occurred
under high encoding variability (dark green) and low encoding variability (light green).
For a reference, the mean proportion of once-presented items that were subsequently
remembered is shown in grey. Error bars reflect standard errors. The solid red line
indicates the mean proportion of lures that were incorrectly judged as having been seen
previously, with dashed red lines indicating standard error.
Neuroimaging Results
For each of our four ROIs, we tested whether face-specific similarity was present
for subsequently remembered and subsequently forgotten faces. We also tested for a
difference in face-specific similarity between forgotten and remembered faces, for a total
of 12 tests across ROIs. We conducted these tests using a linear mixed-effects regression
105
procedure, which let us control for nuisance regressors (lag between presentations and
repetition modulation), as well as participant-specific and stimulus-specific effects. We
controlled for multiple comparisons using a Benjamini-Hochberg correction (Genovese et
al., 2002) to keep the false discovery rate at 𝛼 = 0.05. Corrected p-values are shown in
parentheses for significant results.
In FFA, face-specific similarity was greater for subsequently remembered faces,
t=3.29, p=0.001 (.007). Furthermore, significant face-specific similarity was found for
remembered faces, t=2.57, p=0.012 (0.046), but not for forgotten faces, t=0.11, p>0.91
(see Figure 13A).
In aIT, face-specific similarity was also greater for remembered than forgotten
items; however, this finding did not survive correction for multiple comparisons t=1.99, p
= 0.045 (0.135). Once again, we found significant face-specific similarity in for
remembered faces, t=2.39, p=0.007 (0.043), but not for forgotten faces, t=0.85, p>.05
(see Figure 13B).
In OFA, an upstream face-sensitive region that is generally believed to process
face components, we found no effect of memory on face-specific similarity, t=-1.16,
p>0.23. Furthermore, we found no evidence of face-specific similarity for either
remembered, t=-0.33, p>0.74, or forgotten faces, t=0.58, p>0.57.
As expected, our control region—the scene-selective PPA—similarly showed no
effect of memory on face-specific similarity, t=0.89, p>.36. PPA also showed no facespecific similarity for remembered, t=0.33, p>0.74, or forgotten faces, t=-0.37, p>0.69.
106
In support of the familiarization hypothesis, we found that memory performance
was a significant predictor of face-specific similarity in down-stream face-sensitive
regions. We found no evidence of face-specific information in OFA, which is consistent
with the idea that subsequent memory for faces—not their individual features—is an
important moderator.
107
Figure 13. Average similarity between face repetitions. Bars show similarity between
first presentation targets and either (dark blue) same or (light blue) different second
presentations, split by memory performance. (A) FFA shows significant face-specific
information–greater similarity between identical faces than between different faces of the
same gender–only for faces that are subsequently remembered. The difference in facespecific similarity between remembered and forgotten faces is also significant. (B) aIT
shows the same pattern, although the difference in face-specific information is no longer
significant after correcting for multiple comparisons. † 𝑝 < 0.05 uncorrected, ∗ 𝑝 < 0.05
corrected, ∗∗ 𝑝 < 0.01 corrected.
108
Discussion
The present results demonstrate that downstream face-processing regions, such as
the FFA and aIT, contain unique face-specific information after minimal familiarization.
We find no evidence of such face-specific information for faces that participants are
unable to subsequently recognize. This was unlikely to be due to participants not
attending to those stimuli because we only included faces in our analyses that the
participants successfully categorized as male or female within 1.2 seconds. We also find
no evidence of face-specific information in early face-selective regions (OFA) or in a
control region that is not specifically sensitive to faces (PPA). We used subsequent
memory as an index of the strength of encoding into memory across both presentations of
a face; however, the mechanism that drives this encoding–similarity relationship has not
yet been explained. We next outline some potential mechanisms.
One possibility is the reactivation account proposed previously (Xue et al., 2010).
According to this account, memory is most improved when the same trace of a stimulus
is reactivated on subsequent presentations. Thus, greater face-specific similarity actually
reflects this stable reactivation, which gives rise to stronger memory encoding during
each presentation of a given face. Importantly, the current findings suggest this
reactivation refers to the representation of the face itself, rather than the face and its
context as it has been previously discussed (Xue et al., 2010).
A second possibility is a feedback account where patterns in face areas reflect
online perceptual processing that are associated with enduring memory representations in
other regions, such as the hippocampus (Buzsáki & Moser, 2013; Eichenbaum, 2000,
109
2004) or regions that have been implicated in person memory, such as the anterior
paracingulate cortex or amygdala (Gobbini & Haxby, 2007). These memory
representations may feed back to and alter the perceptual representations in upstream
areas such as the FFA during the second presentation of a face. Thus, the stronger a
memory representation is after a first presentation, the more that memory representation
may influence a subsequent perceptual representation to be like the first.
An alternate explanation for the current results is that attention, not memory
encoding per se, is the mechanism that produces face-specific similarity. Specifically,
faces observed under greater attention may be encoded with greater fidelity. This may
also produce more distinctive representations of those faces in face-sensitive regions.
Greater attention during encoding also facilitates subsequent memory (Chun & TurkBrowne, 2007; Turk-Browne, Golomb, & Chun, 2013). While we are agnostic with
regard to potential attention-based mechanisms at this time, future research could use
eye-tracking and pupillometry to provide a quantitative indices of attentional focus and
cortical arousal to help shed light on this hypothesis.
A more extreme characterization of this mechanism is that the lack of facespecific similarity for forgotten items is due to a complete lack of attention during
forgotten trials. Specifically, face areas will not represent stimuli if a participant’s eyes
were closed or looking away. This would also hamper memory performance for faces,
producing a correlation. However, our neural analyses only used face pairs that were
correctly categorized as by gender during both presentations. Performing this task
110
accurately requires at least some degree of attention. Thus, the current results are likely
not driven by an absence of attention during subsequently forgotten trials.
Furthermore, the current results may suggest that attention alone may not be
sufficient to produce face-specific information. If attention were the sole mechanism at
work, we would expect face-specific similarity in OFA precisely because we used the
same image for both presentations of our target faces. The OFA principally represents
lower-level features of faces (Haxby et al., 2000; Liu et al., 2009; Pitcher et al., 2011;
Pitcher et al., 2007), so greater attention should also yield greater representational fidelity
of those features in FFA. That is, the low-level features are the same, so attention to those
features could have produced “features-specific” information in OFA. We found no
evidence of face-specific similarity for familiar faces in OFA. This is consistent with
earlier research that found no effect of familiarity in OFA when using univariate methods
(Davies-Thompson, Gouws, & Andrews, 2009). Our null result in OFA suggests
something more than attention is necessary to produce face-specific information in FFA
and aIT.
Another potential explanation is that both face-specific similarity and memory are
driven by the features of our stimuli. That is, some faces may be more visually distinctive
than others. Greater distinctiveness could produce both greater face-specific similarity
and better memory performance. By including face labels as a random factor in our
statistical analyses, we were able to control for variation in similarity that was attributable
to the varying distinctiveness of different faces. However, this procedure does not control
for within-participant variation in face distinctiveness. That is, a particular face may not
111
be seen as distinctive by the entire sample, but could be particularly distinctive for a
particular participant. This distinctiveness could still affect that participant’s memory and
neural response pattern.
The present results are limited by the difficulty in imaging the anterior temporal
lobes (Devlin et al., 2000; Visser et al., 2009). If we had been able to reliably extract
functional-localized anterior temporal face areas from our participants, we may have
found stronger or different effects in that region. Nonetheless, the fact that our atlasbased aIT results followed the same pattern as the FFA results is evidence that familiarity
may serve the same function in both regions. fMRI protocols optimized to image the
anterior temporal lobes may provide better functional localization for future work.
Although the current work relied on repetitions of identical images of the target
faces, future research may benefit from including alternate images of the same face
(e.g. from different viewpoints) for first and second presentations. Previous work has
demonstrated that the FFA and aIT contain patterns that discriminate highly familiar
identities, even if alternate images are used for the same face (Anzellotti & Caramazza,
2014). This image-invariant representation of identity has been shown with respect to
variations in viewing angle (Anzellotti et al., 2013; Natu et al., 2009) and emotional
expression (Nestor et al., 2011). However, each of these studies relied on identities that
were highly familiar to participants due to training or previous experience. Future work
should investigate the extent to which image variation (e.g. head rotation, morphing
along a dimension, or different expressions) affect representations for untrained
identities.
112
Conclusions
These results begin to elucidate how the face-processing network comes to
represent individual faces. Face-specific information is detectable in FFA and aIT, but
not OFA, on a trial level. Extensive training of the stimuli before or during scanning is
not required to elicit this face-specific information. However, we only detect facespecific information for faces that were subsequently recognized. Thus, these results
implicate the development of familiarity as being directly related to the emergence of
face-specific information in ventral-stream face regions.
These results also have important implications for our memory model. In TCMSR, the item layer’s representation is noiseless and stable throughout the presentation of a
given stimulus. Our results here suggest that this implementation is inadequate. If our
feedback hypothesis is correct, then the item layer representation must evolve
dynamically based on signals coming from both the stimulus and long-term memory. If
the reactivation account is correct, then noise must be added to the item layer
representations. Implementing and comparing these model variants, which could be
constrained using both the neural and behavioral data (Turner et al., 2013), could give
support to one account over the other. Either of these models may, in turn, offer new
predictions regarding memory function, prediction generation, or perceptual processing.
113
Chapter 5: General Discussion
The present work investigated the relationship between episodic memory and both
prediction and perception. To accomplish the former, we developed a novel experimental
paradigm that tests for effects of prediction on memory. These effects are not detectable
using conventional memory paradigms; however, our new design found consistent
evidence that prediction influences memory. We also presented preliminary results from
a formal model of episodic memory that employs prediction-based learning. Our results
demonstrate that our model can account for the behavioral effects we discovered. Finally,
we used fMRI to investigate how neural representations of visual stimuli relate to
subsequent memory. In particular, we found that face-sensitive brain regions represent
information about specific faces only when those faces will be subsequently remembered.
We first review these findings in more detail and discuss their implications. We conclude
with a discussion of future directions.
Memory and Prediction
Adopting a common definition of context as a continuously evolving construct
(Howard & Kahana, 2002; Polyn et al., 2009; Sederberg et al., 2008), in which recent
experience makes up the context of the current experience, we were able to create an
experimental environment in which the repetition of an item’s context sometimes
preceded a repetition of the item itself. This created an experimental setting with
114
predictable regularities. In two experiments, this setting consistently produced enhanced
memory for once-presented items whose contexts were later repeated. The enhancement
was almost as large as the difference between once- and twice-presented items. However,
memory was not enhanced for items that followed a repetition of context. This
asymmetrical context repetition effect is difficult to explain using simple associative
learning mechanisms (Hebb, 1949). However, a learning rule that includes the learning of
predictions (Dayan, 1993; Gershman et al., 2012; White, 1995) has the potential to
account for both aspects of this effect.
Complicating matters is the fact this context repetition effect does not occur in
environments where the repetition of an item’s temporal context never precedes the
repetition of an item (i.e. environments lacking predictability). We interpret this null
context repetition effect, found in two additional behavioral experiments, as evidence that
the episodic memory system is sensitive to the overall predictability of the current
environment. In environments where predictions have little to no chance of being
accurate, the memory system may disregard those predictions instead of learning them.
This sensitivity to the presence of environmental regularities has obvious value.
However, it complicates explanations of how and when the brain produces predictionbased learning.
As a first step towards an adequate explanation, we developed a computational
model of recognition memory that uses a prediction-based learning mechanism adapted
from reinforcement learning (Dayan, 1993; White, 1995). This algorithm produces
behavior equivalent to Hebbian learning in standard memory paradigms. However, in
115
environments where prediction is possible it deviates from Hebbian models (Gershman et
al., 2012). We fit our model to two of our behavioral experiments: one that featured a
high-predictability environment and a significant context repetition effect, and another
that featured a low-predictability environment and a null context repetition effect. The
magnitude of the model’s context repetition effect is determined by a free parameter, 𝛾,
which scales the degree to which predictions are learned by the memory system.
Using a hierarchical Bayesian framework and a custom sampler, we estimated
posterior parameter distributions for each participant and a set of hyperpriors. To account
for behavioral findings in both high- and low-predictability experiments, we included two
hyperpriors for the prediction coefficient, 𝛾. Hyperparameters for a given participant’s 𝛾
parameter were drawn from one of these two hyperpriors, depending whether the
participant experienced the high- or low-predictability environment. Two of our three
model variants qualitatively reproduced the context repetition effect in a highpredictability environment and mostly reproduced the null context repetition effect in a
low-predictability environment. They also accurately recreated hit rates for once and
twice-presented targets, as well as false-alarm rates for lures. Finally, our model was also
able to capture individual differences in the context repetition effect.
Collectively, the empirical and computational modeling results presented here
provide strong evidence that the episodic memory system learns based on prediction.
This is not implausible given that the medial temporal lobe system is both the locus of
episodic memory (Davachi, 2006; Eichenbaum, 2000, 2004) and known to be critical for
generating predictions (Buckner, 2010; Johnson et al., 2007; Levy, 1989; Levy et al.,
116
2005; Lisman & Redish, 2009; Schacter et al., 2007). However, no other episodic
memory model currently makes use of prediction. Given that our experimental paradigm
reflects the regularity and moderate predictability of many real-world scenarios, we argue
that prediction-based learning may be an important—and overlooked—feature of human
memory. Future empirical work is needed to better define the boundary conditions of the
context repetition effect, as well as the pervasiveness of prediction-based learning more
generally. Additionally, more advanced computational models will be needed to explain
how the memory system calibrates 𝛾 to the current environment. Finally, future versions
of our memory model should include perceptual processes, in an effort to elucidate the
interactions between memory and perception.
Memory and Perception
Turning to the relationship between episodic memory and perception, we used
fMRI to investigate the degree to which face-selective brain regions within the ventral
processing stream encode information related to specific faces, rather than faces as a
category. Using multivariate techniques (Haxby et al., 2001; Xue et al., 2010), we found
evidence of face-specific information in two regions—the fusiform face area (Kanwisher
& Yovel, 2006; Kanwisher et al., 1997) and the anterior inferior temporal lobe (Collins &
Olson, 2014). Both regions are believed to process faces holistically (Collins & Olson,
2014; Harris & Aguirre, 2010; Liu et al., 2009). However, this face-specific information
was only found for faces that were recognized in a subsequent memory test. Additionally,
no face-specific information was found in regions that are not sensitive to faces or only
process the low-level features of faces.
117
To our knowledge, this experiment is the first demonstration that face-selective
regions in the ventral pathway can represent information about individual faces that have
never been seen before. This research also contributes to the growing literature that
explores the relationship between perception and memory. Our findings suggest that
researchers who study the neural pathways of perception—especially, though not
exclusively, regarding faces—should consider incorporating a subsequent memory task
into their research. In doing so, they may be able to control for variance in neural or
behavioral data that is due to memory.
Bridging Domains
The present research focused on three popular topics in the cognitive sciences:
memory, perception, and prediction. Our results demonstrate that the cognitive and neural
processes discussed in each of these domains do not operate in isolation. Rather, the
processes interact and influence one another. This statement no doubt holds true for
domains not discussed here, including: attention, decision-making, language, and affect.
Thus, completely understanding one process requires an understanding of the processes
that interact with it. The present research shows the importance of integrating different
domains. We are certainly not the first to take this approach, and we hope the coming
years will see an even greater emphasis placed on such efforts.
Limitations and Future Directions
Recognition vs. Recall
Every experiment in the present research used a typical item recognition test to
assess memory performance. An alternative testing method would have asked
118
participants to freely recall list items in any order. Free recall enables researchers to
investigate which items a participant would retrieve from memory without any
prompting, as well as the order in which these items would come to mind. Free recall
paradigms have been critical to our understanding of memory search and how the
memory system moves from one episodic retrieval to another (Kahana, 2012). However,
recognition tests afford researchers more control because they allow us to assess memory
for every item of interest, and in the order we choose. Recognition tests are also the only
feasible method of testing memory for unfamiliar faces and scenes. Finally, because
recalling items is much more difficult than recognizing them, free recall tests are
typically performed after each study list. Thus participants quickly learn that their
memory will be tested and adopt rehearsal strategies. Consequently, it is not feasible to
investigate incidental learning using free recall. Nevertheless, using words as stimuli, the
present research could be altered to test participants’ recall.
Testing for the presence of a context repetition effect during free recall would
require significant modifications to the paradigm. The number of items in each study list
would need to be reduced. This would then affect the number of conditions that would
occur in each list. Additionally, participants’ recalls would likely focus on the twicepresented items, perhaps to the point of ignoring the more difficult—but quite critical—
once-presented items. A potential solution may be to direct participants to recall the
relevant words by presenting them in one color and the less relevant words in another
color. The recall phase would then begin by asking participants to recall the relevantly
colored words. Significant pilot testing would be required to refine the paradigm.
119
However, the end result would enable us to explore how likely participants are to
transition from repeated contexts to the first and second novel targets (RCNT1 and
RCNT2, respectively). The results could either lend converging support to our current
prediction-based memory model or force us to modify it.
Recall could also tell us more about how memory and perception interact. For
instance, we might find that the amount of stimulus-specific information present in the
perceptual pathway correlates with the recall position of recalled items. Specifically,
study items with greater stimulus-specific representations may have higher probabilities
of being recalled first. Once again, testing recall would require significant modifications
to our experimental design. List length would need to be shortened, and words would
need to be used as stimuli. Fortunately previous research has already identified faceselective regions in the ventral processing stream (Dehaene et al., 2002; McCandliss et
al., 2003). Focusing on those regions, we may find an analogous relationship between
word-specific representations and subsequent memory.
The Neural Correlates of the Context Repetition Effect
The neuroimaging findings presented here focused on the perceptual
representation of faces. In principle, because that experiment contained the same
conditions used in our behavioral analysis of the context repetition effect, those data can
also be used to investigate the neural correlates of the behavioral context repetition effect.
However, this type of analysis presents several challenges. The first issue is
timing: When exactly prediction-based learning occurs is not entirely clear. When a
context is repeated, a prediction about the next target will follow. However, our current
120
model does not specify exact timing. In fact, the prediction-generation and predictionlearning processes may occur right up to, and potentially during, the onset of the actual
target. Furthermore, given the sluggish nature of the BOLD response, any signal
representing the prediction would overlap with BOLD signals coming from both the
repeated context stimuli and the subsequent novel target. The uncertainty about when
prediction-based learning occurs coupled with the overlapping BOLD responses presents
a serious challenge.
Recent advancements in multivariate imaging analysis (Nielson & Sederberg,
submitted) may enable us to detect these prediction signals. The basic goal is to find
specific patterns of neural activation, similar to our prior analyses on the perception of
faces. We may find patterns indicative of predictions in face-selective regions, suggesting
that predictions generated in the MTL affect activation in perceptual layers. This would
be consistent with the feedback account of our current neural findings. However,
information may also be detected in the MTL itself. Additionally, we might find a BOLD
response corresponding to prediction error when the unexpected novel target (RCNT2)
actually follows the repeated context. Such a signal would be indicative of predictiongeneration and would be expected in regions where similar prediction errors have been
reported previously, such as the lateral prefrontal cortex and intraparietal sulcus
(Gläscher, Daw, Dayan, & O’Doherty, 2010)7.
7
Although previous research has also implicated the dopaminergic system (e.g. ventral tegmental area,
substantia nigra, and ventral striatum) as an additional source of prediction error (D’Ardenne, McClure,
Nystrom, & Cohen, 2008; S. M. McClure, Berns, & Montague, 2003), signals found in these regions
correspond to reward prediction errors. The memory paradigm used in the present research does not feature
explicit rewards. Consequently, we have listed the lateral prefrontal cortex and intraparietal sulcus, as
BOLD responses in these regions are known to correspond to state prediction errors (Gläscher et al., 2010).
121
A Model of Memory, Perception, and Prediction
As mentioned previously, our current computational model is lacking any realistic
perceptual mechanism. Consequently, it cannot help us discriminate between multiple
explanations that could account for memory-dependent, face-specific information in the
visual pathway. A reactivation account posits that greater reactivation of the first neural
pattern during the second presentation leads to better memory strength for that face. A
feedback account posits that the second presentation cues the memory system, which
feeds back to the perceptual layer, constraining and biasing the new perceptual
representation. However, these verbal explanations are imprecise and can be difficult to
distinguish from one another. In contrast, formal mathematical explanations can avoid
ambiguities and offer precise predictions, which can then be tested.
Such a model would have key differences from the memory model proposed
earlier in this document. To begin with, our memory model featured a localist
representation of stimulus features. That is, the vector representing each stimulus
included only one nonzero element, which was always set to one to make each vector unit
length. However, distributed representations of stimuli will be critical in the modeling of
our perceptual findings. We may retain a localist representation for a layer symbolizing
the hippocampus, which encodes episodes using non-overlapping patterns (Norman &
O’Reilly, 2003). However, input to that hippocampal layer would need to come from a
perceptual layer, utilizing distributed, overlapping representations. Furthermore, this
perceptual layer would receive feedback from the hippocampal layer, enabling us to test
122
the feedback account. The context layer and associative matrix, as well as the learning
and decision rules, would likely remain in their current form.
The resulting model of perceptual representation, memory function, and
prediction generation would formalize and clarify our competing hypotheses regarding
our neural data, hopefully helping us to compare them. A model with distributed
perceptual representations may also lead to new questions regarding the relationship
between memory and prediction. Finally, this model may allow us to develop novel
hypotheses about the relationship between prediction and perception, a topic we have not
investigated so far. Thus, in an era where memory researchers tend to develop models to
account for specific tasks and effects, our model would be able to do something unique:
explain the interactions between memory, perception, and prediction.
The Role of Attention
Attention is key process in all of the research presented here. In the context
repetition behavioral paradigm, a participant must attend to the overall experimental
environment to learn that this environment is predictable. Similarly, attention to specific
stimuli is essential to successful encoding and the generation of predictions. In the
imaging experiment, fluctuations in attention could be at least partially responsible for
variations in both face-specific information and subsequent memory. Additionally, lapses
in attention almost certainly account for some of the noise in both our behavioral and
fMRI data. However, there is a difference between acknowledging the importance of
attention and actually incorporating attentional processes into one’s explanation.
123
Properly incorporating attention requires two additions to our program of
research. First, we must operationalize and quantify participants’ attention over time.
Eye-tracking could be used to track the focus of participants’ eye gaze, providing
information about their overt attention. Furthermore, pupillometry could be used to
provide an index of participants’ cognitive alertness over the course of an experiment.
Secondly, our computational accounts of the cognitive and neural phenomena would need
to be modified to include attentional mechanisms. Previous research argues that
perception and memory are functions of different types of attention, external and internal,
respectively (Chun et al., 2011). This suggests that incorporating attentional processes
into our model may make the interactions between the perceptual, memory, and
prediction components more cohesive and complete. The resulting model may also make
predictions about attention itself.
Conclusions
The present research explored the relationship between episodic memory and
prediction and perception. Using a novel experimental design and a computational
model, we demonstrated that in relatively predictable environments—not unlike realworld scenarios—humans use predictions to enhance their memory. Using advanced
neuroimaging techniques, we also demonstrated a link between the neural representation
of faces and memory for those faces. This research contributes to our understanding of
cognition and advances the methodologies used in computational modeling and fMRI
data analysis. Perhaps most importantly, this work demonstrates the utility of spanning
levels of analysis and bridging the domains of the cognitive sciences.
124
Bibliography
Andersson, J., Jenkinson, M., & Smith, S. (2007). Non-linear registration, aka spatial
normalisation FMRIB technical report TR07JA2 (Technical Report No. TR07JA2).
FMRIB Analysis Group of the University of Oxford.
Ando, T. (2007). Bayesian predictive information criterion for the evaluation of hierarchical
bayesian and empirical bayes models. Biometrika, 94(2), 443–458.
Anzellotti, S., & Caramazza, A. (2014). The neural mechanisms for the recognition of face
identity in humans. Frontiers in Psychology, 5.
Anzellotti, S., Fairhall, S. L., & Caramazza, A. (2013). Decoding representations of face identity
that are tolerant to rotation. Cerebral Cortex, bht046.
Bar, M. (2009). Predictions: A universal principle in the operation of the human brain.
Philosophical Transactions of the Royal Society of London B: Biological Sciences,
364(1521), 1181–1182.
Baxter, M. G. (2009). Involvement of medial temporal lobe structures in memory and perception.
Neuron, 61(5), 667–677.
Bousfield, W. A. (1953). The occurrence of clustering in the recall of randomly arranged
associates. The Journal of General Psychology, 49(2), 229–240.
Bousfield, W. A., & Sedgewick, C. H. W. (1944). An analysis of sequences of restricted
associative responses. The Journal of General Psychology, 30(2), 149–165.
Bray, J. F., Robbins, D., & Witcher, W. B. (1976). Encoding variability theory and the spacing
effect in associate learning. Memory & Cognition, 4(5), 548–552.
Buckner, R. L. (2010). The role of the hippocampus in prediction and imagination. Annual
Review of Psychology, 61(1), 27–48.
125
Busemeyer, J. R., & Diederich, A. (2010). Cognitive modeling. SAGE.
Busigny, T., Van Belle, G., Jemel, B., Hosein, A., Joubert, S., & Rossion, B. (2014). Facespecific impairment in holistic perception following focal lesion of the right anterior
temporal lobe. Neuropsychologia, 56, 312–333.
Bussey, T., & Saksida, L. (2007). Memory, perception, and the ventral visual-perirhinalhippocampal stream: Thinking outside of the boxes. Hippocampus, 17(9), 898–908.
Buzsáki, G., & Moser, E. I. (2013). Memory, navigation and theta rhythm in the hippocampalentorhinal system. Nature Neuroscience, 16(2), 130–138.
Cabeza, R., Mazuz, Y. S., Stokes, J., Kragel, J. E., Woldorff, M. G., Ciaramelli, E., …
Moscovitch, M. (2011). Overlapping parietal activity in memory and perception:
Evidence for the attention to memory model. Journal of Cognitive Neuroscience, 23(11),
3209–3217.
Chun, M. M., & Turk-Browne, N. B. (2007). Interactions between attention and memory.
Current Opinion in Neurobiology, 17(2), 177–184.
Chun, M. M., Golomb, J. D., & Turk-Browne, N. B. (2011). A taxonomy of external and internal
attention. Annual Review of Psychology, 62(1), 73–101.
Collins, J. A., & Olson, I. R. (2014). Beyond the FFA: The role of the ventral anterior temporal
lobes in face processing. Neuropsychologia, 61, 65–79.
Corbit, L. H., & Balleine, B. W. (2000). The role of the hippocampus in instrumental
conditioning. The Journal of Neuroscience, 20(11), 4233–4239.
Cox, R. W. (1996). AFNI: Software for analysis and visualization of functional magnetic
resonance neuroimages. Computers and Biomedical Research, 29(3), 162–173.
Cox, R. W. (2012). AFNI: What a long strange trip it’s been. Neuroimage, 62(2), 743–747.
D’Ardenne, K., McClure, S. M., Nystrom, L. E., & Cohen, J. D. (2008). BOLD responses
reflecting dopaminergic signals in the human ventral tegmental area. Science, 319(5867),
1264–1267.
Davachi, L. (2006). Item, context and relational episodic encoding in humans. Current Opinion
in Neurobiology, 16(6), 693–700.
126
Davies-Thompson, J., Gouws, A., & Andrews, T. J. (2009). An image-dependent representation
of familiar and unfamiliar faces in the human ventral stream. Neuropsychologia, 47(6),
1627–1635.
Dayan, P. (1993). Improving generalization for temporal difference learning: The successor
representation. Neural Computation, 5(4), 613–624.
Deese, J. (1959). On the prediction of occurrence of particular verbal intrusions in immediate
recall. Journal of Experimental Psychology, 58(1), 17–22.
Dehaene, S., Le Clec’H, G., Poline, J.-B., Le Bihan, D., & Cohen, L. (2002). The visual word
form area: A prelexical representation of visual words in the fusiform gyrus.
Neuroreport, 13(3), 321–325.
Dennis, S., & Humphreys, M. S. (2001). A context noise model of episodic word recognition.
Psychological Review, 108(2), 452–478.
Devlin, J. T., Russell, R. P., Davis, M. H., Price, C. J., Wilson, J., Moss, H. E., … Tyler, L. K.
(2000). Susceptibility-induced loss of signal: Comparing PET and fMRI on a semantic
task. NeuroImage, 11(6), 589–600.
Ebbinghaus, H. (1913). Memory: A contribution to experimental psychology. University
Microfilms.
Eger, E., Schweinberger, S. R., Dolan, R. J., & Henson, R. N. (2005). Familiarity enhances
invariance of face representations in human ventral visual cortex: fMRI evidence.
NeuroImage, 26(4), 1128–1139.
Eichenbaum, H. (2000). A corticalhippocampal system for declarative memory. Nature Reviews
Neuroscience, 1(1), 41–50.
Eichenbaum, H. (2004). Hippocampus: Cognitive processes and neural representations that
underlie declarative memory. Neuron, 44(1), 109–120.
Epstein, R., & Kanwisher, N. (1998). A cortical representation of the local visual environment.
Nature, 392(6676), 598–601.
Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19(4),
676–685.
127
Estes, W. (1955). Statistical theory of spontaneous recovery and regression. Psychological
Review, 62(3), 145–154.
Falkenberg, P. R. (1972). Recall improves in short-term memory the more recall context
resembles learning context. Journal of Experimental Psychology, 95(1), 39–47.
Fanselow, M. S. (2000). Contextual fear, gestalt memories, and the hippocampus. Behavioural
Brain Research, 110(12), 73–81.
Freiwald, W. A., & Tsao, D. Y. (2010). Functional compartmentalization and viewpoint
generalization within the macaque face-processing system. Science (New York, N.Y.),
330(6005), 845–851.
Freiwald, W. A., Tsao, D. Y., & Livingstone, M. S. (2009). A face feature space in the macaque
temporal lobe. Nature Neuroscience, 12(9), 1187–1196.
Gainotti, G., & Marra, C. (2011). Differential contribution of right and left temporo-occipital and
anterior temporal lesions to face recognition disorders. Frontiers in Human
Neuroscience, 5, 55.
Gauthier, I., Tarr, M. J., Moylan, J., Skudlarski, P., Gore, J. C., & Anderson, A. W. (2000). The
fusiform “face area” is part of a network that processes faces at the individual level.
Journal of Cognitive Neuroscience, 12(3), 495–504.
Geller, A. S., Schleifer, I. K., Sederberg, P. B., Jacobs, J., & Kahana, M. J. (2007). PyEPL: A
cross-platform experiment-programming library. Behavior Research Methods, 39(4),
950–958.
Genovese, C. R., Lazar, N. A., & Nichols, T. (2002). Thresholding of statistical maps in
functional neuroimaging using the false discovery rate. NeuroImage, 15(4), 870–878.
Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., & Sederberg, P. B. (2012). The
successor representation and temporal context. Neural Computation, 24(6), 1553–1568.
Gillund, G., & Shiffrin, R. M. (1984). A retrieval model for both recognition and recall.
Psychological Review, 91(1), 1–67.
Glanzer, M., Hilford, A., & Maloney, L. T. (2009). Likelihood ratio decisions in memory: Three
implied regularities. Psychonomic Bulletin & Review, 16(3), 431–455.
128
Gläscher, J., Daw, N., Dayan, P., & O’Doherty, J. P. (2010). States versus rewards: Dissociable
neural prediction error signals underlying model-based and model-free reinforcement
learning. Neuron, 66(4), 585–595.
Glenberg, A. M. (1979). Component-levels theory of the effects of spacing of repetitions on
recall and recognition. Memory & Cognition, 7(2), 95–112.
Gobbini, M. I., & Haxby, J. V. (2007). Neural systems for recognition of familiar faces.
Neuropsychologia, 45(1), 32–41.
Godden, D. R., & Baddeley, A. D. (1975). Context-dependent memory in two natural
environments: On land and underwater. British Journal of Psychology, 66(3), 325–331.
Goodhill, G. J., & Barrow, H. G. (1994). The role of weight normalization in competitive
learning. Neural Computation, 6(2), 255–269.
Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysicsJohn wiley. New
York.
Grill-Spector, K., Henson, R., & Martin, A. (2006). Repetition and the brain: Neural models of
stimulus-specific effects. Trends in Cognitive Sciences, 10(1), 14–23.
Grill-Spector, K., Knouf, N., & Kanwisher, N. (2004). The fusiform face area subserves face
perception, not generic within-category identification. Nature Neuroscience, 7(5), 555–
562.
Harris, A., & Aguirre, G. K. (2010). Neural tuning for face wholes and parts in human fusiform
gyrus revealed by fMRI adaptation. Journal of Neurophysiology, 104(1), 336–345.
Hasher, L., & Zacks, R. T. (1979). Automatic and effortful processes in memory. Journal of
Experimental Psychology: General, 108(3), 356–388.
Hawkins, J., George, D., & Niemasik, J. (2009). Sequence memory for prediction, inference and
behaviour. Philosophical Transactions of the Royal Society of London B: Biological
Sciences, 364(1521), 1203–1209.
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001).
Distributed and overlapping representations of faces and objects in ventral temporal
cortex. Science, 293(5539), 2425–2430.
129
Haxby, J. V., Hoffman, E. A., & Gobbini, M. I. (2000). The distributed human neural system for
face perception. Trends in Cognitive Sciences, 4(6), 223–233.
Hebb, D. O. (1949). The organization of behavior (Vol. 50). New York: Wiley.
Hintzman, D. L. (1988). Judgments of frequency and recognition memory in a multiple-trace
memory model. Psychological Review, 95(4), 528–551.
Howard, M. W., & Kahana, M. J. (2002). A distributed representation of temporal context.
Journal of Mathematical Psychology, 46(3), 269–299.
Howard, M. W., Fotedar, M. S., Datey, A. V., & Hasselmo, M. E. (2005). The temporal context
model in spatial navigation and relational learning: Toward a common explanation of
medial temporal lobe function across domains. Psychological Review, 112(1), 75–116.
Jenkinson, M., & Smith, S. (2001). A global optimisation method for robust affine registration of
brain images. Medical Image Analysis, 5(2), 143–156.
Jenkinson, M., Bannister, P., Brady, M., & Smith, S. (2002). Improved optimization for the
robust and accurate linear registration and motion correction of brain images.
NeuroImage, 17(2), 825–841.
Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., & Smith, S. M. (2012).
FSL. NeuroImage, 62(2), 782–790.
Jenkinson, M., Pechaud, M., & Smith, S. (2005, June). BET2: MR-based estimation of brain,
skull and scalp surfaces.
Johnson, A., Meer, M. A. van der, & Redish, A. D. (2007). Integrating hippocampus and
striatum in decision-making. Current Opinion in Neurobiology, 17(6), 692–697.
Kahana, M. J. (2012). Foundations of human memory. Oxford University Press.
Kanwisher, N., & Yovel, G. (2006). The fusiform face area: A cortical region specialized for the
perception of faces. Philosophical Transactions of the Royal Society B: Biological
Sciences, 361(1476), 2109–2128.
Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: A module in
human extrastriate cortex specialized for face perception. The Journal of Neuroscience,
17(11), 4302–4311.
130
Kanwisher, N., Tong, F., & Nakayama, K. (1998). The effect of face inversion on the human
fusiform face area. Cognition, 68(1), B1–B11.
Kennedy, P. J., & Shapiro, M. L. (2004). Retrieving memories via internal context requires the
hippocampus. The Journal of Neuroscience, 24(31), 6979–6985.
Kenward, M. G., & Roger, J. H. (1997). Small sample inference for fixed effects from restricted
maximum likelihood. Biometrics, 53(3), 983–997.
Kimball, D. R., Smith, T. A., & Kahana, M. J. (2007). The fSAM model of false recall.
Psychological Review, 114(4), 954–993.
Kirkland, T., & Cunningham, W. A. (2011). Neural basis of affect and emotion. Wiley
Interdisciplinary Reviews: Cognitive Science, 2(6), 656–665.
Kriegeskorte, N., Formisano, E., Sorger, B., & Goebel, R. (2007). Individual faces elicit distinct
response patterns in human anterior temporal cortex. Proceedings of the National
Academy of Sciences, 104(51), 20600–20605.
Levy, W. B. (1989). A computational approach to hippocampal function. Computational Models
of Learning in Simple Neural Systems, 23, 243–305.
Levy, W. B., Hocking, A. B., & Wu, X. (2005). Interpreting hippocampal function as recoding
and forecasting. Neural Networks, 18(9), 1242–1264.
Lisman, J., & Redish, A. D. (2009). Prediction, sequences and the hippocampus. Philosophical
Transactions of the Royal Society of London B: Biological Sciences, 364(1521), 1193–
1201.
Liu, J., Harris, A., & Kanwisher, N. (2009). Perception of face parts and face configurations: An
fMRI study. Journal of Cognitive Neuroscience, 22(1), 203–211.
Maguire, E. A., & Hassabis, D. (2011). Role of the hippocampus in imagination and future
thinking. Proceedings of the National Academy of Sciences, 108(11), E39–E39.
Marr, D. (1982). Vision: A computational investigation into the human representation and
processing of visual information. new york, NY, USA: Henry holt and co. Inc. June.
Martin, A. (2007). The representation of object concepts in the brain. Annual Review of
Psychology, 58(1), 25–45.
131
Martin, E. (1968). Stimulus meaningfulness and paired-associate transfer: An encoding
variability hypothesis. Psychological Review, 75(5), 421–441.
McCandliss, B. D., Cohen, L., & Dehaene, S. (2003). The visual word form area: Expertise for
reading in the fusiform gyrus. Trends in Cognitive Sciences, 7(7), 293–299.
McClure, S. M., Berns, G. S., & Montague, P. R. (2003). Temporal prediction errors in a passive
learning task activate human striatum. Neuron, 38(2), 339–346.
Melton, A. W. (1970). The situation with respect to the spacing of repetitions and memory.
Journal of Verbal Learning and Verbal Behavior, 9(5), 596–606.
Milton, F., Muhlert, N., Butler, C. R., Smith, A., Benattayallah, A., & Zeman, A. Z. (2011). An
fMRI study of long-term everyday memory using SenseCam. Memory, 19(7), 733–744.
Mitchell, J. P. (2009). Inferences about mental states. Philosophical Transactions of the Royal
Society of London B: Biological Sciences, 364(1521), 1309–1316.
Mizumori, S. J., Ragozzino, K. E., Cooper, B. G., & Leutgeb, S. (1999). Hippocampal
representational organization and spatial context. Hippocampus, 9(4), 444–451.
Murdock, B. B. (1993). TODAM2: A model for the storage and retrieval of item, associative,
and serial-order information. Psychological Review, 100(2), 183–203.
Mystkowski, J. L., Mineka, S., Vernon, L. L., & Zinbarg, R. E. (2003). Changes in caffeine
states enhance return of fear in spider phobia. Journal of Consulting and Clinical
Psychology, 71(2), 243–250.
Natu, V. S., Jiang, F., Narvekar, A., Keshvari, S., Blanz, V., & O’Toole, A. J. (2009).
Dissociable neural patterns of facial identity across changes in viewpoint. Journal of
Cognitive Neuroscience, 22(7), 1570–1582.
Nelson, D. L., McEvoy, C. L., & Schreiber, T. A. (2004). The university of south florida free
association, rhyme, and word fragment norms. Behavior Research Methods, Instruments,
& Computers, 36(3), 402–407.
Nestor, A., Plaut, D. C., & Behrmann, M. (2011). Unraveling the distributed neural code of
facial identity through spatiotemporal pattern analysis. Proceedings of the National
Academy of Sciences, 108(24), 9998–10003.
132
Nielson, D., & Sederberg, P. B. (submitted). MELD paper. Submitted Manuscript.
Nielson, D., Smith, T. A., Sreekumar, V., & Sederberg, P. B. (submitted). Experience sampling
paper. Submitted Manuscript.
Norman, K. A., & O’Reilly, R. C. (2003). Modeling hippocampal and neocortical contributions
to recognition memory: A complementary-learning-systems approach. Psychological
Review, 110(4), 611–646.
Olson, I. R., Ezzyat, Y., Plotzker, A., & Chatterjee, A. (2014). The end point of the ventral visual
stream: Face and non-face perceptual deficits following unilateral anterior temporal lobe
damage. Neurocase, 0(0), 1–9.
Olson, I. R., McCoy, D., Klobusicky, E., & Ross, L. A. (2013). Social cognition and the anterior
temporal lobes: A review and theoretical framework. Social Cognitive and Affective
Neuroscience, 8(2), 123–133.
Pitcher, D., Walsh, V., & Duchaine, B. (2011). The role of the occipital face area in the cortical
face perception network. Experimental Brain Research, 209(4), 481–493.
Pitcher, D., Walsh, V., Yovel, G., & Duchaine, B. (2007). TMS evidence for the involvement of
the right occipital face area in early face processing. Current Biology, 17(18), 1568–
1573.
Polyn, S. M., Norman, K. A., & Kahana, M. J. (2009). A context maintenance and retrieval
model of organizational processes in free recall. Psychological Review, 116(1), 129–156.
Raaijmakers, J. G., & Shiffrin, R. M. (1981). Search of associative memory. Psychological
Review, 88(2), 93–134.
Ratcliff, R., Hockley, W., & McKoon, G. (1985). Components of activation: Repetition and
priming effects in lexical decision and recognition. Journal of Experimental Psychology:
General, 114(4), 435–450.
Rescorla, R. A., & Wagner, A. R. (1972). A theory of pavlovian conditioning: Variations in the
effectiveness of reinforcement and nonreinforcement. In Classical conditioning: Current
research and theory.
Rissman, J., Gazzaley, A., & D’Esposito, M. (2004). Measuring functional connectivity during
distinct stages of a cognitive task. NeuroImage, 23(2), 752–763.
133
Roediger, H. L., & McDermott, K. B. (1995). Creating false memories: Remembering words not
presented in lists. Journal of Experimental Psychology: Learning, Memory, and
Cognition, 21(4), 803–814.
Ross, L. A., & Olson, I. R. (2010). Social cognition and the anterior temporal lobes.
NeuroImage, 49(4), 3452–3462.
Schacter, D. L., & Addis, D. R. (2009). On the nature of medial temporal lobe contributions to
the constructive simulation of future events. Philosophical Transactions of the Royal
Society of London B: Biological Sciences, 364(1521), 1245–1253.
Schacter, D. L., Addis, D. R., & Buckner, R. L. (2007). Remembering the past to imagine the
future: The prospective brain. Nature Reviews Neuroscience, 8(9), 657–661.
Sederberg, P. B., Howard, M. W., & Kahana, M. J. (2008). A context-based theory of recency
and contiguity in free recall. Psychological Review, 115(4), 893–912.
Shankar, K. H., & Howard, M. W. (2012). A scale-invariant internal representation of time.
Neural Computation, 24(1), 134–193.
Shankar, K. H., Jagadisan, U. K. K., & Howard, M. W. (2009). Sequential learning using
temporal context. Journal of Mathematical Psychology, 53(6), 474–485.
Shiffrin, R. M., & Steyvers, M. (1997). A model for recognition memory: REMretrieving
effectively from memory. Psychonomic Bulletin & Review, 4(2), 145–166.
Skaggs, W. E., McNaughton, B. L., Wilson, M., & Barnes, C. (1996). Theta phase precession in
hippocampal. Hippocampus, 6, 149–172.
Smith, S. M., Jenkinson, M., Woolrich, M. W., Beckmann, C. F., Behrens, T. E. J., JohansenBerg, H., … Matthews, P. M. (2004). Advances in functional and structural MR image
analysis and implementation as FSL. NeuroImage, 23 Suppl 1, S208–219.
Smith, T. A., Hasinski, A. E., & Sederberg, P. B. (2013). The context repetition effect: Predicted
events are remembered better, even when they don’t happen. Journal of Experimental
Psychology: General, 142(4), 1298–1308.
Stokes, M. G., Atherton, K., Patai, E. Z., & Nobre, A. C. (2012). Long-term memory prepares
neural activity for perception. Proceedings of the National Academy of Sciences, 109(6),
E360–E367.
134
Summerfield, C., Trittschuh, E. H., Monti, J. M., Mesulam, M.-M., & Egner, T. (2008). Neural
repetition suppression reflects fulfilled perceptual expectations. Nature Neuroscience,
11(9), 1004–1006.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning, a bradford book. MIT Press,
Cambridge, MA.
Suzuki, W. A. (2009). Perception and the medial temporal lobe: Evaluating the current evidence.
Neuron, 61(5), 657–666.
Suzuki, W. A., & Baxter, M. G. (2009). Memory, perception, and the medial temporal lobe: A
synthesis of opinions. Neuron, 61(5), 678–679.
Tistarelli, M., Bicego, M., & Grosso, E. (2009). Dynamic face recognition: From human to
machine vision. Image and Vision Computing, 27(3), 222–232.
Tsao, D. Y., Freiwald, W. A., Tootell, R. B. H., & Livingstone, M. S. (2006). A cortical region
consisting entirely of face-selective cells. Science, 311(5761), 670–674.
Tullis, J. G., Benjamin, A. S., & Ross, B. H. (2014). The reminding effect: Presentation of
associates enhances memory for related words in a list. Journal of Experimental
Psychology: General, 143(4), 1526–1540.
Turk-Browne, N. B., Golomb, J. D., & Chun, M. M. (2013). Complementary attentional
components of successful memory encoding. NeuroImage, 66, 553–562.
Turk-Browne, N. B., Simon, M. G., & Sederberg, P. B. (2012). Scene representations in
parahippocampal cortex depend on temporal context. The Journal of Neuroscience,
32(21), 7202–7207.
Turner, B. M., & Sederberg, P. B. (2013). A generalized, likelihood-free method for posterior
estimation. Psychonomic Bulletin & Review, 21(2), 227–250.
Turner, B. M., Forstmann, B. U., Wagenmakers, E.-J., Brown, S. D., Sederberg, P. B., &
Steyvers, M. (2013). A bayesian framework for simultaneously modeling neural and
behavioral data. NeuroImage, 72, 193–206.
Ucros, C. G. (1989). Mood state-dependent memory: A meta-analysis. Cognition and Emotion,
3(2), 139–169.
135
Van Bavel, J. J., Packer, D. J., & Cunningham, W. A. (2008). The neural substrates of in-group
bias a functional magnetic resonance imaging investigation. Psychological Science,
19(11), 1131–1139.
Visser, M., Jefferies, E., & Lambon Ralph, M. A. (2009). Semantic processing in the anterior
temporal lobes: A meta-analysis of the functional neuroimaging literature. Journal of
Cognitive Neuroscience, 22(6), 1083–1094.
Ward, B. (2002). Deconvolution analysis of fMRI time series data. Milwaukee, WI: Biophysics
Research Institute, Medical College of Wisconsin.
Weingartner, H., Adefris, W., Eich, J. E., & Murphy, D. L. (1976). Encoding-imagery specificity
in alcohol state-dependent learning. Journal of Experimental Psychology: Human
Learning and Memory, 2(1), 83–87.
White, L. (1995). Temporal difference learning: Eligibility traces and the successor
representation for actions. Unpublished master’s thesis, De- partment of Computer
Science, University of Toronto.
Xue, G., Dong, Q., Chen, C., Lu, Z., Mumford, J. A., & Poldrack, R. A. (2010). Greater neural
pattern similarity across repetitions is associated with better memory. Science, 330(6000),
97–101.
Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S., & Reynolds, J. R. (2007). Event
perception: A mind-brain perspective. Psychological Bulletin, 133(2), 273–293.
136