Prefrontal Cortex and the Flexibility of Cognitive Control:
Rules Without Symbols
Nicolas P. Rougier
David C. Noelle
Todd S. Braver
Dept of Psychology, CU Boulder
Department of Computer Science
Department of Psychology
INRIA Lorraine, France
Vanderbilt University
Washington University
Jonathan D. Cohen†
Randall C. O’Reilly†
Department of Psychology
Department of Psychology
Princeton University
University of Colorado Boulder
Abstract:
A hallmark of human behavior is the flexibility of cognitive control, which is believed to depend on prefrontal cortex (PFC). Previous work has suggested that specific PFC mechanisms, including active maintenance of rule or goal representations, adaptive gating of these representations, and the top-down biasing
of processing in other cortical areas, are critical for flexible cognitive control. Neural network models have
demonstrated that these mechanisms are sufficient to accurately simulate flexible cognitive control in a
variety of tasks. However, these models have not addressed a fundamental question: How is information
represented in PFC and, critically, how does this develop? Here, we show that PFC-specific mechanisms
interact with the breadth of training experience to produce abstract, rule-like representations that support
generalization of performance in novel task circumstances. We also show that these rule-like representations support patterns of performance characteristic of neurologically intact and frontally-damaged people
on benchmark tasks of cognitive control (Stroop and WCST). Although the rule-like representations that
developed in our model of PFC support flexible cognitive control, they do so in a way that is fundamentally different from symbolic representations characteristic of more traditional unified theories of cognition.
Therefore, these results bear on both the organization and development of PFC at the neurobiogical level,
as well as debates regarding the nature of cognitive flexibility and rule-like behavior at the psychological
level.
† corresponding authors: [email protected],
345 UCB, Boulder, CO 80309-0345 or [email protected],
Green Hall, Princeton, NJ 08544. Supported by ONR grants
N00014-00-1-0246 and N00014-03-1-0428, and NIH grants
MH64445. Last authorship reflects equal contribution; order was determined by a flip of coin. We thank Tim Curran,
Michael Frank, Tom Hazy, Dave Jilk, Ken Norman, Yuko Munakata and members of the CCN lab for helpful comments.
One of the fundamental faculties that distinguishes human cognition and behavior from other species is
our capacity for cognitive control: the ability to behave in accord with rules, goals, or intentions, even when
this runs counter to reflexive or otherwise highly compelling competing responses (e.g., the ability to keep
typing rather than scratch a mosquito bite). A hallmark of the human capacity for cognitive control is its
remarkable flexibility. This flexibility is exhibited in many ways, including the ability to perform a familiar
task in novel environments (generalization; e.g., playing cards with a novel type of deck with new players in
a new location), and the ability to rapidly configure the behavior needed to perform arbitrary new tasks with
relatively little or no prior training (generativity; e.g., playing a new card game for the first time). These
abilities pose a major challenge to theories of cognitive control: How can they be explained in terms of selforganizing mechanisms (that develop on their own, without recourse to unexplained sources of influence or
intelligence; i.e., a “homunculus”), and extensive but not infinite computational resources (1)?
Classic unified theories of cognition, such as ACT-R and SOAR (2), have addressed this issue by assuming that human cognition has the capacity for symbol processing — that is, the ability to execute procedures
for any values of the relevant variables, and to arbitrarily combine these procedures in new ways. While
such theories provide a powerful account of human behavior, they fail to answer several important questions. First, most models based on these theories pre-specify the basic set of procedures that are available,
without explaining how these are acquired. Thus, the modeler is required to imbue such models with much
of the intelligence that is to be explained. Second, one of the great mysteries of neuroscience is how coordinated, goal directed behavior can arise from the highly distributed activity of billions of neurons. Symbolic
models have not addressed this question.
In this paper, we explore an alternative explanation for the origin of cognitive flexibility that addresses
both of these questions: how neural mechanisms can support flexibility of cognitive control, and how this
can develop through experience. Our approach builds on the idea that a primary function of prefrontal cortex
(PFC) is to support rule-like representations that guide behavior (3). Unlike symbolic models, these rule-like
representations do not support arbitrary symbol binding operations, but rather are more like standard neural
network representations, in that they derive their meaning from specific connections between neurons in
PFC that support these representations and neurons in other parts of the brain responsible for carrying out
behavior. By virtue of these connections, sustained patterns of activity in PFC can modulate processing in
other parts of the brain, and thereby govern rule-like behavior. Flexibility of control derives from having a
suitably general set of representations in PFC, and the ability to activate these in task-appropriate contexts.
Extensive neurobiological and theoretical work supports this view. They indicate that PFC exhibits:
1. Robust maintenance of patterns of neural activity over time and against interference from distracting
inputs, so that currently relevant information can be actively held in working memory (4, 5, 6).
2. Adaptive gating of these activity patterns, allowing them to be rapidly and flexibly updated by dynamically switching between robust maintenance and rapid updating (7, 8, 9).
3. Extensive interconnectivity with other cortical areas (e.g., in posterior cortex) (4) allowing patterns of
activity in PFC to modulate processing in pathways responsible for task execution (10, 11).
Neural network models that simulate these PFC mechanisms can explain performance in a wide range
of tasks that rely on cognitive control (10, 12 13, 14), the ability to flexibly shift task sets (15, 16, 17, 14), as
well as behavioral deficits in such tasks associated with disturbances in PFC (18, 9, 14). A critical feature of
these models is the incorporation of rule-like representations in PFC that allow the PFC to exercise control
over processing, and to rapidly shift between tasks. This is consistent with accumulating neurobiological
data indicating that PFC neurons can represent the goals of a task, or the rules needed to perform it (3).
A fundamental limitation of existing models, however, is their use of prespecified representations in PFC,
that are designed “by hand” to perform the particular tasks of interest. Thus, like symbolic models, these
models do not address the question of how these representations develop, and acquire the properties needed
to support the flexible control of behavior. We propose that the development of representations in PFC relies
critically on an interaction between the functional specializations that are characteristic of PFC, as reviewed
above, and exposure to a broad range of tasks during learning. This is consistent with the protracted course
of development of cognitive control, and the human PFC itself (19).
In this article, we describe modeling results that support this proposal, by demonstrating that when a neural network with mechanisms specific to PFC is trained on several different tasks, it develops abstract, rulelike representations that support performance in novel task contexts. Networks without a PFC, or with more
limited training, fail to exhibit this form of flexibility. We present the results of two simulation experiments
a)
b)
No
Cross−Task Generalization
AG
pon
se
PF
(25 C Con
unit text
s)
A1 A2 A3 A4
E1 E2 E3 E4
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Left Stimulus
Cu
(16 e Hidd
unit en
s)
Hid
den
Tas
(16 k Hidd
unit en
s)
(83
uni
ts)
Res
C1 C2 C3 C4
B1 B2 B3 B4
E1 E2 E3 E4
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Right Stimulus
NF MF SF LF
Task
A
B
C
D
Dimension Cue
E
Generalization % Correct
E1 E2 E3 E4
D1 D2 D3 D4
75
Training Regimen
Task Pairs
All Tasks
50
25
r
c
e
l
f
rio +Re Sel SRN PFC Gat Ful
e
t
− o
s
P
P+
Po
RN N
No PFC S
PFC
Network Configuration
Figure 1: a) The model with the complete PFC system. Stimuli are presented in two possible locations (left, right).
Rows represent different dimensions, labeled A-E, and columns represent different features (1-4). Other inputs include
a task input indicating current task to perform (NF = name feature, MF = match features, SF = smaller feature, LF =
larger feature), and, for the explicit cue condition, a cue as to the currently relevant dimension. Output responses are
generated over the response layer. The AG unit is the adaptive gating unit, providing a temporal-differences (22) based
dynamic gating signal to the PFC context layer. To evaluate the features of this architecture, the following variants
were tested: Posterior: a single large hidden unit layer between inputs and response — a simple model of posterior
cortex without any special active maintenance abilities; P + Rec: posterior + full recurrent connectivity among hidden
units — allows hidden to maintain information over time via attractors; P + Self: posterior + self recurrent connections
from hidden units to themselves — allows individual units to maintain activations over time; SRN: simple recurrent
network, with a context layer that is a copy of the hidden layer on the prior step — widely used form of temporal
maintenance; SRN-PFC: an SRN context layer applied to the PFC layer in the full model (identical to the full PFC
model except for this difference) — tests for role of separated hidden layers; NoGate: the full PFC model without
the AG adaptive gating unit. b) Cross-task generalization results (% correct on test set) for the full PFC network
and a variety of control networks, with either only two tasks (Task Pairs) or all four tasks (All Tasks) used during
training. Overall, the full PFC model generalizes substantially better than the other models, and this interacts with the
level of training such that performance on the All Tasks condition is substantially better than the Task Pairs condition
(with no differences in numbers of training trials or training stimuli). With one feature left out of training for each of
4 dimensions, training represented only 31.6% (324) of the total possible stimulus inputs (1024); The roughly 85%
generalization performance on the remaining test items therefore represents good productive abilities.
using the model. The first tests whether the model’s mechanisms are sufficient to support generalization of
task performance to novel environments, and explores the relationship between this generalization ability
and the factors that lead to the development of rule-like task representations. The second tests whether these
learned PFC representations can account for human performance in tasks known to rely on PFC.
Simulation Study 1
Our model includes mechanisms for actively maintaining representations in a PFC layer, an adaptive
gating mechanism for updating these, and connections that allow activity patterns in PFC to influence processing in other layers of the model (we will refer to these other layers as “posterior cortex”, although in
reality this can include subcortical areas). Active maintenance in our model (Figure 1b) is subserved by
complementary mechanisms at the cellular and circuit levels, which allow individual units to be “latched,”
(i.e., robustly maintained via intrinsic bistability) as well as recurrent connectivity that supports the formation of stable attractor states (6, 20, 17). Adaptive gating is achieved through a mechanism based on phasic
dopamine signals, which allows units in PFC to be transiently responsive to input from posterior structures,
and to latch their state once the input has subsided. In the absence of dopamine signals, representations in
PFC are insensitive to exogenous input, and thereby resistant to interference. The concurrent effects of the
dopamine signal on reinforcement learning (21, 22) allow the system to associate appropriate stimuli with
the gating signal, and thus learn how and when to update representations in PFC (9, 8, 16, 17). Feedback
connections from the PFC to units in posterior areas in our model support the ability of the PFC to influence
processing in pathways responsible for task execution.
We trained a model having all of these mechanisms (the Full PFC model), as well as versions of it
lacking these mechanisms by varying degree, on either 2 or 4 different tasks designed to simulate simple
stimulus processing and active maintenance as demanded by standard laboratory tasks (e.g., the Stroop and
WCST). Each task involved processing multidimensional stimuli (e.g., varying along dimensions such as
size, shape, color, etc) in different ways, including naming a stimulus feature value in a given dimension
(e.g., “blue” for the color dimension), matching features of two stimuli in a given dimension, and ordinal
comparisons of stimulus features along a given dimension (23). All of these tasks required the network to
process stimulus features in one dimension while ignoring features in the other dimensions, maintain this
task set (i.e., knowledge of the relevant dimension) across sequences of trials, and flexibly update it when
the relevant dimension or the task itself changed. To measure generalization, we trained each network on a
subset of stimuli in each task, and then tested it on stimuli that it had not previously seen in that task. Thus,
the network had to generalize what it had learned about these stimuli across the different tasks. To determine
the contribution of the breadth of training experience, we manipulated the range of tasks using two training
regimens (pairs of tasks vs. the full set of four tasks)
Figure 1b shows the effects of network configuration and training regimen on generalization. First, note
that most networks failed to surpass 50% generalization, irrespective of training regimen. This included
networks with full recurrence (and therefore the capacity for active maintenance) throughout the network, as
well as networks with a PFC-like layer including recurrence but lacking an adaptive gating mechanism. Only
the network with a PFC layer that possessed both recurrent connectivity and an adaptive gating mechanism
exhibited substantial generalization, achieving 85% accuracy on stimuli for which it had no prior same-task
experience. This network produced only one-third as many errors on novel stimuli than any other network.
Interestingly, however, this was only the case for the Full PFC network trained in the all-tasks regimen. Even
though the network configuration was identical, training on pairs of tasks exhibited more than four times
as many generalization errors as training on all tasks. This indicates that breadth of experience interacted
with and was critical for exploiting the mechanisms present in the PFC. Note that the increased breadth of
experience in the all-tasks condition was strictly with respect to the range of task contexts in which stimuli
were experienced, and was not confounded by the total amount of training or range of stimulus inputs
presented during training (see (23) for details).
An important goal of this study was to examine the types of representations that develop in the PFC
layer of the network, and relate these to the capacity for generalization. Figure 2 shows examples of representations that developed in four different network configurations. In the full model trained on all tasks,
distinct representations for each stimulus dimension developed in the PFC layer. That is, each PFC unit
came to represent a single dimension, and to represent all features in that dimension. More precisely, these
representations collectively formed a basis set of orthogonal vectors that spanned the space of task-relevant
stimuli, and that were aligned with the dimensions along which features had to be distinguished for task
performance. This effect was only partially apparent in the configuration having a PFC but lacking an adaptive gating mechanism, as well as the full PFC model trained only on task pairs, and was essentially absent
from the model entirely lacking a PFC. Furthermore, the degree to which representations developed that
were dimensional in character was closely related to the network’s generalization performance, as shown
in Figure 2e. A measure of the orthogonality of the PFC representations and their alignment with feature
a) Posterior – All Tasks
b) No Gate – All Tasks
c) Full PFC – Task Pairs
d) Full PFC – All Tasks
e)
f)
texture
position
shape
size
color
Generalization % Correct
Rule−Likeness Predicts Generalization
FullPFC−All
75
r=.97
NoGate−All
50
Posterior−All
25
FullPFC−Pair
0.25
0.50
0.75
Rule Representation Measure
Figure 2: Representations that developed in four different network configurations: a) Posterior cortex (no PFC)
trained on all tasks; b) PFC without the adaptive gating mechanism (all tasks); c) Full PFC trained only on task pairs
(NF & MF in this case); and d) Full PFC (all tasks). Each panel shows the weights from the hidden units (a) or PFC
(b–d) to the response layer. Larger squares correspond to units (all 30 in the PFC, and a random and representative
subset of 30 from the 145 hidden units in the posterior model), and the smaller squares designate the strength of the
connection (lighter = stronger) from that unit to each of the units in the response layer. Note that each row designates
connections to response units representing features in the same stimulus dimension (as illustrated in e) and Figure 1).
It is evident, therefore, that each of the PFC units in the full model (d) represents a single dimension and, conversely,
that each dimension is represented by a distinct subset of PFC units. This pattern is less evident in the model lacking an
adaptive gating mechanism (b), and in the PFC model trained only on task pairs (c), and is almost entirely absent in the
posterior model (a) in which the hidden units appear to encode arbitrary combinations of features across dimensions.
Panel (f) shows the correlation of generalization performance in these cases with the extent to which the units distinctly
and orthogonally encode stimulus dimensions (the rule representation measure, described in (24)).
dimensions was highly correlated with generalization performance across network configurations (r=0.97).
(24) These findings suggest that the model’s ability to generalize task performance to new stimuli is closely
related to the extent to which it developed representations that abstractly and discretely encoded the relevant
stimulus dimensions needed to perform the task (if the tasks emphasized different distinctions, the system
would learn different dimensional representations aligned with these distinctions). These results agree with
those of recent neurophysiological studies, in which PFC neurons have been identified that seem to encode,
in similarly explicit form, the rules that govern the current task (3).
Examining the model, and the conditions under which it was trained, provides insights into why these
rule-like representations developed and were related to generalization performance. During training, when
the network produced an incorrect response, the adaptive gating mechanism temporarily inhibited PFC units
that were active during that response, favoring activation of other units on the next trial. This implemented a
simple form of search (random sampling with delayed replacement). The influence of this search mechanism
interacted with the requirement that, for each task, the network had to attend to features in one dimension and
ignore the others (the “rule” for the task). Together, these put pressure on the network to commit individual
units to a single dimension. If a unit represented several dimensions, then even if one of these was the
correct one, nevertheless that unit would often be inhibited because the other dimensions it represented were
incorrect (and supported spurious responses). As a result of this inhibition, connection weights supporting
the activation of these units were weakened. In contrast, units that represented a single dimension were
allowed to remain fully active as long as that was the correct dimension, and thus their connection weights
were strengthened. As a consequence, the network developed PFC representations in which each unit was
committed to a single dimension.
Complementing this pressure to represent a single dimension, training also put pressure on each PFC
unit to represent all of the features in a given dimension, constituting an abstract representation of that
dimension. This is because within a block of trials the relevant dimension remained the same, while the
specific features within that dimension to which the network had to respond varied. This interacted with the
capacity of the PFC to actively maintain dimension representations. Networks with this capacity learned
that it was more efficient to use a single pattern of activity to represent an entire dimension, and maintain
this pattern of activity across a block of trials, than to switch the pattern activity for each new stimulus (as
the specific features changed). Interestingly, this effect was accentuated by training over all tasks versus just
pairs of tasks: The orthogonality and dimensional alignment of PFC representations was greater for the all
task regimen than the two task regimen, paralleling the effects on generalization. Note that for both training
regimens non-identical subsets of features were used for the different tasks, so that there was no direct
pressure on the network to develop the same dimensional representations for both (or all) tasks. In principle,
the network could have developed dimensional representations dedicated to each task, that spanned only
Normalized Epoch Wt Diff
Relative Stability of PFC & Hidden
1.00
0.75
PFC
Hidden
0.50
0.25
0.00
0 10 20 30 40 50 60 70 80 90 100
Epochs
Figure 3: Relative stability of PFC and hidden layer (posterior cortex) in the model, as indexed by euclidean distance
between weight states at the end of subsequent epochs (one epoch is 2,000 trials). The PFC takes longer to stabilize
(i.e., exhibits greater levels of weight change across epochs) than the posterior cortex. For PFC, within-PFC recurrent
weights were used. For Hidden, weights from stimulus input to Hidden were used. Both sets of weights are an
equivalent distance from error signals at the output layer. The learning rate is reduced at 10 epochs, producing a blip
at that point.
those features needed to perform that task. However, with an increased number of tasks, the overlap among
different subsets of features across tasks, and the constancy of dimensions within task blocks, combined to
support the development of representations that spanned full dimensions across all tasks. This then led to
greater generalization across tasks.
In summary, the development within the PFC layer of an orthogonal basis set corresponding to the taskrelevant dimensions made it possible for the network to identify the appropriate dimension for a given task,
maintain this while needed, and then switch to another one when called upon to do so. These functional
characteristics correspond well to those of rule-governed behavior, and the role ascribed to PFC in such
behaviors. Task rules are typically well-defined and stable over extended periods of time. Thus, the model
illustrates how the mechanisms of active maintenance and adaptive gating, together with appropriate training, can give rise to representations sufficient to support rule-governed behavior. An important feature of
our results is that this ability depended critically upon an interaction of processing mechanisms with an
appropriate range of task experience. We suggest that this accords well with the protracted developmental
course in humans of the PFC and associated dopaminergic systems (extending into late adolescence), allowing a broad range of experience to shape PFC representations (19). Accordingly, we hypothesized that PFC
representations in our model should stabilize later in development (training) than those in posterior areas.
We tested this by measuring the average magnitude of weight changes from projections into the main hidden
layer and in the PFC layer. The hidden layer stabilized within 20 epochs (2,000 trials per epoch), while the
PFC did not stabilize until 70 epochs (Figure 3).
Simulation Study 2
One important question that might be asked is whether the rule-like PFC representations learned by our
model will produce the appropriate patterns of performance in tasks specifically associated with prefrontal
function. To address this question, we conducted simulations of the Stroop task and the Wisconsin Card
Sort Task (WCST), two tasks that have been used widely as benchmarks of prefrontal function (25, 26).
In the Stroop task (27), participants are presented with color words printed in various colors, and are
asked to either read the word or name the color in which it is printed. A universal finding is that, in adults,
color naming is slower than word reading, and that an incongruent word (e.g., “green” displayed in red)
interferes with color naming (saying “red”) while word reading is relatively unaffected by the color of the
display (28). A common interpretation of these findings is that word reading is more automatic (“overlearned”) than color naming, and therefore that color naming demands more cognitive control, especially
when the word conflicts with the response. This demand for cognitive control has been linked directly to
prefrontal cortex function (26). To simulate Stroop task performance, one of the stimulus dimensions was
trained less than the other four dimensions, with all other factors unchanged from the first study. As in previous models of the Stroop task (10), this simulated the asymmetries of experience human participants have
had with color naming versus word reading. Figure 4a compares the model to human performance (data
from 29) in the principal conditions of the Stroop task. The model captures not only the slower responses
for color naming versus word reading, but also the asymmetry of interference effects. These results replicate those of previous modeling work, which have shown that the effects can be explained in terms of an
interaction between PFC representations of the dimensions that define each task (colors vs. words) and the
relative strength of the posterior pathways that map the stimuli onto their corresponding responses (30, 10,
13). Thus, our findings suggest that the present model has discovered representations that function similarly
to those that have been manually imposed in previous models of the role of the PFC in performance on the
Stroop task (10, 13).
The Stroop task illustrates the capacity for cognitive control (i.e., to respond to one stimulus dimension
a)
Reaction Time (msec)
Stroop Task
900
People: Color
People: Word
Model: Lo Freq
Model: Hi Freq
800
700
600
500
Neutral
Conflict
Perseverations/Errors
b)
WCST Task
30
Perseverations
Other Errors
20
10
0
0
25
50
75
PFC Damage (%)
Figure 4: a) Performance of the full PFC network on a simulated Stroop task, demonstrating the classic pattern of
conflict effects on the subordinate task of color naming with unaffected performance on the dominant word reading
task (human data from 29). This was simulated by training one dimension (A) with 1/3 the frequency of the others,
making it weaker. Dimension B was used for the strong (word reading) pathway. To simulate the neutral condition,
inputs were presented with a single stimulus feature active in either the weak or strong dimension, and reaction time
was measured as the number of cycles to name this feature in the response layer with activation > .75. In the
conflict condition, features in both dimensions were activated, and the task input specified which dimension was to be
named. b) Performance of the full PFC network with various levels of damage (units removed) in a simulated WCST
task, demonstrating the classic pattern of increasing perseveration with increased damage. We simplified the task by
measuring for each input “card” the model’s feature naming response, without giving it any cue input specifying the
currently-relevant dimension. Therefore, the model had to discover this dimension based solely on error-feedback
provided to the AG unit (if it responded correctly a reward value of 1 was delivered to this unit; otherwise, a 0
was delivered). This AG error feedback would destabilize the PFC representation until it represented the currently
relevant dimension. After 8 correct trials in a row, the dimension was switched. Performance is plotted in terms of the
number of sequential productions of feature names corresponding to the previously-relevant dimension after a switch
(perseverations). Clearly, the simulated PFC is critical for rapid, flexible switching.
and override responses associated with a more compelling one), but what about the flexibility of control
(i.e., the ability to switch rapidly from processing one dimension to another)? This is illustrated by the
WCST, a task that has long been recognized as being sensitive to frontal lobe damage. In the WCST task
(31), participants are provided with a deck of cards bearing multidimensional stimuli that vary in shape,
size, color, and number. These must be sorted, one by one, according to a particular dimension (the “rule”).
Critically, the participant is not told the rule, and so must discover this from feedback provided about whether
or not each card was sorted correctly. The rule is held constant until the participant makes a criterion number
of correct responses in sequence (e.g, 8), at which point the rule is switched (again without informing the
participant) and the procedure repeated. Neurologically intact individuals rapidly discover and switch to
the new sorting dimension. Patients with frontal damage typically are able to discover the first rule without
difficulty. However, once it switches, they perseverate in sorting according to the previous rule. This has led
many authors to conclude that PFC plays a critical role in the cognitive flexibility required to switch “mental
set” from one rule to another (32).
The WCST is analogous to the form of the feature naming task learned by our model where dimension
cues are not presented to the network: The model is presented with a stimulus and must identify its feature
in the relevant dimension (i.e., discover the appropriate rule). We used this task to compare the performance
of the full model under varying degrees of degradation to the PFC layer. Figure 4b shows that such damage
produces a disproportionate increase in perseverative responding relative to other types of errors. This is
consistent with findings from previous modeling studies, which have accounted for performance deficits in
the WCST in terms of a degradation of rule-like representations in PFC (14, 16, 17). Again, we emphasize
that our model learned the relevant representations in PFC through learning mechanisms, instead of having
them manually imposed as in most of these existing models.
Discussion
The findings reported here support the conjecture that the contribution made by PFC to cognitive control
rests on a critical combination of anatomic and functional specializations coupled with sufficiently broad
learning experiences. When these conditions are satisfied, a “vocabulary” of rule-like representations develop within the PFC that contribute to the flexibility of cognitive control. Our simulations shed light on
several issues that are central in the effort to understand the neural mechanisms underlying cognitive control. First, they illustrate that breadth of training experience has a critical influence on the development of
representations in PFC. This may be one reason why development of the PFC and associated dopaminergic systems is so protracted relative to other brain areas, extending into late adolescence. Our examination
of the developmental trajectory of representations in the model suggested a corollary reason: It is necessary for representations in posterior systems to stabilize before the PFC can extract the dimensions of these
representations relevant to task performance.
Although we found that abstract, rule-like PFC representations supported good generalization in the fully
regular domains that we explored here, we do not claim that these representations are universally beneficial.
In particular, it is unlikely that such discrete, abstract representations are as useful in domains characterized
by more graded knowledge structures. Here distributed representations may perform better (for example, in
the coding and retrieval of semantic knowledge in posterior cortical structures). Thus, there may be a tradeoff
between PFC and posterior cortical types of representations, in which each is better suited for a different
kind of domain. This is consistent with data showing that posterior cortex may be better at learning complex,
similarity-based categories, whereas PFC can more quickly acquire simple rule-based categories (33). More
work is needed to explore these potential tradeoffs, for example in the richness of complex domains such
as language, wherein our model may provide a productive middle ground between the neural network and
symbolic modeling perspectives in the long-standing “rules and regularities in language processing” debates
(e.g., (34).
The model illustrates another critical factor that contributes to flexibility of control: The use of patterns
of activity rather than changes in synaptic weights as a means of exerting control over processing (35). As
with learning in all neural network models, PFC representations developed through synaptic modification.
However, once these were learned, then adaptive behavior in novel circumstances was mediated by a search
for the appropriate pattern of activity, rather than the need to learn a new set of connection strengths. This
may clarify the mechanisms underlying the adaptive coding hypothesis (36), which holds that PFC dynamically reconfigures itself for the task at hand. Whereas this could be construed as involving rapid learning
processes, our theory instead suggests that it depends on flexible updating of activity patterns in PFC. Similarly, this activation-based processing differs fundamentally from the kinds of arbitrary symbol binding
mechanisms present in traditional symbolic models, where the meaning of the underlying representations
can be arbitrarily bound to novel inputs to achieve flexible performance.
Our primary focus in this work has been on one of the factors that contributes to the flexibility of cognitive
control: the ability to generalize task performance to novel environments, and to switch task representations
dynamically in response to changing demands (as in the WCST task). However, there is another equally
important factor in the flexibility of control: the ability to generate entirely new behaviors. Our model does
not directly address this capacity for generativity. While the model was tested on its ability to perform tasks
on new combinations of features, it was never asked to perform entirely new tasks. One might contend
that the same is true for humans — the flexibility of cognitive control has its limits. People cannot actually
perform entirely new tasks without training, at least not with any alacrity (e.g., one cannot play complex
card games such as bridge very well when first learning them). Nevertheless, people can recombine familiar
behaviors in novel ways. Such recombination, when built upon a rich vocabulary of primitives, may support
a broad range of behavior and thus contribute to generativity. Indeed, it is this ability to flexibly recombine
learned behaviors that may best distinguish humans from other species, and account for characteristically
human abilities such as planning and problem solving.
We believe that the PFC representations learned by our model provide a rudimentary example of the type
of task or rule “vocabulary” that can support recombination, and thus constitute an important step toward
understanding the neural mechanisms underlying generativity. In particular, the discrete, orthogonal nature
of these representations should facilitate the formation of novel combinations. As a result, the mechanisms
and processes that we have identified as critical to generalization may also set the stage for understanding
generativity. Generativity also highlights the centrality of search processes to find and activate the appropriate combination of representations for a given task, a point that has long been recognized by symbolic
models of cognition (37). Such models have explored a range of search algorithms and heuristics that vary
in complexity and power (e.g., grid search, hill-climbing, means-ends, etc.). Our model implemented a relatively simple form of search: random sampling with delayed replacement. This appeared to be sufficient
for performance of the simple tasks addressed, consistent with previous results (14, 16). An important focus
of future research will be to identify neural mechanisms that implement more sophisticated forms of search.
This is likely to be necessary to account for human performance in more complex problem solving domains,
in which the PFC plays a critical role (38).
Finally, it should be clear that our model of PFC function relies heavily on the nature of its representations, which contrasts with other approaches that focus instead on processes (e.g., manipulation, inhibition,
planning). Although it is often difficult to clearly distinguish these approaches, it has been argued that
representational accounts provide a better fit with the body of existing data regarding how the brain stores
and processes information (39). We concur with this view, and add that neural network models provide a
particularly clear mechanistic perspective on these issues. Nevertheless, there may be important differences
in the types of representations developed in our model compared to others (e.g., 39), that we hope to explore
in future modeling and empirical work.
In conclusion, we have argued that the capacity of the PFC to actively maintain rule-like task representations, update these as circumstances demand, generalize them to novel environments, and recombine them
in novel ways, are critical factors in the flexibility of cognitive control. However, it is important to recognize
that while these factors may be necessary to understand the richness of human cognitive function, they are
certainly not sufficient. No doubt, an understanding of how these mechanisms interact with other systems
(such as those supporting episodic memory, language function, and affect) will be equally important in developing a full understanding of how cognitive control is implemented in the brain. We hope that the work
presented here contributes to this understanding, and will help foster more detailed studies of this fascinating
and important domain.
*
References and Notes
1. S. Monsell. Control of mental processes. In V. Bruce, editor, Unsolved mysteries of the mind: Tutorial
essays in cognition, pages 93–148. Psychology press, Hove, UK, (1996).
2. J. R. Anderson and C. Lebiere. The Atomic Components of Thought. Erlbaum, Mahwah, NJ, (1998).
A. Newell. Unified Theories of Cognition. Harvard University Press, Cambridge, MA, (1990).
3. I. M. White and S. P. Wise, Experimental Brain Research, 126, 315, (1999). D. Freedman, M. Riesenhuber, T. Poggio, and E. Miller, Science, 291, 312–316, (2001). J. D. Wallis, K. C. Anderson, and E. K.
Miller, Nature, 411, 953–956, (2001). K. Sakai and R. E. Passingham, Nature Neuroscience, 6, 75–81,
(2003). M. Watanabe, Experimental Brain Research, 89, 233–247, (1992).
4. J. M. Fuster. The Prefrontal Cortex: Anatomy, Physiology and Neuropsychology of the Frontal Lobe,
3rd Edition. Lippincott-Raven, New York, (1997).
5. K. Kubota and H. Niki, Journal of Neurophysiology, 34, 337–347, (1971). P. S. Goldman-Rakic,
Handbook of Physiology — The Nervous System, 5, 373–417, (1987). E. K. Miller, C. A. Erickson, and
R. Desimone, Journal of Neuroscience, 16, 5154, (1996).
6. J. M. Fellous, X. J. Wang, and J. E. Lisman, Nature Neuroscience, 1, 273–275, (1998). X.-J. Wang,
Journal of Neuroscience, 19, 9587, (1999). D. Durstewitz, J. K. Seamans, and T. J. Sejnowski, Journal
of Neurophysiology, 83, 1733, (2000).
7. J. D. Cohen, T. S. Braver, and R. C. O’Reilly, Philosophical Transactions of the Royal Society (London)
B, 351, 1515–1527, (1996). S. Hochreiter and J. Schmidhuber, Neural Computation, 9, 1735–1780,
(1997).
8. R. C. O’Reilly, T. S. Braver, and J. D. Cohen. A biologically based computational model of working
memory.
In A. Miyake and P. Shah, editors, Models of Working Memory: Mechanisms of Active
Maintenance and Executive Control., pages 375–411. Cambridge University Press, New York, (1999).
9. T. S. Braver and J. D. Cohen. On the control of control: The role of dopamine in regulating prefrontal
function and working memory. In S. Monsell and J. Driver, editors, Control of Cognitive Processes:
Attention and Performance XVIII, pages 713–737. MIT Press, Cambridge, MA, (2000).
10. J. D. Cohen, K. Dunbar, and J. L. McClelland, Psychological Review, 97, 332–361, (1990).
11. E. K. Miller and J. D. Cohen, Annual Review of Neuroscience, 24, 167–202, (2001).
12. J. D. Cohen, D. Servan-Schreiber, and J. L. McClelland, American Journal of Psychology, 105, 239–
269, (1992)., S. Dehaene and J. P. Changeux, Journal of Cognitive Neuroscience, 1, 244–261, (1989).
13. R. C. O’Reilly and Y. Munakata. Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain. MIT Press, Cambridge, MA, (2000).
14. S. Dehaene and J. P. Changeux, Cerebral Cortex, 1, 62–79, (1991).
15. S. J. Gilbert and T. Shallice, Cognitive Psychology, 44, 297–337, (2002).
16. R. C. O’Reilly, D. Noelle, T. S. Braver, and J. D. Cohen, Cerebral Cortex, 12, 246–257, (2002).
17. N. P. Rougier and R. C. O‘Reilly, Cognitive Science, 26, 503–520, (2002).
18. J. D. Cohen and D. Servan-Schreiber, Psychological Review, 99, 45–77, (1992). T. S. Braver and J. D.
Cohen, Cognitive Processing, 2, 25–55, (2001). E. Ruppin, Network : Computation in Neural Systems,
6, 635, (1995).
19. A. Diamond and P. S. Goldman-Rakic, Society for Neuroscience Abstracts, 12, 742, (1986). P. R.
Huttenlocher, Neuropsychologia, 28, 517–527, (1990). D. A. Lewis, Neuropsychopharmacology, 16,
385–398, (1997). B. J. Casey, S. Durston, and J. A. Fossella, Clinical Neuroscience Research, 1,
267–282, (2001). J. B. Morton and Y. Munakata, Developmental Science, 5, 435–440, (2002).
20. M. J. Frank, B. Loughry, and R. C. O’Reilly, Cognitive, Affective, and Behavioral Neuroscience, 1,
137–160, (2001). D. Zipser, Neural Computation, 3, 179–193, (1991).
21. P. R. Montague, P. Dayan, and T. J. Sejnowski, Journal of Neuroscience, 16, 1936–1947, (1996).
22. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA,
(1998).
23. See online supplemental materials for complete details of the network’s architecture and training and
testing procedures. See Figure 1b for a description of the network architecture. Each of the four
tasks required the network to respond to feature values in one dimension and ignore the other four
dimensions. In the naming task (NF), the network was provided with a single stimulus, and required
to identify (“name”) the value of the feature in the selected dimension. In the matching task (MF),
the network was presented with two stimuli and, if they shared the same feature value in the selected
dimension, it had to identify this value, otherwise respond “no.” Finally, two magnitude comparison
tasks required the network to compare the feature values of the two stimuli in the selected dimension
and, in one case identify the larger value (LF) and in the other the smaller value (SF) (with value
arbitrarily assigned according to increasing feature number from left to right across the dimensional
row). Each task was presented in an instructed and an uninstructed form. In the instructed form,
the input units corresponding to both the current task and the relevant dimension were activated. In
the uninstructed form, information about the task but not the relevant dimension was provided to the
network. In this case, the relevant dimension remained the same for sequences of trials, and the network
had to use the feedback it received after each trial to discover the relevant dimension. Note that this
procedure required the network, once it discovered the relevant dimension, to “remember” this for
subsequent trials in the sequence, until the dimension changed, at which point it had to discover the
next dimension.
For a given task the model saw only a subset of the feature values along each dimension, and a relatively
small fraction (about 30%) of all possible stimuli (i.e., combinations of features across dimensions).
A given training run consisted of 100 epochs of 2,000 trials per epoch; it took the networks only
roughly 10 epochs to achieve near-perfect performance on the training items, but we measured crosstask generalization performance every 5 epochs throughout the duration to find the best generalization
for each network, unconfounded by any differences in architecture or in the raw amount of exposure
to features across different training scenarios. Generalization testing measured the network’s ability to
respond to stimuli it had not seen in that task. To evaluate how the range of task experience influenced
performance, the model was trained with two regimens: one involving all possible pairs of tasks, and
the other involving all four tasks.
Finally, we trained and tested different network configurations, in order to test the contribution made
by constituent mechanisms to learning and performance. All network configurations had the same total
number of processing units, in order to control for the effects of overall computing resources. The only
differences among configurations were the patterns of connectivity and the presence or absence of the
adaptive gating mechanism. The various configurations are described in Figure 1. These ranged from
a simple feedforward network with 145 hidden units (equaling the number of hidden plus PFC units in
the full PFC model) to the complete model including full recurrent connectivity within the PFC and an
adaptive gating mechanism.
24. The rule-like representation measure was computed by comparing each unit’s pattern of weights to
the set of 5 orthogonal, complete dimensional target patterns (i.e., the A dimension target pattern has
a 1 for each A feature, and 0’s for the features in all other dimensions, etc.). A numeric value between
0 and 1, where 1 represents a completely orthogonal and complete dimensional representation was
P k |wi ·tk | where tk is the dimensional target pattern k and w i is the
computed for unit i as: di = max
|w ·t |
k
i
k
weight vector for unit i, and |wi · tk | represents the normalized dot product of the two vectors (i.e., the
cosine). This value was then averaged across all units in the layer (PFC or Hidden, as shown in Figure
2) and then correlated with that network’s generalization performance.
25. D. R. Weinberger, K. F. Berman, and D. G. Daniel. Prefrontal cortex dysfunction in schizophrenia.
In H. S. Levin, H. M. Eisenberg, and A. L. Benton, editors, Frontal Lobe Function and Dysfunction,
pages 276–285. Oxford University Press, New York, (1991). P. Vendrell, C. Junque, and J. Grafman,
Neuropsychologia, 33, 341, (1995). D. T. Stuss, B. Levine, M. P. Alexander, J. Hong, C. Palumbo,
L. Hamer, K. J. Murphy, and D. Izukawa, Neuropsychologia, 38, 388–402, (2000). B. Milner, Archives
of Neurology, 9, 90–100, (1963).
26. A. W. MacDonald III, J. D. Cohen, and C. S. Carter, Science, 288, 1835, (2000). M. T. Banich,
M. P. Milham, R. Atchley, N. J. Cohen, A. Webb, T. Wszalek, A. F. Kramer, Z. P. Liang, A. Wright,
J. Shenker, and R. Magin, Journal of Cognitive Neuroscience, 12, 988–1000, (2000).
27. J. R. Stroop, Journal of Experimental Psychology, 18, 643–662, (1935).
28. C. M. MacLeod, Psychological Bulletin, 109, 163–203, (1991).
29. K. Dunbar and C. M. MacLeod, Journal of Experimental Psychology: Human Perception and Performance, 10, 622–639, (1984).
30. R. H. Phaf, A. Heijden, and P. T. Hudson, Cognitive Psychology, 22, 273–341, (1990).
31. D. A. Grant and E. A. Berg, Journal of Experimental Psychology, 38, 404–411, (1948).
32. T. Shallice. From Neuropsychology to Mental Structure. Cambridge University Press, New York,
(1988). J. Duncan, H. Emslie, P. Williams, R. Johnson, and C. Freer, Cognitive Psychology, 30, 257–
303, (1996).
33. F. G. Ashby, L. A. Alfonso-Reese, and E. M. Waldron, Psychological Review, 105, 442, (1998). E. E.
Smith, A. L. Patalano, and J. Jonides, Cognition, 65, 167–196, (1998).
34. J. L. McClelland and K. Patterson, Trends in Cognitive Sciences, 6, 465–472, (2002). S. Pinker and
M. T. Ullman, Trends in Cognitive Sciences, 6, 456–463, (2002).
35. R. C. O’Reilly and Y. Munakata. Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain. MIT Press, Cambridge, MA, (2000), pp 380-381. Y. Munakata,
Developmental Science, 1, 161–184, (1998).
36. J. Duncan, Nature Reviews Neuroscience, 2, 820–829, (2001).
37. A. Newell and H. A. Simon. Human Problem Solving. Prentice-Hall, Englewood Cliffs, NJ, (1972).
38. S. C. Baker, R. D. Rogers, A. M. Owen, C. D. Frith, R. J. Dolan, R. S. J. Frackowiak, and T. W.
Robbins, Neuropsychologia, 34, 515, (1996).
39. J. N. Wood and J. Grafman, Nature Reviews Neuroscience, 4, 139–147, (2003).
Prefrontal Cortex and the Flexibility of Cognitive Control:
Rules Without Symbols
Supplemental Material
Nicolas P. Rougier
David C. Noelle
Todd S. Braver
Dept of Psychology, CU Boulder
Department of Computer Science
Department of Psychology
INRIA Lorraine, France
Vanderbilt University
Washington University
Jonathan D. Cohen†
Randall C. O’Reilly†
Department of Psychology
Department of Psychology
Princeton University
University of Colorado Boulder
Abstract:
.
† corresponding authors: [email protected], 345 UCB,
Boulder, CO 80309-0345 or [email protected], Green Hall, Princeton,
NJ 08544. Supported by ONR grants N00014-00-1-0246 and N0001403-1-0428, and NIH grants MH64445. Last authorship reflects equal
contribution; order was determined by a flip of coin. We thank Tim
Curran, Michael Frank, Tom Hazy, Dave Jilk, Ken Norman, Yuko Munakata and members of the CCN lab for helpful comments.
No
E1 E2 E3 E4
AG
Res
pon
se
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
E1 E2 E3 E4
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Left Stimulus
Cu
(16 e Hidd
unit en
s)
Hid
den
Tas
(16 k Hidd
unit en
s)
(83
unit
s)
PF
(30 C Con
unit text
s)
A1 A2 A3 A4
E1 E2 E3 E4
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Right Stimulus
NF MF SF LF
Task
A
B
C
D
E
Dimension Cue
Figure 1: Full PFC model (as was presented in Figure 1 of the main text). Blue represents input/output layers (two Stimulus
locations, Task and Dimension Cue inputs, and Response output), green represents hidden layers (Task & Cue Hidden, each 16
units, and main Hidden layer with 83 units), and red represents PFC active maintenance layers and related units (i.e., the AG
adaptive gating unit).
This supplementary material provides a detailed explication of the model architectures, equations, and training/testing procedures used. A number of different variations of model architectures and training/testing procedures
were used to determine the basis for the results reported in the main paper.
Model Architectures
The architecture of the full PFC model is shown in Figure 1 of the main paper, and reproduced here (Figure 1). All
models contained the same input/output layers (two Stimulus locations, Task and Dimension Cue inputs, and Response
output). The full model has three hidden layers (Task & Cue Hidden, each 16 units, and main Hidden layer with 83
units) which serve to encode the relevant stimuli and map them onto the appropriate responses. These are thought to
correspond to association areas in posterior cortex. The Task and Cue hidden layers enable the network to develop
internal representations of these inputs that are influenced by the other layers of the network, in particular the PFC.
For example, the PFC representation of a stimulus dimension can activate the Cue hidden layer with a corresponding
representation during the uninstructed task (when there is no dimension cue input), and this helps to support the
network in focusing on the appropriate dimension, through weights that were trained on the instructed cases when a
dimension cue is present. We think of this as representing the process of subvocalization (inner speech), where people
generate internal verbal representations to support task performance (1). As described in the main paper, the PFC and
associated adaptive gating unit (AG) support robust active maintenance that is also capable of rapid updating. Note that
the PFC layer (30 units) had full self connectivity (each unit connected to all other units in the layer), which supported
maintenance in conjunction with a bistable intracellular “latching” mechanism that is described in detail later.
We tested a number of variations on this basic architecture to determine the importance of different features. In all
No
E1 E2 E3 E4
Res
pon
se
D1 D2 D3 D4
C1 C2 C3 C4
Context (30 units)
B1 B2 B3 B4
Context at (t−1)
(30 units)
E1 E2 E3 E4
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Left Stimulus
Cu
(16 e Hidd
unit en
s)
Hid
den
Tas
(16 k Hidd
unit en
s)
(83
unit
s)
A1 A2 A3 A4
E1 E2 E3 E4
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Right Stimulus
NF MF SF LF
Task
A
B
C
D
E
Dimension Cue
Figure 2: SRN-PFC model: PFC layer has associated context layer that is updated as a simple recurrent network (SRN). The
context is copied from the PFC layer after each trial of processing, providing a maintained activity pattern representing the prior
trial of processing on the next trial.
cases, the total number of internal (hidden + PFC) units was the same, to rule out changes in overall representational
power as a confounding factor. In all cases, we found that these alternative architectures generalized significantly
worse than the full PFC model (Figure 1b in the main paper). This is despite the fact that all network architectures
were capable of learning the training patterns to a similarly high level of performance.
PFC No Gate
This variation was identical to the full PFC model, except it lacked the AG adaptive gating unit. The PFC layer
was still capable of maintenance through its recurrent excitatory connections, but this maintenance was not subject to
adaptive gating signals in response to differences between predicted and actual rewards (as computed by the AG unit).
SRN-PFC
The simple recurrent network (SRN) architecture (2) is perhaps the most widely used mechanism in cognitive
models for simulating the maintenance of contextual information for temporally extended tasks. The SRN uses a
context layer that is updated from an associated hidden layer in an alternating, out-of-phase manner. That is, after a
trial of processing, the hidden layer activations are copied to the context layer, which provides input to the hidden layer
on the next trial. In this way, processing on the next trial can be contextualized by information present on the previous
trial. We have noted that this regular context updating represents a simple form of a gating mechanism, where gating
occurs on every trial (3). To determine how effective this form of gating is compared to the adaptive gating provided
by the AG unit, the SRN-PFC model provides an SRN context layer for the PFC layer (Figure 2).
No
E1 E2 E3 E4
R es
pon
se
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
Co
nte
xt
(14
5u
nits
)
Hid
den
(14
5u
nits
)
A1 A2 A3 A4
E1 E2 E3 E4
E1 E2 E3 E4
D1 D2 D3 D4
D1 D2 D3 D4
C1 C2 C3 C4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Left Stimulus
B1 B2 B3 B4
NF MF SF LF
A1 A2 A3 A4
Right Stimulus
A
Task
B
C
D
E
Dimension Cue
Figure 3: SRN (No PFC) model: A more standard type of SRN network without the PFC or other hidden layers. The SRN Context
layer is a copy of a single Hidden processing layer that has the same number of total units as those in all the Hidden and PFC layers
of the full PFC model.
No
E1 E2 E3 E4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Hid
den
(14
5u
nits
)
Res
pon
se
D1 D2 D3 D4
E1 E2 E3 E4
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Left Stimulus
E1 E2 E3 E4
D1 D2 D3 D4
C1 C2 C3 C4
B1 B2 B3 B4
A1 A2 A3 A4
Right Stimulus
NF MF SF LF
Task
A
B
C
D
E
Dimension Cue
Figure 4: Simple Posterior-Cortex model, with a single Hidden layer having no specialized active maintenance abilities mediating
the input-output mapping. Variations of this model included some maintenance abilities through either self (each unit connected to
itself) or full (each unit connected to all other units) recurrent connections.
SRN (No PFC)
The SRN architecture is usually implemented where the SRN context layer copies a single hidden layer. In many
respects, the SRN context layer can be considered as a simple version of the PFC layer in the original full PFC model.
Therefore, we tested a more standard SRN architecture with a single Hidden layer having the same number of units
present in all the hidden and PFC layers in the original full PFC model (145 units), and an SRN context layer that
copies this Hidden layer (Figure 3).
Posterior-Cortex Model and Variations
The final set of architectures consisted of a single large (145 units) hidden layer, representing posterior cortical
areas, mediating the input-output mapping. This represents a standard three-layer architecture as commonly used in
cognitive modeling. In its most basic form, the hidden layer has no capability of maintenance whatsoever. We also
explored two variations having either self recurrent connectivity (each hidden unit connected to itself) or full recurrent
Feature level 1 2 3 4
Dimension
E
D
C
B
A
Figure 5: Stimulus representation for multi-dimensional stimuli. There were five dimensions, which are thought to correspond to
things like size, color, shape, etc. Each dimension had four feature levels (e.g., small, medium, large, XL for the size dimension). For
the naming feature (NF) task, the network would output the name of the feature along one of the five dimensions (e.g., responding
2 (medium) for the A (size) dimension).
connectivity (each hidden unit connected to all other hidden units).
Given that generalization is of primary concern in these networks, and that in some cases larger networks have been
shown to generalize worse than smaller ones (i.e., due to overfitting from too many degrees of freedom), one might
want to also see results from networks with smaller hidden layers. These networks were run and performed worse than
those with larger hidden layers. This is consistent with the fact that the present task is completely deterministic and
has no opportunities for overfitting to represent noise. In such conditions, larger networks are better because the larger
sample of hidden unit activities reduces the chances that the idiosyncrasies of individual units dominate the overall
responding (i.e., a law of large numbers effect). See (4) for further discussion, results, and analysis.
Training and Testing
All models were trained on four types of tasks, with each task type having instructed and uninstructed versions, for
a total of eight different task conditions. Each task involved performing a simple operation on one or two multi-featured
stimulus inputs (Figure 5), such as naming one stimulus feature (NF), matching features across two stimuli (MF),
and comparing the relative magnitude of features across two stimuli (smaller, SF or larger, LF). Each stimulus was
composed by selecting one of four feature values along five different dimensions, which we think of as corresponding
to simple perceptual dimensions such as number, size, color, shape, and texture (e.g., as in the cards used in the WCST
task). For convenience, the five dimensions were abstractly labeled with the letters A-E, and the features numbered 1-4
(e.g., A2 refers to the 2nd feature in dimension A). No attempt was made to impose any real-world semantic constraints
on these stimuli. There are a total of 45 = 1024 different stimuli. Stimuli could be presented in two different locations
in the input, meaning that there are more than one million distinct stimulus input patterns (1024×1024). The Response
output layer contained the same stimulus representation as the input, with an additional “No” output response.
In each task, one of the five dimensions is relevant at any given time. The relevant dimension can either be
explicitly specified in the Dimension Cue input (instructed task) or it must be guessed by the network through trial and
error (uninstructed task). The task type is always specified via the Task input layer. For example, in the NF-I (naming
feature-instructed) task, the single input stimulus (presented in either input location) might correspond to three large,
rough, blue, squares, and if the relevant dimension in the Dimension Cue is size, the correct output would be “large”.
In the other three task types, the input display consists of two stimuli (in the left and right input locations), and these
stimuli are compared along the relevant dimension. Consistent with the NF task, the output for these tasks is either the
name of a feature, or “No.” For example, if the left stimulus and relevant dimension (size) are as in the prior example,
and the right stimulus is medium sized, then the correct output for MF (matching) is “No”, SF (smaller) is “medium”,
and LF (larger) is “large”.
Each network was trained 10 times with random initial weights for 100 epochs consisting of 2,000 individual
training trials per epoch. Testing for cross-task generalization performance occurred every 5 epochs throughout training, and the best level of generalization performance for each training run was recorded. This procedure eliminated
possible confounds associated with differential amounts of feature exposure per epoch, etc depending on the different
training conditions. The presence of the uninstructed versions of tasks required that relevant dimensions be organized
into contiguous blocks of trials where a given dimension was always relevant. The size and ordering of these blocks
was systematically varied in different “training protocols”. Testing was always performed on the instructed tasks, so
that the differential memory abilities of the various model architectures would not confound the results; no memory of
the relevant dimension is required in the instructed version.
Cross-task generalization testing requires that certain stimuli are not experienced on a given task during training;
these stimuli are then used during testing. Specifically, features B3, C2, D3 and E2 were always excluded from training
on all the instructed tasks, and testing consisted only of instructed task cases where one of these non-trained stimuli
was the relevant stimulus. This means that the instructed tasks were trained on
324
1024
= 31.64% of the full stimulus
space (324 = 4 × 3 × 3 × 3 × 3). The uninstructed versions of each of the four task types were trained with one omitted
stimulus feature. Specifically, the B3 stimulus was never seen during the NF-U task, the C2 stimulus was omitted for
the MF-U, etc. Generalization therefore required the learning on uninstructed tasks to transfer to instructed tasks.
Although some of the generalization tested was within the same task type between uninstructed and instructed (e.g.,
training on NF-U C2 could promote generalization of NF-I on C2), these task types were actually quite different from
the network’s perspective. Specifically, the critical relevant dimension must come from two different sources between
the instructed and uninstructed versions of the tasks, and this means that different representations and weights are
involved. The ability to produce an appropriate response across both cases requires that the internal representation
of the relevant dimension be consistent, and consistently used, across both cases. This generalization of dimensional
representation across tasks was exactly what we sought to test, and it applies equally well to generalization across
instructed and uninstructed as to generalization across naming and matching.
Two training/testing conditions were run; one in which all four task types were used, and another in which only 2
out of the four were used for any given network, with results averaged across all possible combinations of such task
pairs. Due to the structuring of the omitted features, these two conditions did not differ in the range of features trained,
but only in the extent to which a given feature appeared across multiple tasks. Thus, the differences in generalization
between these two conditions reveal the benefits of having processed a given feature in two versus four different task
contexts.
Finally, the sequencing of tasks and relevant dimensions was organized in a hierarchically blocked fashion, with
the outermost block consisting of a loop through all instructed tasks followed by all uninstructed tasks, with each
task being performed for a block of 25 input/output trials. The relevant dimension was switched after every 2 of the
outermost blocks. This relatively low rate of switching allows the networks without adaptive gating mechanisms plenty
of time to adapt to the relevant dimension. In other preliminary experiments, we tested a variety of blocking strategies,
including: a) interleaved instructed and uninstructed tasks within a 50 trial block, followed by a dimension switch; b)
the protocol described above (the “standard” protocol, outer blocks of tasks, instructed first, with inner-blocks of 25
trials); b) the standard protocol but with uninstructed cases first; c) standard but with uninstructed first; d) standard
but with inner-block sizes of 50 with 1 outer-block per dimension switch; e) like d but with uninstructed first; f) loop
through all instructed tasks with 50 trials per relevant dimension, followed by a loop through all uninstructed tasks
with 50 trials per dimension. g) like f but with uninstructed first. The impact of these manipulations never made more
than a 10% impact on performance in tests run on the generalization between U and I versions of the NF task (e.g., the
Full PFC model performance varied between 13.6% generalization error to 23.3% generalization error across these
versions).
Model Equations
All of the models were implemented using the Leabra framework, which incorporates a number of standard neural network mechanisms (e.g., point-neuron biophysical activation function, bidirectional excitatory connectivity, inhibitory competition, Hebbian and error-driven learning) (5, 6). This framework has been used to simulate over 40
different models in (6), and a number of other research models, including some that are closely related to that used
here (7, 8). Thus, models can be viewed as instantiations of a systematic modeling framework using standardized
mechanisms, instead of constructing new mechanisms for each model. All models and supplementary code can be
obtained by email at [email protected].
Pseudocode
The pseudocode for Leabra is given here, showing exactly how the pieces of the algorithm described in more detail
in the subsequent sections fit together.
Outer loop: Iterate over events (trials) within an epoch. For each event:
1. Iterate over minus and plus phases of settling for each event.
(a) At start of settling, for all units:
i. Initialize all state variables (activation, v m, etc).
ii. Apply external patterns (clamp input in minus, input & output in plus).
(b) During each cycle of settling, for all non-clamped units:
i. Compute excitatory netinput (ge (t) or ηj , eq 2).
ii. Compute kWTA inhibition for each layer, based on giΘ (eq 6):
A. Sort units into two groups based on giΘ : top k and remaining k + 1 to n.
B. If basic, find k and k + 1th highest; if avg-based, compute avg of 1 → k & k + 1 → n.
Θ
C. Set inhibitory conductance gi from gkΘ and gk+1
(eq 5).
iii. Compute point-neuron activation combining excitatory input and inhibition (eq 1).
(c) After settling, for all units:
i. Record final settling activations as either minus or plus phase (yj− or yj+ ).
2. After both phases update the weights (based on linear current weight values), for all connections:
(a) Compute error-driven weight changes (eq 8) with soft weight bounding (eq 9).
(b) Compute Hebbian weight changes from plus-phase activations (eq 7).
(c) Compute net weight change as weighted sum of error-driven and Hebbian (eq 10).
(d) Increment the weights according to net weight change.
Point Neuron Activation Function
Leabra uses a point neuron activation function that models the electrophysiological properties of real neurons,
while simplifying their geometry to a single point. This function is nearly as simple computationally as the standard sigmoidal activation function, but the more biologically-based implementation makes it considerably easier to
Parameter
El
Ei
Ee
Vrest
τ
k Hidden
k PFC
Value
0.15
0.15
1.00
0.15
.02
15%
10%
.01
Parameter
gl
gi
ge
Θ
γ
k Output
khebb
to AG Value
0.10
1.0
1.0
0.25
600
1 unit*
.01
.04*
Table 1: Parameters for the simulation (see equations in text for explanations of parameters). All are standard default parameter
values except for those with a * (most of which have no default because they are intrinsically task-dependent). The faster learning
rate () for connections into the AG was important for ensuring rapid learning of reward.
model inhibitory competition, as described below. Further, using this function enables cognitive models to be more
easily related to more physiologically detailed simulations, thereby facilitating bridge-building between biology and
cognition.
The membrane potential Vm is updated as a function of ionic conductances g with reversal (driving) potentials E
as follows:
∆Vm (t) = τ
X
c
gc (t)gc (Ec − Vm (t))
(1)
with 3 channels (c) corresponding to: e excitatory input; l leak current; and i inhibitory input. Following electrophysiological convention, the overall conductance is decomposed into a time-varying component g c (t) computed as
a function of the dynamic state of the network, and a constant g c that controls the relative influence of the different
conductances.
The excitatory net input/conductance ge (t) or ηj is computed as the proportion of open excitatory channels as a
function of sending activations times the weight values:
ηj = ge (t) = hxi wij i =
1X
xi wij
n i
(2)
The inhibitory conductance is computed via the kWTA function described in the next section, and leak is a constant.
Activation communicated to other cells (yj ) is a thresholded (Θ) sigmoidal function of the membrane potential
with gain parameter γ:
yj (t) = 1
1+
1
γ[Vm (t)−Θ]+
(3)
where [x]+ is a threshold function that returns 0 if x < 0 and x if X > 0. Note that if it returns 0, we assume
yj (t) = 0, to avoid dividing by 0. As it is, this function has a very sharp threshold, which interferes with graded
learning learning mechanisms (e.g., gradient descent). To produce a less discontinuous deterministic function with a
softer threshold, the function is convolved with a Gaussian noise kernel (µ = 0, σ = .005), which reflects the intrinsic
processing noise of biological neurons:
Z
yj∗ (x) =
∞
−∞
√
2
2
1
e−z /(2σ ) yj (z − x)dz
2πσ
(4)
where x represents the [Vm (t)−Θ]+ value, and yj∗ (x) is the noise-convolved activation for that value. In the simulation,
this function is implemented using a numerical lookup table.
k-Winners-Take-All Inhibition
Leabra uses a kWTA (k-Winners-Take-All) function to achieve inhibitory competition among units within a layer
(area). The kWTA function computes a uniform level of inhibitory current for all units in the layer, such that the k+1th
most excited unit within a layer is below its firing threshold, while the kth is above threshold. Activation dynamics
similar to those produced by the kWTA function have been shown to result from simulated inhibitory interneurons
that project both feedforward and feedback inhibition (6). Thus, although the kWTA function is somewhat biologically implausible in its implementation (e.g., requiring global information about activation states and using sorting
mechanisms), it provides a computationally effective approximation to biologically plausible inhibitory dynamics.
kWTA is computed via a uniform level of inhibitory current for all units in the layer as follows:
Θ
Θ
gi = gk+1
+ q(gkΘ − gk+1
)
(5)
where 0 < q < 1 (.25 default used here) is a parameter for setting the inhibition between the upper bound of g kΘ
Θ
and the lower bound of gk+1
. These boundary inhibition values are computed as a function of the level of inhibition
necessary to keep a unit right at threshold:
giΘ =
ge∗ g¯e (Ee − Θ) + gl g¯l (El − Θ)
Θ − Ei
(6)
where ge∗ is the excitatory net input without the bias weight contribution — this allows the bias weights to override the
kWTA constraint.
Hebbian and Error-Driven Learning
For learning, Leabra uses a combination of error-driven and Hebbian learning. The error-driven component is
the symmetric midpoint version of the GeneRec algorithm (9), which is functionally equivalent to the deterministic Boltzmann machine and contrastive Hebbian learning (CHL). The network settles in two phases, an expectation
(minus) phase where the network’s actual output is produced, and an outcome (plus) phase where the target output is experienced, and then computes a simple difference of a pre and postsynaptic activation product across these
two phases. For Hebbian learning, Leabra uses essentially the same learning rule used in competitive learning or
mixtures-of-Gaussians which can be seen as a variant of the Oja normalization. The error-driven and Hebbian learning
components are combined additively at each connection to produce a net weight change.
The equation for the Hebbian weight change is:
+
+
+ +
∆hebb wij = x+
i yj − yj wij = yj (xi − wij )
(7)
and for error-driven learning using CHL:
+
− −
∆err wij = (x+
i yj ) − (xi yj )
(8)
which is subject to a soft-weight bounding to keep within the 0 − 1 range:
∆sberr wij = [∆err ]+ (1 − wij ) + [∆err ]− wij
(9)
The two terms are then combined additively with a normalized mixing constant k hebb :
∆wij = [khebb (∆hebb ) + (1 − khebb )(∆sberr )]
(10)
Temporal Differences and Adaptive Critic Gating Mechanisms
The adaptive gating mechanism was based on an adaptive critic unit (AG) that updated its activations according
to the temporal-differences (TD) algorithm. The specific mechanism and its motivations are discussed in detail in (8).
It was implemented in Leabra using plus-minus phase states that correspond to the expected reward at the previous
time step (minus) and the current time step (plus). The difference between these two states is the TD error δ, which
is essentially equivalent to the more standard kinds of error signals computed by the error-driven learning component
of Leabra, except that it represents an error of prediction over time, instead of an instantaneous error in the network
output. This δ value then modulates the strength of an excitatory ionic current (labeled m here) that helps to maintain
PFC activations:
gm (t − 1) = 0 if |δ(t)| > r
(11)
gm (t)j = gm (t − 1) + δ(t)yj (t)
(12)
Thus, a positive δ increases this maintenance current for active units, and a negative δ decreases it. Furthermore, if δ
is sufficiently large in magnitude (greater than the reset threshold r = .5), it resets any existing currents.
To prevent gating from occurring for occasional errors, we included a fault-tolerant mechanism where two errors
in a row had to be made before the AG unit was delivered a 0 reward value. Furthermore, we included rapidly
adapting and decaying negative-only bias weights, which serve to inhibit prior task representations (again see 8) for
more detailed motivations and explorations of this mechanism). These bias weights in the PFC layer learn on the
plus-minus phase difference in unit activations, as normal bias weights do, but they only learn negative values, with
a learning rate of .2 (compared to the standard .01 for the remainder of the network). These biases decay with a time
constant of .5 (i.e., .5 of the existing weight value is subtracted on each trial).
*
References and Notes
1. M. J. Emerson and A. Miyake, Journal of Memory and Language, 48, 148–168, (2003). A. R. Luria. The role
of speech in the regulation of normal and abnormal behavior. Liveright Publishing Corporation, New York,
(1961). L. Vygotsky. Thought and Language. MIT Press (Original work published in 1934), Cambridge, MA,
(1934/1986).
2. J. L. Elman, Cognitive Science, 14, 179–211, (1990).
3. R. C. O’Reilly, Neural Computation, (submitted).
4. R. C. O’Reilly, Neural Computation, 13, 1199–1242, (2001).
5. R. C. O’Reilly, Trends in Cognitive Sciences, 2, 455–462, (1998).
6. R. C. O’Reilly and Y. Munakata. Computational Explorations in Cognitive Neuroscience: Understanding the
Mind by Simulating the Brain. MIT Press, Cambridge, MA, (2000).
7. R. C. O’Reilly, D. Noelle, T. S. Braver, and J. D. Cohen, Cerebral Cortex, 12, 246–257, (2002).
8. N. P. Rougier and R. C. O‘Reilly, Cognitive Science, 26, 503–520, (2002).
9. R. C. O’Reilly, Neural Computation, 8, 895–938, (1996).