PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
1
Neuromorphic electronic circuits for building
autonomous cognitive systems
arXiv:1403.6428v1 [cs.ET] 25 Mar 2014
Elisabetta Chicca, Member, IEEE, Fabio Stefanini, Chiara Bartolozzi, Member, IEEE and Giacomo Indiveri Senior
Member, IEEE
Abstract—Several analog and digital brain-inspired electronic
systems have been recently proposed as dedicated solutions for
fast simulations of spiking neural networks. While these architectures are useful for exploring the computational properties
of large-scale models of the nervous system, the challenge of
building low-power compact physical artifacts that can behave
intelligently in the real-world and exhibit cognitive abilities
still remains open. In this paper we propose a set of neuromorphic engineering solutions to address this challenge. In
particular, we review neuromorphic circuits for emulating neural
and synaptic dynamics in real-time and discuss the role of
biophysically realistic temporal dynamics in hardware neural
processing architectures; we review the challenges of realizing
spike-based plasticity mechanisms in real physical systems and
present examples of analog electronic circuits that implement
them; we describe the computational properties of recurrent
neural networks and show how neuromorphic Winner-Take-All
circuits can implement working-memory and decision-making
mechanisms. We validate the neuromorphic approach proposed
with experimental results obtained from our own circuits and
systems, and argue how the circuits and networks presented in
this work represent a useful set of components for efficiently and
elegantly implementing neuromorphic cognition.
I. I NTRODUCTION
Machine simulation of cognitive functions has been a challenging research field since the advent of digital computers.
However, despite the large efforts and resources dedicated
to this field, humans, mammals, and many other animal
species including insects, still outperform the most powerful
computers in relatively routine functions such as sensory processing, motor control and pattern recognition. The disparity
between conventional computing technologies and biological
nervous systems is even more pronounced for tasks involving
autonomous real-time interactions with the environment, especially in presence of noisy and uncontrolled sensory input.
One important aspect is that the computational and organizing
principles followed by the nervous system are fundamentally
different from those of present day computers. Rather than using Boolean logic, precise digital representations and clocked
operations, nervous systems carry out robust and reliable
computation using hybrid analog/digital unreliable processing
elements; they emphasize distributed, event-driven, collective,
E. Chicca is with the Cognitive Interaction Technology - Center of Excellence, Bielefeld University and Faculty of Technology, Bielefeld, Germany
email:chicca[at]cit-ec.uni-bielefeld.de
C. Bartolozzi is with the iCub Facility, Istituto Italiano di Tecnologia,
Genova, Italy
F. Stefanini and G. Indiveri are with the Institute of Neuroinformatics,
University of Zurich and ETH Zurich, Switzerland
Manuscript received Month DD, YEAR; revised MONTH DD, YEAR.
and massively parallel mechanisms and make extensive use of
adaptation, self-organization and learning.
Several approaches have been recently proposed for building custom hardware, brain-like neural processing architectures [1]–[9]. The majority of them are proposed as an
alternative electronic substrate to traditional computing architectures for neural simulations [2], [4], [5], [7]. These
systems can be very useful tools for neuroscience modeling,
e.g., by accelerating the simulation of complex computational
neuroscience models by three or more orders of magnitude [4],
[7], [10]. However, our work focuses on an alternative approach aimed at the realization of compact, real-time, and
energy efficient computational devices that directly emulate
the style of computation of the brain, using the physics of
Silicon to reproduce the bio-physics of the neural tissue. This
approach, on one hand, leads to the implementation of compact
and low-power behaving systems ranging from brain-machine
interfaces to autonomous robotic agents. On the other hand,
it serves as a basic research instrument for exploring the
computational properties of the neural system they emulate
and hence gain a better understanding of its operational
principles. These ideas are not new: they follow the original
vision of Mead [11], Mahowald [12], and colleagues [13].
Indeed, analog Complementary Metal–Oxide–Semiconductor
(CMOS) technology has been effectively employed for the
construction of simple neuromorphic circuits reproducing basic dynamical properties of their biological counterparts, e.g.,
neurons and synapses, at some level of precision, reliability
and detail. These circuits have been integrated into Very
Large Scale Integration (VLSI) devices for building real-time
sensory-motor systems and robotic demonstrators of neural
computing architectures [14]–[19]. However, these systems,
synthesized using ad-hoc methods, could only implement very
specific sensory-motor mappings or functionalities. The challenge that remains open is to bridge the gap from designing
these types of reactive artificial neural modules to building
complete neuromorphic behaving systems that are endowed
with cognitive abilities. The step from reaction to cognition
in neuromorphic systems is not an easy one, because the
principles of cognition remain to be unraveled. A formal
definition of these principles and their effective implementation in hardware is now an active domain of research [20]–
[23]. The construction of brain-like processing systems able to
solve cognitive tasks requires sufficient theoretical grounds for
understanding the computational properties of such a system
(hence its necessary components), and effective methods to
combine these components in neuromorphic systems. During
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
the last decade we pursued this goal by realizing neuromorphic
electronic circuits and systems and using them as building
blocks for the realization of simple neuromorphic cognitive
systems [20]. Here we describe these circuits, analyze their dynamics in comparison with other existing solutions and present
experimental results that demonstrate their functionalities. We
describe the limitations and problems of such circuits, and
propose effective design strategies for building larger brainlike processing systems. We conclude with a discussion on
the advantages and disadvantages of the approach we followed
and with a description of the challenges that need to be
addressed in order to progress in this domain. Specifically, in
the following sections we show how the building blocks we
propose, based on dynamic synapse circuits, hardware models
of spiking neurons, and spike-based plasticity circuits, can be
integrated to form multi-chip spiking recurrent and WinnerTake-All neural networks, which in turn have been proposed
as neural models for explaining pattern recognition [24], [25],
working memory [9], [26], decision making [27], [28] and
state-dependent computation [29], [30] in the brain.
II. N EURAL DYNAMICS IN ANALOG VLSI
Unlike a von Neumann computing architecture, neuromorphic architectures are composed of massively parallel arrays
of simple processing elements in which memory and computation are co-localized. In these architectures time represents
itself and so the synapse and neuron circuits must process
input data on demand, as they arrive, and must produce
their output responses in real-time. Consequently, in order to
interact with the environment and process real-world sensory
signals efficiently, neuromorphic behaving systems must use
circuits that have biologically plausible time constants (i.e.,
of the order of tens of milliseconds). In this way, they are
well matched to the signals they process and are inherently
synchronized with the real-world events. This constraint is not
easy to satisfy using analog VLSI technology. Standard analog
circuit design techniques either lead to bulky and silicon-area
expensive solutions [31] or fail to meet this condition, resorting
to modeling neural dynamics at “accelerated” unrealistic timescales [32], [33].
One elegant solution to this problem is to use currentmode design techniques [34] and log-domain subthreshold
circuits [35]–[39]. When Metal Oxide Semiconductor Field
Effect Transistors (MOSFETs) are operated in the subthreshold domain, the main mechanism of carrier transport is that
of diffusion, as it is for ions flowing through proteic channels
across neuron membranes. As a consequence, MOSFETs have
an exponential relationship between gate-to-source voltage and
drain current, and produce currents that range from femto- to
nano-Ampères. As the time constants of log-domain circuits
are inversely proportional to their reference currents, in addition to being directly proportional to the circuit capacitance,
the subthreshold domain allows the integration of relatively
small capacitors in VLSI to implement temporal filters that are
both compact and have biologically realistic time constants,
ranging from tens to hundreds of milliseconds.
Neuron conductance dynamics and synaptic transmission
can be faithfully modeled by first order differential equa-
2
Vdd
Vdd
Ith
Iin
Iout
VS
VG
I1
VC
I2
Iτ
IC
(a)
Vdd
Vdd
Iτ
Vdd
IC
Vdd
VC
Iout
I1
Iin
(b)
Vdd
Vdd
Vdd
Iin
Iτ
Iout
I1
VC
Vref
2Iτ
IC
Vref
(c)
Fig. 1: Current-mode low-pass filter circuits. Red arrows show
the translinear loop considered for the log-domain analysis. (a)
The Differential Pair Integrator circuit diagram. (b) The LowPass Filter circuit diagram. (c) The “Tau-Cell” circuit diagram.
tions [40], therefore subthreshold log-domain circuits that
implement first order low pass filters can faithfully reproduce
biologically plausible temporal dynamics. Several examples
of such circuits have been proposed as basic building blocks
for the implementation of silicon neurons and synapses.
Among them, the Differential Pair Integrator (DPI) [41], [42],
the log-domain Low-Pass Filter (LPF) [43], and the “TauCell” [44] circuits offer a compact and low-power solution.
These circuits, shown in Fig. 1, can be analyzed by applying
the translinear principle, whereby the sum of voltages in a
chain of transistors that obey an exponential current-voltage
characteristic can be expressed as multiplication of the currents
flowing across them [45]. For example, if we consider the
DPI circuit of Fig. 1a, and we assume that all transistor have
same parameters and operate in the subthreshold regime and
in saturation [37], we can derive circuit solution analytically.
Specifically, we can write:
κVC
Iout = I0 e UT
Iin = I1 + I2
d
VC
dt
I2 = Iτ + IC
IC = C
(1)
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
3
where the term I0 represents the transistor dark current, UT
represents the thermal voltage and κ the subthreshold slope
factor [37]. By applying the translinear principle across the
loop made by the arrows in the circuit diagram of Fig. 1a we
can write: Ith · I1 = I2 · Iout . Then, by replacing I1 and
expanding I2 from eq. (1) we get:
Ith · (Iin − Iτ − IC ) = (Iτ + IC ) · Iout .
(2)
Thanks to the properties of exponential functions, we can
express IC as a function of Iout :
UT d
Iout
(3)
IC = C
κIout dt
Finally, by replacing IC from this equation and dividing
everything by Iτ in eq. (2), we get:
Ith d
Ith Iin
τ 1+
Iout + Iout =
− Ith
(4)
Iout dt
Iτ
DPI
LPF
Tau-Cell
Circuit equations
κVC
κ(Vdd −VC )
UT
κV2 −Vre f
UT
Iout = I0 e UT
Iout = I0 e
C
IC = C dV
dt
C
IC = −C dV
dt
C
IC = C dV
dt
Iin = I1 + Iτ + IC
I1 = Iτ + IC
I1 = Iτ + IC
T
IC = C κIUout
T
IC = C κIUout
dIout
dt
dIout
dt
Iout = I0 e
IC = C IUoutT
dIout
dt
Translinear Loop
Ith · I1 = (Iτ + IC ) · Iout
Iin · I0 = I1 · Iout
Iin · Iτ = I1 · Iout
Solution
τ
dIout
Ith
+ Iout =
Iin
dt
Iτ
τ=
CUT
κIτ
τ
dIout
I0
+ Iout = Iin
dt
Iτ
τ=
CUT
κIτ
τ
dIout
+ Iout = Iin
dt
τ=
CUT
Iτ
where τ , CUT /κIτ .
This is a first-order non-linear differential equation that
cannot be solved explicitly. However, in the case of sufficiently
large input currents (i.e., Iin Iτ ) the term −Ith on the right
side of eq. (4) can be neglected. Furthermore, under this
assumption and starting from an initial condition Iout = 0,
Iout will increase monotonically and eventually the condition
th
Iout Ith will be met. In this case also the term IIout
on the
left side of eq. (4) can be neglected. So the response of the
DPI reduces to a first-order linear differential equation:
TABLE I: Characteristic equations of the DPI, LPF, and TauCell log-domain filters.
d
Ith
Iout + Iout = Iin
dt
Iτ
III. S ILICON NEURONS
Several VLSI implementations of conductance-based models of neurons have been proposed in the past [50]–[54].
Given their complexity, these circuits require significant silicon
real-estate and a large number of bias voltages or currents
to configure the circuit properties. Simplified Integrate-andFire (I&F) models typically require far less transistors and
parameters but often fail at reproducing the rich repertoire of
behaviors of more complex ones [55], [56].
A recently proposed class of generalized I&F models however has been shown to capture many of the properties of
biological neurons, while requiring less and simpler differential equations compared to more elaborate conductance-based
models, such as the Hodgkin & Huxley (H&H) one [56], [57].
Their computational simplicity and compactness make them
valuable options for VLSI implementations [32], [47], [48],
[58], [59].
We describe here a generalized I&F neuron circuit originally presented in [59], which makes use of the DPI circuit
described in the previous Section and which represents an
excellent compromise between circuit complexity and computational power: the circuit is compact, both in terms of transistor count and layout size; it is low-power; it has biologically
realistic time constants; and it implements refractory period
and spike-frequency adaptation, which are key ingredients for
producing resonances and oscillatory behaviors often emphasized in more complex models [55], [57].
The circuit schematic is shown in Fig. 2. It comprises
an input DPI circuit used as a low-pass filter (ML1−3 ), a
τ
(5)
The general solution of the other two log-domain circuits
shown in Fig. 1b and Fig. 1c can be derived analytically
following a similar procedure. Table I shows the equations
used for the derivation of all three circuits, and their general
solution.
The LPF circuit of Fig. 1 is the one that has the least number
of components. However it is not the most compact, because to
apply the translinear principle correctly, it is necessary to use a
p-type Field Effect Transistor (FET) with its bulk connected to
its source node (see p-FET with I1 current flowing through it
in Fig. 1b). This requires an isolated well in the circuit layout,
which leads to larger area usage, and makes the overall size of
the circuit comparable to the size of the other two solutions.
Furthermore, the requirement of an isolated well for the p-FET
does not allow the design of the complementary version of the
circuit in standard CMOS processes (e.g., to have negative
currents). The Tau-Cell circuit does not have this problem,
but it requires precise matching of the two current sources
producing Iτ and −2Iτ , which can also lead to large area
usage at the layout level. The DPI can implement in a compact
way both positive and negative currents (e.g., by using the
complementary version of the schematic of Fig. 1a). An other
advantage of the DPI, with respect to the other two solutions,
is the availability of the additional control parameter Ith that
can be used to change the gain of the filter.
The LPF circuit has been used to model both synaptic
excitation and shunting inhibition [46]. The Tau-Cell has been
used to implement log-domain implementation of MihalasNiebur [47] and Izhikevich [48] neuron models, and the
DPI has been used to implement both synapse and neuron
models [41], [49]. In the next sections we will show examples
of neurons and synapses that exploit the properties of the DPI
to implement the relevant dynamics.
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
Vdd
Vdd
MG1
MA5
Vdd
4
Vdd
MR1
MA1
MG2
Vahp
Vdd
Iin
Vthr
ML1 ML2
Vlk
Iahp
VP
CP
Ir
Vlkahp MR6
MG5
MR3
MA3
Imem
MA4
Vref
MR4
Ia
(Imem + Ith )
Iτ
(7)
In [49] the authors measured Imem experimentally and showed
how f (Imem ) could be fitted with an exponential function of
Imem . The other parameters of eq. (6) are defined as:
MR2
MA2
Cmem M
G6
ML3
Ia M
A6
MG4
Vthrahp
f (Imem ) =
/REQ
ICa
MG3
variable Imem and the positive-feedback current Ia of Fig. 2:
/ACK
CR
MR5
Fig. 2: Adaptive exponential I&F neuron circuit schematic.
The input DPI circuit (ML1−3 ) models the neuron’s leak
conductance. A spike event generation amplifier (MA1−6 )
implements current-based positive feedback (modeling both
sodium activation and inactivation conductances) and produces
address-events at extremely low-power operation. The reset
block (MR1−6 ) resets the neuron and keeps it in a resting
state for a refractory period, set by the Vref bias voltage. An
additional low-pass filter (MG1−6 ) integrates the spikes and
produces a slow after hyper-polarizing current Iahp responsible
for spike-frequency adaptation.
spike-event generating amplifier with current-based positive
feedback (MA1−6 ), a spike reset circuit with refractory period functionality (MR1−6 ) and a spike-frequency adaptation
mechanism implemented by an additional DPI low-pass filter
(MG1−6 ). The DPI block ML1−3 models the neuron’s leak
conductance; it produces exponential sub-threshold dynamics
in response to constant input currents. The neuron’s membrane
capacitance is represented by the capacitor Cmem while Sodium
channel activation and inactivation dynamics are modeled by
the positive-feedback circuits in the spike-generation amplifier
MA1−6 . The reset MR1−6 block models the Potassium conductance and refractory period functionality. The spike-frequency
adaptation block MG1−6 models the neuron’s Calcium conductance that produces the after-hyper-polarizing current Iahp ,
which is proportional to the neuron’s mean firing rate.
By applying the current-mode analysis of Section II to
both the input and the spike-frequency adaptation DPI circuits
we derive the complete equation that describes the neuron’s
subthreshold behavior:
Iahp
d
Ith
1+
τ Imem + Imem 1 +
= Imem∞ + f (Imem )
Imem
dt
Iτ
d
τahp Iahp + Iahp = Iahp∞ u(t)
(6)
dt
where Imem is the sub-threshold current that represents the real
neuron’s membrane potential variable, Iahp is the slow variable
responsible for the spike-frequency adaptation mechanisms,
and u(t) is a step function that is unity for the period in which
the neuron spikes and null in other periods. The term f (Imem )
is a function that depends on both the membrane potential
τ,
CmemUT
,
κIτ
κ
Iτ , I0 e UT
Imem∞ ,
Vlk
τahp ,
C pUT
κIτahp
κ
,
Ith
(Iin − Iahp − Iτ ),
Iτ
V
Iτahp , I0 e UT lkahp
Ithahp
Iahp∞ ,
ICa
Iτahp
where Ith and Iτahp represent currents through n-type
κ
V
MOSFETs not present in Fig. 2, and defined as Ith , I0 e UT thr ,
κ V
and Ithahp , I0 e UT thrahp respectively.
In addition to emulating Calcium-dependent after-hyperpolarization Potassium currents observed in real neurons [60],
the spike-frequency adaptation block MG1−6 reduces power
consumption and bandwidth usage in networks of these neurons. For values of Iin Iτ we can make the same simplifying
assumptions made in Section II. Under these assumptions, and
ignoring the adaptation current Iahp , eq. (6) reduces to:
τ
d
Ith
Imem + Imem = Iin + f (Imem )
dt
Iτ
(8)
where f (Imem ) ≈ IIτa Imem .
So under these conditions, the circuit of Fig. 2 implements
a generalized I&F neuron model [61], which has been shown
to be extremely versatile and capable of faithfully reproducing
the action potentials measured from real cortical neurons [62],
[63]. Indeed, by changing the biases that control the neuron’s time-constants, refractory period, and spike frequency
adaptation dynamics this circuit can produce a wide range of
spiking behaviors ranging from regular spiking to bursting (see
Section VII).
While this circuit can express dynamics with time constants
of hundreds of milliseconds, it is also compatible with fast
asynchronous digital circuits (e.g., < 100 nanosecond pulse
widths), which are required to build large spiking neural
network architectures (see the /REQ and /ACK signals of
Fig. 2 and Section VI). This allows us to integrate multiple
neuron circuits in event-based VLSI devices and construct
large distributed re-configurable neural networks.
IV. S ILICON SYNAPSES
Synapses are fundamental elements for computation and
information transfer in both real and artificial neural systems,
and play a crucial role in neural coding and learning. While
modeling the non-linear properties and the dynamics of large
ensembles of synapses can be extremely onerous for Software
(SW) simulations (e.g., in terms of computational power and
simulation time), dedicated neuromorphic Hardware (HW)
can faithfully reproduce synaptic dynamics in real-time using
massively parallel arrays of pulse (spike) integrators. In this
case, the bottleneck is not in the complexity of the spiking
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
5
processes being modeled, but in the number of spikes being
received and transmitted (see Section VI for more details).
An example of a full excitatory synapse circuit is shown
in Fig. 3. This circuit, based on the DPI circuit described
in Section II, produces biologically realistic Excitatory Post
Synaptic Currents (EPSCs), and can express short term plasticity, N-Methyl-D-Aspartate (NMDA) voltage gating, and
conductance-based behaviors. The input spike (the voltage
pulse Vin ) is applied to both MD3 and MS3 . The output current
Isyn , sourced from MD6 and through MG2 , rises and decays
exponentially with time. The temporal dynamics are implemented by the DPI block MD1−6 . The circuit time constant
is set by Vτ while the synaptic efficacy, which determines the
EPSC amplitude, depends on both Vw0 and Vthr [41].
A. Short term depression and short-term facilitation
Short term plasticity mechanisms can be extremely effective tools for processing temporal signals and decoding
temporal information [64], [65]. Several circuit solutions have
been proposed to implement these types of dynamics, using
different types of devices and following a wide range of
design techniques [66]–[71]. These short-term dynamic mechanisms are subdivided into short-term depression and shortterm facilitation. The circuit block MS1−3 is responsible for
implementing short-term depression: with every voltage pulse
Vin the synaptic weight voltage Vw decreases, at a rate set by
Vstd . When no spikes are being received, the Vw “recovers”
toward the resting state set by Vw0 . In [67] the authors
demonstrate that this sub-circuit is functionally equivalent to
the one described in theoretical models, and often used in
computational neuroscience simulations [72], [73]. In addition
to short-term depression, this DPI synapse is capable also
of short-term facilitation: if the bias Vthr of MD1 is set so
that Ith Isyn at the onset of the stimulation (i.e., during the
first spikes), the circuit equation, derived from eq. (4) in the
analysis of Section II reduces to:
τ
2
Isyn
Iw
d
Isyn +
− Isyn ( + 1) = 0
dt
Ith
Iτ
(9)
which can be further simplified to:
τ
Iw
d
Isyn = Isyn ( + 1)
dt
Iτ
(10)
In other words, the change in circuit response increases with
every spike, by an amount greater than one, for as long as the
condition Isyn Ith is satisfied. As Isyn increases this condition
starts to fail, and eventually the opposite condition (Isyn Ith )
is reached. This is the condition for linearity, under which
the circuit starts to behave as a first order low-pass filter, as
described in Section II.
B. NMDA voltage gating and conductance behavior
The output differential pairs of Fig. 3 (MN1−2 and MG1−2 )
are responsible for implementing NMDA voltage gated channels and conductance-based behavior respectively. The response properties of these circuits have been thoroughly characterized in [41].
Vdd
Vτ
Vdd
Vw0
Vthr
MS1
Vstd
Vin
MS2
MS3
Iτ
Vdd
Vdd
Csyn
MD5
MD6
MD1 MD4
Vw
Cstd
Vin
Vnmda
MD2
Iw
MD3
VG
MG1
MN1
MN2
MG2
Vmem
Isyn
Fig. 3: Complete DPI synapse circuit, including short term
plasticity, NMDA voltage gating, and conductance-based functional blocks. The short-term depression block is implemented by MOSFETs MS1−3 ; the basic DPI dynamics are
implemented by the block MD1−6 ; the NMDA voltage gated
channels are implemented by MN1−2 , and conductance based
voltage dependence is achieved with MG1−2 .
C. Homeostatic plasticity: synaptic scaling
Synaptic scaling is a stabilizing homeostatic mechanism
used by biological neural systems to keep the network’s
activity within proper operating bounds. It operates by globally
scaling the synaptic weights of all the synapses afferent to
a neuron, for maintaining the neuron’s firing rate within a
functional range, in face of chronic changes of their activity
level, while preserving the relative differences between individual synapses [74]. In VLSI, synaptic scaling is an appealing
mechanism that can be used to compensate for undesired
behaviors that can arise for example because of temperature
drifts or sudden changes in the system input activity levels.
Thanks to its independent controls on synaptic efficacy set by
Vw and Vthr , the DPI synapse of Fig. 3 is compatible with
both conventional spike-based learning rules, and homeostatic
synaptic scaling mechanisms. Specifically, while learning circuits can be designed to locally change the synaptic weight
by acting on the Vw of each individual synapse (e.g., see
Section V), it is possible to implement adaptive circuits that
act on the Vthr of all the synapses connected to a given neuron
to keep its firing rate within desired control boundaries. This
strategy has been recently demonstrated in [75].
V. S YNAPTIC PLASTICITY: SPIKE - BASED LEARNING
CIRCUITS
One of the key properties of biological synapses is their
ability to exhibit different forms of plasticity. Plasticity mechanisms produce long-term changes in the synaptic strength of
individual synapses in order to form memories and learn about
the statistics of the input stimuli. Plasticity mechanisms that
induce changes that increase the synaptic weights are denoted
as Long Term Potentiation (LTP) mechanisms, and those that
induce changes that decrease synaptic weights are denoted as
Long Term Depression (LTD) mechanisms [76].
In neuromorphic VLSI chips, implementations of longterm plasticity mechanisms allow us to implement learning
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
algorithms and set synaptic weights automatically, without
requiring dedicated external read and write access to each
individual synapse.
As opposed to the case of theory, or software simulation,
the realization of synapses in hardware imposes a set of
important physical constraints. For example synaptic weights
can only have bounded values, and with a limited (and
typically small) precision. These constraints have dramatic
effects on the memory capacity of the neural network that uses
such synapses [77], [78]. So when developing computational
models of biological synapses that will be mapped onto neuromorphic hardware, it is important to develop plasticity mechanisms that work with limited resolution and bounded synaptic
weights [24]. Another important constraint that should be
taken into account when developing hardware learning systems
that are expected to operate continuously (as is the case
for real-time behaving systems) is related to the blackout
effect [79]. Classical Hopfield networks are affected by this
effect: in Hopfield networks the memory capacity is limited,
and is related to the number of synapses available. Learning
new patterns uses memory resources and if the number of
stored patterns reaches a critical value the storage of even
one single new pattern destroys the whole memory because
none of the old patterns can be recalled. Unfortunately, this
catastrophic condition is unavoidable in most practical scenarios, since continuous, uninterrupted learning will always
lead to the blackout effect. However, it is possible to avoid
this effect, by building networks that can progressively forget
old memories to make room for new ones, thus exhibiting the
palimpsest property [80]. It has been demonstrated that the
optimal strategy for implementing this palimpsest property,
while maintaining a high storage capacity, is to use synapses
that have a discrete number of stable states and that exhibit
stochastic transitions between states [81]. Specifically, it was
demonstrated that by modifying only a random subset of the
network synapses with a small probability, memory lifetimes
increase by a factor inversely proportional to the probability
of synaptic modification [82]. In addition, the probability of
synaptic transitions can be used as a free parameter to set the
trade-off between the speed of learning against the memory
capacity.
These types of plastic synapse circuits can be implemented
in a very compact way by reducing to the minimum the
resolution of the synaptic weight (i.e., just two stable states)
and using variability in the input spike trains as the source of
stochasticity for the transition of the synaptic weights (e.g.,
from an LTD to an LTP stable state). The low resolution in
the synaptic weights can be compensated by redundancy (i.e.,
using large numbers of synapses) and the variability in the
input spike trains can be obtained by encoding signals with the
mean rates of Poisson distributed spike-trains [83]–[85]. An
important advantage of delegating the onus of generating the
stochasticity to the input spiking activity is that no additional
circuitry is needed for the stochastic state transitions [86].
Furthermore, since the spiking activity controls the speed
of learning, the network can easily switch between a slowlearning regime (i.e., to learn pattern of mean firing rates
with uncorrelated stimuli) to a fast learning one (i.e., to
6
learn highly correlated patterns) without changing its internal
parameters [84], [87].
In addition to allowing compact circuit designs, these types
of plastic synapse circuits do not require precisely matched
analog devices. As the dominant source of variability lies
in the (typically Poisson distributed) input spikes driving the
learning, additional sources of variability, for example induced
by device mismatch, do not affect the main outcome of the
learning process. As a consequence, analog VLSI designers
do not have to allocate precious Silicon real-estate resources
to minimize device mismatch effects in these circuits.
An example of a circuit that implements a weight update
mechanism compatible with this stochastic learning rule, is
shown in Fig. 4a. The circuit comprises three main blocks:
an input stage MI1−2 , a spike-triggered weight update block
ML1−4 , and a bi-stability weight storage/refresh block (see
transconductance amplifier in Fig. 4a). The input stage receives
spikes from pre-synaptic neurons and triggers increases or decreases in weights, depending on the two signals VUP and VDN
generated downstream by the post-synaptic neuron. The bistability weight refresh circuit is a positive-feedback amplifier
with very small “slew-rate” that compares the weight voltage
Vw to a set threshold Vthw and slowly drives it toward one of
the two rails Vwhi or Vwlo , depending on whether Vw > Vthw or
Vw < Vthw respectively. This bi-stable drive is continuous and
its effect is superimposed to the one from the spike-triggered
weight update circuit. The analog, bi-stable, synaptic weight
voltage Vw is then used to set the amplitude of the EPSC
generated by the synapse integrator circuit (e.g., the circuit
shown in Fig. 3). Note that while the weight voltage Vw is
linearly driven by the bi-stability circuit, its effect on the EPSC
produced by the connected DPI synapse is exponential. This
non-linearity can in principle affect adversely the dynamics of
learning and is more relevant at small scales (tens of synapses)
since the contribute of each synapse is important. However
the non-linearity has a negligible effect in practice because
in the slow-learning regime only a small subset of a much
larger number of synapses is involved in the learning process,
each one participating with a small contribution. The circuit
presented here can be easily modified to better reproduce the
linear dynamics of the theoretical model by decoupling the
synaptic weight from the internal variable, as in [88].
The two signals VUP and VDN of Fig. 4a that determine
whether to increase or decrease the synaptic weight are shared
globally among all synapses afferent to a neuron. The circuits
that control these signals can be triggered by the neuron’s
post-synaptic spike, to implement standard Spike–timing–
dependent Plasticity (STDP) learning rules [76]. In general,
STDP mechanisms that update the synaptic weight values
based on the relative timing of pre- and post-synaptic spikes
can be implemented very effectively in analog [83], [89]–
[92] or mixed analog-digital VLSI technology [93]. However,
while standard STDP mechanisms can be effective in learning
to classify spatio-temporal spike patterns [93], [94], these
algorithms and circuits are not suitable for both encoding
information represented in a spike correlation code and a
means rate code without spike correlations [95], [96]. For
this reason, we focus on more elaborate plasticity mecha-
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
nisms that not only depend on the timing of the pre-synaptic
spikes but also on other state variables present at the postsynaptic terminal, such as the neuron membrane potential
or its Calcium concentration. An example of such type of
learning rule is the one proposed in [25], which has been
shown to be able to classify patterns of mean firing rates, to
capture the rich phenomenology observed in neurophysiological experiments on synaptic plasticity, and to reproduce the
classical STDP phenomenology both in hardware [9], [85],
[88] and in software simulations [25], [97]. This rule can
be used to implement unsupervised and supervised learning
protocols, and to train neurons to act as perceptrons or binary
classifiers [24]. Typically, input patterns are encoded as sets
of spike trains that stimulate the neuron’s input synapses with
different mean frequencies, while the neuron’s output firing
rate represents the binary classifier output.
Examples of circuits that implement such a learning rule
are shown in Fig. 4b. The spikes produced by the postsynaptic neuron are integrated by the DPI circuit MD1−5
to produce a voltage VCa which represents a post-synaptic
Calcium concentration and is a measure of the recent spiking
activity of the neuron. The three current-mode winner-take-all
circuits [98] MW1−19 compare VCa to three different thresholds
Vthk1 , Vthk2 , and Vthk3 . In parallel, the neuron’s membrane
potential Vmem is compared to a fixed threshold Vthm by a
voltage comparator. The outcomes of these comparisons set
VUP and VDN such that, whenever a pre-synaptic spike Vspk
reaches the synapse weigh-update block of Fig. 4a:
(
Vw = Vw + ∆w if Vmem > Vmth and Vthk1 < VCa < Vthk3
Vw = Vw − ∆w if Vmem < Vmth and Vthk1 < VCa < Vthk2
where ∆w is a factor that depends on V∆w of Fig. 4b, and
is gated by the eligibility traces VUP or VDN . If none of the
conditions above are met, ∆w is set to zero by setting VUP =
Vdd , and VDN = 0.
The conditions on VCa implement a “stop-learning” mechanism that greatly improves the generalization performance of
the system by preventing over-fitting when the input pattern
has already been learned [24], [25]. For example, when the
pattern stored in the synaptic weights and the pattern provided
in input are highly correlated, the post-synaptic neuron will fire
with a high rate and VCa will rise such that VCa > Vthk3 , and
no more synapses will be modified.
In [85], [88] the authors show how such types of circuits
can be used to carry out classification tasks with a supervised
learning protocol, and characterize the performance of these
types of VLSI learning systems. Additional experimental
results from the circuits shown in Fig. 4 are presented in
Section VII.
VI. F ROM CIRCUITS TO NETWORKS
The silicon neuron, synapse, and plasticity circuits presented
in the previous Sections can be combined together to form full
networks of spiking neurons. Typical spiking neural network
chips have the elements described in Fig. 5. Multiple instances
of these elements can be integrated onto single chips and
connected among each other either with on-chip hard-wired
7
Vdd
ML3
MI2
Vspk
VUP
ML4
Vdd
MI1
Vwhi
+
Vthw
ML2
Vw
−
Vwlo
VDN
ML1
(a)
Vdd
Vmem
Vthm
MT2
Vmhi
−
+
Vdd
MT4
Vmlo
MT1
MT3
Vdd
MO3
VDN
Vdd
Vdd
Vdd
MO4
Vdd
MW12
Ik2
Vthk2
MW8
Vdd
Vdd
VCa
MW13
MW11
MW9
MW18
ICa
ICa
MW10
MW14
Vdd
Vdd
Vdd
VthrCa
MW16
MW15
VwCa
VREQ
MD3
Ik3
MW17
Vmhi
Vdd
MD2
Vthk3
Vdd
MD5
MD4
Vdd
MW19
Vmlo
V τCa
VUP
MO1
MO2
VCa
VCa
Vdd
MW7 Vthk1
MW6
ICa
MW2
MD1
Vdd
MW5
MW3
Ik1
MW4
V Δw
MW1
(b)
Fig. 4: Spike-based learning circuits. (a) Pre-synaptic weightupdate module (present at each synapse). (b) Post-synaptic
learning control circuits (present at the soma).
connections (e.g., see Fig. 6a), or via off-chip reconfigurable
connectivity infrastructures [99]–[103].
A. Recurrent neural networks
In the most general Recurrent Neural Network (RNN) each
neuron is connected to every other neuron (fully recurrent network). Unlike feed-forward networks, the response of RNNs
to the input does not only depend on the external input but
also on their internal dynamics, which in turn is determined by
the connectivity profile. Thus, specific changes in connectivity,
for example through learning, can tune the RNN behavior,
which corresponds to the storage of internal representations of
different external stimuli. This property makes RNNs suitable
for implementing, among other properties, associative memo-
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
8
Spiking inputs
Spiking output
Projections
Soma block
inhibitory
excitatory
Synapse and
learning block
Homeostatic adaptation
Neurons
Synaptic
scaling block
ries [81], working memory [104], context-dependent decision
making [30].
There is reason to believe that, despite significant variation
across cortical areas, the pattern of connectivity between
cortical neurons is similar throughout neocortex. This fact
would imply that the remarkably wide range of capabilities of
the cortex are the results of a specialization of different areas
with similar structures to the various tasks [105], [106]. An
intriguing hypothesis about how computation is carried out by
the brain is the existence of a finite set of computational primitives used throughout the cerebral cortex. If we could identify
these computational primitives and understand how they are
implemented in hardware, then we would make a significant
step toward understanding how to build brain-like processors.
There is an accumulating body of evidence that suggests that
one potential computational primitive consists of a RNN with
a well defined excitatory/inhibitory connectivity pattern [106]
typically referred as soft Winner-Take-All (sWTA) network.
In sWTA neural networks, group of neurons compete with
each other in response to an input stimulus. The neurons with
highest response suppress all other neurons to win the competition. Competition is achieved through a recurrent pattern
of connectivity involving both excitatory and inhibitory connections. Cooperation between neurons with similar response
properties (e.g., close receptive fields or stimulus preference)
is mediated by excitatory connections. Competition and cooperation make the output of an individual neuron depend on the
activity of all neurons in the network and not just on its own
input [107]. As a result, sWTAs perform not only common
linear operations but also complex non-linear operations [108].
The linear operations include analog gain (linear amplification
of the feed-forward input, mediated by the recurrent excitation
and/or common mode input), and locus invariance [109]. The
non-linear operations include non-linear selection [110]–[112],
signal restoration [13], [111], and multi-stability [110], [112].
The computational abilities of these types of networks
are of great importance in tasks involving feature-extraction,
signal restoration and pattern classification problems [113].
For example, localized competitive interactions have been used
to detect elementary image features (e.g., orientation) [114],
inhibitory
excitatory
(a)
AER INPUT
AER INPUT
Fig. 5: Silicon neuron diagram. This is a schematic representation of a typical circuital block comprising multiple synapse
blocks, an I&F soma block, and a homeostatic plasticity
control block. The synapses receive input spikes, integrate
them and and convey the resulting currents to the soma. The
soma integrates these currents and produces output spikes with
a mean rate that is proportional to the total net input current.
Synapse circuits can implement both local plasticity mechanisms to change their efficacy, and global scaling mechanisms
via additional homeostatic control block.
I
I
E
I
I
E
I
I
E
I
I
E
I
I
E
I
I
E
I
I
E
E
E
E
E
E
E
I
E
E
E
E
E
E
I
E
E
E
E
E
E
I
E
E
E
E
E
E
I
E
E
E
E
E
E
I
E
E
E
E
E
E
I
E
E
E E E
E E E
AER OUTPUT
(b)
Fig. 6: sWTA network topology. (a) Schematic representation
of the connectivity pattern of the sWTA network. These
connections are implemented by synapses with hardwired
connections to pre- and post-synaptic neurons. Empty circles
represent excitatory neurons and the filled circle represents the
global inhibitory neuron. Solid/dashed lines represent excitatory/inhibitory connections. Connections with arrowheads are
mono-directional, all the others are bidirectional. Only 8 excitatory neurons are shown for simplicity. (b) Chip architecture.
Squares represent excitatory (E) and inhibitory (I) synapses,
small unlabeled trapezoids represent I&F neurons. The I&F
neurons transmit their spikes off-chip and/or to locally connected synapses implementing the network topology depicted
in (a). Adapted from [117].
[115]. In these networks, each neuron represents one feature
(e.g., vertical or horizontal orientation); when a stimulus is
presented the neurons cooperate and compete to enhance the
response to the features they are tuned to and to suppress
background noise. When sWTA networks are used for solving
classification tasks, common features of the input space can be
learned in an unsupervised manner. Indeed, it has been shown
that competition supports unsupervised learning because it
enhances the firing rate of the neurons receiving the strongest
input, which in turn triggers learning on those neurons [116].
B. Distributed multi-chip networks
The modularity of the cortex described in the theoretical
works and suggested by the experimental observations above
mentioned, constitutes a property of great importance related
to the scalability of the system. If we understood the principles by which such computational modules are arranged
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
together and what type of connectivity allows for coherent
communication also at large distances, we would be able
to build scalable systems, i.e., systems whose properties are
qualitatively reproduced at all scales.
The idea of modularity poses some technological questions
as to how the communication between the systems should
be implemented. Large VLSI networks of I&F neurons can
already be implemented on single chips, using today’s technology. However implementations of pulse-based neural networks
on multi-chip systems offer greater computational power and
higher flexibility than single-chip systems and constitute a
tool for the exploration of the properties of scalability of
the neuromorphic systems. Because inter-chip connectivity
is limited by the small number of input-output connections
available with standard chip packaging technologies, it is
necessary to adopt time-multiplexing schemes for constructing
large multi-chip networks. This scheme should also allow for
an asynchronous type of communication, where information is
transmitted only when available and computation is performed
only when needed in a distributed, non-clocked manner.
In recent years, we have witnessed the emergence of a
new asynchronous communication standard that allows analog
VLSI neurons to transmit their activity across chips using
pulse-frequency modulated signals (in the form of events,
or spikes). This standard is based on the Address Event
Representation (AER) communication protocol [12]. In AER
input and output signals are real-time asynchronous digital
pulses (events or spikes) that carry analog information in their
temporal relationships (inter-spike intervals). If the activity of
the VLSI neurons is sparse and their firing rates are biologically plausible (e.g., ranging from a few spikes per second to
a few hundred spikes per second), then it is possible to tradeoff space with speed very effectively, by time-multiplexing a
single (very fast) digital bus to represent many (very slow)
neuron axons. For example, it has been recently demonstrated
how these time-multiplexing schemes can sustain more then
60 M events/sec, representing the synchronous activity of 1 M
neurons firing at a rate of 60 Hz [99], [118]. In general,
AER communication infrastructures provide the possibility
to implement arbitrary custom multi-chip architectures, with
flexible connectivity schemes. Address events can encode
the address of the sending node (the spiking neuron) or of
the receiving one (the destination synapse). The connectivity
between different nodes is implemented by using external
digital components and is typically defined as a look-up table
with source and destination pairs of addresses, or by more
resource-efficient schemes e.g., using multicast or multi-stage
routing [6], [119], [120]. This asynchronous digital solution
permits flexibility in the configuration (and re-configuration)
of the network topology, while keeping the computation analog
and low-power at the neuron and synapse level.
To handle cases in which multiple sending nodes attempt
to transmit their addresses at exactly the same time (event
collisions), on-chip digital asynchronous arbitration schemes
have been developed [12], [118], [121]. These circuits work by
queuing colliding events, so that only one event is transmitted
at a time. Multiple colliding events are therefore delayed
by a few nano-seconds or fractions of microseconds. For
9
neuromorphic architectures that use biologically plausible time
constants (i.e., of the order of milliseconds), these delays are
negligible and do not affect the overall performance of the
network. For example, assuming a tolerance of 1 ms jitter [122]
it is possible to process up to 4 K coincident input events
without introducing sensible delays, even with an outdated
350 nm CMOS technology [102]. On the other hand, in
accelerated-time systems, such as those proposed in [7] whose
circuits operate at 104 the speed of their biological counterpart,
communication delays are much more critical, because their
duration does not scale. In general, the performance of any
AER neuromorphic system will be bound by communication
memory and bandwidth constraints, which trade-off the speed
of the neural processing elements with the size of the network
that can be implemented.
C. A SW/HW echo-system
In order to promptly explore the computational properties
of different types of large-scale multi-chip computational
architectures, it is important to develop a dedicated HW and
SW infrastructure, which allows a convenient, user-friendly
way to define, configure, and control in real-time the properties
of the HW [123], [124] spiking neural networks, as well as
a way to monitor in real-time their spiking and non-spiking
activity.
The definition of a SW infrastructure for neuromorphic
systems pertains to an issue of increasing importance. Indeed,
as reconfigurable neuromorphic platforms are scaled to larger
sizes, it is necessary to develop efficient tools to interpret
the neural network model, e.g., through programming or
scripting languages, and configure the hardware parameters
correspondingly for the neural and synaptic dynamics and for
the events routing. Hence, the SW should provide means to
configure, control, interact and monitor the electronic hardware. Fortunately, while the specific electronic implementation
of each neuromorphic system can differ substantially, several
common properties can be identified, such as the use of
an AER scheme for communication. Therefore a SW echosystem can be defined to assemble and control the system in a
modular, fully reconfigurable way. In this respect, several SW
interfaces for neuromorphic and neuro-computing platforms
have already been developed. The scopes of these tools are
diverse and so are their peculiarities due to the specificities
of the corresponding system. Both digital neuro-computing
platforms and analog neuromorphic systems typically require
a “neuromorphic compiler” able to parse the network topology
and configure correspondingly memories, processors or digital
interfaces to properly simulate the neural and synaptic dynamics and route the spiking events through the network [125]–
[128]. On top of the compilers, a number of SW frameworks
have been developed as scripting and programming languages
for neural networks at the level of the single network elements,
e.g., neurons, synapses and connectivity [123] and also including a system-level description for building large-scale, brain
simulators [129].
A promising example of an open-source SW framework
that interprets generalized hardware specification files and
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
constructs an abstract representation of the neuromorphic
devices compatible with high-level neural network programming libraries is available at http://inincs.github.com/pyNCS/.
This framework is based on reconfigurable and extensible
Application Programming Interfaces (APIs) and includes a
high-level scripting front-end for defining neural networks. It
constitutes a bridge between applications using abstract resources (i.e., neurons and synapses) and the actual processing
done at the hardware level through the management of the
system’s resources, much like a kernel in modern computers [130], and it is compatible with most existing software. The
HW and SW infrastructure can be complemented with tools
for dynamic parameter estimation methods [131], [132] as
well as automated methods for measuring and setting circuitlevel parameters using arbitrary cost-functions at the network
level [124].
VII. E XPERIMENTAL RESULTS
The circuits and architectures described in this paper have
been designed and developed over the course of several years.
Therefore the experimental data presented in this Section
has been collected from multiple neuromorphic VLSI devices
and systems. The results presented demonstrate the correct
behavior of the circuits described in the previous Sections.
10
Fig. 7: Membrane potential of I&F neuron in response to a
50 Hz pre-synaptic input spike train for different values of
short-term depression adaptation rate, which is controlled by
Vstd bias (see Fig. 3). The dashed trace in background corresponds to the response without STD. Black dots correspond
to input spike-times.
A. Synaptic and neural dynamics
To show the combined effect of synaptic and neural dynamics, we stimulated a silicon neuron via an excitatory DPI
synapse circuit, while sweeping different Short-Term Depression (STD) parameter settings. The typical phenomenology of
STD manifests as a reduction of EPSC amplitude with each
presentation of a pre-synaptic spike, with a slow (e.g., of the
order of 100 ms) recovery time [133]. In Fig. 7 we plot the
neuron’s membrane potential Vmem during the stimulation of
one of its excitatory synapses with a regular pre-synaptic input
spike train of 50 Hz, for different STD adaptation settings.
Small parameter settings for the STD bias voltage have no
or little effect. But for larger settings of this bias voltage the
effect of STD is prominent: the synaptic efficacy decreases
with multiple input spikes to a point in which the net input
current to the soma becomes lower than the neuron’s leak
current, thus making the neuron membrane potential decrease,
rather than increase over time.
Another important adaptation mechanism discussed in Section III, is that of spike-frequency adaptation. To show the
effect of this mechanism, we set the relevant bias voltages
appropriately, stimulated the silicon neuron with a constant
input current, and measured it’s membrane potential. Figure 8
shows an example response to the step input current, in
which Vlkahp = 0.05V , Vthrahp = 0.14V , Vahp = 2.85V . As
shown, we were able to tune the adaptation circuits in a
way to produce bursting behavior. This was achieved by
simply increasing the gain of the negative feedback adaptation
mechanism (Vthrahp > 0). This is equivalent to going from an
asymptotically stable regime to a marginally stable one, that
produces ringing in the adaptation current Iahp , which in turn
produces bursts in the neuron’s output firing rate. This was
Fig. 8: Silicon neuron response to a step input current, with
spike frequency adaptation mechanism enabled and parameters
tuned to produce bursting behavior. The figure inset represents
a zoom of the data showing the first 6 spikes. Adapted
from [49].
possible due to the flexibility of the DPI circuits, which allow
us to take advantage of the extra control parameter Vthrahp ,
in addition to the adaptation rate parameter Vahp , and the
possibility of exploiting its non-linear transfer properties as
described in Section IV, without requiring extra circuits or
dedicated resources that alternative neuron models have to
use [32], [57], [58].
B. Spike-based learning
In this section we present measurements from the circuits
implementing the STDP learning mechanism described in Section V. To stimulate the synapses we generated pre-synaptic
input spike trains with Poisson distributions. Similarly, the
post-synaptic neuron was driven by a current produced via
a non-plastic synapse (a DPI circuit with a constant synaptic
weight bias voltage) stimulated by software-generated Poisson
spike trains. These latter inputs are used to drive the I&F
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
Fig. 9: Stochastic transitions in synaptic states. The non-plastic
synapse is stimulated with a Poisson distributed spikes train.
The neuron fires at an average rate of 30 Hz. The presynaptic input (Vpre ) is stimulated with Poisson distributed
spike trains with a mean firing rate of 60 Hz. The updates in
the synaptic weight produced an LTD transition that remains
consolidated. VH and VL show the potentiated and depressed
levels respectively while w denotes the synaptic weight, and
θ the bi-stability threshold. Adapted from [85].
neuron towards different activity regimes which regulate the
probabilities of synaptic transitions [25], [134], effectively
modulating the learning rate in unsupervised learning conditions, or acting as teacher signals in supervised learning
conditions.
The Poisson nature of the spike-trains used in this way
represents the main source of variability required for implementing stochastic learning [83], [84]. In Fig. 9 we show
measurements from a stochastic learning experiment in which
the neuron is driven to a regime where both potentiation and
depression are possible but depression has a higher probability
to occur. As shown, the weight voltage undergoes both positive
and negative changes, depending on the timing of the input
spike and the state of the post-synaptic neuron (as explained
in Section V). In addition, the weight voltage is slowly driven
toward one of the two stable states, depending on whether it
is above or below the threshold θ (where θ corresponds to
the voltage Vthw of Fig. 4a). Long-term transitions occur when
a series of pre-synaptic spikes arrive in a short time-frame
causing the weight to cross the threshold θ . As a consequence,
the probability of synaptic state transitions depends on the
probability that such events occur, hence it depends on the
firing rate of the pre-synaptic neuron [82], [89]. In the case of
the experiment of Fig. 9 an LTD transition has occurred upon
the presentation of an input stimulus of 60 Hz for 400 ms.
In conclusion, the bi-stability of the synapses and the spikebased plasticity concur in a mechanism that (1) ensures that
only a random fraction of the stimulated bi-stable synapses
undergo long-term modifications and (2) that synaptic states
are resilient to changes due to spontaneous activity, thus
increasing the robustness to noise.
If Fig. 10a we show the results of another stochastic learning
experiment in which we stimulated the post-synaptic neuron
11
with a high-frequency Poisson-like spike train through a nonplastic excitatory input synapse, in order to produce Poissonlike firing statistics in the output. The dashed line on the Vmem
plot represents the learning threshold voltage Vthm of Fig. 4b.
The VUP (active low) and VDN (active high) signals are the
same shown in Fig. 4b and represent the currents that change
the synaptic values when triggered by pre-synaptic spikes.
They can be considered as eligibility traces that enable the
weight update mechanism when they are active.
In Fig. 10b we show the results of an experiment where
we trained a matrix of 28 × 124 = 3472 plastic synapses,
constituting the total input of a neuron, with multiple presentations of the same input pattern representing the “INI”
acronym. Initially all the neuron’s input synaptic weights
are set to their low state (black pixels). Then, the postsynaptic neuron is driven by a teacher signal that makes it fire
stochastically with a mean rate of 40 Hz. At the same time,
input synapses are stimulated according to the image pattern:
in the input image (top left image), each white pixel represents
a Poisson spike train of 55 Hz, sent to the corresponding
synapse; similarly, each black pixel represents a low rate spike
train (5 Hz) which is transmitted to its corresponding synapse.
Because the probability of LTP depends on the pre-synaptic
firing rate, elements of the input matrix that correspond to a
white pixel have are more likely to make a transition to the
potentiated state compared to the other ones. Because of the
stochastic nature of the input patterns, only a random subset
of synapses undergoes LTP, leaving room available to store
other memories. By repeating the presentation of the input
pattern multiple times, this pattern gets gradually stored in the
synaptic matrix. The bottom left image of Fig. 10b represents
the synaptic matrix at the end of the experiment. Furthermore,
the stop-learning mechanism described in Sec. V causes a drop
in the number of synapses that undergo LTP because as the
pattern is stored in the memory the post-synaptic firing rate
increases (Fig. 10c).
The above experiments demonstrate the properties of the
learning circuits implemented in the VLSI chips. In a feedforward configuration, the neuron can be controlled by an
external spiking teacher signal, which indirectly controls the
transition probabilities. This “perceptron-like” configuration
allows the realization of supervised learning protocols for
building real-time classification engines. But, as opposed to
conventional perceptron-like learning rules, the spike-triggered
weight updates implemented by these circuits overcome the
need for an explicit control (e.g., using error back-propagation)
on every individual synapse. In “Hopfield-network” like
RNN configurations the same neuron and plasticity circuits
can implement Attractor Neural Network (ANN) learning
schemes [9], [135], exploiting the neural network dynamics to
form memories through stochastic synaptic updates, without
the need for explicit random generators at each synapse.
C. sWTA networks of I&F neurons
Two characteristic features of sWTA networks that make
them ideal building blocks for cognitive systems are their
ability to selectively enhance the contrast between localized
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
12
(a)
(b)
(c)
Fig. 10: Stochastic learning. (a) Single neuron stochasticity.
Traces from a VLSI multi-neuron chip with I&F neurons and
plasticity circuits as in Fig. 4a. The VUP and VDN signals (top
traces) are set by the circuits in Fig. 4b. A Poisson spiketrain of high firing rate is sent to the excitatory synapse of a
I&F neuron whose Vmem trace is reported in the lower trace.
The strong input current generated by the synapse has been
compensated by a strong leakage current (Vleak = 0.39 V). This
parameter choice allows to exploit the stochasticity of the input
spike-trains to produce the highly irregular dynamics of Vmem .
The non-ideal rounding in the rising part of the VUP trace has
negligible effects on the synaptic weight given the exponential
nature of the current generated through transistor ML3 of
Fig. 4a. (b) An image of the “INI” acronym is converted
into a series of Poisson spike-trains and gradually stored in
the memory by repeated presentations. See text for details. (c)
Normalized frequency of occurrence of LTP transitions during
the experiment of Fig. b, fitted by an exponential function
(dashed line).
inputs and to exhibit activity that persists even after the input
stimulus has disappeared. We configured the local hardwired
connectivity of a multi-neuron chip to implement an sWTA
network and carried out test experiments to show both selective
amplification and state-dependent computation. Specifically,
we configured a chip comprising a network of 128 I&F
neurons with local nearest neighbor excitatory connectivity
and global inhibition: each neuron was configured to excite its
first nearest neighbors, its second neighbors and a population
of four global inhibitory neurons (the top four neurons in the
Fig. 11: Selective amplification experiments. The network is
stimulated in two regions, one centered around unit 20 and
the other around unit 60, with Poisson spike trains of mean
firing rate 180 Hz and 240 Hz. The figures show the networks
response to these inputs (black) and their respective steady
state firing rates on the right panels (calculated for time
> 500 ms). Neurons 124 to 127 are the 4 inhibitory neurons
of the soft WTA network. In the right and left panel the input
amplitudes are swapped. The results show smooth activity
profiles that are invariant to input swapping, demonstrating
that the mismatch in the local weights has been partially
compensated. Adapted from [136].
array of 128 neurons). In a first experiment, we calibrated the
settings and input stimuli to minimize the effect of device
mismatch, following the event-based techniques described
in [124], [131] and stimulated the network with two distinct
regions of activation, centered around units 20 and 60 (see
shaded areas in Fig. 11). In one case the top region had
a higher mean firing rate than the bottom one and in the
other case the bottom region had a higher activation (see top
and bottom plots in Fig. 11 respectively). As expected from
theory [108], [109], [111], the population of silicon neurons
receiving the strongest input won the competition, enhancing
its activity by means of the local recurrent connections, while
suppressing the activity of the competing population via the
global inhibitory connections (selective amplification feature).
In a second experiment we demonstrate the behavior of a
sWTA architecture used to construct state-holding elements,
which are the basic blocks for building Finite State Machines
(FSMs) using spiking neurons, and in which the FSM states
are represented by sub-populations of neurons. The network
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
13
silent, and the top population remains active, in a self-sustained
activity regime. In full FSM systems the state transition signals
would be produced by other neuronal populations (transition
populations) responding to both incoming input stimuli and to
neurons representing the current state. A complete description
and analysis of these neural network based FSMs is presented
in [29], and working examples of multi-neuron chips implementing spiking FSMs are described in [131], [132].
VIII. D ISCUSSION
Fig. 12: Finite State Machine state-holding behavior using a
VLSI sWTA architecture. States are represented by two recurrently connected populations of I&F neurons using the hardwired, on-chip connectivity. Population 1 (bottom half of the
raster plot) is stimulated by synthesized Poisson spike trains
for the initial 500 ms. It’s activity persists due to the recurrent
excitatory connectivity, until population 2 (top half of the
raster plot) is stimulated. The width and position of the subpopulations depend on the properties of the local connectivity
and on their variability. Line-plots superimposed to the rasterplot represent the mean firing-rates computed across each
population. The colored bars below the plot represent input
stimulus presentations. Input stimuli are composed of Poisson
spike trains of 200 Hz lasting for 500 ms, and are applied to
all the neurons of one population. The higher variability in
the output, e.g., compared with Fig. 11, is due to the absence
of mismatch compensation techniques, deliberately omitted to
highlight the differences.
topology supporting the FSM functionality and used in the following experiments resembles the ones of ANN with discrete
or line-attractors. As mentioned in the previous sections, this
type of networks can support a diverse range of functionalities
and have employed in hardware implementation, e.g., for headdirection tracking [137] and memory recall [9]. In particular
we concentrated our experiments on demonstration of two
of their main properties useful for implementing the FSM,
namely selective amplification and state-switching due to
external inputs.
In this experiment we present localized and transient inputs
to two groups of neurons using synthetically generated Poisson
trains (see Fig. 12). After the presentation of each input
stimulus the activity of the stimulated population persists, reverberating in time, by means of the local recurrent excitatory
connectivity. Note that, because of the global competition, only
a sub-set of the stimulated neurons remains active. To obtain
the results shown in Fig. 12, we first stimulated the bottom
population for 500 ms, and then after subsequent 500 ms we
stimulated the top population. When the second stimulus is
applied a “state transition” is triggered: as the top population
becomes active the bottom one is suppressed. When the second
stimulus is removed, the bottom population is completely
The set of low-power hybrid analog/digital circuits presented in the previous sections can be used as basic building blocks for constructing adaptive fully-parallel, real-time
neuromorphic architectures. While several other projects have
already developed dedicated hardware implementations of
spiking neural networks, using analog [4], digital [23], [138]
and mixed mode analog/digital [2], [8] approaches few [5],
[14], [139]–[141] follow the neuromorphic approach originally proposed in the early nineties [11]. The foundations of
this neuromorphic approach were established by pointing out
that the implementation of compact and low-power hardware
models of biological systems requires the use of transistors in
the sub-threshold analog domain and the exploitation of the
physics of the VLSI medium. We argue that the circuits and
architectures presented here adhere to this approach and can
therefore be used to build efficient biophysically realistic realtime neural processing architectures and autonomous behaving
systems.
A. Device mismatch and noise
One common criticism to this sub-threshold analog VLSI
design approach is that circuits operating in this domain
have a high degree of noise. However sub-threshold currentmode circuits have lower noise energy (noise power times
bandwidth), and superior energy efficiency (bandwidth over
power) than above-threshold ones [142], [143]. Another common criticism is that device mismatch in sub-threshold circuits
is more prominent than in above threshold circuits. While this
observation is correct, device mismatch is a critical problem
in any analog VLSI implementation of neural networks (e.g.,
see the post-calibration neuronal variability measurements
of above-threshold accelerated time silicon neuron circuits,
presented in [10]). In principle it is possible to minimize
the effect of device mismatch following standard electrical
engineering approaches and adopting appropriate analog VLSI
design techniques, however we argue that it is not necessary
to adopt aggressive mismatch reduction techniques in the
type of neuromorphic systems we propose: these techniques
would lead to very large transistor or circuit designs, which
could in turn significantly reduce the number of neurons and
synapses integrated onto a single chip (see for example [31],
where a whole VLSI device was used to implement a single
synapse). Rather than attempting to minimize mismatch effects
using brute-force engineering techniques at the circuit design
level, the neuromorphic engineering approach we promote
in this work aims to address these effects at the network
and system level, with collective computation, adaptation, and
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
feedback mechanisms. For example, the plasticity mechanisms
presented in Section V are intrinsically robust to mismatch
by design, and do not require precisely matched transistors.
Moreover, it has been shown how both short- and longterm plasticity mechanisms can be effectively used to reduce
the effects of device mismatch in VLSI circuits [68], [144],
and how homeostatic plasticity mechanisms can be used to
compensate for large changes in the signals affecting the
operation of the neurons in multi-neuron VLSI systems [75].
In addition, the approach of building distributed multi-chip
systems interfaced among each other via the AER protocol
(e.g., see Section VI-B), lends itself well to the adoption
of event-based mismatch reduction techniques, such as the
one proposed in [136], that can be effective even for very
large-scale systems, (e.g., comprising 1 million silicon neurons) [145]. In addition to being useful for compensating
mismatch effects across neurons, homeostatic synaptic scaling circuits, such as the ones described in Section IV-C,
can provide another approach to compensating the effects
of temperature drifts, complementing dedicated sub-threshold
bias generator approaches [146], [147]. In summary, this
neuromorphic approach makes it possible to tolerate noise,
temperature, and mismatch effects at the single device level by
exploiting the adaptive features of the circuits and architectures
designed, leading to robustness at the system level.
B. Exploiting variability and imprecision
The strategy proposed by this approach essentially advocates the construction of distributed and massively parallel
computing systems by integrating very compact, but inaccurate
and inhomogeneous circuits into large dense arrays, rather than
designing systems based on small numbers of very precise, but
large and homogeneous computing elements. Indeed, intrinsic
variability and diverse activation patterns are often identified
as fundamental aspects of neural computation for information
maximization and transmission [30], [148]–[150]. The strategy
of combining large numbers of variable and imprecise computing elements to carry out robust computation is also followed
by a wide set of traditional machine learning approaches.
These approaches work on the principle of combining the
output of multiple inaccurate computational modules that have
slightly different properties, to optimize classification performances and achieve or even beat the performances of single
accurate and complex learning systems [151], [152]. A set of
similar theoretical studies showed that the coexistence of multiple different time-scales of synaptic plasticity (e.g., present
due to mismatch in the time-constants of the DPI synapse
circuits) can dramatically improve the memory performance
of ANN [153]. The coexistence of slow and fast learning
processes has been shown to be crucial for reproducing the
flexible behavior of animals in context-dependent decisionmaking (i.e., cognitive) tasks and the corresponding single cell
recordings in a neural network model [154].
C. Towards autonomous cognitive systems
Building cognitive systems using noisy and inhomogeneous
subthreshold analog VLSI circuits might appear as a daunting
14
task. The neural circuits and architectures presented in this
paper represent a useful set of building blocks paving the
way toward this goal. These circuits, as well as analogous
one proposed in the literature [155], have been used to build
compact, low-power, scalable, computing systems that can
interact with the environment [3], [145], [156], learn about
the input signals they have been designed to process [85], and
exhibit adaptive abilities analogous to those of the biological
systems they model [75], [157], [158]. We showed in this
paper how the sWTA networks and circuits presented can
implement models of working memory and decision making, thanks to their selective amplification and reverberating
activity properties, which are often associated to high-level
cognitive abilities [21]. Multi-chip systems employing these
architectures can reproduce the results of a diverse set of
theoretical studies based on models of sWTA and ANN to
demonstrate cognitive properties: for example, Schöner and
Sandamirskaya [28], [159] link the types of neural dynamics
described in Section VI to cognition by applying similar network architectures to sensory-motor processes and sequence
generation; Rutishauser and Douglas [29] show how the sWTA
networks described in this paper can be configured to implement finite state machines and conditional branching between
behavioral states [160]; Rigotti and colleagues [30], [161]
describe neural principles, compatible with the ones implemented by the circuits described in Section V, for constructing
recurrent neural networks able to produce context-dependent
behavioral responses; Giulioni and colleagues [9] demonstrate
working memory in a spiking neural network implemented
using the same type of silicon neuron circuits and plasticity
mechanisms [135] described in Sections III and V.
We recently demonstrated how the circuits and networks
presented in Sections III, IV, and VI can be used to synthesize
cognition on neural processing systems [20]. Specifically, the
neuromorphic multi-chip system proposed was used to carry
out a context-dependent task selection procedure, analogous
to the sensory-motor tasks adopted to probe cognition in primates. This is a concrete example showing how neuromorphic
systems, built using variable and imprecise circuits, can indeed
be configured to express cognitive abilities comparable to
those described in [21], [30].
D. Challenges and progress in Neuromorphic Engineering
Many years have passed since the first publication on neuromorphic electronic systems [11], and remarkable progress has
been made by the small but vibrant Neuromorphic Engineering
(NE) community [162], [163]. For example the NE community
has mastered the art of building real-time sensory-motor
reactive systems, by interfacing circuits and networks of the
type described in this paper with neuromorphic event-based
sensors [164]; new promising neural-based approaches have
been proposed that link neuromorphic systems to machine
learning [165]–[169]; substantial progress has been made in
the field of neuromorphic robots [170]; and we are now able
to engineer both large scale neuromorphic systems (e.g., that
comprise of the order of 106 neurons [171]) and complex
multi-chip neuromorphic systems (e.g., that can exhibit cognitive abilities [20]). However, compared to the progress made in
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
more conventional standard engineering and technology fields,
the rate of progress in NE might appear to be disappointingly
small. On one hand, this is due to the fact that NE is still a
small community involving a small number of research groups
worldwide (e.g., compared to the number of engineers that
are assigned to the industrial development of new Graphical
Processing Units (GPUs) or Central Processing Units (CPUs)),
which lacks the technological infrastructure for automatized
design, verification and configuration tools available for conventional digital Integrated Circuit (IC) development. On the
other hand, scaling and engineering challenges are not the
main issue: the major limiting factor that hinders the fast
development of neuromorphic engineering is related to our
limited understanding of brain function and neural computation, a concept that Carver Mead himself highlighted already
over 20 years ago in a video interview (that we transcribe
here):
“I think at the present time we have enough technology to build anything we could imagine. Our
problem is, we do not know what to imagine. We
don’t understand enough about how the nervous
system computes to really make more complete
thinking systems.”
Progress on theoretical and computational neuroscience is
accelerating dramatically, also thanks to large-scale funding
initiatives recently announced in both Europe and the United
States [172], [173]. At the same time, an increasing number
of companies is beginning to support research and development in brain-inspired computing technologies [174]–[177].
Supported by these new initiatives, progress in NE is beginning
to accelerate as well [178]. In this perspective, reaching the
ambitious goal of building autonomous neuromorphic systems
able to interact with the environment in real-time and to
express cognitive abilities is within the realm of possibility.
To reach this goal, however, it is important to follow a truly
multi-disciplinary approach where neuromorphic engineering
serves as a medium for the exploration of robust principles of
brain computation and not only as a technology platform for
the simulation of neuroscience models.
IX. C ONCLUSIONS
In this paper we proposed circuit and system solutions
following the neuromorphic approach originally proposed
in [11] for building autonomous neuromorphic cognitive systems. We presented an in-depth review of such types of
circuits and systems, with tutorial demonstrations of how to
model neural dynamics in analog VLSI. We discussed the
problems that arise when attempting to implement spikebased learning mechanisms in physical systems and proposed
circuit solutions for solving such problems. We described
examples of recurrent neural network implementations that
can be used to implement decision making and workingmemory mechanisms, and argued how, together with the
circuits described in the previous sections, they can be used
to implement cognitive architectures. We discussed about the
advantages and disadvantages of the approach followed (e.g.,
for the subthreshold regime of operation or for mismatch
15
in analog subthreshold circuits), and proposed system-level
solutions that are inspired by the strategies used in biological
nervous systems. Finally, we provided an assessment of the
progress made in the NE field so far and proposed strategies
for accelerating it and reaching the ambitious goal of building
autonomous neuromorphic cognitive systems.
ACKNOWLEDGMENTS
Many of the circuits and concepts presented here were
inspired by the ideas and work of Rodney Douglas, Misha
Mahowald, Kevan Martin, Matthew Cook, and Stefano Fusi.
The HW/SW infrastructure used to characterize the chips
throughout the years and build multi-chip systems was developed in collaboration with Paolo Del Giudice, Vittorio Dante,
Adrian Whatley, Emre Neftci, Daniel Fasnacht, and Sadique
Sheik. We acknowledge also Tobi Delbruck, Shih-Chii Liu and
all our other colleagues at the Institute of Neuroinformatics
for fruitful discussions and collaborations. We would like to
thank the reviewers for their constructive comments. This work
was supported by the EU ERC Grant “neuroP” (257219), the
EU FET Grant “SI-CODE” (284553), and by the Excellence
Cluster 227 (CITEC, Bielefeld University).
R EFERENCES
[1] X. Jin, et al., “Modeling spiking neural networks on SpiNNaker,”
Computing in Science & Engineering, vol. 12, no. 5, pp. 91–97,
September-October 2010.
[2] J. Schemmel, J. Fieres, and K. Meier, “Wafer-scale integration of
analog neural networks,” in Proceedings of the IEEE International Joint
Conference on Neural Networks, 2008.
[3] R. Silver, K. Boahen, S. Grillner, N. Kopell, and K. Olsen, “Neurotech
for neuroscience: unifying concepts, organizing principles, and emerging tools,” Journal of Neuroscience, vol. 27, no. 44, p. 11807, 2007.
[4] J. Wijekoon and P. Dudek, “VLSI circuits implementing computational
models of neocortical circuits,” Journal of Neuroscience Methods, vol.
210, no. 1, pp. 93–109, 2012.
[5] S. Brink, S. Nease, and P. Hasler, “Computing with networks of spiking
neurons on a biophysically motivated floating-gate based neuromorphic
integrated circuit,” Neural Networks, 2013.
[6] E. Painkras, et al., “SpiNNaker: A 1-W 18-core system-on-chip for
massively-parallel neural network simulation,” IEEE Journal of SolidState Circuits, vol. 48, no. 8, pp. –, August 2013.
[7] T. Pfeil, et al., “Six networks on a universal neuromorphic computing
substrate,” Frontiers in neuroscience, vol. 7, 2013.
[8] J. M. Cruz-Albrecht, T. Derosier, and N. Srinivasa, “A scalable neural
chip with synaptic electronics using cmos integrated memristors,”
Nanotechnology, vol. 24, no. 38, p. 384011, 2013.
[9] M. Giulioni, et al., “Robust working memory in an asynchronously
spiking neural network realized in neuromorphic VLSI,” Frontiers in
Neuroscience, vol. 5, no. 149, 2012.
[10] M. Schmuker, T. Pfeil, and M. Nawrot, “A neuromorphic network for
generic multivariate data classification,” Proceedings of the National
Academy of Sciences, vol. 111, no. 6, pp. 2081–2086, 2014.
[11] C. Mead, “Neuromorphic electronic systems,” Proceedings of the IEEE,
vol. 78, no. 10, pp. 1629–36, 1990.
[12] M. Mahowald, “VLSI analogs of neuronal visual processing: a synthesis of form and function,” Ph.D. dissertation, Department of Computation and Neural Systems, California Institute of Technology, Pasadena,
CA., 1992.
[13] R. Douglas, M. Mahowald, and C. Mead, “Neuromorphic analogue
VLSI,” Annu. Rev. Neurosci., vol. 18, pp. 255–281, 1995.
[14] T. Horiuchi and C. Koch, “Analog VLSI-based modeling of the primate
oculomotor system,” Neural Computation, vol. 11, no. 1, pp. 243–265,
January 1999.
[15] G. Indiveri and R. Douglas, “ROBOTIC VISION: Neuromorphic vision
sensor,” Science, vol. 288, pp. 1189–1190, May 2000.
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
[16] G. Indiveri, “A neuromorphic VLSI device for implementing 2-D
selective attention systems,” IEEE Transactions on Neural Networks,
vol. 12, no. 6, pp. 1455–1463, November 2001.
[17] C. Bartolozzi and G. Indiveri, “Selective attention in multi-chip
address-event systems,” Sensors, vol. 9, no. 7, pp. 5076–5098, 2009.
[18] M. Lewis, R. Etienne-Cummings, M. Hartmann, A. Cohen, and Z. Xu,
“An in silico central pattern generator: silicon oscillator, coupling,
entrainment, physical computation and biped mechanism control,”
Biological Cybernetics, vol. 88, no. 2, pp. 137–151, 2003.
[19] R. Serrano-Gotarredona, et al., “CAVIAR: A 45k neuron, 5M synapse,
12G connects/s aer hardware sensory–processing– learning–actuating
system for high-speed visual object recognition and tracking,” IEEE
Transactions on Neural Networks, vol. 20, no. 9, pp. 1417–1438,
September 2009.
[20] E. Neftci, et al., “Synthesizing cognition in neuromorphic electronic
systems,” Proceedings of the National Academy of Sciences, vol. 110,
no. 37, pp. E3468–E3476, 2013.
[21] C. Eliasmith, et al., “A large-scale model of the functioning brain,”
Science, vol. 338, no. 6111, pp. 1202–1205, 2012.
[22] A. Cassidy, et al., “Cognitive computing building block: A versatile and
efficient digital neuron model for neurosynaptic cores,” in International
Joint Conference on Neural Networks, IJCNN 2013. IEEE, 2013.
[23] A. Cassidy, J. Georgiou, and A. Andreou, “Design of silicon brains in
the nano-CMOS era: Spiking neurons, learning synapses and neural
architecture optimization,” Neural Networks, 2013.
[24] W. Senn and S. Fusi, “Learning Only When Necessary: Better
Memories of Correlated Patterns in Networks with Bounded
Synapses,” Neural Computation, vol. 17, no. 10, pp. 2106–2138,
2005.
[25] J. Brader, W. Senn, and S. Fusi, “Learning real world stimuli in a neural
network with spike-driven synaptic dynamics,” Neural Computation,
vol. 19, pp. 2881–2912, 2007.
[26] A. Renart, P. Song, and X.-J. Wang, “Robust spatial working memory
through homeostatic synaptic scaling in heterogeneous cortical networks,” Neuron, vol. 38, pp. 473–485, May 2003.
[27] G. Deco and E. Rolls, “Neurodynamics of biased competition and
cooperation for attention: a model with spiking neurons,” Journal of
Neurophysiology, vol. 94, pp. 295–313, 2005.
[28] G. Schöner, Dynamical systems approaches to cognition. Cambridge,
UK: Cambridge University Press, 2008, pp. 101–126.
[29] U. Rutishauser and R. Douglas, “State-dependent computation using
coupled recurrent networks,” Neural Computation, vol. 21, pp. 478–
509, 2009.
[30] M. Rigotti, D. B. D. Rubin, S. Morrison, C. Salzman, and S. Fusi,
“Attractor concretion as a mechanism for the formation of context
representations,” NeuroImage, vol. 52, no. 3, pp. 833–847, 2010.
[31] G. Rachmuth, Z. Shouval, M. Bear, and C.-S. Poon, “A biophysicallybased neuromorphic model of spike rate- and timing-dependent plasticity,” Proceedings of the National Academy of Science, vol. 108, no. 49,
pp. E1266–E1274, December 2011.
[32] J. Wijekoon and P. Dudek, “Compact silicon neuron circuit with spiking
and bursting behaviour,” Neural Networks, vol. 21, no. 2–3, pp. 524–
534, March–April 2008.
[33] J. Schemmel, D. Brüderle, K. Meier, and B. Ostendorf, “Modeling
synaptic plasticity within networks of highly accelerated I&F neurons,”
in International Symposium on Circuits and Systems, (ISCAS), 2007.
IEEE, 2007, pp. 3367–3370.
[34] C. Tomazou, F. Lidgey, and D. Haigh, Eds., Analogue IC design: the
current-mode approach. Stevenage, Herts., UK: Peregrinus, 1990.
[35] E. Drakakis, A. Payne, and C. Toumazou, ““Log-domain state-space”:
A systematic transistor-level approach for log-domain filtering,” IEEE
Transactions on Circuits and Systems II, vol. 46, no. 3, pp. 290–305,
March 1999.
[36] R. Edwards and G. Cauwenberghs, “Synthesis of log-domain filters
from first-order building blocks,” International Journal of Analog
Integrated Circuits and Signal Processing, vol. 22, pp. 177–186, 2000.
[37] S.-C. Liu, J. Kramer, G. Indiveri, T. Delbruck, and R. Douglas,
Analog VLSI:Circuits and Principles. MIT Press, 2002.
[38] T. Yu and G. Cauwenberghs, “Log-domain time-multiplexed realization
of dynamical conductance-based synapses,” in International Symposium on Circuits and Systems, (ISCAS), 2010. Paris, France: IEEE,
June 2010, pp. 2558 –2561.
[39] S. Mitra, G. Indiveri, and R. Etienne-Cummings, “Synthesis of
log-domain integrators for silicon synapses with global parametric
control,” in International Symposium on Circuits and Systems,
(ISCAS), 2010, IEEE. Paris, France: IEEE, 2010, pp. 97–100.
16
[40] A. Destexhe, Z. Mainen, and T. Sejnowski, Methods in Neuronal
Modelling, from ions to networks.
MIT Press, 1998, ch. Kinetic
Models of Synaptic Transmission, pp. 1–25.
[41] C. Bartolozzi and G. Indiveri, “Synaptic dynamics in analog VLSI,”
Neural Computation, vol. 19, no. 10, pp. 2581–2603, Oct 2007.
[42] C. Bartolozzi, S. Mitra, and G. Indiveri, “An ultra low power
current–mode filter for neuromorphic systems and biomedical signal
processing,” in Biomedical Circuits and Systems Conference, (BioCAS),
2006. IEEE, 2006, pp. 130–133.
[43] J. Arthur and K. Boahen, “Recurrently connected silicon neurons with
active dendrites for one-shot learning,” in IEEE International Joint
Conference on Neural Networks, vol. 3, July 2004, pp. 1699–1704.
[44] A. van Schaik and C. Jin, “The tau-cell: a new method for the
implementation of arbitrary differential equations,” in International
Symposium on Circuits and Systems, (ISCAS), 2003. IEEE, 2003,
pp. 569–572.
[45] B. Gilbert, “Translinear circuits: An historical review,” Analog Integrated Circuits and Signal Processing, vol. 9, no. 2, pp. 95–118, March
1996.
[46] J. Arthur and K. Boahen, “Synchrony in silicon: The gamma rhythm,”
IEEE Transactions on Neural Networks, vol. 18, pp. 1815–1825, 2007.
[47] A. van Schaik, C. Jin, T. Hamilton, S. Mihalas, and E. Niebur, “A
log-domain implementation of the Mihalas-Niebur neuron model,” in
International Symposium on Circuits and Systems, (ISCAS), 2010.
Paris, France: IEEE, 2010, pp. 4249–4252.
[48] A. van Schaik, C. Jin, and T. Hamilton, “A log-domain implementation
of the Izhikevich neuron model,” in International Symposium on
Circuits and Systems, (ISCAS), 2010. Paris, France: IEEE, 2010,
pp. 4253–4256.
[49] G. Indiveri, F. Stefanini, and E. Chicca, “Spike-based learning with
a generalized integrate and fire silicon neuron,” in International
Symposium on Circuits and Systems, (ISCAS), 2010. Paris, France:
IEEE, 2010, pp. 1951–1954.
[50] M. Mahowald and R. Douglas, “A silicon neuron,” Nature, vol. 354,
pp. 515–518, 1991.
[51] D. Dupeyron, S. Le Masson, Y. Deval, G. Le Masson, and J.-P. Dom, “A
BiCMOS implementation of the Hodgkin-Huxley formalism,” in Proceedings of the Fifth International Conference on Microelectronics for
Neural, Fuzzy and Bio-inspired Systems; Microneuro’96, MicroNeuro.
Los Alamitos, CA: IEEE Computer Society Press, February 1996, pp.
311–316.
[52] L. Alvado, et al., “Hardware computation of conductance-based neuron
models,” Neurocomputing, vol. 58–60, pp. 109–115, 2004.
[53] M. Simoni, G. Cymbalyuk, M. Sorensen, and R. D. S. Calabrese, “A
multiconductance silicon neuron with biologically matched dynamics,”
Biomedical Circuits and Systems, IEEE Transactions on, vol. 51, no. 2,
pp. 342–354, February 2004.
[54] T. Yu and G. Cauwenberghs, “Analog VLSI biophysical neurons and
synapses with programmable membrane channel kinetics,” Biomedical
Circuits and Systems, IEEE Transactions on, vol. 4, no. 3, pp. 139–148,
June 2010.
[55] E. Izhikevich, “Simple model of spiking neurons,” IEEE Transactions
on Neural Networks, vol. 14, no. 6, pp. 1569–1572, 2003.
[56] R. Brette and W. Gerstner, “Adaptive exponential integrate-and-fire
model as an effective description of neuronal activity,” Journal of
Neurophysiology, vol. 94, pp. 3637–3642, 2005.
[57] S. Mihalas and E. Niebur, “A generalized linear integrate-and-fire
neural model produces diverse spiking behavior,” Neural Computation,
vol. 21, pp. 704–718, 2009.
[58] F. Folowosele, R. Etienne-Cummings, and T. Hamilton, “A CMOS
switched capacitor implementation of the Mihalas-Niebur neuron,” in
Biomedical Circuits and Systems Conference, (BioCAS), 2009. IEEE,
Nov. 2009, pp. 105–108.
[59] P. Livi and G. Indiveri, “A current-mode conductance-based silicon
neuron for address-event neuromorphic systems,” in International
Symposium on Circuits and Systems, (ISCAS), 2009. IEEE, May
2009, pp. 2898–2901.
[60] B. Connors, M. Gutnick, and D. Prince, “Electrophysiological properties of neocortical neurons in vitro,” Jour. of Neurophysiol., vol. 48,
no. 6, pp. 1302–1320, 1982.
[61] R. Jolivet, T. Lewis, and W. Gerstner, “Generalized integrate-and-fire
models of neuronal activity approximate spike trains of a detailed
model to a high degree of accuracy,” Journal of neurophysiology,
vol. 92, pp. 959–976, 2004.
[62] L. Badel, et al., “Dynamic I-V curves are reliable predictors of naturalistic pyramidal-neuron voltage traces,” Journal of Neurophysiology,
vol. 99, pp. 656–666, 2008.
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
[63] R. Naud, T. Berger, W. Gerstner, B. Bathellier, and M. Carandini,
“Quantitative Single-Neuron Modeling: Competition 2009,” Frontiers
in Neuroinformatics, pp. 1–8, 2009.
[64] D. Buonomano, “Decoding temporal information: A model based on
short-term synaptic plasticity,” The Journal of Neuroscience, vol. 20,
pp. 1129–1141, 2000.
[65] R. Zucker and W. Regehr, “Short-term synaptic plasticity,” Annual
Review of Physiology, vol. 64, pp. 355–405, 2002.
[66] C. Rasche and R. Hahnloser, “Silicon synaptic depression,” Biological
Cybernetics, vol. 84, no. 1, pp. 57–62, 2001.
[67] M. Boegerhausen, P. Suter, and S.-C. Liu, “Modeling short-term
synaptic depression in silicon,” Neural Computation, vol. 15, no. 2,
pp. 331–348, Feb 2003.
[68] J. Bill, et al., “Compensating inhomogeneities of neuromorphic VLSI
devices via short-term synaptic plasticity,” Frontiers in computational
neuroscience, vol. 4, 2010.
[69] M. Noack, C. Mayr, J. Partzsch, and R. Schuffny, “Synapse dynamics
in CMOS derived from a model of neurotransmitter release,” in Circuit
Theory and Design (ECCTD), 2011 European Conference on. IEEE,
2011, pp. 198–201.
[70] T. Ohno, et al., “Short-term plasticity and long-term potentiation
mimicked in single inorganic synapses,” Nature Materials, vol. 10,
no. 8, pp. 591–595, 2011.
[71] T. Dowrick, S. Hall, and L. Mcdaid, “Silicon-based dynamic synapse
with depressing response,” Neural Networks and Learning Systems,
IEEE Transactions on, vol. 23, no. 10, pp. 1513–1525, 2012.
[72] L. Abbott, K. Sen, J. Varela, and S. Nelson, “Synaptic depression and
cortical gain control,” Science, vol. 275, no. 5297, pp. 220–223, 1997.
[73] M. Tsodyks and H. Markram, “The neural code between neocortical
pyramidal neurons depends on neurotransmitter release probability,”
Proceedings of the National Academy of Sciences of the USA, vol. 94,
no. 2, pp. 719–723, Jan 1997.
[74] G. Turrigiano, K. Leslie, N. Desai, L. Rutherford, and S. Nelson,
“Activity-dependent scaling of quantal amplitude in neocortical neurons,” Nature, vol. 391, pp. 892–896, February 1998.
[75] C. Bartolozzi and G. Indiveri, “Global scaling of synaptic efficacy:
Homeostasis in silicon synapses,” Neurocomputing, vol. 72, no. 4–6,
pp. 726–731, Jan 2009.
[76] L. Abbott and S. Nelson, “Synaptic plasticity: taming the beast,” Nature
Neuroscience, vol. 3, pp. 1178–1183, November 2000.
[77] D. Amit and S. Fusi, “Constraints on learning in dynamic synapses,”
Network: Computation in Neural Systems, vol. 3, no. 4, pp. 443–464,
1992.
[78] S. Fusi and L. Abbott, “Limits on the memory storage capacity of
bounded synapses,” Nature Neuroscience, vol. 10, pp. 485–493, 2007.
[79] D. Amit, Modeling brain function: The world of attractor neural
networks. Cambridge University Press, 1992.
[80] J. Nadal, G. Toulouse, J. Changeux, and S. Dehaen, “Networks of
formal neurons and memory palimpsests,” EPL (Europhysics Letters),
vol. 1, no. 10, p. 535, 1986.
[81] D. Amit and S. Fusi, “Dynamic learning in neural networks with
material synapses,” Neural Computation, vol. 6, p. 957, 1994.
[82] S. Fusi, “Hebbian spike-driven synaptic plasticity for learning patterns
of mean firing rates,” Biological Cybernetics, vol. 87, pp. 459–470,
2002.
[83] S. Fusi, M. Annunziato, D. Badoni, A. Salamon, and D. Amit, “Spike–
driven synaptic plasticity: theory, simulation, VLSI implementation,”
Neural Computation, vol. 12, pp. 2227–58, 2000.
[84] E. Chicca and S. Fusi, “Stochastic synaptic plasticity in deterministic
aVLSI networks of spiking neurons,” in Proceedings of the World
Congress on Neuroinformatics, ser. ARGESIM Reports, F. Rattay, Ed.
Vienna: ARGESIM/ASIM Verlag, 2001, pp. 468–477.
[85] S. Mitra, S. Fusi, and G. Indiveri, “Real-time classification of
complex patterns using spike-based learning in neuromorphic VLSI,”
Biomedical Circuits and Systems, IEEE Transactions on, vol. 3, no. 1,
pp. 32–42, Feb. 2009.
[86] J. Seo, et al., “A 45nm CMOS neuromorphic chip with a scalable
architecture for learning in networks of spiking neurons,” in Custom
Integrated Circuits Conference (CICC), 2011 IEEE. IEEE, 2011, pp.
1–4.
[87] S. Sheik, et al., “Emergent auditory feature tuning in a real-time
neuromorphic VLSI system,” Frontiers in Neuroscience, vol. 6, no. 17,
2012.
[88] M. Giulioni, M. Pannunzi, D. Badoni, V. Dante, and P. Del Giudice,
“Classification of correlated patterns with a configurable analog VLSI
neural network of spiking neurons and self-regulating plastic synapses,”
Neural Computation, vol. 21, no. 11, pp. 3106–3129, 2009.
17
[89] G. Indiveri, E. Chicca, and R. Douglas, “A VLSI array of low-power
spiking neurons and bistable synapses with spike–timing dependent
plasticity,” IEEE Transactions on Neural Networks, vol. 17, no. 1, pp.
211–221, Jan 2006.
[90] A. B. i Petit and A. Murray, “Synchrony detection and amplification
by silicon neurons with STDP synapses,” IEEE Transactions on Neural
Networks, vol. 15, no. 5, pp. 1296–1304, September 2004.
[91] P. Häfliger, M. Mahowald, and L. Watts, “A spike based learning neuron
in analog VLSI,” in Advances in neuralinformation processing systems,
M. Mozer, M. Jordan, and T. Petsche, Eds. MIT Press, 1997, vol. 9,
pp. 692–698.
[92] M. R. Azghadi, S. Al-Sarawi, D. Abbott, and N. Iannella, “A neuromorphic VLSI design for spike timing and rate based synaptic plasticity,”
Neural Networks, vol. 45, pp. 70–82, 2013.
[93] J. Arthur and K. Boahen, “Learning in silicon: Timing is everything,”
in Advances in Neural Information Processing Systems 18, Y. Weiss,
B. Schölkopf, and J. Platt, Eds. Cambridge, MA, USA: MIT Press,
2006.
[94] R. Gütig and H. Sompolinsky, “The tempotron: a neuron that learns
spike timing–based decisions,” Nature Neuroscience, vol. 9, pp. 420–
428, 2006.
[95] W. Senn, “Beyond spike timing: the role of nonlinear plasticity and
unreliable synapses,” Biol. Cybern., vol. 87, pp. 344–355, 2002.
[96] J. Lisman and N. Spruston, “Postsynaptic depolarization requirements
for ltp and ltd: a critique of spike timing-dependent plasticity,” Nature
Neuroscience, vol. 8, no. 7, pp. 839–841, Jul 2005.
[97] M. Beyeler, N. Dutt, and J. Krichmar, “Categorization and decisionmaking in a neurobiologically plausible spiking network using a stdplike learning rule,” Neural Networks, 2013.
[98] J. Lazzaro, S. Ryckebusch, M. Mahowald, and C. Mead, “Winner-takeall networks of O(n) complexity,” in Advances in neural information
processing systems, D. Touretzky, Ed., vol. 2.
San Mateo - CA:
Morgan Kaufmann, 1989, pp. 703–711.
[99] D. Fasnacht and G. Indiveri, “A PCI based high-fanout AER mapper
with 2 GiB RAM look-up table, 0.8 µs latency and 66 mhz output
event-rate,” in Conference on Information Sciences and Systems, CISS
2011, Johns Hopkins University, March 2011, pp. 1–6.
[100] S. Scholze, et al., “VLSI implementation of a 2.8 gevent/s packet based
AER interface with routing and event sorting functionality,” Frontiers
in Neuroscience, vol. 5, 2011.
[101] D. Fasnacht, A. Whatley, and G. Indiveri, “A serial communication
infrastructure for multi-chip address event system,” in International
Symposium on Circuits and Systems, (ISCAS), 2008. IEEE, May
2008, pp. 648–651.
[102] E. Chicca, et al., “A multi-chip pulse-based neuromorphic infrastructure
and its application to a model of orientation selectivity,” IEEE
Transactions on Circuits and Systems I, vol. 5, no. 54, pp. 981–993,
2007.
[103] F. Gomez-Rodriguez, et al., “AER tools for communications and
debugging,” in International Symposium on Circuits and Systems,
(ISCAS), 2006. IEEE, 2006, pp. 3253–3256.
[104] G. Mongillo, D. Amit, and N. Brunel, “Retrospective and prospective
persistent activity induced by hebbian learning in a recurrent cortical
network,” European Journal of Neuroscience, vol. 18, no. 7, pp. 2011–
2024, 2003.
[105] R. Douglas, K. Martin, and D. Whitteridge, “A canonical microcircuit
for neocortex,” Neural Computation, vol. 1, pp. 480–488, 1989.
[106] R. Douglas and K. Martin, “Neural circuits of the neocortex,” Annual
Review of Neuroscience, vol. 27, pp. 419–51, 2004.
[107] R. Douglas, C. Koch, M. Mahowald, K. Martin, and H. Suarez,
“Recurrent excitation in neocortical circuits,” Science, vol. 269, pp.
981–985, 1995.
[108] R. Douglas and K. Martin, “Recurrent neuronal circuits in the neocortex,” Current Biology, vol. 17, no. 13, pp. R496–R500, 2007.
[109] D. Hansel and H. Sompolinsky, Methods in Neuronal Modeling. Cambridge, MA, USA: MIT Press, 1998, ch. Modeling Feature Selectivity
in Local Cortical Circuits, pp. 499–567.
[110] S. Amari and M. Arbib, “Competition and cooperation in neural nets,”
in Systems Neuroscience, J. Metzler, Ed. Academic Press, 1977, pp.
119–165.
[111] P. Dayan and L. Abbott, Theoretical Neuroscience: Computational and
Mathematical Modeling of Neural Systems. Cambridge, MA, USA:
MIT Press, 2001.
[112] R. Hahnloser, R. Sarpeshkar, M. Mahowald, R. Douglas, and S. Seung,
“Digital selection and analog amplification co-exist in an electronic
circuit inspired by neocortex,” Nature, vol. 405, no. 6789, pp. 947–
951, 2000.
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
[113] W. Maass, “On the computational power of winner-take-all,” Neural
Computation, 2000.
[114] R. Ben-Yishai, R. Lev Bar-Or, and H. Sompolinsky, “Theory of orientation tuning in visual cortex,” Proceedings of the National Academy
of Sciences of the USA, vol. 92, no. 9, pp. 3844–3848, April 1995.
[115] D. Somers, S. Nelson, and M. Sur, “An emergent model of orientation
selectivity in cat visual cortical simple cells,” The Journal of Neuroscience, vol. 15, pp. 5448–65, 1995.
[116] A. Bennett, “Large competitive networks,” Network, vol. 1, pp. 449–62,
1990.
[117] E. Chicca, G. Indiveri, and R. Douglas, “Context dependent
amplification of both rate and event-correlation in a VLSI network
of spiking neurons,” in Advances in Neural Information Processing
Systems 19, B. Schölkopf, J. Platt, and T. Hofmann, Eds., Neural
Information Processing Systems Foundation. Cambridge, MA, USA:
MIT Press, Dec 2007, pp. 257–264.
[118] K. Boahen, “Point-to-point connectivity between neuromorphic chips
using address-events,” IEEE Transactions on Circuits and Systems II,
vol. 47, no. 5, pp. 416–34, 2000.
[119] S. Carrillo, et al., “Hierarchical network-on-chip and traffic compression for spiking neural network implementations,” in Networks on Chip
(NoCS), 2012 Sixth IEEE/ACM International Symposium on. IEEE,
2012, pp. 83–90.
[120] S. Moradi, N. Imam, R. Manohar, and G. Indiveri, “A memory-efficient
routing method for large-scale spiking neural networks,” in Circuit
Theory and Design, (ECCTD), 2013 European Conference on. IEEE,
2013, pp. 1–4.
[121] K. Boahen, “A burst-mode word-serial address-event link – I: Transmitter design,” IEEE Transactions on Circuits and Systems I, vol. 51,
no. 7, pp. 1269–80, 2004.
[122] N. Hatsopoulos, S. Geman, A. Amarasingham, and E. Bienenstock, “At
what time scale does the nervous system operate?” Neurocomputing,
vol. 52, pp. 25–29, 2003.
[123] A. Davison, et al., “Pynn: a common interface for neuronal network
simulators. front. neuroinform,” Front. Neuroinform., vol. 2, p. 11,
2008.
[124] S. Sheik, F. Stefanini, E. Neftci, E. Chicca, and G. Indiveri,
“Systematic configuration and automatic tuning of neuromorphic
systems,” in International Symposium on Circuits and Systems,
(ISCAS), 2011. IEEE, May 2011, pp. 873–876.
[125] C. Patterson, et al., “Scalable communications for a million-core
neural processing architecture,” Journal of Parallel and Distributed
Computing, vol. 72, no. 11, pp. 1507–1520, 2012.
[126] F. Galluppi, et al., “A hierachical configuration system for a massively
parallel neural hardware platform,” in Proceedings of the 9th conference
on Computing Frontiers. ACM, 2012, pp. 183–192.
[127] K. Minkovich, N. Srinivasa, J. Cruz-Albrecht, Y. Cho, and A. Nogin,
“Programming time-multiplexed reconfigurable hardware using a scalable neuromorphic compiler,” Neural Networks and Learning Systems,
IEEE Transactions on, vol. 23, no. 6, pp. 889–901, 2012.
[128] R. Preissl, et al., “Compass: A scalable simulator for an architecture for
cognitive computing,” in Proceedings of the International Conference
on High Performance Computing, Networking, Storage and Analysis.
IEEE Computer Society Press, 2012, p. 54.
[129] T. C. Stewart, B. Tripp, and C. Eliasmith, “Python scripting in the
nengo simulator,” Frontiers in neuroinformatics, vol. 3, 2009.
[130] W. Wulf, et al., “Hydra: The kernel of a multiprocessor operating
system,” Communications of the ACM, vol. 17, no. 6, pp. 337–345,
1974.
[131] E. Neftci, E. Chicca, G. Indiveri, and R. Douglas, “A systematic
method for configuring VLSI networks of spiking neurons,” Neural
Computation, vol. 23, no. 10, pp. 2457–2497, Oct. 2011.
[132] E. Neftci, B. Toth, G. Indiveri, and H. Abarbanel, “Dynamic state
and parameter estimation applied to neuromorphic systems,” Neural
Computation, vol. 24, no. 7, pp. 1669–1694, July 2012.
[133] H. Markram and M. Tsodyks, “Redistribution of synaptic efficacy
between neocortical pyramidal neurons,” Nature, vol. 382, pp. 807–
10, 1996.
[134] S. Fusi and M. Mattia, “Collective behavior of networks with linear
(VLSI) integrate and fire neurons,” Neural Computation, vol. 11, pp.
633–52, 1999.
[135] M. Giulioni, et al., “A VLSI network of spiking neurons with
plastic fully configurable “stop-learning” synapses,” in International
Conference on Electronics, Circuits, and Systems, ICECS 2008.
IEEE, 2008, pp. 678–681.
18
[136] E. Neftci and G. Indiveri, “A device mismatch compensation method
for VLSI spiking neural networks,” in Biomedical Circuits and
Systems Conference (BioCAS), 2010. IEEE, 2010, pp. 262–265.
[137] T. Massoud and T. Horiuchi, “A neuromorphic VLSI head direction cell
system,” Circuits and Systems I: Regular Papers, IEEE Transactions
on, vol. 58, no. 1, pp. 150–163, 2011.
[138] S. Furber and S. Temple, “Neural systems engineering,” Journal of the
Royal Society interface, vol. 4, no. 13, pp. 193–206, 2007.
[139] K. Boahen, “Neuromorphic microchips,” Scientific American, vol. 292,
no. 5, pp. 56–63, May 2005.
[140] R. Sarpeshkar, “Brain power – borrowing from biology makes for low
power computing – bionic ear,” IEEE Spectrum, vol. 43, no. 5, pp.
24–29, May 2006.
[141] K. Hynna and K. Boahen, “Nonlinear influence of T-channels in
an in silico relay neuron.” Biomedical Circuits and Systems, IEEE
Transactions on, vol. 56, no. 6, p. 1734, 2009.
[142] R. Sarpeshkar, T. Delbruck, and C. Mead, “White noise in MOS
transistors and resistors,” IEEE Circuits and Devices Magazine, vol. 9,
no. 6, pp. 23–29, November 1993.
[143] B. Shi, “The effect of mismatch in current- versus voltage-mode resistive grids,” International Journal of Circuit Theory and Applications,
vol. 37, pp. 53–65, 2009.
[144] K. Cameron and A. Murray, “Minimizing the effect of process
mismatch in a neuromorphic system using spike-timing-dependent
adaptation,” Neural Networks, IEEE Transactions on, vol. 19, no. 5,
pp. 899–913, May 2008.
[145] S. Choudhary, et al., “Silicon neurons that compute,” in Artificial
Neural Networks and Machine Learning – ICANN 2012, ser. Lecture
Notes in Computer Science, A. Villa, W. Duch, P. Érdi, F. Masulli,
and G. Palm, Eds. Springer Berlin / Heidelberg, 2012, vol. 7552, pp.
121–128.
[146] T. Delbruck and A. Van Schaik, “Bias current generators with wide
dynamic range,” Analog Integrated Circuits and Signal Processing,
vol. 43, no. 3, pp. 247–268, 2005.
[147] T. Delbruck, R. Berner, P. Lichtsteiner, and C. Dualibe, “32-bit
configurable bias current generator with sub-off-current capability,”
in International Symposium on Circuits and Systems, (ISCAS), 2010,
IEEE. Paris, France: IEEE, 2010, pp. 1647–1650.
[148] W. Maass, T. Natschläger, and H. Markram, “Real-time computing
without stable states: A new framework for neural computation based
on perturbations,” Neural Computation, vol. 14, no. 11, pp. 2531–2560,
2002.
[149] W. Shew, H. Yang, S. Yu, R. Roy, and D. Plenz, “Information capacity
and transmission are maximized in balanced cortical networks with
neuronal avalanches,” The Journal of Neuroscience, vol. 31, no. 1, pp.
55–63, 2011.
[150] E. Schneidman, W. Bialek, and M. B. II, “Synergy, redundancy, and
independence in population codes,” The Journal of Neuroscience,
vol. 23, no. 37, pp. 11 539–11 553, 2003.
[151] R. Jacobs, M. Jordan, S. Nowlan, and G. Hinton, “Adaptive mixtures
of local experts,” Neural computation, vol. 3, no. 1, pp. 79–87, 1991.
[152] L. Breiman, “Random forests,” Machine learning, vol. 45, no. 1, pp.
5–32, 2001.
[153] S. Fusi, P. Drew, and L. Abbott, “Cascade models of synaptically stored
memories,” Neuron, vol. 45, pp. 599–611, 2005.
[154] S. Fusi, W. Asaad, E. Miller, and X.-J. Wang, “A neural circuit model
of flexible sensori-motor mapping: learning and forgetting,” Neuron,
2007, in press.
[155] G. Indiveri, et al., “Neuromorphic silicon neuron circuits,” Frontiers
in Neuroscience, vol. 5, pp. 1–23, 2011.
[156] E. Neftci, E. Chicca, M. Cook, G. Indiveri, and R. Douglas, “Statedependent sensory processing in networks of VLSI spiking neurons,”
in International Symposium on Circuits and Systems, (ISCAS), 2010.
Paris, France: IEEE, 2010, pp. 2789–2792.
[157] G. Indiveri, “A low-power adaptive integrate-and-fire neuron circuit,”
in International Symposium on Circuits and Systems, (ISCAS), 2003.
IEEE, May 2003, pp. IV–820–IV–823.
[158] R. Mill, S. Sheik, G. Indiveri, and S. Denham, “A model of stimulusspecific adaptation in neuromorphic analog VLSI,” Biomedical Circuits
and Systems, IEEE Transactions on, vol. 5, no. 5, pp. 413–419, 2011.
[159] Y. Sandamirskaya and G. Schöner, “An embodied account of serial
order: How instabilities drive sequence generation,” Neural Networks,
vol. 23, no. 10, pp. 1164–1179, 2010.
[160] E. Neftci, J. Binas, E. Chicca, G. Indiveri, and R. Douglas,
“Systematic construction of finite state automata using VLSI spiking
neurons,” in Biomimetic and Biohybrid Systems, ser. Lecture Notes in
PROCEEDINGS OF THE IEEE, VOL. X, NO. X, MONTH YEAR
[161]
[162]
[163]
[164]
[165]
[166]
[167]
[168]
[169]
[170]
[171]
[172]
[173]
[174]
[175]
[176]
[177]
[178]
Computer Science, T. Prescott, N. Lepora, A. Mura, and P. Verschure,
Eds. Springer Berlin / Heidelberg, 2012, vol. 7375, pp. 382–383.
M. Rigotti, D. B. D. Rubin, X.-J. Wang, and S. Fusi, “Internal
representation of task rules by recurrent dynamics: the importance
of the diversity of neural responses,” Frontiers in Computational
Neuroscience, vol. 4, no. 0, 2010.
“Telluride neuromorphic cognition engineering workshop,” http://ineweb.org/workshops/workshops-overview.
“The Capo Caccia workshops toward cognitive neuromorphic engineering,” http://capocaccia.ethz.ch.
S.-C. Liu and T. Delbruck, “Neuromorphic sensory systems,” Current
Opinion in Neurobiology, vol. 20, no. 3, pp. 288–295, 2010.
B. Nessler, M. Pfeiffer, and W. Maass, “STDP enables spiking neurons
to detect hidden causes of their inputs,” in Advances in Neural
Information Processing Systems, Y. Bengio, D. Schuurmans, J. Lafferty,
C. I. Williams, and A. Culotta, Eds., vol. 22, 2009, pp. 1357–1365.
A. Steimer, W. Maass, and R. Douglas, “Belief propagation in networks
of spiking neurons,” Neural Computation, vol. 21, pp. 2502–2523,
2009.
D. Corneil, et al., “Real-time inference in a VLSI spiking neural
network,” in International Symposium on Circuits and Systems,
(ISCAS), 2012. IEEE, 2012, pp. 2425–2428.
P. O’Connor, D. Neil, S.-C. Liu, T. Delbruck, and M. Pfeiffer,
“Real-time classification and sensor fusion with a spiking deep belief
network,” Frontiers in Neuroscience, vol. 7, no. 178, 2013.
E. Neftci, S. Das, B. Pedroni, K. Kreutz-Delgado, and
G. Cauwenberghs, “Event-driven contrastive divergence for spiking
neuromorphic systems,” Frontiers in Neuroscience, vol. 7, no. 272,
2014.
J. Krichmar and H. Wagatsuma, Neuromorphic and Brain-Based
Robots. Cambridge University Press, 2011.
P. Merolla, J. Arthur, R. Alvarez, J.-M. Bussat, and K. Boahen, “A
multicast tree router for multichip neuromorphic systems,” Circuits and
Systems I: Regular Papers, IEEE Transactions on, vol. 61, no. 3, pp.
820–833, March 2014.
A. P. Alivisatos, et al., “The brain activity map project and the
challenge of functional connectomics,” Neuron, vol. 74, no. 6, pp. 970–
974, 2012.
H. Markram, “The human brain project,” Scientific American, vol. 306,
no. 6, pp. 50–55, 2012.
E. McQuinn, P. Datta, M. D. Flickner, W. P. Risk, and D. S. Modha,
“Connectivity of a cognitive computer based on the macaque brain,”
Science, vol. 339, no. 6119, pp. 513–513, 2013.
“Cognitive computing - artifical intelligence meets business
intelligence,” IBM Research website, 2013.
“Samsung global research outreach (gro) program,” Samsung’s SAIT
Website, 2013.
“Building artificial nervous systems: Technology,” Brain Corporation
Website, 2013.
G. Indiveri and T. Horiuchi, “Frontiers in neuromorphic engineering,”
Frontiers in Neuroscience, vol. 5, no. 118, pp. 1–2, 2011.
Elisabetta Chicca (Member, IEEE) studied physics
at the University of Rome 1 La Sapienza Italy, where
she graduated in 1999. In 2006 she received a PhD
in Natural Sciences from the Physics department of
the Federal Institute of Technology Zurich (ETHZ),
Switzerland, and a PhD in Neuroscience from the
Neuroscience Center Zurich (ZNZ). Immediately
after the PhD, she started a PostDoc at the Institute
of Neuroinformatics at the University of Zurich
and ETH Zurich, where she continued working as
Research Group Leader from May 2010 to August
2011. Since August 2011, she is an assistant professor at Bielefeld University
and is heading the Neuromorphic Behaving Systems Group affiliated to the
Faculty of Technology and the Cognitive Interaction Technology - Center of
Excellence (CITEC). Her current interests are in the development of VLSI
models of cortical circuits for brain-inspired computation, learning in spiking
VLSI neural networks, bio-inspired sensing (olfaction, active electrolocation,
audition).
Elisabetta Chicca is member of the IEEE Biomedical Circuits and Systems
TC and IEEE Neural Systems and Applications TC (currently Secretary).
19
Fabio Stefanini obtained a Laurea Triennale degree
(B.S.) and a ”Laurea Magistrale” degree (M.S.)
in Physics from La Sapienza University of Rome
(Italy) in 2006 and 2009 respectively. He has been
a Research Collaborator at the Institute for Complex Systems, CNR-INFM (Rome, Italy), developing
experimental, software and theoretical methods for
the study of collective behaviour in flocking birds.
He obtained a Ph.D. at the Institute of Neuroinformatics in Zurich (Switzerland) implementing a
brain-inspired, real-time pattern recognition system
using neuromorphic hardware with distributed synaptic plasticity. His main
research interests are in neuromorphic systems with analog VLSI circuits,
learning neural networks and complex systems. He currently owns a PostDoc
position at the Institute of Neuroinformatics of Zurich. His research involves
the development of cortical-inspired smart processing systems for contextaware, embedded processors for resource management in mobile devices. Dr.
Fabio Stefanini is one of the creators of PyNCS, a Python package proposed
as a flexible, kernel-like infrastructure for neuromorphic systems.
Chiara Bartolozzi (Member, IEEE) received the
Laurea (with honors) degree in biomedical engineering from the University of Genova, Genova, Italy,
in 2001 and the Ph.D. degree in Natural Sciences
from the Physics department of the Federal Institute
of Technology Zurich (ETHZ), Switzerland, and a
PhD in Neuroscience from the Neuroscience Center
Zurich (ZNZ) in 2007. She then joined the the
Istituto Italiano di Tecnologia, Genova, Italy, first
as a PostDoc in the Robotics, Brain and Cognitive
Sciences Department and then as Researcher in the
iCub Facility, where she is heading the Neuromorphic Systems and Interfaces
group. Her main research interest is the design of event-driven technology
and their exploitation for the development of novel robotic platforms. To this
aim, she coordinated the eMorph (ICT-FET 231467) project that delivered the
unique neuromorphic iCub humanoid platform, developing both the hardware
integration and the computational framework for event-driven robotics. She is
member of the IEEE Circuits and Systems Society (CASS) Sensory Systems
(SSTC) and Neural Systems and Applications (NSA) Committees.
Giacomo Indiveri (Senior Member, IEEE) is an
Associate Professor at the Faculty of Science, University of Zurich, Switzerland. Indiveri received the
M.Sc. degree in electrical engineering from the
University of Genoa, Italy in 1992. Subsequently,
he was awarded a doctoral postgraduate fellowship
within the National Research and Training Program
on “Technologies for Bioelectronics” from which he
graduated with “summa cum laude” in 1995. He also
obtained a Ph.D. degree in computer science and
electrical engineering from the University of Genoa,
Italy in 2004, and the “Habilitation” certificate in Neuromorphic Engineering
from ETH Zurich, Switzerland in 2006. Indiveri carried out research on
neuromorphic vision sensors as a Postdoctoral Research Fellow in the Division
of Biology at the California Institute of Technology, Pasadena, CA, USA, and
on neuromorphic selective attention systems as a postdoc at the Institute of
Neuroinformatics of the University of Zurich and ETH Zurich, Switzerland.
His current research interests lie in the study of real and artificial neural
processing systems, and in the hardware implementation of neuromorphic
cognitive systems, using full custom analog and digital VLSI technology.
Indiveri is a member of several Technical Committees (TCs) of the IEEE
Circuits and Systems society and a Fellow of the European Research Council.