Biological and cognitive plausibility
in connectionist networks
for language modelling
Maja Anđel
Department for German Studies
University of Zagreb
Connectionist networks
• AI networks for simulating cognitive
processes – language
• implementing the “computer metaphor”
• architectural inspiration: human brain
(network of artificial neurons)
Connectionist networks
• subsymbolic, not symbolic processing =
no division in hardware/software –
software “built in” to the hardware
structure
Connectionist networks
Connectionist networks
Language modeling in connectionism
Biology /
neurological functioning
Computer
implementation
Language processes
(as cognitive processes)
Models in history (’80)
• Rumelhart &
McClelland (1986):
English past tense –
modeling the Ushaped learning curve
9
8
7
6
5
4
3
2
1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
Models in history (’80)
• McClelland &
Kawamoto (1986):
Thematic roles in
sentences
• “How to represent
sequences?”
w i c k e l f e a t u r e s
w i c
i c k
c k e
k e l
e l f
l f e ...
Models in history (’80)
• Elman (1990): Words
of different length
• “How to represent
sequences?”
outputs
hidden units
inputs
context units
Today – biological plausibility
• O’Reilly (1998): Six principles of biological
plausibility:
–
–
–
–
biological realism
distributed representations
inhibitory competition
bidirectional propagation of activation (top-down
and bottom-up)
– error-driven task learning
– Hebbian learning
New algorithms that satisfy all the constraints!
Today – biological plausibility
• O’Reilly (1998): Six principles of biological
plausibility:
– biological realism
Today – biological plausibility
• O’Reilly (1998): Six principles of biological
plausibility:
– distributed representations
Today – biological plausibility
• O’Reilly (1998): Six principles of biological
plausibility:
– distributed representations
Today – biological plausibility
• O’Reilly (1998): Six principles of biological
plausibility:
– inhibitory competition
Today – biological plausibility
• O’Reilly (1998): Six principles of biological
plausibility:
– bidirectional propagation of activation (top-down
and bottom-up)
conceptual
perceptual
dog
Today – biological plausibility
• O’Reilly (1998): Six principles of biological
plausibility:
– error-driven task learning
desired outcome
error computation
transfer function
error backpropagation
input signal
Today – biological plausibility
• O’Reilly (1998): Six principles of biological
plausibility:
– error-driven task learning
network state
transfer function
input signal
desired outcome
error computation
Today – biological plausibility
• O’Reilly (1998): Six principles of biological plausibility:
– Hebbian learning
"The general idea is an old one, that any two cells or systems of
cells that are repeatedly active at the same time will tend to
become 'associated', so that activity in one facilitates activity
in the other." (Hebb 1949, p. 70)
"When one cell repeatedly assists in firing another, the axon of
the first cell develops synaptic knobs (or enlarges them if they
already exist) in contact with the soma of the second cell."
(Hebb 1949, p. 63)
Today – biological plausibility
• New algorithms – combination of new
principles
• Complexity brings better results
• Old models sucessfully transposed
– morphology acquisition
– syntax processing
– semantic categorization
• Computational neuroscience
Thank you!