Neural Modularity Helps Organisms Evolve to
Learn New Skills Without Forgetting Old Skills
PLoS Computational Biology, April 2015
Kai Olav Ellefsen
Jean-Baptiste Mouret
Jeff Clune
Background
and
Motivation
(Barnes and Underwood 1959)
In Artificial Neural Networks:
Catastrophic Interference
(McCloskey and Cohen 1989)
Learning Skill A then Learning Skill B
Natural Neural Networks are Modular
Low Modularity
High Modularity
Modularity Can Reduce Interference
Can We Evolve Modular Neural Networks?
Can We Evolve Modular Neural Networks?
Clune, Mouret and Lipson (2013)
Experimental
Setup
Experiment – Hypothesized result
Experiment – Hypothesized results
Evolving Learning Abilities
• Evolution optimized learning neural networks
• Each individual was subjected to a number of learning episodes, and
awarded a fitness value based on its ability to learn without forgetting
• The learning task abstracted an animal learning food preferences in a
seasonally varying environment
A Single Fitness Test
Skill A
Skill B
Goal:
Learning two independent
skills, A and B
Inputs for A
Inputs for B
Evolution
Selection on
performance alone
Variation
Selection on
performance and
connection costs
Variation
Results
Performance
Modularity
Connection Costs
Performance Alone
The Best Networks From Independent Evolutionary Runs
Skill
module
Reward delivery
module
Two Benefits of Modularity:
Separating Skills
from Learning Signals
Reducing Interference
Between Learned Skills
Connection Costs
Performance Alone
Percent of all
Learned Associations
Conclusion
Summary
• Sequential learning is an important and difficult challenge for neural
networks
• Adding a connection cost during evolution increases modularity and
performance on this task
• Connection-cost individuals are better at retaining learned skills
Important issues for future work
• More complex learning tasks
• Different learning paradigms
• Deeper analyses of the modularity. Is there a functional modularity?
• Separating skills but allowing shared knowledge
Thank You!
Questions
Bonus
Mitigating catastrophic forgetting
1. Makes neural networks more useful
2. Can help us better model human cognition
Neuromodulated Reinforcement Learning
Dw = h*mod*in*out
Soltoggio et al. (2008)
Performance of Evolved Individuals
Connection Costs
Performance Alone
Percent of all
Learned Associations
Catastrophic Interference
Current Robots: Specialists
What we Need: Multiple Learned
Skills
Hypothesis
• Is the natural tendency for neural modularity a reason why animal
brains can learn skills consecutively without forgetting?
Randomizing edibility associations
0 1
0
0
0
0
0
0
0 0
0 1
0
Summer
Winter
Reinforcement
Reinforcement
0
0
0
0
0
1 0
0 0
0
0
1
Summer
Winter
Reinforcement
1
0
0
0
0 0
0 0
0
0
1
0
0
0
0 0
0 0
Summer
Winter
Reinforcement
Reinforcement
0
0
1
0
0
0
0 1
The Neural Networks
Summer
Winter
Reinforcement
0 1
0
0
0
Summer
Winter
Reinforcement
1
0
0
0
0 0
0 1
0
0
0
0
0
0
0 0
0 1
0
Summer
Winter
Reinforcement
Reinforcement
0
0
0
0
0
1 0
Analyzing the Evolved Individuals
• Experiment inspired by the study on
association interference in humans.
• We first trained individuals to learn WINTER
associations, for 50 “days”
• We then exposed them to 20 days of
SUMMER, and studied how old and new
associations developed
Barnes and Underwood (1959)
Analyzing the Evolved Individuals
Connection Costs
Performance Alone
Analyzing the Evolved Individuals
Connection Costs
Performance Alone
The Evolved Skill: Learning Food
Preferences
Summer
Winter
Summer
The Stability-Plasticity Dilemma
How can a brain or machine learn quickly about new objects and
events without just as quickly being forced to forget previously
learned, but still useful, memories?
The Stability-Plasticity Dilemma
Forgetting in Humans
•Train on association set (A-B)
•Train on association set (A-C)
•Test on association set (A-B)
Barnes and Underwood (1959)
Forgetting in ANNs
•McCloskey and Cohen (1989):
–ANN learned ‘one’s addition facts
–Then learned ‘two’s addition facts
–Performance on ‘one’s addition
decreased rapidly
•Within 5 trials: 20 %
•Within 15 trials: 0 %
The Stability-Plasticity Dilemma
How can a brain or machine learn quickly about new objects and
events without just as quickly being forced to forget previously
learned, but still useful, memories?
0
1
0
0
0
0
Summer
Winter
Reinforcement
1
0
0
0
0
0
1
Summer
0
Winter
Reinforcement
0
0
0
0
0
1
0
Analyzing the Evolved Individuals
• We exposed the evolved individuals to 80 new, random environments
• After each season, we counted how many “skills” were learned,
retained and forgotten
Analyzing the Evolved Individuals
Connection Costs
Performance Alone
The Evolved Skill: Learning “Food”
Preferences
Outline
• Background and Motivation
• Experimental Setup
• Results
• Conclusion
“Forced Forgetting”
Consequence: Evolution can no longer mitigate catastrophic
forgetting.
Evolution only generates good within-season learning strategies.
Do connection costs still make a difference?
Neuromodulated Reinforcement Learning
Soltoggio et al. (2008)
Neuromodulated Reinforcement Learning
Soltoggio et al. (2008)
Neuromodulated Reinforcement Learning
Soltoggio et al. (2008)