Evolution, Brains and
Multiple Objectives
By Jacob Schrum
[email protected]
About Me

B.S. from S.U. in 2006
 Majors:
Math, Computer Science and German
 Honors Thesis w/ Walt Potter:
Genetic Algorithms and Neural Networks

Currently Ph.D. student at U.T. Austin
 Received
M.S.C.S. in 2009
 Neural Networks Research Group:
Genetic Algorithms and Neural Networks
Evolution




Change in allele frequencies in population
Alleles = variant gene forms
Genes ⇨ traits
Traits affect:
 Survival
 Reproduction

Natural selection
favors good traits
Genetic Algorithms

Abstraction of evolution
 Genes
= bits, integers, reals
 Natural selection = fitness function
 Mutation = bit flip, integer swap, random perturbation, …
 Crossover = parents swap substrings
 Other
representations, mutation ops, crossover ops, …
Applications
( A  B)  (A  C )  ( D  A)
Boolean Satisfiability
K. A. De Jong and W. M. Spears, “Using Genetic Algorithms
to Solve NP-Complete Problems” ICGA 1989
Applications
( A  B)  (A  C )  ( D  A)
Magic Squares
T. Xie and L. Kang, "An evolutionary algorithm for magic squares" CEC 2003
Applications
( A  B)  (A  C )  ( D  A)
Circuit Design
J. D. Lohn and S. P. Colombano, "A circuit representation
technique for automated circuit design" EC 3:3 Sep. 1999
Applications
( A  B)  (A  C )  ( D  A)
Wing Design/Cost Optimization
J. L. Rogers and J. A. Samareh, "Cost Optimization with a Genetic Algorithm"
NASA Langley Research Center, RTA 705-03-11-03, October 2000
Applications
( A  B)  (A  C )  ( D  A)
Traveling Salesman Problem
P. Jog, J. Y. Suh, and D. van Gucht. "The effects of population size,
heuristic crossover and local improvement on a genetic algorithm for
the traveling salesman problem" ICGA 1989.
Applications
( A  B)  (A  C )  ( D  A)
Resource-Constrained Scheduling
S. Hartmann, "A competitive genetic algorithm for
resource-constrained project scheduling" NRL 45 1998
Applications
( A  B)  (A  C )  ( D  A)
Lens Design
X. Chen and K. Yamamoto, "Genetic algorithm and its
application in lens design", SPIE 1996
Applications
( A  B)  (A  C )  ( D  A)
Weight Selection for Fixed Neural Networks
F.H.F. Leung, H.K. Lam, S.H. Ling and P.K.S. Tam, "Tuning
of the structure and parameters of a neural network using
an improved genetic algorithm" NN 14:1 Jan. 2003
Applications
( A  B)  (A  C )  ( D  A)
What Are Neural Networks?
Artificial Neural Networks
Brain = network of neurons
 ANN = simple model of brain

 Neurons
organized into layers
What Can Neural Networks Do?

In theory, anything!
 Universal Approximation


Theorem
NNs are function approximators
 [0,1] N  [0,1]M
In practice, learning is hard
 Supervised:
Backpropagation
 Unsupervised: Self-organizing maps
 Reinforcement Learning:
Temporal-difference learning and
Evolutionary computation
Neuro-Evolution
Genetic Algorithms + Neural Networks
 Many different network representations

Fixed length string
Subpopulations for each
hidden layer neuron [1]
Evolve topology and weights
[2]
[1] F. Gomez and R. Miikkulainen, "Incremental Evolution Of Complex General Behavior" Adaptive Behavior 5, 1997.
[2] K. O. Stanley and R. Miikkulainen, "Evolving Neural Networks Through Augmenting Topologies" EC 10:2, 2002.
Constructive Neuroevolution
Population of networks w/ no hidden nodes
 Random weights and connections

Constructive Neuroevolution
Evaluate, assign fitness
 Select the fittest to survive

Constructive Neuroevolution
Fill out population
 Crossover and/or cloning

Crossover
Clone
Constructive Neuroevolution
Random mutations
 Perturb weight, add link, splice neuron

No mutation
Perturb weight
Add link
Splice neuron
Constructive Neuroevolution
Can add recurrent links as well
 Provide a form of memory

Neuroevolution Applications
Double Pole Balancing
F. Gomex and R. Miikkulainen, “2-D Pole Balancing With
Recurrent Evolutionary Networks” ICANN 1998
Neuroevolution Applications
Robot Duel
K. O. Stanley and R. Miikkulainen, "Competitive Coevolution
through Evolutionary Complexification" JAIR 21, 2004
Neuroevolution Applications
Vehicle Crash Warning System
N. Kohl, K. Stanley, R. Miikkulainen, M. Samples, and R. Sherony,
"Evolving a Real-World Vehicle Warning System" GECCO 2006
Neuroevolution Applications
http://nerogame.org/
Training Video Game Agents
K. O. Stanley, B. D. Bryant, I. Karpov, R. Miikkulainen, "Real-Time
Evolution of Neural Networks in the NERO Video Game" AAAI 2006
What I Do With Neuroevolution

Discover complex behavior
 Multiagent
domains
 Simulations, robotics, video games
Support for multiple modes of behavior
 Multiobjective optimization

Mutiobjective Optimization

 
Pareto dominance: v  u iff

i  1,, n: vi  ui

i  1,, n: vi  ui
Assumes maximization
Want nondominated points

NSGA-II [3] used


 Popular
Nondominated
EMO method
[3] K. Deb, S. Agrawal, A. Pratap, T. Meyarivan, "A Fast Elitist Non-dominated sorting
genetic algorithm for multi-objective optimization: NSGA-II" PPSN VI, 2000
Non-dominated Sorting Genetic Algorithm II




Population P with size N; Evaluate P
Use mutation to get P´ size N; Evaluate P´
Calculate non-dominated fronts of {P P´} size 2N
New population size N from highest fronts of {P  P´}
Evolve Game AI

Game where opponents have multiple objectives
 Inflict
damage as a group
 Avoid damage individually
 Stay alive individually

Objectives are contradictory and distinct
Opponents take damage from bat
Player is knocked back by NPC
Intelligent Baiting Behavior
How to avoid stagnation
Some trade-offs are too easy to reach
 Focus on difficult objectives
 TUG: Targeting Unachieved Goals

 Avoids
Hard Objectives
need for incremental evolution
Smaller Team w/ Expert Timing
Multitask Domains
Perform separate tasks
 Predator/Prey

 Prey:
run away
 Pred: prevent escape

Front/Back Ramming
 Attack
with ram on front
 Attack with ram on back
Multimodal Networks

One network, multiple policies
 Multitask
[4] = one mode per task
 Mode mutation = network chooses mode to use
Multitask
Two tasks, two modes
Appropriate mode used for task
Mode Mutation
Start with one mode, mutation adds another
Preference neurons control mode choice
[4] R. A. Caruana, "Multitask learning: A knowledge-based source of inductive bias" ICML 1993
Multimodal Predator/Prey Behavior
Learned with Mode Mutation
Runs away in Prey task
Corralling behavior in Predator task
Multimodal Front/Back Ramming
Behavior
Learned with Multitask
Efficient front ramming
Immediately turn around to attack
with back ram
What about “real” domains?

Unreal Tournament 2004
 Commercial
video game
 Basis for BotPrize competition:
Bot Turing Test

Placed 2nd with our bot: UT^2
UT^2 Behavior/Judging Game
Summary




Neural networks can represent complex behavior
Neuroevolution = way to discover this behavior
Multiobjective evolution needed in complex domains
Success in challenging designed/commercial domains
Questions?
E-mail:
[email protected]
Webpage:
http://www.cs.utexas.edu/~schrum2/
Auxiliary Slides

Empirical results
Differences for Alternating and Chasing significant with p < .05