Coevolving Probabilistic Game Playing Agents Using Particle Swarm
Optimization Algorithms
Evangelos Papacostantis
Department of Computer Science
University of Pretoria
[email protected]
Andries P. Engelbrecht
Department of Computer Science
University of Pretoria
[email protected]
Abstract- Coevolutionary techniques in combination
with particle swarm optimization algorithms and neural networks have shown to be very successful in finding strong game playing agents for a number of deterministic games. This paper investigates the applicability of a PSO coevolutionary approach to probabilistic
games. For the purposes of this paper, a probabilistic
variation of the tic-tac-toe game is used. Initially, the
technique is applied to a simple deterministic game (tictac-toe), proving its effectiveness with such games. The
technique is then applied to a probabilistic 4x4x4 tic-tactoe game, illustrating scalability to more complex, probabilistic games. The performance of the probabilistic
game agent is compared against agents that move randomly. To determine how these game agents compete
against strong non-random game playing agents, coevolved solutions are also compared against agents that
utilize a strong hand-crafted static evaluation function.
Particle swarm optimization parameters/topologies and
neural network architectures are experimentally optimized for the probabilistic tic-tac-toe game.
1 Introduction
The work on computer games has been one of the most successful and visible results of artificial intelligence research
[17]. This is due to the fact that games provide challenging
“puzzles” that require a great level of sophistication in order
to be played/solved. Games can be considered as problems,
which operate by following strict rules within a game environment. The manner in which moves may be executed and
games may be won or drawn are strictly defined and may
not be deviated from. Game environments are an ideal domain to investigate the effectiveness of an array of different
AI techniques. Pioneers of AI research used games as problem domains in their research, for example, Arthur Samuel
[16], Claude Shannon [18] and Alan Turning [20].
This paper investigates the effectiveness of using a coevolutionary technique in combination with particle swarm
optimization algorithms and neural networks to find game
playing agents for probabilistic games from pure self-play.
This means that no game information or game strategies are
provided to the learning algorithm and the agent learns its
own playing strategy by competing against other players.
A similar approach has been used by Messerschmidt and
Engelbrecht [13] and Franken et al [2, 3, 4, 5] for the deterministic games of tic-tac-toe [13, 5, 2], checkers [3, 2]
and the iterated prisoners dilemma [4, 2]. The PSO coevo-
Nelis Franken
Department of Computer Science
University of Pretoria
[email protected]
lutionary approach used is an adaptation of the evolutionary
algorithm approach developed by Chellapilla and Fogel [1].
Probabilistic games contain hidden game information
and players compete against each other based on elements
of chance, which are beyond their control. Games that are
played with dice or cards are usually probabilistic games,
which include backgammon, poker, bridge and scrabble.
Deterministic games on the other hand are games that provide perfect information to all players at all times. There
are no hidden elements and the players can execute moves
in any manner they wish to, within the rules of the game,
without any probabilistic elements affecting their game decisions. Such games include tic-tac-toe, checkers, chess and
go.
Probabilistic games can not be solved, meaning that it is
not possible to play a probabilistic game in such a way that
you are always guaranteed to either win or draw. This is due
to the probabilistic element that may favor any player during the course of the game. This is the reason why games
of this nature are similar to real world problems. With real
world problems it is very difficult for one to define constraints and such problems almost never contain perfect information. Successful techniques applied to probabilistic
games may therefore be more scalable to real world problems in comparison to techniques that are successful when
applied to deterministic games.
Game trees have contributed a tremendous amount to
game learning, providing the ability to see favorable or nonfavorable moves in the future of a game. A number of game
tree construction methods exist, with the most popular being minmax [6], alpha-beta [12] and NegaScout [15]. To
elaborate a bit more, game trees are constructed by adding
every possible move that can be played by each player, alternating the player each time the game tree depth increases.
For probabilistic games the construction of a tree becomes
impractical, since the tree has to represent all possible outcomes of the probabilistic elements used by the game. This
usually causes the tree to become extremely large, hampered by time consuming efforts for its construction and
use. Game trees can be constructed based on the probabilities of certain moves that can be executed. Moves that have
higher probabilities in being selected are used to construct
the tree, while less probable options are excluded. Game
trees are not extensively used in this report, with only simple minmax trees, expanded to a depth of 1.
Coevolutionary techniques in combination with other
learning algorithms have successfully been applied to probabilistic games, specifically in backgammon[19, 14]. Tem-
poral difference learning is one of these, which is based
on Samuel’s machine learning research [16]. This learning algorithm has been successfully applied to backgammon by Gerry Tesauro [19]. TD-Gammon, the name given
by Tesauro to the program, is a neural network that takes
a board state as an input and returns the score of the board
state as an output. The weights of the neural network were
optimized using temporal difference. TD-Gammon is regarded to be as good as the best human backgammon players in the world, and possibly even better [17]. Another
successful learning technique in combination with coevolution has been applied by Blair and Pollack, which managed to use a simple hill-climbing learning algorithm to coevolve competitive backgammon agents [14]. All the examples given above illustrate that coevolution is a very powerful learning machine, which when combined with even the
simplest learning algorithms, it may return sound results.
Simulation-based techniques have also shown to be very
effective [7, 8]. These techniques allow the simulation of
games from their current state to completion, by using different randomly selected possible outcomes to replace the
probabilistic elements. By doing so, statistical information
about the game is gathered for each possible move. Based
on the probability of a win, draw or loss for each move, a
decision is made on which move to select. There have been
some notable applications of this technique: The University
of Alberta developed a poker-playing program called Loki
that utilizes simulation-based techniques [7]. Loki is the
first serious academic effort to build a strong poker playing program and at best is rated as a strong intermediatelevel poker player [17]. Mathew Ginsberg’s bridge playing
program GIB [8] also makes use of simulation-based techniques. GIB forced the frequent bridge world champion Zia
Mahmood to withdraw in 1999 a prize award of £1000000
for any program that manages to beat him, after narrowly
beating GIB in an exhibition match [17]. GIB also utilizes
another technique frequently applied to probabilistic games,
the Monte Carlo simulation method, where a representative
sample of all possible moves is chosen to give a statistical
profile of an outcome.
2 Particle Swarm Optimization
The particle swarm optimization algorithm is a populationbased algorithm that enables a number of individual solutions, called particles, to move through a hyper dimensional
search space in a methodical way. The movement of the
particles is done in such a way that it enables them to incrementally find better solutions. The algorithm is based
on the simulation of the social behavior of birds within a
flock and was first described by Kennedy and Eberhart [10].
What mainly drives a PSO algorithm, is the social interaction between its particles. Particles within a swarm share
their knowledge with each other, specifically with regards
to the quality of the solutions they have found at specific
points in the search space. The best solution discovered by
a specific particle is referred to as a personal best solution.
Particles then move towards other personal best solutions
using certain velocities, in an attempt to discover improved
solutions.
It is obvious that the pattern of communication between
the particles will ultimately affect the manner by which the
particles move within the search space. Different information sharing patterns/structures will enable the search space
to be explored in different ways. Topology is a term that
refers to a pattern by which particles communicate with
each other. The following topologies are most commonly
used:
• Global Best: All particles communicate with each
other, forming a fully interconnected social network.
With this topology all particle movements are affected by their own personal best solution and a
global best solution. The global best solution forms
the best solution of the entire swarm.
• Local Best: A neighborhood size is defined for this
topology, which determines the number of particles
with which each particle can communicate and share
information with. If a neighborhood size is 3, for example, neighborhoods of 3 particles are formed by selecting the two adjacent neighbors of each particle in
variable space. With this topology all particle movements are affected by their own personal best solution and a local best solution. The local best solution
forms the best solution within the neighborhood the
particle belongs to.
• Von Neuman: This topology is very similar to the local best topology, which allows each particle to form
a neighborhood with its immediate top, bottom, left
and right particles in variable space [11].
A particle swarm optimization algorithm has a number
of parameters which allow it to be fine-tuned for better performance. The swarm size and the topology form two parameters, which have already been discussed. Furthermore,
four other parameters determine the behavior of the particle
movement. Two acceleration constants determine the degree by which a personal best and neighborhood best solution affects a particle’s movement. c1 forms the personal acceleration constant and c2 the global acceleration constant.
The inertia weight variable φ determines how much previous particle velocities influence new particle velocities. Finally, Vmax is a value that sets an upper limit for velocities
in all dimensions, which limits particles from moving too
rapidly.
3 Coevolution
Coevolution is a competitive process between a number of
species that enables the species to continuously evolve in
order to overcome and outperform each other. Consider the
example of a lion and a buck, where the two are competing
in a survival “game”. The lions’ survival depends on capturing the buck for food, while the bucks’ survival on the other
hand depends on outwitting the lion so it never gets caught.
The buck can initially run faster than the lion, avoiding its
capture. The lion fails in numerous attempts, but in the process strengthens its leg muscles, enabling it to run faster
and eventually to capture the buck. The buck then develops
a technique that allows it to continuously dodge the lion,
since it can not run any faster and gets caught. In return the
lion manages to increase its stamina in the process of trying
to keep up with the buck, allowing it to follow the buck until
it gets exhausted from its dodging maneuvers and then capturing it. The two continuously discover different ways that
will enable them to survive in turn. This pattern is similar
to the one seen in the case of arms races, where a number of
countries compete against each other to produce more destructive and technologically advanced weapons. Another
excellent example of coevolution is described by Holland
[9].
This continuous coevolution allows each of the species
competing to incrementally become stronger, with a tit-fortat relationship fueling the process. The example given
above is a demonstration of predator-pray coevolution,
where there is an inverse fitness between the two species.
A win for the one means ultimately a lose for the other,
with loosing species improving in order to challenge winning species. A different form of coevolution exists, called
symbiotic coevolution. In this case species do not compete
against each other, but rather cooperate for the general good
of all the other species that are coevolving. A success for
one of the species, means the improved survival fitness of
all other species too.
For the purposes of this report, a combination of symbiotic and predator-prey coevolution has been chosen. The
actual algorithm is described in Section 6. This coevolutionary approach is based on the coevolution of two separate
populations of game playing agents that compete against
each other. A score scheme is used that enables the awarding of points to game playing agents that are successful in
winning and drawing games, while loosing agents are penalized. Agents within a population cooperate in an attempt
to improve the overall fitness of the population. A PSO algorithm is applied to each population separately to adapt
agents.
The size of each population and the scoring scheme used
have an influence on the performance of the coevolutionary
process.
4 Tic-Tac-Toe Variation
The variation introduced below in section 4.2, extends the
original tic-tac-toe game by adding and modifying rules that
make the game more complex and probabilistic.
4.1 Tic-Tac-Toe
The original game is a deterministic 2 player game that is
played on a 3x3 grid, which initially contains empty spaces.
The player who competes first must place an X piece in one
of the 9 spaces of the grid with the second player following
by doing the same with an O piece. The players may not
place a piece in an already occupied space and they may
not skip a turn. The objective of the game is for a player to
complete a full row, column or diagonal with his own pieces
in sequence, with the win going to the player that manages
to do so. Both players compete until a player successfully
completes the objective or until no more empty spaces exist.
In the last case, this implies a draw between the two players.
Table 1 shows the probabilities of a win, draw and
loss between two players when playing tic-tac-toe. These
probabilities were calculated using two random playing
agents competing against each other. P layer1st plays first
while P layer2nd plays second for a total of 100000 games.
Games
%
P layer
58277 58.277
P layer2nd 28968 28.968
Draw
12755 12.755
Table 1: Tic-tac-toe: probabilities.
1st
The table clearly shows an advantage for P layer1st . This
advantage is due to two facts. The first being the priority
of P layer1st to capture the center empty space of the 3x3
grid, giving him a significant advantage over P layer2nd .
This is because the center space forms a part of one row,
one column and two diagonals, and by placing a mark there,
P layer1st already has secured 4 winning options to his favor while denying 4 winning options for P layer2nd . No
other space gives such an advantage. The second advantage
is that P layer1st will have the opportunity to place more
pieces on the board, since the board initially consists of an
odd number of empty spaces.
4.2 Probabilistic 4x4x4
The tic-tac-toe variation described here is a probabilistic 2
player game, played on 4 layers consisting of 4x4 grids.
Another way to visualize the game board is by seeing it as a
3 dimensional cube, consisting of 64 smaller separate cube
spaces which make up the positions in which a player can
place a piece. The figure below shows this.
Figure 1: A probabilistic 4x4x4 game board.
The complexity of the game is increased by introducing the new dimension and increasing the board size by 1.
The first player, P layer1st , will no longer have a large advantage over P layer2nd as in the original tic-tac-toe game.
This is because a center space does not exist any longer due
to the even sized board edge. Furthermore, the total number
of spaces is an even number too, allowing P layer1st and
P layer2nd to have an equal number of pieces on the board.
The game is played similar to the original tic-tac-toe
game, with P layer1st and P layer2nd alternating turns
and respectively placing an X or O piece on one of the
board layers. A player does not have the freedom though of
placing a piece in any of the 4 available layers. The layer
in which a player has to make a move is determined by a
“4 sided dice”. Just before executing a move, the player
rolls this dice to determine the level to play. If a player has
to play on a layer where all spaces are occupied by pieces,
he misses that round and the game moves on to the next
player. The game only ends when there are no more empty
spaces to place a piece. When the board is full, each player
counts the number of rows, columns and diagonals he has
completed and gets a point awarded for each successful 4
pieces placed in sequence. The player with the most points
wins the game. If the players have an equal score, the game
is a draw. Figure 2 shows different combinations in which
a player can score points. All three dimensions can be
used and any 4 pieces lined up in sequence can score a point.
Figure 2:Probabilistic 4x4x4 point combinations.
Table 2 shows the probability of a win for both
P layer1st and P layer2nd , and of a draw for a total of
100000 games that are played randomly by both players.
Games
P layer
50776
P layer2nd 44367
Draw
4857
Table 2: Probabilistic 4x4x4:
1st
%
50.776
44.367
4.857
probabilities.
The advantage of P layer1st over P layer2nd has been considerably reduced compared to the original tic-tac-toe game.
Only a 6.4% winning advantage seperates P layer1st from
P layer2nd in the probabilistic variation.
5 The Game Playing Agents
The game playing agents are represented by standard 3layer feed forward neural networks, consisting of summation units. The size of the input layer is dependant on the
size of the game board. The input layer therefore consists
of 9 neurons for a standard 3x3 tic-tac-toe game, while 64
neurons are required for a probabilistic 4x4x4 tic-tac-toe
game. The size of the hidden layer varies, depending on
the complexity of the game. Only one neuron is used in the
output layer. The architecture explained above excludes all
bias units. The sigmoid activation function is used for each
neuron in the hidden and output layers, with the steepness
value λ set to 1. The weights of the neural network are randomly initialized between the range [ √ −1 ], [ √ 1 ],
f anin
f anin
where f anin represents the number of incoming weights
to the neuron.
The neural network is used to evaluate a given state by
accepting the actual state as an input and returning as an
output a value that represents how advantageous the state
is, with states returning higher values preferred.
Assume that P layerx plays against P layery and that
P layerx needs to plan a new move. Let Statecurrent denote the current game state. The following steps are used to
determine the next state.
1. Build a game tree with a depth of N , using
Statecurrent as the root node and by adding all
possible moves for P layerx for all odd depths and
P layery for all even depths.
2. Evaluate all leaf nodes by using the neural network as
an evaluation function in the following manner:
i. For all valid positions on the board assign a
value of 0.5 for every P layerx piece on the
board, a value of -0.5 for every P layery piece
and a value of 0 if there is no piece on a specific
position.
ii. Supply these values as inputs to the neural network and perform a feed forward process to determine the output.
iii. Assign the value of the neural network output as
the evaluation value of the node.
3. Using the minmax [6] algorithm, determine the most
beneficial state to execute the next move.
Instead of using minmax, the alpha-beta [12] or NegaScout [15] algorithms can be used to optimize the game tree.
Only a single depth for the game tree has been considered
throughout this paper.
The input representation scheme used in the first point
of step 2.i, results in identical board states to be inversely
represented depending on whether the player played first or
not [3].
Since a neural network is used to evaluate how good a
state is, the objective is to find a set of weights which can
differentiate between good states and bad states. Usually,
supervised training would be used to adjust the neural network weights. With supervised training there exists a training set consisting of both inputs and the associated desired
outputs. The most popular supervised training algorithm is
back propagation [22]. With back propagation, each pattern of the training set is used and the difference between
the actual output and the target output is used to adjust the
weights. After repeating this process for a number of times
for the full training set, the neural network eventually fits
a curve over the training set to relate inputs with desired
outputs. In the case of game learning one does not have a
training set and the weights can therefore not be adjusted using back propagation or any other supervised training technique. This problem is overcome with the use of coevolution and particle swarm optimization algorithms.
6 The Game Training Process
This section explains exactly how the coevolution and particle swarm optimization algorithms are combined to train
game playing agents. Initially, two populations of game
playing agents are instantiated by generating a number of
neural networks with randomly initialized weights. These
neural networks form possible game playing agents that will
coevolve, growing stronger for each new generation. The
agents represent particles in a swarm. Each particle represents the weights of one neural network. Each agent has
the ability to store two sets of weights: its current weights
(W eightscurrent ) and its best weights (W eightsbest ) it has
encountered during the training process. Each agent in the
first population competes against all other agents in the second population, and vice-versa. A scoring scheme is then
used to evaluate the agents. Agents are rewarded/penalized
based on whether a game has been lost, won or drawn.
Points are accumulated for each agent over all games played
by the agent. Higher scoring agents are considered better
than lower scoring agents. These scores are used to determine personal, local and global best solutions, as needed for
the PSO algorithm. To discover the overall best agent, the
two best agents of each population compete against a random player for a total of 10000 games. Based on the score
scheme used, the agent with the highest score is regarded as
the overall best agent and stored in a hall of fame. In no way
does this agent affect training and the sole reason the agent
is stored is to preserve the overall best agent during training. The evaluation against a random player may be time
consuming, but the diversity of game playing strategies that
they offer is valuable, making them appropriate to be used
for evaluation purposes. A detailed step-by-step algorithm
is given in the following subsection.
3. The scores of all W eightscurrent and W eightsbest
for each agent in the two separate populations are
compared. If the score of an agents’ W eightscurrent
is larger than the score its W eightsbest , then its
W eightscurrent becomes the new W eightsbest and
therefore its new personal best.
4. All the scores of the W eightsbest in each separate
population are compared. The agent with the highest
score for its W eightsbest , becomes the local/global
best of the population.
5. The W eightsbest of both agents with the highest
scores that exist in both populations compete against
a random playing agent for a total of 10000 games,
5000 of which the agent plays first and the remaining 5000 the agent playing second. A score for the
W eightsbest for both best agents that belong to the
two populations is determined. If a score is found that
is the highest score observed thus far during training,
the weights are stored as the best weights encountered during training. This set of weights is called
W eightssupreme .
6. Update all W eightscurrent based on the PSO algorithm used for both populations.
7. If the algorithm has not converged at a specific solution, go to step 2 and repeat the process.
6.1 Step-by-Step
1. Instantiate two new populations of agents. Each agent
is initialized in the following way:
current
• The W eights
plained in Section 5.
are initialized as ex-
• The W eightsbest are set equal to the
W eightscurrent for the first generation.
2. Agents compete against agents in the opposing population, as explained in Section 5. Agents use both
their W eightscurrent and W eightsbest to compete.
Competing agents use a preselected score scheme,
based upon which each agent receives a specific
score. The scheme adopted by this paper awards the
following: 3 points for a win, 1 for a draw and 0 for
a lose. The weights of each agent are used as follows
to compete against all other agents in the other population:
• All W eightscurrent of the one population
compete against all W eightscurrent of the
other population by playing both first and
second.
Based on each W eightscurrent
wins/losses/draws, a score is assigned to each
W eightscurrent .
• All W eightsbest of the one population compete
against all W eightscurrent of the other population by playing both first and second. Based on
each W eightsbest wins/losses/draws, a score is
assigned to each W eightsbest .
7 Results
The following section reports the results of the coevolutionary technique, as applied to the two tic-tac-toe games.
Each simulation was executed 30 times, with the
W eightssupreme stored for each. The evaluation of two
agents is determined by using a sample of 100000 games.
Each agent competes for 50000 games by playing first while
the remaining 50000 games the agent competes second. The
percentage of wins, loses and draws is given in each case,
together with F , its Franken performance measure [2].
P layerstatic refers to players utilizing a hand-crafted
static evaluation function, P layerran refers to players playing randomly and P layersupreme to players that utilize the
W eightssupreme that were found with the coevolutionary
PSO method. In the case of P layersupreme , the average of
all 30 best solutions is shown in the given tables.
The hand-crafted evaluation function used by P layerx
to compete against P layery for both games is defined as:
nP layery
nP layerx
X
k=1
piecesk −
X
piecesk
(1)
k=1
where nP layerx is the total number of rows, columns and
diagonals (in all dimensions) of the game that only contain
pieces belonging to P layerx , and piecesk is the total number of pieces in that specific row, column or diagonal. In
the case where piecesk = max, then piecesk = +∞. The
value of max represents the maximum number of pieces
that can be placed in sequence. The value of 3 is used
for tic-tac-toe and 4 is used for probabilistic 4x4x4 tic-tactoe. The reason this assignment is done is to allow agents
using equation (1) to immediately take opportunities that
will allow them to complete full sequences which enables
them to win games/score points. Higher values returned by
the hand-crafted evaluation function represents better board
states.
7.1 Tic-Tac-Toe
7.1.1 Hand-Crafted Evaluation Results
Table 3 shows the results of how the hand-crafted function
performs against P layerran .
indicates the performance of the two best set of weights
in each population and the performance of the overall best
set of weights (W eightssupreme ) over the generations for
one of the executed simulations. The performance measure
used in the graph is the score when competing against
P layerran for 10000 games. The covered white part
of the graph conveys the performance of W eightsbest1 ,
which belongs to the first population, while the black
part conveys the performance of W eightsbest2 which
belongs to the second population. The two gray lines are
sixth degree polynomials fitted through W eightsbest1 and
W eightsbest2 .
%
F
P layerstatic 93.55
P layerran
1.78 95.88
Draw
4.67
Table 3: Tic-tac-toe: hand-crafted evaluation.
It is clear that the function works very well, since it only
looses 1.78% in total. One could argue that this is not satisfactory, since the game is very simple and could be played in
such a way that games are only won or drawn. This is where
the depth of the minmax tree comes into play. In order for
the agents to achieve perfect play for the tic-tac-toe game
using the provided hand-crafted static evaluation function, it
is required for them to be able to construct deeper trees that
will enable them to explore possible future moves. When a
depth of 4 (ply-depth of 2) is used, perfect play is achieved.
In order to keep the complexity of the learning algorithms as
low as possible, depths of only 1 are used, keeping in mind
that increased depth sizes may return improved results.
7.1.2 Coevolutionary PSO Results
The initial configurations of the coevolutionary and particle
swarm optimization algorithms are taken from [5], specifically the configuration that returned the best results was
chosen. The Von Neuman topology was selected, with c1 ,
c2 and φ all initialized to the value of 1. No Vmax value
was selected, meaning that the velocity of the particles
are not restricted in any way. The swarm size for each
population was set to 10 (20 particles are used in total),
with each particle having 7 neurons in the hidden layer. A
score scheme that awarded 3 points for a win, 1 point for a
draw and 0 points for a loss was used. Table 4 reveals the
results of this parameter configuration.
%
F
Variance
P layer
72.07
P layerran
22.83 74.61
±2.66
Draw
5.1
Table 4: Tic-tac-toe: initial setup.
supreme
The coevolutionary PSO algorithm does not manage to
produce agents that perform very well, with a mediocre improvement when compared to Table 1. The P layersupreme
under performs in comparison to P layerstatic . Figure 3
Figure 3: Best agents performance: initial setup.
The graph clearly indicates a premature convergence taking
place. The best agents in both populations are clearly
struggling to find better solutions and do not improve at
all in the later stages during training. The performance
of the weights belonging to both agents remain constant
throughout training, with no arms race pattern being
visible. By investigating the velocity values during training,
it was noticed that these grew considerably large in all
dimensions. The maximum velocity value Vmax was
therefore set to 1, a very small value, to investigate how this
affects training. The results are given in Table 5, clearly
showing an improvement.
%
F
Variance
P layersupreme 80.34
P layerran
13.76 83.48
±3.65
Draw
5.9
Table 5: Tic-tac-toe: V max = 1.
Figure 4 shows how this change has affected the performance of the best agents in each population. Both
agents are now alternating and continously finding newer
weights with improved solutions during training, clearly
revealing an “arms race” effect, as described in Section
3. Sixth degree polynomials have been fitted through the
performance of both sets of weights, making this more
apparent.
%
F
Variance
P layer
85.02
GBest
P layerran
12.42 86.29
±3.23
Draw
2.56
P layersupreme 85.57
LBest
P layerran
11.77 86.9
±2.93
Draw
2.66
Table 8: Probabilistic 4x4x4: Different topologies.
Topology
supreme
%
F
Variance
P layer
85.22
10
P layerran
12.19 86.50
±2.54
Draw
2.59
P layersupreme 84.65
15
P layerran
12.61 86.02
±3.06
Draw
2.74
Table 9: Probabilistic 4x4x4: Different hidden layer sizes.
Hidden
supreme
Figure 4: Best agents performance: V max = 1.
7.2 Probabilistic 4x4x4 Tic-Tac-Toe
7.2.1 Hand-Crafted Evaluation
The results of the hand-crafted static evaluation function for
the probabilistic 4x4x4 tic-tac-toe game are shown in table
6.
%
F
static
P layer
99.43
P layerran
0.32 99.55
Draw
0.25
Table 6: Probabilistic 4x4x4: hand-crafted function.
The results indicate that the hand-crafted evaluation function for this game is extremely good, but not perfect. This
is expected though, since the game is probabilistic, making it impossible for a player to constantly win or draw
games, since the probabilistic element can not always favor
P layerstatic .
7.2.2 Coevolutionary PSO Results
Using the exact same setup that proved successful for
tic-tac-toe, the results for probabilistic 4x4x4 tic-tac-toe is
shown in Table 7.
%
F
Variance
P layersupreme 86.19
P layerran
11.21 87.49
±2.95
Draw
2.6
Table 7: Probabilistic 4x4x4: Initial setup.
The results look very promising, with 86.19% of the games
won. A series of other simulations were done, investigating different topologies, neural network architectures and
swarm sizes. Table 8 shows the results when the Global
Best and Local Best topologies were used, indicating that
the Von Neuman topology performs marginally better (Table 7). The Von Neuman topology had managed to succeeded in finding solutions that win an average of 1.17%
more than the solutions found by Global Best and an average of 0.62% more than solutions found by Local Best.
%
F
Variance
P layer
92.57
15
P layerran
5.93
93.32
±2.83
Draw
1.5
P layersupreme 95.06
20
P layerran
3.85 95.615
±2.48
Draw
1.09
P layersupreme 96.55
25
P layerran
2.94
96.8
±2.12
Draw
0.51
Table 10: Probabilistic 4x4x4: Different swarm sizes.
Size
supreme
Table 9 indicates that no performance improvement was
gained with an increase in the number of the hidden neurons. Therefore the hidden layer size is not increased and
remained as a value of 7. Different population sizes were
investigated, with the results shown in Table 10. Population sizes of up to 25 were used, which included sizes of 15,
20 and 25 agents. The results clearly show that there is a direct relation on the improvement of agents as the population
sizes increase. Larger swarm sizes offer a larger diversity
of solutions, enabling the PSO algorithm to discover better
solutions. One must not forget though the increase in complexity of the coevolution process, since a larger population
requires more games to be played to evaluate agents. Even
though a population size of 25 produced a 1.49% improved
winning result over a 20 sized population, the complexity
increase does not make this worth while. The population
size of 20 would therefore be more favorable.
8 Conclusions and Future Work
The coevolutionary technique in combination with a particle swarm optimization algorithm has shown to be very successful in finding strong agents for the probabilistic 4x4x4
tic-tac-toe game. The most optimal setup presented in this
paper was capable in producing a network that could almost match the performance of the hand-crafted evaluation function. Future work includes a more detailed study
with regards to the PSO parameters and the application of
the technique to more complex probabilistic games such as
backgammon and poker. One important coevolutionary aspect was not approached in this report, which forms the next
step in improving the technique. This is concerning the selection of agents within populations which are used for fitness sampling. Different selection strategies should be examined which might prove more efficient than the selection
of the entire population which is adopted in this paper. In
addition to that, a more in depth investigation must be done
to examine how different scoring schemes affect training.
Scoring schemes that award equal points for winning and
drawing would encourage defensive strategies to be found,
while strategies that only award wins with points would encourage aggressive strategies to be found.
Bibliography
[1] Chellapilla K, Fogel D. (1999) Evolving neural
networks to play checkers without expert knowledge. IEEE transactions on neural networks,
10(6):1382-1391.
[2] Franken N. (2004) PSO-based coevolutionary
game learning. MSc thesis, Department of Computer Science, University of Pretoria, South
Africa.
[3] Franken N, Engelbrecht AP. (2003) Comparing PSO structures to learn the game of checkers from zero knowledge. In proceedings of
the IEEE congress on evolutionary computation
(CEC2003), Canberra, Australia.
[4] Franken N, Engelbrecht AP. (2004) PSO approaches to co-evolve IPD strategies. In proceedings of the IEEE congress on evolutionary computation (CEC2004), Portland, USA.
[5] Franken N, Engelbrecht AP. (2004) Evolving
intelligent game playing agents. South African
Computer Journal.
[10] Kennedy J, Eberhart RC. (1995) The particle swarm: social adaptation in informationprocessing systems. (eds) Corne D, Dorigo M,
Glover F. New ideas in optimization, McGrawHill, pages 379-387.
[11] Kennedy J, Mendes R. (2002) Population structure and particle swarm performance. In proceedings of congress on evolutionary computation
(CEC 2002), Honolulu, Hawaii, USA.
[12] Knuth D, Moore R. (1975) An analysis of alphabeta pruning. Artificial intelligence, 6(4):293326.
[13] Messerschmidt L, Engelbrecht AP. (2002) Learning to play games using a pso-based competitive learning approach. In proceedings of the 4th
Asia-Pacific conference on simulated evolution
and learning, Singapore.
[14] Pollack B, Blair AD. (1998) Co-evolution in the
successful learning of backgammon strategy. Machine learning, 32(3),pages 225-240.
[15] Reinfeld A. (1983) An improvement of the scout
tree-search algorithm. Journal of the international
computer chess association, 6(4):4-14.
[16] Samuel A. (1959,1967) Some studies in machine
learning using the game of checkers. IBM journal
of research and development.
[17] Schaeffer J. (2001) The games computers (and
people) play. Academic Press, Vol.50, pages 189266. (eds) Zelkowitz MV.
[18] Shannon C. (1950) Programming a computer for
playing chess. Philosophical magazine, 41:256275.
[19] Teasuro G. (1995) Temporal difference learning
and TD-Gammon. Communications of the ACM,
38(3):58-68.
[6] Brudno A. (1963) Bounds and valuations for
abridging the search for estimates. Problem of cybernetics, 10:225-241. Translation in Problemy
Kibernetiki, 10:141-150.
[20] Turing A. (1953) Digital computers applied to
games. (eds) B Bowden. Faster than thought,
pages 286-295, Pitman.
[7] Billings D, Pena L, Schaeffer J, Szafrom D.
(1999) Using probabilistic knowledge and simulation to play poker. In AAAI national conference, pages 697-703.
[21] van den Bergh F. (2002) An analysis of particle swarm optimizers. PhD thesis, Department of
Computer Science, University of Pretoria, South
Africa.
[8] Ginsberg M. (1999) GIB: Steps toward an expertlevel bridge-playing program. In international
joint conference on artificial intelligence, pages
584-589.
[22] Werbos PJ. (1974) Beyond regression: new tools
for prediction and analysis in the behavioral sciences. PhD thesis, Harvard University, Boston,
USA.
[9] Holland JH. (1990) ECHO: explorations of evolution in a miniature world. (eds) Farmer JD, Doyne
J. Proceedings of the second conference on artificial life, Addison-Wesely.