Video Game Strategy Learning System using Reinforcement

Video Game Strategy Learning System
using Reinforcement Learning and
Inference Rules
T H E S I S
Master of Science in Intelligent Systems
Instituto Tecnológico y de Estudios Superiores de Monterrey
By
Humberto Javier Quijano García
May 2009
Video Game Strategy Learning System
using Reinforcement Learning and
Inference Rules
THESIS
Master of Science in
Intelligent Systems
Instituto Tecnologico y de Estudios Superiores de Monterrey
By
Humberto Javier Quijano García
May 2009
Instituto Tecnológico y de Estudios Superiores de
Monterrey
Graduate Division in Mechatronics and Information
Technology
Thesis committee members recommend the present thesis of Humberto Javier
Quijano García to be accepted in partial fulfillment of the requirements for the degree
of Master of Science in:
Intelligent Systems
Thesis committee:
Leonardo Garrido Luna, Ph.D.
Thesis advisor
Carlos Astengo Noguez, M.S.
Olivia Maricela Barrón Cano, Ph.D.
Examiner
Examiner
Joaquín Acevedo Mascarúa, Ph.D.
Chairman of the Graduate Programs
Engineer School
May de 2009
Video Game Strategy Learning System
using Reinforcement Learning and
Inference Rules
By
Humberto Javier Quijano García
THESIS
Submitted to the Graduate Program in Mechatronics and Information Technology
in partial fulfillment of the requirements for the degree of Master of Science in
Intelligent Systems
Instituto Tecnológico y de Estudios Superiores de Monterrey
Campus Monterrey
Monterrey, N.L. May 2009
To my parents, Humberto Quijano and Lucila García, whom always have been there
giving me their support in the hard times.
To my brother Arturo Quijano, who loves video games.
Acknowledgments
HUMBERTO JAVIER
Instituto Tecnológico y de Estudios Superiores de Monterrey
May 2009
vi
QUIJANO
GARCÍA
Video Game Strategy Learning System
using Reinforcement Learning and
Inference Rules
Humberto Javier Quijano García, M.S.
Instituto Tecnológico y de Estudios Superiores de Monterrey, 2009
Thesis advisor: Leonardo Garrido Luna
Contents
Acknowledgments
vi
Abstract
vii
List of Tables
x
List of Figures
xi
List of Algorithms
Chapter 1 Introduction
1.1 Problem definition .
1.2 Hypotheses . . . . .
1.3 Objectives . . . . . .
1.4 Scope . . . . . . . .
1.5 Document structure .
xii
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Chapter 2 Background and Related Work
2.1 Repetitive Games . . . . . . . . . . . . .
2.2 Space Search Games . . . . . . . . . . .
2.3 Real Time Strategies Games . . . . . . .
2.4 Summary . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
1
2
4
6
7
.
.
.
.
8
9
10
10
14
Chapter 3 IRBREL: Inference Rule-Based Reinforcement Learning Algorithm
3.1 IRBREL Algorithm Specifications . . . . . . . . . . . . . . . . . . . . .
3.2 The Knowledge Base . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.1 World States . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2.2 Inference Rules . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 The Learning Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.1 Prior-Knowledge Phase . . . . . . . . . . . . . . . . . . . . . . .
3.3.2 Play Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3.3 Improvisation Phase . . . . . . . . . . . . . . . . . . . . . . . .
viii
16
16
17
18
19
21
22
23
25
3.3.4 Meditation Phase . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 4 The IRBREL Algorithm in Operation
4.1 Missionary-Cannibal Game . . . . . . . . . . . . . .
4.2 Satisfying the Special Requirements of the IRBREL
4.3 Prior-Knowledge Phase in Operation . . . . . . . .
4.4 Play Phase in Operation . . . . . . . . . . . . . . .
4.5 Meditation Phase in Operation . . . . . . . . . . .
4.6 Improvisation Phase in Operation . . . . . . . . . .
4.7 The Learning Cycle Continues . . . . . . . . . . . .
4.8 Summary . . . . . . . . . . . . . . . . . . . . . . .
28
32
.
.
.
.
.
.
.
.
34
34
35
36
38
40
42
44
48
Time Strategy Game
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
50
50
53
53
56
57
58
59
61
63
64
Chapter 6 Conclusions
6.1 Summary and Principal Results . . . . . . . . . . . . . . . . . . . . . .
6.2 Contributions and Relevance of the IRBREL Algorithm . . . . . . . . .
6.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
68
68
72
73
Bibliography
76
Vita
79
Chapter 5 Using IRBREL Algorithm in a Real
5.1 Flag’s Capture Game . . . . . . . . . . . . . .
5.2 The Performance Analyzer . . . . . . . . . . .
5.3 Special Considerations . . . . . . . . . . . . .
5.4 Flag’s Capture Experiments . . . . . . . . . .
5.4.1 Brainless Agent . . . . . . . . . . . . .
5.4.2 Dummy Agent . . . . . . . . . . . . .
5.4.3 Kamikaze Agent . . . . . . . . . . . .
5.4.4 Commando Agent . . . . . . . . . . . .
5.4.5 Experiment Conclusions . . . . . . . .
5.5 Summary . . . . . . . . . . . . . . . . . . . .
ix
. . . . . .
Algorithm
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
List of Tables
4.1 List of Missionary-Cannibal Actions. . . . . . . . . . . . . . . . . . . .
4.2 List of Missionary-Cannibal Variables. . . . . . . . . . . . . . . . . . .
x
35
36
List of Figures
2.1 Learning agent process using Reinforcement Learning.[23] . . . . . . . .
11
3.1 Inference rule. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Learning cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Assignation of the utility to the inference rules. . . . . . . . . . . . . .
19
21
31
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
4.10
4.11
4.12
4.13
Missionary-Cannibal beginning game. . . . . . . . .
First possible actions. . . . . . . . . . . . . . . . . .
The first inference rules learned. . . . . . . . . . . .
Consequences of the first action. . . . . . . . . . . .
Consequences of the second action. . . . . . . . . .
Inference rule update. . . . . . . . . . . . . . . . . .
The new inference rules learned. . . . . . . . . . . .
The updated inference rules. . . . . . . . . . . . . .
Second possible actions. . . . . . . . . . . . . . . .
Consequences of the improvisation. . . . . . . . . .
Learned inferences rules after improvisation phase. .
World states with the same utility. . . . . . . . . .
Assignation of the utility to the inference rules. . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
35
37
37
39
40
40
41
42
43
44
44
45
47
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
5.10
Flag’s Capture video game in action. . . .
Flag’s Capture video game - start position.
Game progress in Brainless AI category. .
Game score in Brainless AI category. . . .
Game progress in Dummy AI categoryy. .
Game score in Dummy AI category. . . . .
Game progress in Kamikaze AI category. .
Game score in Kamikaze AI category. . . .
Game progress in Commando AI category.
Game score in Commando AI category. . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
51
52
58
59
60
60
61
62
63
64
6.1 Humberto Javier Quijano Garcı́a. . . . . . . . . . . . . . . . . . . . . .
79
xi
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
List of Algorithms
3.1
3.2
3.3
3.4
3.5
3.6
Algorithm that gets a list with all the inference rules that fire.
Prior-Knowledge Phase Algorithm. . . . . . . . . . . . . . . .
Play Phase Algorithm. . . . . . . . . . . . . . . . . . . . . . .
Algorithm that Calculates the Probability to Improvise. . . . .
Improvisation Phase Algorithm. . . . . . . . . . . . . . . . . .
Meditation Phase Algorithm. . . . . . . . . . . . . . . . . . .
xii
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
20
22
24
26
27
29
Chapter 1
Introduction
Machine Learning is synonymous of advanced computing. Machine Learning is
a growing body of work that exists on the use of such techniques to solve real world
problems. The complex and/or ill understood nature of many problem domains, such
as data mining or process control, has led to the need for technologies which can adapt
to the task they face.
Machine Learning has been greeted with a certain amount of caution by video
games developers, and until recently, has not been used in any major games releases.
Many video game companies are currently looking at the possibility of making video
games that can match the player’s ability by altering tactics and strategy, rather than
by improving the ability of the opponents using a difficulty system.
Even on the toughest difficulty settings of most video games (specially on First
Person Shouter games like Doom, released in 1993 ), most players have a routine, which
if successful, will mean that they win more often than not. However, they would surely
not be so smug if the artificial intelligence (AI) could work out their favorite hiding
places, or uncover their winning tactics and adapt to them. This could become a very
important feature of future releases, as it would prolong game life considerably.
The importance for a video game to learn depends on the video game itself. For
some video games, the average player does not appreciate any significant advance if the
game learns and the learning effort is a waste of time and money. Nevertheless, video
games like Black & White (2001) have not been half as successful if it was not for the
creatures that could be taught, through kindness or cruelty, to mimic their master.
1.1
Problem definition
There are not any major game releases that use Machine Learning techniques to
improve AI performance. Most games train AI by just controlling the frequency of
errors in tactical decisions. Then, once the game is released, AI just implements the
tactical decision with fewer errors and there is not further learning.
On released versions of video games, very few still uses learning techniques, like
1
reinforcement learning[23, 24], to improve the agent’s performance. However, on Real
Time Strategies Video Games, the decisions and learning must be made quickly to not
negative affect the performance. If an agent takes too long thinking what to do, then
another agent could take advantage of that moment of inactivity. This quick decision
need is a problem for learning agents in Real Time Strategy Video Games. The learned
knowledge could help to use a better strategy. Notwithstanding, the learning and
decision process should not take too long. Otherwise, it will hinder the performance of
the agent.
Artificial Intelligence has helped to create automatic systems that find a specific
problem solution. Specialized techniques are used commonly to find a solution making
world state space searches or using probabilistic and logical techniques. Nonetheless,
there are still a lot of limitations. A particular intelligent system made to solve a
specific problem could not work if it is applied to another similar problem. Besides,
intelligent systems made to solve a lot of kind of problems have not the same efficiency
than if the intelligent system was made to solve just a particular problem. This leads
to the no free lunch problem. Consequently, if the intelligent system is made to manage
learning in all kind of video games, then the efficiency is smaller than if the intelligent
system was made to manage learning in just one video game.
One of the main capabilities of the human beings is they capacity to improvise in
unexpected situations. Besides, the improvisation is a factor that helps a lot in Real
Time Strategy Video Games. Nevertheless, the Artificial Intelligence Systems are not
good making improvisations.
1.2
Hypotheses
There are three hypotheses in this research. The first one refers to the way the
knowledge could be represented. The second refers to the representatively and usefulness of that way to represent knowledge. Finally, the third hypothesis refers to the way
that the represented knowledge lets to take quick decisions.
The cognition, knowledge and memory of the human beings are not located
in specific places of the brain crust. All that knowledge is represented by neurons
through cortical networks, which distribute and interconnect neurons among them via
synapses[13, 7]. These groups of neurons (cortical network) receive, individually, impulses to fire or not fire. Once all the neurons of the network fires then it means the
representation of some knowledge.
Some neurons have more weight than others. Besides some cortical networks
behave as flags, which make some cortical networks fire or not fire. There is a branch
of AI which tries to emulate that behavior which is named Neural Networks[26]. Neural
Networks are supposed to work as cortical networks once they are completely trained.
2
The human brain does not behave as Neural Networks. The human brain has
the capability to represent knowledge and use that knowledge, even if the knowledge
represented is not true or not all true. For the human brain, contrary to Neural
Networks, there is not need to train the entire cortical network, just is needed to
add a new representation of the knowledge.
The human brain creates, modify or forget knowledge by experience, observation
and reinforcement learning. The brain updates all the knowledge learned during the
day while the person is sleeping. This process consists in convert short time memory in
long time memory[2, 3].This process makes the brain to remember and updates relevant
knowledge meanwhile irrelevant knowledge is forgotten.
The hypothesis is that it is possible to emulate the behavior of the human brain
learning process using a different approach than the Neural Networks. This proposed
different approach uses variables and inference rules to emulate the cortical networks.
The neurons are represented by variables which will represent a stimulation, sensor or
perception in an agent. For example, a variable could represent if is raining or if the
agent is on the wild, etc. The cortical network is represented by an inference rule. The
inference rule could be used to represent any kind of knowledge. For the purpose of
this research, the inference rules will be used to represent strategies, consequences of
the chosen actions and the conditions in which those strategies are useful and valid.
For example, an inference could say something like this: if the weather is raining and
the agent is on the wild then the agent will get wet.
The knowledge represented by the inference rules could be true or valid in some
cases and false or invalid in other cases. However, the idea is that inference rules begin
very naive and general. Then, inaccurate inference rules could be updated using the
learned knowledge. For example, once the last inference rule is found to be inaccurate,
it could be updated as follows: if is raining, the agent is on the wild and the agent is
not in cover, then the agent will get wet.
The human beings have a similar learning process to learn cause-effect events.
They begin with a very general knowledge and as needed that knowledge gets more
specific or general. This could lead to learn some superstitions, in which cause-effect
events are attributed to wrong factors. This events happen because, in most cases, there
are not observable all the factors that provoke the observed consequences. Examples
of these superstitions could be the famous dance to make rain or the cargo cult.
The cargo cult[16, 37, 17] is a superstition that happened during the Second World
War. The American soldiers arrived to islands near Australia and created provisional
landing platforms to prepare the attack against Japan. Notwithstanding, the islands
were inhabited by natives. In order to stay in good mood with the natives, the Americans gave them some food supplies. The natives noticed that the food was brought from
the airplanes that landed in the landing platforms. After the war ended, the natives
began to build their own landing platforms to invoke the airplanes and get more food
3
supplies. This happened because the natives did not had knowledge of all factors that
made the airplanes landed with food supplies.
This leads to a second hypothesis: The knowledge represented by a naive inference
rule is useful even if the knowledge is not completely accurate or if it is a superstition.
It is assumed that after a learning process those superstitions or naive strategies will
lead eventually to very useful knowledge and strategies. Then, the system will use that
knowledge to decide what to do or what not to do.
In Real Time Strategy Video Games, the agents must take decisions almost instantly. If an agent takes too long thinking what to do, then another agent could take
advantage of that moment of inactivity.
The third hypothesis says that using a system based in inference rules will help
to take decisions quickly. The inference rules represent strategies in the game. If an
inference rule fires, it means that the strategy is available or it is viable to use in the
current circumstances. It is assumed that instead of making a whole world state space
search, if the system only takes in account the inference rules that fired and execute
the best of them, then the system will save time.
1.3
Objectives
The goal of this research is to create a system module for Real Time Strategy
Video Games. The system module will learn from its experience and observations
using Reinforcement Learning.
The system module will be used as the AI of a Real Time Strategy Video Game.
The idea is that the video game has agents with an AI that improves with time. The
AI will learn from its chosen actions and behave according to the learned knowledge.
That way, the human players could not use the same strategies to beat the game.
Consequently, the game will force the human players to keep improving their skills.
One of the most important capabilities of this system will be its ability to improvise
under unknown situations. Besides, it is intended that the system module could adapt
and update its knowledge. That way, the next time the system module is under the
same circumstances, the system will know what to do or what not to do.
In order to create the proposed system, there are five main problems that must
be solved:
Knowledge Representation: Must be defined how the learned knowledge will be
represented. The learned knowledge should be easily accessed and should let the
system to choose quickly what to do.
The Use of the Learned Knowledge: In Real Time Strategy Video Games there
is not much time to take decisions. If the system takes much time thinking what
4
do, other agents could take advantage of this moment of inactivity. Consequently,
the system must quickly examine its learned knowledge and decide what to do.
Improvisation: The system should improvise to try something new and to try to
improve its performance. Nonetheless, if the system improvises all the time, then
the agent will execute random actions all the time and he will never achieve
anything. The system should balance the improvisation with the use of the its
learned knowledge. Besides, the system should decide when is better to use the
learned knowledge and when is better to improvise and how improvise.
Update Learned Knowledge: The learned knowledge should be updated with the
time. If some learned knowledge is no longer useful, then that knowledge should
be updated or deleted. Nevertheless, the main problem is to know what knowledge
is useful and what knowledge is obsolete.
Generate Prior-Knowledge: Some times, there is no previous knowledge of the
game. In those circumstances should be decided how generate a prior-knowledge
that let the system begins to play the game.
To accomplish that, the system will be divided in four phases. Each phase solves
a particular problem or need. The first phase is focused in create prior-knowledge to
let system begins to play the game. If there is prior-knowledge of the game, then this
phase is skipped. The second phase will use all the learned knowledge to quickly decide
which is the best action to execute. If at some point the system does not know what
to do or the performance is poor, then the third phase will handle all improvisations.
This phase, will decide how and when improvise. Finally, the fourth phase will update
all the learned knowledge.
In order to have a knowledge representation that can be quickly accessed by the
system and let it to take quick decisions, the knowledge will be represented using
inference rules. The inference rules will represent strategies or actions that could be
executed in certain conditions. Besides, the inference rules represent how the world
changed after these actions were executed.
If the prerequisites in which the strategy could be used are meet, then the inference
rule will fire. So, instead of having a system that analyses its learned knowledge and
then decide what to do, the system should only pay attention to the inference rules
that fired. Then, from the inference rules that fired, it is used the one that gives the
system the best performance.
To create and update the inference rules, the system needs an additional kind
of knowledge, which is the world states. The world states will represent a picture of
how the game is at that particular instant and how good or bad is for the agent that
particular world state. Besides, the system will need to know what actions are available
to its use.
5
Using the world states and the available actions, the system could create inference
rules that represent how the world state was before the action was executed and how
world state will be after the action is executed. The system will learn, modify and
forget inference rules using Reinforcement Learning.
This research is different from any other previous work and is important to make
this research because these several reasons:
1. There are not any major game releases that uses learning.
2. From the few video games that uses learning to improve the AI, mostly only
trains the AI if it is in the develop phase and on the final release the AI is static.
3. Most artificial intelligence systems has the capability to learn and use that knowledge. However, there are not intelligence systems that his main capability is the
improvisation.
4. The proposed learning method is a variant of the well known Reinforcement
Learning. This variant is inspired by the nature of the cortical networks of the
human brain. This method combines Reinforcement Learning techniques and the
use of inference rules to represent the learned knowledge.
5. The learned knowledge represented by the inference rules will be useful even if
the system is not fully trained. This will let the system to take decisions even if
are not known the consequences of all possible actions.
6. If the system only focus on the inference rules that fire, the time of respond will
be faster than if a system make a search in the whole world states space, which
is important for Real Time Strategies Games.
1.4
Scope
The scope of this research is to create a system module that is able to learn and
improvise in Real Time Strategy Video Games. Besides, the system should be able to
decide quickly what to do.
The learned knowledge will be represented using inference rules and world states.
The world states represent a picture of how is the world in that instant and how good
or bad is this world state for the agent. The inference rules represent how useful is to
execute a particular action or strategy under certain conditions.
The system will need to know how valuable the world states and inference rules
are. It is not intended that the system module learns how to evaluate the world states
and inference rules. That evaluation should be calculated by an external module or by
6
the video game itself, then the system will use those values to learn how useful are the
chosen actions.
It is not scope of this research the creation of a representation of the world states.
The variables that will be used to define a world state should be defined by an external module or by the video game itself and these variables must have integer values.
Notwithstanding, the system will be able to learn how the world states are interconnected among them and how to change from one world state to another one.
The system module is intended to work only in deterministic and completely
observable environments. That way the system will be able to learn why the world
state changed after an action was executed.
The scope of this work is to only prove that the proposed learning algorithm
works in Real Time Strategy Video Games. The comparison of the efficiency of this
learning method against any other Machine Learning techniques, like Neural Networks,
Q-Learning, etc, will be a scope of a future work.
1.5
Document structure
This chapter introduces a brief view of the Machine Learning Techniques applied
to the video game’s industry. Besides, it is described the problems to use Machine
Learning techniques in Real Time Strategy Video Games. Furthermore, the chapter
describes the ideas and hypotheses used to create the proposed learning system module
and the objectives and scopes of this research.
The chapter 2 makes a background of what has been done in the video game
industry to improve the AI and what is the common technique to make video games
that learns from experience to adapt human players. Besides, the chapter makes a
short description of what are doing other researches in this area and the differences
among this research and their work.
In the chapter 3 is described in detail how the learned knowledge is represented
and how the learning process works. Besides, the chapter 4 describes step by step,
using the Missionary-Cannibal game, how the learning algorithm works.
The chapter 5 measures the performance of the learning algorithm in a Real Time
Strategy Video Game. Besides, are described special considerations that should be
taken in account to improve the performance of the learning algorithm.
Finally, chapter 6 describes the contributions made by this research, the future
work and possible applications of this research.
7
Chapter 2
Background and Related Work
There many ways in which video games could be categorized. This depends on
what is taken in account. The following list describes the most commonly used categories:
Platform: NES, SNES, Nintendo 64, PlayStation, X Box, PC, etc.
Genre: Action, Adventure, RPG, Sports, Simulation, etc.
Classification: Everyone, Teens, Adults.
Graphical Style: First Person, 3D, Side-Scrolling Perspective, Top-Down Perspective.
Machine Learning: Deterministic or Non-Deterministic, Completely Observable or
Not Completely Observable, Single Agent or Multi Agent.
For the purposes of this research, the video games will be categorized according
to the strategies used to win the game.
Repetitive Games: These kind of games are very simple. All these kind of games are
Zero-Sum games, like the Prisoner’s Dilemma game. If some of these Sum-Zero
games are played many times, the agents should take in count what the other
agents could do. For example, in Prisoner’s Dilemma game, if an agent always
betrays the other agents, then the betrayer agent should know that in future
games, the other agents will be less eager to not betray him.
Space Search Games: Examples of this kind of games are Chess, Checkers and TicTac-Toe. For these kinds of games, the solution is to make a whole world state
space search to look for the best action to execute. Usually this kind of games
uses a Min-Max function[26] to maximize the game results.
Real Time Strategies Games: This kind of games could be solved using techniques
of the other kinds of games. Nonetheless, the agent must take decisions in short
8
periods of time. If the agent takes too long to decide what to do, the other agents
could take advantage of its inactivity, like in the simulated RoboCup Soccer’s
Game[1].
The proposed learning system module is designed to work only in Real Time
Strategy Video Games that are deterministic and completely observable. The learning
system is intended to be used in single agent games. Nevertheless, it is possible to use
the system in multi-agent games if the agents do not cooperate among them.
In the following sections are described how typically is managed the AI in different kinds of video games and the works that has been done at the moment. Since this
research is focused in Real Time Strategy Video Games, there will be only a brief description about video games categorized as Repetitive games and Space Search Games.
2.1
Repetitive Games
The Minimum Regret is a popular criterion in decision making processes. Regret
is defined as the difference between the payoff that could have been achieved, given the
nature of the choices, and what was actually achieved. An expert selection rule is said
to minimize regret if it yields an average payoff as large as that of any single expert,
against any fixed sequence of actions chosen by the opponent.
The Minimum Regret uses algorithms to choose, at each stage, an expert from a
probability distribution that is related to the payoff accumulated by the expert prior to
that stage. Since the experts are compared on a sequence by sequence basis, it makes
sense only under the assumption that nature’s choices are independent of the decision
maker’s choices.
In Repetitive games, the assumption that the opponent’s choices are independent
of the player’s choices is not justified. This is because the opponent is likely to base
his choices of actions on the past history of the game.
Typically in Repetitive games are used approaches that often focus on algorithms
that asymptotically achieve an equilibrium of the one-shot game. Few others shift away
from that approach to consider learning strategies that attempt to play long-term bestresponse to the opponent’s observed strategy[10].
For some strategies of the opponent in a repetitive game, it is impossible to learn
the best response. In those cases, typically is designed a relatively small number of
strategies which can be expected to work well in real-life situations. If only a limited
number of strategies are available (like scissors, rock and paper game), there are used
expert algorithms. The goal of these expert algorithms is to learn from experience how
to combine the advice from multiple experts and to make good sequential decisions
on-line.
9
2.2
Space Search Games
Most AI game researches has focused on games that can be described in a compact
form using symbolic representations, such as board games and card games[15]. Besides,
most of these games, like Checkers or Chess, have a relative small number of world
states. The number of world states is small if we compare them against Real Time
Strategy games which have hundreds or even thousands times world states than a
game like Checkers.
These kind of games are usually solved using a world state space search. The
most commonly world state space search algorithm used is the Min-Max function. For
games with a lot of world states, there are not used whole world state space searches.
Instead of a whole space state, there are used world state space searches only in the
“n” neighbor world states.
In more complex games, like Chess, there are used additional AI techniques[12].
These techniques consists in learn from the opponent. The system analyses what the
other did in similar conditions to predict his next action. Examples of these techniques
are the Chinook[27], the Checkers program which became the world champion in 1994,
or Deep blue[8], the Chess program that defeated the world champion in 1997.
There are also used Reinforcement Learning techniques in multi-agent competitive
card games[20]. The Reinforcement Learning techniques are useful for games with a
lot of world states, like Chess and Go. In these kind of games are created thousands
or millions of small fragments of the board[30], then using Reinforcement Learning is
selected the best action to execute.
2.3
Real Time Strategies Games
The game industry has a big interest and demand for these kind of games[29, 35].
In video games like Mario Bros (1985), Zelda (1986), Black & White (2001), Doom
(1993), the AI or the human players need to take decisions in short periods of time.
Those decisions have a big impact in how the game evolves. Many games developers
use learning algorithms in their games to create a better AI[28]. However, typically the
AI is frozen once the game is released.
For some designers of these kind of games is desirable to create a false impression
of AI learning. It is commonplace within the video game industry to create cheating
AI systems which gives a false impression of learning. An impression of learning can be
easily implemented by controlling the frequency of errors in tactical decisions made by
the AI. Then, the AI just executes the strategy with the least number of tactical errors.
This creates a realistic illusion of an intelligent learning process. Notwithstanding, this
cannot be used unless the desired behavior is already known. In other words, this is
useless for learning to counter human player strategies.
10
The central process of learning is the adaptation of behavior. This adaptation
is intended to improve the performance. Fundamentally, there are two methods of
achieving this:
• Directly changing behavior by testing modifications to it.
• Indirectly by making alterations to certain aspects of behavior based on observations.
There are positive and negative sides to each one, but direct adaptation does have
the advantage of not limiting behavior, which means that ultimately, a better goal may
be achievable.
For the purposes of explanation of the typical learning game strategies process[9,
4, 23], will be used the Flag’s Capture game. There are two agents, which are enemies.
The goal of the game is to capture the opponent’s flag. They must do this without
getting hit by paint-bullets.
Figure 2.1: Learning agent process using Reinforcement Learning.[23]
11
As shown in the figure 2.1, a typical learning agent is composed by a four fundamental parts: a learning element, a performance element, a curiosity element or
problem generator and a performance analyzer or critic.
The learning element is the part of the agent which modifies the agent’s behavior and creates improvements. The performance element is responsible for choosing
external actions based on the percepts it has received.
To illustrate this, consider that one of the agents is in the woods. He is aware that
an enemy agent is nearby. Then, the agent could choose an action like moving behind
a tree. This choice of action is made by the performance element.
The performance analyzer judges the performance of the agent against some suitable performance measure. This measure could be how close the agent is to get hit by
a paint-bullet, or how close is the agent to the rival’s flag. The performance must be
judged on the same percepts as those received by the performance element (the state
of affairs known to the agent). Once the analysis of performance has been made, the
agent must decide whether or not a better performance could be made under the same
circumstances. This decision is then passed to the learning element, which decides on
the appropriate alteration for future behavior, and modifies the performance element
accordingly.
The last part of the process makes sure that the agent advances in its learning,
and does not merely confine itself to previously observed behavior. This is dealt by
the curiosity element, called that way because it searches for a better solution. To
achieve an optimal performance, this element will pose new challenges to the agent in
an attempt to prevent bad habits developing.
To understand the benefits of this, it is needed to go back to the example. The
soldier agent is now hiding behind a tree. From his past experience, he knows that he
is safe to stay where he is and this would result in an adequate performance. However,
the curiosity element kicks in and suggests that he makes a break from his cover and
heads to a nearby tree which is closer to the enemy flag. That may result in the agent
ultimately get shot, but could also achieve a more desirable goal. It is then up to the
performance analyzer and the learning element to consider whether there is a benefit
to this change in strategy.
This typical style of learning process is known as Reinforcement Learning. This
means that the agent can see the result of its actions, but is not told directly what it
should have done instead. Consequently, the agent must use trial and error to evaluate
its performance and learn from its mistakes.
The advantage to this style of learning is that there is no limitation on the behavior. The limit is only restricted by the alterations suggested through the curiosity
element.
As the learning agent is ultimately part of a game, the agents should not be left
to learn for themselves how to play. The agents should be imparted with a fair degree
12
of prior-knowledge about the way to behave. In the case of the Flag’s Capture game,
this could include methods for avoiding fire by using cover, which may later be adapted
during the learning process.
This process is the typical style to create a learning AI, which adapts to the
situation for Real Time Strategy Video Games. Nonetheless, in the practice is common
to combine other techniques of artificial intelligence to complement the learning.
For complex Real Time Strategy Video Games, such as Role Playing Games
(RPGs), the number of choices at each turn ranges from hundreds to even thousands,
the incorporation of a learning AI is hard. To implement a game AI for complex
games, most developers resort to scripts, which are lists of rules that are executed sequentially. Scripts are a common choice in the game industry to implement game AI,
because they are understandable, predictable, adaptable to specific circumstances, easy
to implement, easily extendable, and usable by non-programmers[36].
Scripts are generally static and tend to be quite long and complex[5], which gives
rise to two problems. The first problem is that, due to their complexity, scripts are
likely to contain weaknesses[22]. These weaknesses can be exploited by human players
to defeat supposedly tough opponents with ease. The second problem is due to their
static nature. This static nature can not deal with unforeseen human player tactics.
Besides, the static nature does not let to scale the difficulty level to cater novice and
experienced human players at the same time. These two problems, which are common
to state of the art game AI[6, 31], hamper the entertainment value of games. Very few
uses dynamic scripting as learning technique[32, 25]. This technique uses an adaptive
rule base for the generation of game AI on the fly.
The use of these rules or scripts is similar to the idea of this research to use inference rules. Both represent a tactics or strategies to follow. Nevertheless, scripts differ
from the proposed inference rules in this research. The scripts are only instructions
of what to do once the preconditions are meet. Meanwhile, the inference rules also
represent what will happen if an action is executed under some preconditions. Besides,
it is up to the system to choose from all inference rules that fires which action take.
Furthermore, scripts tent to be static meanwhile the proposed inference rules try to
adapt and learn.
There are few game developers who use a different approach to manage the AI and
create adaptive games. This approach uses evolutionary computation, neural networks
or a combination of both[11, 34, 38, 33]. This approach is called Neuroevolution[21].
Several methods have been developed for evolving neural networks. One particularly appropriate for video games is called Neuroevolution of Augmenting Topologies
(NEAT), which was originally developed for learning behavioral strategies. The neural
networks control agents that select actions in their output based on sensory inputs.
NEAT is unique in that because it begins evolution with a population of small, simple networks and get complex versions of those networks over generations, leading to
13
increasingly sophisticated behaviors.
The idea of this research works in a similar way that NEAT does. Instead of
a small population of neural networks, the system begins with a small population of
inference rules. These inference rules represent a generalized knowledge and strategies.
However, the advantage against NEAT is that, with the proposed learning method, the
system should be able to learn even if there is not any inference rule in the knowledge
base. If there is no previous knowledge represented by the inference rules, the system
will be able to create a general knowledge via trial and error.
There is no commonly used, but there are few researches that use Vector Machine
techniques [18, 14, 19] in video games. Vector Machine techniques are used to classify
the data and decisions that the agents could take. It is more common to find Vector
Machine and Bayesian Networks techniques in the RoboCup Soccer’s simulation game.
2.4
Summary
There many ways in which video games could be categorized. This depends on
what is taken in account. For the purposes of this research, the video games will be
categorized according to the strategies used to win the games.
Repetitive Games: This kind of games are very simple. All these kind of games are
Zero-Sum games, like the Prisoner’s Dilemma game. If some of these Sum-Zero
games are played many times, the agents should take in count what the other
agents could do. For example, in Prisoner’s Dilemma game, if an agent always
betrays the other agents, then the betrayer agent should know that in future
games, the other agents will be less eager to not betray him.
Space Search Games: Examples of this kind of games are Chess, Checkers and TicTac-Toe. For these kinds of games, the solution is to make a whole world state
space search to look for the best action to execute. Usually this kind of games
uses a Min-Max function to maximize the game results.
Real Time Strategies Games: This kind of games could be solved using techniques
of the other kinds of games. Notwithstanding, the agent must take decisions in
short periods of time. If the agent takes too long to decide what to do, then the
other agents could take advantage of its inactivity.
The proposed learning system module is designed to work only in Real Time
Strategy Video Games that are deterministic and completely observable. The learning
system is intended to be used in single agent games. Nonetheless, it is possible to use
the system in multi-agent games if the agents do not cooperate among them.
14
A typical learning agent is composed by a four fundamental parts: a learning
element, a performance element, a curiosity element or problem generator and a performance analyzer or critic.
The learning element is the part of the agent that modifies the agent’s behavior
and creates improvements. The performance element is responsible for choosing external actions based on the percepts it has received. The performance analyzer judges
the performance of the agent against some suitable performance measure. Once the
analysis of performance has been made, the agent must decide whether or not a better
performance could be made under the same circumstances. The curiosity element poses
new challenges to the agent in an attempt to prevent the development of bad habits.
This typical style of learning process is known as Reinforcement Learning. This
means that agent can see the result of its actions, but is not told directly what it should
have done instead. Consequently, the agent must use trial and error, to evaluate its
performance and learn from its mistakes.
This process is the typical style to create a learning AI, which adapts to the
situation for Real Time Strategy Video Games. Nevertheless, in the practice is common
to combine other techniques of artificial intelligence to complement the learning, like
scripts, evolutionary computation, neural networks, etc.
15
Chapter 3
IRBREL: Inference Rule-Based Reinforcement
Learning Algorithm
The Inference Rule-Based Reinforcement Learning algorithm or “IRBREL” algorithm uses the same commonly used principles of reinforcement learning, which were
described in the chapter 2. The learning process is very similar to one shown in the
figure 2.1. However, instead of the common structure used in Reinforcement Learning
algorithms, the IRBREL algorithm uses inference rules to represent strategies and to
model the world.
The inference rules are created and modified using the learned world states and
the utility to be in that world state. The learning process of the IRBREL algorithm
works in cycles of four phases. In the following sections is described in detail how the
knowledge is represented and how the learning cycle works. Besides, in the chapter 4
is explained, using an example, how the IRBREL algorithm works.
3.1
IRBREL Algorithm Specifications
The IRBREL algorithm is designed as a module, which can be adapted to be used
in Real Time Strategy Video Games. Notwithstanding, in order to work properly, there
are some special requirements that must be satisfied.
The IRBREL algorithm needs a module that generates the world states of the
video game. The world states are used by the IRBREL algorithm to know the consequences of the chosen actions and to generate the inference rules.
The world state is defined by a group of variables, which describes how the world of
the video game is at that moment. A world state can be represented using any number
of variables. The variables can have any value from negative infinite to positive infinite.
The only restriction is that the values must be integers.
The developers of the world state generator have complete liberty of using any
number of variables and value ranges for those variables. A world state could be
represented using a single variable with thousands of possible values or thousands of
variables with few possible values. Nonetheless, before design how a world state is
16
represented, it is important to take in account that the more representative is a variable,
the more easily will for the IRBREL algorithm to learn.
The IRBREL algorithm also needs a performance analyzer module. The purpose
of the performance analyzer module is to calculate the utility of the agent to be in the
current world state. The IRBREL algorithm will choose actions that better moves the
agent from the current world state to the best of the known world states. The world
states that are between the current world state and the best of the known world states
should also have utility values between those two values.
The video game world is assumed to be deterministic, which means that the
changes in the world state are only determined by the actions executed by agents.
Changes made by environmental factors, like day and night, should be constant. Besides, the world must be also deterministic to agents’ perspective. Consequently, the
world must be completely observable or observable enough to let the agent identify why
the world state changed.
The IRBREL algorithm needs to know what actions are available to its use. Every
time the IRBREL algorithm is initialized, the user sends to the IRBREL algorithm a list
containing the available actions. For each unit of time of the video game, The IRBREL
algorithm asks the video game for the current world state. Then the IRBREL algorithm
chooses one of the actions of the list that the user sent when initialized the IRBREL
algorithm. Once the action is chosen, the IRBREL algorithm tells the agent to execute
that action. Finally, at the beginning of the next unit of time the IRBREL algorithm
analyzes the consequences of the taken action and chooses a new one.
Some actions are defined as complex actions. Simple actions are instantaneous;
only require a single unit of time to execute. Consequently, their consequences can be
analyzed at the next unit of time. Nevertheless, complex actions are not instantaneous
and require several units of time to accomplish. The IRBREL algorithm can manage
complex actions but the user must tell to the algorithm, using an internal flag, if there
is a complex action still in execution. That way the IRBREL algorithm is able to know
when analyze the consequences of the chosen action. Besides, the IRBREL algorithm
is able to interrupt complex actions if sees that are not getting good results or if the
conditions of the game changed. Section 3.3 and chapter 5 explain how the complex
actions are handled.
3.2
The Knowledge Base
The knowledge base is composed by four elements:
World States: Each world state represents a picture of how is the world in that
instant. Furthermore, each world state has an utility value, which is the utility
for the agent to be in that world state.
17
Inference Rules: Represent actions or strategies which can be done under some conditions. Besides, represents how the world is modified by executing those actions
and how useful are those world modifications.
Original World State: This is a world state that is linked to a specific inference
rule. There is one of this world states for each inference rule in the knowledge
base. The inference rules could be used to represent how the world states are
interconnected among them. Using that perspective, the premises represents the
original world state and the conclusions represents the new world state after
executing the described action. Inference rules are created very general (with
only one premise). These inference rules could become more specific as needed.
However, sometimes it is needed to know the original world state. If the inference
rule is too general, then the premises do not represent the original world state
and is needed this additional linked world state to know that information.
Know All Consequences Flag: This is a Boolean flag that is linked to a specific
inference rule. There is one of this Boolean flag for each inference rule in the
knowledge base. This flag is turned on if there are known the consequences of
executing every possible action under the same conditions of the linked inference
rule. This lets the IRBREL algorithm to know that there is no need to improvise
because all possible results are known.
In the following subsections are explained in more detail the use and representation
of world states and inference rules.
3.2.1
World States
World states represent a picture of how is the world in that instant. Furthermore,
each world state has an utility value, which is the utility for the agent to be in that
world state.
A world state is divided in two parts. The first represents the value of all variables
of the world in that instant. The second is just an utility value, which represents the
utility of the agent to be in that world state. That value must be generated by the
Performance Analyzer element (figure 2.1) as described by section 3.1.
The IRBREL algorithm learns each world state where the agent has been. The
world states are used to learn how the world is affected by the taken actions. Using
that knowledge, the IRBREL algorithm creates inference rules which represent those
changes.
The IRBREL algorithm uses world states to check if the used inference rules are
valid or invalid. Besides, uses the world states values to modify or update invalid
inference rules. Furthermore, the IRBREL algorithm keeps track of the current world
18
state to know if the agent gets stuck or is not making any progress at all, which leads
the IRBREL algorithm to improvise and try something new. This process is explained
in detail in section 3.3.
3.2.2
Inference Rules
Inference rules represent actions or strategies which can be done under some conditions. Besides, represents how the world is modified by executing those actions and
how useful are those world modifications to the agent.
Figure 3.1: Inference rule.
As shown in the figure 3.1, a inference rule has three elements:
Premises: These are composed by a single action and some variables with their respective values. The simplest and general inference rule is composed only by one
action. Premises represents the circumstances in which could be used the strategy
represented by the inference rule.
Conclusions: These are only composed by variables and represent how the world state
will be after the described action is executed.
Utility: This is a numeric value which represents the utility given to the agent if
executes the action under the circumstances expressed by the variables of the
premises. There is also taken in account the number of times the inference rule
was used and was valid or invalid.
Figure 3.1 shows how a typical inference rule looks. The value of each variable
is represented by the variables inside the square brackets. Usually the inference rules
are used for logical reasoning in Boolean operations. Consequently, the variables could
only take binary values. Notwithstanding, the IRBREL algorithm uses inference rules
which variables could take any value. The only restriction to these values is that the
range of values, which a variable can take, must be discrete (see section 3.1 for more
details).
The premises represent in which circumstances is valid to execute the strategy
described in the inference rule. Executing the described action or strategy under the
conditions specified by the premises will cause the conditions expected by the conclusions.
19
The variables of the conclusions section represent the new values that the variables
will have after executed the specified action. Some variables could have the same values
than the previous world state.
In some cases the consequences of the chosen actions will be different from the
consequences expected or defined by the conclusions section. In a deterministic world,
this only happens because inference rule was too general and the IRBREL algorithm
assumed that the results will work for all the cases and applied the inference rule when
it should not do it. For these cases, the inference rule is considered invalid and is
modified to consider additional conditions. The process of how an inference rule is
modified and updated is described in section 3.3.
In order to choose which action or strategy is available to execute, the IRBREL
algorithm only takes in account the inference rules that fires. An inference rule fires only
if the strategy is available to execute under the current circumstances. An inference
rule fires if its variables and values, specified by the premises, match with the current
world state.
Most inference rules in the knowledge base are general, which means that not
every variable is taken in account in the premises. For those cases, if a variable does
not appear in the premise section, then that variable is not taken in account to consider
if the inference rule fires or does not fire. The algorithm 3.1 shows the process to get
a list with all inference rules that fire.
Algorithm 3.1 Algorithm that gets a list with all the inference rules that fire.
list ← newList()
inf erences ←Number of Inference Rules in the KB
for i = 1 to inf erences do
if PremisesMatch(InferenceRule[i], currentWorldState) then
list.Add(Inf erenceRule[i])
end if
end for
return list
Once a game is running, some inference rules fire and others no fire, which depends
on the values of the world state. All the inference rules that fire, represents the possible
actions or strategies that the agent can follow. From all the inference rules that fire,
the IRBREL algorithm chooses the one which give the agent the biggest utility.
Both, world states and inference rules have utility values. These values have a
strict relation but could not be the same. The utility of the world state represents the
utility of the agent to be in that state. Meanwhile, the utility of the inference rule
represents the utility of executing that action. The agent only uses the utility of the
inference rule to take decisions. Nonetheless, the utility of the world states is used to
create, or modify utility values of the inference rules.
20
3.3
The Learning Cycle
The IRBREL algorithm works basically in cycles of three phases. Nevertheless,
for some cases is necessary to add an additional phase to help the agent to begin his
learning. In figure 3.2 could be appreciate a diagram of how this learning algorithm
works.
Figure 3.2: Learning cycle.
In the following subsections is described how each learning phase works. Each
phase has a very specific task. Once the task is accomplished the IRBREL algorithm
chooses which phase execute next.
The main goal of the learning cycle is to learn enough strategies to always win
the game. Inference rules represents the strategies to win the game and the conditions
in which these strategies work. At the beginning of the learning process all inference
rules are very simple and general. Throw the learning cycles, the inferences rules get
more specific and complex as needed. If the game or the agents are constant, then will
be a point where the IRBREL algorithm will learn enough strategies to always win
the game. However, if the game or the agents change with the time, then the learning
process will never end to the IRBREL algorithm. It is important to consider that the
game or the agents can change its behavior, but the world must still be deterministic.
21
3.3.1
Prior-Knowledge Phase
As the learning agent is the ultimately part of a game, usually it is not left alone
to figure out for itself how to play. Before any learning process begins, agents already
should know some basic strategies to help them to survive. Then, the agent after
some observations and experience modifies these basic strategies. Notwithstanding, in
some cases there is not any prior-knowledge and the agent must begin without any
knowledge.
Algorithm 3.2 Prior-Knowledge Phase Algorithm.
oldW orldState ← GetCurrentW orldState()
newW orldState ← null
newInf erenceRule ← null
availableActions ←Number of Available Actions (list given by the user)
for i = 1 to availableActions do
ExecuteAction(i)
W aitUntilActionEnds()
newW orldState ← GetCurrentW orldState()
if WorldStateChanged(oldW orldState, newW orldState) then
newInf erenceRule ← new InferenceRule(i, newW orldState)
Inf erenceRule.Add(newInf erenceRule)
end if
RestartGame()
end for
P layP hase()
At this point, the knowledge base is empty. The IRBREL algorithm has no idea
of what will happen if he executes certain action. The goal of the Prior-Knowledge
phase is to learn enough basic strategies to let the agent begin to play the game.
Consequently, the IRBREL algorithm does the only possible solution, experimenting
with the actions and world states. As shown by algorithm 3.2, the IRBREL algorithm
executes every action one time and checks if the world state changed. This process is
similar to any person that plays a video game that he never has seen before. The person
begins pressing all buttons to figure out what happened and how the game works. The
IRBREL algorithm uses the same strategy.
With that information, the IRBREL algorithm creates naive inference rules, which
represents the observed knowledge. At the beginning, each new inference rule created
assumes that the consequences only depend on the chosen action. These naive inference
rules could be not completely accurate. Perhaps some of these learning strategies only
work in a very specific situation. After some learning cycles, the inference rules become
more specific as needed.
For example, imagine that the IRBREL algorithm represents a person in the wild.
22
If the IRBREL algorithm makes the agent dance and observes that a rain begins, then
could be assumed to the IRBREL algorithm that every time the agent dances a rain
will come. Nonetheless, the rain is independent of the dance, it was just a coincidence.
The conclusion that the event was coincidence will be deduced after some learning
cycles. Consequently, the assumption that dance will provoke rain will be eventually
discarded by the IRBREL algorithm.
Generalized inference rules are very good to save memory space. That way, instead of having a knowledge base with several inference rules for each world state, the
IRBRREL algorithm has several inference rules in the knowledge base that applies to
several world states. There is no need to learn exactly all the conditions of the world
if some elements are irrelevant and some conditions can be generalized.
This Prior-Knowledge phase is executed only if there are not any inference rules
stored in the knowledge base. At the end of this phase, it is executed the Play phase.
3.3.2
Play Phase
The goal of the Play phase is to use the learned knowledge to win the game. At this
point, the IRBREL algorithm has a knowledge achieved via the Prior-Knowledge phase
or via previous learning cycles. The knowledge is represented by inference rules. Each
inference rule represents a strategy and each inference rule, (no matter how general is
it) is assumed to be completely accurate. Using that knowledge, the IRBREL algorithm
tries to play the game the best he can.
As shown by algorithm 3.3, the IRBREL algorithm only considers the inference
rules that fire (see algorithm 3.1). If an inference rule fires, means that are satisfied the
specific conditions in which the strategy works. From all inferences rules that fire, the
IRBREL algorithm chooses the inference rule that has the biggest real utility value. As
shown by equation 3.1, the real utility value is calculated using the utility value of the
inference rule and the percentage of times the inference rule is valid. That way most
frequently used and valid inference rules become more valuable than invalid and no too
frequently used inference rules.
valid
× utility
(3.1)
valid + invalid
After an inference rule is chosen, the IRBREL algorithm tells the agent (that the
IRBREL algorithm is controlling) the action to execute. Then agent of the video game
should tell the IRBREL algorithm if the chosen action is a complex action. Besides,
the agent of the video game should inform to the IRBREL algorithm if the complex
action ended. For more details see section 3.1.
If a complex action is running and the performance (utility value) gets worse from
one unit of time to another, then the IRBREL algorithm tells the agent of the video
RealUtility =
23
Algorithm 3.3 Play Phase Algorithm.
if GameEnded() then
MeditationP hase()
else
inf erenceRulesF ired ← newList()
oldUtility ← GetCurrentUtility()
newUtility ← 0
bestUtility ← −∞
pointer ← 1
inf erenceRulesF ired ← GetFiredInferenceRules()
worldState ← null
counter ← 0
if StuckInT heSameW orldState(GetCurrentW orldState()) then
ImprovistationP hase()
end if
if inf erenceRulesF ired.Count = 0 then
ImprovistationP hase()
end if
for i = 1 to inferenceRulesFired.Count do
if inf erenceRulesF ired[i].Utility > bestUtility then
bestUtility ← inferenceRulesFired[i].Utility
pointer ← i
end if
end for
ExecuteAction(inf erenceRulesF ired[pointer].Action)
while ActionRunning() do
newUtility ← GetCurrentUtility()
if oldUtility > newUtility then
AbortAction()
ImprovistationP hase()
else
oldUtility ← newUtility
end if
end while
worldState ← GetCurrentWorldState()
if worldState 6= inf erenceRulesF ired[pointer].Conclusions then
counter ← inf erenceRulesF ired[pointer].Invalid + 1
inf erenceRulesF ired[pointer].Invalid ← counter
MeditationP hase()
else
counter ← inf erenceRulesF ired[pointer].V alid + 1
inf erenceRulesF ired[pointer].V alid ← counter
P layP hase()
end if
end if
24
game to abort the current action. The IRBREL algorithm only chooses actions that
make the agent get a better world state or at least get into a world state with the
same utility value than the current one. If the utility value gets worse, means that
the circumstances of the game changed and it is better to choose a different strategy.
Consequently, if this happens, then the Play phase is over and the IRBREL algorithm
begins running the Improvisation phase. In the Improvisation phase, the IRBREL
algorithm will chose to execute a different action from his known strategies or will
improvise and try something different.
Each time an actions ends (the interrupted actions are not taken in account), The
IRBREL algorithm compares the current world state with the expected world state
defined by the inference rule. If both world states match, then the inference rule is
valid. If the world states are different, then the inference rule is invalid. Each inference
rule has a counter for the number of times that the inference rule was used and was
valid and number of times that inference rule was invalid. These values are updated
each time an inference rule is used. Besides, these values are used to calculate the real
utility described by equation 3.1.
In some cases, the IRBREL algorithm will discover that some of his actions provoke
things that were not expected. This occurs because some inference rules are too general
and are not considering that a certain condition must be satisfied before using this
strategy. Besides, this also could happen because the chosen action has no relation
with the changes of the world state.
Returning to the previous example of subsection 3.3.1, the IRBREL algorithm
thinks that dancing will provoke rain and the agent needs water. Consequently, the
IRBREL algorithm makes the agent to dance. Nevertheless, this time the IRBREL algorithm observes that is not raining. This kind of inconsistence is noted by the IRBREL
algorithm and updates the number of times that the inference rule was invalid. Invalid
inference rules are not very valuable to the IRBREL algorithm. Consequently, each
time this happens, the IRBREL algorithm executes the Meditation phase to update
the invalid inference rule.
If the IRBREL algorithm gets into a world state where are not any inference rule
that fires or after executing several actions the world state is the same, then IRBREL
algorithm executes the Improvisation phase to try to make get a better world state.
Once the IRBREL achieves to his desired final world state or the game is ended
for any reason, like winning or losing the game, the IRBREL algorithm executes the
Meditation phase to update the inference rules.
3.3.3
Improvisation Phase
The goal of the Improvisation phase is to try something new and learn new strategies. That way the IRBREL algorithm has more strategies to choose when executes
25
the Play phase. The Improvisation phase is executed if an action is interrupted, there
were not any available strategies to execute or the IRBREL algorithm got stuck in the
same world state.
Algorithm 3.4 Algorithm that Calculates the Probability to Improvise.
inf erenceRulesF ired ← GetFiredInferenceRules()
availableActions ←Number of Available Actions (list given by the user)
knownActions ← newList()
probability ← 0
for i = 1 to inferenceRulesFired.Count do
knownActions.AddIF NewElement(inf erenceRulesF ired.Action)
end for
× 100
probability ← availableActions−knownActions.Count
availableActions
RETURN probability
In either case, as shown in algorithm 3.5, the IRBREL algorithm finds the better
way to improvise. First, the IRBREL algorithm calculates the probability to completely
improvise (see algorithm 3.4). The IRBREL algorithm checks which inference rules
fires with the current world state. Then checks which action is available. The IRBREL
algorithm knows how many actions are available to its use, that information is given
by the user when initialized the IRBREL algorithm (see section 3.1 for more details).
If all actions are available, then there is no need to improvise because the IRBREL
algorithm knows the consequences of every action for the current circumstances. If the
IRBREL algorithm only knows the consequences of the half of the total actions, then
there is a 50% of chances to improvise.
Once the IRBREL algorithm knows the probability to improvise, the IRBREL
algorithm analyzes the inference rules that fired. One element of the knowledge base is
the Know All Consequences flag or KAC flag. As described in section 3.2, this element
(linked to each inference rule) represents that the linked inference rule was used in a
situation in which the consequences of all possible actions are known.
If the probability to improvise is zero, then the KAC flag is turned on for every
inference rule that fired. However, if the probability to improvise is greater than zero
but one of the inference rules that fired has the KAC flag turned on, then the IRBREL
algorithm does not improvise and uses one of these inference rules.
If the probability to improvise is greater than zero and all the inference rules that
fired have not the KAC flag on, then the IRBREL algorithm makes a random check to
decide if improvises or if uses one of the known strategies.
If the IRBREL algorithm decides to improvise, then randomly chooses to execute
one of the actions which consequences are not known for the current world state. If the
executed action changed the world state, then the IRBREL algorithm creates a naive
inference rule like the ones created by the Prior-Knowledge phase. After execute the
26
Algorithm 3.5 Improvisation Phase Algorithm.
inf erenceRulesF ired ← GetFiredInferenceRules()
improviseP robability ← GetImproviseProbability()
randomNumber ← GetRandomNumber(1, 101)
randomAction ← 1
availableActions ←Number of Available Actions (list given by the user)
knownActions ← newList()
notKnownActions ← newList()
oldW orldState ← GetCurrentW orldState()
newW orldState ← null
newInf erenceRule ← null
inf erenceRulesF ired ← GetFiredInferenceRules()
for i = 1 to inferenceRulesFired.Count do
knownActions.AddIF NewElement(inf erenceRulesF ired.Action)
end for
if randomNumber = 101 then
randomAction ← GetRandomNumber(1, availableActions)
ExecuteAction(randomAction)
W aitUntilActionEnds()
newW orldState ← GetCurrentW orldState()
if WorldStateChanged(oldW orldState, newW orldState) then
newInf erenceRule ← new InferenceRule(randomAction, newW orldState)
Inf erenceRule.AddIF NewElement(newInf erenceRule)
end if
P layP hase()
else if randomNumber > improviseP robability then
notKnownActions ← GetMissingActions(availableActions, knownActions)
randomAction ← GetRandomNumber(1, notKnownActions.Count)
randomAction ← notKnownActions[randomAction]
ExecuteAction(randomAction)
W aitUntilActionEnds()
newW orldState ← GetCurrentW orldState()
if WorldStateChanged(oldW orldState, newW orldState) then
newInf erenceRule ← new InferenceRule(randomAction, newW orldState)
Inf erenceRule.AddIF NewElement(newInf erenceRule)
end if
P layP hase()
else
P layP hase()
end if
27
improvised action and checks the consequences of that action, the IRBREL algorithm
begins to execute again the Play phase.
If the IRBREL algorithm decides to just use the learned strategies, then chooses
a strategy with the best real utility value using the same process described in the Play
phase. After execute the chosen action and checks the consequences of that action, the
IRBREL algorithm begins to execute again the Play phase.
Even if the probability to improvise is zero, there is always a very small chance of
1% to improvise.
3.3.4
Meditation Phase
The goal of the Meditation phase is to update the inference rules. The invalid
inference rules are destroyed and are created more accurate versions of the original
ones. Besides, the utility values of the inference rules could be modified in order to
improve the performance of the IRBREL algorithm.
Every time a human being sleeps occurs an internal process in the brain. The
short time memory gets converted into long time memory. Some irrelevant learned
knowledge is forgotten. Meanwhile, all relevant learned knowledge is updated and made
efficient for future uses. The Meditation phase was inspired in that mental process. The
Meditation phase is executed if an inference rule was discovered to be invalid or if the
game is over for some reason, like a game lost or won.
As shown by algorithm 3.6, the IRBREL algorithm search for all inference rules
that are invalid. In the previous phases, the IRBREL algorithm took note of every
inference rule that did not behave as it predicted. This could happen for two reasons,
the inference rule is too general or the executed action is not the consequence of the
changes in the world state.
The IRBREL algorithm assumes that every invalid inference rule was invalid because they are too general. If an inference rule is too general, then there are other
involved variables not taken in account. For example, the IRBREL algorithm knows
that the agent get wet if the agent is in the wild and begins to rain. Consequently,
the next time the agent is in the wild and begins to rain, the IRBREL algorithm will
assume that the agent will get wet. That knowledge is true if the agent is not under
cover of something like an umbrella. If the IRBREL algorithm discovers that this time
the agent is not getting wet, then the IRBREL algorithm will figure out that getting
wet depends also to at least one additional element, like the umbrella element.
The IRBREL algorithm creates new versions of invalid inference rules only if there
are not more general versions of that inference rule or if the inference rule has not any
sister inference rule. If there are more general versions of the invalid inference rule in
the knowledge base, then the invalid inference is just deleted.
A new version of an inference rule is the same inference rule but with an additional
28
Algorithm 3.6 Meditation Phase Algorithm.
bestW orldState ← null
newInf erenceRules ← newList()
bestUtility ← −∞
for i = 1 to InferenceRule.Count do
if Inf erenceRule[i].Invalid > 0 then
newInf erenceRules ← GetUpdatedInf erenceRules(Inf erenceRule[i])
Inf erenceRule.Remove[i]
i← i−1
for j = 1 to newInferenceRules.Count do
Inf erenceRule.Add(newInf erenceRules[j])
end for
end if
end for
if GameEnded() ∧ enabledUpdate then
for i = 1 to WorldStates.Count do
if W orldStates[i].Utility > bestUtility then
bestUtility ← W orldStates[i].Utility
bestW orldState ← W orldStates[i]
end if
end for
UpdateInf erenceRuleUtilities(bestW orldState)
SaveAndClose()
else
P layP hase()
end if
29
variable in the premise. The additional variable and its value is obtained from the
Original World State element, which was described in section 3.2, that is linked to each
inference rule. The Original World State element or OWS element has a picture of how
the world state was before the action was executed.
Each invalid inference rule creates several new inference rules. The number of
inference rules generated depends, as shown by equation 3.2, on the number of variables
that the original inference rule has in its premises and the number of variables not taken
in account. Consequently, if the original inference rule has one variable in its premises
and the world state has five variables, then are created four new inference rules with
two variables in its premises.
NewInf erenceRules = T otalW orldV ariables−V ariablesInOriginalP remise (3.2)
The IRBREL algorithm has no way to know which variable should take in account.
That is why the IRBREL algorithm creates several inference rules each one considering
a different variable. These new inference rules are considered sisters. So, if one of
these sisters inference rules gets invalid, there will not be generated new versions of
itself until all its sisters are deleted. An invalid inference rule with sisters does not
generate new inference rules; this invalid inference rule is just deleted. In this way the
IRBREL algorithm eventually discards inference rules with premise variables which are
not important.
A totally specific inference rule is an inference rule which in its premises has all
the variables of the world state. If a totally specific inference rule gets invalid, then it
means that the executed action did not cause the changes in the world state, like in
the example of the agent that dances to make rain. Totally specific inference rules that
gets invalid are just deleted from the knowledge base.
The human brain has the capability to specify or generalize the conditions in which
the learned strategies are valid. Notwithstanding, the IRBREL algorithm only has the
capability to specify the conditions in which the learned strategies are valid. The only
way to the IRBREL algorithm to learn generalized versions of the learned strategies is
throw the 1% improvisation factor which was described in subsection 3.3.3.
Once every invalid inference rule is deleted and/or generates new versions of itself,
then the IRBREL algorithm checks if the game is still running. If the game is still
running, then it is executed again the Play phase. Nonetheless, if the game is ended for
any reason (like a game won or lost), then the IRBREL algorithm updates the utility of
every inference. The IRBREL algorithm uses that period of game inactivity to update
the learned knowledge. In the same way, the human brain converts short time memory
in long time memory while the person is sleeping.
The process to update the utility values of the inference rules is not strictly necessary to execute. The user has the option to enable or disable this option when the
30
IRBREL algorithm is initialized. The process to update the utility values of the inference rules is useful if the utility values assigned to the world states do not exactly
reflect the real utility of the agent to be in that world state. The chapter 4 has an
example of a case where the use of this process is necessary. Nevertheless, if the utility
values of the world states are well managed, then the user could disable this step.
Figure 3.3: Assignation of the utility to the inference rules.
In this step, the IRBREL algorithm finds the world state with the best utility
value. Using that world state, the IRBREL algorithm assigns that utility value to all
inference rules that are supposed to move the agent to that world state. After that,
the IRBREL algorithm finds the neighbor world states of the world state with the best
utility value.
The neighbor states are found using the inference rules. The conclusion section of
an inference rule represents the world state where the agent will be after executing the
specified action. The OWS element linked to each inference rule represents the original
world state before the action was executed. Consequently, each inference rule represents
how the world states are interconnected among them. Using that information could be
31
created a map with all world states connections.
The utility value of the best world state is decremented by a delta value. This new
utility value is assigned to all inference rules that connect the neighbor world states
with the world state with the best utility value.
The process can be exemplified by the figure 3.3. In that figure, the world states
are represented by boxes and the inference rules by red arrows. The inference rules
connect the world states. The inference rules represents how can be changed from one
world state to another using a particular action.
The process of finding neighbor world states and propagate the last assigned utility
value minus a delta continues until all inference rules have been revised.
3.4
3.4 Summary
The Inference Rule-Based Reinforcement Learning algorithm or “IRBREL” algorithm uses the same commonly used principles of reinforcement learning. However,
instead of the common structure used in Reinforcement Learning algorithms, the IRBREL algorithm uses inference rules to represent strategies and to model the world.
The IRBREL algorithm is designed as a module, which can be adapted to be used
in Real Time Strategy Video Games. Notwithstanding, in order to work properly, there
are five special requirements that must be satisfied.
World State Generator Module: A module that generates the world states of the
video game.
Performance Analyzer Module: Module that calculates the utility of the agent to
be in the current world state.
Deterministic Game: The changes in the world state must be only determined by
the actions executed by agents.
Available Actions: The IRBREL algorithm needs to know what actions are available
to its use.
Feedback for Complex Actions: The agent must inform to the IRBREL algorithm,
using an internal flag, if there is a complex action still in execution
In order to represent the learned knowledge, the IRBREL algorithm uses a knowledge base composed by four elements:
World States: Each world state represents a picture of how is the world in that
instant. Furthermore, each world state has an utility value, which is the utility
for the agent to be in that world state.
32
Inference Rules: Represent actions or strategies which can be done under some conditions. Besides, represents how the world is modified by executing those actions
and how useful are those world modifications.
Original World State: This is a world state that is linked to a specific inference rule.
This world state represents the original world state before the action described
in the inference rule is executed.
Know All Consequences Flag: This is a Boolean flag that is linked to a specific
inference rule. This flag is turned on if there are known the consequences of
executing every possible action under the same conditions of the linked inference
rule.
The IRBREL algorithm learns in cycles of four phases. Each phase is executed as
needed. The Prior-Knowledge is used if there is no previous knowledge of the game. In
this phase the IRBREL algorithm learns using a trial and error strategy. The Play phase
uses all the learned knowledge to find the best strategy to win the game. If at some
point the IRBREL algorithm finds something unexpected, then, in the Improvisation
phase, the IRBREL algorithm looks for the best way to improvise and try something
new. Finally, in the Meditation phase, the IRBREL algorithm updates the learned
knowledge to make it accurate.
33
Chapter 4
The IRBREL Algorithm in Operation
In order to clarify how the IRBREL algorithm works, will be used the MissionaryCannibal game. This game is very simple and is not a Real Time Strategy Game, it
could be solved using a whole world state search. Nonetheless, its simplicity will be
used to show how the IRBREL algorithm could work using any other kind of game.
In the following sections is described the Missionary-Cannibal Game and how the
world states and the Performance Analyzer module are handled. Besides, is described
how the IRBREL algorithm uses the Prior-Knowledge, Play, Improvisation and Meditation phases to learn how to solve the game.
4.1
Missionary-Cannibal Game
The game consists in a group of three missionaries and three cannibals at the left
side of a river and their goal is to cross to the right side of the river. In order to cross
the river, they will need to use a boat. The problem is that the boat has only room
enough for two passengers. Besides, cannibals are very aggressive. If cannibals are not
under missionary care, they will eat any meat they could find, even missionaries. To
solve that problem, the number of cannibals (in both sides of the river) never must be
bigger than the number of missionaries.
The Missionary-Cannibal game is a single agent game. The only agent in the
game is the human player who controls how to move the missionaries, the cannibals
and the boat. The game is completely observable for the human player. Besides, this
is deterministic game. The changes in the world are only caused by the human player
actions.
The Missionary-Cannibal game begins as shown by figure 4.1. From that point,
the human player has to move missionaries, cannibals and the boat in certain order to
cross them all to the right side of the river.
If the number of cannibals is greater than the missionaries in either side of the
river, then human player loses the game. The only way to win the game is to move all
missionaries and cannibals to right side of the river without letting that the number of
34
Figure 4.1: Missionary-Cannibal beginning game.
cannibals exceeds the missionaries in either side of the river.
4.2
Satisfying the Special Requirements of the IRBREL Algorithm
As described by chapter 3, the IRBREL algorithms needs that the game be deterministic and that the world be observable enough to let the agent learns why the world
changed after executing an action. These prerequisites are already satisfied by the
Missionary-Cannibal game. Nevertheless, the IRBREL algorithm also needs a module
that generates the world states, a module that evaluates the performance of the agent
and a list of the available actions of the agent.
In game terms there are six possible actions that human player could execute. The
IRBREL algorithm will have access to same actions available to human players. The
table 4.1 shows the available actions and the ID that IRBREL algorithm will use to
tell the game which action execute. All these actions are simple actions. Consequently,
there is no need to notify the IRBREL algorithm about complex actions.
ID
1
2
3
4
5
6
Action
Get a missionary inside the boat.
Get a cannibal inside the boat.
Get a missionary outside the boat.
Get a cannibal outside the boat.
Move the boat to the right side of the river.
Move the boat to the left side of the river.
Table 4.1: List of Missionary-Cannibal Actions.
The world states are represented using seven integer variables. As shown by table 4.2, each variable has an ID or variable name that is used by the IRBREL algorithm
35
to handle the inference rules and world states. The world state module creates an array of these seven variables and their values. These arrays are used by the IRBREL
algorithm as world states.
ID
1
2
3
4
5
6
7
Variable
Number of Missionaries at the left side of the river.
Number of Cannibals at the left side of the river.
Number of Missionaries at the right side of the river.
Number of Cannibals at the right side of the river.
Number of Missionaries inside the boat.
Number of Cannibals inside the boat.
Position of the boat (left or right).
Table 4.2: List of Missionary-Cannibal Variables.
In order to let the IRBREL algorithm knows the utility of the actions and the
utility to be in a certain world state, the game will have a score. This score is used as
the Performance Analyzer module required by the IRBREL algorithm.
At the beginning of a game the score will be zero. Some actions will increase the
score value and others will decrease it. The score is increased by five points for each
missionary or cannibal inside the boat. The score is increased by ten points for each
missionary or cannibal at the right side of the river. If the game is won, the score will
be positive infinite. However, if the game is lost, the score will be negative infinite.
4.3
Prior-Knowledge Phase in Operation
The IRBREL algorithm begins knowing only the available actions and the initial
world state. The IRBREL algorithm does not know the consequences of those actions
or how win or lose the game. Besides, at the beginning, the knowledge base of the
IRBREL algorithm is empty. If the knowledge base is empty and only if is empty, the
Prior-Knowledge phase begins.
At this point, the IRBREL algorithm has no idea of what will happen if he executes
certain action. Consequently, the IRBREL algorithm does the only possible solution:
experiment with the actions and world states.
The IRBREL algorithm executes every action one time and checks what happens
to the world states and restarts the game. This process is similar to any person which
is with a video game that he never has seen before. The person begins pressing all
buttons to figure out what happened and how the video game works.
As shown by the figure 4.2, at the beginning of the game there are only two
available actions: get a missionary inside the boat or get a cannibal inside the boat.
The other actions are invalid and the game will ignore them.
36
Figure 4.2: First possible actions.
Figure 4.3: The first inference rules learned.
At the end of the Prior-Knowledge phase the agent will have three world states
and two naive inference rules saved in his knowledge base. The saved world states
are: the beginning game world state and its two neighbor world states. The two new
inference rules, saved in the knowledge base, are shown in the figure 4.3.
The “A[number]” means which action is executed, in the first case “A[1]” means
the action: get a missionary inside the boat. The “Vnumber[number]” means the
variable. The first number is the variable ID. The number inside the squared parenthesis
is the value of the variable. Each variable could contain any integer value. For future
information about the list of variables and actions see table 4.1 and table 4.2. In
the conclusions section, the values inside the brackets are: the utility to execute that
strategy, the number of times the inference rule was valid and the number of times the
inference rule was invalid.
The generated inference rules are very simple and naive. For the premises, only
was taken in account the executed action. These naive inference rules assume that the
consequences only depend on the chosen actions. Notwithstanding, in many cases the
results depend on more variables. This problem could give inconsistencies problems
which will be resolved latter by the Meditation phase.
The Prior-Knowledge phase ends once all actions are tried and the IRBREL algorithm learned the naive inference rules. At the end of this phase the game is restarted
and the Play phase begins.
37
4.4
Play Phase in Operation
At this point the Prior-Knowledge phase is over and begins the first cycle of the
Play phase. Besides, there are three world states saved in the knowledge base and there
are two inference rules saved (see figure 4.3).
The goal of the Play phase is to maximize the utility. Right now the IRBREL
algorithm only knows three world states, one with utility of zero, another with utility
negative infinite and the last one with utility of five. Consequently, the current goal of
the IRBREL algorithm is to achieve world state with the utility of five. To accomplish
that goal, the IRBREL algorithm will use his learned inference rules.
A new game begins and the world state is the shown by figure 4.1. The IRBREL
algorithm checks which strategies are available to its use. Each inference rule represents
a strategy. If an inference rule fires, means that strategy is available to execute under
the current circumstances.
The premises of the inference rules represent the conditions under which the strategy is available. An inference rule fires if all the variables values in its premises are
match with the variables values of the current world state. World state variables that
are not in the premise section of an inference rule are not taken in account to determine
if that inference rule fires. The variables of the conclusions section represent, how the
world is affected after the strategy is executed.
If an inference rule fires and is executed, could lead to two possible results. First,
the values specified by the conclusions are the same as the new world state. Or second,
the values specified by the conclusions are different than the new world state. In the
first case the inference rule is valid. In the second case the inference rule is invalid.
It is important to take note of every invalid inference rule. An invalid inference
rule means that rule is too general and it needs to take in account other variables.
These inconsistencies will be managed latter by the Meditation phase.
The IRBREL checks which of its two learned inference rules fires. In this case
both inference rules fires because all premises conditions are meet. Then, the IRBREL
algorithm decides which fired inference rule use. There is one inference rule that gives
the IRBREL algorithm an utility of five and another that gives an utility of negative
infinite (game lost). The IRBREL algorithm chooses the inference rule with the highest real utility value. For this case, the IRBREL algorithm chooses the inference rule
number two, which gives it a real utility value of five. If there are two or more inference rules with the same real utility values, then IRBREL algorithm chooses the most
frequently used.
To avoid using inaccurate inference rules and improve the efficiency, the IRBREL
algorithm uses the real utility value of an inference rule instead of its utility value to
choose which inference rule use. The real utility of the inference rule is calculated
using the number of times it was valid or invalid as shown by equation 4.1. In this
38
way old and inaccurate inference rules have less weight than accurate inference rules.
Inaccurate inference rules are managed latter by the Meditation phase.
valid
× utility
(4.1)
valid + invalid
After executing the first action, the IRBREL algorithm checks the results. If the
world state is the same specified by the used inference rule, then this rule was valid
and the counter of number of success is increased by one. Nonetheless, if the world
state is different from the specified by the inference rule, then this rule was invalid and
the counter of number of failures is increased by one. Besides, if an inference rule is
invalid, the IRBREL algorithm executes the Meditation phase (this is done in order to
make the inference rule more accurate).
RealUtility =
Figure 4.4: Consequences of the first action.
At this point, the IRBREL algorithm has chosen and executed the action two (get
a cannibal inside the boat). That action makes the world state be as shows figure 4.4.
Then the IRBREL algorithm checked that the current world state is the same described
by the inference. So, the number of success by the inference rule is increased by one.
Now, the IRBREL algorithm checks which inference rules fires. At this moment,
the two inference rules stored in the knowledge base are too general and both fire again.
However, if there are not any inference rules that fire, then the agent is considered stuck
and the Improvisation phase begins.
For this case, the IRBREL algorithm chooses to execute the same inference rule
of the last time. Notwithstanding, instead of getting to the world state shown by
figure 4.4, the world state is the one shown by figure 4.5.
That inconsistency is noted and is executed the Meditation phase to update the
invalid inference rule.
39
Figure 4.5: Consequences of the second action.
4.5
Meditation Phase in Operation
The Meditation phase is executed if an inference rule was found invalid or the
game itself is over, like in a game lost or won. In this case, this phase is executed
because an inference rule was used and instead of getting in the world state shown by
figure 4.4, the world state is the one shown by figure 4.5.
In the Play phase, the IRBREL algorithm took note of the inference rule that
did not behave as it was predicted. This happened because the inference rule is too
general. Consequently, the IRBREL algorithm will generate new versions of the invalid
inference rule using the OWS element described in chapter 3.
Figure 4.6: Inference rule update.
The IRBREL algorithm examined all inference rules and found only one of them
invalid. As shown in figure 4.6, the IRBREL algorithm uses the linked OWS element
of this inference rule to create more accurate versions of the original one. The new
inference rules have the same elements of the original inference rule plus an additional
variable in the premise. This additional variable and its value is obtained from the
OWS element.
NewInf erenceRules = T otalW orldV ariables−V ariablesInOriginalP remise (4.2)
The number of new inference rules generated, for each invalid inference rule, is
the difference between the total number of variables in the world and the number of
40
variables contained by the premises of the original inference rule. This is shown by
equation 4.2. After every new inference rule is generated, the original inference rule
is considered obsolete and it is deleted from the knowledge base. Figure 4.7 shows
the current inference rules stored in the IRBREL knowledge base at this point. It is
important to notice, that each time an new version of an inference rule is created, the
counter of number of times the inference rule was valid and invalid are restarted.
Figure 4.7: The new inference rules learned.
If the game is over for any reason, like the game is lost, then the IRBREL algorithm
updates the utilities of the inference rules. Nonetheless, the game is not over and the
Play phase is executed again. The game is at the world state shown by figure 4.5.
The Play phase is executed again and the IRBREL algorithm will notice that
inference rules one, two, four, five, six and eight fire (see figure 4.7). Nevertheless, after
choosing and executing an inference rule, the IRBREL algorithm will notice that the
inference rule used was invalid again and the Meditation phase is executed again.
After several cycles of executing the Play and Meditation phases, the IRBREL
algorithm discards inference rules two, four, five, six and eight (see figure 4.7). These
inference rules do not generate newer versions of themselves because there are still sister
inference rules in the knowledge base. The world state remains the same because the
action of getting a cannibal inside the boat is invalid for the current world state and
the game ignores that action.
Now, the only inference rule that fires is the number one. The IRBREL algorithm
will try to use it in Play phase. Then, after find that the inference rule is invalid, the
IRBREL algorithm executes the Meditation phase again. The process described for
the inference rule two of figure 4.3 is repeated for this inference rule. Eventually, the
IRBREL algorithm will have a knowledge base like the one shown by figure 4.8.
After this process, the world state remain the same described by figure 4.5. Finally,
the Play phase is executed again and the IRBREL algorithm did not found any inference
rule that fired. Consequently, is executed the Improvisation phase.
41
Figure 4.8: The updated inference rules.
4.6
Improvisation Phase in Operation
The goal of the Improvisation phase is to try something new and learn new strategies. That way the IRBREL algorithm has more strategies to choose when executes
the Play phase. The Improvisation phase is executed if an action is interrupted, there
were not any available strategies to execute or the IRBREL algorithm got stuck in the
same world state.
For this particular game, there are not complex actions. Consequently, the IRBREL algorithm never has the chance to interrupt a complex action. The only way to
execute the Improvisation phase is that IRBREL algorithm gets stuck in a world state
or if there are not any inference rules in the knowledge base that fire.
At this point the IRBREL algorithm got stuck at the world state shown by figure 4.5. The IRBREL algorithm has no inference rules which tell him how to improve
its utility (figure 4.8). Consequently, if the IRBREL algorithm wants to improve its
utility is needed to try something new.
The improvisation could lead the IRBREL algorithm to four possible scenarios:
1. The new world state has smaller utility than the original one.
2. The new world state has bigger utility than the original one.
3. The new world state has the same utility than the original. However, both world
states are different.
4. After trying something new, the world state is the same as the original one.
The most desirable scenario is one that, after trying something new, the IRBREL
algorithm improved its utility. Notwithstanding, the least desirable scenario is one that,
after trying something new, the IRBREL algorithm is in the same world state. This
last scenario is a problem because the IRBREL algorithm is still considered stuck.
In order to avoid that, the IRBREL algorithm has memory of the world state where
has been stuck. Besides, the IRBREL algorithm remembers how many times arrived to
the same world state. If the number of times the IRBREL algorithm gets stuck in the
same world state gets bigger, the improvisation of the IRBREL algorithm gets bigger
42
too. Nonetheless, the more knows the IRBREL algorithm about the consequences of
the available actions for the current world state, the smaller is the improvisation factor.
Figure 4.9: Second possible actions.
At this point there are two valid actions of the six actions available in the game
(see table 4.1 for future reference). The possible results of those two actions are shown
by figure 4.9. However, the IRBREL algorithm has no idea of which actions are valid
or invalid. Besides, in the current world state there are not inference rules that fire.
Consequently, the only available option is to randomly choose one of the six available
actions. If the IRBREL algorithm tries an invalid action, the game will simply ignore
that action.
After the random action is chosen and executed, the IRBREL algorithm checks if
the current state world is the same than the original one or if it is different. If the world
state is different from the original one, the IRBREL algorithm is no longer considered
stuck and the Play phase is executed again. If the world state is the same than the
original one, then it means that the chosen random action had no effect and the agent
needs to choose a new random action.
Suppose that the IRBREL algorithm chooses to move the boat to the left side of
the river as a random action. Then, IRBREL algorithm checks the world state and
noticed that is the same world state. In game terms, the boat was already at the left
side of the river so there was no change in the world state. In response of that, the
IRBREL algorithm now tries a new action. The IRBREL algorithm chooses the action:
move the boat to the right side of the river. The IRBREL algorithm checks again the
world state and noticed that is a different world state so it is no longer considered
stuck.
Every time the IRBREL algorithm gets out of stuck, checks its knowledge base.
If the world state is not already in his knowledge base, the IRBREL algorithm adds
it. Besides, The IRBREL algorithm generates a naive inference rule that represents
43
the chosen action. This inference rule is very general, like the ones generated in the
Prior-Knowledge phase.
Figure 4.10: Consequences of the improvisation.
At this point, the IRBREL algorithm is no longer considered stuck and the current
world state is shown by figure 4.10. This world state is not already in the IRBREL
algorithm’s knowledge base. Consequently, this world state is added to the knowledge
base. Besides, there is also generated a new naive inference rule that describes these
changes in the world state. Figure 4.11 shows the inference rules stored in the knowledge
base.
Figure 4.11: Learned inferences rules after improvisation phase.
4.7
The Learning Cycle Continues
At this point, the IRBREL algorithm is in the world state shown by figure 4.10.
Besides, the IRBREL algorithm is no longer considered stuck and the Play phase is
executed again.
Once the Play phase is executed, the IRBREL algorithm executes the only inference rule that fires, which is the inference rule number five (see figure 4.11). The
execution of this strategy will not make inference rule invalid but will get stuck the
IRBREL algorithm in the same world state. Consequently, the IRBREL algorithm
executes the Improvisation phase again.
44
The IRBREL algorithm keeps executing the Improvisation phase until gets out
of the stuck. Once out of stuck, the IRBREL algorithm executes the Play phase.
Sometimes, the used inference rules will be invalid. Consequently, the Meditation
phase is executed to update those inference rules. These cycles continues until the
game is stopped or until the IRBREL algorithm wins or loses the game.
Once a game is over, the Meditation phase is executed again. Then, as described
by section 4.5, the IRBREL algorithm checks for invalid inference rules and generates
new versions of them. Besides, once a game is over, the Meditation phase executes an
additional step. In this step the IRBREL algorithm updates the utility of all inference rules. This additional step is optional and is done in order to help the IRBREL
algorithm to choose the best strategy to execute.
Figure 4.12: World states with the same utility.
In the section 4.2 was described how the Performance Analyzer module evaluated
the utility of the current world state. The use of that particular evaluation system
creates some world states, one better than the other one, with the same utility values.
As shown by figure 4.12, there are some world states with the same utility value.
Nevertheless, in the world state of the bottom is closer to win the game than the one
of the top. This issue could lead the IRBREL algorithm to get stuck in the game.
That kind of issues should be avoided by the developer of the Performance Analyzer module. In this case, the Performance Analyzer module was created in this way
for IRBREL algorithm explanation purposes. From the initial world state to the winner
world state, the utility should have a gradual increment. World states that get away
from the winner world state should have a gradual utility decrement. If each world
state is evaluated properly, then there is no need of this additional step. However,
45
executing this additional step gives the IRBREL algorithm some degree of tolerance to
this issue.
To solve that issue, the IRBREL algorithm will try to internally give the world
state of the bottom an utility of ten (see figure 4.12). Besides, the IRBREL algorithm
will try to internally give the world state of the top an utility of ten minus a delta
value. This is done in order to make the IRBREL algorithm prefer one world state over
the other world states with the same utility value.
Using the inference rules, world states and OWS elements stored in the knowledge
base, the IRBREL algorithm creates a hierarchical tree structure of how the world
states are interconnected among them. The world states stored in the knowledge base
are used as nodes of the tree. The inference rules and OWS elements stored in the
knowledge base are used to know how the nodes are interconnected among them.
Inference rules are composed by three elements: premises, conclusions and utility.
The premises describe how the world state was before the action was executed. The
conclusions describe how the world state will be after the action is executed. Most
inference rules are too general and the premises could not completely define the original
world state. Notwithstanding, the linked OWS element, that each inference rule has,
defines the original world state. Using that information, it is possible to know how the
world states are interconnected among them.
After the hierarchical tree structure is generated, the IRBREL algorithm looks
for the node that represents the world state with the highest utility value. If there are
more than one world states with the same utility, then it is randomly chosen one world
state.
The chosen world state (node) is assigned the same utility described by its counterpart world state. Then all the neighbor world states (nodes) receive that utility
value minus a delta value. The same process is repeated with the next neighbors. This
process is repeated until all nodes have an utility value. Figure 4.13 shows an example
of how this process works.
Now that the hierarchical tree with utilities is generated, it is needed to update
the utilities of all inference rules to make them match with the ones of the nodes. The
conclusion section of each inference rule represents how the world state is after executing
the action. Consequently, using the world state represented by the conclusions of each
inference rule, it is assigned the utility value of its counterpart node represented by the
hierarchical tree. Finally, once the utility value of all inference rules are updated, the
hierarchical tree is destroyed.
Once this step ends, the IRBREL algorithm is ready to play a new game. The
phases described in sections 4.3, 4.4, 4.5, 4.6 and 4.7 are repeated as needed. Eventually, after several learning cycles, the IRBREL algorithm will learn enough strategies
(inference rules) to win this game.
46
Figure 4.13: Assignation of the utility to the inference rules.
47
4.8
Summary
The Missionary-Cannibal game consists in a group of three missionaries and three
cannibals at the left side of a river and their goal is to cross to the right side of the
river. In order to cross the river, they will need to use a boat. The problem is that the
boat has only room enough for two passengers. Besides, cannibals are very aggressive.
If cannibals are not under missionary care, they will eat any meat they could find,
even missionaries. To solve that problem, the number of cannibals (in both sides of the
river) never must be bigger than the number of missionaries.
The Missionary-Cannibal game is a single agent and completely observable game.
Besides, this is deterministic game. This game is used to explain how the IRBREL
algorithm learns and how the learning cycle works.
To work properly, the IRBREL algorithm needs a list of the available actions of
the agent, a module that generates the world states and module that evaluates the
performance of the agent.
The IRBREL algorithm has access to the following actions:
• Get a missionary inside the boat.
• Get a cannibal inside the boat.
• Get a missionary outside the boat.
• Get a cannibal outside the boat.
• Move the boat to the right side of the river.
• Move the boat to the left side of the river.
The world states are composed by the following variables:
• Number of Missionaries at the left side of the river.
• Number of Cannibals at the left side of the river.
• Number of Missionaries at the right side of the river.
• Number of Cannibals at the right side of the river.
• Number of Missionaries inside the boat.
• Number of Cannibals inside the boat.
• Position of the boat (left or right).
48
The Performance Analyzer module calculates the utility of each world state using
the number of missionaries and cannibals at each side in the river and in the boat. The
utility of the initial world state is zero. The utility is increased by five points for each
missionary or cannibal inside the boat. The utility is increased by ten points for each
missionary or cannibal at the right side of the river. If the game is won, the utility will
be positive infinite. Nonetheless, if the game is lost, the utility will be negative infinite.
The IRBREL algorithm began playing the game with no previous knowledge of
the Missionary-Cannibal game. Consequently, The IRBREL algorithm executed the
Prior-Knowledge phase to learn some basic knowledge that helps it to begin playing
the game.
Once the IRBREL algorithm learned some basic knowledge, the Play phase was
executed. After some choices, the IRBREL algorithm found the learned strategies too
general and executed the Meditation phase to improve those strategies.
After several cycles of executing the Play and Meditation phases, the IRBREL
algorithm refined the learned strategies (inference rules). Nevertheless, at some point,
the IRBREL algorithm did not found any strategy that helps it to continue making
progress in the game. Consequently, the IRBREL algorithm executed the Improvisation
phase to try something new.
The IRBREL algorithm continued executing the Play, Meditation and Improvisation phases. Eventually, the IRBREL algorithm arrived to a final world state (game
lost or game won). Then, the IRBREL algorithm executed the Meditation phase to
update invalid inference rules. Finally, the IRBREL algorithm updated the utility values of the inference rules. This additional step is done in order to help the IRBREL
algorithm to choose the best strategy to execute. This last step solves the issue of
having two world states, one better than the other, with the same utility values.
With the exception of the Prior knowledge phase, this learning cycle process is
repeated as needed in each game. Eventually, the IRBREL algorithm learns enough
strategies to win the game.
49
Chapter 5
Using IRBREL Algorithm in a Real Time Strategy
Game
In the following sections are described the experiments made using the IRBREL
algorithm in a Real Time Strategy Video Game. Besides, there are described special
considerations that should be taken in account when the IRBREL algorithm is used
in a Real Time Strategy Game. The IRBREL algorithm was set to compete against
different AI agents and was registered the learning progression and the game results.
The video game used for these experiments is named Flag’s Capture. This video game
is a 2-D version game of the real Gotcha with some additional changes.
5.1
Flag’s Capture Game
The Flag’s Capture game is a 2-D video game version of the real Gotcha game.
The game is completely observable for the agents. The game itself is deterministic;
every change in the world is only the consequence of the agents’ actions. There are
only two agents interacting in the video game. The video game is a real time strategy
game. This means that the agents must take decisions quickly. The game does not
wait agents to take decisions; the time keeps running and the world keeps moving like
in real life. If an agent takes too long thinking what to do next, the other agent could
take advantage of this moment of inactivity.
The video game consists in two rival agents (soldiers), a battlefield and two flags.
The goal of each agent is to capture the enemy flag which is located in the opposite
side of the battlefield. Each agent is equipped with a paint-ball pistol with unlimited
ammunition. The paint-ball pistols are used to attack and defend the owner against
the rival agent. If an agent is hit by a paint-bullet, then the agent is considered dead
and leaves the battlefield. The paint-bullets move three times faster than soldiers.
Figure 5.1 shows how the video game looks in action. The screen views the world as if
the observer is positioned in the sky and looks to the ground. Each agent has a red or
blue uniform and all his belongings (flag and bullets) are made of the same color. The
flag is represented by the colored red or blue circle.
50
Figure 5.1: Flag’s Capture video game in action.
The video game was designed with five different agents or difficulties.
Brainless: This is the easiest difficulty of the game. As the name says, the agent itself
does not have any behavior or strategy. The agents controlled by this kind of AI
just stay without executing any action. Consequently, it is like the rival agent is
playing alone.
Dummy: The agents controlled by this AI always tries to capture the enemy flag
without taking care anything else. Since the beginning of the game, these AI
makes the agent moves directly to the enemy flag and capture it. This agent does
not try to attack the rival or to defend himself against incoming paint-bullets.
Kamikaze: This AI is very similar to the Dummy agent. The difference is that the
Kamikaze AI makes the agent shoots paint-bullets to everything that crosses his
path to the enemy flag. Like the Dummy agent, this AI does not try to evade
incoming paint-bullets.
Commando: This is the best static AI of the game. The AI makes the agent to
aim and shoot paint-bullets against the rival agent. If there is a paint-bullet in
collision route to him, the AI makes the agent evade that paint-bullet. Once the
rival agent is considered dead, the AI makes the agent to go and capture the
enemy flag.
51
IRBREL: The AI is dynamic. This AI uses de IRBREL algorithm to decide what
to do. As described by chapters 3 and 4, the AI begins knowing nothing. After
several learning cycles, the AI knows the consequences of his actions in many
scenarios and behaves according to get the best known performance.
For the human players, there are five available actions.
Move Forward: Moves the soldier one step to the front.
Move Backward: Moves the soldier one step back.
Turn Left: Rotates the soldier a minus one degree.
Turn Right: Rotates the soldier one positive degree.
Fire: Shoots one paint-bullet from his paint-gun.
At the beginning of each game, the red flag is positioned at the center of the left
side of the battlefield. On the other side, the blue flag is positioned at the center of the
right side of the battlefield. Each soldier is positioned one step ahead of his color flag.
Both soldiers begin targeting their respective rival. Figure 5.2 Shows how the game
looks at the beginning of each encounter.
Figure 5.2: Flag’s Capture video game - start position.
52
5.2
The Performance Analyzer
According to the specified requirements of the IRBREL algorithm, the IRBREL
algorithm needs a module which serves as performance analyzer of its actions. In other
words, the IRBREL algorithm needs a module that tells him the utility of the current
world state. This utility is used by IRBREL algorithm to know the consequences of his
actions and to choose the best available known strategy.
The objective of the game is to capture the enemy flag. Consequently, the Performance Analyzer module will award with an infinite utility to all world states where the
soldier wins the game. Besides, will award with a negative infinite utility to all world
states where the soldier loses or is considered dead. For games in which the result is a
draw, the soldier is awarded with five thousand utility points.
The closer is the soldier to the enemy flag, the closer he is to the goal of the game.
So, the Performance Analyzer module awards the soldier with one utility point for each
step closer to the enemy flag. However, the closer his rival is form his flag, the closer
the soldier is to lose the game. So, the Performance Analyzer module penalizes the
soldier with one utility point for each step closer the rival is from his flag.
Having a paint-bullet in collision route to the soldier is a bad thing. Notwithstanding, having a paint-bullet in collision route to the enemy is a good thing. Consequently,
the Performance Analyzer module awards with one thousand utility points if there is
a paint-bullet in collision route to the enemy. Besides, penalizes with one thousand
utility points if there is a paint-bullet in collision route to him.
5.3
Special Considerations
If we give to the IRBREL algorithm the variables values of the world state and the
same simple actions available to human players (move forward, move backward, turn
left, turn right and fire), the IRBREL algorithm will be able to learn the consequences
of his actions and how this world works. Nonetheless, there are some considerations
that should be taken in account.
The IRBREL algorithm is similar to the human learning. If humans do not know
what they are trying to learn, then will be harder and in some cases very improvable to
learn that thing. It is not completely necessary that humans know exactly what they
want to learn. Nevertheless, they need to have an idea of what they want to learn. An
example of these is the little children that learn to crawl. They do not exactly want
to learn to crawl, but they are trying to learn a way to move to the place they want
to be. This will lead the little children to learn how to crawl and then to learn how
to walk, run, etc. These principles apply to the IRBREL algorithm too. The IRBREL
algorithm does not necessary needs to know that is learning to evade paint-bullets. The
IRBREL algorithm could only know that it is just learning how survive.
53
The knowledge of what the IRBREL algorithm is trying to learn is represented by
the utility given from the Performance Analyzer module. Consequently, a completely
learned behavior is rewarded with a positive infinite utility. Besides, a complete failure
in learning the behavior is rewarded with a negative infinite utility. Any other behavior
is rewarded according to of how close is the behavior from the desired one. The IRBREL
algorithm uses reinforcement learning. So, for optimal results, the utilities given by the
Performance Analyzer module should reflect, in the best possible way, what is wanted
to learn. Should be avoided the cases where two world states, one better than the other
one, have the same utility values.
It is important to consider what is the goal of the IRBREL algorithm. Is the goal
to learn complex behaviors like learn to evade a paint-bullet? Is the goal is just to learn
how and when use the known abilities to win the game?
For the first case, if is wanted that the IRBREL algorithm learns complex behaviors like evade a paint-bullet, then the whole Performance Analyzer module described
in the section 5.2 needs to be modified. The Performance Analyzer should reward with
positive infinite all cases where the soldier evades a bullet. Besides, the Performance
Analyzer should penalize actions that do not help to avoid paint-bullets and awards
actions that help to avoid paint-bullets.
For the second case, if is wanted that the IRBREL algorithm learns how and when
use the known abilities to win the game, then should be used a Performance Analyzer
module similar to one described in the previous section.
The meaning or representation of the variables is a factor that could improve or
delay the learning process. The representation of a paint-bullet collision could be done
in several ways. This could be handled by learning under which trajectories of paintbullets could collide with the soldier. That learning should be repeated by each possible
position that the soldier could be in the battlefield. Another way to handle this is to
add an additional variable in the world state. The new variable indicates if the soldier
will be hit by a paint-bullet if he does not make anything to change that situation. In
that way, the IRBREL algorithm could learn valid inference rules faster because the
premises depend in fewer variables. Besides, that makes inference rules more general
and applies to all possible positions in the battlefield instead of learning inference rules
for each particular position.
Humans beings have the amazing capability of generate abstract data, like the
paint-bullet collision detection. Unfortunately, the IRBREL algorithm does not have
that capability yet. Which means that by now all abstract variables should be managed
by the world state generator module.
Another important consideration is the variables used to generate the world states.
If there are used abstract variables, then some other variables could become redundant.
For example, if there is used the abstract variable that represents paint-bullet collisions,
then other variables like position, orientation and trajectory of paint-bullets could be54
came redundant. The new single variable could represent the same information than
the other three variables together. If redundant variables have no other use, then is safe
to ignore these variables for the world state. However, if these variables are used for
other considerations (like for example evade paint-bullets), then these variables should
be included in for the world state.
Ignoring important variables for the world state could make a deterministic game
to become a non-deterministic game to the IRBREL algorithm perspective. The IRBREL algorithm is not capable to function properly in non-deterministic environments.
In the other hand, ignoring not used variables simplifies the world and speeds up the
learning of the IRBREL algorithm.
In some cases, some world states could be redundant. If the goal of the game is
to capture the enemy flag, then it is not so important if the rival soldier is still alive
or if the soldier captures the flag while is looking to the north. For these cases, if it is
not important how the game was won, then some world states could be grouped into
a single one that represents a won state. The same applies to the cases of game lost,
game draw, etc.
Grouping world states into single ones reduces the number of world states and simplifies the work for the IRBREL algorithm, which makes the learning process faster.
Grouping world states could make a deterministic game to become a non-deterministic
game to the IRBREL algorithm perspective. Consequently, is very important to consider how the IRBREL algorithm will perceive the world with these changes.
Learning in video games should be used to improve the agents’ performance.
Learning of how play the video game should only be used in the developing phase
not in the finished version. For finished versions of the video game, the agents should
know some skills and through learning the agents could perfection those skills knowing
how and when use them. For the Flag’s Capture video game, the skills could be basic skills like move forward, turn left, fire, etc. Besides, the skills could include some
complex skills like target enemy, evade paint-bullets, etc.
A simple skill or simple action is an actions that take one game action or unit of
time to be executed, like move one step forward or shoot a paint-bullet. In most cases,
the simple actions are those actions available for human players. A complex skill or
complex action is the execution of simple actions in certain order. For example, evading
a paint-bullet requires several turn lefts or turn rights and several move forwards or
move backwards. Complex actions take more than one game action to be executed.
The IRBREL algorithm is able to manage simple and complex actions. The IRBREL algorithm just needs to know, from his available actions repertory, which actions
are simple and which are complex. Besides, the IRBREL algorithm has the ability to
abort the execution of complex actions if notice that the performance is not going well
or if the conditions of the game changed.
55
5.4
Flag’s Capture Experiments
The learning in video games should be used to improve the agents’ performance.
Learning of how to play the video game should only be used in the developing phase
not in the finished version. For finished versions of the video game, the agents should
know some skills and through learning the agents could perfection those skills knowing
how and when use them. The knowledge base of the IRBREL algorithm should not
be empty in finished versions. Notwithstanding, in order to measure how the IRBREL
algorithm improves its performance, the knowledge base will be empty at the beginning
of each experiment.
The propose of these experiments is not the development of new complex actions.
The goal of these experiments is just to measure how the IRBREL algorithm improves
its performance. Thus, the IRBREL algorithm will know the basic simple actions
available to human players plus some additional complex actions.
For the IRBREL algorithm, there are eight available actions. The first five are
simple actions. The last three are complex actions.
Move Forward: Moves the soldier one step to the front.
Move Backward: Moves the soldier one step back.
Turn Left: Rotates the soldier a minus one degree.
Turn Right: Rotates the soldier one positive degree.
Fire: Shoots one paint-bullet from his paint-gun.
Evade Paint-Bullets: Makes the soldier to evade all paint-bullets in collision trajectory to him.
Target Enemy: The soldier targets his enemy using his paint-gun.
Go to Enemy Flag: Moves the soldier to the position of the enemy flag and captures
it.
The evade paint-bullets action is not perfect. This complex action fails if the
incoming paint-bullets are too close to the soldier to escape. The paint-bullets are
three times faster than soldiers. So, in some cases it is impossible for the soldier to
escape a paint-bullet collision. Besides, this complex action does not work for very
complex bursts of paint-bullets. Nevertheless, every AI, that has the ability of evade
paint-bullets, use the same algorithm. So, any AI that uses the ability of evade paintbullets has the same advantages and disadvantages.
56
To measure the ability of the IRBREL algorithm to learn how and when use its
skills, the IRBREL algorithm was set to play against every AI of the video game. The
behavior of the different kinds of AI is explained in section 5.1.
The experiments are split in four categories. In each category the IRBREL agent
was set to play against a specific AI. For all the games played in the same category is
used the same kind of AI.
Each category is split in fifty game sessions. In each game session the IRBREL
agent was set to play three hundred times against the AI agent specified by the current
category. The IRBREL algorithm’s knowledge base is erased before each game session
begins. Consequently, at the beginning of each game session, the IRBREL agent begins
using the Prior-Knowledge phase. Finally after the Prior-Knowledge phase ends, the
first game begins.
At the end of each game session the video game created a log file with the results.
The results show how the game ended in each of the three hundred played games. There
are three possible endings: The IRBREL agent won the game, lost the game or the
game is a draw.
In the next subsections are shown the results of the experiments for each category.
The first graph shows the results of the ending values of the three hundred played games.
The “X” axis represents the number of the played game. The “Y” axis represents the
ending result of each game. The graph values go from minus one to one. One represents
a game won by the IRBREL agent. Zero represents a game draw. Minus one represents
a game lost by the IRBREL agent. The values of the graph are the average ending
results of the fifty game sessions. The second graph shows the percentage of games
won, lost or draw in the current category.
5.4.1
Brainless Agent
The Brainless agent is the easiest AI to beat of the game. As the name says,
the Brainless agent itself does not have any behavior or strategy. The Brainless agent
just stays without executing any action and represents no harm for the IRBREL agent.
Consequently, it is like the IRBREL agent is playing alone.
In order to win the game, the IRBREL agent just needs to learn that is safe to
go and capture the enemy flag. Since the Brainless agent will not take any action, the
IRBREL agent does not have to learn any special precondition in which use the known
actions.
As shown by figure 5.3, all the games were won by the IRBREL agent. All the
games were won by the IRBREL agent because the Brainless agent did not execute
any action. So, eventually the IRBREL agent could improvise enough to capture the
enemy flag and learn how to win the game.
Figure 5.4 shows the percentage of games won, lost or draw. For this case, all
57
Figure 5.3: Game progress in Brainless AI category.
games were won because the Brainless agent did not execute any action and the IRBREL agent kept improvising.
5.4.2
Dummy Agent
The Dummy agent always tries to capture the enemy flag without taking care of
anything else. At the beginning of each game, the Dummy AI makes the soldier to
move directly to the enemy flag and capture it. The Dummy agent does not try to
attack the rival or to defend himself against incoming paint-bullets.
In order to win the game, the IRBREL agent needs to learn how to capture the
enemy flag before the Dummy agent does. If the IRBREL agent just go and capture
the enemy flag, then all the games will be a draw because both soldiers will capture the
flag at the same time. The only way to win the game is to first kill the enemy soldier
and then capture the flag. Consequently, in order to win the game, the IRBREL agent
needs to learn to do those actions in that order.
As shown by figure 5.5, at the beginning all games were lost by the IRBREL agent.
After a few games, the IRBREL agent found a way to win or draw the game. By the
game number seven, the IRBREL agent found a way to win the game. Nonetheless,
the learned inference rules were too general and the IRBREL agent applied them in
wrong situations. Eventually the inference rules became accurate and the IRBREL
agent began to apply them in the right situations.
The desire of improvisation by the IRBREL agent made that sometimes tried
something new in different situations. The intension with those improvisations was to
58
Figure 5.4: Game score in Brainless AI category.
see if the IRBREL agent got better results than using the already known strategies. Finally, by the game number 150, all desire of improvisation was satisfied by the IRBREL
agent and only used its best known strategies.
Figure 5.6 shows the percentage of games won, lost or draw. For this case, the
IRBREL agent won 98.29% of all games. Only 1.38% of the games were a draw and
0.33% of the games were lost. These results show that the IRBREL agent learned very
fast how to apply the available actions.
5.4.3
Kamikaze Agent
The Kamikaze agent is very similar to the Dummy agent. The difference is that
the Kamikaze AI makes the soldier shoots paint-bullets to everything that crosses his
path to the enemy flag. Like the Dummy agent, the Kamikaze agent does not try to
evade incoming paint-bullets.
In order to win the game, the IRBREL agent needs to learn how to capture the
enemy flag without getting hit by a paint-bullet. Besides, the IRBREL agent needs
to do that before the Kamikaze agent captures the flag. If the IRBREL agent just
evades the paint-bullets and go to capture the enemy flag, then the IRBREL agent
will always lose the game because the Kamikaze agent will arrive sooner to the flag.
If the IRBREL agent just shoots paint-bullets to the Kamikaze agent without evading
bullets, then both will die and the game will be a draw. Consequently, in order to
win the game, the IRBREL agent needs to learn to kill the Kamikaze agent, evade all
paint-bullets and capture the enemy flag.
59
Figure 5.5: Game progress in Dummy AI categoryy.
Figure 5.6: Game score in Dummy AI category.
60
There are several ways to win the game and some actions could be done in different
order. The Kamikaze agent is more sophisticate and takes longer to the IRBREL agent
to find a way to always win the game. Figure 5.7 shows the average learning progress
of the IRBREL agent. By the game number seven, the IRBREL agent found a way
to win the game. Nevertheless, the generalized assumptions made the IRBREL agent
lost or draw the game from game 15 to 40. From the game number 40 to 205 the
IRBREL agent continued doing little improvisations to try to get better results, some
of these improvisations made the performance worse. Finally, by the game number 205
the behavior of the IRBREL agent was stable and always won the game.
Figure 5.7: Game progress in Kamikaze AI category.
Figure 5.8 shows the percentage of games won, lost or draw. For this case, the
IRBREL agent won 95.03% of all games. Only 0.53% of the games were a draw and
4.43% of the games were lost. Contrary to the results obtained in the Dummy agent
category (figure 5.6), there are more cases of game lost than game draw. This can be
explained by the aggressive behavior of the Kamikaze agent and the passive behavior
of the Dummy agent.
5.4.4
Commando Agent
The Commando agent is the best static AI of the game. The Commando AI
makes the soldier to aim and shoot paint-bullets against the rival soldier. If there is a
paint-bullet in collision route to him, the Commando AI makes the soldier evade that
paint-bullet. Once the rival agent is considered dead, the Commando AI makes the
soldier to go and capture the enemy flag. The human players have many troubles to
61
Figure 5.8: Game score in Kamikaze AI category.
just draw a game against this agent.
In order to win the game, the IRBREL agent needs to learn how to capture the
enemy flag after eliminating the rival soldier. The only way to beat the Commando
agent is shooting paint-bullets to the enemy and evading incoming paint-bullets. The
Commando agent will keep shooting paint-bullets and evading the ones that will hit
him. The tricky part is to make the Commando agent to flee to a corner and then kill the
soldier. However, the Commando agent tries to avoid those situations. Consequently,
human players and the IRBREL agent will have a lot of trouble to win a single game.
In order to win the game, the IRBREL agent will need to learn how combine the known
actions in very specific situations.
The Commando agent has a lot of resources and each game change a lot from each
other. This make harder for the IRBREL agent to learn an absolute winner strategy.
Figure 5.9 shows the average learning progress of the IRBREL agent. From game
number 1 to 5, the IRBREL agent learned how to draw the game. Notwithstanding,
from game number 5 to 10, the generalized assumptions made the IRBREL agent to
lose more often. From game 13 to 269, the IRBREL agent began to learn how win
a game more often than draw o lose it. The graph in that section shows how the
values slowly have a tendency towards one, which means, that by each game played,
the IRBREL agent won more often. By game number 270, the IRBREL agent found a
way to almost always win a game. At that point, the rare cases were the IRBREL agent
does not always won the game is because the improvisation desire of the IRBREL agent.
Nonetheless, eventually the behavior was stabilized and the IRBREL agent always won
the game.
62
Figure 5.9: Game progress in Commando AI category.
Figure 5.10 shows the percentage of games won, lost or draw. For this case,
the IRBREL agent won 81.68% of all games. Only 1.03% of the games were a draw
and 17.23% of the games were lost. Like the results obtained with Kamikaze agent
category (figure 5.8), there are more cases of game lost than game draw. Besides it
took longer for the IRBREL agent to absolutely beat the Commando agent than in
any other category. This can be explained by the extremely well performance of the
Commando agent. Nevertheless, the IRBREL agent eventually found a way to always
win the game.
5.4.5
Experiment Conclusions
At this point was been proven that the IRBREL algorithm has the capability
to learn, take quick decisions and improvise, which is its most important capability.
Besides was proven that IRBREL algorithm has the capability to learn and to take quick
decisions in Real Time Strategy Games. Taking in account the special considerations
described in section 5.3, there is possible to use the IRBREL algorithm in other video
games.
The improvisation factor is the only way for the IRBREL algorithm to learn
new strategies. As shown in almost every category, the improvisation could hinder
the performance if the IRBREL algorithm already knows almost every good strategy.
However, for these cases, the IRBREL algorithm does not have a way to know if
the not tested actions will be a better or worse choices until try them in the current
circumstances.
63
Figure 5.10: Game score in Commando AI category.
The generalized inference rules or strategies, in the early learning stage of the
IRBREL algorithm, harm its performance. Every time the IRBREL algorithm found
a better strategy, assumed that only depended on the taken action. So, after its first
won game, the IRBREL algorithm got a worse performance. This can be appreciate in
figures 5.5, 5.7 and 5.9 where there is a drastic performance fall in the early learning
stage. Notwithstanding, the IRBREL algorithm makes the inference rule more specific
quickly and the lost performance is quickly recovered.
The capability of the IRBREL algorithm to improvise, interrupt complex actions
if the performance gets worse and then improvise again is what lets the IRBREL
algorithm to quickly learn useful strategies. Besides, the interruption of complex actions
is what let the IRBREL algorithm to change the used strategy to adapt itself against
the Commando AI behavior.
5.5
Summary
The Flag’s Capture game is a 2-D video game version of the real Gotcha game.
The game is completely observable for the agents. The game itself is deterministic;
every change in the world is only the consequence of the agents’ actions. There are
only two agents interacting in the video game. The video game is a real time strategy
game. This means that the agents must take decisions quickly. The game does not
wait agents to take decisions; the time keeps running and the world keeps moving like
in real life. If an agent takes too long thinking what to do next, the other agent could
take advantage of this moment of inactivity.
64
The video game consists in two rival agents (soldiers), a battlefield and two flags.
The goal of each agent is to capture the enemy flag which is located in the opposite
side of the battlefield. Each agent is equipped with a paint-ball pistol with unlimited
ammunition. The paint-ball pistols are used to attack and defend the owner against
the rival agent. If an agent is hit by a paint-bullet, then the agent is considered dead
and leaves the battlefield.
In order to measure the learning performance of the IRBREL algorithm, it was
set to compete in the Flag’s Capture game against different agents.
Brainless: This is the easiest difficulty of the game. As the name says, the agent itself
does not have any behavior or strategy. The agents controlled by this kind of AI
just stay without executing any action. Consequently, it is like the rival agent is
playing alone.
Dummy: The agents controlled by this AI always tries to capture the enemy flag
without taking care anything else. Since the beginning of the game, these AI
makes the agent moves directly to the enemy flag and capture it. This agent does
not try to attack the rival or to defend himself against incoming paint-bullets.
Kamikaze: This AI is very similar to the Dummy agent. The difference is that the
Kamikaze AI makes the agent shoots paint-bullets to everything that crosses his
path to the enemy flag. Like the Dummy agent, this AI does not try to evade
incoming paint-bullets.
Commando: This is the best static AI of the game. The AI makes the agent to
aim and shoot paint-bullets against the rival agent. If there is a paint-bullet in
collision route to him, the AI makes the agent evade that paint-bullet. Once the
rival agent is considered dead, the AI makes the agent to go and capture the
enemy flag.
For those competitions, the IRBREL agent had access to eight actions. The first
five are simple actions. The last three are complex actions.
Move Forward: Moves the soldier one step to the front.
Move Backward: Moves the soldier one step back.
Turn Left: Rotates the soldier a minus one degree.
Turn Right: Rotates the soldier one positive degree.
Fire: Shoots one paint-bullet from his paint-gun.
Evade Paint-Bullets: Makes the soldier to evade all paint-bullets in collision trajectory to him.
65
Target Enemy: The soldier targets his enemy using his paint-gun.
Go to Enemy Flag: Moves the soldier to the position of the enemy flag and captures
it.
According to the specified requirements of the IRBREL algorithm, the IRBREL
algorithm needs a module which serves as performance analyzer of its actions. The
Performance Analyzer module awards with an infinite utility if the game is won, a
negative infinite utility if the game is lost or five thousand utility points if the game is
a draw. The soldier is awarded with one utility point for each step closer to the enemy
flag and penalized with one utility point for each step closer the rival is from his flag.
The Performance Analyzer module awards the soldier with one thousand utility points
if there is a paint-bullet in collision route to the enemy. Besides, penalizes the soldier
with one thousand utility points if there is a paint-bullet in collision route to him.
There are some special considerations that should be taken in account before
using the IRBREL algorithm. These considerations could help to improve the IRBREL
algorithm performance.
• The utility given by the Performance Analyzer should reflect what the IRBREL
algorithm wants to learn.
• There could be used additional variables that represent abstract knowledge. This
abstract knowledge helps the IRBREL algorithm to have simpler inference rules
and to learn faster.
• Some world states could be grouped in a single world state. This simplifies the
game and helps the IRBREL algorithm to learn faster.
• Complex actions helps the IRBREL algorithm to keep focused in learning how to
apply the learned actions to win the game and avoids the need of learning new
complex behaviors.
The IRBREL algorithm was set to play three hundred games against each agent.
These experiments were repeated fifty times and was registered the average learning
progress. The learning progress was measured using the number of times the game was
lost, draw or won by the IRBREL agent.
The IRBREL agent was able to learn enough strategies (inference rules) to always
win the game. These experiments proves that the IRBREL algorithm has the capability
to learn, take quick decisions and improvise, which is its most important capability.
Besides was proven that IRBREL algorithm has the capability to learn and to take
quick decisions in Real Time Strategy Games.
The improvisation factor is the only way for the IRBREL algorithm to learn new
strategies. The improvisation could hinder the performance if the IRBREL algorithm
66
already knows almost every good strategy. Nonetheless, for these cases, the IRBREL
algorithm does not have a way to know if the not tested actions will be a better or
worse choices until try them in the current circumstances.
The generalized inference rules or strategies, in the early learning stage of the
IRBREL algorithm, harm its performance. Every time the IRBREL algorithm found a
better strategy, assumed that only depended on the taken action. So, after its first won
game, the IRBREL algorithm got a worse performance. Nevertheless, the IRBREL
algorithm makes the inference rule more specific quickly and the lost performance is
quickly recovered.
The capability of the IRBREL algorithm to improvise, interrupt complex actions
if the performance gets worse and then improvise again is what lets the IRBREL
algorithm to quickly learn useful strategies. Besides, the interruption of complex actions
is what let the IRBREL algorithm to change the used strategy to adapt itself against
the Commando agent.
67
Chapter 6
Conclusions
In the following sections are described in summary how the IRBREL algorithm
works and its principal results. Besides, it is described the contributions made by the
IRBREL algorithm and its relevance. Finally, are described several ways in which the
IRBREL algorithm could be modified to be used in a wide range of Real Time Strategy
Games. These additional considerations were not scope of this research but could be
used for future work.
6.1
Summary and Principal Results
The goal of this research is to create a system module for Real Time Strategy
Video Games. The system module learns from its experience and observations using
Reinforcement Learning.
The system module is used as the AI of a Real Time Strategy Video Game. The
idea is that the video game has agents with an AI that improves with the time. The
AI will learn from its chosen actions and behave according to the learned knowledge.
That way the human players could not use the same strategies to beat the game.
Consequently, the game will force the human players to keep improving their skills.
One of the most important capabilities of this system is its ability to improvise
under unknown situations. Besides, it is able to adapt and update its knowledge as
needed. That way, the next time the system module is under the same circumstances,
the system will know what to do or what not to do.
The scope of this research is to create a system module able to learn and improvise
in Real Time Strategy Video Games. The system module is intended to work only in
deterministic and completely observable environments.
The Inference Rule-Based Reinforcement Learning algorithm or “IRBREL” algorithm uses the same commonly used principles of reinforcement learning. However,
instead of the common structure used in Reinforcement Learning algorithms, the IRBREL algorithm uses inference rules to represent strategies and to model the world.
The IRBREL algorithm is designed as a module, which can be adapted to be used
68
in Real Time Strategy Video Games. Notwithstanding, in order to work properly, there
are five special requirements that must be satisfied.
World State Generator Module: A module that generates the world states of the
video game.
Performance Analyzer Module: Module that calculates the utility of the agent to
be in the current world state.
Deterministic Game: The changes in the world state must be only determined by
the actions executed by agents.
Available Actions: The IRBREL algorithm needs to know what actions are available
to its use.
Feedback for Complex Actions: The agent must inform to the IRBREL algorithm,
using an internal flag, if there is a complex action still in execution
In order to represent the learned knowledge, the IRBREL algorithm uses a knowledge base composed by four elements:
World States: Each world state represents a picture of how is the world in that
instant. Furthermore, each world state has an utility value, which is the utility
for the agent to be in that world state.
Inference Rules: Represent actions or strategies which can be done under some conditions. Besides, represents how the world is modified by executing those actions
and how useful are those world modifications.
Original World State: This is a world state that is linked to a specific inference rule.
This world state represents the original world state before the action described
in the inference rule is executed.
Know All Consequences Flag: This is a Boolean flag that is linked to a specific
inference rule. This flag is turned on if there are known the consequences of
executing every possible action under the same conditions of the linked inference
rule.
The IRBREL algorithm learns in cycles of four phases. Each phase is executed as
needed. The Prior-Knowledge is used if there is no previous knowledge of the game. In
this phase the IRBREL algorithm learns using a trial and error strategy. The Play phase
uses all the learned knowledge to find the best strategy to win the game. If at some
point the IRBREL algorithm finds something unexpected, then, in the Improvisation
phase, the IRBREL algorithm looks for the best way to improvise and try something
69
new. Finally, in the Meditation phase, the IRBREL algorithm updates the learned
knowledge to make it accurate.
There are some special considerations that should be taken in account before
using the IRBREL algorithm. These considerations could help to improve the IRBREL
algorithm performance.
• The utility given by the Performance Analyzer should reflect what the IRBREL
algorithm wants to learn.
• There could be used additional variables that represent abstract knowledge. This
abstract knowledge helps the IRBREL algorithm to have simpler inference rules
and to learn faster.
• Some world states could be grouped in a single world state. This simplifies the
game and helps the IRBREL algorithm to learn faster.
• Complex actions helps the IRBREL algorithm to keep focused in learning how to
apply the learned actions to win the game and avoids the need of learning new
complex behaviors.
The performance of the IRBREL algorithm was measured using the Flag’s Capture
game, which is a Real Time Strategy Video Game. The Flag’s Capture game is a 2-D
video game version of the real Gotcha game. The game is completely observable for
the agents. The game itself is deterministic; every change in the world is only the
consequence of the agents’ actions. There are only two agents interacting in the video
game. The video game is a real time strategy game. This means that the agents must
take decisions quickly. The game does not wait agents to take decisions; the time keeps
running and the world keeps moving like in real life. If an agent takes too long thinking
what to do next, the other agent could take advantage of this moment of inactivity.
The video game consists in two rival agents (soldiers), a battlefield and two flags.
The goal of each agent is to capture the enemy flag which is located in the opposite
side of the battlefield. Each agent is equipped with a paint-ball pistol with unlimited
ammunition. The paint-ball pistols are used to attack and defend the owner against
the rival agent. If an agent is hit by a paint-bullet, then the agent is considered dead
and leaves the battlefield.
In order to measure the learning performance of the IRBREL algorithm, it was
set to compete in the Flag’s Capture game against different agents.
Brainless: This is the easiest difficulty of the game. As the name says, the agent itself
does not have any behavior or strategy. The agents controlled by this kind of AI
just stay without executing any action. Consequently, it is like the rival agent is
playing alone.
70
Dummy: The agents controlled by this AI always tries to capture the enemy flag
without taking care anything else. Since the beginning of the game, these AI
makes the agent moves directly to the enemy flag and capture it. This agent does
not try to attack the rival or to defend himself against incoming paint-bullets.
Kamikaze: This AI is very similar to the Dummy agent. The difference is that the
Kamikaze AI makes the agent shoots paint-bullets to everything that crosses his
path to the enemy flag. Like the Dummy agent, this AI does not try to evade
incoming paint-bullets.
Commando: This is the best static AI of the game. The AI makes the agent to
aim and shoot paint-bullets against the rival agent. If there is a paint-bullet in
collision route to him, the AI makes the agent evade that paint-bullet. Once the
rival agent is considered dead, the AI makes the agent to go and capture the
enemy flag.
For those competitions, the IRBREL agent had access to eight actions. The first
five are simple actions. The last three are complex actions.
Move Forward: Moves the soldier one step to the front.
Move Backward: Moves the soldier one step back.
Turn Left: Rotates the soldier a minus one degree.
Turn Right: Rotates the soldier one positive degree.
Fire: Shoots one paint-bullet from his paint-gun.
Evade Paint-Bullets: Makes the soldier to evade all paint-bullets in collision trajectory to him.
Target Enemy: The soldier targets his enemy using his paint-gun.
Go to Enemy Flag: Moves the soldier to the position of the enemy flag and captures
it.
According to the specified requirements of the IRBREL algorithm, the IRBREL
algorithm needs a module which serves as performance analyzer of its actions. The
Performance Analyzer module awards with an infinite utility if the game is won, a
negative infinite utility if the game is lost or five thousand utility points if the game is
a draw. The soldier is awarded with one utility point for each step closer to the enemy
flag and penalized with one utility point for each step closer the rival is from his flag.
The Performance Analyzer module awards the soldier with one thousand utility points
71
if there is a paint-bullet in collision route to the enemy. Besides, penalizes the soldier
with one thousand utility points if there is a paint-bullet in collision route to him.
The IRBREL algorithm was set to play three hundred games against each agent.
These experiments were repeated fifty times and was registered the average learning
progress. The learning progress was measured using the number of times the game was
lost, draw or won by the IRBREL agent.
The IRBREL agent was able to learn enough strategies (inference rules) to always
win the game. These experiments proves that the IRBREL algorithm has the capability
to learn, take quick decisions and improvise, which is its most important capability.
Besides was proven that IRBREL algorithm has the capability to learn and to take
quick decisions in Real Time Strategy Video Games.
The improvisation factor is the only way for the IRBREL algorithm to learn new
strategies. The improvisation could hinder the performance if the IRBREL algorithm
already knows almost every good strategy. Nonetheless, for these cases, the IRBREL
algorithm does not have a way to know if the not tested actions will be a better or
worse choices until try them in the current circumstances.
The generalized inference rules or strategies, in the early learning stage of the
IRBREL algorithm, harm its performance. Every time the IRBREL algorithm found a
better strategy, assumed that only depended on the taken action. So, after its first won
game, the IRBREL algorithm got a worse performance. Nevertheless, the IRBREL
algorithm makes the inference rule more specific quickly and the lost performance is
quickly recovered.
The capability of the IRBREL algorithm to improvise, interrupt complex actions
if the performance gets worse and then improvise again is what lets the IRBREL
algorithm to quickly learn useful strategies. Besides, the interruption of complex actions
is what let the IRBREL algorithm to change the used strategy to adapt itself against
the Commando agent.
6.2
Contributions and Relevance of the IRBREL
Algorithm
The IRBREL algorithm is the first learning system module designed for Real
Time Strategy Video Games that uses Reinforcement Learning and Inference Rules.
The main contribution of the IRBREL algorithm is its capability to improvise under
known circumstances and learn from that improvisation.
The use of inference rules to represent the learned knowledge helps the IRBREL
algorithm to quickly decide which strategy execute. Instead of analyzing the whole
world state space, the IRBREL algorithm just needs to look which inference rule fires.
Then, from the inference rule that fire, executes the inference rule that gives the agent
72
the highest utility value. This process saves a lot of time because instead of analyzing
multiple variables and world states, the IRBREL algorithm only needs to analyze two
values: The Boolean value that indicates if the inference rule fired and the utility value.
The knowledge represented by general inference rules is useful even if the knowledge is not completely accurate. In many cases the general inference rules hinder the
performance in the early learning cycles. However, these general inference rules serve
as the base for more specific and accurate inference rules.
The use of the IRBREL algorithm in Real Time Strategy Video Games improves
the video game in several ways:
• The agents using the IRBREL algorithm will learn new strategies and try different
things. Consequently, the video game is not always the same and that could be
attractive for human players.
• The improvisation and learning capabilities of the IRBREL algorithm will make
that the human players could not use the same strategy to win the game. The
agents will learn from the human player as the human players will learn from the
AI.
• The change of behavior of the IRBREL agents will give the video game an additional challenge. This additional challenge will help to avoid that human players
get bored of the video game too quickly. Consequently, the life time of the video
game will be increased.
• If the life time of the video game is increased and it keeps popular for a longer
time frame, then the developers could sell more copies of the video game.
6.3
Future Work
At this moment, the IRBREL algorithm only works in Real Time Strategy Video
Games that are completely observable and deterministic. Besides the IRBREL algorithm needs a module that generates the world states and another module that generates the utility to be in the current world state. For future work, it is intended to
modify the IRBREL algorithm to make it able to work in a more wide range of Real
Time Strategy Video Games.
The completely observable environment is a restriction that helps the IRBREL
algorithm to learn why the world changes. Using the IRBREL algorithm in not completely observable environments could lead to the generation of superstitions, like the
dance to make rain superstition or the cargo cult described in chapter 1. The learning
of that kind knowledge should have an evolutionary advantage. Otherwise, the humans
73
could have evolved to avoid those situations. Notwithstanding, in order to handle that
kind of knowledge, the IRBREL algorithm will need several modifications.
The Science says that everything have a scientific explanation. Nonetheless, if
there are not known or are not observable all the factors that are involved, then there
is not enough knowledge to learn accurate cause-effect knowledge. Another additional
skill of the IRBREL algorithm could be its use in non-deterministic environments. The
IRBREL algorithm will need to know that sometimes the consequences of its actions
could be different. To manage non-deterministic environments could be used inference
rules that could have multiple conclusions. These multiple conclusions should have
their own utility value and a probability value in which the consequences specified by
the conclusion come true.
The IRBREL algorithm is only able to handle integer values in its variables.
Nevertheless, the IRBREL algorithm could be modified to be able to use real values in
its variables. The inference rules could be adapted to handle Fuzzy logic. This could
lead to a new way to use and learn strategies.
The abstraction of knowledge is one of the best capabilities of the humans. However, the IRBREL algorithm does not have that capability yet. The IRBREL algorithm
only works with the variables given by the world state generator. It could be created an
additional module, for the IRBREL algorithm, that learns to generate abstract variables. That way the IRBREL algorithm could use a single abstract variable that is
more representative than a group of simple variables. Learning to generate abstract
variables could lead to simpler inference rules that become accurate quickly.
The IRBREL algorithm is designed to be used as the AI of an agent. Notwithstanding, if a video game uses many copies of the same agent (like a group of soldiers),
then the learned knowledge of one agent could be shared to the other agents. Each
agent will have a different perspective of the world and different experiences. If multiple
agents are set to learn at the same time and share their knowledge, then the learning
process could be speeded up.
The use of inference rules to represent the learned knowledge helps the IRBREL
algorithm to quickly decide which strategy execute. Instead of analyzing the whole
world state space, the IRBREL algorithm just needs to look which inference rule fires.
Then, from the inference rule that fire, executes the inference rule that gives the agent
the highest utility value. This process saves a lot of time because instead of analyzing
multiple variables and world states, the IRBREL algorithm only needs to analyze two
values: The Boolean value that indicates if the inference rule fired and the utility value.
Nonetheless, this advantage is limited by the available hardware.
If a computer only has one microprocessor, then that microprocessor needs to
check if an inference rule fires or not. Then, the microprocessor needs to find, form the
inference rules that fire, the one with the best utility value. Besides, that microprocessor
must handle the video game’s graphics, manage the input of human players and manage
74
other agents that could be also in that game.
To solve the hardware problem, could be created a hardware module that manages
the inference rules. Instead of using inference rules defined in software, could be used
inference rules defined in hardware. This hardware module could use a variant of PALs
and GALs technology to create, modify and update inference rules. That way could be
checked almost instantly which inference rule fired and which one has the best utility
value. This hardware module could be also adapted to be used in robotics.
75
Bibliography
[1] RoboCup. http://www.robocup.org/, 2008.
[2] R. C. Atkinson and R. M. Shiffrin. Human memory: A proposed system and its
control processes. The Psychology of Learning and Motivation, Vol 2. London:
Academic Press, 1968.
[3] A. D. Baddeley, N. Thomson, and M. Buchanan. Word length and the structure
of short term memory. pages 575–589, 1975.
[4] David M. Bourg and Glenn Seemann. AI for Game Developers: Creating Intelligent Behavior in Games. O’Reilly Media, Inc, 2004.
[5] M. Brockington and M. Darrah. How not to Implement a Basic Scripting Language.
AI Game Programming Wisdom, 2002.
[6] M. Buro. Rts games as test-bed for real-time ai research. pages 481–484, 2003.
[7] Edward Callaway. Fino-escalar la especificidad de redes corticales depende de tipo
y de conectividad inhibitorios de la célula. Instituto de Salk, 2005.
[8] M. Campbell, A. J. Hoane Jr., and F. H. Hsu. Deep blue. pages 57–83, 2002.
[9] Alex J. Champandard. AI Game Development: Synthetic Creatures with Learning
and Reactive Behaviors. New Riders Publishing.
[10] D. P. de Farias and N. Megiddo. Learning and teaching in repeated games: A
machine learning approach to long-term best-response play. 2003.
[11] J. Schmidhuber F. Gomez and and R. Miikkulainen. Efficient non-linear control
through neuroevolution. pages 654–662, 2006.
[12] J. Furnkranz. Machine learning in games: A survey in machines that learn to play
games. pages 11–59, 2001.
[13] Joaquı́n Fuster. Cortical Dynamics of Working Memory. IBM Almaden Institute,
2006.
76
[14] Hisashi Handa and Satoshi Kajiyama. Decision-making of football agents with
support vector machine. 2007.
[15] J. Haugeland. Artificial intelligence: The very idea. 1985.
[16] Judy Inglis. Cargo cults: The problem of explanation. 1957.
[17] Holger Jebens. Cargo, cult, and culture critique. 2004.
[18] Y. Jiang, J. Jiang, and I. J. Palmer. Computerized interactive gaming via supporting vector machines. 2008.
[19] J. Ma and S. Perkinsn. Time-series novelty detection using one-class support vector
machines. pages 1741–1745, 2003.
[20] Yoichiro Matsuno, Tatsuya Yamazaki, Jun Matsuda, and Shin Ishii. A multiagent
reinforcement learning method based on the model inference of the other agents.
pages 67–76, 2002.
[21] Risto Miikkulainen. Creating intelligent agents in games. 2006.
[22] A. Nareyek. Intelligent agents for computer games. pages 414–422, 2002.
[23] Nick Palmer. Machine learning in games development. AI Depot, 2008.
[24] Marc Ponsen and Pieter Spronck. Improving adaptive game ai with evolutionary
learning. pages 389–396, 2004.
[25] Marc Ponsen and Pieter Spronck. Improving adaptive game ai with evolutionary
learning. pages 389–396, 2004.
[26] S. J. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Pearson
Education, Inc, 2005.
[27] J. Schaeffer. One jump ahead. 1997.
[28] J. Schaeffer. A gamut of games. pages 29–46, 2001.
[29] B. Scott. The illusion of intelligence. pages 16–20, 2002.
[30] David Silver, Richard Sutton, and Martin Muller. Reinforcement learning of local
shape in the game of Go. 2007.
[31] P. Spronck, I. Sprinkhuizen-Kuyper, and E. Postma. Improving opponent intelligence through offline evolutionary learning. pages 20–27, 2003.
77
[32] Pieter Spronck, Marc Ponsen, Ida Sprinkhuizen-Kuyper, and Eric Postma. Adaptive game ai with dynamic scripting. Technical report, Machine Learning, Vol. 63,
No. 3, 2006.
[33] K. O. Stanley, B. D. Bryant, and R. Miikkulainen. Real-time neuroevolution in
the nero video game. pages 653–668, 2005.
[34] K. O. Stanley and R. Miikkulainen. Evolving neural networks through augmenting
topologies. pages 99–127, 2002.
[35] P. Tozour. The evolution of game ai. pages 3–15, 2002.
[36] P. Tozour. The Perils of AI Scripting. AI Game Programming Wisdom, 2002.
[37] Peter Worsley. The trumpet shall sound: a study of cargo cults in melanesia. 1957.
[38] X. Yao. Evolving artificial neural networks. pages 1423–1447, 1999.
78