UNIVERSITATEA BABEŞ-BOLYAI CLUJ-NAPOCA
FACULTATEA DE MATEMATICǍ ŞI INFORMATICǍ
SPECIALIZAREA INFORMATICĂ ROMÂNĂ
LUCRARE DE DIPLOMǍ
Machine Learning-based gameplay
Conducător ştiinţific
Prof. univ. dr. Czibula Gabriela
Absolvent
Cîmpean Alexandru
2012
Table of contents
Introduction ................................................................................................................................... 3
I. Game perspective in Artificial Intelligence ............................................................................. 5
1.1 A general game description................................................................................................. 5
1.2 The context of video games ................................................................................................. 5
1.3 Artificial intelligence in video games ............................................................................... 13
II. Learning to play ..................................................................................................................... 17
2.1 Software agents: autonomous programs ......................................................................... 17
2.2 Machine learning: making computers act ....................................................................... 22
2.3 Embedding machine learning in computer ga mes ......................................................... 23
2.3.1 The history of machine learning and games............................................................. 23
2.3.2 What do the games have to learn? ............................................................................ 23
2.3.3 Creating a Learning Agent ........................................................................................ 25
2.3.4 Proble ms with Learning ............................................................................................. 28
III. Relevant applications of machine learning in games ........................................................ 30
3.1 TD-Gammon ...................................................................................................................... 30
3.2 Samuel's checkers program .............................................................................................. 31
3.3 NeuroChess......................................................................................................................... 32
3.4 SAL...................................................................................................................................... 32
IV. “A Tale of Two Catapults” – machine learning techniques applied in games................ 34
4.1 Proble m statement ............................................................................................................. 34
4.2 Motivation .......................................................................................................................... 34
4.3 Analysis and design............................................................................................................ 35
4.3.1 Proble m analysis ......................................................................................................... 35
4.3.2 Solution’s design ......................................................................................................... 38
4.4 Implementation .................................................................................................................. 42
4.5 User manual ....................................................................................................................... 49
4.6 Results obtained ................................................................................................................. 51
4.7 Future work........................................................................................................................ 53
Conclusions .................................................................................................................................. 54
Bibliography ................................................................................................................................ 55
2
Introduction
Since the beginning of computer games, whether they were inspired from traditional games such
as chess, backgammon, checkers, whether they exploited the new opportunities of the virtual
worlds, creating new and innovative genres, the problem of an artificial player appeared. The
players of those video games desired a program that knows how to play a game, and will play it
in collaboration or against him, using similar skills as themselves.
The first games had a component that would play against the human player, but that proved to
be very poor and limited according to the players’ demands. Then, in order to beat the game,
some games had a collaborative part which helped the player in different situations that mostly
couldn’t be done alone. To make such programs the developers inspired themselves from a
classical branch of artificial intelligence, more exactly game theory. Using this method the
programs used different algorithms to choose the best moves based on different heuristics and
classical moves. The game theory approach proved to be a good one until its limits were
discovered. This method requires a well established strategy and when the state space is big they
will run very slow. Another deficit of this method is the incapacity to adapt to new situations,
making the radical change of the game parameters difficult or even impossible.
Later, the concept of agent, which we will explain on this work, introduced new possibilities for
programs to play games. Thus the artificial intelligence in video games has grown according to
the need of better non-human players and was split in multiple types: traditional game playing,
mimicking intelligence and human player behavior and some of them tried to adapt to methods
from another branch of artificial intelligence – machine learning. Machine learning is a concept
we will explain further in this work and applied in games, agents are trying to learn a game
without knowing its rules and without considering a strategy, only based on the feedback he
receives after playing repeated games with himself.
3
In the first chapter of this work we will present theoretical aspects of video games in general,
artificial intelligence in video games as well as different traditional techniques of making a game
playing program.
In the second chapter we will present other techniques of artificial intelligence. In the first place
we will describe agents and their characteristics; we will see what an agent needs to be
considered intelligent and what features are required; we will classify the agents according to
different criteria and give examples of some of them. Further in this chapter we will present the
concept of machine learning; we will describe its subtypes and characteristics; we will present
the steps needed to make a learning algorithm; also here we will describe the concept of learning
to play, give examples of learning methods used in game play and see how they work;
In the third chapter of this work we will talk about the recent work in this domain and analyze
some of the most relevant learning games together with their characteristics.
The fourth chapter of this work is a practical application of machine learning in games. The
game is called “A Tale of Two Catapults”, it is a game in which the player controls a catapult
and in order to win the game he has to hit the opponent catapult. He can choose the required
angle and initial velocity to shoot a projectile at some distance the opponent is situated.
Furthermore the wind can blow from any direction and influence the trajectory of the projectile.
In order to win the game the player has to hit the opponent three times. The opponent is in fact a
program designed to learn to shoot after playing many games with himself. In the fourth chapter
we will present all the steps in developing such a program including the design, the algorithms
and methods used, difficulties met and the results obtained.
4
I. Game perspective in Artificial Intelligence
1.1 A general game description
Below we have some definitions of games from different theoreticians and specialists in games:

"A game is a system in which players engage in an artificial conflict, defined by rules,
that results in a quantifiable outcome." (Katie Salen and Eric Zimmerman)[1]

"A game is a form of art in which participants, termed players, make decisions in order to
manage resources through game tokens in the pursuit of a goal." (Greg Costikyan)[2]

"A game is an activity among two or more independent decision- makers seeking to
achieve their objectives in some limiting context." (Clark C. Abt)[3]

"At its most elementary level then we can define game as an exercise of voluntary control
systems in which there is an opposition between forces, confined by a procedure and
rules in order to produce a disequilibrial outcome." (Elliot Avedon and Brian SuttonSmith)[4]

"A game is a form of play with goals and structure." (Kevin J. Maroney)[5]
1.2 The context of video games
There are two principal reasons to continue to do research on games ... First, human fascination
with game playing is long-standing and pervasive. Anthropologists have catalogued popular
games in almost every culture....Games intrigue us because they address important cognitive
functions....The second reason...is that some difficult games remain to be won, games that people
play very well but computers do not. These games clarify what our current approach lacks. They
set challenges for us to meet, and they promise ample rewards [6]. - Susan L. Epstein, Game
Playing: The Next Moves
5
Computer game designer Chris Crawford attempted to define the term game using a series of
dichotomies:

Creative expression is art if made for its own beauty, and entertainment if made for
money.

A piece of entertainment is a plaything if it is interactive. Movies and books are cited as
examples of non- interactive entertainment.

If no goals are associated with a plaything, it is a toy. (Crawford notes that by his
definition, (a) a toy can become a game element if the player makes up rules, and (b) The
Sims and SimCity are toys, not games.) If it has goals, a plaything is a challenge.

If a challenge has no "active agent against whom you compete," it is a puzzle; if there is
one, it is a conflict. (Crawford admits that this is a subjective test. Video games with
noticeably algorithmic artificial intelligence can be played as puzzles; these include the
patterns used to evade ghosts in Pac-Man.)

Finally, if the player can only outperform the opponent, but not attack them to interfere
with their performance, the conflict is a competition. (Competitions include racing and
figure skating.) However, if attacks are allowed, then the conflict qualifies as a game.[7]
Video Games encompass a range of electronic games, including coin-operated arcade games,
console and cartridge games, various handheld electronics, and games on diskette and CD-ROM
ranging in sophistication from simple geometric shapes to virtual reality programs with
movielike qualities. [8]
Most games fall within a particular category. Some different gaming styles and, thus, could
appear under more than one category simultaneously. And others pioneer new approaches to
electronic entertainment. Those often fall outside of any pre-conceived genre. If such a title
becomes popular enough that others try to duplicate the experience, a new genre comes into
being.[9]
Below is a list of some of the most common video game genres with a brief description and some
examples of games that fit within that category. This list is not comprehensive and is limited to
larger classifications. It is easy to classify particular titles much more narrowly and, thus, create
6
dozens of genres and/or sub- genres. This is an attempt to give a broader perspective of types of
video games.[9]

Shooter:
One of the oldest genres of video game is the classic shooter. It has roots in the early 60s with
Steve Russell's Spacewar! Shooters are games that require the player to blow away enemies or
objects in order to survive and continue gameplay. They usually fall into one of two categories:
1) horizontal, or 2) vertical. However, like Spacewar!, Star Castle, and Asteroids, there are
shooters that are neither horizontal or vertical. These involve moving around the screen and
shooting in whatever direction necessary to keep from being destroyed. Other classic examples
include Defender, Galaga, R-Type, Phoenix, Space Invaders, Tempest, Xevious, and Zaxxon.

First-Person-Shooter (or FPS):
This is an example of a sub-genre that has grown enough to become its own genre. In fact,
because of the prevalence of these games, many people use the term "shooter" to refer to firstperson-shooters. These games are realtime fast-paced action games in which the player navigates
an environment from a first-person perspective and, usually, blows everything and everyone
away whenever possible. Though Wolfenstein 3D is regarded as the first sucessful example of
this genre, it wasn't until the release of Doom that people began to recognize the true potential of
this type of gaming. Doom enabled multiple game players to share in the same game
simultaneously via modem and LAN. This would become the standard of this genre, opening the
game format up to multi-player deathmatches that would become so important to the format that
some put little effort into story and the single-player experience in general (i.e., Unreal
Tournament and Quake III). Though this is a relatively new genre (since the early 1990s), it has
grown in popularity. Examples of first-person-shooter franchises include Wolfenstein 3D, Doom,
Duke Nukem 3D, Descent, Marathon, GoldenEye, Halo, Quake, and Time Splitters.
7

Adventure:
Another of the first video game genres, especially from the computer platforms, was the
adventure game. These were initially text-based games like Will Crowther's Collossal Cave and
the original Zork games. However, as the power of the gaming systems grew, developers tried to
tap into the visual capabilities of each consecutive platform. The Atari VCS offered a game
entitled Adventure. Roberta Williams began develping the King's Quest series for Sierra Online
in an attempt to add interactive graphics and point-and-click funtionality to the more puzzleoriented traditional text-based adventure. There has always been a strong following for this genre
because of the challenge of puzzle-solving and the general lack of violence. This has also made it
popular for many non-traditional gaming demographics. In recent years, LucasArts and Cyan
have been known for their contributions to the adventure genre. Other examples of adventure
franchises include Gabriel Knight, Indiana Jones, Maniac Mansion, Monkey Island, Myst, Police
Quest, and Syberia.

Platform:
It is believed that the platform genre began in 1981 with the release of the games Donkey Kong
and Space Panic. Games within this genre are usually identified by navigating environments that
require timing and jumping in order to reach a desitination while avoiding and/or disposing of
enemies. Many of these, like Donkey Kong, have a series of screens, each with its own
individual pattern of challenges. As companies began to develop platform games for home
consoles and computers instead of arcade machines (i.e. Super Mario Bros for the Famicom and
Nintendo Entertainment system), they took advantage of the evolving processors and and greater
memory capacity by transcending individual screens and utilizing actively side-scrolling worlds.
This evolutionary step in platform games moved them closer to immersive stories rather than
challenging puzzles. Platform video games continued to evolve as gaming became more 3D. One
of the greatest 3D platform games was introduced with the launch of the Nintendo 64 and was
called Super Mario 64. Examples of 2D screen-based platform franchises include Bubble
Bobble, Burgertime, Donkey Kong, Lode Runner, Mario Bros., and Space Panic. Examples of
2D scrolling platform franchises include Bonk, Donkey Kong Country, Sonic the Hedgehog,
8
Super Mario Bros., and Vectorman. Examples of 3D platform franchises include Crash
Bandicoot, Pac-Man World, Spyro the Dragon, and the aforementioned Super Mario 64.

Role-Playing Games (RPGs):
Evolving from pen-and-paper games like Dungeons and Dragons, RPGs are a special type of
adventure game that usually incorporate three major elements: 1) a specific quest, 2) a process
for evolving a character through experience to improve his/her ability to handle deadlier foes, 3)
the careful acquisition and management if inventory items for the quest (i.e., weapons, armor,
healing items, food, and tools). Having said that, these games still have many variations and
appearances.

Puzzle:
In many ways, puzzle video games are not dissimilar from traditional puzzles. What they offer
are unique environments that are not as easily introduced in one's living room. For example,
Wetrix enables the player to build up a series of walls that would be able to contain a deluge of
water when it falls. Successful completion of a level involves capturing enough water. Other
examples include Tetris, Intelligent Qube, Puzzle Bobble, Puyo Puyo, Devil Dice, and Mercury.

Simulations:
By their nature, simulations are attempts to accurately re-create an experience. These can be in
the form of management simulations like SimCity and Theme Hospital, or more hands on like
MicroSoft Flight Simulator or Gran Turismo.

Strategy/Tactics:
Like simulations, strategy/tactics games attempt to capture a sense of realism for the game player
to experience. However, these titles are often turn-based as opposed to realtime and they give the
player a greater sense of specific control over a situation. Franchises that fall into this genre
include Ogre Tactics, Command and Conquer, Final Fantasy Tactics, and Worms.
9

Sports
As you can imagine, sports games are those that simulate the playing of sports. Many of these
have incorporated novel aspects beyond the games themselves. For example, most football video
games like the Madden series enable the player to create and customize teams and play them for
an entire season. Furthermore, many sports games include management elements beyond the
games themselves. There is quite a bit of variety in this genre for fans of the games, the players,
and the behind the scenes responsibilities of owning a team.

Fighting:
These titles pit player against player (usually 2 players head-to-head) and involve one triumphing
over the other. Many of these games include a single player mode, but the real draw to this genre
is the ability to demonstrate one's gaming prowess against a friend. Examples of franchises in
this genre include Street Fighter, Soul Calibur, Mortal Kombat, Tekken, Virtua Fighter, Dead or
Alive, King of Fighters, and Bloody Roar.

Dance/Rhythm:
Dance Dance Revolution is probably the single largest franchise in this genre. Of the rest, many
require a specialized controller like DDR, but several don't. This grouping of games is
differentiated by the timed elements usually synched to music somehow. Other good examples of
this form include Parappa the Rapper, Bust a Groove, Gitaroo Man, Space Channel 5,
Frequency, Beatmania, Para Para Paradise, Donkey Konga, and Eyetoy Groove.

Survival Horror:
As the name suggests, these titles are an interactive evolutionary step of the horror genre. The
main gameplay mechanic in these is to "survive" the environment that includes fantastic or
supernatural elements that are very frightening and often disturbing. Many of these titles are
rated mature because of they are not intended for younger audiences and often include graphic
scenes.
10

Hybrids:
It's important to recognize that many games are not limited to a single genre. Some are the
combination of two or more game types. In fact, as gaming evolves, we see lines blurred
between genres more frequently than not. Since the introduction of 3D gaming, the
action/adventure genre has grown dramatically. It is practically a catch-all category that
incorporates 3D games with real-time combat and puzzle-solving in a fairly cohesive storyline.
Many of these games are also first-person-shooters. Some are 3D platform titles. And most
survival horror titles qualify as Action/Adventure games too. Another example of a hybrid is
Myst. It is both an adventure game and a puzzle game. However, it is most certainly not an
Action/Adventure game. [9]
Below we have three examples of games that show the evolution of video games throughout the
decades:
Figure 1 – OXO (Noughts And Crosses) – made in 1952 by Alexander S. Douglas
Figure 2 - Super Mario Bros. – made in 1985 by Nintendo Co., Ltd
Figure 3 – Assassin’s Creed – made in 2007 by Ubisoft Entertainment S.A.
11
Figure 1 - OXO
Figure 2 - Super Mario Bros.
Figure 3 - Assassin's Creed
12
1.3 Artificial intelligence in video games
Artificial intelligence in video games can be implemented using two types of algorithms:
machine learning, which we will discuss in the next chapter and search algo rithms. Next we will
make a classification of search algorithms and describe them.
Search in artificial intelligence
Search plays a major role in solving many Artificial Intelligence (AI) problems. Search is a
universal problem-solving mechanism in AI. In many problems, sequence of steps required to
solve is not known in advance but must be determined by systematic trial-and-error exploration
of alternatives.
The problems that are addressed by AI search algorithms fall into three general classes:
single-agent path-finding problems, two-players games, and constraint-satisfaction problems [10]

Path-finding problems
Classic examples in the AI literature of path- finding problems are sliding- title puzzles, Rubik’s
Cube and theorem proving. The single-title puzzles are common test beds for research in AI
search algorithms as they are very simple to represent and manipulate. Real-world problems
include the traveling salesman problem, vehicle navigation, and the wiring of VLSI circuits. In
each case, the task is to find a sequence of operations that map an initial state to a goal state.

Two-players games
Two-players games are two-player perfect information games. Chess, checkers, and othello are
some of the two-player games.
13

Constraint-satisfaction problems
Eight Queens problem is the best example. The task is to place eight queens on an 8*8
chessboard such that no two queens are on the same row, column or diagonal. Real- world
examples of constraint satisfaction problems are planning and scheduling applications.
Proble m space
Problem space is a set of states and the connections between to represent the problem. Problem
space graph is used to represent a problem space. In the graph, the states are represented by
nodes of the graph, and the operators by edges between nodes. Although most problem spaces
correspond to graphs with more than one path between a pair of nodes, for simplicity they are
often represented as trees, where the initial state is the root of the tree. The cost of the
simplification is that any state that can be reached by two different paths will be represented by
duplicate nodes thereby increasing the tree size. The benefit of using tree is that the absence of
cycles greatly simplifies many search algorithms.
One feature that distinguishes AI search algorithms from other graph-searching algorithms is the
size of the graph involved. For example, the entire chess graph is estimated to contain over
10^40 nodes. Even a simple problem like twenty-four puzzle contains almost 10^25 nodes. As a
result, the problem-space graphs of AI problems are never represented explicitly by listing each
state, but rather are implicitly represented by specifying an initial state and a set of operators to
generate new states from existing states. Moreover the size o f an AI problem is rarely expressed
as the number of nodes in its problem-space graph. The two parameters of a search tree that
determine the efficiency of various search algorithms are its branching factor and its solution
depth. The branching factor is nothing but average number of children of a given node. For
example in eight-puzzle problem, the average branching factor is square-root(3) or about 1.732.
The solution depth of a problem instance is the length of a shortest path from the initial state to a
goal state or the length of a shortest sequence of operators that solve the problem. [10]
14
Classes of search algorithms

For virtual search spaces
Algorithms for searching virtual spaces are used in constraint satisfaction problem, where the
goal is to find a set of value assignments to certain variables that will satisfy specific
mathematical equations and inequations. They are also used when the goal is to find a variable
assignment that will maximize or minimize a certain function of those variables. Algorithms for
these problems include the basic brute-force search (also called "naïve" or "uninformed" search),
and a variety of heuristics that try to exploit partial knowledge about structure of the space, such
as linear relaxation, constraint generation, and constraint propagation.
An important subclass are the local search methods, that view the elements of the search space as
the vertices of a graph, with edges defined by a set of heuristics applicable to the case; and scan
the space by moving from item to item along the edges, for example according to the steepest
descent or best- first criterion, or in a stochastic search. This category includes a great variety of
general metaheuristic methods, such as simulated annealing, tabu search, A-teams, and genetic
programming, that combine arbitrary heuristics in specific ways.
This class also includes various tree search algorithms, that view the elements as vertices of a
tree, and traverse that tree in some special order. Examples of the latter include the exhaustive
methods such as depth- first search and breadth-first search, as well as various heuristic-based
search tree pruning methods such as backtracking and branch and bound. Unlike general
metaheuristics, which at best work only in a probabilistic sense, many of these tree-search
methods are guaranteed to find the exact or optimal solution, if given enough time.
Another important sub-class consists of algorithms for exploring the game tree of multiple-player
games, such as chess or backgammon, whose nodes consist of all possible game situations that
could result from the current situation. The goal in these problems is to find the move that
provides the best chance of a win, taking into account all possible moves of the opponent(s).
Similar problems occur when humans or machines have to make successive decisions whose
outcomes are not entirely under one's control, such as in robot guidance or in marketing,
financial or military strategy planning. This kind of problem - combinatorial search - has been
15
extensively studied in the context of artificial intelligence. Examples of algorithms for this class
are the minimax algorithm, alpha-beta pruning, and the A* algorithm.[11]

For sub-structures of a given structure
The name combinatorial search is generally used for algorithms that look for a specific substructure of a given discrete structure, such as a graph, a string, a finite group, and so on. The
term combinatorial optimization is typically used when the goal is to find a sub-structure with a
maximum (or minimum) value of some parameter. (Since the sub-structure is usually represented
in the computer by a set of integer variables with constraints, these problems can be viewed as
special cases of constraint satisfaction or discrete optimization; but they are usually formulated
and solved in a more abstract setting where the internal representation is not explicitly
mentioned.)
An important and extensively studied subclass are the graph algorithms, in particular graph
traversal algorithms, for finding specific sub-structures in a given graph — such as subgraphs,
paths, circuits, and so on. Examples include Dijkstra's algorithm, Kruskal's algorithm, the nearest
neighbour algorithm, and Prim's algorithm.
Another important subclass of this category are the string searching algorithms, that search for
patterns within strings. Two famous examples are the Boyer–Moore and Knuth–Morris–Pratt
algorithms, and several algorithms based on the suffix tree data structure. [11]
16
II. Learning to play
2.1 Software agents: autonomous programs
There have been many debates about how a software agent should be defined, and a universally
accepted definition has not been given. We have below some definitions of the software agents
from the point of view of theoreticians and other people in this domain.
Software agents are computational systems that inhabit some complex dynamic environment,
sense and act autonomously in this environment, and by doing so realize a set of goals
or tasks for which they are designed (P.Maes) [12].
Russel and Norvig point out: “the notion of an agent is meant to be a tool for analyzing systems,
not an absolute characterization that divides the world into agents and non-agents” [14]. We can
say that ‘software agent’ or ‘intelligent agent’ can best be seen as an umbrella term for programs
that to some extent display attributes commonly associated with agency. (Nwana H.) [13]
Franklin and Graesser (1996) give their own definitions of what an agent is supposed to be: “an
autonomous agent is a system situated within and part of an environment that senses that
environment and acts on it, over time, in pursuit of its own agenda and so as to effect what it
senses in the future.”[18]
17
Figure 4 -Franklin and Graesser’s (1996) agent taxonomy
Agent characte ristics
There are some characteristics that define a so ftware agent. In the section below we will
enumerate and describe each characteristic as defined and accepted in literature.

Reactive
In order for an agent to function autonomously in any given environment it must be able to
perceive its environment and act in a timely fashion upon changes that occur in it. A software
agent may employ any type and number of sensors to sense its environment. The agent can react
to sensory input using its actuators. We can differentiate between various degrees of reactivity,
ranging from purely reactive software agents on the one hand, to software agents that deliberate
extensively before reacting on the other hand. [15]

Pro-active and goal-oriented
Pro-activity is a more specialized form of autonomy. When an agent is said to be pro-active, it
does not simply act in response to its environment, but it will exhibit goal-directed behavior and
take initiative to attain its goals or design objectives [15].
18

Deliberative
More sophisticated agents are not merely reactive (i.e., operating in a stimulus response manner)
but are able to reason about their actions. The ability to reason enables agents to act pro-actively
and perform more difficult tasks in complex environments. [16]

Continual
In order for a software agent to accomplish its goals, it must have temporal continuity. The agent
must be able to function over a certain period of time with persistence of identity and state.
Agents that have an episodic memory are able to learn from previous experiences [17]

Adaptive
Making agents adaptive is one way of attaining flexibility. Adaptivity can range from (1) being
able to adapt flexibly to short-term, smaller changes in the environment, to (2) dealing with more
significant and long-term (lasting) changes in the environment [12]. Software agents that are
able to deal with longterm changes are able to improve themselves and their performance over
time by storing the knowledge of past experiences within their internal state and taking this
knowledge into account when executing (similar) actions in the future. [16]

Communicative
Agents should be able to communicate with other software agents and even humans in order to
complete their tasks and help other agents complete theirs. [15] Especially in multi-agent
systems the ability to communicate with other agents is important. Agents communicate with
other agents using a common agent language, such as FIPA ACL or KQML. When agents
communicate with humans they must communicate using natural language. [16]

Mobile
Although mobility is neither a necessary nor a sufficient condition for agency, many scholars
(Gilbert et al. 1995; Nwana 1996; Brazier et al. 2003) include mobility when describing agent
characteristics. It is oftentimes better for an agent to interact with a remote system at the location
of the remote system than to do it over a distance. Several reasons for this preferred form of
19
interaction can be specified. A first reason is efficiency. Network traffic can be reduced when the
agent and the remote system are at the same location. For instance, when an agent queries a
remote database, data has to be sent back and forth between the remote database and the agent.
This communication can be kept local when the agent and the remote system are at the same
location, thereby reducing the strain on external networks such as the Internet. A second reason
is that data need not be exchanged over (public) networks but can be handled locally. It also
means that agents can operate more secure. A third reason is that the remote system only allows
for agents to operate locally, thereby forcing the agent to migrate to the remote location.[16]
Agent topologies
In the next section we will investigate different agent topologies and try to place different agents
into classes. A typology refers to the study of types of entities. There are several dimensions to
classify existing software agents.
Firstly, agents may be classified by their mobility, i.e. by their ability to move around some
network. This yields the classes of static or mobile agents. [13]
Secondly, they may be classed as either deliberative or reactive. Deliberative agents derive from
the deliberative thinking paradigm: the agents possess an internal symbolic, reasoning model and
they engage in planning and negotiation in order to achieve coordination with other agents. Work
on reactive agents originate from research carried out by Brooks (1986) and Agre & Chapman
(1987). These agents on the contrary do not have any internal, symbolic models of their
environment, and they act using a stimulus/respo nse type of behavior by responding to the
present state of the environment in which they are embedded (Ferber, 1994). Indeed, Brooks has
argued that intelligent behavior can be realized without the sort of explicit, symbolic
representations of traditional AI (Brooks, 1991b). [13]
20
Figure 5 - Typology based on Nwana’s (Nwana 1996) primary attribute dimension
Thirdly, agents may be classified along several ideal and primary attributes which agents should
exhibit. We have identified a minimal list of three: autonomy, learning and cooperation. We
appreciate that any such list is contentious, but it is no more or no less so than any other
proposal. Hence, we are not claiming that this is a necessary or sufficient set. Autonomy refers to
the principle that agents can operate on their own without the need for human guidance, even
though this would sometimes be invaluable. Hence agents have individual internal states and
goals, and they act in such a manner as to meet its goals on behalf of its user. A key element of
their autonomy is their proactiveness, i.e. their ability to take the initiative rather than acting
simply in response to their environment (Wooldridge & Jennings, 1995a). Cooperation with
other agents is paramount: it is the raison d’être for having multiple agents in the first place in
contrast to having just one. In order to cooperate, agents need to possess a social ability, i.e. the
ability to interact with other agents and possibly humans via some communication language
(Wooldridge & Jennings, 1995a). Having said this, it is possible for agents to coordinate their
actions without cooperation (Nwana et al., 1996). Lastly, for agent systems to be truly smart,
they would have to learn as they react and/or interact with their external environment. In our
view, agents are (or should be) disembodied bits of intelligence. Though, we will not attempt to
define what intelligence is, we maintain that a key attribute of any intelligent being is its ability
to learn. The learning may also take the form of increased performance over time. [13]
21
2.2 Machine learning: making computers act
Tom M.Mitchell is one of the most prominent theoreticians in the field of machine learning and
he gave a widely quoted definition about this term : “A computer program is said to learn from
experience E with respect to some class of tasks T and performa nce measure P, if its
performance at tasks in T, as measured by P, improves with experience E”[19]
Machine learning algorithms are organized into taxonomy, based on the desired outcome of
the algorithm. Common algorithm types include:
• Supervised learning - where the algorithm generates a function that maps inputs
to desired outputs. One standard formulation of the supervised learning task is the
classification problem: the learner is required to learn (to approximate the behavior
of) a function which maps a vector into one of several classes by looking at several
input-output examples of the function.
• Unsupervised learning - which models a set of inputs: labeled examples are not
available.
• Semi-supervised learning - which combines both labeled and unlabeled examples
to generate an appropriate function or classifier.
• Reinforcement learning - where the algorithm learns a policy of how to act given
an observation of the world. Every action has some impact in the environment, and
the environment provides feedback that guides the learning algorithm.
• Transduction - similar to supervised learning, but does not explicitly construct a
function: instead, tries to predict new outputs based on training inputs, training
outputs, and new inputs.
• Learning to learn - where the algorithm learns its own inductive bias based on
previous experience. [19]
Of course the list above can be enlarged with other types of machine learning and all kinds of
hybrid algorithms.
22
2.3 Embedding machine learning in computer games
2.3.1 The history of machine learning and games
The history of the interaction of machine learning and computer game-playing started at the
same time Artificial Intelligence appeared. Arthur Samuel was the first one to apply machine
learning techniques in games, working on his checker-playing program, this way being a pioneer
in machine- learning and game-playing techniques (Samuel, 1959, 1967). He facilitated the
evolution of both fields and since then the intersection of the two can be found regularly in
conferences in their respective fields and have evolved throughout research along the years.
In recent years, the computer games industry has discovered AI as a necessary element
to make games more entertaining and challenging and, vice versa, AI has discovered computer
games as an interesting and rewarding application area. The industry’s perspective is
witnessed by a multitude of recent books on introductions to AI techniques for game
programmers or a series of edited collections of articles. AI research on computer
games began to follow developments in the games industry early on, but since John Laird’s
keynote address at the AAAI 2000 conference, in which he advocated Interactive Computer
Games as a challenging and rewarding application area for AI (Laird & van Lent, 2001) [20]
which demonstrates the growing importance of game-playing applications for Artificial
Intelligence.[21]
2.3.2 What do the games have to learn?
The need of AI in computer games is growing, and with a good user experience must come an
artificial intelligence part as good as it. These needs can be recast as a call for new practical and
theoretical tools to help with [21]:

learning to play the game: Game worlds provide excellent test beds for investigating the
potential to improve agents’ capabilities via learning. The environment can be
constructed with varying characteristics, from deterministic and discrete as in classical
23
board and card games to non-deterministic and continuous as in action computer games.
Learning algorithms for such tasks have been studied quite thoroughly. Probably the bestknown instance of a learning game-playing agent is the Backgammon-playing program
TD-Gammon (Tesauro, 1995) [23].

learning about players: Opponent modeling, partner modeling, team modeling, and
multiple team modeling are fascinating, interdependent and largely unsolved challenges
that aim at improving play by trying to discover and exploit the plans, strengths, and
weaknesses of a player’s opponents and/or partners. One of the grand challenges in this
line of work are games like Poker, where opponent modeling is crucial to improve over
game-theoretically optimal play (Billings et al., 2002) [22].

behavior capture of players: Creating a convincing avatar based on a player’s in- game
behavior is an interesting and challenging supervised learning task. For example, in
Massive Multiplayer Online Role-playing Games (MMORGs) an avatar that is trained to
simulate a user’s game-playing behavior could take his creator’s place at times when the
human player cannot attend to his game character. First steps in this area have been made
in commercial video games such as Forza Motorsport (Xbox) where the player can train a
“Drivatar” that learns to go around the track in the style of the player by observing and
learning from the driving style of that player and generalizing to new tracks and cars.
model selection and stability: Online settings lead to what is effectively the unsupervised
construction of models by supervised algorithms. Methods for biasing the proposed
model space without significant loss of predictive power are critical not just for learning
efficiency, but interpretive ability and end-user confidence. optimizing for adaptivity:
Building opponents that can just barely lose in interesting ways is just as important for
the game world as creating world-class opponents. This requires building highly adaptive
models that can substantively personalize to adversaries or partners with a wide range of
competence and rapid shifts in play style. By introducing a very different set of update
and optimization criteria for learners, a wealth of new research targets are created [21].

model interpretation: “What’s my next move” is not the only query desired of models in
a game, but it is certainly the one which gets the most attention. Creating the illusion of
intelligence requires “painting a picture” of an agent’s thinking process. The ability to
describe the current state of a model and the process of inference in that model from
24
decision to decision enables queries that provide the foundation for a host of social
actions in a game such as predictions, contracts, counter- factual assertions, advice,
justification, negotiation, and demagoguery. These can have as much or more influence
on outcomes as actual in- game actions [21].

performance: Resource requirements for update and inference will always be of great
importance. The AI does not get the bulk of the CPU or memory, and the machines
driving the market will always be underpowered compared to typical desktops at any
point in time [21].
2.3.3 Creating a Learning Agent
We will now see how to implement a learning system into a game, because of the need of the
games to be adaptable. The example we will use is given by Nick Palmer in the article Machine
Learning in Games development and is the example of a team-strategy based paintball game.
The aim of the program is for a team of seven agents to capture the opponent's flag and bring it
back to their starting position. They must do this without being hit by the opposing team's
paintballs. For a start, we shall exclude the team strategy from our discussions, and concentrate
on the individual agents - it is no good having a perfect strategy if the agents aren't smart enough
to survive on their own [24].
We must consider the factors which will influence an agent’s performance in the game. Terrain
is an obvious start point, as this is all that stands between the two teams, so it must be used to the
agent's advantage. Secondly, there must be an element of stealth behind the behavior of each
agent, as otherwise it will be simple to undermine any tactics used during the game by simply
picking off the naïve agents one-by-one [24].
A learning agent is composed of a few fundamental parts: a learning element, a performance
element, a curiosity element (or 'problem generator'), and a performance analyzer (or 'critic').
The learning element is the part of the agent which modifies the agent's behavior and creates
improvements. Tom M.Mitchell names this component in his book “Machine Learning” The
Generalizer [19]. The performance element is responsible for choosing external actions based
25
on the percepts it has received (percepts being information that is known by the agent about its
environment). To illustrate this, consider that one of our agents is in the woods playing paintball.
He is aware of an opposing paintballer nearby. This would be the percept that the agent responds
to, by selecting an action - moving behind a tree. This choice of action is made by the
performance element. This component is called The Performance System by Tom M. Mitchell.
The performance analyzer judges the performance of the agent against some suitable
performance measure. The performance must be judged on the same percepts as those received
by the performance element - the state of affairs 'known' to the agent. When the analysis of
performance has been made, the agent must decide whether or not a better performance could be
made in the future, under the same circumstances. This decision is then passed to the learning
element, which decides on the appropriate alteration to future behavior, and modifies the
performance element accordingly. This component is called The Critic in Tom M. Mitchell’s
book [19][24].
Below we have an illustrated example of the four components above working together as
mentioned in Tom M. Mitchell’s book.
Figure 6 - Tom M. Mitchell's Learning Model
26
How do we make sure that the agent advances in its learning, and doesn't merely confine itself to
previously observed behavior? This is dealt with by the curiosity element (so-called because it
searches for a better solution) which has a knowledge of the desirable behavior of the agent (i.e.
it knows that being shot is not desirable, and that finding the opponent's flag is). To achieve
optimal performance, this element will pose new challenges to the agent in an attempt to prevent
bad habits developing. This component is called The Expe riment Generator by Tom Mitchell
[19].
To understand the benefits of this, we will consider a paintballer who is hiding behind a
tree. From his past experience, he knows that he is safe to stay where he is, and this would result
in an adequate performance. However, the curiosity element kicks in, and suggests that he makes
a break from his cover and heads to a nearby tree which is closer to the enemy flag. This may
result in the agent ultimately being shot at, but could also achieve a more desirable goal. It is
then up to the performance analyzer and the learning element to consider whether there is a
benefit to this change in strategy [24].
Figure 7 - Learning architecture proposed by Nick Palmer
This style of learning is known as reinforcement learning, which means that agent can see the
result of its actions, but is not told directly what it should have done instead. This means that the
agent must use, what is trial and error, to evaluate its performance and learn from mistakes. The
27
advantage to this is that there is no limitation on the behavior, other than the limit to alterations
suggested through the curiosity element. If after each action, the agent was told what its mistake
was and how it should correct its behavior, then the desired behavior must already be understood
[24].
As the learning agent is ultimately part of a game, it must not be left simply to work out for itself
how to play. The agents must be imparted with a fair degree o f prior knowledge about the way to
behave. In the case of paintball, this could include methods for avoiding fire by using cover,
which may later be adapted during the learning process [24].
2.3.4 Proble ms with Learning
Although machine learning can bring a very big improvement in gameplay there are certain
problems that can occur when tying to implement such an agent [24]:

Mimicking Stupidity - When teaching an AI by copying a human player's strategy, we
will find that the computer could be taught badly. This is more than likely when the
player is unfamiliar with a game. In this situation, a reset function may be required to
bring the AI player back to its initial state, or else a minimum level must be imposed on
the computer player to prevent its performance dropping below a predetermined standard.

Overfitting - This can occur if an AI agent is taught only a certain section of a game, and
then expected to display intelligent behavior based on its experience. Using a FPS as an
example, an agent which has learnt from its experience over one level will encounter
problems when attempting a new level, as it may not have learnt the correct 'lessons' from
its performance.

Local Optimality - When choosing a parameter on which the agent is to base its learning,
be sure to choose one which has no dependency on earlier actions. As an example, we
will take a snow-boarding game. The agent learns, through the use of an optimization
algorithm, the best course to take down the ski slope, using its rotation as a parameter.
This may mean that a non-optimal solution is reached, in which any small change cannot
improve performance.
28

Set Behavior - Once an agent has a record of its past behavior and the resulting
performance analysis, will it stick to the behavior which has been successful in the past,
or will it try new methods in an attempt to improve? This is a problem which must be
addressed or else an agent may either try to evaluate every possible behavior, or else stick
to one without finding the optimal solution [24].
29
III. Relevant applications of machine learning in games
In this chapter we will present the most relevant games with machine learning techniques and try
to analyze their structure and algorithms used.
3.1 TD-Gammon
One of the most impressive applications of reinforcement learning lately is presented by Gerry
Tesauro in backgammon (Tesauro, 1992, 1994, 1995). Tesauro's program, TD-Gammon, needed
little knowledge of table, but he learned to play very well, close to the world masters. Learning
algorithm TD-Gammon was a simple combination of algorithm TD (λ) and nonlinear function
approximation using a multilayer neural network trained by TD's errors calculated wit h
backpropagation algorithm. [23]
To apply the learning rule a source of backgammon games was needed. Tesauro obtained an
unending sequence of playing games that teach against the agent himself. To choose movements,
TD-Gammon considered each of the approximately 20 ways he could get the dice and
corresponding positions that would result. The artificial neuronal network was consulted to
estimate each of their values. The movement that would lead to the estimated value was selected.
Continuing in this way, it was possible to easily generate a large number o f table games. Each
game was treated as an episode, the sequence of positions as s0, s1, s2 ... that the rule applied
after each movement of TD.
The network’s weights were established initially as low random values. Initial assessments were
thus completely arbitrary. Since movements were selected based on these assessments, initial
movements were inevitably weak and original games often lasted hundreds or thousands of
moves up to win one. After dozens of games, however, performance has improved rapidly.
30
After playing about 300,000 games against itself, TD-Gammon 0.0 as described above, learned
to play as good as the best previous backgammon computer programs. This was a striking result,
because all previous performance implementations have used the extensive knowledge of table.
For example, the champion at that time was undoubtedly Neurogammon, another program
written by Tesauro, who used neural network learning but not TD. The network used by
Neurogammon was trained with copies provided by a backgammon expert, and, moreover, began
with a set of features specifically designed for backgammon. TD-Gammon 0.0, on the other
hand, was constructed essentially with zero backgammon knowledge. The fact that TD-Gammon
was as good as Neurogammon and the other previous programs demonstrates potential of
learning algorithms in games [23].
3.2 Samuel's checkers program
An important precursor of TD-Gammon was one part of Arthur Samuel’s work (1959, 1967) to
build the learning programs for playing checkers. Samuel was one of the first to effectively use
search heuristics and what we now call temporal-difference learning. Samuel wrote a checkers
program for the IBM 701 in 1952. Its learning program was completed in 1955 and was shown
on television in 1956. Future versions have acquired an ability to play good, but not expert [25].
Samuel used two main methods of learning, the easiest one being called rote learning. It
consisted simply in saving a description of each position of the board met during the game, along
with its value determined by back-up procedure Min-Max. The result was that, if a position has
already been seen again it appeared as a terminal position of a search tree, the search depth was
amplified effectively because the position value is cached earlier.
Rote learning produced slow but continuous improvement, which was most effective for opening
and closing the game. Samuel's program has become a "better than average" program, having
learned from many games with itself, a broad array of human opponents, and in a supervised
learning mode from examples in books [25].
31
3.3 NeuroChess
NeuroChess is a program that learned the game of chess created by Sebastian Thrun. It learns
from a combination of playing with itself and the observation of experts. The search algorithm is
taken from the GNU Chess, but the assessor is from a neural network.
NeuroChess is an application of "explanation-based neural networks" (EBNN). EBNN adjusts
explanation based learning theory for neural networks. NeuroChess represents a chess position as
a feature vector of 175 data by given by hand. It has two neural networks, V and M. V is the
evaluation function, which takes as input an array of features and produces an assessment used to
play. M is a model of chess, which has as input an array of features and the predicted value of
two half- moves later. The result of M is used in the formation of V [26].
After training of M with 120,000 games of master level, and V with other 2400 games,
NeuroChess defeated the GNU Chess about 13% of the games. The same preparation, a version
not using model M, NeuroChess chess wins about 10% of the games against GNU Chess. EBNN
thus improves its performance. Of course these percents will grow with the number of games
played.
Learning from the games against itself has failed. However, learning from the experts first
introduced some artifacts in program evaluations. A clear example of learning from experts is
that the program observes that the probability of winning is greater when experts move their
queen in the middle. The program does not realize that it is moved only when threatened.
Therefore the best results are provided by combining the games with itself and learning from
experts [26].
3.4 SAL
SAL is a system developed by Michael Gherrity learners to play more games.
SAL can apparently learn any two-player board game of perfect information, provided the game
can be played on a rectangular board and uses fixed types of pieces [27].
32
He works the following way:
The user supplies a move generator, written in C, and some constants that declare the size of the
board and the number and type of pieces used in the game. This is SAL’s only knowledge of the
rules of the game.
SAL chooses its moves with a search, like any chess program. It uses a two-ply, full-width
alpha-beta search, plus a consistency search for quiescence.
SAL’s evaluation function is a backpropagation neural network (actually, it uses separate
evaluators for the two sides of the game). The inputs to the network are features representing the
board position, the number of pieces of each type on the board, the type of piece just moved, the
type of piece just captured, and a bunch of ad hoc features having to do with pieces and squares
under attack and threats to win the game [27] .
The evaluator of the neural network is trained by temporal difference methods to estimate the
outcome of the game based on the current state.
SAL has been tested on games tic-tac-toe (X and 0), Connect-four, and chess.
 In tic-tac-toe, SAL learned to play perfect in 20,000 games, which means it is a bit slow.
 At Connect-four, a more complex game, after 100,000 games SAL has learned to
overcome an opponent about 80% of the time.

In chess, SAL has played 4,200 games against GNU Chess, with GNU Chess set to play
at a rate of one move per second. SAL has won eight games and lost others. The first
game shows random play by SAL, since it hadn’t learned anything yet. A drawn game
shows how much SAL improved, though it still plays very weakly. The graphs show that
SAL continues to improve [27].
33
IV. “A Tale of Two Catapults” – machine learning techniques
applied in games
4.1 Problem statement
The application is meant to be a Windows Phone application, a turn-based game with catapults.
The player controls a catapult and he must hit the opponent (in our case an AI opponent). In
order to win, the player needs to hit the opponent three times directly. If the player is hit three
times by the opponent, then the game is lost. The logic of the application is the essential part
because the AI opponent is not a normal one. The application is, in essence an example of how
machine learning techniques are used in video games and more than that. The central idea of this
project is not the game itself but the design of a framework, an age nt-oriented architecture that
can be applied to any turn-based two player game.
4.2 Motivation
Our motivation for this application is the growing need of computer game players for a better
experience and a more capable AI. Through the last decades games evolved drastically due to the
new limits of graphic processors. Along with this evolution came the evolution of the video
gamer, whose needs for a more competent opponent are still a problem nowadays. The problems
of the recent games are that their AI component is too weak, because it is static and limited.
There were some very good examples of machine learning techniques used in video games that
had shown us the great potential of machine learning, but unfortunately they are just a few.
Inspired by Tom M. Mitchell’s book “Machine Learning” we have decided to make a computer
game with a learning component and to show the results obtained.
34
4.3 Analysis and design
4.3.1 Proble m analysis
Being a game that can be only played on the computer, we do not have any human expert to
teach our program different strategies, besides the fact that the game itself is pretty simple. So we
have chosen a learning technique that doesn’t require a direct feedback. The most recommended
learning strategy for this kind of task is reinforcement learning or any other learning that is
works on the same principle, more exactly playing repeated games with itself in order to learn.
That being said we have chosen gradient descent learning for our program, a learning technique
in which the player has to learn a function, called target function which will reflect the
mathematical function that calculates the distance of a projectile that is being shot with an initial
velocity and angle. In order to evaluate a state of the game we will give every state a value
assigned by the target function. For the final game states it is very easy, we decide that the value
of the target function is V(s) = 2000 if the game is won or V(s) = -2000 if the game is lost, s
being the current state of the game we have to assign a value. For the intermediate states we need
an approximation function that must be as relevant as it can in reflecting the real value of the
state.
The mathematical function the program has to learn is D = v0 2 / g · sin (2θ), where v0 is the initial
velocity, g is the gravitational acceleration (9,8 m/s2 ) and θ is the angle formed between the
gound and the catapult arm which fires, given in degrees. Because the above function is too
complex to be learned in that form, it will be expressed in a approximation form, a linear
combination of the features of the environment in a given state.
We consider the following features of the environment to be relevant for a state:
 x1 – the number of hits player one can take;
 x2 – the number of hits player two can take;
 x3 – the distance between the two catapults (in meters);
 x4 – the intensity of the wind (in m/s) and it’s direction;
 x5 – the initial velocity of the projectile;
35
 x6 – the angle where the projectile in shot from, in degrees.
To calculate the approximation function correctly we assign to each feature a weight represented
by a float number and a bias. These weights are changed throughout the learning making the
function converge to what we want. This being said, the approximatio n function can be
expressed in the following way:
Ṽ(s) = w0 +
wi · xi
where x i represents the values of the features,
w i represent the values of the weights and w0
represents the bias.
In order to learn properly, the agent needs some kind of feedback from a critic.In our case that
feedback is given indirectly by the value of a function called training function. For every state s,
the training function can be expressed in the following way:
Vtrain(s) = Ṽ(Succesor(s))
The Successor(s) function is the state that was right after s in the list of training examples
provided by the game’s performance system. We need just oane more adjustment to make the
algorithm work properly: the way weights are recalculated every turn. For that we have chosen
the LMS(least mean squares) algorithm in which every turn wehave to minimize the value of E,
which represents the error made by the sum of squared differences between the target function
and the training function. The ideal situation, in which the agent has learned the function
perfectly the target function is the same with the training function. In this way, each turn,for
every single training example(made from a state and the associated value) every weight will
recalibrate after the following function:
wi = wi + η· (Vtrain(s) - Ṽ(s))· xi
where
η
is a very small constant called learning rate,and the other values have been explained
before.
36
After repeated simulations the target function should represent more and more correctly the
desired function and finally it will be the same.
As stated before, our learning model was inspired by Tom M. Mitchell’s book „Machine
Learning”[19], and our learning model can be modeled in the same way using the following
components:
Experiment
Generator
Updated target function
New initial state
Performance
System
Generalizer
Training examples
Game History
Critic
Figure 8 - Catapult Learning Model
37
4.3.2 Solution’s design
Use case diagram
The application being a simple leads to a simple use case diagram. The player can start the game,
play the game or exit the game. The characteristics of the game will be detailed in future
sections.
Figure 9 - Use case diagram
Class diagram
Because the application has many classes including the UI, made with the XNA Game Studio 4.0
framework together with the logic of the application which include s the abstract classes
representing the agent architecture as well as the implementation of those classes with some
specific algorithms used for this game we will show only the class diagram for the abstract agent
model to clarify the relationship between the classes.
38
Figure 10 - Class diagram
 The diagrams were made using the CASE instrument StarUML
Description of classes
In this section we will describe every component of the model, and present the way it works and
what is its relationship with other classes. The main classes of the agent architecture are: Play,
Simulation, Player, AI, Action, Environment and State. Further, the machine learning part is
given by the classes Learner, Critic, Generalizer, TargetFunction and PerformanceSystem.
Play – this class is practically the core of the model. It has only one abstract method called
“Start” which takes as parameter an initial state. It also has as class members the two players and
the environment the game takes place. From this class two abstract classes are derived:
Simulation and Game.
39
Simulation – this class is as important as its base class Play because it implements the actual
logic of the game on an abstract level, independent of the implementation of the involved
classes. It also has a function “IsComplete” which checks the environment if the game is over.
In the implementation section we will present a code snippet from the algorithm. It also has two
components that facilitate machine learning: the Learner and History which will be discussed
later.
Player – this class represents the abstract form of a player. It has as a member his number in the
turn-based game. This class is further expanded in two classes: AI and Human. It has only one
method ”SelectAction” which chooses the best action from the current state in order to win the
game.
AI – the representation of an AI player in a game, having all the components required to play it
properly. As its parent class it has the “SelectAction” method but in addition it also has two new
elements which are part of the machine learning system. The classes TargetFunction and
PerformanceSystem are specific to a player and each of them has its role in the process of
learning. Details about how these two classes work will be discussed later in this chapter.
Action – this class represents a basic action an agent can do when is his turn. It has only one
method “Execute”, which applies the effect of the selected action of the player over the current
state.
Environment – this class represents the environment in which the game takes place. Usually it
has a current state and a method to update it. The environment is an important element in the
model because it’s one of the key elements in the interaction process. The State class is a
property of the environment that gives a clear description of a state in the whole process as well
as the parameters needed for evaluation.
Learner – this is the main component of the learning mechanism, consisting of the Critic and
Generalizer and combining the two processes using a history of the game as an input. The
History is created during the simulation and based on it the learner adjusts the target function.
40
Critic – this is a component of the learning part which takes the history of the game and
transforms it into training examples which consist primarily of the state of the game and its value
as a float number or any desired format.
Generalizer – this component takes the training examples made by the Critic and adjusts the
weights using the LMS algorithm described above. After the adjustments are made the Learner
stores them so they can persist.
TargetFunction – this class has only one method and no members. Its job is to calculate the
value of the state according to the method we select in the implementation of it.
PerformanceSystem – this is one of the most important parts of the learning system. It is a
property of the AI class and it chooses the best move to make based on the target function and
other action selection algorithms. Because in the beginning the target function is totally wrong
we need a method to expand the search of the best solutions in all the cases. That’s way we
implement an epsilon- greedy mechanism for the selection of the best moves, the epsilon’s value
increasing along with the performance of the system. For the generation of random numbers we
used an algorithm proposed by George Marsaglia that generates random numbers according to a
Gaussian continuous distribution [28].
41
4.4 Implementation
The implementation of the program is in C# programming language using the XNA Game Studio
4.0 framework and Windows Phone 7 SDK and as an IDE we used Microsoft Visual Studio
2010. In the following section we will present the frameworks above along with their
characteristics.
Windows Phone 7
The Windows Phone Application Platform enables developers to create engaging consumer
experiences running on a Windows® Phone. It is built upon existing Microsoft® tools and
technologies such as Visual Studio, Expression Blend®, Silverlight®, and the XNA Framework.
Developers already familiar with those technologies and their related tools will be able to create
new applications for Windows Phone without a steep learning curve [29].
Runtimes
With Silverlight and the XNA Framework, all development is done in managed code, in a
protected sandbox allowing for the rapid development of safe and secure applications.
Applications written for Silverlight or the XNA Framework today will run on Windows Phone
with only a minor number of adjustments, such as for screen size or for device-specific features.
The two frameworks, along with Windows Phone–specific components and the Common Base
Class Library provide a substantial number of components for developers to construct
applications on [29].
 Silverlight - Silverlight is the ideal framework for creating Rich Internet Applicationstyle user interfaces. A Windows Phone Silverlight application exposes its UI through a
set of pages. Windows Phone controls that match the Windows Phone visual style can be
used to enhance the look and feel of the UI. Visual Studio or Expression Blend can be
used for designing the XAML-based interfaces. Visual Studio can be used to implement
42
the application logic by utilizing the media rich Silverlight libraries or the base
functionality provided by the Common Base Library.
 XNA Framework - The XNA Framework is composed of software, services, and
resources focused on enabling game developers to be successful developing on Microsoft
gaming platforms. Microsoft provides technology that allows professional developers to
quickly enable games on platforms like Windows Phone, Xbox 360, Zune HD, and
Windows 7. The XNA Framework provides a complete set of managed APIs for game
development. This includes 2D sprite-based APIs that support rotation, scaling,
stretching, and filtering as well as 3D graphics APIs for 3D geometry, textures, and
standard lighting and shading.
 Sensors - A variety of sensors will return data that can be consumed by developers. For
example, multi-touch input, accelerometer, compass, gyroscope, and microphone sensors
will all be accessible by APIs.
 Media – Both Silverlight and the XNA Framework pro vide developers with a
programming model for building rich user experiences that incorporate graphics,
animation, and media. The managed APIs support a variety of media formats and allow
for discovery and enumeration of media on the device and for playback of that media.
 Data - Isolated storage allows an application to create and maintain data in a sandboxed
isolated virtual folder. All I/O operations are restricted to isolated storage and do not have
direct access to the underlying operating system file system. This prevents unauthorized
access and data corruption by other applications.

Location - The Microsoft Location Service for Windows Phone allows application
developers to access the user’s physical location information from a single API.
Developers can query for the current location, subscribe to Location Changed events, set
the desired accuracy of the data, get access to the device heading and speed, and calculate
the distance between points. The Location APIs on the phone will work in conjunction
with the Location Cloud Services [29].
XNA Frame work
The XNA Framework class library is a library of classes, interfaces, and value types that are
included in XNA Game Studio. This library provides access to XNA Framework functionality
43
and is designed to be the foundation on which XNA Game Studio applications, components, and
controls are built [30].
Namespaces

Microsoft.Xna.Framework
Provides commonly needed game classes such as timers and game loops.

Microsoft.Xna.Framework.Audio
Contains low- level application programming interface (API) methods that can load and
manipulate XACT-created project and content files to play audio.

Microsoft.Xna.Framework.Content
Contains the run-time components of the Content Pipeline.

Microsoft.Xna.Framework.Design
Provides a unified way of converting types of values to other types.

Microsoft.Xna.Framework.GamerServices
Contains classes that implement various services related to gamers. These services communicate
directly with the gamer, the gamers’ data, or otherwise reflect choices the gamer makes. Gamer
services include input device and profile data APIs.

Microsoft.Xna.Framework.Graphics
Contains low- level application programming interface (API) methods that take advantage of
hardware acceleration capabilities to display 3D objects.

Microsoft.Xna.Framework.Graphics.PackedVector
Represents data types with components that are not multiples of 8 bits.

Microsoft.Xna.Framework.Input
Contains classes to receive input from keyboard, mouse, and Xbox 360 Controller devices.

Microsoft.Xna.Framework.Input.Touch
Contains classes that enable access to touch-based input on devices that support it.

Microsoft.Xna.Framework.Media
Contains classes to enumerate, play, and view songs, albums, playlists, and pictures.

Microsoft.Xna.Framework.Net
Contains classes that implement support for Xbox LIVE, multiplayer, and networking for XNA
Framework games.
44

Microsoft.Xna.Framework.Storage
Contains classes that allow reading and writing of files [30].
Code snippets of important functions
In this section we will present relevant fragments of the main functions of the algorithms.
The Start() function which represents the actual functionality.
public override void Start(State initState)
{
env.SetInitialState(initState);
env.CurrentState().Display();
history.AddToHistory(env.CurrentState());
while (!IsComplete()) {
if (playersTurn)
{
Action action = player.SelectAction(env.CurrentState());
env.UpdateState(player, action);
playersTurn = false;
}
else
{
Action action = opponent.SelectAction(env.CurrentState());
env.UpdateState(opponent, action);
playersTurn = true;
}
env.CurrentState().Display();
history.AddToHistory(env.CurrentState());
}
learner.Learn(history);
Console.WriteLine("END of simulation");
}
In the function above the variable env represents the environment, the variables player and
opponent represent the two players of the game, learner is the component who actually does the
learning, after the game is over (IsComplete() function) based on the history which is a list of
game states.
45
The Learn() function of the Learner component who combines the critic and Generalizer.
public void Learn(History history)
{
List<TrainingExample> examples = new List<TrainingExample>();
examples = critic.CalculateTrainingFunction(history);
State state = generalizer.AdjustWeights(examples);
SaveValues(state);
}
The AdjustWeights() function of the Generalizer component which implements the LMS
algorithm.
public override State AdjustWeights(List<TrainingExample> exemple)
{
float eta = 0.0000001f;
GameState savedFeatures = new GameState();
State sta = exemple[0].State;
if (sta is GameState)
{
savedFeatures = (GameState)sta;
}
foreach (TrainingExample te in exemple)
{
GameState newState = null;
State st = te.State;
if (st is GameState)
{
newState = (GameState)st;
}
float vtrain = te.Vtrain;
float vaprox = agent.TF.CalculateValue(te.State);
foreach (Feature f in newState.Features)
{
Feature newFeature = savedFeatures.getFeatureById(f.FeatureID);
float weight = 0;
weight = newFeature.Weight + eta * (vtrain - vaprox) * f.Value;
newFeature.Weight = weight;
}
}
return savedFeatures;
}
46
The ChooseBestMove() function of the implementation of the PerformanceSystem is the
component which chooses the next move based on the target function and using the epsilongreedy technique to increase the exploring rate.
public override State ChooseBestMove(State s)
{
GameState current = null;
if (s is GameState)
{
current = (GameState)s;
}
currentState = current;
List<GameState> nextStates = new List<GameState>();
nextStates = GeneratePossibleSuccessors(current);
RandomGenerator.SetSeedFromSystemTime();
GameState bestState = new GameState(current);
float bestValue = agent.TF.CalculateValue(nextStates.ElementAt(randomIndex.Next(0,
nextStates.Count-1)));
double minThreshold = 1;
double maxThreshold = 0;
foreach (GameState g in ScrambleArrayList(nextStates))
{
float approxVal = agent.TF.CalculateValue(g);
if (approxVal > bestValue)
{
double treshold = RandomGenerator.GetNormal(0.5, 0.15);
if (treshold < minThreshold)
minThreshold = treshold;
if (treshold > maxThreshold)
maxThreshold = treshold;
if (Math.Abs(treshold) < 1 - epsilon)
{
bestValue = approxVal;
bestState = g;
}
}
}
nextStates.Clear();
return bestState;
}
47
The CalculateValue() function from the implementation of the TargetFunction class which
implements the formula the target function is calculated.
public override float CalculateValue(State s)
{
GameState state = null;
if (s is GameState)
{
state = (GameState)s;
}
float val = 0;
int p1 = state.Features.Where(x => x.Name == "P1 Lives").First().Value;
int p2 = state.Features.Where(x => x.Name == "P2 Lives").First().Value;
if (p1 == 0) { val = -2000; }
else if (p2 == 0) { val = 2000; }
else
{
foreach (Feature f in state.Features)
{
val += f.Value * f.Weight;
}
}
return val;
}
The Execute() function from the Shoot class, and implementation of Action, where the effects
over the environment are made.
public override State Execute(Player p, State s)
{ ..............
int varDist = dist - wind;
float g = 9.8f;
int range = (int)((Math.Pow(velocity, 2) * Math.Sin(2 * (Math.PI * angle / 180.0))) / 9.8f);
if (Math.Abs(varDist - range) < 5)
{
if (p.playerNr == 1)
{
state.Features.Where(x => x.Name == "P2 Lives").First().Value--;
} else if (p.playerNr == 2)
{
state.Features.Where(x => x.Name == "P1 Lives").First().Value--;
}
}
.............
return state;
}
48
4.5 User manual
As the player starts the application the game will automatically start. The screen will look like
the following screenshot, a simple environment with two catapults: blue and red.
Figure 11 - Screenshot from "A Tale of Two Catapults"
To select the angle and velocity swipe freely on the screen of the phone and release when you
want to shoot. After you hit the opponent three times and he hit you three times the game is over.
You can start over by selecting the game again. Anytime during the application you can exit the
game by pressing the “Back” button on the phone.
In the next section we have a flow chart of how the game should work.
49
Figure 12 - Game flowchart
We tried to optimize the game for the best user experience, but there can be improvements in the
future on the UI as well as on the logic of the game
*The chart was made using the online tool Gliffy, www.gliffy.com
50
4.6 Results obtained
In the following section we will present the performance obtained by our machine learning
program using logs from the application. The first chart represents the perce ntage of games won
over many simulations against himself. The second table will represent the evolution of the
weights over the simulations.
Figure 13 - Percentage of games won
As we can see from the graph above there is little difference between the wins and losses, both of
them being kept almost equal. Although the improvements seem insignificant, the small
differences reflect the evolution of the algorithm and because he is playing against himself about
half of the games are won. The ups and downs of the evolution line are in fact the more
significant changes in the weights’ values, changes who affect the entire function.
51
Initial
Value
0.68
Value after
200 games
0.6731037
Value after
400 games
0.6733158
Value after
600 games
0.6732157
Value after
800 games
0.6702238
Value after
1000 games
0.6715883
Player 1
Lives
Player 2
Lives
Distance
0.1
0.1301638
0.1617171
0.1838343
0.2016241
0.2248101
0.1
0.09603488
0.09260094
0.08035062
0.06279583
0.05061561
3
-0.2506298
0.04197381
0.3638703
-0.1142112
0.5817273
Wind
12
6.957904
4.366641
2.584595
1.598882
0.9481826
Velocity
7
3.011064
2.094112
1.595586
0.9499249
0.8309253
Angle
6
1.566059
0.07657859
-0.01758801
-0.05721566
-0.01786953
Name
Bias
We can observe that the values either grow or decrease, and they tend to converge to a very
small number. But in order to get to that number, they need more training, maybe more relevant
features and maybe other action selectors. Nonetheless, this is the proof the learning is a good
mechanism in artificial intelligence, but also that is one that needs perfection and research.
The values used in the chart and table were taken from the files “Log.txt” and “weightLog.txt”
located in the application resources folder.
*The chart was made using the online tool FooPlot, www.foofplot.com
52
4.7 Future work
53
Conclusions
54
Bibliography
[1] Salen, Katie; Zimme rman, Eric (2003). Rules of Play: Game Design Fundamentals.
MIT Press. p. 80. ISBN 0-262-24045-9
[2] Costikyan, Greg (1994). "I Have No Words & I Must Design". Interactive Fantasy, 2
(1994), 3
[3] Abt, Clark C. Serious Games. Viking Press. 1970. p. 6. ISBN 0-670-63490-5
[4] Avedon, Elliot; Sutton-Smith, Brian (1971). The Study of Games. J. Wiley. p. 405. ISBN
0-471-03839-3
[5] Maroney, Kevin (2001). My Entire Waking Life. The Games Journal. May 2001, 1
[6] Epstein, Susan L. Game Playing: The Next Moves. 1999, Sixteenth National Conference
on Artificial Intelligence, 987 - 993. Menlo Park, Calif.: AAAI Press.
[7] Crawford, Chris (2003). Chris Crawford on Game Design. New Riders. ISBN 0-88134117-7.
[8] Kent, Steven L. The Ultimate History of Video Games. Roseville, Calif.: Prima, 2001
[9] Ted Stahl, Video Game Genres, 2005,
http://www.thocp.net/software/games/reference/genres.htm
[10] Robin, AI Search Techniques. December 13th, 2009
http://intelligence.worldofcomputing.net/ai-search/ai-search-techniques.html
[11] Knuth, Donald. The Art of Computer Programming. Volume 3: Sorting and
Searching. ISBN 0-201-89685-0.
[12] Maes, P. 1995. Modeling Adaptive Autonomous Agents. In Artificial Life: An
Overvie w, Ed. C. G. Langton, 135–162. Cambridge, Mass.: MIT Press
[13] Nwana, H. S. 1996. Software Agents: An Overview. Knowledge Engineering Review,
11(3): 205-244.
[14] Russell, S.J., Norvig, P. (1995). Artificial-intelligence: A Modern Approach,
Engle wood Cliffs, NJ: Prentice Hall
[15] Wooldridge, M., Jennings, N.R. (1995). Intelligent Agents: Theory and Practice,
Knowledge Engineering Review
55
[16] Schermer, B.W., Software agents, surveillance, and the right to privacy: A legislative
frame work for agent-enabled surveillance. Leiden University Press, 2007.
[17] Bradshaw, J. (1998). Software Agents, Menlo Park, California: AAArtificialintelligence Press
[18] Franklin, S., Graesser, A. 1996. Is It an Age nt or Just a Program? A Taxonomy for
Autonomous Agents. In Proceedings of the Third Inte rnational Workshop on Agent
Theories Architectures, and Languages. New York: Springer-Verlag.
[19] Mitchell, T. (1997). Machine Learning, McGraw Hill. ISBN 0-07-042807-7
[20] Laird, J. E., & van Lent, M. (2001). Human-level AI’s killer application: Interactive
computer games. AI Magazine, 22(2), 15–26.
[21] Bowling, M., Furnkranz, J., Graepel, T., Musick, R. Machine learning and games,
Mach Learn (2006) 63:211–215 DOI 10.1007/s10994-006-8919-x
[22] Billings, D., Pena, L., Schaeffer, J., & Szafron, D. (2002). The challenge of poker.
Artificial Intelligence, 134(1–2), 201–240, Special Issue on Games, Computers and Artificial
Intelligence
[23] Tesauro, G. (1995). Temporal diffe rence learning and TD-Gammon. Communications
of the ACM, 38(3), 58–68.
[24] Palmer, Nick Machine Learning in Games Development (June 2002) – http://www.aidepot.com/GameAI/Learning.html
[25] Arthur, Samuel (1959-03-03). "Some Studies in Machine Learning Using the Game of
Checkers" (PDF). IBM Journal 3 (3): 210–229. Retrieved 2011-10-31
[26] Thrun, Sebastian Learning to play the game of chess (1995) http://www.robots.stanford.edu/papers/thrun.nips7.neuro -chess.html
[27] Gherrity, Michael (1993). A Game Learning Machine. Ph.D. Thesis, University of
California, San Diego
[28] G. Marsaglia, "Random numbers fall mainly in the planes", Proc. Natl. Acad. Sci.
61(1), 25–28 (1968).
[29] MSDN, Windows Phone, Application Platform Ove rvie w for Windows Phone. March
22, 2012 http://www.ms dn.microsoft.com/en-us/library/ff402531(v=vs.92)
[30] MSDN, XNA Game Studio 4.0, XNA Frame work Class Library.
http://www.ms dn.microsoft.com/en-us/library/bb203940(XNAGameStudio.40).as px
56
Images and diagrams were taken from
1. Figure 1 – OXO http://www.dreamauthentics.blogspot.ro/2011/09/noughts-and-crossesfirst-real.html
2. Figure 2 – Super Mario Bros. http://nintendo-okie.com/tag/new-super-mario-bros-wii/
3. Figure 3 – Assassin’s Creed - http://www.elder-geek.com/2009/09/re-reviewed-assassinscreed/
4. Figure 4 - Franklin and Graesser’s (1996) agent taxonomy - Franklin, S., and Graesser,
A. 1996. Is It an Agent or Just a Program? A Taxonomy for Autonomous Agents. In
Proceedings of the Third International Workshop on Agent Theories Architectures, and
Languages. New York: Springer-Ve rlag.
5. Figure 5 - Typology based on Nwana’s (Nwana 1996) primary attribute dimension Nwana, H. S. 1996. Software Agents: An Overview. Knowledge Engineering Review, 11(3):
205-244
6. Figure 6 – Tom M. Mitchell's Learning Model - Mitchell, T. (1997). Machine Learning,
McGraw Hill. ISBN 0-07-042807-7
7. Figure 7 – Learning architecture proposed by Nick Palme r - Palme r, Nick Machine
Learning in Games Development (June 2002) –http://www.aidepot.com/GameAI/Learning.html
57