Video Game Strategy Learning System using Reinforcement Learning and Inference Rules T H E S I S Master of Science in Intelligent Systems Instituto Tecnológico y de Estudios Superiores de Monterrey By Humberto Javier Quijano García May 2009 Video Game Strategy Learning System using Reinforcement Learning and Inference Rules THESIS Master of Science in Intelligent Systems Instituto Tecnologico y de Estudios Superiores de Monterrey By Humberto Javier Quijano García May 2009 Instituto Tecnológico y de Estudios Superiores de Monterrey Graduate Division in Mechatronics and Information Technology Thesis committee members recommend the present thesis of Humberto Javier Quijano García to be accepted in partial fulfillment of the requirements for the degree of Master of Science in: Intelligent Systems Thesis committee: Leonardo Garrido Luna, Ph.D. Thesis advisor Carlos Astengo Noguez, M.S. Olivia Maricela Barrón Cano, Ph.D. Examiner Examiner Joaquín Acevedo Mascarúa, Ph.D. Chairman of the Graduate Programs Engineer School May de 2009 Video Game Strategy Learning System using Reinforcement Learning and Inference Rules By Humberto Javier Quijano García THESIS Submitted to the Graduate Program in Mechatronics and Information Technology in partial fulfillment of the requirements for the degree of Master of Science in Intelligent Systems Instituto Tecnológico y de Estudios Superiores de Monterrey Campus Monterrey Monterrey, N.L. May 2009 To my parents, Humberto Quijano and Lucila García, whom always have been there giving me their support in the hard times. To my brother Arturo Quijano, who loves video games. Acknowledgments HUMBERTO JAVIER Instituto Tecnológico y de Estudios Superiores de Monterrey May 2009 vi QUIJANO GARCÍA Video Game Strategy Learning System using Reinforcement Learning and Inference Rules Humberto Javier Quijano García, M.S. Instituto Tecnológico y de Estudios Superiores de Monterrey, 2009 Thesis advisor: Leonardo Garrido Luna Contents Acknowledgments vi Abstract vii List of Tables x List of Figures xi List of Algorithms Chapter 1 Introduction 1.1 Problem definition . 1.2 Hypotheses . . . . . 1.3 Objectives . . . . . . 1.4 Scope . . . . . . . . 1.5 Document structure . xii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 2 Background and Related Work 2.1 Repetitive Games . . . . . . . . . . . . . 2.2 Space Search Games . . . . . . . . . . . 2.3 Real Time Strategies Games . . . . . . . 2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1 2 4 6 7 . . . . 8 9 10 10 14 Chapter 3 IRBREL: Inference Rule-Based Reinforcement Learning Algorithm 3.1 IRBREL Algorithm Specifications . . . . . . . . . . . . . . . . . . . . . 3.2 The Knowledge Base . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 World States . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Inference Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 The Learning Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Prior-Knowledge Phase . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Play Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Improvisation Phase . . . . . . . . . . . . . . . . . . . . . . . . viii 16 16 17 18 19 21 22 23 25 3.3.4 Meditation Phase . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chapter 4 The IRBREL Algorithm in Operation 4.1 Missionary-Cannibal Game . . . . . . . . . . . . . . 4.2 Satisfying the Special Requirements of the IRBREL 4.3 Prior-Knowledge Phase in Operation . . . . . . . . 4.4 Play Phase in Operation . . . . . . . . . . . . . . . 4.5 Meditation Phase in Operation . . . . . . . . . . . 4.6 Improvisation Phase in Operation . . . . . . . . . . 4.7 The Learning Cycle Continues . . . . . . . . . . . . 4.8 Summary . . . . . . . . . . . . . . . . . . . . . . . 28 32 . . . . . . . . 34 34 35 36 38 40 42 44 48 Time Strategy Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 50 53 53 56 57 58 59 61 63 64 Chapter 6 Conclusions 6.1 Summary and Principal Results . . . . . . . . . . . . . . . . . . . . . . 6.2 Contributions and Relevance of the IRBREL Algorithm . . . . . . . . . 6.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 68 72 73 Bibliography 76 Vita 79 Chapter 5 Using IRBREL Algorithm in a Real 5.1 Flag’s Capture Game . . . . . . . . . . . . . . 5.2 The Performance Analyzer . . . . . . . . . . . 5.3 Special Considerations . . . . . . . . . . . . . 5.4 Flag’s Capture Experiments . . . . . . . . . . 5.4.1 Brainless Agent . . . . . . . . . . . . . 5.4.2 Dummy Agent . . . . . . . . . . . . . 5.4.3 Kamikaze Agent . . . . . . . . . . . . 5.4.4 Commando Agent . . . . . . . . . . . . 5.4.5 Experiment Conclusions . . . . . . . . 5.5 Summary . . . . . . . . . . . . . . . . . . . . ix . . . . . . Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of Tables 4.1 List of Missionary-Cannibal Actions. . . . . . . . . . . . . . . . . . . . 4.2 List of Missionary-Cannibal Variables. . . . . . . . . . . . . . . . . . . x 35 36 List of Figures 2.1 Learning agent process using Reinforcement Learning.[23] . . . . . . . . 11 3.1 Inference rule. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Learning cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Assignation of the utility to the inference rules. . . . . . . . . . . . . . 19 21 31 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 4.10 4.11 4.12 4.13 Missionary-Cannibal beginning game. . . . . . . . . First possible actions. . . . . . . . . . . . . . . . . . The first inference rules learned. . . . . . . . . . . . Consequences of the first action. . . . . . . . . . . . Consequences of the second action. . . . . . . . . . Inference rule update. . . . . . . . . . . . . . . . . . The new inference rules learned. . . . . . . . . . . . The updated inference rules. . . . . . . . . . . . . . Second possible actions. . . . . . . . . . . . . . . . Consequences of the improvisation. . . . . . . . . . Learned inferences rules after improvisation phase. . World states with the same utility. . . . . . . . . . Assignation of the utility to the inference rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 37 37 39 40 40 41 42 43 44 44 45 47 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 Flag’s Capture video game in action. . . . Flag’s Capture video game - start position. Game progress in Brainless AI category. . Game score in Brainless AI category. . . . Game progress in Dummy AI categoryy. . Game score in Dummy AI category. . . . . Game progress in Kamikaze AI category. . Game score in Kamikaze AI category. . . . Game progress in Commando AI category. Game score in Commando AI category. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 52 58 59 60 60 61 62 63 64 6.1 Humberto Javier Quijano Garcı́a. . . . . . . . . . . . . . . . . . . . . . 79 xi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of Algorithms 3.1 3.2 3.3 3.4 3.5 3.6 Algorithm that gets a list with all the inference rules that fire. Prior-Knowledge Phase Algorithm. . . . . . . . . . . . . . . . Play Phase Algorithm. . . . . . . . . . . . . . . . . . . . . . . Algorithm that Calculates the Probability to Improvise. . . . . Improvisation Phase Algorithm. . . . . . . . . . . . . . . . . . Meditation Phase Algorithm. . . . . . . . . . . . . . . . . . . xii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 22 24 26 27 29 Chapter 1 Introduction Machine Learning is synonymous of advanced computing. Machine Learning is a growing body of work that exists on the use of such techniques to solve real world problems. The complex and/or ill understood nature of many problem domains, such as data mining or process control, has led to the need for technologies which can adapt to the task they face. Machine Learning has been greeted with a certain amount of caution by video games developers, and until recently, has not been used in any major games releases. Many video game companies are currently looking at the possibility of making video games that can match the player’s ability by altering tactics and strategy, rather than by improving the ability of the opponents using a difficulty system. Even on the toughest difficulty settings of most video games (specially on First Person Shouter games like Doom, released in 1993 ), most players have a routine, which if successful, will mean that they win more often than not. However, they would surely not be so smug if the artificial intelligence (AI) could work out their favorite hiding places, or uncover their winning tactics and adapt to them. This could become a very important feature of future releases, as it would prolong game life considerably. The importance for a video game to learn depends on the video game itself. For some video games, the average player does not appreciate any significant advance if the game learns and the learning effort is a waste of time and money. Nevertheless, video games like Black & White (2001) have not been half as successful if it was not for the creatures that could be taught, through kindness or cruelty, to mimic their master. 1.1 Problem definition There are not any major game releases that use Machine Learning techniques to improve AI performance. Most games train AI by just controlling the frequency of errors in tactical decisions. Then, once the game is released, AI just implements the tactical decision with fewer errors and there is not further learning. On released versions of video games, very few still uses learning techniques, like 1 reinforcement learning[23, 24], to improve the agent’s performance. However, on Real Time Strategies Video Games, the decisions and learning must be made quickly to not negative affect the performance. If an agent takes too long thinking what to do, then another agent could take advantage of that moment of inactivity. This quick decision need is a problem for learning agents in Real Time Strategy Video Games. The learned knowledge could help to use a better strategy. Notwithstanding, the learning and decision process should not take too long. Otherwise, it will hinder the performance of the agent. Artificial Intelligence has helped to create automatic systems that find a specific problem solution. Specialized techniques are used commonly to find a solution making world state space searches or using probabilistic and logical techniques. Nonetheless, there are still a lot of limitations. A particular intelligent system made to solve a specific problem could not work if it is applied to another similar problem. Besides, intelligent systems made to solve a lot of kind of problems have not the same efficiency than if the intelligent system was made to solve just a particular problem. This leads to the no free lunch problem. Consequently, if the intelligent system is made to manage learning in all kind of video games, then the efficiency is smaller than if the intelligent system was made to manage learning in just one video game. One of the main capabilities of the human beings is they capacity to improvise in unexpected situations. Besides, the improvisation is a factor that helps a lot in Real Time Strategy Video Games. Nevertheless, the Artificial Intelligence Systems are not good making improvisations. 1.2 Hypotheses There are three hypotheses in this research. The first one refers to the way the knowledge could be represented. The second refers to the representatively and usefulness of that way to represent knowledge. Finally, the third hypothesis refers to the way that the represented knowledge lets to take quick decisions. The cognition, knowledge and memory of the human beings are not located in specific places of the brain crust. All that knowledge is represented by neurons through cortical networks, which distribute and interconnect neurons among them via synapses[13, 7]. These groups of neurons (cortical network) receive, individually, impulses to fire or not fire. Once all the neurons of the network fires then it means the representation of some knowledge. Some neurons have more weight than others. Besides some cortical networks behave as flags, which make some cortical networks fire or not fire. There is a branch of AI which tries to emulate that behavior which is named Neural Networks[26]. Neural Networks are supposed to work as cortical networks once they are completely trained. 2 The human brain does not behave as Neural Networks. The human brain has the capability to represent knowledge and use that knowledge, even if the knowledge represented is not true or not all true. For the human brain, contrary to Neural Networks, there is not need to train the entire cortical network, just is needed to add a new representation of the knowledge. The human brain creates, modify or forget knowledge by experience, observation and reinforcement learning. The brain updates all the knowledge learned during the day while the person is sleeping. This process consists in convert short time memory in long time memory[2, 3].This process makes the brain to remember and updates relevant knowledge meanwhile irrelevant knowledge is forgotten. The hypothesis is that it is possible to emulate the behavior of the human brain learning process using a different approach than the Neural Networks. This proposed different approach uses variables and inference rules to emulate the cortical networks. The neurons are represented by variables which will represent a stimulation, sensor or perception in an agent. For example, a variable could represent if is raining or if the agent is on the wild, etc. The cortical network is represented by an inference rule. The inference rule could be used to represent any kind of knowledge. For the purpose of this research, the inference rules will be used to represent strategies, consequences of the chosen actions and the conditions in which those strategies are useful and valid. For example, an inference could say something like this: if the weather is raining and the agent is on the wild then the agent will get wet. The knowledge represented by the inference rules could be true or valid in some cases and false or invalid in other cases. However, the idea is that inference rules begin very naive and general. Then, inaccurate inference rules could be updated using the learned knowledge. For example, once the last inference rule is found to be inaccurate, it could be updated as follows: if is raining, the agent is on the wild and the agent is not in cover, then the agent will get wet. The human beings have a similar learning process to learn cause-effect events. They begin with a very general knowledge and as needed that knowledge gets more specific or general. This could lead to learn some superstitions, in which cause-effect events are attributed to wrong factors. This events happen because, in most cases, there are not observable all the factors that provoke the observed consequences. Examples of these superstitions could be the famous dance to make rain or the cargo cult. The cargo cult[16, 37, 17] is a superstition that happened during the Second World War. The American soldiers arrived to islands near Australia and created provisional landing platforms to prepare the attack against Japan. Notwithstanding, the islands were inhabited by natives. In order to stay in good mood with the natives, the Americans gave them some food supplies. The natives noticed that the food was brought from the airplanes that landed in the landing platforms. After the war ended, the natives began to build their own landing platforms to invoke the airplanes and get more food 3 supplies. This happened because the natives did not had knowledge of all factors that made the airplanes landed with food supplies. This leads to a second hypothesis: The knowledge represented by a naive inference rule is useful even if the knowledge is not completely accurate or if it is a superstition. It is assumed that after a learning process those superstitions or naive strategies will lead eventually to very useful knowledge and strategies. Then, the system will use that knowledge to decide what to do or what not to do. In Real Time Strategy Video Games, the agents must take decisions almost instantly. If an agent takes too long thinking what to do, then another agent could take advantage of that moment of inactivity. The third hypothesis says that using a system based in inference rules will help to take decisions quickly. The inference rules represent strategies in the game. If an inference rule fires, it means that the strategy is available or it is viable to use in the current circumstances. It is assumed that instead of making a whole world state space search, if the system only takes in account the inference rules that fired and execute the best of them, then the system will save time. 1.3 Objectives The goal of this research is to create a system module for Real Time Strategy Video Games. The system module will learn from its experience and observations using Reinforcement Learning. The system module will be used as the AI of a Real Time Strategy Video Game. The idea is that the video game has agents with an AI that improves with time. The AI will learn from its chosen actions and behave according to the learned knowledge. That way, the human players could not use the same strategies to beat the game. Consequently, the game will force the human players to keep improving their skills. One of the most important capabilities of this system will be its ability to improvise under unknown situations. Besides, it is intended that the system module could adapt and update its knowledge. That way, the next time the system module is under the same circumstances, the system will know what to do or what not to do. In order to create the proposed system, there are five main problems that must be solved: Knowledge Representation: Must be defined how the learned knowledge will be represented. The learned knowledge should be easily accessed and should let the system to choose quickly what to do. The Use of the Learned Knowledge: In Real Time Strategy Video Games there is not much time to take decisions. If the system takes much time thinking what 4 do, other agents could take advantage of this moment of inactivity. Consequently, the system must quickly examine its learned knowledge and decide what to do. Improvisation: The system should improvise to try something new and to try to improve its performance. Nonetheless, if the system improvises all the time, then the agent will execute random actions all the time and he will never achieve anything. The system should balance the improvisation with the use of the its learned knowledge. Besides, the system should decide when is better to use the learned knowledge and when is better to improvise and how improvise. Update Learned Knowledge: The learned knowledge should be updated with the time. If some learned knowledge is no longer useful, then that knowledge should be updated or deleted. Nevertheless, the main problem is to know what knowledge is useful and what knowledge is obsolete. Generate Prior-Knowledge: Some times, there is no previous knowledge of the game. In those circumstances should be decided how generate a prior-knowledge that let the system begins to play the game. To accomplish that, the system will be divided in four phases. Each phase solves a particular problem or need. The first phase is focused in create prior-knowledge to let system begins to play the game. If there is prior-knowledge of the game, then this phase is skipped. The second phase will use all the learned knowledge to quickly decide which is the best action to execute. If at some point the system does not know what to do or the performance is poor, then the third phase will handle all improvisations. This phase, will decide how and when improvise. Finally, the fourth phase will update all the learned knowledge. In order to have a knowledge representation that can be quickly accessed by the system and let it to take quick decisions, the knowledge will be represented using inference rules. The inference rules will represent strategies or actions that could be executed in certain conditions. Besides, the inference rules represent how the world changed after these actions were executed. If the prerequisites in which the strategy could be used are meet, then the inference rule will fire. So, instead of having a system that analyses its learned knowledge and then decide what to do, the system should only pay attention to the inference rules that fired. Then, from the inference rules that fired, it is used the one that gives the system the best performance. To create and update the inference rules, the system needs an additional kind of knowledge, which is the world states. The world states will represent a picture of how the game is at that particular instant and how good or bad is for the agent that particular world state. Besides, the system will need to know what actions are available to its use. 5 Using the world states and the available actions, the system could create inference rules that represent how the world state was before the action was executed and how world state will be after the action is executed. The system will learn, modify and forget inference rules using Reinforcement Learning. This research is different from any other previous work and is important to make this research because these several reasons: 1. There are not any major game releases that uses learning. 2. From the few video games that uses learning to improve the AI, mostly only trains the AI if it is in the develop phase and on the final release the AI is static. 3. Most artificial intelligence systems has the capability to learn and use that knowledge. However, there are not intelligence systems that his main capability is the improvisation. 4. The proposed learning method is a variant of the well known Reinforcement Learning. This variant is inspired by the nature of the cortical networks of the human brain. This method combines Reinforcement Learning techniques and the use of inference rules to represent the learned knowledge. 5. The learned knowledge represented by the inference rules will be useful even if the system is not fully trained. This will let the system to take decisions even if are not known the consequences of all possible actions. 6. If the system only focus on the inference rules that fire, the time of respond will be faster than if a system make a search in the whole world states space, which is important for Real Time Strategies Games. 1.4 Scope The scope of this research is to create a system module that is able to learn and improvise in Real Time Strategy Video Games. Besides, the system should be able to decide quickly what to do. The learned knowledge will be represented using inference rules and world states. The world states represent a picture of how is the world in that instant and how good or bad is this world state for the agent. The inference rules represent how useful is to execute a particular action or strategy under certain conditions. The system will need to know how valuable the world states and inference rules are. It is not intended that the system module learns how to evaluate the world states and inference rules. That evaluation should be calculated by an external module or by 6 the video game itself, then the system will use those values to learn how useful are the chosen actions. It is not scope of this research the creation of a representation of the world states. The variables that will be used to define a world state should be defined by an external module or by the video game itself and these variables must have integer values. Notwithstanding, the system will be able to learn how the world states are interconnected among them and how to change from one world state to another one. The system module is intended to work only in deterministic and completely observable environments. That way the system will be able to learn why the world state changed after an action was executed. The scope of this work is to only prove that the proposed learning algorithm works in Real Time Strategy Video Games. The comparison of the efficiency of this learning method against any other Machine Learning techniques, like Neural Networks, Q-Learning, etc, will be a scope of a future work. 1.5 Document structure This chapter introduces a brief view of the Machine Learning Techniques applied to the video game’s industry. Besides, it is described the problems to use Machine Learning techniques in Real Time Strategy Video Games. Furthermore, the chapter describes the ideas and hypotheses used to create the proposed learning system module and the objectives and scopes of this research. The chapter 2 makes a background of what has been done in the video game industry to improve the AI and what is the common technique to make video games that learns from experience to adapt human players. Besides, the chapter makes a short description of what are doing other researches in this area and the differences among this research and their work. In the chapter 3 is described in detail how the learned knowledge is represented and how the learning process works. Besides, the chapter 4 describes step by step, using the Missionary-Cannibal game, how the learning algorithm works. The chapter 5 measures the performance of the learning algorithm in a Real Time Strategy Video Game. Besides, are described special considerations that should be taken in account to improve the performance of the learning algorithm. Finally, chapter 6 describes the contributions made by this research, the future work and possible applications of this research. 7 Chapter 2 Background and Related Work There many ways in which video games could be categorized. This depends on what is taken in account. The following list describes the most commonly used categories: Platform: NES, SNES, Nintendo 64, PlayStation, X Box, PC, etc. Genre: Action, Adventure, RPG, Sports, Simulation, etc. Classification: Everyone, Teens, Adults. Graphical Style: First Person, 3D, Side-Scrolling Perspective, Top-Down Perspective. Machine Learning: Deterministic or Non-Deterministic, Completely Observable or Not Completely Observable, Single Agent or Multi Agent. For the purposes of this research, the video games will be categorized according to the strategies used to win the game. Repetitive Games: These kind of games are very simple. All these kind of games are Zero-Sum games, like the Prisoner’s Dilemma game. If some of these Sum-Zero games are played many times, the agents should take in count what the other agents could do. For example, in Prisoner’s Dilemma game, if an agent always betrays the other agents, then the betrayer agent should know that in future games, the other agents will be less eager to not betray him. Space Search Games: Examples of this kind of games are Chess, Checkers and TicTac-Toe. For these kinds of games, the solution is to make a whole world state space search to look for the best action to execute. Usually this kind of games uses a Min-Max function[26] to maximize the game results. Real Time Strategies Games: This kind of games could be solved using techniques of the other kinds of games. Nonetheless, the agent must take decisions in short 8 periods of time. If the agent takes too long to decide what to do, the other agents could take advantage of its inactivity, like in the simulated RoboCup Soccer’s Game[1]. The proposed learning system module is designed to work only in Real Time Strategy Video Games that are deterministic and completely observable. The learning system is intended to be used in single agent games. Nevertheless, it is possible to use the system in multi-agent games if the agents do not cooperate among them. In the following sections are described how typically is managed the AI in different kinds of video games and the works that has been done at the moment. Since this research is focused in Real Time Strategy Video Games, there will be only a brief description about video games categorized as Repetitive games and Space Search Games. 2.1 Repetitive Games The Minimum Regret is a popular criterion in decision making processes. Regret is defined as the difference between the payoff that could have been achieved, given the nature of the choices, and what was actually achieved. An expert selection rule is said to minimize regret if it yields an average payoff as large as that of any single expert, against any fixed sequence of actions chosen by the opponent. The Minimum Regret uses algorithms to choose, at each stage, an expert from a probability distribution that is related to the payoff accumulated by the expert prior to that stage. Since the experts are compared on a sequence by sequence basis, it makes sense only under the assumption that nature’s choices are independent of the decision maker’s choices. In Repetitive games, the assumption that the opponent’s choices are independent of the player’s choices is not justified. This is because the opponent is likely to base his choices of actions on the past history of the game. Typically in Repetitive games are used approaches that often focus on algorithms that asymptotically achieve an equilibrium of the one-shot game. Few others shift away from that approach to consider learning strategies that attempt to play long-term bestresponse to the opponent’s observed strategy[10]. For some strategies of the opponent in a repetitive game, it is impossible to learn the best response. In those cases, typically is designed a relatively small number of strategies which can be expected to work well in real-life situations. If only a limited number of strategies are available (like scissors, rock and paper game), there are used expert algorithms. The goal of these expert algorithms is to learn from experience how to combine the advice from multiple experts and to make good sequential decisions on-line. 9 2.2 Space Search Games Most AI game researches has focused on games that can be described in a compact form using symbolic representations, such as board games and card games[15]. Besides, most of these games, like Checkers or Chess, have a relative small number of world states. The number of world states is small if we compare them against Real Time Strategy games which have hundreds or even thousands times world states than a game like Checkers. These kind of games are usually solved using a world state space search. The most commonly world state space search algorithm used is the Min-Max function. For games with a lot of world states, there are not used whole world state space searches. Instead of a whole space state, there are used world state space searches only in the “n” neighbor world states. In more complex games, like Chess, there are used additional AI techniques[12]. These techniques consists in learn from the opponent. The system analyses what the other did in similar conditions to predict his next action. Examples of these techniques are the Chinook[27], the Checkers program which became the world champion in 1994, or Deep blue[8], the Chess program that defeated the world champion in 1997. There are also used Reinforcement Learning techniques in multi-agent competitive card games[20]. The Reinforcement Learning techniques are useful for games with a lot of world states, like Chess and Go. In these kind of games are created thousands or millions of small fragments of the board[30], then using Reinforcement Learning is selected the best action to execute. 2.3 Real Time Strategies Games The game industry has a big interest and demand for these kind of games[29, 35]. In video games like Mario Bros (1985), Zelda (1986), Black & White (2001), Doom (1993), the AI or the human players need to take decisions in short periods of time. Those decisions have a big impact in how the game evolves. Many games developers use learning algorithms in their games to create a better AI[28]. However, typically the AI is frozen once the game is released. For some designers of these kind of games is desirable to create a false impression of AI learning. It is commonplace within the video game industry to create cheating AI systems which gives a false impression of learning. An impression of learning can be easily implemented by controlling the frequency of errors in tactical decisions made by the AI. Then, the AI just executes the strategy with the least number of tactical errors. This creates a realistic illusion of an intelligent learning process. Notwithstanding, this cannot be used unless the desired behavior is already known. In other words, this is useless for learning to counter human player strategies. 10 The central process of learning is the adaptation of behavior. This adaptation is intended to improve the performance. Fundamentally, there are two methods of achieving this: • Directly changing behavior by testing modifications to it. • Indirectly by making alterations to certain aspects of behavior based on observations. There are positive and negative sides to each one, but direct adaptation does have the advantage of not limiting behavior, which means that ultimately, a better goal may be achievable. For the purposes of explanation of the typical learning game strategies process[9, 4, 23], will be used the Flag’s Capture game. There are two agents, which are enemies. The goal of the game is to capture the opponent’s flag. They must do this without getting hit by paint-bullets. Figure 2.1: Learning agent process using Reinforcement Learning.[23] 11 As shown in the figure 2.1, a typical learning agent is composed by a four fundamental parts: a learning element, a performance element, a curiosity element or problem generator and a performance analyzer or critic. The learning element is the part of the agent which modifies the agent’s behavior and creates improvements. The performance element is responsible for choosing external actions based on the percepts it has received. To illustrate this, consider that one of the agents is in the woods. He is aware that an enemy agent is nearby. Then, the agent could choose an action like moving behind a tree. This choice of action is made by the performance element. The performance analyzer judges the performance of the agent against some suitable performance measure. This measure could be how close the agent is to get hit by a paint-bullet, or how close is the agent to the rival’s flag. The performance must be judged on the same percepts as those received by the performance element (the state of affairs known to the agent). Once the analysis of performance has been made, the agent must decide whether or not a better performance could be made under the same circumstances. This decision is then passed to the learning element, which decides on the appropriate alteration for future behavior, and modifies the performance element accordingly. The last part of the process makes sure that the agent advances in its learning, and does not merely confine itself to previously observed behavior. This is dealt by the curiosity element, called that way because it searches for a better solution. To achieve an optimal performance, this element will pose new challenges to the agent in an attempt to prevent bad habits developing. To understand the benefits of this, it is needed to go back to the example. The soldier agent is now hiding behind a tree. From his past experience, he knows that he is safe to stay where he is and this would result in an adequate performance. However, the curiosity element kicks in and suggests that he makes a break from his cover and heads to a nearby tree which is closer to the enemy flag. That may result in the agent ultimately get shot, but could also achieve a more desirable goal. It is then up to the performance analyzer and the learning element to consider whether there is a benefit to this change in strategy. This typical style of learning process is known as Reinforcement Learning. This means that the agent can see the result of its actions, but is not told directly what it should have done instead. Consequently, the agent must use trial and error to evaluate its performance and learn from its mistakes. The advantage to this style of learning is that there is no limitation on the behavior. The limit is only restricted by the alterations suggested through the curiosity element. As the learning agent is ultimately part of a game, the agents should not be left to learn for themselves how to play. The agents should be imparted with a fair degree 12 of prior-knowledge about the way to behave. In the case of the Flag’s Capture game, this could include methods for avoiding fire by using cover, which may later be adapted during the learning process. This process is the typical style to create a learning AI, which adapts to the situation for Real Time Strategy Video Games. Nonetheless, in the practice is common to combine other techniques of artificial intelligence to complement the learning. For complex Real Time Strategy Video Games, such as Role Playing Games (RPGs), the number of choices at each turn ranges from hundreds to even thousands, the incorporation of a learning AI is hard. To implement a game AI for complex games, most developers resort to scripts, which are lists of rules that are executed sequentially. Scripts are a common choice in the game industry to implement game AI, because they are understandable, predictable, adaptable to specific circumstances, easy to implement, easily extendable, and usable by non-programmers[36]. Scripts are generally static and tend to be quite long and complex[5], which gives rise to two problems. The first problem is that, due to their complexity, scripts are likely to contain weaknesses[22]. These weaknesses can be exploited by human players to defeat supposedly tough opponents with ease. The second problem is due to their static nature. This static nature can not deal with unforeseen human player tactics. Besides, the static nature does not let to scale the difficulty level to cater novice and experienced human players at the same time. These two problems, which are common to state of the art game AI[6, 31], hamper the entertainment value of games. Very few uses dynamic scripting as learning technique[32, 25]. This technique uses an adaptive rule base for the generation of game AI on the fly. The use of these rules or scripts is similar to the idea of this research to use inference rules. Both represent a tactics or strategies to follow. Nevertheless, scripts differ from the proposed inference rules in this research. The scripts are only instructions of what to do once the preconditions are meet. Meanwhile, the inference rules also represent what will happen if an action is executed under some preconditions. Besides, it is up to the system to choose from all inference rules that fires which action take. Furthermore, scripts tent to be static meanwhile the proposed inference rules try to adapt and learn. There are few game developers who use a different approach to manage the AI and create adaptive games. This approach uses evolutionary computation, neural networks or a combination of both[11, 34, 38, 33]. This approach is called Neuroevolution[21]. Several methods have been developed for evolving neural networks. One particularly appropriate for video games is called Neuroevolution of Augmenting Topologies (NEAT), which was originally developed for learning behavioral strategies. The neural networks control agents that select actions in their output based on sensory inputs. NEAT is unique in that because it begins evolution with a population of small, simple networks and get complex versions of those networks over generations, leading to 13 increasingly sophisticated behaviors. The idea of this research works in a similar way that NEAT does. Instead of a small population of neural networks, the system begins with a small population of inference rules. These inference rules represent a generalized knowledge and strategies. However, the advantage against NEAT is that, with the proposed learning method, the system should be able to learn even if there is not any inference rule in the knowledge base. If there is no previous knowledge represented by the inference rules, the system will be able to create a general knowledge via trial and error. There is no commonly used, but there are few researches that use Vector Machine techniques [18, 14, 19] in video games. Vector Machine techniques are used to classify the data and decisions that the agents could take. It is more common to find Vector Machine and Bayesian Networks techniques in the RoboCup Soccer’s simulation game. 2.4 Summary There many ways in which video games could be categorized. This depends on what is taken in account. For the purposes of this research, the video games will be categorized according to the strategies used to win the games. Repetitive Games: This kind of games are very simple. All these kind of games are Zero-Sum games, like the Prisoner’s Dilemma game. If some of these Sum-Zero games are played many times, the agents should take in count what the other agents could do. For example, in Prisoner’s Dilemma game, if an agent always betrays the other agents, then the betrayer agent should know that in future games, the other agents will be less eager to not betray him. Space Search Games: Examples of this kind of games are Chess, Checkers and TicTac-Toe. For these kinds of games, the solution is to make a whole world state space search to look for the best action to execute. Usually this kind of games uses a Min-Max function to maximize the game results. Real Time Strategies Games: This kind of games could be solved using techniques of the other kinds of games. Notwithstanding, the agent must take decisions in short periods of time. If the agent takes too long to decide what to do, then the other agents could take advantage of its inactivity. The proposed learning system module is designed to work only in Real Time Strategy Video Games that are deterministic and completely observable. The learning system is intended to be used in single agent games. Nonetheless, it is possible to use the system in multi-agent games if the agents do not cooperate among them. 14 A typical learning agent is composed by a four fundamental parts: a learning element, a performance element, a curiosity element or problem generator and a performance analyzer or critic. The learning element is the part of the agent that modifies the agent’s behavior and creates improvements. The performance element is responsible for choosing external actions based on the percepts it has received. The performance analyzer judges the performance of the agent against some suitable performance measure. Once the analysis of performance has been made, the agent must decide whether or not a better performance could be made under the same circumstances. The curiosity element poses new challenges to the agent in an attempt to prevent the development of bad habits. This typical style of learning process is known as Reinforcement Learning. This means that agent can see the result of its actions, but is not told directly what it should have done instead. Consequently, the agent must use trial and error, to evaluate its performance and learn from its mistakes. This process is the typical style to create a learning AI, which adapts to the situation for Real Time Strategy Video Games. Nevertheless, in the practice is common to combine other techniques of artificial intelligence to complement the learning, like scripts, evolutionary computation, neural networks, etc. 15 Chapter 3 IRBREL: Inference Rule-Based Reinforcement Learning Algorithm The Inference Rule-Based Reinforcement Learning algorithm or “IRBREL” algorithm uses the same commonly used principles of reinforcement learning, which were described in the chapter 2. The learning process is very similar to one shown in the figure 2.1. However, instead of the common structure used in Reinforcement Learning algorithms, the IRBREL algorithm uses inference rules to represent strategies and to model the world. The inference rules are created and modified using the learned world states and the utility to be in that world state. The learning process of the IRBREL algorithm works in cycles of four phases. In the following sections is described in detail how the knowledge is represented and how the learning cycle works. Besides, in the chapter 4 is explained, using an example, how the IRBREL algorithm works. 3.1 IRBREL Algorithm Specifications The IRBREL algorithm is designed as a module, which can be adapted to be used in Real Time Strategy Video Games. Notwithstanding, in order to work properly, there are some special requirements that must be satisfied. The IRBREL algorithm needs a module that generates the world states of the video game. The world states are used by the IRBREL algorithm to know the consequences of the chosen actions and to generate the inference rules. The world state is defined by a group of variables, which describes how the world of the video game is at that moment. A world state can be represented using any number of variables. The variables can have any value from negative infinite to positive infinite. The only restriction is that the values must be integers. The developers of the world state generator have complete liberty of using any number of variables and value ranges for those variables. A world state could be represented using a single variable with thousands of possible values or thousands of variables with few possible values. Nonetheless, before design how a world state is 16 represented, it is important to take in account that the more representative is a variable, the more easily will for the IRBREL algorithm to learn. The IRBREL algorithm also needs a performance analyzer module. The purpose of the performance analyzer module is to calculate the utility of the agent to be in the current world state. The IRBREL algorithm will choose actions that better moves the agent from the current world state to the best of the known world states. The world states that are between the current world state and the best of the known world states should also have utility values between those two values. The video game world is assumed to be deterministic, which means that the changes in the world state are only determined by the actions executed by agents. Changes made by environmental factors, like day and night, should be constant. Besides, the world must be also deterministic to agents’ perspective. Consequently, the world must be completely observable or observable enough to let the agent identify why the world state changed. The IRBREL algorithm needs to know what actions are available to its use. Every time the IRBREL algorithm is initialized, the user sends to the IRBREL algorithm a list containing the available actions. For each unit of time of the video game, The IRBREL algorithm asks the video game for the current world state. Then the IRBREL algorithm chooses one of the actions of the list that the user sent when initialized the IRBREL algorithm. Once the action is chosen, the IRBREL algorithm tells the agent to execute that action. Finally, at the beginning of the next unit of time the IRBREL algorithm analyzes the consequences of the taken action and chooses a new one. Some actions are defined as complex actions. Simple actions are instantaneous; only require a single unit of time to execute. Consequently, their consequences can be analyzed at the next unit of time. Nevertheless, complex actions are not instantaneous and require several units of time to accomplish. The IRBREL algorithm can manage complex actions but the user must tell to the algorithm, using an internal flag, if there is a complex action still in execution. That way the IRBREL algorithm is able to know when analyze the consequences of the chosen action. Besides, the IRBREL algorithm is able to interrupt complex actions if sees that are not getting good results or if the conditions of the game changed. Section 3.3 and chapter 5 explain how the complex actions are handled. 3.2 The Knowledge Base The knowledge base is composed by four elements: World States: Each world state represents a picture of how is the world in that instant. Furthermore, each world state has an utility value, which is the utility for the agent to be in that world state. 17 Inference Rules: Represent actions or strategies which can be done under some conditions. Besides, represents how the world is modified by executing those actions and how useful are those world modifications. Original World State: This is a world state that is linked to a specific inference rule. There is one of this world states for each inference rule in the knowledge base. The inference rules could be used to represent how the world states are interconnected among them. Using that perspective, the premises represents the original world state and the conclusions represents the new world state after executing the described action. Inference rules are created very general (with only one premise). These inference rules could become more specific as needed. However, sometimes it is needed to know the original world state. If the inference rule is too general, then the premises do not represent the original world state and is needed this additional linked world state to know that information. Know All Consequences Flag: This is a Boolean flag that is linked to a specific inference rule. There is one of this Boolean flag for each inference rule in the knowledge base. This flag is turned on if there are known the consequences of executing every possible action under the same conditions of the linked inference rule. This lets the IRBREL algorithm to know that there is no need to improvise because all possible results are known. In the following subsections are explained in more detail the use and representation of world states and inference rules. 3.2.1 World States World states represent a picture of how is the world in that instant. Furthermore, each world state has an utility value, which is the utility for the agent to be in that world state. A world state is divided in two parts. The first represents the value of all variables of the world in that instant. The second is just an utility value, which represents the utility of the agent to be in that world state. That value must be generated by the Performance Analyzer element (figure 2.1) as described by section 3.1. The IRBREL algorithm learns each world state where the agent has been. The world states are used to learn how the world is affected by the taken actions. Using that knowledge, the IRBREL algorithm creates inference rules which represent those changes. The IRBREL algorithm uses world states to check if the used inference rules are valid or invalid. Besides, uses the world states values to modify or update invalid inference rules. Furthermore, the IRBREL algorithm keeps track of the current world 18 state to know if the agent gets stuck or is not making any progress at all, which leads the IRBREL algorithm to improvise and try something new. This process is explained in detail in section 3.3. 3.2.2 Inference Rules Inference rules represent actions or strategies which can be done under some conditions. Besides, represents how the world is modified by executing those actions and how useful are those world modifications to the agent. Figure 3.1: Inference rule. As shown in the figure 3.1, a inference rule has three elements: Premises: These are composed by a single action and some variables with their respective values. The simplest and general inference rule is composed only by one action. Premises represents the circumstances in which could be used the strategy represented by the inference rule. Conclusions: These are only composed by variables and represent how the world state will be after the described action is executed. Utility: This is a numeric value which represents the utility given to the agent if executes the action under the circumstances expressed by the variables of the premises. There is also taken in account the number of times the inference rule was used and was valid or invalid. Figure 3.1 shows how a typical inference rule looks. The value of each variable is represented by the variables inside the square brackets. Usually the inference rules are used for logical reasoning in Boolean operations. Consequently, the variables could only take binary values. Notwithstanding, the IRBREL algorithm uses inference rules which variables could take any value. The only restriction to these values is that the range of values, which a variable can take, must be discrete (see section 3.1 for more details). The premises represent in which circumstances is valid to execute the strategy described in the inference rule. Executing the described action or strategy under the conditions specified by the premises will cause the conditions expected by the conclusions. 19 The variables of the conclusions section represent the new values that the variables will have after executed the specified action. Some variables could have the same values than the previous world state. In some cases the consequences of the chosen actions will be different from the consequences expected or defined by the conclusions section. In a deterministic world, this only happens because inference rule was too general and the IRBREL algorithm assumed that the results will work for all the cases and applied the inference rule when it should not do it. For these cases, the inference rule is considered invalid and is modified to consider additional conditions. The process of how an inference rule is modified and updated is described in section 3.3. In order to choose which action or strategy is available to execute, the IRBREL algorithm only takes in account the inference rules that fires. An inference rule fires only if the strategy is available to execute under the current circumstances. An inference rule fires if its variables and values, specified by the premises, match with the current world state. Most inference rules in the knowledge base are general, which means that not every variable is taken in account in the premises. For those cases, if a variable does not appear in the premise section, then that variable is not taken in account to consider if the inference rule fires or does not fire. The algorithm 3.1 shows the process to get a list with all inference rules that fire. Algorithm 3.1 Algorithm that gets a list with all the inference rules that fire. list ← newList() inf erences ←Number of Inference Rules in the KB for i = 1 to inf erences do if PremisesMatch(InferenceRule[i], currentWorldState) then list.Add(Inf erenceRule[i]) end if end for return list Once a game is running, some inference rules fire and others no fire, which depends on the values of the world state. All the inference rules that fire, represents the possible actions or strategies that the agent can follow. From all the inference rules that fire, the IRBREL algorithm chooses the one which give the agent the biggest utility. Both, world states and inference rules have utility values. These values have a strict relation but could not be the same. The utility of the world state represents the utility of the agent to be in that state. Meanwhile, the utility of the inference rule represents the utility of executing that action. The agent only uses the utility of the inference rule to take decisions. Nonetheless, the utility of the world states is used to create, or modify utility values of the inference rules. 20 3.3 The Learning Cycle The IRBREL algorithm works basically in cycles of three phases. Nevertheless, for some cases is necessary to add an additional phase to help the agent to begin his learning. In figure 3.2 could be appreciate a diagram of how this learning algorithm works. Figure 3.2: Learning cycle. In the following subsections is described how each learning phase works. Each phase has a very specific task. Once the task is accomplished the IRBREL algorithm chooses which phase execute next. The main goal of the learning cycle is to learn enough strategies to always win the game. Inference rules represents the strategies to win the game and the conditions in which these strategies work. At the beginning of the learning process all inference rules are very simple and general. Throw the learning cycles, the inferences rules get more specific and complex as needed. If the game or the agents are constant, then will be a point where the IRBREL algorithm will learn enough strategies to always win the game. However, if the game or the agents change with the time, then the learning process will never end to the IRBREL algorithm. It is important to consider that the game or the agents can change its behavior, but the world must still be deterministic. 21 3.3.1 Prior-Knowledge Phase As the learning agent is the ultimately part of a game, usually it is not left alone to figure out for itself how to play. Before any learning process begins, agents already should know some basic strategies to help them to survive. Then, the agent after some observations and experience modifies these basic strategies. Notwithstanding, in some cases there is not any prior-knowledge and the agent must begin without any knowledge. Algorithm 3.2 Prior-Knowledge Phase Algorithm. oldW orldState ← GetCurrentW orldState() newW orldState ← null newInf erenceRule ← null availableActions ←Number of Available Actions (list given by the user) for i = 1 to availableActions do ExecuteAction(i) W aitUntilActionEnds() newW orldState ← GetCurrentW orldState() if WorldStateChanged(oldW orldState, newW orldState) then newInf erenceRule ← new InferenceRule(i, newW orldState) Inf erenceRule.Add(newInf erenceRule) end if RestartGame() end for P layP hase() At this point, the knowledge base is empty. The IRBREL algorithm has no idea of what will happen if he executes certain action. The goal of the Prior-Knowledge phase is to learn enough basic strategies to let the agent begin to play the game. Consequently, the IRBREL algorithm does the only possible solution, experimenting with the actions and world states. As shown by algorithm 3.2, the IRBREL algorithm executes every action one time and checks if the world state changed. This process is similar to any person that plays a video game that he never has seen before. The person begins pressing all buttons to figure out what happened and how the game works. The IRBREL algorithm uses the same strategy. With that information, the IRBREL algorithm creates naive inference rules, which represents the observed knowledge. At the beginning, each new inference rule created assumes that the consequences only depend on the chosen action. These naive inference rules could be not completely accurate. Perhaps some of these learning strategies only work in a very specific situation. After some learning cycles, the inference rules become more specific as needed. For example, imagine that the IRBREL algorithm represents a person in the wild. 22 If the IRBREL algorithm makes the agent dance and observes that a rain begins, then could be assumed to the IRBREL algorithm that every time the agent dances a rain will come. Nonetheless, the rain is independent of the dance, it was just a coincidence. The conclusion that the event was coincidence will be deduced after some learning cycles. Consequently, the assumption that dance will provoke rain will be eventually discarded by the IRBREL algorithm. Generalized inference rules are very good to save memory space. That way, instead of having a knowledge base with several inference rules for each world state, the IRBRREL algorithm has several inference rules in the knowledge base that applies to several world states. There is no need to learn exactly all the conditions of the world if some elements are irrelevant and some conditions can be generalized. This Prior-Knowledge phase is executed only if there are not any inference rules stored in the knowledge base. At the end of this phase, it is executed the Play phase. 3.3.2 Play Phase The goal of the Play phase is to use the learned knowledge to win the game. At this point, the IRBREL algorithm has a knowledge achieved via the Prior-Knowledge phase or via previous learning cycles. The knowledge is represented by inference rules. Each inference rule represents a strategy and each inference rule, (no matter how general is it) is assumed to be completely accurate. Using that knowledge, the IRBREL algorithm tries to play the game the best he can. As shown by algorithm 3.3, the IRBREL algorithm only considers the inference rules that fire (see algorithm 3.1). If an inference rule fires, means that are satisfied the specific conditions in which the strategy works. From all inferences rules that fire, the IRBREL algorithm chooses the inference rule that has the biggest real utility value. As shown by equation 3.1, the real utility value is calculated using the utility value of the inference rule and the percentage of times the inference rule is valid. That way most frequently used and valid inference rules become more valuable than invalid and no too frequently used inference rules. valid × utility (3.1) valid + invalid After an inference rule is chosen, the IRBREL algorithm tells the agent (that the IRBREL algorithm is controlling) the action to execute. Then agent of the video game should tell the IRBREL algorithm if the chosen action is a complex action. Besides, the agent of the video game should inform to the IRBREL algorithm if the complex action ended. For more details see section 3.1. If a complex action is running and the performance (utility value) gets worse from one unit of time to another, then the IRBREL algorithm tells the agent of the video RealUtility = 23 Algorithm 3.3 Play Phase Algorithm. if GameEnded() then MeditationP hase() else inf erenceRulesF ired ← newList() oldUtility ← GetCurrentUtility() newUtility ← 0 bestUtility ← −∞ pointer ← 1 inf erenceRulesF ired ← GetFiredInferenceRules() worldState ← null counter ← 0 if StuckInT heSameW orldState(GetCurrentW orldState()) then ImprovistationP hase() end if if inf erenceRulesF ired.Count = 0 then ImprovistationP hase() end if for i = 1 to inferenceRulesFired.Count do if inf erenceRulesF ired[i].Utility > bestUtility then bestUtility ← inferenceRulesFired[i].Utility pointer ← i end if end for ExecuteAction(inf erenceRulesF ired[pointer].Action) while ActionRunning() do newUtility ← GetCurrentUtility() if oldUtility > newUtility then AbortAction() ImprovistationP hase() else oldUtility ← newUtility end if end while worldState ← GetCurrentWorldState() if worldState 6= inf erenceRulesF ired[pointer].Conclusions then counter ← inf erenceRulesF ired[pointer].Invalid + 1 inf erenceRulesF ired[pointer].Invalid ← counter MeditationP hase() else counter ← inf erenceRulesF ired[pointer].V alid + 1 inf erenceRulesF ired[pointer].V alid ← counter P layP hase() end if end if 24 game to abort the current action. The IRBREL algorithm only chooses actions that make the agent get a better world state or at least get into a world state with the same utility value than the current one. If the utility value gets worse, means that the circumstances of the game changed and it is better to choose a different strategy. Consequently, if this happens, then the Play phase is over and the IRBREL algorithm begins running the Improvisation phase. In the Improvisation phase, the IRBREL algorithm will chose to execute a different action from his known strategies or will improvise and try something different. Each time an actions ends (the interrupted actions are not taken in account), The IRBREL algorithm compares the current world state with the expected world state defined by the inference rule. If both world states match, then the inference rule is valid. If the world states are different, then the inference rule is invalid. Each inference rule has a counter for the number of times that the inference rule was used and was valid and number of times that inference rule was invalid. These values are updated each time an inference rule is used. Besides, these values are used to calculate the real utility described by equation 3.1. In some cases, the IRBREL algorithm will discover that some of his actions provoke things that were not expected. This occurs because some inference rules are too general and are not considering that a certain condition must be satisfied before using this strategy. Besides, this also could happen because the chosen action has no relation with the changes of the world state. Returning to the previous example of subsection 3.3.1, the IRBREL algorithm thinks that dancing will provoke rain and the agent needs water. Consequently, the IRBREL algorithm makes the agent to dance. Nevertheless, this time the IRBREL algorithm observes that is not raining. This kind of inconsistence is noted by the IRBREL algorithm and updates the number of times that the inference rule was invalid. Invalid inference rules are not very valuable to the IRBREL algorithm. Consequently, each time this happens, the IRBREL algorithm executes the Meditation phase to update the invalid inference rule. If the IRBREL algorithm gets into a world state where are not any inference rule that fires or after executing several actions the world state is the same, then IRBREL algorithm executes the Improvisation phase to try to make get a better world state. Once the IRBREL achieves to his desired final world state or the game is ended for any reason, like winning or losing the game, the IRBREL algorithm executes the Meditation phase to update the inference rules. 3.3.3 Improvisation Phase The goal of the Improvisation phase is to try something new and learn new strategies. That way the IRBREL algorithm has more strategies to choose when executes 25 the Play phase. The Improvisation phase is executed if an action is interrupted, there were not any available strategies to execute or the IRBREL algorithm got stuck in the same world state. Algorithm 3.4 Algorithm that Calculates the Probability to Improvise. inf erenceRulesF ired ← GetFiredInferenceRules() availableActions ←Number of Available Actions (list given by the user) knownActions ← newList() probability ← 0 for i = 1 to inferenceRulesFired.Count do knownActions.AddIF NewElement(inf erenceRulesF ired.Action) end for × 100 probability ← availableActions−knownActions.Count availableActions RETURN probability In either case, as shown in algorithm 3.5, the IRBREL algorithm finds the better way to improvise. First, the IRBREL algorithm calculates the probability to completely improvise (see algorithm 3.4). The IRBREL algorithm checks which inference rules fires with the current world state. Then checks which action is available. The IRBREL algorithm knows how many actions are available to its use, that information is given by the user when initialized the IRBREL algorithm (see section 3.1 for more details). If all actions are available, then there is no need to improvise because the IRBREL algorithm knows the consequences of every action for the current circumstances. If the IRBREL algorithm only knows the consequences of the half of the total actions, then there is a 50% of chances to improvise. Once the IRBREL algorithm knows the probability to improvise, the IRBREL algorithm analyzes the inference rules that fired. One element of the knowledge base is the Know All Consequences flag or KAC flag. As described in section 3.2, this element (linked to each inference rule) represents that the linked inference rule was used in a situation in which the consequences of all possible actions are known. If the probability to improvise is zero, then the KAC flag is turned on for every inference rule that fired. However, if the probability to improvise is greater than zero but one of the inference rules that fired has the KAC flag turned on, then the IRBREL algorithm does not improvise and uses one of these inference rules. If the probability to improvise is greater than zero and all the inference rules that fired have not the KAC flag on, then the IRBREL algorithm makes a random check to decide if improvises or if uses one of the known strategies. If the IRBREL algorithm decides to improvise, then randomly chooses to execute one of the actions which consequences are not known for the current world state. If the executed action changed the world state, then the IRBREL algorithm creates a naive inference rule like the ones created by the Prior-Knowledge phase. After execute the 26 Algorithm 3.5 Improvisation Phase Algorithm. inf erenceRulesF ired ← GetFiredInferenceRules() improviseP robability ← GetImproviseProbability() randomNumber ← GetRandomNumber(1, 101) randomAction ← 1 availableActions ←Number of Available Actions (list given by the user) knownActions ← newList() notKnownActions ← newList() oldW orldState ← GetCurrentW orldState() newW orldState ← null newInf erenceRule ← null inf erenceRulesF ired ← GetFiredInferenceRules() for i = 1 to inferenceRulesFired.Count do knownActions.AddIF NewElement(inf erenceRulesF ired.Action) end for if randomNumber = 101 then randomAction ← GetRandomNumber(1, availableActions) ExecuteAction(randomAction) W aitUntilActionEnds() newW orldState ← GetCurrentW orldState() if WorldStateChanged(oldW orldState, newW orldState) then newInf erenceRule ← new InferenceRule(randomAction, newW orldState) Inf erenceRule.AddIF NewElement(newInf erenceRule) end if P layP hase() else if randomNumber > improviseP robability then notKnownActions ← GetMissingActions(availableActions, knownActions) randomAction ← GetRandomNumber(1, notKnownActions.Count) randomAction ← notKnownActions[randomAction] ExecuteAction(randomAction) W aitUntilActionEnds() newW orldState ← GetCurrentW orldState() if WorldStateChanged(oldW orldState, newW orldState) then newInf erenceRule ← new InferenceRule(randomAction, newW orldState) Inf erenceRule.AddIF NewElement(newInf erenceRule) end if P layP hase() else P layP hase() end if 27 improvised action and checks the consequences of that action, the IRBREL algorithm begins to execute again the Play phase. If the IRBREL algorithm decides to just use the learned strategies, then chooses a strategy with the best real utility value using the same process described in the Play phase. After execute the chosen action and checks the consequences of that action, the IRBREL algorithm begins to execute again the Play phase. Even if the probability to improvise is zero, there is always a very small chance of 1% to improvise. 3.3.4 Meditation Phase The goal of the Meditation phase is to update the inference rules. The invalid inference rules are destroyed and are created more accurate versions of the original ones. Besides, the utility values of the inference rules could be modified in order to improve the performance of the IRBREL algorithm. Every time a human being sleeps occurs an internal process in the brain. The short time memory gets converted into long time memory. Some irrelevant learned knowledge is forgotten. Meanwhile, all relevant learned knowledge is updated and made efficient for future uses. The Meditation phase was inspired in that mental process. The Meditation phase is executed if an inference rule was discovered to be invalid or if the game is over for some reason, like a game lost or won. As shown by algorithm 3.6, the IRBREL algorithm search for all inference rules that are invalid. In the previous phases, the IRBREL algorithm took note of every inference rule that did not behave as it predicted. This could happen for two reasons, the inference rule is too general or the executed action is not the consequence of the changes in the world state. The IRBREL algorithm assumes that every invalid inference rule was invalid because they are too general. If an inference rule is too general, then there are other involved variables not taken in account. For example, the IRBREL algorithm knows that the agent get wet if the agent is in the wild and begins to rain. Consequently, the next time the agent is in the wild and begins to rain, the IRBREL algorithm will assume that the agent will get wet. That knowledge is true if the agent is not under cover of something like an umbrella. If the IRBREL algorithm discovers that this time the agent is not getting wet, then the IRBREL algorithm will figure out that getting wet depends also to at least one additional element, like the umbrella element. The IRBREL algorithm creates new versions of invalid inference rules only if there are not more general versions of that inference rule or if the inference rule has not any sister inference rule. If there are more general versions of the invalid inference rule in the knowledge base, then the invalid inference is just deleted. A new version of an inference rule is the same inference rule but with an additional 28 Algorithm 3.6 Meditation Phase Algorithm. bestW orldState ← null newInf erenceRules ← newList() bestUtility ← −∞ for i = 1 to InferenceRule.Count do if Inf erenceRule[i].Invalid > 0 then newInf erenceRules ← GetUpdatedInf erenceRules(Inf erenceRule[i]) Inf erenceRule.Remove[i] i← i−1 for j = 1 to newInferenceRules.Count do Inf erenceRule.Add(newInf erenceRules[j]) end for end if end for if GameEnded() ∧ enabledUpdate then for i = 1 to WorldStates.Count do if W orldStates[i].Utility > bestUtility then bestUtility ← W orldStates[i].Utility bestW orldState ← W orldStates[i] end if end for UpdateInf erenceRuleUtilities(bestW orldState) SaveAndClose() else P layP hase() end if 29 variable in the premise. The additional variable and its value is obtained from the Original World State element, which was described in section 3.2, that is linked to each inference rule. The Original World State element or OWS element has a picture of how the world state was before the action was executed. Each invalid inference rule creates several new inference rules. The number of inference rules generated depends, as shown by equation 3.2, on the number of variables that the original inference rule has in its premises and the number of variables not taken in account. Consequently, if the original inference rule has one variable in its premises and the world state has five variables, then are created four new inference rules with two variables in its premises. NewInf erenceRules = T otalW orldV ariables−V ariablesInOriginalP remise (3.2) The IRBREL algorithm has no way to know which variable should take in account. That is why the IRBREL algorithm creates several inference rules each one considering a different variable. These new inference rules are considered sisters. So, if one of these sisters inference rules gets invalid, there will not be generated new versions of itself until all its sisters are deleted. An invalid inference rule with sisters does not generate new inference rules; this invalid inference rule is just deleted. In this way the IRBREL algorithm eventually discards inference rules with premise variables which are not important. A totally specific inference rule is an inference rule which in its premises has all the variables of the world state. If a totally specific inference rule gets invalid, then it means that the executed action did not cause the changes in the world state, like in the example of the agent that dances to make rain. Totally specific inference rules that gets invalid are just deleted from the knowledge base. The human brain has the capability to specify or generalize the conditions in which the learned strategies are valid. Notwithstanding, the IRBREL algorithm only has the capability to specify the conditions in which the learned strategies are valid. The only way to the IRBREL algorithm to learn generalized versions of the learned strategies is throw the 1% improvisation factor which was described in subsection 3.3.3. Once every invalid inference rule is deleted and/or generates new versions of itself, then the IRBREL algorithm checks if the game is still running. If the game is still running, then it is executed again the Play phase. Nonetheless, if the game is ended for any reason (like a game won or lost), then the IRBREL algorithm updates the utility of every inference. The IRBREL algorithm uses that period of game inactivity to update the learned knowledge. In the same way, the human brain converts short time memory in long time memory while the person is sleeping. The process to update the utility values of the inference rules is not strictly necessary to execute. The user has the option to enable or disable this option when the 30 IRBREL algorithm is initialized. The process to update the utility values of the inference rules is useful if the utility values assigned to the world states do not exactly reflect the real utility of the agent to be in that world state. The chapter 4 has an example of a case where the use of this process is necessary. Nevertheless, if the utility values of the world states are well managed, then the user could disable this step. Figure 3.3: Assignation of the utility to the inference rules. In this step, the IRBREL algorithm finds the world state with the best utility value. Using that world state, the IRBREL algorithm assigns that utility value to all inference rules that are supposed to move the agent to that world state. After that, the IRBREL algorithm finds the neighbor world states of the world state with the best utility value. The neighbor states are found using the inference rules. The conclusion section of an inference rule represents the world state where the agent will be after executing the specified action. The OWS element linked to each inference rule represents the original world state before the action was executed. Consequently, each inference rule represents how the world states are interconnected among them. Using that information could be 31 created a map with all world states connections. The utility value of the best world state is decremented by a delta value. This new utility value is assigned to all inference rules that connect the neighbor world states with the world state with the best utility value. The process can be exemplified by the figure 3.3. In that figure, the world states are represented by boxes and the inference rules by red arrows. The inference rules connect the world states. The inference rules represents how can be changed from one world state to another using a particular action. The process of finding neighbor world states and propagate the last assigned utility value minus a delta continues until all inference rules have been revised. 3.4 3.4 Summary The Inference Rule-Based Reinforcement Learning algorithm or “IRBREL” algorithm uses the same commonly used principles of reinforcement learning. However, instead of the common structure used in Reinforcement Learning algorithms, the IRBREL algorithm uses inference rules to represent strategies and to model the world. The IRBREL algorithm is designed as a module, which can be adapted to be used in Real Time Strategy Video Games. Notwithstanding, in order to work properly, there are five special requirements that must be satisfied. World State Generator Module: A module that generates the world states of the video game. Performance Analyzer Module: Module that calculates the utility of the agent to be in the current world state. Deterministic Game: The changes in the world state must be only determined by the actions executed by agents. Available Actions: The IRBREL algorithm needs to know what actions are available to its use. Feedback for Complex Actions: The agent must inform to the IRBREL algorithm, using an internal flag, if there is a complex action still in execution In order to represent the learned knowledge, the IRBREL algorithm uses a knowledge base composed by four elements: World States: Each world state represents a picture of how is the world in that instant. Furthermore, each world state has an utility value, which is the utility for the agent to be in that world state. 32 Inference Rules: Represent actions or strategies which can be done under some conditions. Besides, represents how the world is modified by executing those actions and how useful are those world modifications. Original World State: This is a world state that is linked to a specific inference rule. This world state represents the original world state before the action described in the inference rule is executed. Know All Consequences Flag: This is a Boolean flag that is linked to a specific inference rule. This flag is turned on if there are known the consequences of executing every possible action under the same conditions of the linked inference rule. The IRBREL algorithm learns in cycles of four phases. Each phase is executed as needed. The Prior-Knowledge is used if there is no previous knowledge of the game. In this phase the IRBREL algorithm learns using a trial and error strategy. The Play phase uses all the learned knowledge to find the best strategy to win the game. If at some point the IRBREL algorithm finds something unexpected, then, in the Improvisation phase, the IRBREL algorithm looks for the best way to improvise and try something new. Finally, in the Meditation phase, the IRBREL algorithm updates the learned knowledge to make it accurate. 33 Chapter 4 The IRBREL Algorithm in Operation In order to clarify how the IRBREL algorithm works, will be used the MissionaryCannibal game. This game is very simple and is not a Real Time Strategy Game, it could be solved using a whole world state search. Nonetheless, its simplicity will be used to show how the IRBREL algorithm could work using any other kind of game. In the following sections is described the Missionary-Cannibal Game and how the world states and the Performance Analyzer module are handled. Besides, is described how the IRBREL algorithm uses the Prior-Knowledge, Play, Improvisation and Meditation phases to learn how to solve the game. 4.1 Missionary-Cannibal Game The game consists in a group of three missionaries and three cannibals at the left side of a river and their goal is to cross to the right side of the river. In order to cross the river, they will need to use a boat. The problem is that the boat has only room enough for two passengers. Besides, cannibals are very aggressive. If cannibals are not under missionary care, they will eat any meat they could find, even missionaries. To solve that problem, the number of cannibals (in both sides of the river) never must be bigger than the number of missionaries. The Missionary-Cannibal game is a single agent game. The only agent in the game is the human player who controls how to move the missionaries, the cannibals and the boat. The game is completely observable for the human player. Besides, this is deterministic game. The changes in the world are only caused by the human player actions. The Missionary-Cannibal game begins as shown by figure 4.1. From that point, the human player has to move missionaries, cannibals and the boat in certain order to cross them all to the right side of the river. If the number of cannibals is greater than the missionaries in either side of the river, then human player loses the game. The only way to win the game is to move all missionaries and cannibals to right side of the river without letting that the number of 34 Figure 4.1: Missionary-Cannibal beginning game. cannibals exceeds the missionaries in either side of the river. 4.2 Satisfying the Special Requirements of the IRBREL Algorithm As described by chapter 3, the IRBREL algorithms needs that the game be deterministic and that the world be observable enough to let the agent learns why the world changed after executing an action. These prerequisites are already satisfied by the Missionary-Cannibal game. Nevertheless, the IRBREL algorithm also needs a module that generates the world states, a module that evaluates the performance of the agent and a list of the available actions of the agent. In game terms there are six possible actions that human player could execute. The IRBREL algorithm will have access to same actions available to human players. The table 4.1 shows the available actions and the ID that IRBREL algorithm will use to tell the game which action execute. All these actions are simple actions. Consequently, there is no need to notify the IRBREL algorithm about complex actions. ID 1 2 3 4 5 6 Action Get a missionary inside the boat. Get a cannibal inside the boat. Get a missionary outside the boat. Get a cannibal outside the boat. Move the boat to the right side of the river. Move the boat to the left side of the river. Table 4.1: List of Missionary-Cannibal Actions. The world states are represented using seven integer variables. As shown by table 4.2, each variable has an ID or variable name that is used by the IRBREL algorithm 35 to handle the inference rules and world states. The world state module creates an array of these seven variables and their values. These arrays are used by the IRBREL algorithm as world states. ID 1 2 3 4 5 6 7 Variable Number of Missionaries at the left side of the river. Number of Cannibals at the left side of the river. Number of Missionaries at the right side of the river. Number of Cannibals at the right side of the river. Number of Missionaries inside the boat. Number of Cannibals inside the boat. Position of the boat (left or right). Table 4.2: List of Missionary-Cannibal Variables. In order to let the IRBREL algorithm knows the utility of the actions and the utility to be in a certain world state, the game will have a score. This score is used as the Performance Analyzer module required by the IRBREL algorithm. At the beginning of a game the score will be zero. Some actions will increase the score value and others will decrease it. The score is increased by five points for each missionary or cannibal inside the boat. The score is increased by ten points for each missionary or cannibal at the right side of the river. If the game is won, the score will be positive infinite. However, if the game is lost, the score will be negative infinite. 4.3 Prior-Knowledge Phase in Operation The IRBREL algorithm begins knowing only the available actions and the initial world state. The IRBREL algorithm does not know the consequences of those actions or how win or lose the game. Besides, at the beginning, the knowledge base of the IRBREL algorithm is empty. If the knowledge base is empty and only if is empty, the Prior-Knowledge phase begins. At this point, the IRBREL algorithm has no idea of what will happen if he executes certain action. Consequently, the IRBREL algorithm does the only possible solution: experiment with the actions and world states. The IRBREL algorithm executes every action one time and checks what happens to the world states and restarts the game. This process is similar to any person which is with a video game that he never has seen before. The person begins pressing all buttons to figure out what happened and how the video game works. As shown by the figure 4.2, at the beginning of the game there are only two available actions: get a missionary inside the boat or get a cannibal inside the boat. The other actions are invalid and the game will ignore them. 36 Figure 4.2: First possible actions. Figure 4.3: The first inference rules learned. At the end of the Prior-Knowledge phase the agent will have three world states and two naive inference rules saved in his knowledge base. The saved world states are: the beginning game world state and its two neighbor world states. The two new inference rules, saved in the knowledge base, are shown in the figure 4.3. The “A[number]” means which action is executed, in the first case “A[1]” means the action: get a missionary inside the boat. The “Vnumber[number]” means the variable. The first number is the variable ID. The number inside the squared parenthesis is the value of the variable. Each variable could contain any integer value. For future information about the list of variables and actions see table 4.1 and table 4.2. In the conclusions section, the values inside the brackets are: the utility to execute that strategy, the number of times the inference rule was valid and the number of times the inference rule was invalid. The generated inference rules are very simple and naive. For the premises, only was taken in account the executed action. These naive inference rules assume that the consequences only depend on the chosen actions. Notwithstanding, in many cases the results depend on more variables. This problem could give inconsistencies problems which will be resolved latter by the Meditation phase. The Prior-Knowledge phase ends once all actions are tried and the IRBREL algorithm learned the naive inference rules. At the end of this phase the game is restarted and the Play phase begins. 37 4.4 Play Phase in Operation At this point the Prior-Knowledge phase is over and begins the first cycle of the Play phase. Besides, there are three world states saved in the knowledge base and there are two inference rules saved (see figure 4.3). The goal of the Play phase is to maximize the utility. Right now the IRBREL algorithm only knows three world states, one with utility of zero, another with utility negative infinite and the last one with utility of five. Consequently, the current goal of the IRBREL algorithm is to achieve world state with the utility of five. To accomplish that goal, the IRBREL algorithm will use his learned inference rules. A new game begins and the world state is the shown by figure 4.1. The IRBREL algorithm checks which strategies are available to its use. Each inference rule represents a strategy. If an inference rule fires, means that strategy is available to execute under the current circumstances. The premises of the inference rules represent the conditions under which the strategy is available. An inference rule fires if all the variables values in its premises are match with the variables values of the current world state. World state variables that are not in the premise section of an inference rule are not taken in account to determine if that inference rule fires. The variables of the conclusions section represent, how the world is affected after the strategy is executed. If an inference rule fires and is executed, could lead to two possible results. First, the values specified by the conclusions are the same as the new world state. Or second, the values specified by the conclusions are different than the new world state. In the first case the inference rule is valid. In the second case the inference rule is invalid. It is important to take note of every invalid inference rule. An invalid inference rule means that rule is too general and it needs to take in account other variables. These inconsistencies will be managed latter by the Meditation phase. The IRBREL checks which of its two learned inference rules fires. In this case both inference rules fires because all premises conditions are meet. Then, the IRBREL algorithm decides which fired inference rule use. There is one inference rule that gives the IRBREL algorithm an utility of five and another that gives an utility of negative infinite (game lost). The IRBREL algorithm chooses the inference rule with the highest real utility value. For this case, the IRBREL algorithm chooses the inference rule number two, which gives it a real utility value of five. If there are two or more inference rules with the same real utility values, then IRBREL algorithm chooses the most frequently used. To avoid using inaccurate inference rules and improve the efficiency, the IRBREL algorithm uses the real utility value of an inference rule instead of its utility value to choose which inference rule use. The real utility of the inference rule is calculated using the number of times it was valid or invalid as shown by equation 4.1. In this 38 way old and inaccurate inference rules have less weight than accurate inference rules. Inaccurate inference rules are managed latter by the Meditation phase. valid × utility (4.1) valid + invalid After executing the first action, the IRBREL algorithm checks the results. If the world state is the same specified by the used inference rule, then this rule was valid and the counter of number of success is increased by one. Nonetheless, if the world state is different from the specified by the inference rule, then this rule was invalid and the counter of number of failures is increased by one. Besides, if an inference rule is invalid, the IRBREL algorithm executes the Meditation phase (this is done in order to make the inference rule more accurate). RealUtility = Figure 4.4: Consequences of the first action. At this point, the IRBREL algorithm has chosen and executed the action two (get a cannibal inside the boat). That action makes the world state be as shows figure 4.4. Then the IRBREL algorithm checked that the current world state is the same described by the inference. So, the number of success by the inference rule is increased by one. Now, the IRBREL algorithm checks which inference rules fires. At this moment, the two inference rules stored in the knowledge base are too general and both fire again. However, if there are not any inference rules that fire, then the agent is considered stuck and the Improvisation phase begins. For this case, the IRBREL algorithm chooses to execute the same inference rule of the last time. Notwithstanding, instead of getting to the world state shown by figure 4.4, the world state is the one shown by figure 4.5. That inconsistency is noted and is executed the Meditation phase to update the invalid inference rule. 39 Figure 4.5: Consequences of the second action. 4.5 Meditation Phase in Operation The Meditation phase is executed if an inference rule was found invalid or the game itself is over, like in a game lost or won. In this case, this phase is executed because an inference rule was used and instead of getting in the world state shown by figure 4.4, the world state is the one shown by figure 4.5. In the Play phase, the IRBREL algorithm took note of the inference rule that did not behave as it was predicted. This happened because the inference rule is too general. Consequently, the IRBREL algorithm will generate new versions of the invalid inference rule using the OWS element described in chapter 3. Figure 4.6: Inference rule update. The IRBREL algorithm examined all inference rules and found only one of them invalid. As shown in figure 4.6, the IRBREL algorithm uses the linked OWS element of this inference rule to create more accurate versions of the original one. The new inference rules have the same elements of the original inference rule plus an additional variable in the premise. This additional variable and its value is obtained from the OWS element. NewInf erenceRules = T otalW orldV ariables−V ariablesInOriginalP remise (4.2) The number of new inference rules generated, for each invalid inference rule, is the difference between the total number of variables in the world and the number of 40 variables contained by the premises of the original inference rule. This is shown by equation 4.2. After every new inference rule is generated, the original inference rule is considered obsolete and it is deleted from the knowledge base. Figure 4.7 shows the current inference rules stored in the IRBREL knowledge base at this point. It is important to notice, that each time an new version of an inference rule is created, the counter of number of times the inference rule was valid and invalid are restarted. Figure 4.7: The new inference rules learned. If the game is over for any reason, like the game is lost, then the IRBREL algorithm updates the utilities of the inference rules. Nonetheless, the game is not over and the Play phase is executed again. The game is at the world state shown by figure 4.5. The Play phase is executed again and the IRBREL algorithm will notice that inference rules one, two, four, five, six and eight fire (see figure 4.7). Nevertheless, after choosing and executing an inference rule, the IRBREL algorithm will notice that the inference rule used was invalid again and the Meditation phase is executed again. After several cycles of executing the Play and Meditation phases, the IRBREL algorithm discards inference rules two, four, five, six and eight (see figure 4.7). These inference rules do not generate newer versions of themselves because there are still sister inference rules in the knowledge base. The world state remains the same because the action of getting a cannibal inside the boat is invalid for the current world state and the game ignores that action. Now, the only inference rule that fires is the number one. The IRBREL algorithm will try to use it in Play phase. Then, after find that the inference rule is invalid, the IRBREL algorithm executes the Meditation phase again. The process described for the inference rule two of figure 4.3 is repeated for this inference rule. Eventually, the IRBREL algorithm will have a knowledge base like the one shown by figure 4.8. After this process, the world state remain the same described by figure 4.5. Finally, the Play phase is executed again and the IRBREL algorithm did not found any inference rule that fired. Consequently, is executed the Improvisation phase. 41 Figure 4.8: The updated inference rules. 4.6 Improvisation Phase in Operation The goal of the Improvisation phase is to try something new and learn new strategies. That way the IRBREL algorithm has more strategies to choose when executes the Play phase. The Improvisation phase is executed if an action is interrupted, there were not any available strategies to execute or the IRBREL algorithm got stuck in the same world state. For this particular game, there are not complex actions. Consequently, the IRBREL algorithm never has the chance to interrupt a complex action. The only way to execute the Improvisation phase is that IRBREL algorithm gets stuck in a world state or if there are not any inference rules in the knowledge base that fire. At this point the IRBREL algorithm got stuck at the world state shown by figure 4.5. The IRBREL algorithm has no inference rules which tell him how to improve its utility (figure 4.8). Consequently, if the IRBREL algorithm wants to improve its utility is needed to try something new. The improvisation could lead the IRBREL algorithm to four possible scenarios: 1. The new world state has smaller utility than the original one. 2. The new world state has bigger utility than the original one. 3. The new world state has the same utility than the original. However, both world states are different. 4. After trying something new, the world state is the same as the original one. The most desirable scenario is one that, after trying something new, the IRBREL algorithm improved its utility. Notwithstanding, the least desirable scenario is one that, after trying something new, the IRBREL algorithm is in the same world state. This last scenario is a problem because the IRBREL algorithm is still considered stuck. In order to avoid that, the IRBREL algorithm has memory of the world state where has been stuck. Besides, the IRBREL algorithm remembers how many times arrived to the same world state. If the number of times the IRBREL algorithm gets stuck in the same world state gets bigger, the improvisation of the IRBREL algorithm gets bigger 42 too. Nonetheless, the more knows the IRBREL algorithm about the consequences of the available actions for the current world state, the smaller is the improvisation factor. Figure 4.9: Second possible actions. At this point there are two valid actions of the six actions available in the game (see table 4.1 for future reference). The possible results of those two actions are shown by figure 4.9. However, the IRBREL algorithm has no idea of which actions are valid or invalid. Besides, in the current world state there are not inference rules that fire. Consequently, the only available option is to randomly choose one of the six available actions. If the IRBREL algorithm tries an invalid action, the game will simply ignore that action. After the random action is chosen and executed, the IRBREL algorithm checks if the current state world is the same than the original one or if it is different. If the world state is different from the original one, the IRBREL algorithm is no longer considered stuck and the Play phase is executed again. If the world state is the same than the original one, then it means that the chosen random action had no effect and the agent needs to choose a new random action. Suppose that the IRBREL algorithm chooses to move the boat to the left side of the river as a random action. Then, IRBREL algorithm checks the world state and noticed that is the same world state. In game terms, the boat was already at the left side of the river so there was no change in the world state. In response of that, the IRBREL algorithm now tries a new action. The IRBREL algorithm chooses the action: move the boat to the right side of the river. The IRBREL algorithm checks again the world state and noticed that is a different world state so it is no longer considered stuck. Every time the IRBREL algorithm gets out of stuck, checks its knowledge base. If the world state is not already in his knowledge base, the IRBREL algorithm adds it. Besides, The IRBREL algorithm generates a naive inference rule that represents 43 the chosen action. This inference rule is very general, like the ones generated in the Prior-Knowledge phase. Figure 4.10: Consequences of the improvisation. At this point, the IRBREL algorithm is no longer considered stuck and the current world state is shown by figure 4.10. This world state is not already in the IRBREL algorithm’s knowledge base. Consequently, this world state is added to the knowledge base. Besides, there is also generated a new naive inference rule that describes these changes in the world state. Figure 4.11 shows the inference rules stored in the knowledge base. Figure 4.11: Learned inferences rules after improvisation phase. 4.7 The Learning Cycle Continues At this point, the IRBREL algorithm is in the world state shown by figure 4.10. Besides, the IRBREL algorithm is no longer considered stuck and the Play phase is executed again. Once the Play phase is executed, the IRBREL algorithm executes the only inference rule that fires, which is the inference rule number five (see figure 4.11). The execution of this strategy will not make inference rule invalid but will get stuck the IRBREL algorithm in the same world state. Consequently, the IRBREL algorithm executes the Improvisation phase again. 44 The IRBREL algorithm keeps executing the Improvisation phase until gets out of the stuck. Once out of stuck, the IRBREL algorithm executes the Play phase. Sometimes, the used inference rules will be invalid. Consequently, the Meditation phase is executed to update those inference rules. These cycles continues until the game is stopped or until the IRBREL algorithm wins or loses the game. Once a game is over, the Meditation phase is executed again. Then, as described by section 4.5, the IRBREL algorithm checks for invalid inference rules and generates new versions of them. Besides, once a game is over, the Meditation phase executes an additional step. In this step the IRBREL algorithm updates the utility of all inference rules. This additional step is optional and is done in order to help the IRBREL algorithm to choose the best strategy to execute. Figure 4.12: World states with the same utility. In the section 4.2 was described how the Performance Analyzer module evaluated the utility of the current world state. The use of that particular evaluation system creates some world states, one better than the other one, with the same utility values. As shown by figure 4.12, there are some world states with the same utility value. Nevertheless, in the world state of the bottom is closer to win the game than the one of the top. This issue could lead the IRBREL algorithm to get stuck in the game. That kind of issues should be avoided by the developer of the Performance Analyzer module. In this case, the Performance Analyzer module was created in this way for IRBREL algorithm explanation purposes. From the initial world state to the winner world state, the utility should have a gradual increment. World states that get away from the winner world state should have a gradual utility decrement. If each world state is evaluated properly, then there is no need of this additional step. However, 45 executing this additional step gives the IRBREL algorithm some degree of tolerance to this issue. To solve that issue, the IRBREL algorithm will try to internally give the world state of the bottom an utility of ten (see figure 4.12). Besides, the IRBREL algorithm will try to internally give the world state of the top an utility of ten minus a delta value. This is done in order to make the IRBREL algorithm prefer one world state over the other world states with the same utility value. Using the inference rules, world states and OWS elements stored in the knowledge base, the IRBREL algorithm creates a hierarchical tree structure of how the world states are interconnected among them. The world states stored in the knowledge base are used as nodes of the tree. The inference rules and OWS elements stored in the knowledge base are used to know how the nodes are interconnected among them. Inference rules are composed by three elements: premises, conclusions and utility. The premises describe how the world state was before the action was executed. The conclusions describe how the world state will be after the action is executed. Most inference rules are too general and the premises could not completely define the original world state. Notwithstanding, the linked OWS element, that each inference rule has, defines the original world state. Using that information, it is possible to know how the world states are interconnected among them. After the hierarchical tree structure is generated, the IRBREL algorithm looks for the node that represents the world state with the highest utility value. If there are more than one world states with the same utility, then it is randomly chosen one world state. The chosen world state (node) is assigned the same utility described by its counterpart world state. Then all the neighbor world states (nodes) receive that utility value minus a delta value. The same process is repeated with the next neighbors. This process is repeated until all nodes have an utility value. Figure 4.13 shows an example of how this process works. Now that the hierarchical tree with utilities is generated, it is needed to update the utilities of all inference rules to make them match with the ones of the nodes. The conclusion section of each inference rule represents how the world state is after executing the action. Consequently, using the world state represented by the conclusions of each inference rule, it is assigned the utility value of its counterpart node represented by the hierarchical tree. Finally, once the utility value of all inference rules are updated, the hierarchical tree is destroyed. Once this step ends, the IRBREL algorithm is ready to play a new game. The phases described in sections 4.3, 4.4, 4.5, 4.6 and 4.7 are repeated as needed. Eventually, after several learning cycles, the IRBREL algorithm will learn enough strategies (inference rules) to win this game. 46 Figure 4.13: Assignation of the utility to the inference rules. 47 4.8 Summary The Missionary-Cannibal game consists in a group of three missionaries and three cannibals at the left side of a river and their goal is to cross to the right side of the river. In order to cross the river, they will need to use a boat. The problem is that the boat has only room enough for two passengers. Besides, cannibals are very aggressive. If cannibals are not under missionary care, they will eat any meat they could find, even missionaries. To solve that problem, the number of cannibals (in both sides of the river) never must be bigger than the number of missionaries. The Missionary-Cannibal game is a single agent and completely observable game. Besides, this is deterministic game. This game is used to explain how the IRBREL algorithm learns and how the learning cycle works. To work properly, the IRBREL algorithm needs a list of the available actions of the agent, a module that generates the world states and module that evaluates the performance of the agent. The IRBREL algorithm has access to the following actions: • Get a missionary inside the boat. • Get a cannibal inside the boat. • Get a missionary outside the boat. • Get a cannibal outside the boat. • Move the boat to the right side of the river. • Move the boat to the left side of the river. The world states are composed by the following variables: • Number of Missionaries at the left side of the river. • Number of Cannibals at the left side of the river. • Number of Missionaries at the right side of the river. • Number of Cannibals at the right side of the river. • Number of Missionaries inside the boat. • Number of Cannibals inside the boat. • Position of the boat (left or right). 48 The Performance Analyzer module calculates the utility of each world state using the number of missionaries and cannibals at each side in the river and in the boat. The utility of the initial world state is zero. The utility is increased by five points for each missionary or cannibal inside the boat. The utility is increased by ten points for each missionary or cannibal at the right side of the river. If the game is won, the utility will be positive infinite. Nonetheless, if the game is lost, the utility will be negative infinite. The IRBREL algorithm began playing the game with no previous knowledge of the Missionary-Cannibal game. Consequently, The IRBREL algorithm executed the Prior-Knowledge phase to learn some basic knowledge that helps it to begin playing the game. Once the IRBREL algorithm learned some basic knowledge, the Play phase was executed. After some choices, the IRBREL algorithm found the learned strategies too general and executed the Meditation phase to improve those strategies. After several cycles of executing the Play and Meditation phases, the IRBREL algorithm refined the learned strategies (inference rules). Nevertheless, at some point, the IRBREL algorithm did not found any strategy that helps it to continue making progress in the game. Consequently, the IRBREL algorithm executed the Improvisation phase to try something new. The IRBREL algorithm continued executing the Play, Meditation and Improvisation phases. Eventually, the IRBREL algorithm arrived to a final world state (game lost or game won). Then, the IRBREL algorithm executed the Meditation phase to update invalid inference rules. Finally, the IRBREL algorithm updated the utility values of the inference rules. This additional step is done in order to help the IRBREL algorithm to choose the best strategy to execute. This last step solves the issue of having two world states, one better than the other, with the same utility values. With the exception of the Prior knowledge phase, this learning cycle process is repeated as needed in each game. Eventually, the IRBREL algorithm learns enough strategies to win the game. 49 Chapter 5 Using IRBREL Algorithm in a Real Time Strategy Game In the following sections are described the experiments made using the IRBREL algorithm in a Real Time Strategy Video Game. Besides, there are described special considerations that should be taken in account when the IRBREL algorithm is used in a Real Time Strategy Game. The IRBREL algorithm was set to compete against different AI agents and was registered the learning progression and the game results. The video game used for these experiments is named Flag’s Capture. This video game is a 2-D version game of the real Gotcha with some additional changes. 5.1 Flag’s Capture Game The Flag’s Capture game is a 2-D video game version of the real Gotcha game. The game is completely observable for the agents. The game itself is deterministic; every change in the world is only the consequence of the agents’ actions. There are only two agents interacting in the video game. The video game is a real time strategy game. This means that the agents must take decisions quickly. The game does not wait agents to take decisions; the time keeps running and the world keeps moving like in real life. If an agent takes too long thinking what to do next, the other agent could take advantage of this moment of inactivity. The video game consists in two rival agents (soldiers), a battlefield and two flags. The goal of each agent is to capture the enemy flag which is located in the opposite side of the battlefield. Each agent is equipped with a paint-ball pistol with unlimited ammunition. The paint-ball pistols are used to attack and defend the owner against the rival agent. If an agent is hit by a paint-bullet, then the agent is considered dead and leaves the battlefield. The paint-bullets move three times faster than soldiers. Figure 5.1 shows how the video game looks in action. The screen views the world as if the observer is positioned in the sky and looks to the ground. Each agent has a red or blue uniform and all his belongings (flag and bullets) are made of the same color. The flag is represented by the colored red or blue circle. 50 Figure 5.1: Flag’s Capture video game in action. The video game was designed with five different agents or difficulties. Brainless: This is the easiest difficulty of the game. As the name says, the agent itself does not have any behavior or strategy. The agents controlled by this kind of AI just stay without executing any action. Consequently, it is like the rival agent is playing alone. Dummy: The agents controlled by this AI always tries to capture the enemy flag without taking care anything else. Since the beginning of the game, these AI makes the agent moves directly to the enemy flag and capture it. This agent does not try to attack the rival or to defend himself against incoming paint-bullets. Kamikaze: This AI is very similar to the Dummy agent. The difference is that the Kamikaze AI makes the agent shoots paint-bullets to everything that crosses his path to the enemy flag. Like the Dummy agent, this AI does not try to evade incoming paint-bullets. Commando: This is the best static AI of the game. The AI makes the agent to aim and shoot paint-bullets against the rival agent. If there is a paint-bullet in collision route to him, the AI makes the agent evade that paint-bullet. Once the rival agent is considered dead, the AI makes the agent to go and capture the enemy flag. 51 IRBREL: The AI is dynamic. This AI uses de IRBREL algorithm to decide what to do. As described by chapters 3 and 4, the AI begins knowing nothing. After several learning cycles, the AI knows the consequences of his actions in many scenarios and behaves according to get the best known performance. For the human players, there are five available actions. Move Forward: Moves the soldier one step to the front. Move Backward: Moves the soldier one step back. Turn Left: Rotates the soldier a minus one degree. Turn Right: Rotates the soldier one positive degree. Fire: Shoots one paint-bullet from his paint-gun. At the beginning of each game, the red flag is positioned at the center of the left side of the battlefield. On the other side, the blue flag is positioned at the center of the right side of the battlefield. Each soldier is positioned one step ahead of his color flag. Both soldiers begin targeting their respective rival. Figure 5.2 Shows how the game looks at the beginning of each encounter. Figure 5.2: Flag’s Capture video game - start position. 52 5.2 The Performance Analyzer According to the specified requirements of the IRBREL algorithm, the IRBREL algorithm needs a module which serves as performance analyzer of its actions. In other words, the IRBREL algorithm needs a module that tells him the utility of the current world state. This utility is used by IRBREL algorithm to know the consequences of his actions and to choose the best available known strategy. The objective of the game is to capture the enemy flag. Consequently, the Performance Analyzer module will award with an infinite utility to all world states where the soldier wins the game. Besides, will award with a negative infinite utility to all world states where the soldier loses or is considered dead. For games in which the result is a draw, the soldier is awarded with five thousand utility points. The closer is the soldier to the enemy flag, the closer he is to the goal of the game. So, the Performance Analyzer module awards the soldier with one utility point for each step closer to the enemy flag. However, the closer his rival is form his flag, the closer the soldier is to lose the game. So, the Performance Analyzer module penalizes the soldier with one utility point for each step closer the rival is from his flag. Having a paint-bullet in collision route to the soldier is a bad thing. Notwithstanding, having a paint-bullet in collision route to the enemy is a good thing. Consequently, the Performance Analyzer module awards with one thousand utility points if there is a paint-bullet in collision route to the enemy. Besides, penalizes with one thousand utility points if there is a paint-bullet in collision route to him. 5.3 Special Considerations If we give to the IRBREL algorithm the variables values of the world state and the same simple actions available to human players (move forward, move backward, turn left, turn right and fire), the IRBREL algorithm will be able to learn the consequences of his actions and how this world works. Nonetheless, there are some considerations that should be taken in account. The IRBREL algorithm is similar to the human learning. If humans do not know what they are trying to learn, then will be harder and in some cases very improvable to learn that thing. It is not completely necessary that humans know exactly what they want to learn. Nevertheless, they need to have an idea of what they want to learn. An example of these is the little children that learn to crawl. They do not exactly want to learn to crawl, but they are trying to learn a way to move to the place they want to be. This will lead the little children to learn how to crawl and then to learn how to walk, run, etc. These principles apply to the IRBREL algorithm too. The IRBREL algorithm does not necessary needs to know that is learning to evade paint-bullets. The IRBREL algorithm could only know that it is just learning how survive. 53 The knowledge of what the IRBREL algorithm is trying to learn is represented by the utility given from the Performance Analyzer module. Consequently, a completely learned behavior is rewarded with a positive infinite utility. Besides, a complete failure in learning the behavior is rewarded with a negative infinite utility. Any other behavior is rewarded according to of how close is the behavior from the desired one. The IRBREL algorithm uses reinforcement learning. So, for optimal results, the utilities given by the Performance Analyzer module should reflect, in the best possible way, what is wanted to learn. Should be avoided the cases where two world states, one better than the other one, have the same utility values. It is important to consider what is the goal of the IRBREL algorithm. Is the goal to learn complex behaviors like learn to evade a paint-bullet? Is the goal is just to learn how and when use the known abilities to win the game? For the first case, if is wanted that the IRBREL algorithm learns complex behaviors like evade a paint-bullet, then the whole Performance Analyzer module described in the section 5.2 needs to be modified. The Performance Analyzer should reward with positive infinite all cases where the soldier evades a bullet. Besides, the Performance Analyzer should penalize actions that do not help to avoid paint-bullets and awards actions that help to avoid paint-bullets. For the second case, if is wanted that the IRBREL algorithm learns how and when use the known abilities to win the game, then should be used a Performance Analyzer module similar to one described in the previous section. The meaning or representation of the variables is a factor that could improve or delay the learning process. The representation of a paint-bullet collision could be done in several ways. This could be handled by learning under which trajectories of paintbullets could collide with the soldier. That learning should be repeated by each possible position that the soldier could be in the battlefield. Another way to handle this is to add an additional variable in the world state. The new variable indicates if the soldier will be hit by a paint-bullet if he does not make anything to change that situation. In that way, the IRBREL algorithm could learn valid inference rules faster because the premises depend in fewer variables. Besides, that makes inference rules more general and applies to all possible positions in the battlefield instead of learning inference rules for each particular position. Humans beings have the amazing capability of generate abstract data, like the paint-bullet collision detection. Unfortunately, the IRBREL algorithm does not have that capability yet. Which means that by now all abstract variables should be managed by the world state generator module. Another important consideration is the variables used to generate the world states. If there are used abstract variables, then some other variables could become redundant. For example, if there is used the abstract variable that represents paint-bullet collisions, then other variables like position, orientation and trajectory of paint-bullets could be54 came redundant. The new single variable could represent the same information than the other three variables together. If redundant variables have no other use, then is safe to ignore these variables for the world state. However, if these variables are used for other considerations (like for example evade paint-bullets), then these variables should be included in for the world state. Ignoring important variables for the world state could make a deterministic game to become a non-deterministic game to the IRBREL algorithm perspective. The IRBREL algorithm is not capable to function properly in non-deterministic environments. In the other hand, ignoring not used variables simplifies the world and speeds up the learning of the IRBREL algorithm. In some cases, some world states could be redundant. If the goal of the game is to capture the enemy flag, then it is not so important if the rival soldier is still alive or if the soldier captures the flag while is looking to the north. For these cases, if it is not important how the game was won, then some world states could be grouped into a single one that represents a won state. The same applies to the cases of game lost, game draw, etc. Grouping world states into single ones reduces the number of world states and simplifies the work for the IRBREL algorithm, which makes the learning process faster. Grouping world states could make a deterministic game to become a non-deterministic game to the IRBREL algorithm perspective. Consequently, is very important to consider how the IRBREL algorithm will perceive the world with these changes. Learning in video games should be used to improve the agents’ performance. Learning of how play the video game should only be used in the developing phase not in the finished version. For finished versions of the video game, the agents should know some skills and through learning the agents could perfection those skills knowing how and when use them. For the Flag’s Capture video game, the skills could be basic skills like move forward, turn left, fire, etc. Besides, the skills could include some complex skills like target enemy, evade paint-bullets, etc. A simple skill or simple action is an actions that take one game action or unit of time to be executed, like move one step forward or shoot a paint-bullet. In most cases, the simple actions are those actions available for human players. A complex skill or complex action is the execution of simple actions in certain order. For example, evading a paint-bullet requires several turn lefts or turn rights and several move forwards or move backwards. Complex actions take more than one game action to be executed. The IRBREL algorithm is able to manage simple and complex actions. The IRBREL algorithm just needs to know, from his available actions repertory, which actions are simple and which are complex. Besides, the IRBREL algorithm has the ability to abort the execution of complex actions if notice that the performance is not going well or if the conditions of the game changed. 55 5.4 Flag’s Capture Experiments The learning in video games should be used to improve the agents’ performance. Learning of how to play the video game should only be used in the developing phase not in the finished version. For finished versions of the video game, the agents should know some skills and through learning the agents could perfection those skills knowing how and when use them. The knowledge base of the IRBREL algorithm should not be empty in finished versions. Notwithstanding, in order to measure how the IRBREL algorithm improves its performance, the knowledge base will be empty at the beginning of each experiment. The propose of these experiments is not the development of new complex actions. The goal of these experiments is just to measure how the IRBREL algorithm improves its performance. Thus, the IRBREL algorithm will know the basic simple actions available to human players plus some additional complex actions. For the IRBREL algorithm, there are eight available actions. The first five are simple actions. The last three are complex actions. Move Forward: Moves the soldier one step to the front. Move Backward: Moves the soldier one step back. Turn Left: Rotates the soldier a minus one degree. Turn Right: Rotates the soldier one positive degree. Fire: Shoots one paint-bullet from his paint-gun. Evade Paint-Bullets: Makes the soldier to evade all paint-bullets in collision trajectory to him. Target Enemy: The soldier targets his enemy using his paint-gun. Go to Enemy Flag: Moves the soldier to the position of the enemy flag and captures it. The evade paint-bullets action is not perfect. This complex action fails if the incoming paint-bullets are too close to the soldier to escape. The paint-bullets are three times faster than soldiers. So, in some cases it is impossible for the soldier to escape a paint-bullet collision. Besides, this complex action does not work for very complex bursts of paint-bullets. Nevertheless, every AI, that has the ability of evade paint-bullets, use the same algorithm. So, any AI that uses the ability of evade paintbullets has the same advantages and disadvantages. 56 To measure the ability of the IRBREL algorithm to learn how and when use its skills, the IRBREL algorithm was set to play against every AI of the video game. The behavior of the different kinds of AI is explained in section 5.1. The experiments are split in four categories. In each category the IRBREL agent was set to play against a specific AI. For all the games played in the same category is used the same kind of AI. Each category is split in fifty game sessions. In each game session the IRBREL agent was set to play three hundred times against the AI agent specified by the current category. The IRBREL algorithm’s knowledge base is erased before each game session begins. Consequently, at the beginning of each game session, the IRBREL agent begins using the Prior-Knowledge phase. Finally after the Prior-Knowledge phase ends, the first game begins. At the end of each game session the video game created a log file with the results. The results show how the game ended in each of the three hundred played games. There are three possible endings: The IRBREL agent won the game, lost the game or the game is a draw. In the next subsections are shown the results of the experiments for each category. The first graph shows the results of the ending values of the three hundred played games. The “X” axis represents the number of the played game. The “Y” axis represents the ending result of each game. The graph values go from minus one to one. One represents a game won by the IRBREL agent. Zero represents a game draw. Minus one represents a game lost by the IRBREL agent. The values of the graph are the average ending results of the fifty game sessions. The second graph shows the percentage of games won, lost or draw in the current category. 5.4.1 Brainless Agent The Brainless agent is the easiest AI to beat of the game. As the name says, the Brainless agent itself does not have any behavior or strategy. The Brainless agent just stays without executing any action and represents no harm for the IRBREL agent. Consequently, it is like the IRBREL agent is playing alone. In order to win the game, the IRBREL agent just needs to learn that is safe to go and capture the enemy flag. Since the Brainless agent will not take any action, the IRBREL agent does not have to learn any special precondition in which use the known actions. As shown by figure 5.3, all the games were won by the IRBREL agent. All the games were won by the IRBREL agent because the Brainless agent did not execute any action. So, eventually the IRBREL agent could improvise enough to capture the enemy flag and learn how to win the game. Figure 5.4 shows the percentage of games won, lost or draw. For this case, all 57 Figure 5.3: Game progress in Brainless AI category. games were won because the Brainless agent did not execute any action and the IRBREL agent kept improvising. 5.4.2 Dummy Agent The Dummy agent always tries to capture the enemy flag without taking care of anything else. At the beginning of each game, the Dummy AI makes the soldier to move directly to the enemy flag and capture it. The Dummy agent does not try to attack the rival or to defend himself against incoming paint-bullets. In order to win the game, the IRBREL agent needs to learn how to capture the enemy flag before the Dummy agent does. If the IRBREL agent just go and capture the enemy flag, then all the games will be a draw because both soldiers will capture the flag at the same time. The only way to win the game is to first kill the enemy soldier and then capture the flag. Consequently, in order to win the game, the IRBREL agent needs to learn to do those actions in that order. As shown by figure 5.5, at the beginning all games were lost by the IRBREL agent. After a few games, the IRBREL agent found a way to win or draw the game. By the game number seven, the IRBREL agent found a way to win the game. Nonetheless, the learned inference rules were too general and the IRBREL agent applied them in wrong situations. Eventually the inference rules became accurate and the IRBREL agent began to apply them in the right situations. The desire of improvisation by the IRBREL agent made that sometimes tried something new in different situations. The intension with those improvisations was to 58 Figure 5.4: Game score in Brainless AI category. see if the IRBREL agent got better results than using the already known strategies. Finally, by the game number 150, all desire of improvisation was satisfied by the IRBREL agent and only used its best known strategies. Figure 5.6 shows the percentage of games won, lost or draw. For this case, the IRBREL agent won 98.29% of all games. Only 1.38% of the games were a draw and 0.33% of the games were lost. These results show that the IRBREL agent learned very fast how to apply the available actions. 5.4.3 Kamikaze Agent The Kamikaze agent is very similar to the Dummy agent. The difference is that the Kamikaze AI makes the soldier shoots paint-bullets to everything that crosses his path to the enemy flag. Like the Dummy agent, the Kamikaze agent does not try to evade incoming paint-bullets. In order to win the game, the IRBREL agent needs to learn how to capture the enemy flag without getting hit by a paint-bullet. Besides, the IRBREL agent needs to do that before the Kamikaze agent captures the flag. If the IRBREL agent just evades the paint-bullets and go to capture the enemy flag, then the IRBREL agent will always lose the game because the Kamikaze agent will arrive sooner to the flag. If the IRBREL agent just shoots paint-bullets to the Kamikaze agent without evading bullets, then both will die and the game will be a draw. Consequently, in order to win the game, the IRBREL agent needs to learn to kill the Kamikaze agent, evade all paint-bullets and capture the enemy flag. 59 Figure 5.5: Game progress in Dummy AI categoryy. Figure 5.6: Game score in Dummy AI category. 60 There are several ways to win the game and some actions could be done in different order. The Kamikaze agent is more sophisticate and takes longer to the IRBREL agent to find a way to always win the game. Figure 5.7 shows the average learning progress of the IRBREL agent. By the game number seven, the IRBREL agent found a way to win the game. Nevertheless, the generalized assumptions made the IRBREL agent lost or draw the game from game 15 to 40. From the game number 40 to 205 the IRBREL agent continued doing little improvisations to try to get better results, some of these improvisations made the performance worse. Finally, by the game number 205 the behavior of the IRBREL agent was stable and always won the game. Figure 5.7: Game progress in Kamikaze AI category. Figure 5.8 shows the percentage of games won, lost or draw. For this case, the IRBREL agent won 95.03% of all games. Only 0.53% of the games were a draw and 4.43% of the games were lost. Contrary to the results obtained in the Dummy agent category (figure 5.6), there are more cases of game lost than game draw. This can be explained by the aggressive behavior of the Kamikaze agent and the passive behavior of the Dummy agent. 5.4.4 Commando Agent The Commando agent is the best static AI of the game. The Commando AI makes the soldier to aim and shoot paint-bullets against the rival soldier. If there is a paint-bullet in collision route to him, the Commando AI makes the soldier evade that paint-bullet. Once the rival agent is considered dead, the Commando AI makes the soldier to go and capture the enemy flag. The human players have many troubles to 61 Figure 5.8: Game score in Kamikaze AI category. just draw a game against this agent. In order to win the game, the IRBREL agent needs to learn how to capture the enemy flag after eliminating the rival soldier. The only way to beat the Commando agent is shooting paint-bullets to the enemy and evading incoming paint-bullets. The Commando agent will keep shooting paint-bullets and evading the ones that will hit him. The tricky part is to make the Commando agent to flee to a corner and then kill the soldier. However, the Commando agent tries to avoid those situations. Consequently, human players and the IRBREL agent will have a lot of trouble to win a single game. In order to win the game, the IRBREL agent will need to learn how combine the known actions in very specific situations. The Commando agent has a lot of resources and each game change a lot from each other. This make harder for the IRBREL agent to learn an absolute winner strategy. Figure 5.9 shows the average learning progress of the IRBREL agent. From game number 1 to 5, the IRBREL agent learned how to draw the game. Notwithstanding, from game number 5 to 10, the generalized assumptions made the IRBREL agent to lose more often. From game 13 to 269, the IRBREL agent began to learn how win a game more often than draw o lose it. The graph in that section shows how the values slowly have a tendency towards one, which means, that by each game played, the IRBREL agent won more often. By game number 270, the IRBREL agent found a way to almost always win a game. At that point, the rare cases were the IRBREL agent does not always won the game is because the improvisation desire of the IRBREL agent. Nonetheless, eventually the behavior was stabilized and the IRBREL agent always won the game. 62 Figure 5.9: Game progress in Commando AI category. Figure 5.10 shows the percentage of games won, lost or draw. For this case, the IRBREL agent won 81.68% of all games. Only 1.03% of the games were a draw and 17.23% of the games were lost. Like the results obtained with Kamikaze agent category (figure 5.8), there are more cases of game lost than game draw. Besides it took longer for the IRBREL agent to absolutely beat the Commando agent than in any other category. This can be explained by the extremely well performance of the Commando agent. Nevertheless, the IRBREL agent eventually found a way to always win the game. 5.4.5 Experiment Conclusions At this point was been proven that the IRBREL algorithm has the capability to learn, take quick decisions and improvise, which is its most important capability. Besides was proven that IRBREL algorithm has the capability to learn and to take quick decisions in Real Time Strategy Games. Taking in account the special considerations described in section 5.3, there is possible to use the IRBREL algorithm in other video games. The improvisation factor is the only way for the IRBREL algorithm to learn new strategies. As shown in almost every category, the improvisation could hinder the performance if the IRBREL algorithm already knows almost every good strategy. However, for these cases, the IRBREL algorithm does not have a way to know if the not tested actions will be a better or worse choices until try them in the current circumstances. 63 Figure 5.10: Game score in Commando AI category. The generalized inference rules or strategies, in the early learning stage of the IRBREL algorithm, harm its performance. Every time the IRBREL algorithm found a better strategy, assumed that only depended on the taken action. So, after its first won game, the IRBREL algorithm got a worse performance. This can be appreciate in figures 5.5, 5.7 and 5.9 where there is a drastic performance fall in the early learning stage. Notwithstanding, the IRBREL algorithm makes the inference rule more specific quickly and the lost performance is quickly recovered. The capability of the IRBREL algorithm to improvise, interrupt complex actions if the performance gets worse and then improvise again is what lets the IRBREL algorithm to quickly learn useful strategies. Besides, the interruption of complex actions is what let the IRBREL algorithm to change the used strategy to adapt itself against the Commando AI behavior. 5.5 Summary The Flag’s Capture game is a 2-D video game version of the real Gotcha game. The game is completely observable for the agents. The game itself is deterministic; every change in the world is only the consequence of the agents’ actions. There are only two agents interacting in the video game. The video game is a real time strategy game. This means that the agents must take decisions quickly. The game does not wait agents to take decisions; the time keeps running and the world keeps moving like in real life. If an agent takes too long thinking what to do next, the other agent could take advantage of this moment of inactivity. 64 The video game consists in two rival agents (soldiers), a battlefield and two flags. The goal of each agent is to capture the enemy flag which is located in the opposite side of the battlefield. Each agent is equipped with a paint-ball pistol with unlimited ammunition. The paint-ball pistols are used to attack and defend the owner against the rival agent. If an agent is hit by a paint-bullet, then the agent is considered dead and leaves the battlefield. In order to measure the learning performance of the IRBREL algorithm, it was set to compete in the Flag’s Capture game against different agents. Brainless: This is the easiest difficulty of the game. As the name says, the agent itself does not have any behavior or strategy. The agents controlled by this kind of AI just stay without executing any action. Consequently, it is like the rival agent is playing alone. Dummy: The agents controlled by this AI always tries to capture the enemy flag without taking care anything else. Since the beginning of the game, these AI makes the agent moves directly to the enemy flag and capture it. This agent does not try to attack the rival or to defend himself against incoming paint-bullets. Kamikaze: This AI is very similar to the Dummy agent. The difference is that the Kamikaze AI makes the agent shoots paint-bullets to everything that crosses his path to the enemy flag. Like the Dummy agent, this AI does not try to evade incoming paint-bullets. Commando: This is the best static AI of the game. The AI makes the agent to aim and shoot paint-bullets against the rival agent. If there is a paint-bullet in collision route to him, the AI makes the agent evade that paint-bullet. Once the rival agent is considered dead, the AI makes the agent to go and capture the enemy flag. For those competitions, the IRBREL agent had access to eight actions. The first five are simple actions. The last three are complex actions. Move Forward: Moves the soldier one step to the front. Move Backward: Moves the soldier one step back. Turn Left: Rotates the soldier a minus one degree. Turn Right: Rotates the soldier one positive degree. Fire: Shoots one paint-bullet from his paint-gun. Evade Paint-Bullets: Makes the soldier to evade all paint-bullets in collision trajectory to him. 65 Target Enemy: The soldier targets his enemy using his paint-gun. Go to Enemy Flag: Moves the soldier to the position of the enemy flag and captures it. According to the specified requirements of the IRBREL algorithm, the IRBREL algorithm needs a module which serves as performance analyzer of its actions. The Performance Analyzer module awards with an infinite utility if the game is won, a negative infinite utility if the game is lost or five thousand utility points if the game is a draw. The soldier is awarded with one utility point for each step closer to the enemy flag and penalized with one utility point for each step closer the rival is from his flag. The Performance Analyzer module awards the soldier with one thousand utility points if there is a paint-bullet in collision route to the enemy. Besides, penalizes the soldier with one thousand utility points if there is a paint-bullet in collision route to him. There are some special considerations that should be taken in account before using the IRBREL algorithm. These considerations could help to improve the IRBREL algorithm performance. • The utility given by the Performance Analyzer should reflect what the IRBREL algorithm wants to learn. • There could be used additional variables that represent abstract knowledge. This abstract knowledge helps the IRBREL algorithm to have simpler inference rules and to learn faster. • Some world states could be grouped in a single world state. This simplifies the game and helps the IRBREL algorithm to learn faster. • Complex actions helps the IRBREL algorithm to keep focused in learning how to apply the learned actions to win the game and avoids the need of learning new complex behaviors. The IRBREL algorithm was set to play three hundred games against each agent. These experiments were repeated fifty times and was registered the average learning progress. The learning progress was measured using the number of times the game was lost, draw or won by the IRBREL agent. The IRBREL agent was able to learn enough strategies (inference rules) to always win the game. These experiments proves that the IRBREL algorithm has the capability to learn, take quick decisions and improvise, which is its most important capability. Besides was proven that IRBREL algorithm has the capability to learn and to take quick decisions in Real Time Strategy Games. The improvisation factor is the only way for the IRBREL algorithm to learn new strategies. The improvisation could hinder the performance if the IRBREL algorithm 66 already knows almost every good strategy. Nonetheless, for these cases, the IRBREL algorithm does not have a way to know if the not tested actions will be a better or worse choices until try them in the current circumstances. The generalized inference rules or strategies, in the early learning stage of the IRBREL algorithm, harm its performance. Every time the IRBREL algorithm found a better strategy, assumed that only depended on the taken action. So, after its first won game, the IRBREL algorithm got a worse performance. Nevertheless, the IRBREL algorithm makes the inference rule more specific quickly and the lost performance is quickly recovered. The capability of the IRBREL algorithm to improvise, interrupt complex actions if the performance gets worse and then improvise again is what lets the IRBREL algorithm to quickly learn useful strategies. Besides, the interruption of complex actions is what let the IRBREL algorithm to change the used strategy to adapt itself against the Commando agent. 67 Chapter 6 Conclusions In the following sections are described in summary how the IRBREL algorithm works and its principal results. Besides, it is described the contributions made by the IRBREL algorithm and its relevance. Finally, are described several ways in which the IRBREL algorithm could be modified to be used in a wide range of Real Time Strategy Games. These additional considerations were not scope of this research but could be used for future work. 6.1 Summary and Principal Results The goal of this research is to create a system module for Real Time Strategy Video Games. The system module learns from its experience and observations using Reinforcement Learning. The system module is used as the AI of a Real Time Strategy Video Game. The idea is that the video game has agents with an AI that improves with the time. The AI will learn from its chosen actions and behave according to the learned knowledge. That way the human players could not use the same strategies to beat the game. Consequently, the game will force the human players to keep improving their skills. One of the most important capabilities of this system is its ability to improvise under unknown situations. Besides, it is able to adapt and update its knowledge as needed. That way, the next time the system module is under the same circumstances, the system will know what to do or what not to do. The scope of this research is to create a system module able to learn and improvise in Real Time Strategy Video Games. The system module is intended to work only in deterministic and completely observable environments. The Inference Rule-Based Reinforcement Learning algorithm or “IRBREL” algorithm uses the same commonly used principles of reinforcement learning. However, instead of the common structure used in Reinforcement Learning algorithms, the IRBREL algorithm uses inference rules to represent strategies and to model the world. The IRBREL algorithm is designed as a module, which can be adapted to be used 68 in Real Time Strategy Video Games. Notwithstanding, in order to work properly, there are five special requirements that must be satisfied. World State Generator Module: A module that generates the world states of the video game. Performance Analyzer Module: Module that calculates the utility of the agent to be in the current world state. Deterministic Game: The changes in the world state must be only determined by the actions executed by agents. Available Actions: The IRBREL algorithm needs to know what actions are available to its use. Feedback for Complex Actions: The agent must inform to the IRBREL algorithm, using an internal flag, if there is a complex action still in execution In order to represent the learned knowledge, the IRBREL algorithm uses a knowledge base composed by four elements: World States: Each world state represents a picture of how is the world in that instant. Furthermore, each world state has an utility value, which is the utility for the agent to be in that world state. Inference Rules: Represent actions or strategies which can be done under some conditions. Besides, represents how the world is modified by executing those actions and how useful are those world modifications. Original World State: This is a world state that is linked to a specific inference rule. This world state represents the original world state before the action described in the inference rule is executed. Know All Consequences Flag: This is a Boolean flag that is linked to a specific inference rule. This flag is turned on if there are known the consequences of executing every possible action under the same conditions of the linked inference rule. The IRBREL algorithm learns in cycles of four phases. Each phase is executed as needed. The Prior-Knowledge is used if there is no previous knowledge of the game. In this phase the IRBREL algorithm learns using a trial and error strategy. The Play phase uses all the learned knowledge to find the best strategy to win the game. If at some point the IRBREL algorithm finds something unexpected, then, in the Improvisation phase, the IRBREL algorithm looks for the best way to improvise and try something 69 new. Finally, in the Meditation phase, the IRBREL algorithm updates the learned knowledge to make it accurate. There are some special considerations that should be taken in account before using the IRBREL algorithm. These considerations could help to improve the IRBREL algorithm performance. • The utility given by the Performance Analyzer should reflect what the IRBREL algorithm wants to learn. • There could be used additional variables that represent abstract knowledge. This abstract knowledge helps the IRBREL algorithm to have simpler inference rules and to learn faster. • Some world states could be grouped in a single world state. This simplifies the game and helps the IRBREL algorithm to learn faster. • Complex actions helps the IRBREL algorithm to keep focused in learning how to apply the learned actions to win the game and avoids the need of learning new complex behaviors. The performance of the IRBREL algorithm was measured using the Flag’s Capture game, which is a Real Time Strategy Video Game. The Flag’s Capture game is a 2-D video game version of the real Gotcha game. The game is completely observable for the agents. The game itself is deterministic; every change in the world is only the consequence of the agents’ actions. There are only two agents interacting in the video game. The video game is a real time strategy game. This means that the agents must take decisions quickly. The game does not wait agents to take decisions; the time keeps running and the world keeps moving like in real life. If an agent takes too long thinking what to do next, the other agent could take advantage of this moment of inactivity. The video game consists in two rival agents (soldiers), a battlefield and two flags. The goal of each agent is to capture the enemy flag which is located in the opposite side of the battlefield. Each agent is equipped with a paint-ball pistol with unlimited ammunition. The paint-ball pistols are used to attack and defend the owner against the rival agent. If an agent is hit by a paint-bullet, then the agent is considered dead and leaves the battlefield. In order to measure the learning performance of the IRBREL algorithm, it was set to compete in the Flag’s Capture game against different agents. Brainless: This is the easiest difficulty of the game. As the name says, the agent itself does not have any behavior or strategy. The agents controlled by this kind of AI just stay without executing any action. Consequently, it is like the rival agent is playing alone. 70 Dummy: The agents controlled by this AI always tries to capture the enemy flag without taking care anything else. Since the beginning of the game, these AI makes the agent moves directly to the enemy flag and capture it. This agent does not try to attack the rival or to defend himself against incoming paint-bullets. Kamikaze: This AI is very similar to the Dummy agent. The difference is that the Kamikaze AI makes the agent shoots paint-bullets to everything that crosses his path to the enemy flag. Like the Dummy agent, this AI does not try to evade incoming paint-bullets. Commando: This is the best static AI of the game. The AI makes the agent to aim and shoot paint-bullets against the rival agent. If there is a paint-bullet in collision route to him, the AI makes the agent evade that paint-bullet. Once the rival agent is considered dead, the AI makes the agent to go and capture the enemy flag. For those competitions, the IRBREL agent had access to eight actions. The first five are simple actions. The last three are complex actions. Move Forward: Moves the soldier one step to the front. Move Backward: Moves the soldier one step back. Turn Left: Rotates the soldier a minus one degree. Turn Right: Rotates the soldier one positive degree. Fire: Shoots one paint-bullet from his paint-gun. Evade Paint-Bullets: Makes the soldier to evade all paint-bullets in collision trajectory to him. Target Enemy: The soldier targets his enemy using his paint-gun. Go to Enemy Flag: Moves the soldier to the position of the enemy flag and captures it. According to the specified requirements of the IRBREL algorithm, the IRBREL algorithm needs a module which serves as performance analyzer of its actions. The Performance Analyzer module awards with an infinite utility if the game is won, a negative infinite utility if the game is lost or five thousand utility points if the game is a draw. The soldier is awarded with one utility point for each step closer to the enemy flag and penalized with one utility point for each step closer the rival is from his flag. The Performance Analyzer module awards the soldier with one thousand utility points 71 if there is a paint-bullet in collision route to the enemy. Besides, penalizes the soldier with one thousand utility points if there is a paint-bullet in collision route to him. The IRBREL algorithm was set to play three hundred games against each agent. These experiments were repeated fifty times and was registered the average learning progress. The learning progress was measured using the number of times the game was lost, draw or won by the IRBREL agent. The IRBREL agent was able to learn enough strategies (inference rules) to always win the game. These experiments proves that the IRBREL algorithm has the capability to learn, take quick decisions and improvise, which is its most important capability. Besides was proven that IRBREL algorithm has the capability to learn and to take quick decisions in Real Time Strategy Video Games. The improvisation factor is the only way for the IRBREL algorithm to learn new strategies. The improvisation could hinder the performance if the IRBREL algorithm already knows almost every good strategy. Nonetheless, for these cases, the IRBREL algorithm does not have a way to know if the not tested actions will be a better or worse choices until try them in the current circumstances. The generalized inference rules or strategies, in the early learning stage of the IRBREL algorithm, harm its performance. Every time the IRBREL algorithm found a better strategy, assumed that only depended on the taken action. So, after its first won game, the IRBREL algorithm got a worse performance. Nevertheless, the IRBREL algorithm makes the inference rule more specific quickly and the lost performance is quickly recovered. The capability of the IRBREL algorithm to improvise, interrupt complex actions if the performance gets worse and then improvise again is what lets the IRBREL algorithm to quickly learn useful strategies. Besides, the interruption of complex actions is what let the IRBREL algorithm to change the used strategy to adapt itself against the Commando agent. 6.2 Contributions and Relevance of the IRBREL Algorithm The IRBREL algorithm is the first learning system module designed for Real Time Strategy Video Games that uses Reinforcement Learning and Inference Rules. The main contribution of the IRBREL algorithm is its capability to improvise under known circumstances and learn from that improvisation. The use of inference rules to represent the learned knowledge helps the IRBREL algorithm to quickly decide which strategy execute. Instead of analyzing the whole world state space, the IRBREL algorithm just needs to look which inference rule fires. Then, from the inference rule that fire, executes the inference rule that gives the agent 72 the highest utility value. This process saves a lot of time because instead of analyzing multiple variables and world states, the IRBREL algorithm only needs to analyze two values: The Boolean value that indicates if the inference rule fired and the utility value. The knowledge represented by general inference rules is useful even if the knowledge is not completely accurate. In many cases the general inference rules hinder the performance in the early learning cycles. However, these general inference rules serve as the base for more specific and accurate inference rules. The use of the IRBREL algorithm in Real Time Strategy Video Games improves the video game in several ways: • The agents using the IRBREL algorithm will learn new strategies and try different things. Consequently, the video game is not always the same and that could be attractive for human players. • The improvisation and learning capabilities of the IRBREL algorithm will make that the human players could not use the same strategy to win the game. The agents will learn from the human player as the human players will learn from the AI. • The change of behavior of the IRBREL agents will give the video game an additional challenge. This additional challenge will help to avoid that human players get bored of the video game too quickly. Consequently, the life time of the video game will be increased. • If the life time of the video game is increased and it keeps popular for a longer time frame, then the developers could sell more copies of the video game. 6.3 Future Work At this moment, the IRBREL algorithm only works in Real Time Strategy Video Games that are completely observable and deterministic. Besides the IRBREL algorithm needs a module that generates the world states and another module that generates the utility to be in the current world state. For future work, it is intended to modify the IRBREL algorithm to make it able to work in a more wide range of Real Time Strategy Video Games. The completely observable environment is a restriction that helps the IRBREL algorithm to learn why the world changes. Using the IRBREL algorithm in not completely observable environments could lead to the generation of superstitions, like the dance to make rain superstition or the cargo cult described in chapter 1. The learning of that kind knowledge should have an evolutionary advantage. Otherwise, the humans 73 could have evolved to avoid those situations. Notwithstanding, in order to handle that kind of knowledge, the IRBREL algorithm will need several modifications. The Science says that everything have a scientific explanation. Nonetheless, if there are not known or are not observable all the factors that are involved, then there is not enough knowledge to learn accurate cause-effect knowledge. Another additional skill of the IRBREL algorithm could be its use in non-deterministic environments. The IRBREL algorithm will need to know that sometimes the consequences of its actions could be different. To manage non-deterministic environments could be used inference rules that could have multiple conclusions. These multiple conclusions should have their own utility value and a probability value in which the consequences specified by the conclusion come true. The IRBREL algorithm is only able to handle integer values in its variables. Nevertheless, the IRBREL algorithm could be modified to be able to use real values in its variables. The inference rules could be adapted to handle Fuzzy logic. This could lead to a new way to use and learn strategies. The abstraction of knowledge is one of the best capabilities of the humans. However, the IRBREL algorithm does not have that capability yet. The IRBREL algorithm only works with the variables given by the world state generator. It could be created an additional module, for the IRBREL algorithm, that learns to generate abstract variables. That way the IRBREL algorithm could use a single abstract variable that is more representative than a group of simple variables. Learning to generate abstract variables could lead to simpler inference rules that become accurate quickly. The IRBREL algorithm is designed to be used as the AI of an agent. Notwithstanding, if a video game uses many copies of the same agent (like a group of soldiers), then the learned knowledge of one agent could be shared to the other agents. Each agent will have a different perspective of the world and different experiences. If multiple agents are set to learn at the same time and share their knowledge, then the learning process could be speeded up. The use of inference rules to represent the learned knowledge helps the IRBREL algorithm to quickly decide which strategy execute. Instead of analyzing the whole world state space, the IRBREL algorithm just needs to look which inference rule fires. Then, from the inference rule that fire, executes the inference rule that gives the agent the highest utility value. This process saves a lot of time because instead of analyzing multiple variables and world states, the IRBREL algorithm only needs to analyze two values: The Boolean value that indicates if the inference rule fired and the utility value. Nonetheless, this advantage is limited by the available hardware. If a computer only has one microprocessor, then that microprocessor needs to check if an inference rule fires or not. Then, the microprocessor needs to find, form the inference rules that fire, the one with the best utility value. Besides, that microprocessor must handle the video game’s graphics, manage the input of human players and manage 74 other agents that could be also in that game. To solve the hardware problem, could be created a hardware module that manages the inference rules. Instead of using inference rules defined in software, could be used inference rules defined in hardware. This hardware module could use a variant of PALs and GALs technology to create, modify and update inference rules. That way could be checked almost instantly which inference rule fired and which one has the best utility value. This hardware module could be also adapted to be used in robotics. 75 Bibliography [1] RoboCup. http://www.robocup.org/, 2008. [2] R. C. Atkinson and R. M. Shiffrin. Human memory: A proposed system and its control processes. The Psychology of Learning and Motivation, Vol 2. London: Academic Press, 1968. [3] A. D. Baddeley, N. Thomson, and M. Buchanan. Word length and the structure of short term memory. pages 575–589, 1975. [4] David M. Bourg and Glenn Seemann. AI for Game Developers: Creating Intelligent Behavior in Games. O’Reilly Media, Inc, 2004. [5] M. Brockington and M. Darrah. How not to Implement a Basic Scripting Language. AI Game Programming Wisdom, 2002. [6] M. Buro. Rts games as test-bed for real-time ai research. pages 481–484, 2003. [7] Edward Callaway. Fino-escalar la especificidad de redes corticales depende de tipo y de conectividad inhibitorios de la célula. Instituto de Salk, 2005. [8] M. Campbell, A. J. Hoane Jr., and F. H. Hsu. Deep blue. pages 57–83, 2002. [9] Alex J. Champandard. AI Game Development: Synthetic Creatures with Learning and Reactive Behaviors. New Riders Publishing. [10] D. P. de Farias and N. Megiddo. Learning and teaching in repeated games: A machine learning approach to long-term best-response play. 2003. [11] J. Schmidhuber F. Gomez and and R. Miikkulainen. Efficient non-linear control through neuroevolution. pages 654–662, 2006. [12] J. Furnkranz. Machine learning in games: A survey in machines that learn to play games. pages 11–59, 2001. [13] Joaquı́n Fuster. Cortical Dynamics of Working Memory. IBM Almaden Institute, 2006. 76 [14] Hisashi Handa and Satoshi Kajiyama. Decision-making of football agents with support vector machine. 2007. [15] J. Haugeland. Artificial intelligence: The very idea. 1985. [16] Judy Inglis. Cargo cults: The problem of explanation. 1957. [17] Holger Jebens. Cargo, cult, and culture critique. 2004. [18] Y. Jiang, J. Jiang, and I. J. Palmer. Computerized interactive gaming via supporting vector machines. 2008. [19] J. Ma and S. Perkinsn. Time-series novelty detection using one-class support vector machines. pages 1741–1745, 2003. [20] Yoichiro Matsuno, Tatsuya Yamazaki, Jun Matsuda, and Shin Ishii. A multiagent reinforcement learning method based on the model inference of the other agents. pages 67–76, 2002. [21] Risto Miikkulainen. Creating intelligent agents in games. 2006. [22] A. Nareyek. Intelligent agents for computer games. pages 414–422, 2002. [23] Nick Palmer. Machine learning in games development. AI Depot, 2008. [24] Marc Ponsen and Pieter Spronck. Improving adaptive game ai with evolutionary learning. pages 389–396, 2004. [25] Marc Ponsen and Pieter Spronck. Improving adaptive game ai with evolutionary learning. pages 389–396, 2004. [26] S. J. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Pearson Education, Inc, 2005. [27] J. Schaeffer. One jump ahead. 1997. [28] J. Schaeffer. A gamut of games. pages 29–46, 2001. [29] B. Scott. The illusion of intelligence. pages 16–20, 2002. [30] David Silver, Richard Sutton, and Martin Muller. Reinforcement learning of local shape in the game of Go. 2007. [31] P. Spronck, I. Sprinkhuizen-Kuyper, and E. Postma. Improving opponent intelligence through offline evolutionary learning. pages 20–27, 2003. 77 [32] Pieter Spronck, Marc Ponsen, Ida Sprinkhuizen-Kuyper, and Eric Postma. Adaptive game ai with dynamic scripting. Technical report, Machine Learning, Vol. 63, No. 3, 2006. [33] K. O. Stanley, B. D. Bryant, and R. Miikkulainen. Real-time neuroevolution in the nero video game. pages 653–668, 2005. [34] K. O. Stanley and R. Miikkulainen. Evolving neural networks through augmenting topologies. pages 99–127, 2002. [35] P. Tozour. The evolution of game ai. pages 3–15, 2002. [36] P. Tozour. The Perils of AI Scripting. AI Game Programming Wisdom, 2002. [37] Peter Worsley. The trumpet shall sound: a study of cargo cults in melanesia. 1957. [38] X. Yao. Evolving artificial neural networks. pages 1423–1447, 1999. 78
© Copyright 2026 Paperzz