GDC 2017 研習分享
AI IN GAMES
台科資工 戴文凱博士
AI in Games
• The Hierarchical Portfolio Search AI
– http://www.gdcvault.com/play/1024207/Playing-Your-Cards-Right-The
• Unproven Game Ai Techniques That Might Possibly Work
– http://www.gdcvault.com/play/1024217/Crackpot-AI-Devs-UnprovenGame
• Techniques For Leveraging Ai In Content Generation
– http://www.gdcvault.com/play/1024146/PCG-Shotgun-6-Techniques-for
wktai@GAME Lab
3
The Hierarchical Portfolio
Search AI
Introduction to Prismata
• A strategy game developed by Lunarch Studios
– Combines “concepts from real-time strategy games,
collectible card games, and
table-top strategy games
– http://www.prismata.net
• Prismata in 60 Seconds
• Prismata in 600 Seconds
wktai@GAME Lab
5
Prismata- Game Properties
• Two
Prismata
: focus on the more popular and competitive
• Alternating
: Players take
form of
performing moves
– Turns may consists of multiple
taken by the same player (such as buying units
or attacking
– The turn is over when the active player declares no additional actions and passes, or a
time limit is reached
• Perfect Information: All players have access to all of the game’s information
– No decks, hands, or fog of war to keep information secret from your opponent like in
some other strategy games.
• Deterministic - After the initial state (a random set of 5 or 8 units is added to
the base pool of purchasable units), all game rules are deterministic.
• Zero Sum: The outcome of a game of Prismata is a win, loss or a draw
(stalemate)
– A winner being declared if they have destroyed all enemy units
wktai@GAME Lab
6
Prismata- Game Properties
• Two player: focus on the more popular and competitive 1 vs. 1 form of
Prismata
wktai@GAME Lab
7
Prismata- Game Properties
• Alternating
: Players take
performing moves
– Turns may consists of multiple
taken by the same player (such as buying
units or attacking
– The turn is over when the active player declares no additional actions and passes, or
a time limit is reached
wktai@GAME Lab
8
Prismata- Game Properties
• Perfect Information: All
players have access to all
of the game’s
information
– No decks, hands, or
fog of war to keep
information secret from
your opponent like in
some other strategy
games.
wktai@GAME Lab
9
Prismata- Game Properties
• Deterministic - After the initial state (a random set of 5 or 8 units
is added to the base pool of purchasable units), all game rules
are deterministic.
wktai@GAME Lab
10
AI Design Goals- To provide the best
experience
• New Player Tutorial
– Aids new players in learning the
game rules and strategies
– Offer different difficulty settings
so that players have a gradual
introduction
• Experienced Player Training
– Experienced and competitive
players often want to practice
– Players can practice these
strategies with some resistance
wktai@GAME Lab
• Single Player Replayability
– Add replay value with more
dynamic, ensuring the player
doesn’t fight against the same
tactics every time they play
– Games may have properties that
need to be
over time
for strategic balancing
– Require maintenance every time
an object was updated,
valuable time for programmers
11
wktai@GAME Lab
12
Why is using Heuristic Search for
Strategy Game AI Difficult?
• With Lower Bound: 10120 States
wktai@GAME Lab
13
Why is using Heuristic Search for
Strategy Game AI Difficult?
wktai@GAME Lab
14
Hierarchical Portfolio Search (HPS)
• A bottom-up, two level hierarchical search system
• At the
– A portfolio of algorithms generate multiple
suggestions for each of several tactically
decomposed areas of the game turn
• At the
– All possible combinations of these
suggestions are iterated over by a high-level search
technique
• such as MinMax or Monte-Carlo tree search which makes the
final decision on which move to perform
wktai@GAME Lab
15
Basic Components For Most AI Systems
• State s = <t, U1, U2>, containing all
relevant game state information
– Current game time t
– Sets of units Ui under control of player i
• Unit u= <p, hp, ta, tm, type>
– Position p = <x, y> in R2
– Current hit points hp
– Time step when unit can next attack ta,
or move tm
• Move m = <a1,…, ak>, a sequence of
Actions ai
wktai@GAME Lab
16
Basic Components For Most AI Systems
• Player function p [m = p(s, U)]
– Input state s and units U under
player’s control
– Performs Move decision logic
– Returns move m generated by p at
state s
• Game function g [s’= g(s, p1, p2)]
– Initial state s and Players p1, p2
– Performs game rules / logic
– Returns final game state s’ (win,
lose, or draw)
wktai@GAME Lab
function
[m =
(s)]
– Input state s
– Performs decision logic for a subset of a
turn
– Returns partial Move m to perform at
state s
• Portfolio P = <
,
, …,
>
– Portfolio is a collection of Partial Player
functions
17
Hierarchical Porfolio Search in Prismata
• Game Phases in Prismata
Defense
• Develop algorithms (Partial Players)
to choose good actions for each
individual phase
• For example
Action
Buying
units
Breach
Using
abilities
– In the defense phase, have one Partial Player to minimize the amount of
resources you will lose if you block a certain way
– To maximize the amount of attack remaining to punish opponent
• Portfolio P = <
1,
– A set of Partial Players
wktai@GAME Lab
2,
3,
i
4>
corresponding to each of the four phases
18
Can use existing scripts/algorithms to
generate a Portfolio
wktai@GAME Lab
19
wktai@GAME Lab
20
wktai@GAME Lab
21
10120 => 48(2X3X4X2)
wktai@GAME Lab
22
High Level Search Algorithms
• Use any algorithm you like/want!
• Ex.
–
–
–
–
–
–
Monte-Carlo Tree Search
Alpha Beta
Random Selection
Hill-Climbing
Genetic Algorithms
Machine/Reinforcement Learning
wktai@GAME Lab
23
AI Configuration and Difficulty Settings
• Difficulty settings
–
–
–
–
–
–
Master Bot- using Monte-Carlo Tree Search
Expert Bot- performs a 2-ply alpha-beta search
Medium Bot- makes poor unit purchase choices
Easy Bot- makes moor defensive choices
Random Bot- random moves
Docile Bot- never attacks
• All of these difficulties were created by creating new combinations
of Partial Players within the AI settings file
wktai@GAME Lab
24
AI Configuration
• All AI components modularly described in a text configuration
file
– AI Components: Players, Partial Players, Portfolios,
States, and Tournaments
,
• Partial Players:
– Arranged via tactical category and can be combined in any order to form
full Players or Portfolios
:
– Parameters such as search depth, time limits, evaluation methods, and
portfolio move iterators are specified
wktai@GAME Lab
25
Benefits of HPS
• Search adapts to dynamic scenarios
– AI can rush, econ, defend (no hard-coding)
– Determined automatically from search
• Emergent behavior is scary
– HPS only selects from know set of moves
• Designer friendly
– Designers can tweak portfolios easily
wktai@GAME Lab
26
Experiments
• Difficulty Settings
– Test if the playing strength rank of the various difficulty settings matched their
descriptive rank
• Descriptions of each bot difficulty are as follows:
– Master - Uses a Portfolio of 12 Partial Players and does a 3000ms UCT search within
HPS
- Uses the same Portfolio as Master bot, does an millisecond UCT search
within HPS
- Uses the same Portfolio as Master bot, does an millisecond AlphaBeta
search within HPS
- Uses the same Portfolio as Master Bot, does a 2-ply
search within HPS
– Medium - Picks a random move from Master Bot’s Portfolio
- Medium, but with weaker defensive purchasing
– Random - All actions taken are randomly chosen until no more legal actions remain
and the turn is passed
wktai@GAME Lab
27
wktai@GAME Lab
28
wktai@GAME Lab
29
Experiments
• AI vs. Human Players
– The AI player was Master Bot, used UCT as its high-level search with a 3
second time limit
– AI was configured to secretly play games in the human ranked
matchmaking system over the course of a
hour period, approximate
games against human
• The bot achieved a ranking of Tier 6 with 48% progression
toward Tier 7
– Table 1, placing the bot’s skill within the top 25% of human players on
the Prismata ranked ladder
wktai@GAME Lab
30
wktai@GAME Lab
31
Final Thought
• “Can’t Spare Cycles”?
• Search isn’t limited to in game behavior
• Offline search for QA:
– Bug Testing
– Balance Testing
– Stress Testing
wktai@GAME Lab
Quiz: the most
fundamental
part of HPS?
32
References
• Alpha beta tree search
– https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning
– http://will.thimbleby.net/algorithms/doku.php?id=minimax_search_with_alph
a-beta_pruning
• Hill climbing algorithm
– https://en.wikipedia.org/wiki/Hill_climbing
• Elo rating system
– https://en.wikipedia.org/wiki/Elo_rating_system
• Symmetric Game playout
• State Evaluation
wktai@GAME Lab
33
Utility AI
Utility AI
• Demands for
with non-player characters
and
• As game developers wants more complex AI that features
emergent behavior and
under
incomplete information
• Killzone 2, F.E.A.R, Civilization, Quake bots, etc.
– Used a utility system for making dynamic tactical decisions for the AI
without knowing the game world in advance
wktai@GAME Lab
36
Finite State Machines (FSM)
• Pros
– Fantastic for simple AI
– Simply defining states, and defining conditions for
translating between states, AIs can be invoked in infinite
loops of decision-making
• Cons
– difficult to debug
• In professional games, FSMs can easily have hundreds of
states
• Damian Isla outlined this in detail regarding the AI of Halo
2 (GDC 2005)
wktai@GAME Lab
37
Finite State Machines (FSM)
• Evolved into
them easier to design and debug
to make
– Did not remove the problem that sooner or later
FSMs become unmanageable as they grow in size
and complexity
• This idea of hierarchical organisation of tasks,
lead to organising tasks in tree-like structures,
– a.k.a.
wktai@GAME Lab
38
Behavior Tree
• Pros
– Provide a framework for designing more comprehensible
and easier-to-read AIs than hierarchical FSMs
– The nicely organized tree makes for easier visual
debugging in practice
• Cons
– For very large behavior trees, the costs of evaluating the
whole tree can be prohibitive
• The idea of sub trees has been introduced
Complex Behavior Tree with
conditionals, concurrent
• However, this reintroduces some of the complexities associated (simultaneous) behaviors, and
with controlling and debugging transitioning between states in
a sequence of behaviors.
Finite State Machines
– The sub tree can continue executing without invoking the
whole tree, until some condition is met to exit the subtree
– Does not provide a model for improving decision-making
• The decision-making is locked to the conditional nodes,
without specifying how decisions are made to invoke different
subtrees.
wktai@GAME Lab
39
Features needed of state-of-the-art AI
• Requirements: AI can support more complex, deep and
immersive game worlds
– Games: Hitman, Witcher, Assassin’s Creed
• ||
|| is growing, and the demands for life-like behaviors are increasing
– Games: Call of Duty and The Division
– ||
|| and the requirements for deep, varying and complex
tactical behaviors are exploding
• Static, scripted game design CANNOT support the dynamic and
ever evolving scenarios
wktai@GAME Lab
40
Features needed of state-of-the-art AI
• AI can
–Give rise to emergent behavior,
–Make
in situations that the
game designer did not necessarily foresee,
–Express a large range of behaviors
to what
would be expected by a
wktai@GAME Lab
41
A New Paradigm for AI- Utility AI
• Works by identifying options available to the AI and selecting
the best option by scoring each option based on the
circumstance
• The urge can be different for each
desire.
– The urge to sleep can be based on the
time since the last nap
– The
energy can be based
on time of day, and
– The
can be based on the
time since the last meal
• The curves enable the Utility AI to
make decisions across a large spectrum
of inputs, and gives it a fuzzy-logic
quality
wktai@GAME Lab
42
Behaviors Trees vs. Utility AI
• An example of first-person shooter
• The AI has
:
to Enemy,
Fire at Enemy,
to Cover, and
Load Weapon.
• The AI has
to base its
decisions on:
(actions that should be executed for
different input parameters)
wktai@GAME Lab
43
Example: Behavior Tree
• An initial conditional would
determine what subtree to
traverse, based on whether the
gun is loaded or not.
• Subsequently,
be evaluated
can
• …
• Note that
– The order of input parameters
make the tree more or less
complex, although the resulting
action will be the same
wktai@GAME Lab
Getting to this stage from the table above is already
quite time-consuming and error-prone
44
Example: Utility AI
• Each action is evaluated
separately, and the highest
scoring action is chosen
• The Utility AI requires just
one table with 10 scorers to
be implemented
• Note that
– Hierarchy can be added to the
Utility AI as needed
– Actions can lead to other Utility
AI evaluations
wktai@GAME Lab
45
Utility AI
– Can be designed in natural language, easy for the AI programmer to speak
with game designers
• such as “If the AI is under fire, prioritize finding cover”
– No need to talk about arcane concepts such as conditions, states, sequences
and decorators
– Scorers can easily be added on top of the existing AI, making it easy to extend
AI functionality and fidelity.
• Better Quality
– The simplicity of design and the ease extension reduces bugs and dramatically
improves productivity
– Leaves more room for developing more complex and well-behaving AI within
the given budget and timeframe
wktai@GAME Lab
46
Potential Application areas of Utility AI
• Complex Behavior
• Prioritizing actions
• Tactical reasoning
• Goal-based behaviors
• Strategic planning
• Group-behavior
wktai@GAME Lab
• Dynamic, context-sensitive
dialogues
• Target prioritization
• Desires
• Emotions
• Simulation of human
behavior
47
Limitations of Utility AI
• As with most AI technologies, AI should implement the
specified by the game designers
• Hence, Utility AI can exhibit some emergent behavior based on
its ability to interpolate and make fuzzy decisions
– Still
that the AI developer who designed it
• To make smart decisions requires
ensure correct behavior
to
– Requires developer-time and game testing
• To avoid these limitations, toward machine learning
wktai@GAME Lab
48
References
• http://www.gdcvault.com/play/1012410/Improving-AI-DecisionModeling-Through
• http://www.gamasutra.com/blogs/JakobRasmussen/20160427/271188/A
re_Behavior_Trees_a_Thing_of_the_Past.php
• http://www.gamasutra.com/view/feature/130663/gdc_2005_proceeding_
handling_.php
• http://alumni.media.mit.edu/~jorkin/gdc2006_orkin_jeff_fear.pdf
• http://www.goodai.com/
• http://intrinsicalgorithm.com/
• https://docs.unrealengine.com/latest/INT/Engine/Blueprints/index.html
wktai@GAME Lab
49