Dynamic Task Allocation in
a turn based strategy game
Gilles Schtickzelle
September 2012
ULB
Problem Statement
• Creating an intelligent player for a turn-based
strategy game.
• Working Framework:
• Many possible challenges to meet:
o
o
o
o
Resource management
Adversarial planning
Spatial reasoning
…
A game of FreeCol
• Colonization of America
• Establish settlements, grow and develop them
• Victory: Declare independence & Beat the Royal
Expeditionary Force
Colony Management
• Assigning tasks to units for optimal resources production
Division of labor in insect
societies
• Ants and wasps colonies have efficient distributed task
allocation mechanisms through stygmergy.
Bonabeau, E., Theraulaz, G., & Deneubourg, J.-L. (1996).
Response Threshold
• Ants have probabilistic response to stimuli:
𝑆2
𝑃 𝑋=0 →𝑋=1 = 2
𝑆 + 𝜃2
𝑃 𝑋=1 →𝑋=0 =𝑝
• Varying threshold θ induces specialization
o Reduces switching costs
o Increases individual efficiency
From insects to games
• Ants/Wasps
Colony
• FreeCol
Colony
• Insects
• Units
• Tasks
• Resources
• Specialization
• Expert
units
Resources Dynamics
•
Surplus: Extra workers.
Shortage: Lose worker.
•
Freedom. 50% required to win. Gives bonus or
penalty to workers.
•
required to make hammers
•
Used to produce buildings or artileries
•
required to make tools
•
Used to produce buildings or artilleries
Allocation Mechanism
• One stimulus Sr for each resource r =
• One set of dynamic thresholds θri per unit i
𝛽𝑟 𝑆𝑟2
𝑃 𝑋𝑖 = 0 → 𝑋𝑖 = 𝑟 =
2
𝛽𝑟 𝑆𝑟2 + 𝜃𝑟𝑖
𝑃 𝑋𝑖 = 𝑟 → 𝑋𝑖 = 0 = 𝑝
𝛽𝑟 𝑆𝑟
𝑃𝑜𝑤𝑒𝑟𝑟𝑖 =
𝜃𝑟𝑖
Stimuli and Thresholds
• Simple computation rules for each stimulus
𝑆𝐹𝑂𝑂𝐷=
𝑆𝐻𝐴𝑀𝑀𝐸𝑅𝑆 =
100
𝑖𝑓(𝑆𝑢𝑟𝑝𝑙𝑢𝑠𝐹𝑂𝑂𝐷 < 0)
𝑆𝐹𝑂𝑂𝐷 + 1 𝑖𝑓(0 ≤ 𝑆𝑢𝑟𝑝𝑙𝑢𝑠𝐹𝑂𝑂𝐷 < 6)
1
𝑖𝑓 𝑆𝑢𝑟𝑝𝑙𝑢𝑠𝐹𝑂𝑂𝐷 ≥ 6
𝑆𝑡𝑜𝑟𝑎𝑔𝑒𝐻𝐴𝑀𝑀𝐸𝑅𝑆 ≤ 𝐵𝑢𝑖𝑙𝑑𝐻𝐴𝑀𝑀𝐸𝑅𝑆
𝑜𝑟 𝑆𝑡𝑜𝑟𝑎𝑔𝑒𝐿𝑈𝑀𝐵𝐸𝑅 ≥ 15
0 𝑖𝑓 𝑛𝑜𝑡
10
𝑖𝑓
• One set of dynamic thresholds θru per unit u
0.1
𝜃𝑟𝑢 =
𝑖𝑓 𝑢 𝑖𝑠 𝑒𝑥𝑝𝑒𝑟𝑡 𝑓𝑜𝑟 𝑟
200 − 𝐸𝑥𝑝𝑟
min(10,
+ 1) 𝑖𝑓 𝑛𝑜𝑡
20
• Genetic Algorithm to find appropriate scale factors
βr
Simple Scenario
AI goals
1. Reach the year 1776 with enough bells to be able
to declare independence.
2. Have the best defense possible to resist the attack
of the royal expeditionary force.
3. Allocate workers to
1. minimize famine
2. Keep the production modifier as high as possible
Results (Basic player)
MEAN (100 games)
Size
Famine
Military
100%
14
0
24
91.57 ± 2.39
15.12 ± 0.85
0.18 ± 0.09
15.99 ± 0.62
(B)Evolution of the production
modifier (AI vs Expert)
(C)Evolution of the number of
artilleries (AI vs Expert)
30
2
20
0
10
1
23
45
67
89
111
133
155
177
199
221
243
265
287
309
331
353
375
397
419
441
4
PRODUCTION MODIFIER (AI)
PRODUCTION MODIFIER (EXPERT)
0
1
24
47
70
93
116
139
162
185
208
231
254
277
300
323
346
369
392
415
438
EXPERT
Freedom %
ARTILLERIES (AI)
ARTILLERIES (EXPERT)
Planning approach
• Suboptimal allocation: building too early
(C)Evolution of the number of artilleries (AI vs Expert)
30
20
ARTILLERIES (AI)
10
ARTILLERIES (EXPERT)
1
19
37
55
73
91
109
127
145
163
181
199
217
235
253
271
289
307
325
343
361
379
397
415
433
451
0
• Two planning methods:
o Layered response threshold.
o Rule-based planning.
Planning approach
• Layered response threshold :
𝑃𝐺𝑅𝑂𝑊𝑇𝐻
2
𝑆𝐺𝑅𝑂𝑊𝑇𝐻
= 2
𝑆𝐺𝑅𝑂𝑊𝑇𝐻 + Θ2𝐺𝑅𝑂𝑊𝑇𝐻
o Use two sets of scale factors:
• Optimized for growth
• Optimized for production
• Rule-based planning :
𝑆𝐻𝐴𝑀𝑀𝐸𝑅𝑆
𝑆𝑡𝑜𝑟𝑎𝑔𝑒𝐻𝐴𝑀𝑀𝐸𝑅𝑆 ≤ 𝐵𝑢𝑖𝑙𝑑𝐻𝐴𝑀𝑀𝐸𝑅𝑆
𝑜𝑟 𝑆𝑡𝑜𝑟𝑎𝑔𝑒𝐿𝑈𝑀𝐵𝐸𝑅 ≥ 15
10 𝑖𝑓
=
𝑜𝑟 𝑆𝑖𝑧𝑒𝑐 > 6
0 𝑖𝑓 𝑛𝑜𝑡
Planning Results (1)
Layered AI
Rule-based AI
Evolution of the colony size
Evolution of the colony size
15
15
10
10
5
5
0
0
SIZE (MULTI LAYER AI)
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
381
400
419
438
457
20
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
381
400
419
438
457
20
SIZE (EXPERT)
SIZE (RULE BASED)
Evolution of the number of artilleries
Evolution of the number of artilleries
ARTILLERIES (MULTI LAYER AI)
ARTILLERIES (EXPERT)
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
381
400
419
438
457
30
25
20
15
10
5
0
1
20
39
58
77
96
115
134
153
172
191
210
229
248
267
286
305
324
343
362
381
400
419
438
457
30
25
20
15
10
5
0
SIZE (EXPERT)
ARTILLERIES (RULE BASED)
ARTILLERIES (EXPERT)
Planning Results (2)
Military
SoL %
Size
Famine
24
100
14
0
MEAN (BASIC)
15.99 ± 0.62
91.57 ± 2.39
15.12 ± 0.85
0.18 ± 0.09
MEAN (LAYERED)
17.49 ± 0.70
99.17 ± 0.65
14.53 ± 0.84
0.19 ± 0.10
MEAN (RULE BASED)
18.80 ± 0.54
98.22 ± 1.28
17.49 ± 1.03
0.16 ± 0.09
EXPERT
Statistics for 100 games with the simple scenario.
Modified Threshold rule
12
10
8
6
4
2
0
12
Jobs:
10 1=FOOD
8
2=BELLS
3=NONE
6
4 4=HAMMERS
5=LUMBER
2
6=TOOLS
0
7=ORE
1
18
35
52
69
86
103
120
137
154
171
188
205
222
239
256
273
290
307
324
Thresholds
Evolution of thresholds and jobs for unit 5 (non expert)
FOOD
BELLS
NONE
HAMMERS
LUMBER
TOOLS
ORE
jobs for unit 5
• Unit u produces resource r
𝜃𝑢𝑟 =
0.1 𝑖𝑓 𝑢 𝑖𝑠 𝑒𝑥𝑝𝑒𝑟𝑡 𝑓𝑜𝑟 𝑟
max 1, 𝜃𝑢𝑟 − 1
𝑖𝑓 𝑛𝑜𝑡
• Unit u does not produces resource r
𝜃𝑢𝑟 =
0.1 𝑖𝑓 𝑢 𝑖𝑠 𝑒𝑥𝑝𝑒𝑟𝑡 𝑓𝑜𝑟 𝑟
min 10, 𝜃𝑢𝑟 + 1
𝑖𝑓 𝑛𝑜𝑡
“State of the art” player
Evolution of unit's job over time
(single layer AI player)
8
6
4
2
0
1
19
37
55
73
91
109
127
145
163
181
199
217
235
253
271
289
307
325
343
361
379
397
415
433
451
Jobs:
1 = FOOD
2 = BELLS
3 = NONE
4 = HAMMERS
5 = LUMBER
6 = TOOLS
7 = ORE
unit1
unit2
unit3
unit4
unit5
unit6
unit7
unit8
unit9
unit10
Evolution of unit's job over time
(State-of-the-art AI player)
Jobs
1 = FOOD
2 = BELLS
3 = NONE
4 = HAMMERS
5 = LUMBER
6 = ORE
7 = TOOLS
8
6
4
2
0
1
19
37
55
73
91
109
127
145
163
181
199
217
235
253
271
289
307
325
343
361
379
397
415
433
451
• Modified Threshold
update rule +
rule-based planning
unit1
unit2
unit3
unit4
unit5
unit6
unit7
unit8
unit9
unit10
AI players comparison
AI players compared (number of military units)
30
30
25
Expert = 24
25
20
20
15
15
10
10
5
5
0
0
Basic
Layered
rule-based
State of the art
AI goal completion
EXPERT
MEAN (100 games)
Freedom %
Size
Famine
Military
100%
14
0
24
100% ± 0.92
20.21 ± 0.94
0.04 ± 0.07
20.99 ± 0.55
1. Reach the year 1776 with enough bells to be able
to declare independence.
2. Have the best defense possible to resist the attack
of the royal expeditionary force.
3. Allocate workers to
1. minimize famine
2. Keep the production modifier has high as possible
Conclusions
 Human-level performances can emerge from
simple rules, without cheating.
 Easy to implement (compared to traditional rule-based only
AI).
 Easy to tune down performances (if playing against nonexpert).
 Hybrid system (with planning instructions) improves
on basic RTM
− Tendency to chaos with large number of stimuli
− Difficult to extend to other game aspects (combat,
spatial reasoning, diplomacy,…).