Games Computers (cannot) Play
Graham Kendall
Automated Scheduling, Optimisation and
Planning Research Group (ASAP)
MIU, July 2004
Games Computers (cannot) Play
Contents
Checkers: Why was it considered “beaten”?
Two approaches to Checkers
Poker (if time)
Games Computers (cannot) Play
1959. Arthur Samuel started to look at
Checkers2
The determination of weights through selfplay
39 Features
Included look-ahead via mini-max
2
Samuel A. Some studies in machine learning using the game of checkers. IBM J. Res. Develop. 3 (1959), 210-229
Games Computers (cannot) Play
Samuels’s program defeated Robert Nealy,
although the victory is surrounded in
controversy
Was he state champion?
Did he lose the game or did Samuel win?
Games Computers (cannot) Play
Checkers Starting Position
1
5
2
6
9
21
15
18
19
26
30
12
16
23
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Checkers Moves
1
5
2
6
9
Pieces10 can only move
11
diagonally forward
17
21
15
18
19
26
30
12
16
23
22
25
29
8
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Jumps are forced
Checkers Forced Jumps
1
5
2
6
9
21
15
18
19
26
30
12
16
23
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Red (Samuel’s Program)
K1
5
2
6
9
21
15
18
22
25
29
11
10
17
8
7
14
13
4
3
12
16
19
20
23
Getting
to the back24 row
gives a King
28
26
27
30
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program)
1
5
2
6
4
3
8
7
Forced Jump
9
17
21
18
19
26
30
12
16
23
22
25
29
15
14
13
11
10
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program)
1
5
2
6
9
21
15
18
19
26
30
12
16
23
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Strong
(Try
to
keep)
Red (Samuel’s Program)
Trapped
1
5
2
6
9
21
15
18
19
26
30
16
23
22
25
29
11
10
17
8
7
14
13
4
3
24
28
27
31
White (Nealey)
Only
12
advance
to Square
28
20
32
Games Computers (cannot) Play
Red (Samuel’s Program)
1
What Move?
5
2
6
9
21
15
18
19
26
30
12
16
23
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program)
1
5
2
6
9
21
15
18
19
26
30
12
16
23
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program)
1
5
2
6
9
21
15
18
19
26
30
12
16
23
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
This was a very poor move.
It allowed Samual to retain his “Triangle
of Oreo”
AND.. By moving his checker from 19 to
24 it guaranteed Samuel a King
This questioned how strong a player
Nealy really was
Games Computers (cannot) Play
Red (Samuel’s Program)
1
5
2
6
9
21
15
18
19
26
30
12
16
23
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
This was a very poor move.
It allowed Samual to retain his “Triangle
of Oreo”
AND.. By moving his checker from 19 to
24 it guaranteed Samuel a King
This questioned how strong a player
Nealy really was
Games Computers (cannot) Play
Red (Samuel’s Program)
1
5
2
6
9
21
15
18
19
26
30
12
16
23
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
21
15
18
16
23
26
30
12
19
K
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
11
10
17
21
15
18
26
30
16
23
What Move (5, 13 or 16)?
29
12
19
K
22
25
8
7
14
13
4
3
31
White (Nealey)
20
24
28
27
32
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
21
15
18
16
23
26
30
12
19
K
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
13
29
11
10
K
22
25
12
16
19
23
26
30
8
7
16-12 then
5-1, Chinook
15
14
said would 18be a draw
17
21
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
21
15
18
16
23
26
30
12
19
K
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
21
15
18
16
23
26
30
12
19
K
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
21
15
18
16
23
26
30
12
19
K
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
21
15
18
16
23
26
30
12
19
K
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
21
15
18
16
23
26
30
12
19
K
22
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
13
21
11
15
30
19
23
26
12
16
K
18
22
25
29
8
7
10
This14
checker
17
is lost
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
9
15
26
30
19
23
22
12
16
K
18
25
29
11
10
17
21
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
K
1
5
2
6
9
11
10
17
21
8
7
15
14
13
25
19
23
26
30
20
24
28
27
What Move (3, 6 or 19)?
29
12
16
K
18
22
4
3
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
K
1
5
2
6
9
13
21
11
10
22
25
29
8
7
14
This checker
17 could run 18
(to 10) but..
15
12
16
K
19
23
26
30
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
K
1
5
2
6
9
15
26
30
19
23
22
12
16
K
18
25
29
11
10
17
21
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
K
1
5
2
6
9
21
26
30
16
19
23
22
12
K
15
18
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
K
Forced
Jump
1
5
2
6
9
21
26
30
16
19
23
22
12
K
15
18
25
29
11
10
17
8
7
14
13
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
17
21
26
30
16
19
23
22
12
K
15
18
25
29
11
10
14
13
8
7
K
9
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
5
2
6
17
21
26
30
16
19
23
22
12
K
15
18
25
29
11
10
14
13
8
7
K
9
4
3
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Red (Samuel’s Program) : After Move 25
1
Victory
5
6
9
17
21
11
15
18
19
26
30
12
16
23
22
25
8
7
14
4
3
10
K
13
29
2
20
24
28
27
31
White (Nealey)
32
Games Computers (cannot) Play
Two Mistakes by Nealy
Allowing Samuel to get a King
Playing a move that led to defeat when
there was a draw available
Games Computers (cannot) Play
The next year a six match rematch was won by
Nealy 5-1.
Three years later (1966) the two world
championship challengers (Walter Hellman
and Derek Oldbury) played four games each
against Samuel’s program. They won every
game.
Games Computers (cannot) Play
Checkers
Chinook
Blondie 24 (aka Anaconda)
Games Computers (cannot) Play
Types of Games
Perfect
Each Player has complete knowledge of the
game state
Usually only two players, who take
alternate turns
Examples include Chess, Checkers, Awari,
Connect-Four, Go, Othello
Games Computers (cannot) Play
Types of Games
Imperfect
Some of the game state is hidden
Examples include Poker, Cribbage, Bridge
Games Computers (cannot) Play
Types of Games
Games with an element of chance
The game moves have some stochastic
element
For example, Backgammon
Games Computers (cannot) Play
Types of Games
Solved or
Cracked
Over
World
Champion Champion
Connect-Four
Checkers
(8x8)
Chess
Qubic
Othello
Backgammon
GrandMaster
Go (9x9)
Amateur
Go (19x19)
Nine Men’s
Morris
Go_moku
Awari
6
Jaap van den Herik H., Uiterwijk and van Rijswijck J. Games Solved:
Now and in the future. Artificial Intelligence 134 (2002) 277-311
Games Computers (cannot) Play
Case Study 1: Checkers
Samuel’s work, perhaps, restricted the
research into Checkers until 1989 when
Jonathan Schaeffer began working on
Chinook
He had two aims
To develop the worlds best checkers player
To “solve” the game of checkers
Games Computers (cannot) Play
Case Study 1: Checkers
Chinook, at its heart, had an evaluation
function
Piece count (+30% for a King)
Runaway checker
“Dog Hole”
The weights were hand-tuned
Games Computers (cannot) Play
Case Study 1: Checkers
Opening game database from published work
(with corrections they found)
Initially 4000 openings, leading to an eventual
40,000
“Cooks” – innovative lines of play that could
surprise an opponent
The aim was to take opponents into unknown
territory
Games Computers (cannot) Play
Case Study 1: Checkers
Endgame database: Started writing in May
1989
The 8-piece endgame database finished on
February 20th 1994
Games Computers (cannot) Play
Case Study 1: Checkers
1
2
3
4
5
6
7
8
120
6,972
261,224
7,092,774
148,688,232
2,503,611,964
34,779,531,480
406,309,208,481
Games Computers (cannot) Play
Case Study 1: Checkers
9
10
11
12
13
14
15
16
4,048,627,642,976
34,778,882,769,216
259,669,578,902,016
1,695,618,078,654,976
9,726,900,031,328,256
49,134,911,067,979,776
218,511,510,918,189,056
852,888,183,557,922,816
Games Computers (cannot) Play
Case Study 1: Checkers
17
18
19
20
21
22
23
24
TOTAL
2,905,162,728,973,680,640
8,568,043,414,939,516,928
21,661,954,506,100,113,408
46,352,957,062,510,379,008
82,459,728,874,435,248,128
118,435,747,136,817,856,512
129,406,908,049,181,900,800
90,072,726,844,888,186,880
500,995,484,682,338,672,639
Games Computers (cannot) Play
Case Study 1: Checkers
With a 4-piece database Chinook won the 1989
Computer Olympiad
In the 1990 US National Checkers
Championship Chinook was using a 6-piece
database.
It came second, to Marion Tinsley, defeating
Don Lafferty on the way who was regarded at
the worlds second best player.
Games Computers (cannot) Play
Case Study 1: Checkers
Marion Tinsley
Held the world championship from 1951 to
1994
Before playing Chinook, Tinsley only lost 4
competitive games (no matches)
Games Computers (cannot) Play
Case Study 1: Checkers
The winner of the US Championship has the
right to play for the world championship.
Finishing second (with Tinsley first) entitled
Chinook to play for the world championship
The American Checkers Federation (ACF) and
the European Draughts Association (ADF)
refused to let a machine compete for the title.
Games Computers (cannot) Play
Case Study 1: Checkers
In protest, Tinsley resigned
The ACF and EDF, created a new world
championship, “man versus machine” and
named Tinsley as the world champion.
At this time Tinsley was rated at 2812,
Chinook was rated at 2706
Games Computers (cannot) Play
Case Study 1: Checkers
The match took place 17-29 Aug 1992.
The $300,000 computer used in the
tournament ran at about half the speed of a
1GHz PC
The match finished 4-2 in favour of Tinsley
(with 34 draws)
Games Computers (cannot) Play
Case Study 1: Checkers
A 32 game rematch was held in 1994
8-piece end game
Processors four times as fast (resulted in a
factor of 2 speed up due to more complex
evaluation function and the overhead of
parallel processing)
Opening book of 40,000 moves
In preparation Chinook no losses in 94 games
against Grandmasters
Games Computers (cannot) Play
Case Study 1: Checkers
Six games in (1-1, with 4 draws) Tinsley
resigned for health reasons. His symptoms
were later diagnosed as pancreatic cancer.
Tinsley died on 3rd April 1995 (aged 68).
Undoubtedly the best player ever to have lived
Chinook was crowned the man versus machine
champion. The first automated game player to
have achieved this.
A 20-match with Don Lafferty resulted in a
draw (1-1, with 18 draws)
Games Computers (cannot) Play
Case Study 1: Checkers
…defeating the world who
Opening Game Database
had held the title
(40,000) moves
for 40 years
Schaeffer
J. One
Jump
Ahead:
Won the
World
(Man
Challenging
Human
Supremacy
Versus
Machine)
in checkers,
Springer, 1997
Championship
in 1994…
Marion Tinsley lost his 5th,
End Game Database
6th and 7th games to
(8-pieces)
Chinook
Games Computers (cannot) Play
Case Study 2: Anaconda
Project started in the summer of 1998,
following a conversation between David Fogel
and Kumar Chellapilla
It was greatly influenced by the recent defeat
of Kasparov by Deep Blue
Chess was seen as too complex so “draughts”
was chosen instead
The aim is to evolve a player – rather than
build in knowledge
Games Computers (cannot) Play
Case Study 2: Anaconda
Reject inputting into a neural network what
humans think might be important
Reject inputting any direct knowledge into the
program
Reject trying to optimise the weights for an
evaluation function
Games Computers (cannot) Play
Case Study 2: Anaconda
The Gedanken Experiment
I offer to sit down and play a game with you.
We sit across an 8x8 board and I tell you the
legal moves
We play five games, only then do I say “You
got 7 points.”I don’t tell you if you won or lost
We play another five games and I say “You got
5 points”
You only know “higher is better”
Games Computers (cannot) Play
Case Study 2: Anaconda
The Gedanken Experiment
How long would it take you to become an
expert at this game?
We cannot conduct this experiment but we can
get a computer to do it
Games Computers (cannot) Play
Case Study 2: Anaconda
Samuel’s Challenge: “Can we design a program
that would invent its own features in a game of
checkers and learn how to play, even up to the
level of an expert?”
Games Computers (cannot) Play
Case Study 2: Anaconda
Newell’s Challenge: “Could the program learn
just by playing games against itself and receiving
feedback, not after each game, but only after a
series of games, even to the point where the
program wouldn’t even know which programs had
been won or lost?”
Newell (and Minsky)7 believed that this was
not possible, arguing that the way forward was
to solve the credit assignment problem.
7
Minsky M. Steps Towards Artificial Intelligence. Proceedings of the IRE, 1961, 8-30
Games Computers (cannot) Play
Later changed to
Case Study 2: Anaconda
an explicit piece
differential
I1
.
.
.
HL11
.
.
.
HL21
.
.
.
O
Evaluation
used for
MiniMax
I32
HL140
HL210
# weights=1741
Games Computers (cannot) Play
Case Study 2: Anaconda
1
2
5
6
9
21
18
19
26
30
12
16
23
22
25
29
11
15
14
17
8
7
10
K
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Case Study 2: Anaconda
+1
1
+1
7
+1
8
-K
14
-1
25
-1
27
All other neurons have an value of zero
Games Computers (cannot) Play
Case Study 2: Anaconda
Algorithm
Initialise 30 Networks
Each network played 5 games as red against
random opponents
Games were played to completion or until 100
moves had been made (a draw)
+2 for a win, 0 for a draw, -1 for a loss
15 best performing networks were saved for
the next generation and copies were mutated
Games Computers (cannot) Play
Case Study 2: Anaconda
Observations
The points for a win, lose draw were set such
that wins were encouraged. No
experimentation with different values were
tried
Players could play a different number of
games. This was, purposefully, not taken into
account
Mutation was carried out using an
evolutionary strategy
Games Computers (cannot) Play
Case Study 2: Anaconda
After 10 Generations
After 10 generations the best neural network
was able to beat both its creators and a simple
(undergraduate project) program which, by
the authors admission was “weak”
Note: 400MHz PC
Games Computers (cannot) Play
Case Study 2: Anaconda
ACF Ratings
Grand (Senior) Master
2400+ Class E
1000-1199
Master
2200-2399 Class F
800-999
Expert
2000-2199 Class G
600-799
Class A
1800-1999 Class H
400-599
Class B
1600-1799 Class I
200-399
Class C
1400-1599 Class J
<200
Class D
1200-1399
Games Computers (cannot) Play
Case Study 2: Anaconda
After 100 Generations
Playing on zone.com
Initial rating = 1600
Beat a player ranked at 1800 but lost against a
player in the mid 1900’s
After 10 games their ranking had improved to
1680. After 100 games it had improved to 1750
Typically a 6-ply search but often 8-ply
Games Computers (cannot) Play
Case Study 2: Anaconda
Observations
The highest rating it achieved was 1825
The evolved King value was 1.4, which agrees
with perceived wisdom that a king is worth
about 1.5 of a checker
In 100 generations a neural network had been
created that was competitive with humans
It surpassed Samuel’s program
The challenge set by Newell had been met
Games Computers (cannot) Play
Case Study 2: Anaconda
The Next Development
Alpha-Beta Pruning introduced and evolved
over 250 generations
Over a series of games, Obi_WanThe Jedi
defeated a player rated at 2134 (48 out of
40,000 registered) and also beat a player rated
2207 (ranked 18)
Final rating was 1902 (taking into account the
different orderings of the games)
Games Computers (cannot) Play
Case Study 2: Anaconda
The Next Development
Spatial nature of the board was introduced as
at the moment it just “saw” the board as a
vector of length 32
Games Computers (cannot) Play
Case Study 2: Anaconda
1
5
2
6
9
17
21
36 3x3
Overlapping
squares
10
29
26
12
16
19
23
22
30
11
15
18
25
8
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Case Study 2: Anaconda
1
5
2
6
9
17
21
25 4x4
Overlapping
squares
15
18
26
30
12
16
19
23
22
8
11
10
25
29
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Case Study 2: Anaconda
1
5
2
6
9
17
21
16 5x5
Overlapping
squares
10
29
26
12
16
19
23
22
30
11
15
18
25
8
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Case Study 2: Anaconda
1
5
2
6
9
17
21
9 6x6
Overlapping
squares
10
29
26
12
16
19
23
22
30
11
15
18
25
8
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Case Study 2: Anaconda
1
5
2
6
9
17
21
4 7x7
Overlapping
squares
10
29
26
12
16
19
23
22
30
11
15
18
25
8
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Case Study 2: Anaconda
1
5
2
6
9
17
21
1 8x8
Overlapping
squares
10
29
26
12
16
19
23
22
30
11
15
18
25
8
7
14
13
4
3
20
24
28
27
31
32
Games Computers (cannot) Play
Case Study 2: Anaconda
The Next Development
36+25+16+9+4+1 = 91 inputs
5,046 weights
Games Computers (cannot) Play
Case Study 2: Anaconda
36 3x3
I1
HL1
HL2
HL3
(91 nodes)
(40 nodes)
(10 nodes)
25 4x4
.
.
.
9 6x6
I32
4 7x7
16 5x5
O
1 8x8
Sum of 32 Board Inputs
# weights=5046
Games Computers (cannot) Play
Case Study 2: Anaconda
2 months and 230 generations later!!
After 100 games the rating was 1929
A 27 point increase over the previous network.
Nice but not decisive
Maybe it was due to there being three times
more weights but the training period was the
same?
Games Computers (cannot) Play
Case Study 2: Anaconda
6 months and 840 generations later!!
After 165 games it was rated at 2045.85 (sd
33.94)
Rated in the top 500 at zone.com (of the
120,000 players now registered)
That is better than 99.61% of the players
Games Computers (cannot) Play
Case Study 2: Anaconda
Playing Chinook8
In a ten match series against Chinnok novice
level it had two wins, two losses and 4 draws
Fogel D. B. and Chellapilla K. Verifying Anaconda’s expert rating by competing against Chinook: experiments in
co-evolving a neural checkers player, Neurocomputing 42 (2002) 69-86
8
Games Computers (cannot) Play
Case Study 2: Anaconda
Blondie
The neural checkers player went through a
number of names
David0111
Anaconda
Blondie24
Games Computers (cannot) Play
Case Study 2: Anaconda
Games Computers (cannot) Play
Case Study 2: Anaconda
 References
 Fogel D.B. Blondie24: Playing at the Edge of AI, Morgan Kaufmann, 2002
 Fogel D. B. and Chellapilla K. Verifying Anaconda’s expert rating by competing
against Chinook: experiments in co-evolving a neural checkers player,
Neurocomputing 42 (2002) 69-86
 Chellapilla K and Fogel D. B. Evolving neural networks to play checkers without
expert knowledge. IEEE Trans. Neural Networks 10(6):1382-1391, 1999
 Chellapilla K and Fogel D. B.Evolution, neural networks, games, and intelligence,
Proc. IEEE 87(9):1471-1496. 1999
 Chellapilla K and Fogel D. B. Evolving an expert checkers playing program without
relying on human expertise. IEEE Trans. Evolutionary Computation, 2001
 Chellapilla K and Fogel D. B. Anaconda Defeats Hoyle 6-0: A Case Study Competing
an Evolved Checkers Program Against Commercially Available Software. Proc. Of
CEC 2000:857-863