School of Computing
FACULTY OF ENGINEERING
Tabletop Robotics: Board Games with a Baxter Robot
Pascal Alexander Siddons
Submitted in accordance with the requirements for the degree of
BSc Artificial Intelligence
Session 2014/2015
- ii -
The candidate confirms that the following have been submitted:
Items
Format
Recipient(s) and Date
Final Report (2 copies)
Report
SSO (13/05/15)
Final Report (digital)
Report
VLE (13/05/15)
Demonstration video
URL/File Download
Prof. Tony Cohn & Dr Yiannis
Gatsoulis (13/05/15)
Type of Project: Exploratory Software
The candidate confirms that the work submitted is their own and the appropriate credit has
been given where reference has been made to the work of others.
I understand that failure to attribute material which is obtained from another source may be
considered as plagiarism.
(Signature of student) ______________________________
© 2015 The University of Leeds and Pascal Alexander Siddons
- iii -
Summary
The aim of this project is to write software for a Baxter robot in order to make it capable of
playing a board game with a human player, playing fully legal and non-random moves with
some element of strategy. This requires a suitable game to be designed and prototyped with
components suitable to be handled by a Baxter robot.
- iv -
Acknowledgements
I would like to thank my project supervisors, Professor Tony Cohn and Dr Yiannis Gatsoulis
for all the advice, time and support they have given me throughout this project, as well as the
other researchers working in the robotics lab for putting up with me while I was there.
I would also like to thank my partner Paulina for keeping my sanity in check as the final
deadline loomed and playtesting the game with me, as well as Ben Hirsh for letting me park
outside his house, which is considerably closer to campus than mine.
I would like to thank the late Sir Terry Pratchett for his literature and words of wisdom, which
I grew up with and have instilled in me a great fascination with the wonderful and the out of
the ordinary, the cutting-edge and the awe-inspiring, without which I’d probably be doing a
project involving mid-range predictions in finance or something equally dreadful. The manner
in which he wrote about death (or rather, DEATH) also made his passing away partway through
this project much easier to cope with.
Finally, I would like to give thanks to Tom McMurchie and Calliope Games, for designing and
publishing Tsuro respectively, which heavily inspired the game that this project revolves
around, and has given me hours of enjoyment.
-v-
Table of Contents
Summary .............................................................................................................. iii
Acknowledgements ............................................................................................. iv
Table of Contents .................................................................................................. v
Chapter 1 Introduction.......................................................................................... 1
1.1
Aims ...................................................................................................... 1
1.2
Objectives .............................................................................................. 1
1.3
Minimum Requirements ......................................................................... 1
1.4
The Problem .......................................................................................... 1
1.5
Methodology .......................................................................................... 3
1.6 Project Management ................................................................................ 3
Chapter 2 Background And Literature Review ................................................... 6
2.1 Game AI ................................................................................................... 6
2.1.1 Games From a Computing Perspective ........................................ 6
2.1.2 Adversarial Searches and Pruning ................................................ 6
2.1.2 Machine Learning ......................................................................... 7
2.2 Machine Vision ......................................................................................... 8
2.3 Robotics ................................................................................................... 8
2.3.1 Robotic Behaviour Architectures ................................................... 8
2.3.2 Inverse Kinematics........................................................................ 9
Chapter 3 Materials ............................................................................................. 11
3.1 The Robot .............................................................................................. 11
3.2 The Game .............................................................................................. 12
3.2.1 Adapting the game to work with Lucas ........................................ 13
3.2.2 Strategies ................................................................................... 13
3.3 Original Concept..................................................................................... 15
3.4 Dealing With Limitations ......................................................................... 15
3.4.1 Vision .......................................................................................... 15
3.4.2 Mechanical ................................................................................. 17
3.5 Consequences of Compromise on the Game ......................................... 18
3.6 Breaking Down the Game Logic ............................................................. 18
3.7 Decisions on Development Tools ........................................................... 21
Chapter 4 Methods .............................................................................................. 22
4.1 The Vision System.................................................................................. 22
- vi 4.1.1 Block Identification ...................................................................... 22
4.1.2 Scanning the Board State ........................................................... 25
4.2 The Mechanical System ......................................................................... 26
4.3 The Game Logic System ........................................................................ 28
4.4 Putting It All Together ............................................................................. 30
Chapter 5 Analysis and Evaluation of the System............................................ 33
5.1 System Reliability ................................................................................... 33
5.2 Effectiveness .......................................................................................... 34
5.2.1 Effectiveness of the Vision Systems ........................................... 35
5.2.2 Effectiveness of the Mechanical Systems ................................... 35
5.3 Limitations Imposed By the System ............................................... 35
5.4 Gameplay Data ...................................................................................... 36
5.5 Discussion of Results ............................................................................. 38
5.5.1 Data Gathering Methods ............................................................. 38
5.5.2 Data Analysis .............................................................................. 38
5.6 Evaluation of the Project ......................................................................... 38
5.6.1 Project Schedule ......................................................................... 38
5.6.2 Choice of Development Tools ..................................................... 39
5.6.3 Decisions in Game and Component Design ................................ 39
5.6.4 Decisions in Software Design ..................................................... 39
Chapter 6 Conclusions ....................................................................................... 40
6.1 Conclusions ............................................................................................ 40
6.2 Future Work............................................................................................ 42
List of References ............................................................................................... 44
Appendix A Personal Reflection ........................................................................ 45
Appendix B Ethical Issues Addressed .............................................................. 48
B. i: Ethical issues ....................................................................................... 48
Appendix C Record of External Materials Used ................................................ 49
Appendix D Full Rules of the Game ................................................................... 50
D. i: Components.......................................................................................... 50
D. ii: The Tiles .............................................................................................. 50
D. iii: Setup ................................................................................................... 50
D. iv: Playing the game................................................................................. 50
D. v: Winning the Game ............................................................................... 51
D. vi: FAQ..................................................................................................... 51
- vii Appendix E Project Code and Other Materials of Interest ............................... 52
E. i: coords.py .............................................................................................. 52
E. ii: vision.py ............................................................................................... 52
E. iii: Demonstration Video ........................................................................... 62
E. iv: GitHub Repository ............................................................................... 62
-1-
Chapter 1
Introduction
In this section of the report, I give the reader an overview of the project and the planning that
went behind it.
1.1 Aims
The aim of this project is to design and implement a software system for a Baxter robot to
allow a human to play a simple board game with it.
A secondary aim of this project is for me to learn more about the field of robotics and extend
my knowledge of practical artificial intelligence, more specifically the application of game logic.
1.2 Objectives
The main objectives for this project are as follows:

I am to design or modify a game so that it is suitable for play with a Baxter robot.

I am to implement a computer vision system to allow the Baxter robot to play with no
prior knowledge of the game state or human input.

I am to implement an artificial intelligence (AI) system to allow the robot to play
competently.

I am to implement it in such a way as to allow full game play with a human player.
1.3 Minimum Requirements
The minimum requirements for this project are as follows:

I am to design or modify a game so that it is suitable for play with a Baxter robot.

I am to implement a system allowing the robot to play the game by itself.
1.4 The Problem
In order for Lucas to play the game, there are several tasks it needs to perform in order to
complete a game turn.

Tile Manipulation: Lucas needs to be able to accurately pick tiles up and place them
within a grid space
-2
Tile identification: Lucas needs to be able to identify which type of tile it has just picked
up.

Board state identification: Lucas needs to know what tiles have been placed in each
grid space, and what the implications of that would be on its potential moves.

Game logic: Lucas needs to be able to make a decision as to how to place the tile
based on the position of its piece and the possible paths on the tile.

Turns and player interaction: Lucas needs to be able to indicate when its turn is over
and be able to realise when its turn has come around again.

Detect game end: Lucas needs to be able to detect when the game is over, either
through all other players ejecting themselves from the board, or from Lucas sending
his own path off of the board.
In essence, this can be configured as a looping program with the game end being the break
condition. The structure of a turn loop is as follows:
1
Wait to receive turn start signal
o
2
Identify board state
o
3
The tile is placed in a specific place on the table for Lucas to identify it.
Identify the tile
o
5
If the game has ended during another player’s turn, celebrate victory
Draw a tile
o
4
This is triggered when Lucas detects a new game piece in the draw space.
Lucas needs to know which of the two types of tile it is.
Decide how to place the tile
o
With the “straight ahead” tiles there is only one option, but the diagonal tiles
can be placed in two different ways, going left or going right.
6
Move the tile to the appropriate grid square
7
Rotate the tile if necessary
o
As the diagonal tile has two possible outcomes (left or right), the tile may need
to be rotated to direct the path in the correct direction.
-3-
8
Place the tile
9
Check for game loss (sometimes a loss is unavoidable)
o
If Lucas has lost the game, be gracious in defeat
10 Send turn end signal
11 The loop begins anew.
1.5 Methodology
Throughout this project I intended to use primarily a heavy-planning, heavy-testing agile
modular approach to time management, with each section of the overall software solution
being designed, developed and tested sequentially. As time passed and the full solution came
together it became clear to me that some parts could not be tested without others. Pictured
below is the initial schedule for the project, in the form of a Gantt Chart (Fig. 1.1). The general
method I used for this project is to reduce each part of each task to its simplest form to produce
a clear, well defined and functional program with minimal margin for error in execution.
1.6 Project Management
As with any project, there are several things which must be taken into consideration,
specifically when working with one particular robot. The most important risk to consider is the
possibility of the robot breaking down and being non-functional for the delivery of the project.
In this case the contingency plan would be to have the project running on simulation software
rather than on the actual robot. Fortunately RethinkRobotics provides a Baxter package for
Gazebo1, multi-robot simulation software which allows simulation of interaction with 3D
environments. In the worst-case scenario (Lucas falls down some stairs or explodes or is
otherwise incapacitated for the project delivery), this is the course of action to take.
Other risks to consider would be risks to people near the robot, as the robot’s arms do not
stop when they come into contact with something or someone. To minimise personal risk, no
one should be within arm’s reach of the robot when it’s in operation, and the emergency stop
1
http://sdk.rethinkrobotics.com/wiki/Baxter_Simulator More information on the Gazebo
simulator for use with Baxter robots.
-4-
button should always be to hand, although the inbuilt safety measures in the Baxter Robot
should prevent anything grievous from happening. Just in case, however, the emergency
shutdown button is always close at hand during operation.
Due to Lucas’ tendency to attempt to move its arms through routes which include going
through other game pieces or the table, the code for moving the arms is not as elegant as it
could be, as this is compensated for by taking the arm more precisely through several small
movements in each step as opposed to one large sweeping movement, which holds the
potential for sowing chaos and destruction across the board to a minimum.
-5-
Figure 1.1: Gantt chart showing the initial planned schedule for the project
-6-
Chapter 2
Background And Literature Review
In this section of the report, I provide some background knowledge on the aspects of computer
science tackled in this project.
2.1 Game AI
2.1.1 Games From a Computing Perspective
Russell and Norvig, in their Artificial Intelligence: A Modern Approach define games as:
“competitive environments, in which the agents’ goals are in conflict, giving rise to adversarial
search problems” (Russel, 2010)
From a purely AI point of view, this makes perfect sense, and covers most of the points which
define the kind of games that researchers are generally interested in programming robots to
play, namely chess, backgammon and the game in this project.
2.1.2 Adversarial Searches and Pruning
Adversarial searches function best in two playing, zero-sum games of perfect information such
as chess, where everything is on the board and all of your opponents possible moves after
your next move can be determined in advance. This can lead to some rather enormous search
trees, especially in situations looking for optimal play. The term “Adversarial” comes from the
fact that these are most often applied to competitive games, usually where a robot is
attempting to best a human player, and the search algorithm will analyse possible moves from
both players. This is epitomised by the “Minimax” algorithm which reads through possibilities
for both players for several turns in advance, so called because it calculates the “max” or most
optimal move for the agent, leading to the “min” or least optimal move for the opponent.
As some games are so complex in nature that searching each individual possibility would take
so long as to be impractical (chess, for example, is usually estimated to have around 35100
nodes in a maximal search tree) (Russel, 2010), and a lot of these moves can, at an early
stage, be deemed to be purely detrimental to the state of play (such as a move which sends
the player off of the board). These branches can be removed before too much time is used to
search the rest of the tree, which can have a considerable impact as decision trees in game
solving algorithms expand exponentially with each future turn explored.
There are several factors which can influence the effectiveness of these search algorithms.
They are most effective in situations where the agent has omniscience of the environment, in
games such as chess or checkers where the entire board and all of the pieces which will ever
-7-
take place in the game are visible immediately. In the game presented in this project, however,
future prediction through a minimax algorithm would be redundant, due to the random nature
of the tile dealt, and the fact that there are a maximum of two decisions once a tile has been
drawn.
2.1.2 Machine Learning
Machine learning describes an aspect of computer science where a system learns or is taught
how to evaluate data according to certain weighted parameters. For example, a system for
filtering out news stories relevant to a particular subject could be taught by being fed a test
dataset with the articles with the specific keywords to look for already marked out. From this,
the machine learning algorithm can understand that the marked articles all contain something
in common which is what the system is to look out for. For a game implementation, this would
take the form of sets of moves and which moves lead to a more successful result. This type
of machine learning is known as supervised learning. Supervised learning often uses
Okham’s2 Razor (or lex parsimoniae) for deriving the shared attributes from a set of training
data. Okham’s Razor states that when presented with several hypotheses that obtain the
same result, the correct course of action would be to go with the simplest hypothesis which is
the consistent with the data. (Russel, 2010)
The other main method of machine learning implementation is reinforcement learning, where
the agent performs the task multiple times and analyses the data to see how it can improve
after each iteration. For example, in a machine learning algorithm for the game in this project,
the set of moves performed by both players would be stored and the game would receive a
value depending on how well the agent performed. Higher values would indicate a better
performance, and the dataset could be analysed to see which moves in which board spaces
lead to better results.
Both methods of machine learning have their advantages and disadvantages, and often,
especially in the field of games, semi-supervised learning is encountered, where a small
training data set is provided, and then many un-labelled sets of data (such as playing a game
repeatedly) are sorted.
2
Also seen spelled as “Occam’s”
-8-
2.2 Machine Vision
Machine vision describes the field of computer science used to allow automatic image analysis
and inspection. This has many different applications including robot guidance and medical
imaging. Many machine vision systems are designed to interpret images in real-time and feed
the results to another system to process, such as automatic license plate scanners and robotic
vision systems.
Many machine vision systems make use of such techniques as edge or contour detection in
identifying objects. These techniques make use of image analysis techniques and analyse
pixels in contrast with the surrounding pixels to determine where edges are formed in the
image. Contour detection works in a similar way but instead of simply returning which pixels
are “edge pixels”, returns the contours of objects within the image. (Hornberg, 2007)
Another approach taken for identifying objects is mask or template matching. In a template
matching approach, the program is given a template of an object which is then used to detect
any examples of that object in an image. Advanced variations of template matching also
include rotations and scaling of the template, to detect the desired object within an image even
if it is at an odd angle or depicted at a different size than the template model.
2.3 Robotics
2.3.1 Robotic Behaviour Architectures
All robotics system act in accordance to a certain “paradigm”. The most traditional robotics
paradigm is known as the sense-plan-act paradigm, which was used in some of the first
autonomous robots, namely Shakey, a robot which explored and mapped an environment.
(Nilsson, 1984)
Sense-Plan-Act is structured in just the way that it sounds, a paradigm composed of three
parts:
1: Sense – The agent examines the environment.
2: Plan – The agent takes this information and makes an informed decision on how to
proceed.
3: Act – The agent undertakes the action decided through the “plan” step.
-9-
The most basic form of Sense-Plan-Act is the Hierarchical or Deliberative paradigm (depicted
below in Fig 2.1), with a heavy focus on planning and gathering sensing data to update a
global world model. Other models exist, which are primarily based on reactions to sensory
triggers.
Sense
Plan
Act
Figure 2.1: Graphical representation of the Sense-Plan-Act Deliberative
paradigm.
Not long after the first practical implementations of Sense-Plan-Act, however, some
researchers (namely Rodney Brooks, MIT), argued that sense-plan-act was “detrimental to
the construction of real working robots” (Arkin, 1998) and advocated instead the use of a
layered “subsumption” architecture based instead on a purely reactive behaviour-based model
of operation.
Although Arkin’s points on the subsumption architecture were, and remain, valid, in my opinion
the architecture of the paradigm should be tailored to the intended purpose of the system. As
such, due to the nature of the task in this project, a turn-based game, I think that the most
suited paradigm for this is a traditional Deliberative Sense-Plan-Act paradigm, as at the start
of a game turn each player gathers all of the necessary information, plans what to do with this
information and then enacts the turn actions they choose to do. It seems to be the intuitive,
logical fit, as opposed to convoluting the process with a reaction-based behaviour model.3
2.3.2 Inverse Kinematics
The process involved in inverse kinematics usually begins with a kinematic analysis of the
robots mechanical system, that is, an analysis of the lengths and ranges of the links between
3
There are a plethora of other robotic behaviour paradigms out there, but the most common
comparison that seems to be drawn in academic literature is between these two, with
Sense-Plan-Act being the one that I have deemed most relevant to this project.
- 10 -
each joint in each limb. Kinematics deals with how to move the limbs in a certain range and
finding the end result, usually in terms of where the gripping mechanism at the end of the limb
will end up. Inverse kinematics is the application of the reverse. Given a point in the
environment, either in world coordinates or joint angles, inverse kinematic formulas are used
in order to coordinate the limb to move from its current position to the given world position.
There are several problems which often arise in inverse kinematics, the first and foremost
being that there are multiple solutions to achieve the desired result, and the system has to be
able to choose one. The criteria which is usually used in this decision is the solution that
presents the least amount of movement while still achieving the desired final location. This
ambiguity in the actual route the limb will take can lead to some unwanted effects, such as the
limb taking an unexpected route through some solid objects in the environment, as the route
which seems the closest to the robot may not be the route which would seem to be the closest
to a human, as the robot will choose the closest solution in “joint-space” rather than our
perspective of three-dimensional space (Craig, 2005).
- 11 -
Chapter 3
Materials
In this chapter I go into detail about the design process for the game and the decisions made
to make it suitable for Lucas and the project, as well as the decisions on how to interpret the
game logic and which development tools to use in the project.
3.1 The Robot
The robot used for this project is “Lucas”, the University Of Leeds School Of Computing’s
Baxter robot, manufactured by Rethink Robotics. Baxter robots are composed of a torso
containing their central processing system, a head with a screen for a face and two very mobile
arms, each with 7 degrees of freedom with a camera and parallel grippers on the ends.
Figure 3.1: Lucas in the laboratory.
Notable features of the Baxter robots over other industrial robots include the manually
controllable arms, which allows programs based around fixed positions (such as this project)
to be implemented more easily, as well as rigorous safety protocols hard-coded into its
operation, which prevent the arms from going through each other or surrounding objects.4
4
More information on RethinkRobotics’
http://www.rethinkrobotics.com/baxter/
Baxter
robots
can
be
found
here:
- 12 Baxter robots are produced primarily for use in manufacturing as an alternative to outsourcing
labour, more specifically for monotonous tasks such as line loading and packaging. Notable
features of the Baxter robots over other industrial robots include the manually controllable
arms, as well as the rigorous safety protocols hard-coded into its operation. They have also
found a place in many universities’ robotics laboratories due to the ease with which they can
be programmed and the low risk to personal safety in working with one in comparison to other
industrial robots.
3.2 The Game
The game I designed for this project is heavily based on Tsuro: The Game of the Path5. Tsuro
is played with a stack of tiles, a board, and one playing piece per player. The tiles each have
4 “paths” on them connecting 8 points. Gameplay starts by each player in turn placing their
first tile along the edge of a board and choosing which path to take. In subsequent turns, the
players continue placing tiles in such a way as to form a continuous path that their piece takes,
attempting to ensure that their piece stays on the board and does not collide with other players’
pieces (as this links up their paths, both of which start on a board edge, thus resulting in both
of their paths leading off of the board). A revised version of Tsuro, Tsuro Of The Seas, was
released in 2012, however the additional mechanics added in simply add another element of
chance into the game and as such have not been considered for this particular project. These
added rules also detract from the simplicity and elegance which makes Tsuro so appealing,
and Tsuro of the Seas is generally held in lower regard than the original.
Figure 3.2: A two-player game of Tsuro in
progress.
5
More
information
on
Tsuro
can
https://boardgamegeek.com/boardgame/16992/tsuro
be
found
here:
- 13 -
3.2.1
Adapting the game to work with Lucas
The game designed for this project is similar to – and inspired by - Tsuro, but simplified, with
only 2 paths and 4 points per tile, thus meaning there are only two possible types of tile, as
shown below in Fig 3.3:
Figure 3.3: Schematic for the two tile designs used in the revised game, with
connections to opposite sides on the left, and adjacent on the right.
The pieces are custom made from wood, to appropriate dimensions for Lucas to be able to
pick them up and manipulate them as needed, with electrical tape used for the colouring, due
to its low reflectivity and to get consistent shades of black and white.
The board is a 5 by 4 grid, a compromise between maintaining some complexity to the game,
and the limited reach of Lucas’ arms and the space needed for the larger tiles, with some
leeway included to accommodate for inaccuracy in placement, as well as to allow the arms of
the parallel grippers to move between tiles.
3.2.2 Strategies
There are two basic strategies in Tsuro, both of which can be developed upon, but the basic
premises of both of which can be applied to the game in this project as well.
The first of these is sticking to the edge (Fig 1.4): simply attempting to run around the edge of
the board is a good way to keep space from tiles other players have placed, in an effort to
“buy time” for yourself in order to survive longer in the game. However, with a small slip up or
bad luck of the draw it is very easy to end up with no choice but to send yourself off of the
edge.
- 14 -
Figure 3.4: Possible progression of play for a player
attempting to stick to the edge of the board.
The second strategy, the one which Lucas is programmed to follow in this project, is to stick
as close to the centre as possible. There is a “golden zone” (as seen in Fig. 2.5) in the middle
of the board where the player is at least 1 space away from the edge, and as such it becomes
much harder to throw oneself off the board with a single bad play. This strategy essentially
uses the outer regions of the board as “insurance”, and revolves around making sure that the
player stays as far from the edges as possible. However this is risky as the squares in the
centre of the grid are usually in high demand for these reasons.
Figure 3.5: Showing the "Golden Zone" or "Safe Zone" in the centre of the board.
Lucas will attempt to occupy these spaces for as much of the game as possible.
- 15 -
3.3 Original Concept
I had originally planned to program Lucas to play the full game of Tsuro, using pairings of
coloured squares to define potential paths. However, after discussing this with my supervisors
it became obvious that this was far too ambitious to achieve in the timescale for the project,
due to the enormously increased level of complexity and challenge represented in almost
every part of the project with the inclusion of 4 colour coded paths per tile.
Figure 3.6: Early prototype game pieces for the original concept,
comprised of Tsuro game pieces with path ends marked with
coloured tape.
3.4 Dealing With Limitations
Because the game had to be playable by a Baxter robot, there were several factors I had to
take into consideration. The main change I made was reducing the number of potential paths
on the block to 2, down from 4. This meant that, not only did I have to make custom pieces for
the game, I had to redesign the game around this.
3.4.1 Vision
Every tile that goes on the board needs to be scanned by Lucas, so that it can make a fully
informed decision, as a human player would. As such the game pieces had to be made in
such a way that they could be easily and, most importantly, accurately identified by a computer
vision system.
In order to make the blocks as easily identifiable as possible, the background of the blocks is
black, with the paths shown in white, so that a computer vision system could reduce them to
a binary image (where each pixel is black or white, no grey pixels) very easily with minimal
- 16 -
error. All three (from Lucas’ perspective) types of block are shown below in Fig. 3.76. Once
reduced to a binary image, it would be much easier to determine the pattern on the block.
Another factor affecting the vision element of the project is the size of the blocks. If the blocks
were too small, it would be harder to obtain a clear image of the block face.
Figure 3.7: The physical blocks used in the game. Block types (from
left to right) 1, 2 and 3 shown here.
The board is a black sheet of plywood, marked with an even 4x5 grid of squares, as shown
below in Fig. 3.8 (although the board seems blotchy, the lighter areas are still dark enough to
be classed as black in the binary image). The grid is marked in pencil, and as such is
imperceptible to Lucas (especially after applying smoothing and filtering small areas from the
image), but still easily visible to the human player. The board is black so as to make the
identification of the patterns on the block as easy as possible, and to make all of the vision
tasks easier and more accurate by providing a solid black background.
6
I will refer to the different block types by these designations throughout the course of the
report, whereas for the pieces, I will use the words block, piece and tile interchangeably.
- 17 -
Figure 3.8: The game board, with the spaces marked and the grid
labelled in pencil for the convenience of the human player.
3.4.2 Mechanical
Baxter robots, while precise by modern standards of robotics, still have difficulty picking up
objects thinner than a few centimetres with the electric parallel grippers. Each set of grippers
also has a maximum and minimum width, which means the width of the game pieces has to
be somewhere between these two values for the set of grippers that is to be used to
manipulate them. Baxter robots can also tell when the grippers sense resistance, but may not
stop immediately, or apply a certain amount of force before stopping. In order to avoid damage
to the components through play, I had to make the blocks out of rigid material. Because of the
slight potential for error in the arms placing the blocks and the fact that the grippers will have
to fit between the blocks, the grid spaces need to be made to be larger than the blocks to
accommodate this.
As a result of these considerations, I made the game pieces as square blocks of wood, roughly
63mm square and 45mm deep. The size of the square face allows a large enough facing for
the vision system to identify the block accurately, while also placing the block in the width
range for the set of grippers used for manipulating the blocks to be able to go around the
blocks and also grip them firmly. The blocks are covered in black electrical tape, with white
electrical tape used to define the paths on each block. I chose electrical tape over painting the
blocks as it gives a more even and defined coating of black and white than painting the blocks,
and covers any errant splinters which may be apparent on the blocks. To allow the grippers to
move smoothly between blocks on the grid, each of the grid squares is 80mm square.
- 18 -
3.5 Consequences of Compromise on the Game
With these compromises and the revised blocks in the game, this obviously has a significant
impact on the rules and play of the game. The depth of the blocks mean that, rather than being
picked from a stack (which would be almost a metre tall at the start of the game, far too
cumbersome for Lucas to work around), the human player (or an unbiased 3rd party, to
minimise the potential for foul play) has to deal blocks to Lucas on each of its turns.
In simplifying the game, I had to tread a fine line between making it simple enough, while at
the same time being careful not to remove the “game” from the exercise, simply placing blocks
on a grid at random is merely an activity, there is no problem to solve. To maintain the game
aspect, which is fundamental to this project, it is important that the final played game still fits
within the “human” definition of a game, as well as the computational definition provided in
2.1.1. What exactly separates a game from an activity can seem to be obscure, so I have
taken this definition to apply to the minimum standard of a game:
“A game is a problem-solving activity, approached with a playful attitude” (Schell, 2015)
While it may be hard for a robot to assume a “playful attitude”, I would assume the
responsibility here would lie with the human player, although I intend to make Lucas act a bit
more playfully than its usual solemn demeanour.
3.6 Breaking Down the Game Logic
As detailed in chapter 1.4 The Problem, the game turn has a solid structure, which both human
and robot players go through in typical play. The first stage of each turn is to identify the board
state and recognise the move your opponent has made. After that, the player draws their own
tile, which then has to be identified, before looking for the most optimal placement. Full rules
for play can be found in Appendix C.
A major feature in the game is that if the piece lands in such a way that the player’s path joins
up with a tile already on the board, the path continues along it. This also applies if the opponent
has placed their piece in the space your path advances to. As detailed in chapter 3.4, Lucas
will be using a strategy which focuses on staying in the middle of the board.
Lucas will also start the game by placing a tile on the edge of the board. The player’s path
starts from the edge of the board and follows from there.
There are several factors that have to be taken into account to understand the situation and
make an informed decision:

Current position: The board space the tile will be placed in.

Current direction: The direction from which the space is being entered
- 19 
Tile type: This, in combination with the current position and direction gives the result
of the move, or the next position and next direction, if you will.

The next position and direction: This is where you end up after taking your turn. It may
very well be off the board. If this is occupied, then this is not the final position and
direction, and this needs to be followed up to find where the player is at the end of their
turn.
The basic turn structure in terms of thought processes and actions is displayed below in Figure
3.9 below.
Figure 3.9: The structure of a turn in terms of thought processes and actions.
Each of these points in the turn process can be further expanded upon, but to avoid repeating
myself I’ll go into further detail in chapter 4 when I explain more about how the final program
handles these actions.
There are only 2 parts of this structure where any game logic actually needs to be applied,
and they are somewhat intertwined.
The decision on how to place a piece would, ultimately, depend on where the path leads to,
as the decision would be based on whether one leads you off of the board, or into the central
six spaces (according to the strategy which is used by Lucas in this project). In essence, these
are the two factors that Lucas takes into account in its strategy when playing the game. Lucas
will not take into account how its moves affect my future actions, or indeed how it will affect its
actions in future turns.
Although being taken off the board and being taken into the centre are mutually exclusive, the
option to remain on the board at all costs is still treated with the highest priority. We can then
assume that the decision trees would follow the logic depicted in Figure 3.10.
- 20 -
As per Figure 3.10, there are only two real outcomes from the turn after the piece is placed.
Either the game is lost and the path leads off of the board or the move is good and the next
space needs to be figured out.
Figure 3.10: Flowchart depicting the logic involved in deciding how to place a tile and whether
it results in game loss or finding the space for the next turn.
- 21 -
In order to ensure I had dissected the game logic thoroughly, I played around 50 games
against humans and myself in order to make sure I had a firm grasp on all of the logic involved
in decision making.7
3.7 Decisions on Development Tools
I chose Python over C++ to be the language of choice for this project due to my familiarity with
it (though I am quite experienced with C++ also), and because of the wide array of useful
libraries and resources available for Python when working with C++. I also prefer Python
personally as it is not as pernickety when it comes to variable types and passing variables.
I also had several options to choose from when deciding how to handle the computer vision
aspect of the implementation. All of the computer vision programming I had done before was
in MATLAB, though I chose OpenCV because of the Python libraries available to interface
between ROS and OpenCV (namely cv_bridge , which is invaluable in converting image
types from the raw ROS source to a format which OpenCV can recognise).
For version control, I chose to use a GitHub git repository as, when faced with the options
available, I found that GitHub had all of the features I needed, as well as being easier to set
up, commit to, and retrieve files from remotely than the alternatives. I deemed version control
(as well as a repository) to be a necessity for this project, as in order to write this report I would
need access to the code wherever I am (which GitHub allows through a browser interface for
extra convenience), and keeping track of features over various incarnations of the project is
useful in case a particular “fix” or “improvement” update turns out to be quite detrimental and
needs to be undone.
7
I have also played several dozen games of Tsuro in the past.
- 22 -
Chapter 4
Methods
In this chapter of the report, I go into detail about the design, development and operation of
the Software project.
4.1 The Vision System
There are two functions in the program which form the two distinct parts to the vision element
of this project:

IDBlock(img): This function takes an image from the left-hand camera of the space
underneath the arm at that point, and identifies which type of game piece is in that
space.

ScanBoard(img): This function takes an image from the right-hand camera of an
orthogonal view of the board, and scans each grid space for changes. Once the space
in which a tile has been placed is identified, it calls for the right arm to move to that
space and calls IDBlock(img)
Both of these functions start by using thresholding to convert each pixel to be either black or
white (a “binary image”), depending on which it is closest to, with the components all being
black or white, this reduces noise and essentially makes sure that Lucas is seeing the
environment as clearly defined pieces.
4.1.1 Block Identification
After converting the image to binary, to reduce the capacity for error the image is cropped to
the area containing the desired block that needs identification. This stage of the process is
vital as the full original image contains segments of up to 6 board spaces.
The cropped image is then searched for shapes using OpenCV’s findContours function,
which searches for continuous areas of white and stores outlines of them. In case any small
contours have been drawn around noise anomalies, only contours with an area greater than
a certain thresholding value are kept and used. Using an approach based on taking only the
essential data required to successfully and accurately identify a block, the most basic set of
assumptions we can make to differentiate between the different types of blocks, while still
having complete accuracy is as follows:

The type 1 block has one contour (the cross), and the type 2 and 3 blocks have 2
contours (the two diagonal paths).

To differentiate between type 2 and 3, the simplest way to tell which is which is to
locate the centre points of each of the diagonals, and compare them to each other.
- 23 
If one of the points has both a greater x and y value than the other, then the block is
type 2, otherwise it is type 3.
The entire process can be seen below over the course of figures 4.1.1 to 4.1.4:
Step 1: Lucas captures an image of the area the arm is currently above.
Figure 4.1.1: The image captured by Lucas when positioned over
the desired block.
Step 2: Lucas crops the image to the area containing the desired game piece and converts
the image to binary format.
Figure 4.1.2: The image has been cropped to the right
area and converted to a binary image.
- 24 Step 3: Lucas, using the findContours function, identifies all of the areas of continuous
white in the image (they have been coloured in green here).
Figure 4.1.3: The contours of the areas of white have
been located.
Step 4: Lucas filters the contours to only include those big enough to be relevant, identifies
that there are two of them and draws a line between them. The two points of the line are then
compared to see if it is a type 2 or type 3 piece.
Figure 4.1.4: The relevant contours have been found
and the central points are compared. In this case,
the function would return that this piece is a "type
2" piece.
- 25 -
Another approach that I considered and explored was to find the contour of the entire block
face. Once this was identified, I could attempt to deduce the type of block by comparing the
amount of black with the amount of white in the central third of the block. If the area was mostly
white, it was a type 1 block, as the two paths cross in the centre. If the area was mostly black,
it was a type 2 or 3 block, as both paths on those blocks avoid the centre. I would then have
been able to find the specific type of “diagonal” block by comparing the corners of this central
area to each other. I abandoned this method while designing the vision system as, due to
inconsistencies in the blocks and the difficulty involved in finding the contour of a two-colour
object, it was too unreliable and added unnecessary complications, especially when
contrasted with the final method used, detailed above.
IDBlock() also functions as the trigger to start a new turn. At the end of the previous turn,
Lucas begins checking the Pickup spot for a new piece at short intervals. When this becomes
non-zero as Lucas is dealt a new piece, the turn begins and Lucas then scans the board.
4.1.2 Scanning the Board State
The method I devised for scanning the board and identifying the current board state at the
start of each turn works as follows:

At the start of each turn, move the right arm camera to provide an orthogonal view of
the board.

Capture an image of the entire board area.

The right arm can then be tucked away so as to not obstruct the left arm.

For each space that is still marked as empty within the game logic system, scan the
corresponding area of the image.

If there is a change, move the right arm to that space and run IDBlock(img) on that
area.

As only one space per turn would change (Lucas marks the tiles it places down as they
are placed), Lucas can then move onto the next part of the turn.
This method, while slow, is accurate and very reliable. The image is captured, converted to
binary using thresholding and stored. For each empty board space remaining, a version of the
entire image of the board is cropped to fit that space. If the average pixel intensity, found using
numpy.mean on the cropped image (which takes advantage of the fact that OpenCV treats
images as arrays), in the space is greater than a certain value (Around 0.4, with 0 being
completely black and 1 being completely white), the value for the space and the space
designation are stored. The space in this list with the highest average pixel intensity is then
chosen and the left arm is sent to that space and IDBlock(img) is run to detect which kind
of block has been placed there.
- 26 -
Figure 4.2: The Robot marks the space which is marked as empty that it has perceived to
have a piece in from the pixel intensity of the space.
4.2 The Mechanical System
There are three main methods to control the arms of a Baxter robot:

World coordinates.

Joint angles.

A vision-based system.
In a world coordinates-based system, the robot navigates the environment by moving the arms
to certain positions defined as (x, y, z) coordinates in the area around it. These can be
calculated given the origin and relative positions of objects and points in the environment.
In a joint-angles based system, the robot moves an arm between positions by moving each of
the motors in the arm to a specific pre-set position. These can be gotten from Lucas by placing
the arm in each required position and requesting them through a Python shell.
In essence, these two methods do not differ very much, as they both involve moving the arm
to a predefined destination point without having a clear set path. This means that occasionally
the arm may take an unexpected path which could potentially try to lead the arm through
something solid. This was encountered in development of an early demo version of the project
- 27 -
where Lucas was attempting to take a piece from the top of a stack but the path it attempted
to take to that point in the environment was taking the arm through the stack of tiles. The rather
inelegant solution to this problem is to set more positions at small intervals between the points,
forcing the arm to travel in small, easily predictable steps, rather than sweeping gestures.
Each of the two systems detailed above has its advantages and disadvantages, and I took
these into consideration when choosing which to use in the project. In the end I decided to
use joint angles to determine the positions, as it is much easier to gather the positions from
the hardware in this way than to figure out the 45 world coordinates I needed.
Due to the difficulties mentioned in 4.1.2 with getting contours for entire blocks, I did not
attempt to investigate a vision-based system where the robot identifies objects and moves the
arm autonomously to pick it up, as it would have had considerable difficulty in identifying where
the block is exactly, as well as which block to pick up. Vision-based systems work best when
Lucas is needed to pick up distinctly coloured objects from unpredictable positions, whereas
many of the game pieces are identical and will always need to be picked up from or placed in
exact, pre-decided positions.
It is also possible to program a fully pre-set path through a toolkit provided by RethinkRobotics,
but I didn’t investigate this course of action very far as it seemed to be unintuitive and difficult
to integrate with the rest of the project.
Each movement in the game turn has a specific structure to minimise the opportunity for
disruption caused by unpredictable movement paths. There is a “Reset” point far above the
board, and each space on the board has two sets of joint angles defined in the program to
determine the path for the arm to take to the point, designated as (space)U and (space)D, e.g.
A1U and A1D. As an example, the route the arm would take to pick up a new block and place
it in space C3 would be as shown below in Fig. 4.3:
Figure 4.3:
Flowchart depicting
the orders that the
left arm goes through
in picking up a block
and taking it to a
certain board space.
- 28 -
4.3 The Game Logic System
To program the game logic into the implementation of this project, for the decision making
part, I took a simple approach of a decision tree comprised of nested IF statements describing
the decision logic depicted in the flowchart in Fig 3.8, resulting in the next space (defined by
the global variable next_space) being assigned to the space which either:

For a type 1 game piece, was the next block in that direction

For a type 2 or 3 piece, was the choice which was:

o
If possible, in the centre 6 spaces.
o
If not, then, if possible, was not off of the board.
If the space was off of the board, it was defined as space 668
The main decision tree is contained in the turn loop itself, as the positioning of it within the
loop is of absolute importance.
One important factor I will talk about in more detail in section 4.4 is that each board space is
represented in the code as an object (an instance of the BoardSpace class defined in the
code), and stored in a Python list (board_spaces[] ). The board spaces are referred to
throughout the code almost exclusively as the integer that defines them as a member of this
list, so the global and local constants that refer to board spaces are usually integers, and the
functions usually operate on these integers which are later used to retrieve more information
about the particular board space(s) (though there are exceptions to this).
There are two main functions which comprise the majority of the rest of the game logic:

nextSpace() which, given the current direction, assigns the next space as an
arithmetic operator on the current space, and checks to see if this results in the path
leading off of the board.

If the path leads off the board it re-assigns it to a value of 66

changeDirection() which takes the direction of travel at the start of the turn, and
figures out the new direction based on the piece placed in the turn.

As an example, a direction of “up” and current_piece = 2 would result in the
direction being changed to “right”
While programming the logic I had to take certain factors into account. In an earlier version of
the program, changeDirection() was integrated into nextSpace(), which, when the next
space was calculated for both options when placing a 2 or 3 type piece, calculated the change
8
I chose a value of 66 arbitrarily because it was well outside of the range of possibly assigned
values for current_space .
- 29 -
in direction twice, leading to some very interesting and very illegal game moves on the part of
Lucas.
The nextSpace() function works around the premise of the board_space[] list and the
current_space and next_space variables acting as references to its indices. The list
stores BoardSpace objects referring to each space, starting with space A1 (the top left space)
in list position 0, going across each of the rows of the board and finishing with space D5 in list
position 20.
Because of the structure of this list, moving around the board in terms of the list elements is
incredibly easy.




To move one space up on the board:
o
next_space = current_space – 5
o
e.g. from B3 to A3
To move one space down on the board:
o
next_space = current_space + 5
o
e.g. from B3 to C3
To move one square to the right:
o
next_space = current_space + 1
o
e.g. from D2 to D3
To move one space to the left:
o
next_space = current_space – 1
o
e.g. from B5 to B4
This function is also where the first check for game loss takes place, when the move is
calculated and, if it takes the path off of the board, the current_space variable is assigned to
66. There are two different methods used to check depending on the current direction.

For vertical travel, it simply checks if the current_space value leaves the list index
range.
o
E.g. moving downwards from space D4 (list index 18) would result in a
current_space value of 23, which would be reassigned to 66.

For horizontal travel, it checks the direction of travel and the board space along the
respective edges.
o
E.g. moving left from space C1 (list index 10) would result in a current_space
of 9, however since the direction was “left” and the space was “10”, the move
leads off of the board, which results in current_space being reassigned to 66.
During the early stages of this project’s conception, I considered implementing a machine
learning algorithm or some form of alpha-beta pruning (as described in Section 2.1.2).
However, as the game was analysed and the features in it were scaled back it became
- 30 -
apparent that these were unnecessary complexities due to the random dealing nature of the
game, and that the only real choice is which direction to go in when given a type 2 or 3 tile. If
time allowed, I would have implemented the option to give the players a hand of 2 pieces to
choose from, which may have warranted some advanced AI features such as these given the
increase in game complexity.9
4.4 Putting It All Together
I found the trickiest part of the implementation was what came after making all of the various
pieces of the puzzle separately, the vision systems, the mechanical system and the game
logic, and, essentially, shoving it all together and seeing if it works.
In this part of the implementation I ended up changing a lot with how the global variables are
handled by all of the different functions, as when I had originally written the functions, some of
them took BoardSpace objects as variables when they only needed the current_space
integer to perform.
The BoardSpace class is a simple object class that stores a total of 7 variables about each
board space:

The “Up” position joint angles, for moving the arm to a point above that board space.

The “Down” position joint angles, for moving the arm to the board space proper.

The “piece” integer variable defining which piece is occupying that board space.

Four integers corresponding to x and y coordinates describing the area occupied byu
this board space in the image of the orthographic view of the board used by the
scanBoard() function.
When I was testing each stage of the combined implementation, at some points I also realised
that I needed to put in more functions to manage the global variables to stop confusion (such
as with the issue with the direction variable mentioned in 4.3) as well as refine the mechanical
system (moving back from identifying a tile would take a direct path to the reset point, instead
of traveling through the interval point above the board space, which occasionally disrupted
other board pieces).
The final implementation uses a total of 14 functions, which often refer to each other and are
all utilised in the course of a game.
9
Unfortunately I did not get around to doing this, though I will discuss this further in chapter
6.2.
- 31 -
The vision-based functions are:

get_right(msg) and get_left(msg) which take the images from the
corresponding arm cameras and convert them into a format useable by OpenCV.

IDBlock(img) which uses an image to decipher what type of game piece it is
currently looking at. Explained in more detail in section 4.1.1.

scanBoard(img) which scans the board for changes before identifying any spaces
where a block has been placed and using IDBlock and moving the left arm to identify
the block in that space. Explained in more detail in section 4.1.2.
The mechanical functions are:

place(num) which places a block in the board space at index num in
board_spaces[]

move_to(bs) and move_from(bs), which move the left arm to and from a certain
board space respectively. Used in boardScan() to move to the block which needs to
be identified with IDBlock().

pickUp() which picks up the new block from the draw space and identifies it with
IDBlock().

rotate() which picks up and rotates the block on the draw space before calling
pickUp() to pick it back up and re-identify it.
And finally, the logical functions:

startTurn() which instructs Lucas to wait for a piece to be placed in the draw space
before its turn starts. This also acts as the trigger for detecting victory, if Lucas waits
more than 30 seconds for a piece, it assumes it has won and the game is over. This is
when scanBoard() is called.

nextSpace(piece) given a piece type and the current space and direction, this
function calculates the next space that this path would call it to.

changeDirection() which applies the change in direction implied by the current
piece.

turnOne() which instructs Lucas to make a particular move in the first turn. This
move is the safest move as it places a tile in the centre of one of the longer board
edges, moving straight towards the central point of the board.

boardStatePrint() is not a logical function, but prints a grid depicting the board
state Lucas currently has stored. I put this in more for testing purposes than anything
else.
There is also the final game loop, which follows directly on from a call of the turnOne()
function. This loop is a while loop which ends when the Boolean gameLoss stops being False,
and is formed of three main parts.
- 32 -
The start-of-turn board state inspection and deduction is the first of these parts. In this stage,
startTurn() is called, which waits for the new tile or declares victory, and triggers the
boardScan() function to run. After this, if there is a piece in the space which Lucas is about
to play into, it tracks this path to its end. This is done with the snippet of code shown below in
Figure 4.4:
Figure 4.4: The if and while statements used to track the current path through to
the end and trigger gameLoss if the path leads off of the board.
The main decision tree and its result from the second part of the turn. In this stage, Lucas
uses the logic described in sections 3.6 and 4.3 to decide how to place a tile and see the
resulting next_space where it will be placing a tile in its next turn. As described in section
4.3, this decision tree is a direct translation of the flowchart in figure 3.8. This includes rotating
the piece if necessary and placing it on the appropriate space.
Each turn ends with a short roundup step after the decision tree is executed. The direction is
updated with changeDirection() and boardStatePrint() is called to print the current
stored board state into the console.
If any of the triggers that cause Lucas to register that it has lost the game go off, gameLost
will return True, which will break the loop and cause the final part of the program to trigger.
When this happens, Lucas “refuses to believe” that a human could have bested it and returns
the following message:
“+++Divide
By
Cucumber
Error.
Please
Reinstall
Universe
And
Reboot+++”10
10
This is a reference to HEX, a fictional, magical computer originally powered by ants
navigating through glass tubes before evolving into something much more magnificent
(because, well, magic), from the Discworld of the late, great Sir Terry Pratchett, from
whose work I have gained an appreciation for all things exciting and out of the ordinary,
and a penchant for going off on a tangent in footnotes.
- 33 -
Chapter 5
Analysis and Evaluation of the System
In this chapter I critically analyse the reliability and performance of the system as a whole, and
address any issues which I observed. I also provide, analyse and discuss the results from
game data gathered and evaluate the decisions I made at the start, and throughout, the
project.
5.1 System Reliability
The system in its final incarnation proved to be very reliable, with only one game of the 20
having to be exited early due to a misinterpretation of a tile leading Lucas to make an illegal
move (although not from its perspective). The grippers never dropped a tile in mid-air and the
hardware performed consistently. The game components served their purpose absolutely and
without error or causing or withstanding any damage to themselves, the robot or the
environment throughout the course of the testing and development.
There was only one major bug which manifested repeatedly in testing, and this is due to the
fact that I misallocated the value of “B1U” in “coords.py” as the same value as “A5U”.11 This
leads to Lucas moving its arm (and any block it happens to be holding) rapidly across the
board at board level, scattering any blocks in its path. An attentive human player can catch
this as it happens and minimise the damage caused by replacing the scattered blocks (if any).
It is interesting to note, however, that this has absolutely no effect on Lucas’ play, as it has
already allocated values for those board spaces of the correct tile, and will not scan them in
the scanBoard() function, and will make decisions based on the tiles that were placed there.
Replacing the tiles instead of removing them from the board then is only in the interest of the
human player and the program is quite indifferent, because, as far as it is concerned, those
board spaces have game pieces in them.
One other minor unintended negative effect is that, if Lucas is attempting to identify a block in
a space with other blocks placed horizontally adjacent, quite often the path of the grippers will
be on path to collide with one or both of the blocks placed either side of the piece intended to
be identified. Due to the fact that two solid objects cannot occupy the same point in space at
the same time, the blocks either side must be moved slightly to allow the arm to descend into
the required position for the IDBlock() function to be run properly. If the blocks are moved after
the grippers have made contact, the joints in Lucas’ arms will apply excess pressure, and
when the blocks are removed, will move too far down and may collide with the table, so
11
I have since fixed this by assigning a correct set of joint angles to coords.B1U.
- 34 -
vigilance on this matter is important. This could only be resolved by making a board with larger
spaces, at which point the arm may not be able to reach all of the spaces on the board, or by
playing with smaller pieces and installing smaller grippers on the arm, which would increase
the precision required in the vision and mechanical systems.
5.2 Effectiveness
The system proved relatively effective, with a win rate of 35%, a draw rate of 5% and an error
prompted exit rate of 5%.
The strategy used by Lucas was simple, but effective. The major downfall in its strategy is that
it does not take into account its opponents future moves, or the moves it will have to make
after the next round of play. I, as the human player, took full advantage of further forward
planning and knowing how my moves would affect Lucas’ turn. This worked against the overall
win rate of the system and gave me a clear advantage. However, the aim of this project was
not to produce a system that wins every time (especially difficult if not impossible due to the
randomised nature of the game), but to produce a system that interacts with a human through
the medium of a game and plays sensible moves that stick to a specific strategy.
Using this definition of effectiveness for the system, valuation is subjective along with the term
“sensible”, however I would say that the robot clearly and consistently executed sensible game
moves, never ejecting itself from the board if it had a choice, and consistently staying with the
strategy detailed in sections 3.2.2 and 3.6 as much as the tiles dealt allowed it to, although
several times Lucas boxed itself into a corner where the alternative choice would have been
better (as shown in Figure 5.1 below), but this was not accounted for in the logic programmed
in the decision tree.
Figure 5.110: An example of an objectively illogical move
which would be made by Lucas. By going to the right,
Lucas only has one space free and will lose if dealt a
type 1 tile in the next round. By going left instead,
Lucas would have gained much more space and a
better chance at survival
- 35 -
There is one exception to this, due to the lack of foresight in the space calculations in the
decision tree. In hindsight it’s something which was easy to overlook, and relatively simple to
implement, but the fact that Lucas placed a tile which joined both of our paths together, leading
to a draw in test game 13 instead of the alternative option which would have kept its path on
the board while sending mine off. The decision loop only takes into account the next space. It
doesn’t investigate as to whether that space has a tile in it or where that path leads. That is
not calculated in the game loop until the start of the next turn, by which point the human player
has conceded defeat (as their path leads off of the board) and Lucas therefore assumes
victory.
Each game took on average just over a minute per game round, with an average game lasting
between 6 and 10 minutes. I am quite pleased with the (relatively) swift execution of Lucas’
turns, considering how much arm movement is involved in each game turn, as well as the
number of individual image processing functions called over the course of a game.
5.2.1 Effectiveness of the Vision Systems
In the 20 games from which data was gathered, the vision systems were very effective, with
only 8 misidentification errors over a total of 254 IDBlock() function calls and 1 error over
127 scanBoard() function calls. This gives the combined vision systems an overall error rate
of only 2.36%.
Misidentification errors were detected by comparing the final board state printed by Lucas with
the actual configuration of tiles on the board at the end of the game.
5.2.2 Effectiveness of the Mechanical Systems
With the exception of the aforementioned “B1U” bug (Section 5.1), which has since been fixed
and had no real impact on gameplay from Lucas’ perspective, the mechanical systems
performed as predicted.
I am especially pleased with the effectiveness of the rotate() function, as some of the
blocks are not perfectly square, but Lucas nevertheless picked them up neatly once rotated.
Over the course of the test games, Lucas never dropped a game piece mid-air or performed
any drastic moves which caused the board to shift on the table, both of which had occasionally
been an issue during development.
5.3 Limitations Imposed By the System
There are several limitations that this implementation imposes, namely the fact that a lot of
the time the game ends due to the fact that the player had no choice about how to proceed,
for example, getting dealt several type 1 pieces in a row, leading straight off the edge of the
board. This is an issue more with the game than the software implementation though.
- 36 -
Lucas will only start in one of two spaces, and will only choose one if the other has already
been taken up by the human player. In addition, Lucas will always start with a type 1 tile. This
makes its opening moves fairly predictable.
In order to ensure complete fairness as well (although I dealt tiles to both players randomly in
testing), a third party is required to deal tiles to both players. Failing this, tiles need to be
shuffled and placed in separate, randomly ordered stacks of 10, one stack for each player.
Due to the fixed nature of the joint angles in terms of manoeuvring around the environment,
the board must be placed in exactly the same place every time when playing with Lucas. In
addition to this, due to the materials used for the components and the way the vision system
operates, the vision systems only operate reliably in a dimly lit environment, with no direct
lighting on the game board.
5.4 Gameplay Data
All of the data gathered from the 20 games is shown in table 5-1, with a key to the Issues/Bugs
column of data provided in table 5-1a.
Table 5-1a: Key to Issues/Bugs reported in table 5-2
Ob
Obstruction. Piece had to be moved to
allow grippers to pass so that Lucas could
ID a block placed on the board
MI(x)
MisIdentified. Lucas misidentified a block it
picked up or looked at x times in the course
of the game. The game was continued so
long as this had no impact on Lucas’ play.
DNR
Did Not Recognise. Lucas did not recognise
where a block had been placed and ran
IDBlock on an empty space/no space at all
IM
Illegal Move. Other factors led to Lucas
playing an illegal move.
- 37 -
Table 5-1: Results of the 20 games played against Lucas on the final version of the project
Game Number
Number of Lucas’
Result for Lucas
Issues/Bugs
Turns
1
5
Loss
2
7
Win
3
6
Loss
4
6
N/A
5
6
Loss
6
6
Win
7
8
Loss
8
6
Loss
9
5
Win
10
7
Loss
11
6
Win
12
7
Loss
13
7
Draw
14
6
Win
MI(1)
15
5
Loss
MI(2)
16
8
Win
Ob
17
6
Loss
18
7
Loss
Ob
19
6
Win
Ob
20
7
Loss
Ob
Ob, MI(1)
Ob
Ob, MI(2), DNR -> IM
Ob
Ob, MI(1)
Ob
MI(1)
Ob
- 38 -
5.5 Discussion of Results
5.5.1 Data Gathering Methods
Over the course of the testing stage, before any data was gathered, I decided that the only
results that were of import to the overall conclusion were the number of turns taken by Lucas,
the game result for Lucas and any issues or bugs that arose during the game.
For the gathering of results, I was the human player and dealt tiles to both of us from a
randomised face-down piles. In an ideal situation the human player would be someone who
did not know the exact logic behind Lucas’ actions, and there would have been an impartial
third party dealing tiles to each player. I attempted to make play decisions as if I were facing
a human player, and blindly selected tiles at random from the stack.
During the data gathering stage, the first turn was taken alternately by Lucas and the human
player.
5.5.2 Data Analysis
A first-glance analysis of the data gathered returns this information:

The mean number of turns taken by Lucas in a game was 6.35.

The modal number of turns taken by Lucas in a game was 6.

Lucas was slightly more likely to win when going first, with 4 of its 7 victories occurring
on even numbered games (when Lucas took the first turn).

The mean number of turns in games where Lucas won was 6.29.

Lucas only won 2 of the 8 games which lasted 7 turns or more.

The most common issue that arose was obstruction, where one or more blocks had to
be moved to allow the grippers to pass.

The majority of the time, misidentification of a tile placed by the human player did not
have any effect on Lucas’ play.
5.6 Evaluation of the Project
5.6.1 Project Schedule
After the first few weeks of work on the project it became evident that the original schedule
was not adhered to and it was disregarded from then on. At this point, time allocation became
much more flexible, and I invested a lot of time into meticulous research on the programming
methods and hardware I was going to be using in the project, as well as drawing up full
pseudo-code prototypes for the implementations, none of which exactly resemble the final
product. Once development actually began in the latter half of the project timeframe, the Gantt
chart became a useful resource for tracking milestones and acting as a flowchart for devising
which parts of the project should be implemented before others.
- 39 -
5.6.2 Choice of Development Tools
I feel that the choices I made in terms of development tools were optimal on every front.
The lenience in passing and handling variables in Python was invaluable in a system where
the main object class was comprised of two arrays where each of the entries was comprised
of both a string and a float, and four integer values, and the main form of accessing and
manipulating these objects was through a list. Attempting this in C++ would have been much
harder and probably would have required a different and more convoluted approach.
OpenCV, once I had some experience with the library and the unusual parameters involved
in its function calls, was perfectly suited to the tasks I required it to do with each task I needed
it for being solved in a simple function call or object reassignment.
GitHub was the perfect platform for version control, allowing me to edit the code on the
repository from home, through my web browser, whenever I thought of a small fix or feature
to apply (or comment in in the case of larger edits which would have required testing) without
having to install or download anything to my home computer.
5.6.3 Decisions in Game and Component Design
The game design is solid, easy to understand and one of the things I am most proud of in this
project. It is not too complex for this project nor is it too simple to meet the requirements. All
of the pieces required for the game can be represented as chunky blocks that the robot can
pick up and the designs on them are large enough to be easily recognised by the system.
However, in hindsight, electrical tape was not the best choice for the blocks. This is due to the
reflectivity of the black tape which, while not as reflective as other tapes or certain types of
paint, will reflect what appears to a computer vision system as white light if under a direct light
source. This restricts the game to being played under a suitable lighting condition. In
retrospect, a printed or cut-out matte paper covering would have been the optimal material
choice to represent the patterns on the game pieces.
5.6.4 Decisions in Software Design
I am confident in the methods I used for the software design. They all follow the approach I
set out with when starting this implementation which is to reduce each factor to its minimum
parts to reduce the margin for error (for example, instead of using template matching to identify
blocks, simply counting the number of white areas on the block). This has resulted in a set of
complimentary vision systems with minimal error rates and a solid, logical decision tree. Each
of the functions in the finished project serves a purpose, and no computation time is wasted
on irrelevant or superfluous tasks.
- 40 -
Chapter 6
Conclusions
In which I provide a closing statement on the final state of the project, conclude whether or not
I consider it to be a success, and suggest ways in which it could be expanded given the time
and resources.
6.1 Conclusions
In the conclusion of this report, there are several questions which I have to ask myself about
the project, in order to deem whether or not it was a success.
“Was the project successful?” is the most obvious question, however there are several factors
that have to be addressed and analysed before success of the project can be established.
In order to deduce success, we must look at the aims and objectives that I set for this project
to achieve. Firstly, the minimum requirements for the project were as follows:

I am to design or modify a game so that it is suitable for play with a Baxter robot.

I am to implement a system allowing the robot to play the game by itself.
Both of these have been comfortably met. The data shows that the game is suitable to the
task and, in addition to this, the game meets both definitions of what makes a game presented
in this project. Lucas can play the game by itself if it is continually dealt tiles. It will simply
attempt to stay on the board as long as possible with the tiles given.
Next we need to look at the main objectives:

I am to design or modify a game so that it is suitable for play with a Baxter robot.

I am to implement a computer vision system to allow the Baxter robot to play with no
prior knowledge of the game state or human input.

I am to implement an artificial intelligence (AI) system to allow the robot to play
competently.

I am to implement it in such a way as to allow full game play with a human player.
The game is fit for play with a Baxter robot, as is made evident by the fact that 20 full games
were played. The game is a modified version of Tsuro by my own design, simplifying the game
pieces and modifying the game rules to suit the new pieces. The final design was partially
influenced by suggestions from my supervisors and from observations made during
development.
- 41 -
The computer vision systems allow Lucas to play completely autonomously without any
human input, with only a 2.36% rate of error over 381 separate vision function calls, which
only once over the course of 20 games resulted in the game having to be abandoned.
The AI system results in sensible play, although this is a subjective assessment. While playing
it felt to me very much like a child playing chess with their parent. Lucas was definitely playing
the game independently and making decisions based on the current board state but without
considering the future implications of its play or how its play may affect the other player. Even
with the only decision in the game being where to start and how to place tiles which are not of
type 1, there are a few factors that are not considered in the AI. All things considered I think
that the AI for play is competent, but has a lot of room for improvement.
The final objective is that the full implementation allows play of a full game with a human
player. Having played 20 games with Lucas in full with the test version of the program running,
I can wholeheartedly conclude that play of a full game to completion is possible, and that this
objective has been thoroughly met.
In conclusion then: Was this project a success?
I believe that, overall, taking all of the results and components into account, this project was a
success, (especially since the bugs present during testing and in the demonstration video12
although there is a lot of room for expansion, especially given the subjective nature of a lot of
the evaluative criteria (notably being “competent” at a game, as well as making “sensible”
moves). The implementation is still far from perfect, some of the issues detailed in Chapter 5
still occur, especially the obstruction of the arm movement by other blocks on the board
blocking the grippers, which is a fairly regular occurrence. The test version of the project
derailed one of the 20 games wildly by making an illegal play after having misidentified a block.
I theorise that most of the block misidentifications were due to the way the block was placed
by the human player, as the image is cropped to a narrow window with little margin for error
in IDBlock() and the functionality of IDBlock() can be compromised if the block is placed half
an inch out from the ideal position. The system is inflexible and too precise for its own good,
but despite its faults, the vast majority of the time works perfectly as intended.
The final program is still dependent on the human player placing a block for the robot to draw
each turn, and does not communicate with the player directly, however I think that the solutions
to these lie outside of the basic scope of the project, although they definitely fit the theme of
the project and would be worthwhile expansions.
The secondary aim of this project was for me to gain a deeper understanding of the field of
robotics and extend my knowledge of practical artificial intelligence, more specifically the
12
A link to which can be found in Appendix E
- 42 -
application of game logic. I can attest that throughout the course of this project, from my initial
research into applied game logic systems through to the final stages of development and
testing, I was constantly learning new applications and methods of implementation for the
fundamental logic algorithms in game AI. I have also learnt a significant amount about practical
robotics – a subject which I knew very little about at the start of this project – and applied it to
achieve all of the goals and objectives set for this project.
6.2 Future Work
There are a multitude of ways in which this project could be expanded, given more time and/or
resources.
Expansions that could be implemented without modifying the game include:

A more advanced game AI system, allowing Lucas to predict the possible implications
of its moves on the future moves of both it and its opponent.
o
A minimax algorithm could be implemented here to also encourage a more
competitive style of play.
o
A machine learning algorithm could also be implemented, to allow Lucas to
form its own strategy based on how placing tiles in various ways in certain
spaces leads to a better or worse result.

Implementing a separate array and a second global direction variable to keep track of
its opponents moves and be able to detect when it has won through inspecting the
board state, rather than the current method of assuming game loss when its piece is
not renewed.

Instead of detecting the start of the turn by using IDBlock() and waiting for a new
tile to be dealt, detecting a change in board state (which occurs when the other player
places a tile) by using boardScan() at short intervals to trigger the start of the turn.

Lucas could be made to draw a new tile itself when it receives a start of turn trigger,
from a stack of randomised game pieces, negating the need for an impartial dealer.

Using an expanded vision system to detect where the board grid is, and moving to the
various points defined according to this vision-based perspective rather than the
- 43 -
mechanical world perspective currently used. This also negates the need for the board
to be placed in the exact same position for every game.

Allowing play with multiple human players, and scanning each change in board state
between turns as opposed to only one space. Game AI could also be expanded for
this, taking more variables into consideration with the increased number of players.

Using a randomised starting space and tile. At the moment, in the first turn, Lucas will
only place in one of two spaces, and always uses a type 1 piece in its first turn.
With improvements to the vision system (and a stable lighting environment), it would be
possible to implement the full game of Tsuro, with 4 colour-coordinated paths to each tile.
Each of the paths would be represented with a pair of matching coloured points (as seen in
Fig 7.1). This would also add another layer of complexity in the choice, as each side of each
tile has two paths linked, meaning there is a possible 7 exit paths. Game complexity could be
further increased by also giving Lucas a choice of 3 tiles, bringing it up to a full implementation
of the original Tsuro.
With the game being made so much more complex, it would require more complex AI as well,
rather than just a which would make use of complex decision trees and alpha-beta pruning.
This could also be expanded upon with the application of a machine learning algorithm,
gathering data based on which tiles leads to a greater chance of success, or which tile
placements lead to the opponent careening off the board.
Of course, it would also be possible to play different games using a modified version of this
system. With a sufficiently developed vision system and a much more complicated artificial
intelligence system, I believe it would even be possible to implement it so that Lucas could
play chess against a human player using similar principles to form the base of the vision and
mechanical systems – and assuming the pieces are blocks of the right size.
To develop more on the human interaction side of this project, I would look at implementing a
speech input system and text-to-speech system to allow certain phrases to be “spoken”
between the player and the robot, mostly phrases such as “let’s play”, “your turn” or “good
game, well played” which can obviously be interpreted as triggers for events in the course of
the game, such as a turn ending or victory being declared by a player.
- 44 -
List of References
Arkin, R. C., 1998. Behavior-based Robotics. Cambridge (US-MA): MIT Press.
Craig, J. J., 2005. Introduction to Robotics - Mechanics and Control. 3rd ed. Upper Saddle
River, NJ.: Pearson Education Inc..
Hornberg, A., 2007. Handbook of Machine Vision. 1st ed. Weinheim: John Wiley & Sons.
Nilsson, N. J., 1984. Shakey The Robot, Technical Note 323. Menlo Park: SRI International.
Russel, S. &. N. P., 2010. Artificial Intelligence: A Modern Approach. 3rd ed. Upper Saddle
River: Pearson Education Inc..
Schell, J., 2015. The Art of Game Design. 2nd ed. Boca Raton, FL: CRC Press.
- 45 -
Appendix A
Personal Reflection
When It came to choosing a project, this one immediately stuck out to me. Board games and
tabletop gaming is a big passion of mine, and when I think of the word “Tabletop”, games are
the first thing that come to my mind. My ambition in my career once I graduate is to become a
games developer, and while there were no projects proposed which directly translated to
games development programming, I saw an opportunity here to add an element of “games” to
a project which already had an exciting base in that it involved robotics and, especially, a
Baxter robot. After all, robots are cool, exciting and wonderful. Even now after dozens of hours
working on Lucas, the sci-fi geek in me still revels in the fact that I have made a real life robot
play a board game with a human13.
Such was my excitement that I went a little overboard with my original concept, and my
ambition was rightfully tempered by my supervisors early on so as to save me from setting
goals which would have been impossible to achieve in the time given. Upon entering this
project, I was venturing into a completely unknown realm of computer science: Robotics. This
was the first time I had actually worked on a robot since I had access to a LEGO Mindstorms
kit when I went to LEGOLand as a child. Similarly I had no previous experience of actually
programming an AI algorithm, the second year Artificial Intelligence module I took
unfortunately suffered from some complications, and there was only a small practical
coursework in the end. My experience programming computer vision was also extremely
limited, as we had had only one specific coursework assigned to us during the second year
Computer Vision module.
When I started working on this project, I perceived it to be immensely daunting, a task of
almost Herculean scale, comparable to slaying a hydra. I have an unfortunate habit of shying
away from work that seems impossible, fearing that I would be defeated, proved right and
driven further away from the task at hand. As such, I spent a lot of time designing and planning
solutions for each aspect of the project without actually working on the implementation. Now,
having successfully gone through and implemented the project, having actually made a
program that makes Lucas play a game with me, I am ecstatic that I managed to achieve the
final program that I did, relieved that I managed to do it within the timeframe, but mostly, I am
frustrated at myself for putting myself off of it for so long. I found that once I started working
13
Yes, even when I was encountering those frustrating bugs that make no sense and take
hours to figure out, before turning out to be a stupid mistake such as a misplaced set of
parentheses.
- 46 with the robot itself, it was much clearer than I had anticipated. I feel that if I hadn’t scared
myself off of working on Lucas and exploring the capacity of what it can do, I could have
delivered a much more refined and advanced system incorporating some of the features I
have detailed in the future work section of this report.
The vision systems aren’t the 400-line goliaths I had envisioned. The mechanics are precise
and were the easiest part to implement. The whole program fits into about 500 lines of code if
you remove the variable definitions. Lucas performs its actions smoothly and efficiently.
RethinkRobotics have made a robot that is an absolute pleasure to work with. In positioning
the arms to gather joint angle values I could feel when the arm was reaching the limit on one
of the 7 motors in the arm and adjust accordingly, the arm moving smoothly and in accord with
whatever way I pushed it. At times, Lucas can feel strangely “alive” such as when the arms
are in danger of colliding, it will reflexively stop and move one out of the way.
Despite my slow start, I am pleased with the outcome achieved, although I am annoyed at the
fact that there were a few bugs in the version of the code which I used to gather data and
record the demonstration video. Hearing the subject matter of the final projects of some of my
course-mates, I am very glad to have chosen such a unique and exciting project. I am still
extremely surprised that myself and Ben were the only two students to apply to work with a
Baxter robot for a few months.
Due to the nature of this project, all of the practical development of the implementation had to
be done in the labs, as I couldn’t bring Lucas home to run tests on. I feel this helped me to
focus on my work once I started working as, when I was in the lab, I was not exposed to the
distractions that are legion at my desk at home.
One thing I very much appreciated about working on this project was the environment in the
robotics lab, as well as the resources provided to me so that I could always have access to
what I needed for this project. I had access to the lab 24/7, and even had the cost of the
materials for the game pieces reimbursed. At first I was concerned about working in the lab
with a robot that is a shared resource, but after a while I became comfortable in the lab and it
was made clear that the other people working there were more than willing to give up time
working with Lucas whenever I needed it, and were also willing to offer advice when I was
stuck with a bug or needed to get Lucas to do something specific through the Baxter API. It
reminded me of the internship I had last summer, where I was programming factory operator
interfaces, and also had zero experience in the subject matter prior to turning up and getting
stuck in. At first it seems almost overwhelming, working in the same office as people who have
been doing this for years and must be sick of untrained newcomers to the field asking for
advice constantly, but after a while the realisation dawns that the people working around you
are human too, and were once in the same position.
- 47 -
Overall, I am glad that I chose this project. It was exciting, and appealed greatly to the side of
me that loves the fantastical and futuristic, it pushed me out of my comfort zone and into an
area of computing I had no opportunity to explore before in an academic context, one which
has been romanticised, explored and parodied in popular science-fiction and forms one of the
core aspects of some of my favourite science fiction franchises14. The result of achieving the
final project implementation is extremely satisfying in comparison to a lot of computer science
projects as it has a real, visible, tangible effect on the physical world as Lucas moves blocks
around a board in front of you.
I have learnt a lot of useful programming skills in areas of computer science I hold a great
interest in, I have had the opportunity to work with a fantastic, highly sophisticated modern
robot and I am satisfied with the final result of my project. I do not think I could ask for a better
end to my 5-year journey to achieving a BSc15.
14 One example of
a character who programs a robot that can play board games? Darth Vader.
Anakin Skywalker created C3-P0, who plays Holochess against Chewbacca in Star Wars
Episode IV: A New Hope, where the famous line “Let the Wookiee win” was uttered.
15I
spent 2 years in York studying Astrophysics, which, as it turns out, is quite difficult. During
this time, I made some excellent friends and met my partner, while living in one of the most
wonderful cities in Britain. As such I do not regret one moment of it.
- 48 -
Appendix B
Ethical Issues Addressed
B. i: Ethical issues
I do not think there are any ethical issues to be raised for this project as, although it pits a
robot in direct opposition to a human, it is not with intent to cause harm or allow a human to
come to harm.
- 49 -
Appendix C
Record of External Materials Used
Few external materials were used in this project. The only source from which code was taken
directly, in order to figure out how OpenCV gets and draws contours (although the code which
was taken has since been heavily modified and dissected in this project) was:
http://opencvpython.blogspot.co.uk/2012/06/hi-this-article-is-tutorial-which-try.html
This source is a tutorial on finding contours in images using OpenCV and was accessed in
late April 2015.
- 50 -
Appendix D
Full Rules of the Game
D. i: Components

One board, plywood, marked in pencil.

20 game pieces, wood, marked out with tape.
D. ii: The Tiles
Each game piece or tile shows two possible paths across a game board space, depicted in
white on a black background. These two paths connect the centre of each side to the centre
of another side of the tile face. Paths on a tile may cross each other, but the path continues
along the single most straightforward route across the tile, uninterrupted and independently of
the others.
D. iii: Setup

Place the board on a table. If playing with a Baxter robot, place the board in the exact
configuration required relative to the Baxter robot. This can be determined by moving
the left hand to the PickUpD joint angles found in coords.py and aligning the pickup
space of the board underneath it.

Place the 20 game tiles in a face-down pile on the table.
D. iv: Playing the game
At the start of each turn, the current player draws a tile at random.
Players take turns placing tiles on the board. Each player must start along a board edge. Once
you have placed your first tile, your path starts at the board edge and follows the path on the
tile.
A player may not willingly direct their path off of the game board unless they have no other
option.
A player may place their tile in any orientation so long as it lines up with the grid spaces on
the board.
A player is eliminated from the game if the open end of their path connects to an edge of the
game board.
- 51 -
If their path collides with another tile, the path follows the path on that tile and any subsequent
tiles until the path reaches an empty space or a board edge.
D. v: Winning the Game
A player becomes the winner when they are the only player remaining who has not been
eliminated.
D. vi: FAQ
Q: What happens if the two paths meet?
A: The game ends in a draw, as both paths now lead off the edge. Congratulations,
you have both lost.
Q: How do I keep track of which path belongs to which player?
A: Use your mind. Alternatively, use a small white token to mark the progress of your
path.
Q: On a type 1 tile (straight across), can I use any of the paths off of the tile as they all
connect in the middle?
A: No, on the type 1 tile you can only use the path leading across the tile.
- 52 -
Appendix E
Project Code and Other Materials of Interest
E. i: coords.py
coords.py has been excluded from this Appendix on the grounds that it is roughly 12 A4 pages
of joint angle values for Lucas. If you would like to inspect coords.py, feel free to access it at
https://github.com/sc12pas/TTR/blob/master/src/coords.py
E. ii: vision.py
#vision.py
#Full code for the Final Year Project:
#"Tabletop Robotics: Board Games With a Baxter Robot"
# by Pascal Alexander Siddons, [email protected]
#----------------Importing required libraries-------------------------------#
import numpy as np
import rospy
import cv2
import cv_bridge
import time
import sys
import baxter_interface as bax
from select import select
import coords #The file containing all of the necessary joint angles
from sensor_msgs.msg import Image
rospy.init_node('Game_run')
#-----------------Defining object classes-----------------------------------#
class Point:
"""A class to describe objects representing points in 2D coordinate systems"""
def __init__(self):
self.x = 0
self.y = 0
class BoardSpace:
"""A class to describe objects representing the 20 spaces on the game board"""
def __init__(self,posU,posD,piece,x1,y1,x2,y2):
self.posU = posU
- 53 self.posD = posD
self.piece = piece
self.x1 = x1
self.y1 = y1
self.x2 = x2
self.y2 = y2
#---------------Defining global variables and the parameters for the robots arms,-#
#------------------------Gripper and cameras.-------------------------------------#
limbR = bax.Limb('right')
limbL = bax.Limb('left')
gripper = bax.Gripper('left')
left_camera = bax.CameraController('left_hand_camera')
left_camera.resolution = (640,400)
left_camera.open()
left_image = None
right_camera = bax.CameraController('right_hand_camera')
right_camera.resolution = (640,400)
right_camera.open()
right_image = None
board_spaces = []
current_space = 0
next_space = 0
current_piece = 0
direction = ""
gameLoss = False
#------------------------Defining all of the board spaces-------------------#
h1=195
h2=260
h3=335
h4=400
h5=470
h6=535
v1=44
v2=113
v3=174
v4=250
v5=315
#The first two parameters correspond to the joint angles required to
#navigate to that particular space
A1 = BoardSpace(coords.A1U, coords.A1D, 0, h1,v1,h2,v2)
board_spaces.append(A1)
#The third is the piece currently occupying that space
#While the last four describe the area this space occupies in the orthogonal view
A2 = BoardSpace(coords.A2U, coords.A2D, 0, h2,v1,h3,v2)
board_spaces.append(A2)
A3 = BoardSpace(coords.A3U, coords.A3D, 0, h3,v1,h4,v2)
- 54 board_spaces.append(A3)
A4 = BoardSpace(coords.A4U, coords.A4D, 0, h4,v1,h5,v2)
board_spaces.append(A4)
A5 = BoardSpace(coords.A5U, coords.A5D, 0, h5,v1,h6,v2)
board_spaces.append(A5)
B1 = BoardSpace(coords.B1U, coords.B1D, 0, h1,v2,h2,v3)
board_spaces.append(B1)
B2 = BoardSpace(coords.B2U, coords.B2D, 0, h2,v2,h3,v3)
board_spaces.append(B2)
B3 = BoardSpace(coords.B3U, coords.B3D, 0, h3,v2,h4,v3)
board_spaces.append(B3)
B4 = BoardSpace(coords.B4U, coords.B4D, 0, h4,v2,h5,v3)
board_spaces.append(B4)
B5 = BoardSpace(coords.B5U, coords.B5D, 0, h5,v2,h6,v3)
board_spaces.append(B5)
C1 = BoardSpace(coords.C1U, coords.C1D, 0, h1,v3,h2,v4)
board_spaces.append(C1)
C2 = BoardSpace(coords.C2U, coords.C2D, 0, h2,v3,h3,v4)
board_spaces.append(C2)
C3 = BoardSpace(coords.C3U, coords.C3D, 0, h3,v3,h4,v4)
board_spaces.append(C3)
C4 = BoardSpace(coords.C4U, coords.C4D, 0, h4,v3,h5,v4)
board_spaces.append(C4)
C5 = BoardSpace(coords.C5U, coords.C5D, 0, h5,v3,h6,v4)
board_spaces.append(C5)
D1 = BoardSpace(coords.D1U, coords.D1D, 0, h1,v4,h2,v5)
board_spaces.append(D1)
D2 = BoardSpace(coords.D2U, coords.D2D, 0, h2,v4,h3,v5)
board_spaces.append(D2)
D3 = BoardSpace(coords.D3U, coords.D3D, 0, h3,v4,h4,v5)
board_spaces.append(D3)
D4 = BoardSpace(coords.D4U, coords.D4D, 0, h4,v4,h5,v5)
board_spaces.append(D4)
D5 = BoardSpace(coords.D5U, coords.D5D, 0, h5,v4,h6,v5)
board_spaces.append(D5)
#---------------------------Code to retrieve images from Lucas------------------#
- 55 -
def get_right(msg):
global right_image
right_image
desired_encoding='rgb8')
=
cv_bridge.CvBridge().imgmsg_to_cv2(msg,
def get_left(msg):
global left_image
left_image = cv_bridge.CvBridge().imgmsg_to_cv2(msg, desired_encoding='rgb8')
left_sub = rospy.Subscriber( 'cameras/left_hand_camera/image', Image, get_left )
right_sub = rospy.Subscriber( 'cameras/right_hand_camera/image', Image, get_right )
#--------------------------Functions for everything needed to play the game-----#
def place(num):
"""A function to instruct Lucas to place a block in a particular space"""
global current_piece
global current_space
global board_spaces
space = board_spaces[num]
limbL.move_to_joint_positions(coords.ResetPos)
limbL.move_to_joint_positions(space.posU)
limbL.move_to_joint_positions(space.posD)
gripper.open()
space.piece = current_piece
current_space = num
limbL.move_to_joint_positions(space.posU)
limbL.move_to_joint_positions(coords.ResetPos)
def startTurn():
"""A function instructing Lucas to wait for the start of its turn before
scanning the board to see where the Human has placed their tile"""
global current_piece
current_piece = 0
limbL.move_to_joint_positions(coords.ResetPos)
limbL.move_to_joint_positions(coords.PickU)
limbL.move_to_joint_positions(coords.PickD)
i = 0
while current_piece == 0:
current_piece = IDBlock(left_image)
rospy.sleep(5)
i+=1
if i >= 6:#After 30 seconds of inaction by the human player, Lucas
assumes victory
print "Huzzah! Another glorious victory for your robotic
overlord!"
sys.exit()
limbL.move_to_joint_positions(coords.PickU)
limbL.move_to_joint_positions(coords.ResetPos)
scanBoard()
limbL.move_to_joint_positions(coords.ResetPos)
- 56 def move_to(bs): #A function instructing Lucas to move the left arm to a particular
space
limbL.move_to_joint_positions(coords.ResetPos)
limbL.move_to_joint_positions(bs.posU)
limbL.move_to_joint_positions(bs.posD)
def move_from(bs): #A function instructing Lucas to move back from a particular space
limbL.move_to_joint_positions(bs.posU)
limbL.move_to_joint_positions(coords.ResetPos)
def pickUp(): #A Function instructing Lucas to pick up its tile for the turn
limbL.move_to_joint_positions(coords.PickU)
limbL.move_to_joint_positions(coords.PickD)
current_piece = IDBlock(left_image)
gripper.close()
rospy.sleep(2)
limbL.move_to_joint_positions(coords.PickU)
limbL.move_to_joint_positions(coords.ResetPos)
def rotate(): # A function instructing Lucas to rotate its piece for the turn by 90
degrees
limbL.move_to_joint_positions(coords.PickU)
limbL.move_to_joint_positions(coords.rotateU)
limbL.move_to_joint_positions(coords.rotateD)
gripper.open()
rospy.sleep(2)
limbL.move_to_joint_positions(coords.rotateU)
pickUp()
def nextSpace(piece):
"""A function calculating, given the current piece and with knowledge of the
current space
and direction, what the space Lucas will be placing a piece in in its next
turn will be"""
global current_space
global current_piece
global next_space
global direction
if direction == "up":
print "piece = ", piece
if piece == 1:
next_space = current_space - 5
if next_space <= -1:
next_space = 66
elif piece == 2:
if current_space == (4 or 9 or 14):
next_space = 66
else:
next_space = current_space + 1
elif piece == 3:
if current_space == (0 or 5 or 10):
next_space = 66
- 57 else:
next_space = current_space - 1
elif direction == "down":
if piece == 1:
next_space = current_space + 5
if next_space >= 20:
next_space = 66
elif piece == 2:
if current_space == (5 or 10 or 15):
next_space = 66
else:
next_space = current_space - 1
elif piece == 3:
if current_space == (9 or 14 or 19):
next_space = 66
else:
next_space = current_space + 1
elif direction == "right":
if piece == 1:
if current_space == 4 or current_space ==9 or current_space ==14
or current_space ==19:
next_space = 66
else:
next_space = current_space + 1
elif piece == 2:
if 1 <= current_space <= 4:
next_space = 66
else:
next_space = current_space - 5
elif piece == 3:
if 16 <= current_space <= 19:
next_space = 66
else:
next_space = current_space + 5
elif direction == "left":
if piece == 1:
if current_space == (0) or current_space == 5 or current_space
== 10 or current_space == (15):
next_space = 66
else:
next_space = current_space - 1
elif piece == 2:
next_space = current_space + 5
if next_space >= 20:
next_space = 66
elif piece == 3:
next_space = current_space - 5
if next_space <= -1:
next_space = 66
else:
print "+++?????+++ Out of Cheese Error. Redo From Start"
sys.exit()
- 58 if next_space <= -1:
next_space = 66
elif next_space >= 20:
next_space = 66
#Defining next_space as 66 tells Lucas the next space is off of the board and
that it has lost
return next_space
def changeDirection():
"""A function instructing Lucas to update the direction it is moving around
the board in
given the type of piece it has just placed"""
global current_space
global current_piece
global next_space
global direction
if direction == "up":
if current_piece == 3:
direction = "left"
elif current_piece == 2:
direction = "right"
elif direction == "down":
if current_piece == 3:
direction = "right"
elif current_piece == 2:
direction = "left"
elif direction == "right":
if current_piece == 3:
direction = "down"
elif current_piece == 2:
direction = "up"
elif direction == "left":
if current_piece == 3:
direction = "up"
elif current_piece == 2:
direction = "down"
def scanBoard():
"""A function which scans all of the empty spaces on the board to
see if any have changed. The one which is most likely to have changed is then
scanned by Lucas"""
global board_spaces
limbL.move_to_joint_positions(coords.ResetPos)
limbL.move_to_joint_positions(coords.left_back)
limbR.move_to_joint_positions(coords.ortho_view)
board = right_image
thresh, bd = cv2.threshold(board, 127,255,cv2.THRESH_BINARY)
array = []
- 59 maxspace = -1
meanmax = 0
for i in range(len(board_spaces)):
if board_spaces[i].piece == 0:
bs = board_spaces[i]
space = bd[bs.y1:bs.y2, bs.x1:bs.x2]
if np.mean(space) >= 0.4:
if np.mean(space) >= meanmax:
meanmax = np.mean(space)
maxspace = i
line = cv2.line(bd, (bs.x1,bs.y1), (bs.x2,bs.y2),
(0,255,0), 1)
if maxspace >= 0:
bs = board_spaces[maxspace]
cv2.imwrite('conts.jpg',bd)
limbR.move_to_joint_positions(coords.tuck_right)
move_to(bs)
bs.piece = IDBlock(left_image)
move_from(bs)
limbR.move_to_joint_positions(coords.tuck_right)
def turnOne():# A function instructing Lucas with what to do on its first turn.
global board_spaces
global direction
global next_space
gripper.calibrate()
limbR.move_to_joint_positions(coords.tuck_right)
limbL.move_to_joint_positions(coords.ResetPos)
startTurn()
pickUp()
if board_spaces[17].piece == 0:
place(17)
direction = "up"
next_space = 12
else:
place(12)
direction = "down"
next_space = 7
def IDBlock(img):# A function to Identify the type of a block
img = img[0:300, 160:520]
thresh, im = cv2.threshold(img, 127,255,cv2.THRESH_BINARY)
imgray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)
ret,thresh = cv2.threshold(imgray,127,255,0)
contours,
hierarchy
cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
cv2.drawContours(im,contours,-1,(0,255,0),-1)
my_contours = []
tile_type = 0
for cnt in contours:
if cv2.contourArea(cnt)>2000:
my_contours.append(cnt)
cv2.drawContours(im,my_contours,-1,(255,0,0),-1)
if(len(my_contours)==1):
tile_type = 1
elif(len(my_contours)==0):
=
- 60 tile_type = 0
else:
centres = []
for i in range(len(my_contours)):
mom = cv2.moments(my_contours[i])
centre = Point()
centre.x = (int(mom['m10']/mom['m00']))
centre.y = (int(mom['m01']/mom['m00']))
centres.append(centre)
line
=
cv2.line(im,
(centres[0].x,centres[0].y),
(centres[1].x,centres[1].y), (0,255,0), 1)
if (centres[0].x > centres[1].x) and (centres[0].y > centres[1].y):
tile_type = 2
elif (centres[1].x > centres[0].x) and (centres[1].y > centres[0].y):
tile_type = 2
else: tile_type = 3
print tile_type
return tile_type
def boardStatePrint():
#This just prints out a nice little display of Lucas' current perspective of
the board state into the console
print "
1
2
3
4
5 "
print
"A","|",board_spaces[0].piece,"|",board_spaces[1].piece,"|",board_spaces[2].piece,"
|",board_spaces[3].piece,"|",board_spaces[4].piece,"|"
print
"B","|",board_spaces[5].piece,"|",board_spaces[6].piece,"|",board_spaces[7].piece,"
|",board_spaces[8].piece,"|",board_spaces[9].piece,"|"
print
"C","|",board_spaces[10].piece,"|",board_spaces[11].piece,"|",board_spaces[12].piec
e,"|",board_spaces[13].piece,"|",board_spaces[14].piece,"|"
print
"D","|",board_spaces[15].piece,"|",board_spaces[16].piece,"|",board_spaces[17].piec
e,"|",board_spaces[18].piece,"|",board_spaces[19].piece,"|"
while left_image==None:
pass
#-------------Calling turnOne() and the main game loop-------------------------------------#
turnOne() #Start the game!
while gameLoss == False: # The main game loop, where the action happens and it all
comes together.
startTurn()
print "State: ", current_piece, " ", direction, " ", current_space,
board_spaces[current_space].piece
current_space = next_space
if board_spaces[current_space].piece != 0:
while board_spaces[current_space].piece != 0:
current_space = nextSpace(board_spaces[current_space].piece)
changeDirection()
if current_space == 66:
gameLoss = True
break
- 61 print "State: ", current_piece,
current_space, board_spaces[current_space].piece
"
",
direction,
"
",
pickUp()
print "State: ", current_piece, " ", direction, " ", current_space
if current_piece == 1:
place(current_space)
next = nextSpace(current_piece)
if next == 66:
gameLoss = True
elif current_piece == 2:
print "State: ", current_piece, " ", direction, " ", current_space
next = nextSpace(current_piece)
altnext = nextSpace(3)
print "next: ", next, " , ", altnext
next == 13:
if next == 66:
if altnext == 66:
place(current_space)
gameLoss = True
else:
rotate()
place(current_space)
elif next == 6 or next == 7 or next == 8 or next == 11 or next == 12 or
place(current_space)
next_space = next
elif altnext == 6 or altnext == 7 or altnext == 8 or altnext == 11 or
altnext == 12 or altnext == 13:
rotate()
place(current_space)
next_space = altnext
else:
place(current_space)
next_space = next
elif current_piece == 3:
next = nextSpace(current_piece)
altnext = nextSpace(2)
print "next: ", next, " , ", altnext
if next == 66:
if altnext == 66:
place(current_space)
gameLoss = True
else:
rotate()
place(current_space)
elif next == 6 or next ==7 or next ==8 or next ==11 or next ==12 or
next == 13:
place(current_space)
next_space = next
elif altnext == 6 or altnext ==7 or altnext ==8 or altnext ==11 or
altnext ==12 or altnext ==13:
rotate()
place(current_space)
next_space = altnext
- 62 else:
place(current_space)
next_space = next
changeDirection()
boardStatePrint()
print next_space
if gameLoss == True:
print "+++Divide By Cucumber Error. Please Reinstall Universe And Reboot+++"
sys.exit()
E. iii: Demonstration Video
The (less-than-perfect) demonstration video16 for the final implementation of this project can
be found on YouTube here:
https://www.youtube.com/watch?v=YEtFgFRH7wk
Alternatively you can download the non-annotated version from DropBox here:
https://www.dropbox.com/s/nbhjdbjevfaxi47/Final%2520Project%2520Demo%2520Tabletop%2520Robotics.mp4?dl=0
E. iv: GitHub Repository
The GitHub repository containing all of the files for this project can be found here:
https://github.com/sc12pas/TTR
16
I went into the labs on 12/5/15 to attempt to re-record the demo video but found that Lucas
had been moved and I could not recreate the lighting environment necessary while it was
in its new position, and struggled to place the board at the right height for the mechanical
system to operate fully.