PORTLAND STATE UNIVERSITY
ISOBOT SOCCER
Padmashri Gargesa
INTELLIGENT ROBOTICS II (Winter 2011)
Contents
Introduction.................................................................................................................................................. 4
Demo Objective .......................................................................................................................................... 4
Hardware ..................................................................................................................................................... 5
i-SOBOT .................................................................................................................................................. 5
Gigaware® 1.3MP webcam .................................................................................................................. 5
USB UIRT ................................................................................................................................................ 5
IR Reception: ...................................................................................................................................... 6
IR Transmission:................................................................................................................................. 6
Software ....................................................................................................................................................... 7
Uuirtdrv .................................................................................................................................................... 7
OPENCV.................................................................................................................................................. 7
Visual Studio C++................................................................................................................................... 7
Environment Setup..................................................................................................................................... 8
Vision System Setup.............................................................................................................................. 8
Soccer Field Setup ................................................................................................................................. 8
Object Color Setup ................................................................................................................................. 9
Object Detection ....................................................................................................................................... 11
Simple Background Cancellation ....................................................................................................... 11
Field Coordinates ................................................................................................................................. 11
LEVEL I .............................................................................................................................................. 12
LEVEL II ............................................................................................................................................. 13
Ball Coordinates ................................................................................................................................... 14
Front Bot Coordinates ......................................................................................................................... 15
Rear Bot Coordinates .......................................................................................................................... 17
Trajectory Planning .................................................................................................................................. 20
Coordinate System............................................................................................................................... 20
Vector Orientation ................................................................................................................................ 20
Design .................................................................................................................................................... 21
State Diagram for the BOT ............................................................................................................. 21
Command Flow................................................................................................................................. 22
Rotation logic .................................................................................................................................... 24
Page 2
Trajectory Plan .................................................................................................................................. 30
Geometric Calculations ................................................................................................................... 30
Challenges Faced .................................................................................................................................... 32
Link and References ................................................................................................................................ 33
Page 3
Introduction
Robo Soccer involves the challenge of combining individually complex subject areas such as
real-time vision, trajectory planning and robot motion control. The idea behind this project was to
gain an introductory experience into the process and approach taken by robo soccer teams
which compete in main stream tournaments. Takara Tomy’s i-SOBOT is the smallest humanoid
robot in the market with a pre-programmed set of normal motion, combat, soccer etc
commands. Since the intention was to work with humanoid robots playing soccer, and also due
to its small size, i-SOBOT was the easier and natural choice to work with.
Demo Objective
To setup an overhead camera overlooking the entire soccer field, which is to act as the vision
system for the soccer robot. Find real-time co-ordinates of the goal, the ball and the bot. Also,
be able to calculate orientation of the bot. Once relative positions of all key components are
determined, calculate a feasible shot region and a trajectory. Issue IR commands to move the
bot along the trajectory thus calculated.
Page 4
Hardware
i-SOBOT
According to the Guinness Book of World Records i-SOBOT, with dimensions H: 6.5" x W: 3.8"
x D: 2.6“ is the world’s smallest mass produced humanoid robot. It comes equipped with 2
gyroscopic sensors which help maintain its balance. It uses 3 rechargeable AAA size NiMH
batteries (Nickel Metal Hydride), which provide about a full hour of continuous play. 3 CPU chips
control the robot’s general processing, voice recognition, and servo control systems. 17 servos
provide i-SOBOT with a truly impressive skill level in general agility and physical movement. It
operates in 4 different modes to respond to roughly 200 commands. All of these modes except
the voice control mode receive commands through the IR sensor mounted on the robot’s back.
Gigaware® 1.3MP webcam
This Gigaware 1.3MP webcam does not require drivers to install and is plug-and-play
compatible. It captures still photos at a maximum (interpolated) resolution of 5.2-megapixels
(2560x2048) and captures video at a maximum resolution of 1.3-megapixels (1280x1024). It
features a built-in microphone and a multipurpose stand, ideal for laptop monitors and flat
surfaces. Comes with a webcam enhancement software.
USB UIRT
This IR Transceiver is used to fire commands to the i-SOBOT. This device is a USB version of
the UIRT (stands for Universal Infrared Receiver Transmitter). The USB version offers a simple
plug-and-play solution without having to deal with some of the hassles of the classic serial-port
(RS-232) version. The USB-UIRT contains a small micro-controller which is capable of 'listening'
Page 5
for Infrared codes transmitted from most remote controls. When IR signals are detected, the
USB-UIRT interprets these signals, decodes them, and sends them to your PC via a USB
connection. Automation Softwares such as ‘EventGhost’ can be programmed to respond to
these codes. In addition, software on the PC can send IR codes to the USB-UIRT to be
transmitted. The USB-UIRT's microcontroller will translate these codes to an Infrared stream
and transmit them using built-in IR emitters or a connected IR extension.
IR Reception:
• 34-40KHz Frequency Range
• *56KHz IR Receiver Add-On Option*
• UIR-compatible Receiver Mode
• >8 Meter Sensitivity
• IR Wavelength Filtered
• USB Wake-from-Standby Capability
• Built-in wideband IR detector for accurate Learning, including carrier frequency.
IR Transmission:
• 20-60KHz Frequency Range
• Two (2) built-in High-Power Emitters
• 1/8" Mini-Jack for External Emitters
• UIRT-compatible Struct and Raw modes
• Extended-length code support in hardware (up to 96burst-pairs)
• Max-length burst code support in driver (up to 2048 burst-pairs)
Page 6
Software
Uuirtdrv
This is a manufacturer supplied API for the UIRT device implemented in C. This can be used to
poll the UIRT for events and programmatically send/receive IR codes to the USB UIRT.
OPENCV
Open CV is a set of open source libraries to help process real time still and video camera
images. Written in C and C++ and originally designed to advance the CPU intensive computer
vision related operations, the different libraries in Open CV are designed specifically to support
and handle data strucures and matrix algebra involved in image transformations, image
processing, machine learning, highlevel graphical user interface, camera interface etc.
Visual Studio C++
The OpenCV and USB UIRT device libraries are integrated into one win32 application, which
polls the webcam continuously for real-time video feed.
Page 7
Environment Setup
Vision System Setup
A 1.3 MP overhead webcam connected to the PC through a 10’ USB extension cord was setup
to overlook the entire soccer field and act as a centralized vision capture and feed to the
decision making system.
Soccer Field Setup
Page 8
The Soccer field is a 53’’X45’’ bounded rectangle. It’s at a vertical distance of 66’’ from the
overhead real-time vision feed. The soccer field was chosen to have a black background to
absorb as much light from the overhead lights and thus to minimize noise/color interference with
other key objects on the field. A green color border is used to calculate the field coordinates in
real-time.
Object Color Setup
The key elements and their color choices are as listed in the below table. The reason for those
color choices was to place them as wide apart in the color scale as possible so that thresholding
based on hue values during blob detection would be easier.
Object
Color
Comments
Field border
Green
To
detect
the
rectangle
bounding the entire soccer field.
To be used as ROI (Region of
interest) for the rest of the
processing.
Ball
Peach
Since all pure colors were
already assigned to other key
players, this less interfering
color was chosen for the Ball.
Page 9
Goal
White
To detect goal coordinates.
Orientation of the goal is
assumed to be parallel to the
nearest edge of the field.
Bot
Front Green Tile and Rear Tile like pattern was chosen
Red Tile
detect bot co-ordinates and
orientation.
Page 10
Object Detection
Simple Background Cancellation
An image of the soccer field sans border and all other objects is captured. This image is
subtracted from every real-time image feed for all of the further object detections. Since area of
interest (soccer field) in the below image is black, there is no significant shift in the color pixel
values when a literal background subtraction is done on the real-time video feed.
Field Coordinates
This is done initially to calibrate the position and dimension of the soccer field. The input image
for this phase is real-time image feed from the overhead camera with just the black soccer field
and the green border. This image is taken through 2 levels of image transformation and
processing to determine precise bounding box representing the soccer field. This approach was
taken to filter out surrounding noise as much as possible through the first phase so that the
second phase had a more accurate detection of the largest bounding contour. The coordinates
thus found are set for the rest of the game. Any changes to the position of the field will need a
re-calibration.
Page 11
LEVEL I
Threshold between
Subtract
background from
the captured image
Convert to HSV
Scale
50 < H < 180
170 < S <256
50 < V < 180
Resulting image is
passed through
Gaussian
smoothing
This transforms the image as shown
Input Image
Output Image
Page 12
Hough transform
to detect longest
lines in the image
lenghtwise and
breadthwise to
determine field
coordinates
LEVEL II
Subtract
background from
the captured image
Convert input
image to grayscale
Pass the resulting
image through
morphological
opening operation
Set the bounding
box determined by
the previous stage
as ROI
Contour detection
to find the
bounding box with
the largest area
and set it as the
final field
coordinates
This transforms the image as shown
Input Image
Output Image
Page 13
Ball Coordinates
The peach colored ball’s dimensions are calibrated initially before game start. The same
process is followed before and after game start to determine ball co-ordinates.
Subtract
background from
the captured
image
Threshold between
6 < H < 35
Convert input
image to HSV
35 < S <256
110 < V < 256
Pass the resulting
image through
morphological
opening
operation and
Gaussian
smoothing
Set ROI based on
the field
coordinates
previously
detrmined
This transforms the image as shown
Input Image
Output Image
Page 14
Contour
detection to find
the bounding box
which
approximately
has the same
dimension as the
pre-recorded ball
dimensions
Front Bot Coordinates
The bot has green and red tiled pattern placed on its head to determine its coordinates and
orientation.
The position of the rear red tile is determined by passing the input image through 2 levels of
transformation since the hue values for red are split between 0-6 and 170-200.
Subtract
background
from the
captured image
Threshold
between
Convert input
image to HSV
0<H<6
84 < S <256
84 < V < 256
Subtract
background
from the
captured image
Threshold
between
Convert input
image to HSV
170 < H < 200
84 < S <256
84 < V < 256
Pass the
resulting image
through
morphological
dilation
operation
Pass the
resulting image
through
morphological
dilation
operation
The images resulting from both the above processes are added to get the contour of the red tile
at all distinguishable levels of hue, saturation and brightness.
Page 15
Set ROI based on the
field coordinates
previously
determined
Contour detection to
determine the rear
bot co-ordinates
This transforms the image as shown
Input Image
Output Image
Page 16
Rear Bot Coordinates
The position of the front green tile is determined by passing the input image through the
following transformation.
Subtract
background
from the
captured image
Threshold
between
Convert input
image to HSV
60 < H < 100
84 < S <256
84 < V < 256
Pass the
resulting image
through
morphological
dilation
operation and
Gaussian
smoothing
Set ROI based
on the field
coordinates
previously
detrmined
Contour
detection to
determine
front bot
coordinates
This transforms the image as shown
Input Image
Output Image
The below image is a result of overlapping the above 2 images separated in the red and green
scales. A line joining the center co-ordinates of these 2 bounding boxes gives the bot
orientation.
Page 17
Goal Coordinates
The goal coordinates and dimensions are calibrated initially before game start and these prerecorded dimensions are used to identity a contour with matching dimensions as the goal when
the game is in progress and when there are other objects in the picture which may be
interfering. In both scenarios, the image transformation procedure for goal detection is as below.
Subtract background
from the captured
image
Convert input image to
grayscale
Pass the resulting
image through
morphological
opening operation
Page 18
Set ROI based on the
field coordinates
previously
determined
Contour detection
to find the bounding
box which does not
correspond to any
of the above object
co-ordinates and
also which
approximately has
the same dimension
as the pre-recorded
goal dimensions
This transforms the image as shown
Input Image
Output Image
Page 19
Trajectory Planning
Coordinate System
The whole soccer field is treated as a 2D Euclidean space. The (x, y) coordinates determined
for the field corners are as shown the.
(a, b)
(a+w, b)
+ve increase in x co-ordinates
+ve increase in y co-ordinates
(a, b+h)
(a+w, b+h)
Vector Orientation
With respect to the above diagram the vector orientation is calculated as shown
90°
0°
180°
270°
Page 20
Design
State Transition Diagram for the BOT
Object Coordinates not found
Poll for
object coordinates
Found all object coordinates
Calculating shot
region and bot
alignment
Inside shot region at an angle from shot
coordinates/Issue Command
Finished command
execution
Outside shot region
Calculating next
point on trajectory
and relative bot
Finished command
alignment
execution
Inside shot region aligned angularly with
shot co-ordinates/ Issue Command
At an angle with the trajectory
coordinates/Issue Command
Aligned angularly with trajectory coordinates/ Issue Command
Rotating along
the calculated
angle
Walking
Forward
Page 21
Command Flow
Start
Yes
Stop
User
clicked
esc?
B
No
Calculate
object
coordinates
No
All
coordinates
available?
Yes
Calculate strike line
Calculate shot region
Bot within
shot
region?
No
A
Page 22
Yes
Bot aligned
with strike
line?
Yes
No
Rotate
Walk Fwd
A
B
1 == 1
D
Yes
Bot within
shot
region?
Yes
Bot aligned
with strike
line?
Yes
No
Calculate bot and
ball coordinates
Yes
Ball
coordinates
changed?
No
Calculate relative position of
bot with the ball and thus
the next point on the
trajectory
C
Page 23
Rotate
Walk Fwd
C
Bot angularly
aligned with
trajectory?
Yes
No
Rotate
D
Rotation logic
Start
No
Bot angle
>=0 and Bot
angle <=90
Yes
B
A
Page 24
Walk Fwd
B
A
Traj angle
>=0 and Traj
angle <=90?
Yes
Bot-traj <
0
Yes
No
No
Traj angle >90
and Traj angle
<=180?
Walk Fwd
Curve Left
Yes
Walk Fwd
Curve Right
Walk Fwd
Curve Left
No
Traj angle
>180 and Traj
angle <=270?
Yes
Bot-traj <
= 180
No
Yes
Walk Fwd
Curve Right
No
Traj angle
>270 and Traj
angle <=360?
No
C
Page 25
Yes
Walk Fwd
Curve Right
Walk Fwd
Curve Left
C
No
Bot angle
>90 and Bot
angle <=180
Yes
Traj angle
>90 and Traj
angle <=180?
Yes
Bot-traj <
0
Yes
No
No
Traj angle >=0
and Traj angle
<=90?
Walk Fwd
Curve Right
Yes
Walk Fwd
Curve Left
Walk Fwd
Curve Right
No
Traj angle
NoTraj
>270 and
angle <=360?
Yes
Bot-traj <
= 180
No
Yes
No
E
D
Page 26
Walk Fwd
Curve Right
Walk Fwd
Curve Left
E
D
Traj angle
>180 and Traj
angle <=270?
Yes
Walk Fwd
Curve Left
No
No
Bot angle
>180 and Bot
angle <=270
Yes
Traj angle
>180 and
Traj angle
<=270?
Yes
Bot-traj <
0
Yes
No
No
Traj angle >90
and Traj angle
<=180?
No
G
F
Page 27
Walk Fwd
Curve Right
Yes
Walk Fwd
Curve Right
Walk Fwd
Curve Left
G
F
Traj angle
>270 and Traj
angle <=360?
Yes
Walk Fwd
Curve Left
No
Traj angle >=0
and Traj angle
<=90?
Yes
Bot-traj <
= 180
No
Yes
No
Walk Fwd
Curve Left
Walk Fwd
Curve Right
No
Bot angle
>270 and Bot
angle <360
Yes
Traj angle
>270 and
Traj angle
<=360?
Yes
Bot-traj <
0
Yes
No
No
I
H
Page 28
Walk Fwd
Curve Right
Walk Fwd
Curve Left
H
H
Traj angle >=0
and Traj angle
<=90?
Yes
Walk Fwd
Curve Left
No
Traj angle >90
and Traj angle
<=180?
Yes
Bot-traj <
= 180
No
Yes
Walk Fwd
Curve Right
No
Traj angle
>180 and Traj
angle <=270?
No
Walk Fwd
Stop
Page 29
Yes
Walk Fwd
Curve Right
Walk Fwd
Curve Left
Trajectory Plan
A smooth cosine curve like trajectory is planned from the bot coordinates to the shot region.
Geometric Calculations
1. Calculate Shot region:
Angle of strike line
𝜃 = 𝑎𝑡𝑎𝑛⁡(𝑌𝐺𝑜𝑎𝑙 − 𝑌𝐵𝑎𝑙𝑙 /𝑋𝐺𝑜𝑎𝑙 − 𝑋𝐵𝑎𝑙𝑙 )
Normal to strike line angle
∝= 𝑎𝑡𝑎𝑛⁡((𝑌𝐺𝑜𝑎𝑙 − 𝑌𝐵𝑎𝑙𝑙 ) × −1/𝑋𝐺𝑜𝑎𝑙 − 𝑋𝐵𝑎𝑙𝑙 )
Shot Coordinates
𝑋𝑆ℎ𝑜𝑡 = 𝑋𝐵𝑎𝑙𝑙 − ⁡𝑘1⁡ × 𝑐𝑜𝑠(𝜃)
𝑌𝑆ℎ𝑜𝑡 = 𝑌𝐵𝑎𝑙𝑙 − ⁡𝑘1⁡ × 𝑠𝑖𝑛(𝜃)
Back Projection Coordinates
𝑋𝐵𝑃𝑟𝑜𝑗 = 𝑋𝐵𝑎𝑙𝑙 − ⁡𝑘2⁡ × 𝑐𝑜𝑠(𝜃)
𝑌𝐵𝑃𝑟𝑜𝑗 = 𝑌𝐵𝑎𝑙𝑙 − ⁡𝑘2 × 𝑠𝑖𝑛(𝜃)
Page 30
Shot Region Vertices
𝑋𝑣1 = 𝑋𝐵𝑃𝑟𝑜𝑗 − ⁡𝑘3⁡ × 𝑐𝑜𝑠(∝)
𝑌𝑣1 = 𝑌𝐵𝑃𝑟𝑜𝑗 − ⁡𝑘3⁡ × 𝑠𝑖𝑛(∝)
𝑋𝑣2 = 𝑋𝐵𝑃𝑟𝑜𝑗 + ⁡𝑘3⁡ × 𝑐𝑜𝑠(∝)
𝑌𝑣2 = 𝑌𝐵𝑃𝑟𝑜𝑗 + ⁡𝑘3⁡ × 𝑠𝑖𝑛(∝)
Shot co-ordinates and shot region vertices together form the shot region.
2. Check if Bot is within shot region:
Angle between front bot coordinates and shot coordinates
𝜃1 = 180 − ⁡𝑎𝑡𝑎𝑛⁡(𝑌𝐹𝐵𝑜𝑡 − 𝑌𝑆ℎ𝑜𝑡 /𝑋𝐹𝐵𝑜𝑡 − 𝑋𝑆ℎ𝑜𝑡 )
Angle between front bot coordinates and shot region vertex 1
𝜃2 = 180 − ⁡𝑎𝑡𝑎𝑛⁡(𝑌𝐹𝐵𝑜𝑡 − 𝑌𝑣1 /𝑋𝐹𝐵𝑜𝑡 − 𝑋𝑣1 )
Angle between front bot coordinates and shot region vertex 1
𝜃3 = 180 − ⁡𝑎𝑡𝑎𝑛⁡(𝑌𝐹𝐵𝑜𝑡 − 𝑌𝑣2 /𝑋𝐹𝐵𝑜𝑡 − 𝑋𝑣2 )
If abs (360 – (𝜃1 + ⁡𝜃2 + ⁡𝜃3)) < 0 then Bot is in shot region
Page 31
Challenges Faced
a. Problem: Considered mounting a mobile camera on the bot and giving it autonomous
vision. The bot being small, mounting the camera shifted its center of gravity resulting in
the gyro motor responsible for keeping the bot erect being on all the time thus draining
the battery much faster
Solution: Considered centralized vision through an overhead camera.
b. Problem: Any change in lighting causes the color values to shift.
Solution: This is not a flexible design and success of the system is tightly coupled to
providing the right environmental setup with specific emphasis on lighting.
c. Problem: ISOBOT is pre-programmed with 140 commands for motion, combat etc.
Relying on these commands to move the bot along the trajectory was a challenge and
was difficult to control.
Solution: A continuous vision feedback on the bot location and accordingly changing
the trajectory in real-time was the only way out of this.
d. Problem: A constant feedback from the vision system was resulting in the ball and goal
coordinates to shift by small amounts constantly and this made it extremely cumbersome
to calculate trajectory and issue commands to the bot.
Solution: A nested loop was introduced which would only monitor bot position in realtime while the goal and ball positions are fixed. This loop breaks only if there is a
significant shift in the ball position.
Page 32
Link and References
i.
ii.
iii.
iv.
v.
http://www.youtube.com/watch?v=SUIOWowloTk
http://opencv.willowgarage.com/wiki/
http://www.usbuirt.com/
http://www.academypublisher.com/proc/wisa09/papers/wisa09p267.pdf
http://www.lirtex.com/robotics/fast-object-tracking-robot-computer-vision/
Page 33