PSU ECE Robotics 2 Class Project
Tony Muilenburg, Dawson Green
Computer vision and control
Baby Schrodinger Cat (CatBot)
Contents
Project summary ........................................................................................................................................... 3
Robot cat ....................................................................................................................................................... 3
Robot firmware code .................................................................................................................................... 3
Robot control code ....................................................................................................................................... 4
Computer vision code ................................................................................................................................... 5
Next steps ..................................................................................................................................................... 7
CatBot Control Introduction ......................................................................................................................... 7
Main Execution File (main.cpp) .................................................................................................................... 7
Compilation Warnings .............................................................................................................................. 7
Namespaces .............................................................................................................................................. 8
Global Definitions File (definitions.h) ........................................................................................................... 8
#define values ........................................................................................................................................... 8
Use of enums ............................................................................................................................................ 8
Use of ref structs & ref classes.................................................................................................................. 8
Port Manager Object (PortManager.h & PortManager.cpp) ........................................................................ 8
Port Available and Robot Ready functions ............................................................................................... 9
Sending BT Messages to an NXT ............................................................................................................... 9
SerialPort object...................................................................................................................................... 10
Catbot Auto Control object (CatbotAutoCtrl.h & CatbotAutoCtrl.cpp) ...................................................... 10
FilePath variable...................................................................................................................................... 10
Reading OpenCV data ............................................................................................................................. 10
Calculating Catbot Motion ...................................................................................................................... 10
Application UI (AppUI.h, AppUI.cpp & AppUI_Events.cpp) ........................................................................ 11
Functions for the Setup Tab .................................................................................................................... 11
Manual Controls...................................................................................................................................... 11
Automatic Control................................................................................................................................... 11
Project summary
With the increase in speed of personal computers many things have become possible. One of the most
interesting applications is computer vision. For this project we used computer vision to control a robot
as part of a robot theater program Dr. Marek Perkowski putting together (Link). Using a single overhead
camera we were able to control a robot, tracking its position and angle, and have it autonomously drive
toward an arbitrary target location. This report will describe the steps necessary to reproduce what we
did, as well as cover the materials used.
Robot cat
Because another team was working on the Schrodinger cat robot, we decided to create use a Lego
Mindstorms robot, and added a baby kitten on top of it. The base for the robot came from the 9797
First Lego League kit. This kit is described on Lego’s website at this link: 9797. Instructions for
assembling the robot can be found here: robot base instructions. To simplify tracking of the robot, a
green circle, and a pink square were taped to the top to the baby kitten toy. Having two different colors
allowed us to calculate the direction the cat was facing. The cat riding the robot is shown below before
the colors were added.
Robot firmware code
Communication with the robot is achieved using Bluetooth. The Lego NXT robot is equipped with
Bluetooth, though sending commands using a laptop is not possible using Lego’s standard code. To
communicate with the NXT and set it up to accept commands, the firmware on the NXT brick is first
overwritten using a program called RobotC. This program can be used for free for ten days on a
computer, after which the program must be purchased for $50. Fortunately, once the firmware has
been written to the robot, RobotC no longer needs to be used.
Robot control code
The code used to send commands to the robot was written in visual studio using visual c++, and is called
catbot control. The following figure outlines the project structure:
The user interface is split up into two tabs. The first allows the user select the Bluetooth device that is
to be used, and configure it. While some laptops come with integrated Bluetooth, the laptop used for
this did not, so a USB dongle was added which added it. After selecting the Bluetooth port, the connect
button will enable communication with the NXT equipped robot that has been flashed with the robotc
firmware created for this project. The computer vision code uses a text file to pass location data to this
robot control code. The path to the file the computer vision code generates should be added to the
textbox using the find file button which pops up a file browse dialog window. The “Test File” button
should then be used to ensure that the path is valid and file permissions are ok (permission to read the
file is granted).
The second tab has buttons to allow for manual or automatic control. In manual mode, the robot can be
turned, or orders can be given for it to go forward or backward. A speed control is available, as well as
an emergency stop (see below).
In automatic mode, the code will read the coordinate from the file the computer vision code generated,
then a heading and distance will be generated from the robot to a blue dot, and commands will be send
to have the robot turn and drive to the blue dot. The coordinates are written and read many times per
second to ensure that the robot does not drive too far, or get off course.
The code for this project can be downloaded from this link: CatBot control code
Computer vision code
By default, the compute vision code (cat tracking opencv) opens many windows. The following figure
show what screen looks like in debug mode.
The top four windows show the threshold image for a particular color. The colors red, blue, green and
pink are tracked, though red is not currently in use (see the next steps section for more on this). In
looking at the green and pink threshold windows, it is clear that these colors exist in the camera feed.
All other colors are filtered out by specifying filter in HSV format (follow this link for a description of
HSV). Along the bottom, there are four windows each of which have sliders that can be used to adjust
the H, S, and V values and filter out all but the desired color to be tracked. The “camera feed” window
in the middle of the lower portion of the screen shows what the camera sees along with superimposed
circles that correspond to areas being tracked.
The code writes the coordinates of each of the discovered colors to a file called “cat_tracking.txt” which
is then read by the cat control code, which guides the cat to the desired location. The format of the file
is given below:
Next steps
While many of the next steps are outlined in Dr. Perkowski’s robot theatre script, some of the most
obvious next steps include:





Put together a third option for scripted control in addition to manual and automatic control of
the baby Schrodinger cat.
Enable tracking and control of other robots, including Schrodinger cat, Einstein, Newton, Bohr,
Marie Curie, etc.
Enable finer granularity for the fuzzy logic speed control
Incorporate depth control using a Kinect camera
Train and use a Haar cascade classifier to recognize robots, and robot orientation (color would
no longer be needed for tracking).
CatBot Control Introduction
This document describes the breakdown of files used in the CatbotControl program for the baby
Schrodinger Cat robot (referred to as catbot throughout). Each section of the project is broken down in
turn and notes on the usage and setup configuration for it are discussed. Be aware that full understanding
of the code will likely only come from reading it and experimenting with it. However, this should hopefully
help expedite that process along. If any objects or styles of code are unfamiliar to you when you come
across them, please refer to the Microsoft MSDN for details about Visual C++ and the Common Language
Runtime (CLR).
NOTE: This document assumes you are using Visual Studio Professional 2013. If you are using a different
IDE to compile this project, directions to edit settings may be different. Please use your best judgement
to interact with your IDE UI and update this document if future classes will use a different program to
interact with this code.
Main Execution File (main.cpp)
The main routine is cleverly hidden in the main.cpp file. This file is fairly barebones and is a useful format
for how to invoke a GUI in Visual C++. The file does contain a few notes on project settings, so make sure
these are configured correctly before compiling the code. To edit them, right click the project in the
solution explorer and select “Properties”. This opens the master set of project properties for modification.
Compilation Warnings
Any time the code produces are warning for C4101, you are free to ignore it. This warning is
generated by the compiler letting you know that a local variable isn’t actually used in its function
and most often caused by try/catch blocks. If in doubt, check the line of code throwing the
warning. There will be a comment if it was expected and should be ignored.
Namespaces
When possible, this code declares and uses namespaces to shorten down conventions. However,
for accuracy, this document will often expand out the namespace of an object so it’s clear what
object is being discussed. Namespaces are separated by the double colon indicator “::”, with the
last keyword on the line being the actual object in question. Please check Google if you need more
information on how these work.
Global Definitions File (definitions.h)
The global definition file is included in all other header files. It is used as a centralized hub for all file
includes, all #defines and all the global variables in this namespace. Note that #defines aren’t restricted
by Visual C++ to a particular namespace, so care should be exercised about naming conventions for them.
#define values
Visual C++ doesn’t provide a way to define string constants when using CLR. While most numerical
values have been locked in as globals, we have not found a good way around using #defines to
drop in strings to path files yet. Please update this section if you find a way.
Use of enums
If you’re not familiar with enums, they provide a good way to turn keywords into integers. Each
enum defaults to numbering its elements from 0 to length-1. While you can manually order these
and change the numbering system, this isn’t encouraged. It is expressly prohibited for the BotID
enum, as the TOTAL_BOTS value would then be inaccurate. Please refer to Google if you need
more information on how these work.
Use of ref structs & ref classes
The really cool thing about Visual C++ is that it has a garbage collector available to it that’s similar
to other garbage collected languages. However, this causes a split between “managed” code and
“native” code. Managed objects can always be identified by the hat (“^”) character at the end of
their declaration (“Object^”). If all references to an object are deleted by destructor or go out of
scope of the code, then the objects destructor will be automatically invoked. You can create your
own managed objects simply by putting the keyword “ref” before the declaration of any struct or
class you create. For the most part, this project uses them to make life easier and more
automagical.
Port Manager Object (PortManager.h & PortManager.cpp)
This object provides a useful abstraction for interacting with the catbot and any future robots you choose
to tie into this project. At its core, it is designed to manage all of the serial ports (COM ports) used to
communicate with robots controlled by this application. From the outside, a user simply passes the
appropriate BotID and this object will handle sending or receiving data to the right robot. While only
catbot is currently established in the code right now, stubs are provided to show how the functions can
be expanded to control a full array of robots. NOTE: This object can only interact with robots registered
to a Windows COM port. If you have another communication system, you’ll need a new class.
Port Available and Robot Ready functions
While the NXT provides the Bluetooth Serial Port Profile (SPP) that allows it to be controlled over
a COM port when paired to the computer, the BT receiver on it is fairly slow. It’s so slow, in fact,
that testing it for availability is a prohibitively time expensive operation. Therefore, this code will
only check if ports are currently owned by this program and configured for this program’s robots
before trying to interact with them. If the port gets hijacked by another process, this code won’t
realize it and will probably start doing weird things.
Sending BT Messages to an NXT
While the catbot has its own code to interpret the messages to the NXT and translate them into
motions, the NXT itself requires messages to be formatted in specific ways. Each communication
is referred to as a “telegram” and a master list can be found in the supplementary documentation
for RobotC. For this project, each message can be broken down and understood in the following
way:
array<Byte>^ BT_Msg = gcnew array < Byte > {0x07, 0x00, 0x80, 0x09, 0x00, 0x03, msg,
msgMag, 0x00};
1. Variable type (“array<Byte>^”) – Managed array of 8 bit bytes gets automatically disposed
of after sending.
2. Message Length (“0x07, 0x00”) – Each message must start with the length of the actual
payload, LSB first. This format indicates that there are 7 bytes being sent not including
the two bytes declaring the length.
3. Command Type (“0x80”) – This command doesn’t need a response from the NXT.
4. Command (“0x09”) – Our code is writing a message to the system
5. Message Mailbox (“0x00”) – The message will be in “mailbox1” for the RobotC code to
read from.
6. Message Size (“0x03”) – Number of bytes of actual message data (i.e. length of the
remainder of the array)
7. Message Contents (“msg, msgMag, 0x00”) – The RobotC code specifies that the robot
command and command speed are bytes 1 and 2 of the message. The NXT requires all
messages to terminate with a null character, so “0x00” must be tacked on to the end of
any message and included in the payload calculations.
Please refer to “Appendix 2 – LEGO MINDSTORMS NXT Direct Commands.pdf” for details of how
to format and send other messages to the NXT.
SerialPort object
Most of the structure of this code is making use of functions and properties provided by the
System::IO::Ports::SerialPort object. MSDN has excellent documentation about the properties of
this object and you’re encouraged to check it out if you need more info on an object function call.
Catbot Auto Control object (CatbotAutoCtrl.h & CatbotAutoCtrl.cpp)
Just as the PortManager helped abstract the interface layer for communicating with the robot, the Auto
Control object is designed to abstract the process of reading data from the OpenCV program and
determining what to do with that data. Because each robot will consume data differently, this object is
only targeted to the catbot. However, it should provide a good template if you want other robots to
consume OpenCV code in a similar fashion.
FilePath variable
While the object used to read data from the file in readData() can theoretically support relative
paths to the file, it is our experience that this can cause some odd read/write issues with the file.
These were fixed by using the absolute path of the variable. If you can ever figure out why, please
update this document with the answers.
Reading OpenCV data
The OpenCV program takes measurements of position of the necessary markers and provides
those in a valid text file. The GUI can find this file in the system and test that it is able to be read,
but reading that file requires a set of keys. Back in definitions.h, there’s a series of keywords that
corresponds to the data relevant to catbot from OpenCV. Note that ORDERING IS IMPORTANT
for the moment, as each keyword is mapped to a specific variable in the data reading. The order
the keywords appear in the text file doesn’t matter (or if unexpected keywords are there), but the
order in the definitions.h file does.
Each keyword expects an integer to be read from the textfile, separated by any of the valid
delimiters found in definitions.h. Currently, a space (‘ ‘) is used. This program will use position
data to calculate angles, so the more accurate those integers can be, the better. Values are
currently scaled to the number of pixels viewed by the webcam.
Calculating Catbot Motion
The algorithm works basically by calculating differences between the target and the current
position/heading of the catbot. Angles are constrained to being between -180 and +180 degrees,
with conversions being provided to transform radians into degrees. The process is that, if the
catbot is “close enough”, it will stop. If it is in the right general direction, it will drive forward. If it
still needs to turn, then it will turn in the “fastest” direction. Catbot motion resulting from this
calculation are then packaged and returned to the caller. NOTE: the calculation will return a null
pointer if the read OpenCV data was invalid or if the calculation causes a repeat in the motion
being sent to the catbot. The caller is expected to NOT send data if it gets a null pointer back.
Application UI (AppUI.h, AppUI.cpp & AppUI_Events.cpp)
We now come to the main application UI, where everything needed for the UI and everything else that
couldn’t be abstracted out has been crammed in. Much of the header file was created automatically by
the Visual Studio GUI creator interface. As such, you only need to refer to the top and bottom of the file
to find declarations made by our team. Also, code is split into UI events and “other” functions. AppUI.cpp
contains all the non-event function definitions.
Functions for the Setup Tab
These functions are mostly presented as is. They are primarily utilitarian and necessary to initialize
data needed for the actual control of the catbot. All functions here will be executed on the main
GUI thread, so we have tried to minimize slowdowns. In order to improve abstraction of the
methods (and allow greater extension of the code), helper functions have been created to gather
and package data necessary to pass to other functions. Similar GUI gather functions should be of
a similar format to catbotGetSettings(), as this allows utility functions further downstream to
update the GUI and modify the system via indirection.
Manual Controls
Once setup is complete, manual control can be enabled under the control tab. Logic for the
motion buttons (except for stop) is to allow user to ramp up motor speed by repeatedly clicking
on the same motion command (start at 10% power and increase by 10% each click). Each time a
new motion is started, it defaults to starting at 10% motor power. There is also a slider for manual
speed control. Updating the slider causes a thread to get started which reads the slider and waits
to send the command until it has been stationary for about 100 ms. This is again to accommodate
the NXT’s BT receiver only being able to handle new data once every 30 ms. On a stop or
emergency stop, the speed bar is set to 0 and disabled.
Also, enabling manual control will cause automatic control to disengage.
Automatic Control
Enabling automatic control will disable manual control and cause a new thread to be spawned.
This thread used to actually control and manage the Catbot Auto Control object. It is designed so
that it is frequently sleeping, then waking up to check if it should do work. The sleep/wake-up
model allows a volatile bool variable to be checked frequently. If this variable is ever set to false,
the thread will assume it should be ended and will exit the controlling function. Otherwise, it will
run through each of the 4 states and direct traffic between the auto controller and the port
manager to make sure messages get where they need to go.