Project Report
Group X-Factor
(Team Multitouch)
Thomas Bryan
Kristin Scudder
Tara Srihari
Senior Projects Spring 2009
CSC4790
2
Table of Contents
1. Introduction
Page 3
1.1 Purpose
Page 3
1.2 Project Description
Page 3
1.3 Intended audience
Page3
2. Hardware/Software Requirements Document
Page 4
2.1 Multitouch Table
Page 4
2.2 Game of Craps
Page 5
2.3 Checking-In Website/Application
Page 11
3. Explanation of design options and choices
Page 16
3.1 Multitouch table
Page 16
3.2 Game of Craps
Page 16
3.3 Checking-in Website/Application
Page 17
4. Suggestions for system extensions
Page 18
4.1 Multitouch table
Page 18
4.2 Game of Craps
Page 18
4.3 Checking-in Website/Application
Page 18
3
1. Introduction
1.1. Purpose
The purpose of this project was to construct a multitouch table and develop applications
that take advantage of the alternate input method of IR reflection location processing.
1.2. Project Description
The table uses Rear Diffused Illumination to allow multiple inputs over the surface of the table
from multiple users. The surface of the table is evenly illuminated with infrared light, which is
reflected from input objects (hands, fingers, or other objects placed on or slightly above the
input surface of the table). The IR light is detected by cameras, which feed the data to image
tracking software, which interfaces with applications over one of two protocols. The
applications under development were chosen to be a multi-purpose check-in application for
use in places such as doctor’s offices, interview sites, or tour sites, and a gaming application.
For the end-product we implemented a version for use at tour sites, namely Villanova’s
Admissions Office’s new student tours. The gaming application was developed as an emulation
of the Game of Craps, a common game for casinos who might find this sort of technology
interesting under the right circumstances.
1.3. Intended Audience
The intended audience is a wide group. The table’s intended audience is anyone interested in a
new method of interfacing with technology through a pseudo-touch interface. This group
ranges from children to the elderly with any number of needs. This technology enables highly
detailed modeling, digital music synthesis, interactive education, or any number of seemingly
unrelated tasks to be easily accomplished. The broad check-in application’s intended audience
is offices, security organizations, or tour sites. The implementation we have brought to
realization has a much narrower audience (Villanova’s Admissions Office), but reversecustomization is very easy. The intended audience for the Game of Craps is anyone interested
in gambling, or the casino industry itself. Fully automating the gambling industry would allow
casinos to make much more profit, with no need to pay dealers. Overall, this technology has an
extremely broad intended audience ranging from small children, the elderly, the education
sector, the engineering sector, the music industry, and many more unrelated sectors of the
world.
4
2. Hardware/Software Requirements Document
2.1. Multitouch Table
Functionality
The multitouch table is intended to offer an alternative method for people to interface
with computers (through “touch”). Through various applications operating over the
flash and/or TUIO protocols, the image tracking software (CCV) will enable input from
the user to be mapped to specific locations on the application level. CCV(Community
Core Vision) is one of several open-source tracking package developed by a community
centered at nuigroup.com.
External Interfaces
The table interacts with people by way of infrared light illumination of hands and fingers
(as well as any other object placed on the surface). The object is illuminated with IR
light, reflecting the light back to two modified cameras, which then feed the video data
to a tracking tool, which then outputs the tracking data over either the flash or TUIO
protocol.
Performance
The table’s speed depends entirely on the computer system to which it is connected by
VGA and USB connections. The projector runs optimally at 800x600 resolution, but can
obtain any resolution desired, as long as the connected computer system can output a
VGA signal in said resolution.
Attributes
The table is fairly portable, as it is mounted on four wheels (two free and two locking).
The plexiglass surface is easily removed to allow the unit to easily fit through doorways.
All components within the table, save the projector, are fastened with screws to
maintain even illumination and reception of images. The interactive surface is
28-7/8”x16-5/8”x1/4” clear plexiglass. The box itself is crafted of 1/4” plywood sheets
with 1”x1” plywood beams forming a skeleton around the sheets. The floor of the box is
1/2” pressure-treated chipboard, giving the box both sturdiness and making it easy to
mount components. The diffusion layer, as well as the projection surface, is a plastic
table cloth stretched across the viewport. There are two ventilation ports, each having
a 40mm fan to create air flow, and one doubling as a cable port for the VGA and USB
cables. An access door is located on the short side of the box to allow easy access to all
of the components within the box. The components mounted within the box are 5 48LED infrared illuminators, 2 modified PS3 Playstation Eye cameras, and a BenQ MP515ST
projector.
5
Finished Product
2.2 Game of Craps
Functionality
The Game of Craps (GoC) is one of two multitouch applications developed to be used in
conjunction with our rear diffused illumination multitouch table. GoC is a Java gaming
application that takes a simplified version of the Game of Craps and incorporates
multitouch capabilities. The inclusion of multitouch features improves upon other
existing casino game simulations by making the experience more interactive for the
user. Up to three players can participate per game, with each player positioned at a side
of the table. Each player inputs their name and how much money they want to start
with by “typing” on a multitouch keyboard. Players then input their desired bet for each
roll by touching the corresponding multitouch buttons. When the game is started, a
multitouch button allows players to “roll” the dice and have the result visually displayed.
A random number generator is utilized to determine the outcome of each roll. Each
player’s current amount of money is updated and displayed according to the outcome of
the dice roll. The game continues until either the players choose to end it, or until all
players run out of money.
External Interface
6
GoC is intended to be used on a multitouch table that utilizes a tracking tool that
outputs the data over the Table-Top User Interfaces Objects (TUIO) protocol. The user’s
interaction with the table via their hands and fingers is thus translated into events that
trigger actions within the application. These events can include the user inputting
information through a multitouch keyboard, pressing a multitouch button, or
moving/resizing/rotating a component through multitouch gestures.
GoC can also simply be run on a computer by using the mouse clicks to simulate the
touch of a user’s single finger. Multiple touches can be simulated by pressing Ctrl + n to
identify the location of the second finger, and then pressing Shift to simulate the touch
of the second finger.
Performance
There is a slight delay upon the startup of the application while the connection to TUIO
is established to transmit tracking data. The reactivity of the application to the user’s
touch depends on the sensitivity of the cameras utilized in the hardware
implementation. Performance otherwise relies on the attributes of the computer being
used to execute the application.
Attributes
Multitouch Components: This application incorporates several multitouch components
which serve to make a uniquely interactive experience for the user as compared to
other casino game simulations. The first multitouch component presented to the user is
the keyboard utilized to input each player’s information. This keyboard is included in the
Multitouch For Java (MT4j) Software Development Kit (SDK) and can be manipulated
with various multitouch gestures. The keyboard can be dragged to change its location,
rotated to face each player, and resized to best facilitate the “typing” of each player’s
information. The second multitouch aspect of GoC involves the entering of each
player’s bet. When all players have entered their information, the display will change to
show a modified Craps table composed of several multitouch buttons representing
different monetary amounts. In order to enter their bets, each player touches the
buttons corresponding to how much they desire to bet on that roll. The multitouch
aspect of the buttons allows several of them to be pressed at once, giving the user the
choice of entering their bet in multiple ways. For example, a bet of $150 could be
entered by pressing the $100 button and the $50 button in turn, or by pressing both
buttons simultaneously. The final multitouch aspect of this application involves the
components of the display where the dice are rolled and the results are displayed. All
the images and text boxes in this display are can be manipulated with multitouch
gestures, much like the multitouch keyboard. The components’ locations can be
changed by dragging, the components can be resized, and the components can be
rotated to change their orientation.
Betting: GoC utilizes a very simplified betting system as compared to an actual game of
Craps. For each roll, players can choose to bet an amount of money less than or equal to
the amount of money they currently have, or no money at all. If a winning sum is rolled,
all players will win the amount they bet; if a losing sum is rolled, all players will lose the
7
amount they bet. Players have the opportunity to change their bet amounts before each
roll and can increase or decrease their bet in accordance with whether they have been
winning or losing.
Game Logic: The logic used in this application follows the rules of an actual Craps game
in a casino. For each dice roll, the sum of the rolls is computed and then checked against
the rules to determine if the players won, lost, or need to roll again. If a sum of 2, 3, or
12 is rolled on the first roll, the players lose. If a sum of 7 or 11 is rolled on the first roll,
the players win. If another sum is rolled (4, 5, 6, 8, 9, 10), the players will roll again. This
sum becomes equal to the players’ point value. If a sum equal to point value is rolled on
the second roll, the players win. If a sum of 7 is rolled on the second roll, the players
lose. If another sum is rolled, the players will continue rolling until they either get a
sumo f 7 or a sum equal to their point value. After the players roll a winning or losing
sum, they will be allowed to change their bets before the next roll.
Design Constraints
The computer running the application must have the latest version of both the Java and
Processing programming languages. The latest version of the MT4j (SDK) is also required
for GoC to function.
Screenshots
Welcome Screen
8
Inputting Player Information via the MT Keyboard
Inputting Player Bets via MT Buttons
9
Roll Again
Losing Roll
10
Winning Roll
End of Game
11
2.3 Checking-In Website/Application
Functionality
The Checking-In website was designed for families desiring to visit Villanova. The
website allows families to pick the date of their visit and sign up for the various
admissions events offered throughout that day, including three different times for
campus tours and general admissions presentations. The website gathers personal
information, education information, and guest information about the visiting student.
Once the user enters all the required fields, their information is sent to the database, in
which each user has a unique id. The front-end application is designed to be utilized in
the admissions office. When prospective students come to visit, they can check-in via
this application. The application displays the names of all students registered for the
specific event about to occur. The user, the prospective student, selects his or her name
and the application pulls up their information card. The users identify that all their
information is correct and then they click “check-in” to indicate that they attended the
event.
External Interfaces
The software interacts with the users via both the website and the application. Both the
website and the application interact with the database. The website uses a Perl script to
extract the user’s data and places it in a Common Gate Interface (cgi) which interfaces
with the server. The Perl script makes a connection to my database. It then creates a
mysql query to insert all user entered data into the database table, visitingFam. The
application also interfaces with the database. It uses JDBC to create a connection to the
database. It then pulls all the users’ data from the database. It displays first and last
name of users on buttons on the main screen. It then displays all the other user
information on the information card page.
Performance
The website interacts with the database at an optimal speed, seemingly instantly.
However, the application does not interact with the database at optimal speed. There is
about a 10 second delay when starting up the application, because of the drawing
information from the database. The website is universally available; any user with
access to the internet can access it. The application; however, is only available to those
who visit the university and check in at the Villanova admissions office. The recovery
time of the website functions are also optimal. However, again for the application the
time efficiency cost of accessing the database is high. Every access into the database is
very costly; thus, the application is only designed to access the database an hour before
12
each event to pull the users and their respective information that are scheduled to
attend this particular event.
Attributes
The website is portable. It can be accessed and maintained from any computer that has
a connection and access to the server, webster in my case. All the code is stored under
the user name vutour. The application is really not portable; however, it does not really
have a need to be. In order to run and edit the application from a computer, it must
have a java runtime environment and java system development kit respectively
installed. I found it simplest to implement in an eclipse environment, but this is not a
necessary requirement. A version of mysql must exist on the computer. The JBDC
connection driver must be installed and imported into the class path. In addition, mt4j
must also be downloaded and imported into the class path. Finally, the vutour project
would have to be imported into the same class path as the rest.
As a result of its high portability, the website can easily be maintained. Multiple people
can work on it at any time very easily. The application; however, can only be
maintained by computers that set up the environment. Thus, its low portability makes
the maintainability not as easy. The cost of setting-up new computers, makes it harder
for multiple people to work on this application.
There is no implemented security in the web-site or application. Any security this
software has is inherited by the server. Webster has a user name and password for
accessing the database, the website code, and the Perl script. The application itself has
no security outside of its secure access into the database.
Design Constraints
The website is written in HTML with a Perl script to access text field data, as mentioned
above. The application is written entirely in Java. It is written on a Windows system.
13
Screen Shots
Website
Database
14
Application (Main Page)
Application (After Clicking Scudder, Kristin
button)
15
Application (After clicking Check-In button)
16
3 Explanation of Design Options and Choices
3.1 Multitouch Table
There were many choices to be made during table construction. The viewport was chosen to be
28-7/8”x16-5/8” to obtain a roughly 16:9 ratio viewport. The surface has about 1’ extra
plexiglass and plywood around the viewport to create a border that can be used as a traditional
table. This also ensures that in the chance that anything spills on the table top, no internal
components can possibly be affected. The box was built to a height that both is compatible with
either standing or sitting operation as well as the throw distance of the projector in use. The
skeleton of the box was built on the outside of the plywood sheets in order to create as uniform
an internal surface as possible to create optimal infrared illumination when reflected off the
white interior walls of the unit. The exterior of the box was painted black to create contrast
between the projected image and the table itself, as well as to conceal any nicks or marks left on
the box during transport or use. Two cameras were used to capture the input in order to obtain
a somewhat sufficient resolution over the viewport. One camera was not able to detect smaller
objects, and the use of three (or the desired four) cameras is not currently possible with the
open source tracking software we chose to use. PS3 cameras were chosen due to their price tag
($40 each), and their ease of modification to block out most of the visible spectrum.
3.2 Game of Craps
There are multitouch SDKs available for various languages including Java, C and C++. For my
software application, I chose to utilize the SDK for Java as that was the language I was most
experienced in and most comfortable using. The MT4j SDK is built on top of the Processing
programming language which allowed me to access some of Processing’s libraries to create my
applications. I chose to develop using the open source integrated development environment
(IDE) Eclipse as I am well versed in its functions and have found it to be a robust IDE.
My goal in designing the user interface for my application was to keep it simple and intuitive for
the user. I favored a clean interface with basic components instead of creating a complex display
that was cluttered with graphics. I chose a color scheme that closely resembled what would be
found on an actual Craps table to help create the feeling of a casino-like environment for the
user. Instead of attempting to create a graphic version of a full Craps table, I chose to produce a
very modified version that only contained the components I needed for my simplified Craps
simulation. I also chose to use different displays for where the players bet and where the
players roll the dice to simplify the amount of information on the screen.
My design was also limited by the capabilities of the MT4j SDK which was lacking many features
that would have simplified the creation of a graphical user interface. For example, there are no
layout managers to oversee the placement of the components in the display; the location of
17
each component must be set explicitly. Another limitation was the fact that all buttons had to
be images. The only way to have text appear on a button was to overlay a text area on the
button. With a more functional SDK it might have been easier to create a more intricate user
interface.
3.3 Checking-in Website/Application
The client-side of the website was developed in HTML. This language was chosen for several
reasons. First, the web-site was not the primary focus of the project and the design itself was
fairly simplistic. HTML, which I have decent amount of exposure to, was able to accomplish all
the design functionality I needed for the web-site. Perl was used for the server-side of the
website. This language was chosen for similar reasons. The only server-side interactions I
needed were to pull information from text fields and combination boxes. Perl is very simplistic
and easy to learn. It was able to extract the data I needed very simply. It also gave me the
ability to connect to and query the database. If the website was a primary focus of the project, I
would have put more emphasis on making it more than just functional. I would have perhaps
spent more time on the appearance of the website, making it more appealing to the eye.
For the database I used mysql, because I am most familiar with and was able to do everything I
needed it to.
The application was written entirely in Java; because it is the language I am strongest with and
there was software to integrate Java with my database and the Multi-touch table. I used the
Multitouch For Java (MT4j) open source Java development platform. I chose this primarily
because Tom recommended it to both Tara and me. However, during development I saw that
this development platform had many limitations and was not really well-suited for the
application I was trying to develop. Buttons are merely images, so there is no way to create an
arbitrary button with a label on it, like you can in Java. In addition, there is no ability to repaint
the Scene. This made it impossible to change the information card based on the id of the user
selected, even though I had all this information available in the Scene. In addition, there is no
text fields in this platform. Again, making it impossible to edit any of the user information
displayed on the info card Scene.
18
4. Suggestions for System Extensions
4.1 Multitouch Table
The extension of this system would involve the open source community patching the TUIO
protocol output bug in the CCV 1.3a code. Physical extension would involve building a dedicated
computer to be located within the table itself. This would make using the table much easier and
faster. Construction could be improved by finding some way to keep the molding fastened to
the edge of the plexiglass surface (superglue does not seem to be doing the trick at the
moment). Using a better projector (with either a shorter or slightly longer throw distance [along
with mirrors]) would allow the entire viewport to be utilized. Managing static electricity
between the plexiglass surface and the projection surface would improve table accuracy, as well
as finding a method to reduce ambient light penetration through the diffuser surface would also
increase accuracy during day time and would allow more ambient light in the room while still
maintaining accuracy of IR detection. Adding two more cameras and segmenting the surface
into quadrants rather than halves would greatly increase the resolution of the IR detection,
allowing for smaller input regions as well.
4.2 Game of Craps
There are several ways to extend my software application to make it a more realistic simulation
of the actual Game of Craps and incorporate more of the multitouch capabilities. To make the
application more realistic, I would expand the game capabilities to incorporate the different
types of bets that are available in a casino Craps game. The user interface would also have to be
modified to more closely resemble an actual Craps table and support the expanded betting.
Another extension would be to incorporate graphical poker chips in the application so the
players could visualize how much money they had. Players could also make their bets by moving
the graphical poker chips through multitouch gestures on the table. A final extension would be
to incorporate three-dimensional (3-D) graphical dice objects that the users could “throw” via
multitouch gestures.
In order to incorporate more of the multitouch capabilities with our table, my application would
need to be re-tooled to utilize the Flash protocol as opposed to the TUIO protocol. This is
because in testing my application on the table, we discovered that there was a bug in the open
source TUIO software that prevented the user’s touches from being translated into events that
could interact with my application. The test applications that we ran on our table that utilized
the Flash protocol worked as expected, proving that the problem did not lie in our hardware but
rather the outside software that we were utilizing. Reworking my application to use a different
protocol was an impossible task given the time constraints of this semester, but it could
certainly be accomplished in a future implementation.
4.3 Checking-in Website/Application
19
For the website, it could be extended in a couple ways. First, there should definitely be a serverscript checking that all the required fields are indeed filled out. In addition, it would also be very
useful to provide the users with an id after they have registered, in case they want to change
any of the information they entered.
For the application, there is definitely a lot of future work. To begin, I would completely use a
different development sdk for the multitouch aspect, perhaps even work with flash instead of
Java. Flash seems to be more suited for integration with the multitouch table. Beyond that,
there needs to be a way to randomly generate the buttons. In addition, there needs to be a way
to repaint the Scene in order to update the user information card based on what button is
clicked. In addition, after the user checks in, I would like all the buttons to move up, instead of
just removing the name from the button. I would also like to create a Cancel button on the info
card Scene, in case a user accidentally selects the wrong name. In addition, the application
could also include a button on the main page for users that have not registered online.