Haptic control and feedback in mobile gaming
Tommi Urtti
University of Tampere
Department of Computer Sciences
“Haptics in mobile contexts”-course thesis
December 2008
Table of Contents
Haptic control and feedback in mobile gaming.................................................................... 1
Introduction................................................................................................................................................... 3
Pervasive elements ..................................................................................................................................... 4
The accelerometer ...................................................................................................................................... 5
Multi-touch screens in gaming............................................................................................................... 8
Tactile feedback ........................................................................................................................................... 9
Conclusion ................................................................................................................................................... 10
References ................................................................................................................................................... 11
Keywords: Pervasive gaming, haptics, gaming, accelerometer, multi-touch
Introduction
Gaming is all about immersion: a good game is one that takes you into the game and
doesn’t let you go, the type of game that you play ”just one round more”. Many
successfully designed features of the game are needed to achieve the best possible
immersion. These include graphics, sound, plot, characters and challenge.[5,6,7]
One element that has not been fully integrated until late has been the use haptic
feedback and use of haptics as input. Including haptics has been shown by Ramsamy
et al. [14] to increase immersion considerably. There are other cases in which haptics
has been integrated as only an inconsistent and irrelevant vibration of the game
controller. In these cases, most of them console or PC games, many players find the
addition annoying instead of positively contributing to the experience.
In the past few years many interesting innovations have electrified the usage of
haptics in games. Nintendo Wii was marketed before announcement as a “revolution”
of gaming instead of an “evolution” like its competitors the Xbox360 and the PS3.
Wii’s graphics are not on par with the other consoles, but its motion-sensing haptic
game controller has made it a success story worldwide, outselling [15] both
Microsoft’s and Sony’s consoles.
Apple’s iPhone was introduced in June 2007. The much-hyped smartphone has many
technical capabilities that profile it as a sound mobile gaming platform. These include
a 3,5 inch multi-touch display, 3D-graphics processing unit from PowerVR and an
accelerometer. Game developers have welcomed the accelerometer and multi-touch
display. We introduce these features through example games.
Sony Playstation’s DualShock controller sparked interest in vibrating controllers in
the 1990’s. Afterwards there has been little progress in designing and giving players
haptic feedback in attempts to achieve a more immersive experience. We’ll take a
look at few innovations in this area, particularly in the field of pervasive gaming,
where haptics is expected to play a bigger role than with mainstream gaming.
Pervasive elements
Pervasive elements are designed to bridge real life objects into virtual game
environments. We decided to include this part in the survey, since they introduce
interesting ideas to gaming in general and mobile gaming in particular.
Ullmer and Ishii [17] have in their work presented two levels of haptic feedback. In
tangible interfaces the first one is the representation level. Immediately by picking up
a tangible object like a baseball bat or a real life PDA you get haptic sensations from
the device, what is its form and functions. The other level is the interaction level, this
the haptic feedback given to the player when the physical objects and digital
information interact in pervasive games. For instance when you hit an image of a
tennis ball with an actual tennis racket (or a controller) the racket vibrates.
Martin Faust has accomplished some interesting work with pervasive elements in
games. Doom 3, a first-person shooter game, has the typical method of guiding the
player with in-game emails and a heads-up-display. Faust modified the game [8] in a
way that instead of reading the emails on the on-screen virtual PDA, they were
pushed to an actual PDA device. Haptic feedback of the game was improved on the
representation level, since the player could hold their real PDA in their hand and read
the emails on it, instead of watching an on-screen image of a PDA.
The new version of Airkanoid [8] is another work by Faust. Airkanoid is an old game
from 1985 where the player controls a paddle with which he reflects a ball around the
screen to break a number of bricks. Once the bricks are broken the player reaches the
next level. A new version of Airkanoid in 2006 took the game concept as it was, but
made a few changes. The paddle movement would not be vertically limited as it had
been before (paddle on the bottom of the screen, horizontal movement for the width
of the screen).
The big invention however was to replace the virtual paddle with actual physical
objects called Airbats. The Airbats in the game are optically detected, so they can
essentially be anything, even a rolled up paper. The game is projected on to a screen.
The image has been altered so that the game elements are added on top of the video
stream of the players. When the players position the Airbat so that the ball hits it, the
ball is reflected in the game (see figure 1).
Figure 1: Haptic Airkanoid.
In “Haptic Airkanoid” Faust and Yoo [10] further improved the concept of Airbats.
By adding two vibrating elements into the Airbats, a stronger bond could be created
between the virtual and physical world. When the image of the physical Airbat hits
the image of the ball, the Airbat vibrates like the collision had actually happened. The
participants described the sensation as not very realistic, but all of them agreed that
the game was more fun with the haptic feedback than without.
The accelerometer
The accelerometer (or motion sensor) is not a new invention, but lately it has
appeared more commonly in consumer devices. Hard drives have integrated the
motion sensor to prevent damage from sudden movement. This is especially useful in
laptops. Mobile phones, including the Nokia N95 and Apple iPhone, are equipped
with accelerometers. Other mobile phones equipped with this feature include models
from Samsung, Motorola and Nokia. In terms of software distribution the iPhone has
an advantage in comparison to competitors. Games and other software can be
purchased exclusively from the App Store. App Store has proven a success; Vodafone
and other operators are now developing similar services for other phone models.
iPhone is the main platform for which we surveyed accelerometer games.
iPhone’s accelerometer composes of a silicon mass, a set of silicon springs, and an
electrical current. Motion is sensed by registering fluctuations in the electrical current
when the silicon mass moves. The silicon springs are in place to lead the current
through the mass. Fluctuations are registered and software developers may use the
data as events. [2]
Figure 2: Accelerometer demonstration [Source: Apple.com].
The accelerometer in the iPhone is used by many of the games available on App
Store; one
ne of these games is Labyrinth. The game’s idea is rather simple, to guide a
small ball through a maze with holes for the ball to fall through. Inn the original game
balancing the board was done via 2 wheels that rotate the axis of the planes.
planes This is
type of game that would be practically impossible to implement with good immersion
using the typical keypad for control. Labyrinth
Labyrinth on the iPhone uses the accelerometer
to measure the angle of the phone. When the phone is slanted the ball rolls with fairly
realistic physics in the correct direction with
with speed relative to the angle.
angle
Figure 3: iPhone game “Labyrinth”.
Other available games that use the accelerometer include racing, snowboarding,
snowboarding
fishing and flying (a bi-plane)
plane). Generally the player moves the phone around using it
as they would a steering wheel or a fishing rod. Skeptics have questioned the actual
immersion of accelerometer games believing it to be something that wears off quicker
than a captivating plot or a well-designed
well
learning curve.
Asier Arranz developed software called NiiMe [3] (Nokia2MovingExperience) in
2008. The software is an innovative design that
that enables the use of Nokia N95 and
other Nokia models equipped with the accelerometer to be used as a control
peripheral of a PC. Software installed in the Symbian S60 phone sends location data
(x,y,z-coordinates)
coordinates) via Bluetooth to a nearby PC that has been paired with the phone.
The phone can then be used as a mouse or a game controller. Game controller usage
has been demoed with games that use the phone as a drumstick, steering wheel,
airplane flight control and playing Super Mario by actually moving your feet to run
and jumping to jump. The last one I believe is done by wearing the phone in the
player’s pocket and registering a slight horizontal coordinate change for running and a
clearer vertical change for jumping. In my opinion this represent a clever usage
u
of
technology together, taking the accelerometer of the Nokia phone and using to control
a gaming platform that can run more advanced games.
It is early to say if the accelerometer will become a common feature of smartphones
aiming the gaming market. Nevertheless it gives developers the possibility of
designing a completely different game experience compared to the traditional keypad.
Multi-touch screens in gaming
Multi-touch has been one of the most discussed features of display techniques since
Jefferson Han’s (NYU's Courant Institute of Mathematical Sciences) impressive
demonstrations in the TED (Technology, Entertainment, Design) conference of 2006.
In the following years a similar concept was implemented with different techniques
into the Apple iPhone and Microsoft Surface. Multi-touch will also be supported in
the upcoming Windows 7 operating system, scheduled for launch late 2009 or early
2010.
Multi-touch makes gestures a possible method for input in a user interface. For
example the iPhone has the flick, tap, pinch and stretch gestures [1]. The gestures are
used in an attempt to create more intuitive interfaces with less buttons, scales and
menus. By reducing the number of physical buttons in mobile devices the frame can
fit a larger display. A larger display providing visual feedback has been found to
increase game immersion. Previous generations of touch screens had error situations
when the touch screen was used too quickly. Multi-touch has been designed for
simultaneous touches and thus will register touches with better accuracy being less
prone to errors. Disadvantages of touch screens quickly become obvious when
multiplayer games are played on a single screen. Fingers block the view of the
display.
AirHockey is one of the games that use multi-touch on the iPhone. Two players can
play the game either on one phone or over the network. AirHockey is a game where
you control a circular tool with which you hit the puck trying to score a goal in the
other player’s end. The game is fast-paced and requires constant attention. As a result
typically you will have both players controlling their sticks all the time. This game
could not have been implemented on a touch screen without the multi-touch
capabilities of the display. A normal touch screen would only register the one player’s
movements.
A different type of example usage of multi-touch is Simcity, a classic city
construction simulation game on the PC. Simcity on the iPhone uses the pinch and
stretch gestures to zoom in and out. Previously it has been published for the Nintendo
DS and the Windows Mobile platform. Once this game is published it could be well
be used for comparing gaming with gestures or a stylus on mobile platforms.
Tactile feedback
Typically haptics are used in games for feedback instead of an input. Most mobile
phones have the vibration function that is used in games for simple feedback. Problem
with the typical vibration function of a mobile phone or a game controller is that it is
too ambiguous: it doesn’t give specific information to the player of what has
happened. In attempts to significantly improve the usefulness of tactile feedback
innovative engineering is needed. Well designed haptic feedback was proven by Jiang
et al to reduce error rates and thus improve performance [11].
TN Games was founded by a medical surgeon who invented a new method of using
old aircompression techniques. The TN Games 3rd Space Vest has 8 pockets that
inflate rapidly creating the sensation of a tap. Depending on a game this will be
getting shot or hit. Giving the player a hint where the attacks are coming from enables
a quick response and a more immersive experience [16].
Figure 4: TN Games 3rd Space Vest.
There are many different ways of giving haptic feedback to the player. Techniques
include usage of ultrasound or deep bass sounds. If a company wants to concentrate
on making a product as mobile as possible sound and air inflation devices are heavyset. Also in order to achieve the most mobile experience, cables must get rid of. 3rd
Space Vest has a USB cable running to the PC and an air tube going to the external
compressor, which has an AC cord. With this setup moving is going to be limited and
the user has to weigh the value of the haptic feedback. Scott Lowe, of IGN Gear, was
one of many that was impressed by the TN Games device, but would not wear it
regularly. [13]
Conclusion
In this paper we have given an overview of some of the recent developments in the
field of enhancing gaming through the use of haptics as either an input or output. As
future work it would be interesting to do user studies comparing using haptics as input
against keypads or joystics.
References
1. Apple Inc., iPhone multitouch specification sheets,
http://www.apple.com/iphone/features/multitouch.html, (1.12.2008)
2. Apple Inc., iPhone accelerometer specification sheets,
http://www.apple.com/iphone/features/accelerometer.html, (1.12.2008)
3. ARRANZ, A. http://www.niime.com/ (13.12.2008)
4. BROWN, E. and CAIRNS, P., 2004. A grounded investigation of game
immersion, CHI’04: CHI '04 extended abstracts on Human factors in
computing systems, 2004, ACM pp. 1297-1300.
5. CHANG, D. "Haptics: Gaming's New Sensation," Computer, vol. 35, no. 8,
pp. 84-86, Aug., 2002
6. CHENG, K. and CAIRNS, P.A., 2005. Behaviour, realism and immersion in
games, CHI'05: CHI '05 extended abstracts on Human factors in computing
systems, 2005, ACM pp. 1272-1275.
7. ERMI, L. and MÄYRÄ, F., 2005. Fundamental components of the gameplay
experience: Analysing immersion. Proceedings of DiGRA 2005 Conference:
Changing Views - Worlds in Play 2005.
8. FAUST M., Airkanoid, http://www.e56.de/Airkanoid.html, 2006 (1.12.2008)
9. FAUST M.: “Doom 3 PDA Mod”, http://www.e56.de/PDA.php 2005.
(1.12.2008)
10. FAUST M., YOO Y.-H.: Haptic Feedback in Pervasive Games. In
Proceedings of the Third International Workshop on Pervasive Gaming
Applications - PerGames 2006
11. JIANG, L. et al.: Reducing error rates with lowcost haptic feedback in virtual
reality-based training applications. In: Proceedings world HAPTICS, Pisa,
Italy (2005) 420–425603-609.
12. KIM, S.-Y.: ”Interactive Racing Game with Graphic and Haptic Feedback”,
p.69-77, LNCS Volume 4813/2007
13. LOWE, S. 3rd Space Gaming Vest Review,
http://gear.ign.com/articles/931/931587p1.html (14.12.2008)
14. NEXGENWARS, http://nexgenwars.com/ (1.12.2008)
15. RAMSAMY, P. et al: Using Haptics to Improve Immersion in Virtual
Environments. In Computational Science – ICCS 2006, Springer 2006, pp.
16. TN GAMES www.tngames.com/ (13.12.2008)
17. ULLMER, B., ISHII, H.: Emerging frameworks for tangible user interfaces. In
Caroll, J.M., ed.: Human-Computer Interaction in the New Millenium.
Addison-Wesley (2001) 579–601
2
18. WANG, Q. et al: “A Haptic Memory Game using the STRESS Tactile
Display” CHI 2006, April 22–27, 2006