An Intelligent Game Engine for the At-Home
Rehabilitation of Stroke Patients
1
N. Alberto Borghese
1
Renato Mainetti
1
2
12
Michele Pirovano , Pier Luca Lanzi
Applied Intelligent Systems Laboratory - Department of Computer Science - University of Milano, Milano - Italy
2
Politecnico di Milano - Dipartimento di Elettronica, Informazione e Bioingegneria, Milano, Italy
{alberto.borghese, renato.mainetti, michele.pirovano}@unimi.it {pirovano,lanzi}@elet.polimi.it
Abstract-The recent availability of advanced video game
interfaces (such as the Microsoft Kinect, the Nintendo WiiMote
and Balance Board)
is creating interesting opportunities to
provide low-cost rehabilitation at-home for patients. In this
context, video games are rising as promising tools to guide
patients through their recovery experience and to increase their
motivation throughout the rehabilitation path. However, to be
applied to clinical scenarios, video games must be designed to
adhere to the clinical requirements and to meet doctors/patients
expectations. They also need to be integrated within multi-level
platforms that can allow different levels of monitoring, e.g., at
a personal level by the therapist, at the hospital level by the
doctors, and at the regional level by the government agencies.
In this paper, we overview an intelligent game engine for the
at-home rehabilitation of stroke patients The engine provides
several games that implement actual rehabilitation exercises and
have been developed in strict collaboration with therapists. It
is integrated in a patient station that provides several types of
monitoring and feedback using virtual and/or human therapists.
I.
INTRODUCTION
Stroke is a major cause of adult disability in developed
countries and it totals about 16 million new first stroke events
per year [1]. Given such increasing figures, the cost of stroke
rehabilitation is expected to saturate National Health Service
Providers which are expected to be forced to shorten the
duration of the rehabilitation support. However, exercising
should be continued also outside the hospital to avoid losing
the benefits of hospital rehabilitation and to stabilize psycho­
physical conditions. Moreover, accumulating evidence sug­
gests that intensive rehabilitation allows recovering function
also months after the stroke event [2]. This requires daily
rehabilitation sessions that presently have to be carried out
in specialized centers, with the support of therapists. Unfortu­
nately, only few patients can afford such an option as it is often
not supported by public health systems. This has an enormous
socio-economic impact also on the patients families who often
feel left alone by the health service providers [3] and patients
who should continue the therapy outside the hospital actually
drop out mostly due to high costs. But even when the costs
were covered, many patients lack the necessary motivation to
continue training for their recovery.
978-1-4673-6165-1/13/$3l.00
©
2013
IEEE
In fact, recent studies show that motivation is easier to be
stimulated in a comfortable environment and often hospitals
are not perceived as such so that patients may be better
motivated at home.
Virtual Reality (VR) has been explored as a viable tool to
support rehabilitation either alone [4], [5] or in combination
with robotics [6], as it was shown the potentiality of rich
graphics environments in capturing patients attention and mo­
tivating them [7]. However, classical VR devices have turned
out to be expensive, intrusive, and unsuitable for a deployment
at home. Starting from Nintendo Wii I, a shift of focus has been
recently observed in the gaming industry, in which Natural
User Interfaces (NUl) for home consoles, have been gaining
much interest. This has also spurred the development of new
low-cost interfaces: devices like Microsoft Kinect2 make it
possible integrating highly reliable patient tracking into a low­
cost platform that can be deployed at home.
In this paper, we show how specifically designing games
and using NUl devices, a platform that can be used to guide
rehabilitation robustly can be built. Such a platform can be
used for rehabilitation at home only if it is integrated inside
a multi-level platform that allows remote supervision by the
hospital clinicians. These ideas have been developed inside
the Fitrehab project3 and are being currently further pursued
inside the Rewire project4 financed under the FP7 framework,
both coordinated by University of Milano.
II.
RELATED WORK
The use of leT technology to help elder people in their
daily life has been explored in many research projects and
application areas. The Oldes projectS offered new technolog­
ical solutions to improve the quality of life of elder people
through low-cost and easy to use entertainment devices and
health care platforms. The Neuroweb project6 tried to improve
healthcare delivery through vertical integration of existing
clinical and genetic databases using an online web platform.
The BrainAble projecr7 conceived, designed, implemented and
1 http://www.nintendo.com
2 http://www.microsoft.com/en-us/kinectforwindowsl
3 http://www.innovation4welfare.eu/2S7/subprojects/fitrehab.htm!
4 http://www.rewire-project.eu/
Shttp://www.oldes.eu/
6http://nuke.neurowebkc.eu!
7 http://www.brainable.org!
validated a Brain Computer Interface combined with affective
computing and virtual environments. The Caalyx project8
supports the elder in her indoor and outdoor life. It was based
on continuous monitoring and measuring specific vital signs
and on the communication with a Central Care Service in
case of emergency. A similar goal is shared by the Dreaming
projecr9 that employed a standard TV set as simplified remote
control, services for teleconferencing, cellular phones for
outdoor localization, alarm messaging and fall location.
Several researchers developed rehabilitation approaches
based on robotics platforms. The Companionable projectlO
proposed an assistive environment at home, where a mobile
robotic companion interacts with a smart home sensors net­
work. In the Smiling project11, special shoes with dynamic
control of equilibrium were developed to improve patients'
mobility and counteract falls. The Domeo project12 proposed a
robotic assistant to the elder, with surveillance and interaction
capabilities to stimulate mobility and rehabilitation. Similar
goals have been pursued in the project Farseeing, based
on using inertial sensors worn by the patient. The Humour
project13 explored the use of efficient robotic strategies to
facilitate the acquisition of motor skills and to rehabilitate the
upper limb.
Companies have also approached the rehabilitation market
through robotics. MIT Manus14 is a weight counterbalanced
robotic arm for the neurorehabilitation of the upper extremi­
ties. A similar approach has been pursued by HocomalS with
the Armeo system. Hocoma has also developed a driven gait
orthosis with a body weight support system, named Lokomat,
that assist rehabilitation of neurological patients walking on a
treadmill. Hand rehabilitation has received a lot of attention
along the years. Besides the Script project, funded by the
European Community, several conunercial systems for hand
rehabilitation have been developed such as the Pablo System16
and the rather inexpensive Gloreha glove17.
All these systems require that the subject is tied to a robot,
which makes the situation unnatural and undesirable. More­
over, because robotic technology is expensive, complicated to
use and requires close maintenance, its use at home is not
feasible at least in the foreseeable future.
A different approach is based on the development of
videogames and gaming platforms to guide rehabilitation.
For instance, a cheap platform based on a web camera, a
projector or TV screen, and a PC has been proposed for
neglect rehabilitation [8]. Its main strengths are (i) a careful
design of the rehabilitation games and their progression, and
(ii) a well-balanced set of targets and cues, which can be
8http://caalyx.eu
9http://www.dreaming-project.org/
10http://www.companionable.neU
Ilhttp://www.smilingproject.eu/index.html
12http://www.aal-europe.eulcalls/funded-projects-caU-l/domeo
13http://www.humourproject.eu/
14 http://web.miLeduinewsoffice/2010/stroke-therapy0419.htm]
15 http://www.hocoma.com/en/products/lokomaU
16http://www.tyromotion.com
17 www.gloreha.com/
specified using scripting language. Results from a first pilot
on a chronic neglect patient has provided excellent results.
Another low-cost framework designed on a similar principle,
but tailored to postural deficits, was proposed in [9]. It is based
on catadioptric velcro strips attached to the patient feet and
2 cameras. Targets on the floor are shown to patients that,
disappear when walked over by them.
The Rehabilitation Gaming System (RGS) project18 focused
on the development of videogames for the rehabilitation of
motor deficits of the upper extremities; game design, game
adaptation and usability were all evaluated in strict collab­
oration with clinicians [10]. The Fitrehab project pursued a
similar approach but focused on closing the rehabilitation loop
to the hospital. A hospital station was employed to prescribe
the rehabilitation exercises in the form of videogames which
could be downloaded and used inside a patient station, de­
ployed at patient's home. The station collected data of the
patient's interaction with the games and transmitted them to
the hospital so that they could be analyzed by the clinicians.
Gaming approaches to rehabilitation have also been ex­
plored by companies. The Silverfit platform19 provides a
set of exercises of varying difficulty and it is based on a
single 3D range camera. The Gesturetek IREX20 represents
the state of the art for non-obtrusive motion tracking for
rehabilitation. The system is based on a portable Green Screen
or a wall which can be painted with Digi Comp Paint, an LCD
ceiling-mounted projector, a 32" or 40" flat-screen television
monitor, and a green mat or Digi Comp painted floor surface.
More recently, the Nirvana system21 tried to provide total
body rehabilitation using only a webcam and a large screen
projector. The system can be arranged to project scenarios
either on the floor or on a frontal wall while the subject can
interact either with his feet or with his hands. However, these
systems are expensive and difficult to customize to different
pathologies.
III. THE REWIRE
PROJECT
The approach pursued in Fitrehab is being further de­
veloped inside the Rewire project that aims at developing
and field testing a platform for rehabilitation at home that
allows patients, discharged from the hospital, to continue
intensive rehabilitation at home under remote monitoring by
the hospital. The underlying idea is to assemble off-the-shelf
components to build a robust and reliable system that can be
massively deployed at the patients homes. The platform is
constituted of three hierarchical components (see Figure 1):
(i) a patient station (PS), installed at home; (ii) a hospital
station (HS); and (iii) a networking station (NS) at the health
provider site.
The patient station (PS) is used by the patient and her
caregivers at home to perform rehabilitation exercises. The
patient follows her exercise plan through a series of simple
18http://rgs-project.eul
19http://www.silverfit.nl
20http://www.gesturetekheal th/prod ucts-rehab-irex.php
21http://www.btsbioengineering.com/
Networking station
Hospital stations
Patient stations
Figure 1. The rehabilitation platform envisaged in Rewire: the patient station
(PS); the remote hospital station (HS); and the networking station (NS).
yet compelling and engaging games. The TV screen displays
either the patient herself or an avatar impersonating the patient
moving and interacting in real-time with the game's virtual
environment. The patient's movement is tracked in real-time
through inexpensive NUl devices (e.g., Microsoft Kinect or
Sony PlayStation Eye). A wide variety of game scenarios, a
balanced scoring system, quantitative and qualitative exercise
evaluation, gameplay level adaptation to patient's status, and
audio-visual feed-back are all aimed at maximizing patient's
motivation. The PS monitors also that the patient, during the
game, does not assume wrong postures or executes movements
in a wrong way.
Patient's indoor and outdoor activity is monitored through
a set of inertial sensors worn by the patient, that constitute a
Body Sensor Network (BSN). This is profiled to obtain infor­
mation about the patient lifestyle [11], [12]. that can be used,
along with environment parameters, both to tune rehabilitation
and to assess potential risks. In particular, the possibility to
assess patient's improvement, not only on the specific exercises
prescribed, but also in the activity of every day life, is of
particular value. Specific scales to assess improvement, like
the ADL index [13], [14], have been proposed to this purpose.
The hospital station (HS) serves two main purposes. First,
it is used as a therapy management tool that allows defining
and monitoring the rehabilitation treatment. The station helps
doctors and therapists to define, tune, and monitor the planned
rehabilitation exercises, taking into account all the activity
data collected by the patient station during the exercises and
by the BSN during patient daily activity. The exercises are
mapped onto adequate games that are configured according to
patient status and rehabilitation goals. The hospital station is
conceived as a Web application to allow maximum flexibility
and responsivity. Secondly, the hospital station manages the
virtual community of patients, caregivers and clinicians in­
volved in the rehabilitation process. and can be used to gather
information on the therapy and to communicate with all the
subjects involved in the rehabilitation process. Patients can
share their experiences and information with other patients
and with therapists so as to create a strong and supporting
community, and decreasing the risk of isolation hidden in at
home rehabilitation.
Multi-player gaming support could also be envisaged in the
future to allow patients to perform rehabilitation in groups
connected through the Internet. Exercising together may turn
out motivating to the patients and it is being explored in
elder communities in North Brabante22 . Through the virtual
community, patients will be able to confront their results on
rehabilitation with other patients, increasing their motivation
to exercise through (moderate) competition.
Community is also expected to strengthen patient to doctor
communication, allowing them to gain more knowledge about
their pathology, to communicate with the clinicians to discuss
their rehabilitation progression, and even ask for support. This
would make the patient and her caregivers more responsible
and involved in the rehabilitation process [15].
The network station (NS) is installed at a regional level
at the health provider site and it is used to analyze all the
data incoming from the individual patients in their day by
day rehabilitation and to compare and interpret the results on
different patient populations. Data are highly heterogeneous
and data mining procedures are applied to discover features
and trends in rehabilitation treatments [16].
IV. THE
PATIENT STATION
The patient station (PS) is used by patients and their
caregivers at home to perform supervised rehabilitation. For
this reason, its design is based on two main principles:
friendliness and efficacy. Friendliness arises from the need
for the patient and its caregivers to be at ease in a familiar
environment. This is supported by the development of simple
and engaging interfaces, and entertaining games. It is also
supported by social aspects, both on-line with discussions
with therapists and other patients and off-line through the
involvement of other family members and caregivers [8].
Efficacy regards the rehabilitation goals. Beneath their game
coat, the exercises that the patient is required to perform follow
a rehabilitation program designed by professional therapists.
Exercises are chosen by the therapist at the hospital according
to the patient status, functional impairment, and progress as
monitored across different sessions. Upon evaluation of these
data, the clinicians can choose to upgrade the rehabilitation
to a more difficult level either by choosing a different set of
games or by modifying the settings of the same games.
A. Structure of the Patient Station
The patient station is structured as an application composed
of four integrated modules: the community, the lifestyle, the
hospital communication and the Intelligent Game Engine for
Rehabilitation (IGER) module. The four modules can be
22http://www.brabant.nIlsubsites/english/portfolio/welfare-and-healthl
smart-care.aspx
accessed by the patient directly through the main menu of the
application. We have adopted an interface based on a hands­
free paradigm based on gesture and voice recognition.
The main menu gives access to the web client of the virtual
community whose server is resident on the hospital station and
it guides on the collection of the environment and lifestyle
data. These data are usually collected before a rehabilitation
session and sent to the hospital along with rehabilitation data
after the session. Moreover, the Patient Station main menu
allows patients to call the therapist when assistance is needed
or to ask questions. Alternatively, the therapist can schedule an
appointment and meet on-line with the patient when needed.
The conununication can be performed through a bi-directional
video chat, allowing the therapist to watch the patient also
during the execution of the rehabilitation exercises. This also
increases the patient's motivation, as the presence of the
therapist can be beneficial thanks to the human factor.
The Intelligent Gaming Engine for Rehabilitation (lGER)
module is the main core of the patient station [17]; its role
is to introduce the patients to the games prompt them to
play, thus performing rehabilitation exercises. The module
comprises three main modules: the game engine, the input
module and the adaptation controller. The patient station and
its modules are shown in figure 2. In the prototype of the
IGER module currently available the lower-level functions
(e.g., input acquisition, input processing, and the time-critical
sections) are implemented in C++, while the actual game
engine is based on the open-source Panda3D framework23
and it has been developed using Python. One of the main
requirements is to keep the cost of the final framework as
low as possible. Accordingly, we have been focusing on free
and open-source software. Due to this, all the meshes and
animations were created using Blender24 , the state-of-the-art
in open-source modelling software, all the textures are being
created using GIMp25 and the music is going to be created
using the Linux MultiMedia Studi026 (LMMS) audio synthesis
software.
B
I II H���;t�l
c,".",
Commu
";�
Figure 2.
I
Adaptation
I
11
Game
engine
The patient station and its modules
The Game Engine. The game engine is the core of the
IGER module and it features all what is needed to run
rehabilitation games: the rendering of 3D and 2D elements, the
avatar animation, collision detection, control and game logic,
23 http://www.panda3d.org/
2 4 http://www.blender.org!
25 http://www.gimp.org/
2 6http://lmms.sourceforge.netl
sound and music playback, feedback and scoring. The engine
also include advanced features, specific to rehabilitation [7],
[10]. For instance, all the game elements are parametrized
to provide an always-changing, never-boring and balanced
game experience. Rules used to move the game elements and
the game pace are also parametrized and can be changed
dynamically.
The Input Module. The game engine conununicates with all
the supported input devices (e.g., Microsoft Kinect, the Nin­
tendo Balance Board, etc.) through the Input Abstraction Layer
(lAL) that allows the same game to be played with different
combinations of devices. Thus, the therapist has the complete
freedom to customize the game interaction mechanisms and
to select the proper device to maximize the rehabilitation
efficacy. In fact, different patients can play the same game
with their feet, hands or whole body according to rehabilitation
requirements.
The Adaptation Module can automatically modify the param­
eters associated to a game, thanks to a smart monitoring of
the patient performance, by swiftly lowering or increasing the
difficulty of the game. For instance, in the Fruit Catcher game
(Section VII), the fruits can fall faster, more laterally, or more
frequently based on the patient's performance; fruits can also
fall from random positions or from a predetermined sequence
of positions, depending on the actual degree of patient postural
control. Moreover, the fruit size can be adapted to the patient
visual acuity: if the patient suffers from hemiparesis, the
adaptation module may modify the distance of the targets on
the patient impaired side to make it easier for her to reach
them.
The module combines real-time monitoring with real-time
adaptation; it also implements two components of artificial
intelligence that we can classify into an implicit or explicit
intelligence. Implicit intelligence aims at increasing the en­
joyment in playing the game and set a game difficulty level
adequate for appropriately and challenging the patient and
thus create a flow-like experience [18], [19]. Accordingly,
implicit intelligence provides real-time adaptation of the game
parameters, defined by the therapist, depending on the real­
time monitoring of patient success rate, its movement speed,
etc. Such adaptation is implemented using heuristics as sug­
gested in [20] but we plan to explore Fuzzy systems and
Bayesian predictors too [21]. Explicit intelligence incorporates
in a declarative form all the a-priori information available on
the correct execution of the exercises. For instance, explicit
intelligence should monitor that the trunk does not tilt when
moving a foot forward or laterally, or that the shoulder is not
moving forward instead of extending the arm when reaching
for an object. Such module will issue a warning to the
player or, in extreme cases, even aborting the task when unfit
exercising is detected.
V.
LOW-COST INPUT HA RDWA RE
Our framework is very flexible in that it integrates a wide
variety of devices and supports visual, audio, pressure, and
haptic interfaces. In fact, the patient station supports the
Sony PlayStation Eye camera27, the Microsoft Kinect camera,
the Wii Balance Board, and two haptic devices: the Omni
Phantom28 and the Novint Falcon29• All these input devices are
interfaced with the game engine through the Input Abstraction
Layer described hereabove.
The Sony PlayStation Eye is a high performance RGB
camera. An efficient background subtraction algorithm and a
reliable identification of the hands position has been developed
[8] and adopted to extract from the video stream a robust
colored silhouette of the patient. The camera is used as an
input device in augmented reality games, projecting the mirror
image of the patient into the game environment. The patient
can use her upper body to interact with the game objects.
The Wii Balance Board is a pressure platform equipped
with four pressure sensors at its corners that allows locating
the projection of the center of mass of the body of the player.
It has already been used as an input device for rehabilitation
of posture and balance [22]. The board is integrated in our
framework through the open-source WiiYourself library 3o.
The Omni Phantom and the Novint Falcon are three­
dimensional haptic devices. They present an end effector that
can be moved by the user and used as a three-dimensional
pointer. The devices provide fully configurable and pro­
grammable dynamic force feedback, providing us with a mean
to create several kinds of force perturbations or aids to the
patient.
The Microsoft Kinect includes in a single device a 640x480
RGB camera, a 320x240 depth camera and a four microphones
array. Contrary to most commercially available depth cameras,
the Kinect has a low cost as it was developed for the gaming
market. Using the Software Development Kit (SDK) provided
by Microsoft3l , we can find the position of the patient in the
room in real-time and obtain a representation of her skeleton
as a set of 20 ordered points, with no need for an initial
calibration phase. From these points, the 3D orientation of
the segments representing the bones of the patient can be
estimated. These data can be used also to monitor and assess
the movement and the posture of the patient.
A. 3D Skeletal Animation with Kinect
The orientations estimated with the Kinect camera can be
mapped to an avatar that can therefore be moved in real-time
replicating the patient's movements, as seen in figure 3a.
However, this mapping is not trivial. We first compute the
3D position of the 20 points identified on the body using the
Microsoft Kinect SDK API, we then link these points in pairs
according to their hierarchy inside the skeleton, creating a
skeleton with 9
1 bones. Each bone i has a starting point P�
and an end point Ph' For each of these bones, we define a joint
2 7 http://us.playstation.comlps3/accessories/playstation-eye-camera-ps3.
htm!
28http://www.sensable.comihaptic-phantom-omni.htm
29http://www.novint.comlindex.php/novintfalcon
30http://wiiyourself.gl.tter.orgl
31http://www.xbox.com/en-US/kinect
V,
(a) RGE stream, 3D points and
final mesh
Figure 3.
(b) The local quatemion
Animation with the Kinect
Ji positioned at point P� and oriented towards point P�, and a
vector V"
P� - Ph' Joints are arranged in kinematics chains
according to their hierarchy, with joint Ji being connected
to joint Ji-l We defined a reference system solid with joint
.
i-I, Si-l that has the Y axis directed as the link Vi-I
.
We estimate the local orientation of the link Vi with respect
=
to Si-l , as a quaternion qLl We thus compute the rotation
'
quaternion from vector V;�::-} to vector v;�l' which is obtained
by projecting Vi on the coordinate system of Ji-l (cf. Fig.
3b). This computation is simplified by the fact that vector
1 ]
0 ,
0 . Since we cannot infer
V;�::-} is always the up vector [,
the rotation of a link around its axis using only its extremes,
we make sure that the computed quaternion is the shortest­
route quaternion, thus implying that the link has zero roll. We
then normalize the quaternion.
vi-1
i-I
"
q
1 0]
0 ,
[,
i-1 Vi Vi-1
i]
[Vi-I
' i-I' i-I X Vi-I
0 - V;�l]
[V;�IY' V;�lz' ,
x
+
1
"
q
q'/ Iq'l
q'
qi-l
(1)
Starting from the root joint and moving towards the end­
points we compute the orientation of all the segments.
In particular, for each bony segment, we update the refer­
ence system associated to joint Ji, Si, to use with its children
joint. To this aim, we apply the quaternion rotations backwards
through the joint chain, as seen in equation 2.
Note that the orientation of the hip joint, selected as being
the root joint, is set to be the world's up.
Ri
C
=
i i
i-I
Ri'
°
qi-l . qi-2 .... . qc
(2)
We have developed a C++ library for the estimation of the
orientations of the joints. In order to do vector and quaternion
math, we have also implemented our own functions.
During the creation of this library, we encountered a few
problems. The first problem laid in the different coordinate
systems of the 3D applications we have integrated with the
Kinect SDK functions: Blender and Panda3D. Blender and
Panda3D share the same coordinate system, with a right­
handed, Z-up reference. Kinect SDK, on the other hand, works
with a right-handed, Y-up reference system. For this reason,
we had to make sure that the positions in the skeleton space
of Kinect would be correctly transferred to Panda3D and
therefore to Blender. We thus included a transformation matrix
inside our code, reported in equation 3.
1
0
0
0
0
1
o
-1
0
(3)
We also had to solve issues with the exporter of Blender
for Panda3D, which creates .egg files, readable by the game
engine, from blender meshes. Exploring the need to procedu­
rally animate the joints of the rigged character, we realized
that the root of the exported armature, when exposed through
Panda3D built-in functions, erroneously rotated its reference
system 90 around the positive X axis, introducing a 90-degrees
pitch, basically switching to a Y-up system. We solved this
problem by adding a -90 pitch to the armature in Blender and
by restoring the correct pitch in the Panda3D scene.
The orientation data we gather is used as input for the
animation of a three-dimensional avatar inside our games.
The avatar is built with an armature that mimics the skeleton
obtained from the Microsoft Kinect SDK. In particular, it must
be noted that the spine joint is positioned behind the hip joint
and that the left and right shoulder joints do not form a classic
T-pose with the mid shoulder joint. For these reason, using
the orientation data with a skinned character not specifically
designed for Kinect would result in wrong animations. The
hierarchy is maintained inside the skeleton, so that we can
directly assign the quaternions returned by our C++ library to
the bones of the avatar.
Using orientations instead of positions, we gain three im­
portant benefits. First, the use of joint orientations for 3D
animation of bodies is the standard in the industry, thus
allowing us to reuse existing meshes, by making sure to fix
their skeletons beforehand, for real-time animation. Second,
the positional data is subject to noise, resulting in the bones not
having a constant length. By directly applying the positional
data to meshes, we would see limbs getting longer or shorter,
which would be utterly unrealistic. Rotational data is still
subject to noise, but the result is just a random twitch of a
muscle, resulting in a more consistent animation. Third, by
using rotational data for the movement of the animated avatars,
we make sure that the anatomy of the 3D avatar does not
change between patients, thus removing the need for adapting
distances in the game world to the patient.
VI.
DESIGNING GAMES FOR REHABILITATION
Commercial games currently available on the market are
not suitable for patients that are typically impaired and in
the 60-70 age range [23]: their fast interaction pace and
the wealth of targets and distractors make usability low and
may produce strain and anxiety. Therefore, games specifically
targeted to rehabilitation have been developed (e.g., [6], [7],
[10]). Unfortunately, most rehabilitation games developed so
far are focused on the clinical constraints and did not take
into account the well-known design principles used in most
commercial games. As a result, rehabilitation games generally
lack the appeal of commercial games because of their limited
graphics and gameplay. In contrast, we believe that both
clinical and design aspects should be considered to deliver
compelling and engaging rehabilitation games which, ideally,
people might want to play even if they do not have to. This
might increase the emotional connection between the patients
and their family members.
In this work, we aim at increasing patients' involvement
by developing games that follow the best practices of game
design, provide attractive graphics, and can create an engaging
rehabilitative experience, hiding the burden of therapeutical
repetitive tasks under the hood of compelling fantasy. Our goal
is twofold. On the one hand, we want to create games that are
effective for therapeutic purposes and result in successful reha­
bilitation. On the other hand, we want games to be compelling,
since patients must typically play them on a regular basis. We
address our first goal through a tight collaboration with on­
field caretakers and by collecting as much feedback as possible
from patients and therapists. We address our second goal
by leveraging state-of-the-art game design theory (e.g., [24],
[25]), which takes into account several principles including
formal and dramatic elements, the interest curve, flow theory
and the sense of presence.
A. Critical Design Aspects
When designing games to implement rehabilitation exer­
cises, there are few but critical factors that must be taken
into account. First, the exercises are repetitive and therefore
they can easily become boring. Accordingly, the corresponding
rehabilitation games need to introduce elements variability, of
fun, surprise, and suitable challenges so to make repetitive
tasks bearable, almost engaging. For instance, an exercise
requiring patients to repeatedly move their hand, can become a
game in which the same patients must catch afish in the water,
in different randomized positions, a fish that changes its aspect
each time, thus changing the patient perspective significantly.
Second, rehabilitation often requires a predefined training
pace (e.g., the trial duration and the time between trials). This
introduces a critical constraint for the game designer who must
set the timing of the game adequately while still making it
engaging, for instance, by reducing boring segments of the
game, while extending its exciting moments. Our approach
aims at reaching the best equilibrium between therapeutic time
and gaming time through the analysis of patient explicit and
implicit feedback.
Third, the patients, our players, have often serious limita­
tions on their motor and/or cognitive functions. Accordingly,
games must be designed to adapt to the patients specific
capabilities so as to provide the right amount of challenge.
Finally, during their rehabilitation, patients are prone to
depression and thus less willing to play games. Accordingly,
designers must do their best to build games that are capable of
motivating players even in a worst case scenario. Our approach
is to focus on a scoring system designed to avoid frustration,
implemented using a slow-growth structure [7], [10], coupled
with a social component that allows patients to share their
achievements with other patients so as to provide a sort of
community-based motivational support.
High
Flow
channel
B. Game Design Principles and Rehabilitation
Our framework has been developed with the established
games design principles in mind [7], [25] While most rehabili­
tation games are conceived as a series of completely unrelated
minigames, in our approach games are developed around a
resonant theme shared by all the games, creating an unifying
experience. In particular, the set of minigames shown here
is built around a farming theme due to its peaceful and calm
atmosphere and its connection with most people. Accordingly,
all the games take place in the same world, a farm, with
the same creatures, people, animals and objects. Rewards,
obtained through the completion of rehabilitation exercises,
give to the patients the ability to add buildings to the farm, to
start tilling flower to create an awesome garden.
When designing all the games, we considered the principle
of meaningful play. When meaningful play is present, every
game action has a direct and clear feedback as well as a lasting
reasonable effect. This helps the patient in understanding what
she can and cannot do and distractions are minimized. We also
considered elegance, which makes games immediate yet never
boring. Basically, to keep the rehabilitation games simple but
still fun and long-lasting, we designed each minigame to have
few very clear rules to follow and few allowed actions that
however spawn many different kinds of interactions with the
virtual environment.
A last important aspect of the design is the patients' view
of the game environment. In this work, we considered and
implemented two representations of the patient's interaction
with the game: one in which the patient has a first person
view (similar to first person shooter games) and one in which
the patient sees herself depicted using a 3D avatar (similar to
Nintendo's Mii). We leave to the clinicians the option to switch
views according to the current exercise and to the goal of the
rehabilitation exercise. Here, we adopted a first person view
in the first game and a third person view in the second one
with the possibility to switch among different camera angles.
C. Flow Theory
Some of the elements previously discussed, if not all,
concur to create the flow experience [18], [19]. The theory
of flow establishes a relationship between the challenge and
the patient's skill level. According to [18], when the skills of
the user are matched by the level of challenge posed by the
game, the user enters a state of complete focus and immersion
in which it loses track of time. The relationship is reported
in Figure 4, where a beneficial trajectory is suggested [25]
that alternates moments of difficulty and moments of easiness
of play. The benefit of the flow state is that the patient is
completely focused on the game and everything else vanishes,
hiding the rehabilitation burden as well as the difficulties
arising from possible impairment. Studies show that physical
pain is also reduced when flow state is reached [26].
Boredom
Skills
Figure 4.
VII.
High
The flow
REHABILITATION MINIGAMES
We have developed two simple rehabilitation videogames
(or minigames) to test the REWIRE platform's features,
namely, Animal Feeder and Fruit Catcher. The games are
aimed at posture and balance rehabilitation.
A. Animal Feeder
The Animal Feeder minigame aims at training patients on
dual tasks. In Animal Feeder, the player must feed three
hungry cows that keep requesting food by mooing while
opening their mouth. The player kneels before the display and
controls a virtual hand in first-person mode. The player must
first collect some hay and then feed it to a hungry cow, and,
when it is fed, the player score increases. If a cow remains
hungry for too long, it groans and the player score decreases.
In addition, a pitchfork positioned to the left of the player
must be kept upright by the player with her other hand. If the
player fails to do so, the pitchfork breaks and the player score
also decreases.
Animal Feeder can be currently played with any device
supported by our prototype. In particular, the pitchfork can
be controlled using a haptic interface which gives a sense
of stiffness and force feedback to the player when the
pitchfork bends. If too much force is applied to the fork,
it breaks. The haptic interface both increases the patient's
immersion/involvement and exercises the patient balancing
skills. Visual and auditory feedback is available as an option,
to increase the clarity of the exercises. A training mode can
demonstrate the raw rehabilitation exercise deprived of its
gaming look-and-feel, and it is used when the patient needs to
learn the correct movements. The final deployment includes
a virtual therapist to provide additional patient support and
warning when wrong movements are executed, either in the
form of an artificial avatar (similar to Nintendo's Mii) or as a
recorded video presentation from an actual therapist.
Notice that, as previously noted, devices not used for
gameplay can still be used for additional patient's monitoring.
For instance, Figure 5a shows how the Nintendo's Balance
Board is used to monitor the oscillation of the user's center of
mass during the game. However, the oscillations do not affect
the gameplay, but provide useful information to monitor the
patient balancing capability.
B. Fruit Catcher
In this second mini-game the patient is required to shift her
body to the left and to the right, requiring a displacement of
the whole body, while keeping the feet still on the ground.
In Fruit Catcher (see Figure 5b), the player must catch fruits
falling from the top of a tree. The player stands below the tree
with a basket on her head and can move the body laterally
to catch the fruits in the basket. When a fruit falls into the
basket, the player's score increases while it decreases when
a fruit falls on the ground. The fruits fall from different
heights and from different positions on the horizontal axis.
The basket size, the fruit size and weight, the fall frequency,
the range on the horizontal axis and the number of falling
fruits are all parametrized. The adaptation module modifies
these values to guarantee a target success rate, measured as the
ratio of successes against the number of trials, according to
the Bayesian Quest criterion [21]. The game can be currently
played either with the Nintendo Balance Board or Microsoft
Kinect. In the latter case, the player can also use her arms to
catch the falling fruits. A warning is issued when the player
tries to compensate from the limited range by displacing her
feet position as the feet position is explicitly monitored during
gaming.
(a) The Animal Feeder mini­
game
Figure 5.
(b) The Fruit Catcher mini­
game
The mini-games of the patient station
VIII.
CONCLUSION
The platform described here can be well considered a
Personalized Health System. To achieve this many possibilities
offered by ICT are explored and implemented realizing a
highly modular, adaptive platform for rehabilitation. We will
start testing this soon on a small pilot.
[5] c. et aI., "Improving left hemispatial neglect using virtual reality,"
Neurology, vol. 62,no. 11, pp. 1958-1962, 2004.
[6] "Hocoma website," http://www.hocoma.com/enl.
[7] J. W. Burke, M. McNeill, D. Charles, P. Morrow, J. Crosbie, and
S. McDonough, "Serious games for upper limb rehabilitation following
stroke," Conference in Games and Virtual Worlds for Serious Applica­
tions, pp. 103-110,2009.
[8] A. Sedda,F. Pasotti,G. Bottini,R. Mainetti,M. Ronchetti,and N. Borgh­
ese, "A new virtual reality program to rehabilitate neglect," Submitted
to Disability and Rehabilitation, Proc ICVR2011 coriference., 2011.
[9] G. et aI, "Low-cost virtual motror rehabilitation system for standing
exercises," IEEE Virtual Rehabilitation, vol. 27,no. 29,pp. 34-38,2007.
[10] M. Cameirao, S. Badia, E. Oller, and P. Verschure, "Neurorehabil­
itation using the virtual reality based rehabilitation gaming system:
Methodology, design,psychometrics, usability and validation," Journal
of NeuroEngineering and Rehabilitation, vol. 7,no. 1, p. 48,2010.
[l1] A. F. B. Coley,B. M. BM. Jolles and K. Aminian,"Arm position during
daily activity," Gait & Posture, pp. 581-587,2008.
[l2] R. L. Ganea,A. Paraschiv-Ionescu,C. Bla,S. Rochat,and K. Aminian,
"Multi-parametric evaluation of sit-to-stand and stand-to-sit transitions
in elderly people," MedicalEngineering & Physics, 2011.
[l3] B. L. et aI, "Exercise and health-related quality of life during the first
year following acute stroke: a randomized controlled trial," Brain Injury,
vol. 22,no. 2, pp. 135-145,2008.
[l4] K. L.Furie,S. E. Kasner,R. J. Adams,G. W. Albers, R. L. Bush, and
S. C. F. et aI, "Guidelines for the prevention of stroke in patients with
stroke or transient ischemic attack," Stroke, 2010.
[l5] W. S. Kamel Boulos M. N., "The emerging web 2.0 social software:
an enabling suite of sociable technologies in health and health care
education," Health Information and Libraries Journal, vol. 24, no. 1,
pp. 20-23, 2007.
[16] S. Dzeroski, B. Goethals, and P. Panov, "Inductive databases and
cosntraint-based data mining," Springer, 2010.
[l7] M. Pirovano, R. Mainetti, G. Baud-Bovy, P. L. Lanzi, and N. A.
Borghese,"Self-adaptive games for rehabilitation at home," Proc. IEEE
Conference on Computational Intelligence and Games, pp. 179-186,
2012.
[18] M. Csikszentmihalyi, "Flow: The psychology of optimal experience,"
Harper Perennial, 1991.
[19] P. Sweetser and P. Wyeth, "Gameflow: a model for evaluating player
enjoyment in games," Comput. Entertain., vol. 3, no. 3, pp. 3-3, Jul.
2005. [Online]. Available: http://doi.acm.org/IO.1145/1077246.l077253
[20] R. Colombo, F. Pisano, A. Mazzone,C. Delconte,S. Micera,M. Car­
rozza, P. Dario, and G. Minuco, "Design strategies to improve pa­
tient motivation during robot-aided rehabilitation," J Neuroeng Rehabil,
vol. 4,no. 3,2007.
[21] A. B. Watson and D. G. Pelli,"Quest: a bayesian adaptive psychometric
method. " Perception And Psychophysics, vol. 33, no. 2, pp. 113120, 1983. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/
6844102
[22] M. Kennedy, J. Schmiedeler, C. Crowell, M. Villano, A. Striegel,
and J. Kuitse, "Enhanced feedback in balance rehabilitation using the
nintendo wii balance board," in e-Health Networking Applications and
Services (Healthcom), 2011 13th IEEE International Coriference on,
ACKNOWLEDGMENT
The authors wish to thank luri Frosio for the clear intro­
duction to quaternions and Gabriel Baud Bovy for the support
on haptics. This work has been partially supported by the FP7
Project REWIRE, grant N. 287713, of the European Union
(EU).
REFERENCES
[1] C. D. Mathers and D. Loncar, "Projections of global mortality and
burden of disease from 2002 to 2030. " PLoS Medicine, vol. 3, no. 11,
p. e442, 2006. [Online]. Available: http://www.pubmedcentral.nih.gov/
articlerender.fcgi?artid=1664601&tooI=pmcentrez&rendertype=abstract
[2] c. F. Langhorne P and P. A.,"Motor recovery after stroke: a systematic
review. " The lancet Neurology, vol. 8,no. 8, pp. 741-754,2009.
[3] C. Warlow,P. Sandercock,and G. Hankey, Stroke: Practical Manage­
ment. Backwell Publishing,2008.
[4] K. et aI., "Cyberpsychology," 2007.
[23]
[24]
[25]
[26]
june 2011, pp. 162-168.
w. Ijsselsteijn, H. H. Nap, Y. de Kort, and K. Poels, "Digital game
design for elderly users," in Proceedings of the 2007 coriference on
Future Play, ser. Future Play '07,2007, pp. l7-22.
K. Salen and E. Zimmerman, Rules of Play - Game Design Fundamen­
tals. MIT Press,2003.
J. Schell, The Art of Game Design: Book of Lenses. Elsevier,2008.
J. Kirk, "Burning man," http://www.gq.com/news-politics/newsmakers/
201202/burning-man-sam-brown-jay-kirk-gq-february-2012?
currentPage=l,2012.