Neuro-physical rehabilitation by means of
novel touch technologies
Michele Confalonieri a, Piergiorgio Tomasid, Miriam Depauld, Giovanni Guandalini c,
Matteo Baldessarib, Daniele Ossb, Fabrizio Pradab, Alessandro Mazzalaib, Mauro Da
Lioa, Mariolino De Ceccoa
a
University of Trento, DII, via Mesiano 77, 38100 Trento, Italy
b
students of the course “Measurement Systems and Applications”, University of Trento,
Faculty of engineering
c
Abilita -Ospedale Villa Rosa, Loc. Vigalzano, 38057 Pergine Valsugana, Italy
d
Ambulatorio Neuropsicologia – UO1 Psicologia Clinica, 38100 Trento, Italy
Abstract. In this work we show the results of a collaboration between an
engineering department and a rehabilitation hospital in using innovative touch
interfaces properly designed for both neuro-cognitive and physical rehabilitation.
The novel touch interface measures also force so as to enable dexterity training
through ‘direct’ manipulation of virtual objects in 3D. Two dimensions via the
touch screen, the third by the force channel. We believe that this tool could
increase the degree of effectiveness of traditional rehabilitation treatments thanks
to its capability to merge physical and cognitive rehabilitation. Furthermore the
exergames implemented allow an easy personalization of the exercise structure
and difficulty level. The rehabilitator has at disposal tools to design the interface
and the exercise structure personalised to the patient kind and level of disability.
The effectiveness of the FP technology compared to more traditional methods of
rehabilitation will be measured according to specific parameters observed in an
experimental group, compared with a control group.
Keywords. Serious games, touch systems, cognitive rehabilitation, physical
rehabilitation.
Introduction
In recent years there has been a significant increase in the research for new means of
evaluation and neuropsychological rehabilitation [25]. The primary objective of
neurocognitive rehabilitation is the recovery of autonomy in the performance of daily
activities (social and work), interpersonal relationships, and more generally in
increasing the quality of life of the patient. It then becomes crucial to develop new
trainings and rehabilitation tools for clinical practice, counting on the help of new
technologies. For this purpose serious games for health were developed including
psychological therapy, cognitive training, physical rehabilitation, as well as
exergaming - games used as a form of exercise. The benefits of using VR technology
for rehabilitation have been the subject of many studies [1]. Several outcomes
demonstrate that people with disabilities are capable of motor learning within virtual
environments and movements learned in VR transfer to real world equivalent motor
tasks, and can even generalize to other untrained tasks [2]. There is evidence that
proprioceptive and exteroceptive feedback, associated with the execution of skilled
tasks, induces brain changes at cellular and synaptic level [15,23]. Learning new skills
is associated with structural changes in specific cortical areas involved [16]. Motor
skills and cognitive skills can be seen as complementary [14]. For a patient with severe
brain injury, the increased levels of autonomy must in fact provide for a modification
of the architecture of the cerebral structures of both the cognitive and the physical
channel [17]. It must be considered that the increase in cognitive skills increases the
ability to develop motor strategies. On the other hand, the improvement of motor skills
increases the Brain Derived Neurotrophic Factor (BDNF), a neurotrophin in the
hippocampus and frontal cortex which plays a fundamental role in the growth,
differentiation, and maintenance of neuronal matter. It is a fact that energy homeostasis
is fundamental for the survival of both the cell and the organism [20,21].
Accumulated evidence indicates that mechanisms of energy metabolism play a key
role in mediating aspects of higher order cognitive function. Physical activity, an event
that intrinsically impacts energy management, has repeatedly been demonstrated to
enhance cognitive function in both animal and human studies [8, 9, 10]. Physical
training has beneficial effects not only on physical functions but also on brain
functioning, including memory [12], increased capillarisation [7, 18], brain plasticity
[6,24], proteasome activation and up-regulation of the antioxidant system [11].The
development of specific skills through the use of both motor, and cognitive
rehabilitation could thus result in a more significative user recovery [19,22].
Touch sensing interfaces have rapidly increased their technological level. 3M
MicroTouch™ provides tactile feedback effects for on-screen video buttons to make
users actually "feel" like they are depressing mechanical buttons. Apple Multi-Touch™
technology enables to detect click, scroll, swipe and rotate independently. The Korean
Institute of Science and Technology developed a touch that measure normal and
tangential forces [3]. This, overtaking the actual limitations of touch screens that
currently fail to properly distinguish force intensity, will certainly open new
possibilities for serious games in the field of dexterity rehabilitation. We believe touch
technology has not yet been completely exploited and, following the path of its huge
improvements, clinical applications will certainly benefit. On this track we developed a
custom instrument, the Force Panel - FP, in the framework of the VERITAS FP7
project with the aim to quantify the dexterity of different categories of impaired
peoples. The same instrument is currently used for serious games in clinical settings.
The novel touch interface measures also force thus enabling dexterity training through
‘direct’ manipulation of virtual objects in 3D: two dimensions via the touch screen, the
third along the (vertical) force channel [4, 5]. In summary touch interface stimulate the
physical rehabilitation the visual interaction can exploit cognitive exergames.
1. Material and Methods
The device is composed of a resistive touch-screen, an embedded system, a 15”LCD
and three load cell with their relative amplification and filtration modules. The figure
below shows the instrument. Two planes compose the resistive touchscreen. One rigid
glass plane, the second plane is flexible and is in PET material that interfaces with the
user. The measure of x and y position of the touch is separately detected alternately
powering the two surfaces. The LCD is mounted onto the base by means of three load
cells that measure the contact force modulus and position. The instrument is made of a
structure based on harmonic wires that constrain the movement along the plane of the
touch screen while allowing the upper part to move along the perpendicular direction.
In this way the device orientation does not affect (or at least compensates for) the
orthogonal pressure exerted by the user measurement. The electronic drivers of the
LCD and the force transducers conditioning system are hosted under the screen. The
embedded system drives the touch and reads the force and position values. The
elaboration and graphical interface is achieved by an external computer that receives
position and force data from the embedded system and generates the visual interface by
its graphics processing unit.
The Force Panel has a spatial resolution of 0.4 mm with an accuracy of ±1.8 mm.
The system composed by the force transducers operates with a resolution of 0.05 N
while its accuracy is ±0.1 N. The maximum working frequency is 100 Hz, limited by
capacitive effects of the resistive touch panel.
Figure 1. A photo of the Force-Panel instrument and functional connections between the principal modules.
To be used by peoples affected by cognitive and physical disabilities we realized a
reclining table in which we inserted the instrument. The instrument in this way can also
be used by wheelchair users.
The software application is written using Processing 1.5.1, an open source
programming language and environment for people who want to program images,
animation, and sound. It is used by students, artists, designers, architects, researchers,
and hobbyists for learning, prototyping, and production. It is created to teach
fundamentals of computer programming within a visual context and to serve as a
software sketchbook and professional production tool.
2. Serious games for cognitive and physical rehabilitation
Some recent scientific studies emphasize the importance of integration between
physical and cognitive rehabilitation in severe brain injury [13]. Cognitive
rehabilitation seems to be more effective when combined with physical stimuli and a
well sensorial challenging. Physical stimuli can facilitate cognitive recovery through
the modulation of neurotrophic factor. The neurotrophic factors are proteins important
both during development of the central nervous system that in the processes of
neuronal survival, with a fundamental role in the phenomena of synaptic plasticity that
are the basis of a central transmission normally functioning. Recently, attention has
focused in particular on Brain Derived Neurotrophic Factor (BDNF), a particularly
neurotrophin present in the hippocampus and frontal cortex, which also plays an
important role in the processes of learning and memory, has a fundamental role in the
growth, differentiation, and maintenance of neuronal survival [21]. It has been
demonstrated, first in animals and then in humans, that the physical stimulation
produces a specific increase of cerebral blood volume of the dentate gyrus, the only
subregion of the hippocampal formation that supports neurogenesis (formation of new
nerve cells) in adulthood [17]. This change correlates selectively with an improvement
of cardiopulmonary and cognitive functions. According to these studies, the physical
activity would also be able to increase the mood and reduce depressive symptoms. It
can therefore be assumed that the physical stimulation promotes the growth of neural
precursors [15,25]. The other side of the cognitive activity, learning and adequate
environmental stimulation, allow the activation and the survival of the new cells. In
conclusion, it can be argued that in the cognitive rehabilitation of patients with
acquired brain injury (traumatic brain injury or ischemia), the integration of
neuropsychological rehabilitation paths with motor stimulation and physiotherapy can
allow the achievement of better and faster results [24].
In neurocognitive rehabilitation, use of computer programs is gaining more and
more interest [25]. These tools allow various benefits, such as greater flexibility, the
ability to be programmed depending on the condition of the patient, a greater
motivation by patients in their use, a greater effectiveness in the measurement of the
variations of performance, the possibility to be also used by patients at home. In this
study identified some software developed for the rehabilitation of specific cognitive
functions. Have been defined and organized functions target on which to develop and
modify individual computer exercises, in order to identify precisely the rehabilitative
purpose-specific training offered to patients. Were defined levels of complexity gradual,
appropriate to the degree of severity of the outcome in individual patients.
The FP instrument is used within the Hospital Villa Rosa as the experimental
protocol. To date has been mainly used as an aid in rehabilitation to facilitate the
rehabilitation of fine motor skills with exercises based on the control of digital pressure
and visuo-motor coordination. The aim of this research is to add the physical channel to
the cognitive rehabilitation. In this light the FP technology only to provides the
rehabilitation of fine motor functions of the upper limbs, but also integrates cognitive
rehabilitation and physiotherapy
We have organized and summarized the cognitive abilities in a pattern (Table 1),
in order to identify the specific functions to be rehabilitated through individual
exercises.
Table 1. Pattern of neuropsychological functions
Macrofunction
Specific
function
Spatial
Orientation
Orientation
Temporal
Orientation
Vigilance/Alert
ness
Selectiv
attention
Attention
Sustained
attention
Divided
attention
Short Term
Memory (MBT)
and Working
memory (WM)
Memory
Executive
functions
Activations level of
arousal
Selection of one
attention target and
inhibition of distractors
To preserve attention in
a long time
Division of attentive
resources between
many simultaneous
stimulus/tasks
Temporary retention
system, elaboration and
selection of visuospatial information,
verbal and write,
finalized to make a
cognitive task
Permanent or part time
retention system, in the
memory stock
Planning
Attention
Control – To
inhibit
inappropriate
responses
Include this capacities
To planning action’s
strategies
To inhibit automated
behaviours
Set Shifting –
Cognitive
flexibility
Abstraction
Motivation
Language
Oral
comprehension
Object
Space
Executive
Motion
Ability to organize the
self-perception in the
space and in in the time
Long Term
Memory (MLT)
Verbal
production
Visual perception
Specification
Strategical
To planning strategies
for problem solving
Environmental
correlations
Orient oneself of
“where and when” in
specific situations
Physiological
quickness in stimulus
responses
Ability of to take the
interested stimoulus out
of contest and to ignore
the distractors, in a
span of time
What time is it? Where
are you?
Atlete that wait the start
in the race
Try to listen to a
conversation while as
other conversations are
in progress
Speak on the telephone
while you are cooking
Pay attention to more
informations or more
tasks
MBT: memory and
recall of just presented
informations
Ability to repeat the last
expression that you have
hear
WM: ability of keep in
memory the
informations for the all
time necessary for
finalize the task
Capacity to organize
events, situation,
learnings, in order to
render available this
informations if
necessary
To be able to organize
own actions and
behaviours, in relation
of the enviromental
requests, social
relations, even in nonordinary situations
For solve algebrical
calculation, I must
remember many prior
changeovers
Ability to interact in
the usual
comunications
To interact through the
word and listening in a
conversation between 2
or more persons
To recognize the
objects
To distinguish between
a pen and a pencil
Distance between me
and an object
Abstaction and
classification of
stimulus and events
Willingness to begin
many actions
Ability to produce
understandable verbal
messages
Ability to understand
verbal messages
Ability to recognize the
objects, through the
visual channel
Ability to elaborate the
surronding space
Patterns of finalized
motor behaviour
For example
To recognize the places
and environments
Programmation of
motor actions, in
relation of space into
take place the action
Serf-memory, memory
of events, learning
knowledges
Write the shopping list:
to planning what you
want to buy. To find the
motivation for leave
home. Go in the
supermarket, according
to necessity, but having
the flexibility to put
down in the shopping
trolley one economic
product, for example.
Go in checkout counter
having patience,
standing in line and
having willingness to
pay
To climb a mountain
and in relation with the
around subjects
3. Some serious games implemented
3.1. Map Game
The exercise require to the user to indicate his actual position on google maps moving
through different zoom levels. By pushing on the screen a rectangle appear around the
touch position, while on the top bar it is indicated the intensity of the force. To go over
the level the user need to choose an area that include the actual position and keep the
force in the green range for a predefined time span indicated in the watch just below the
force bar located on top. Time period and force span can be defined in the option menu.
The feedback is provided both in wrong and correct behaviour cases. It is provided
by sound and visual cues. At the end of the test the application shows to the user a final
score.
Figure 2. Sequence of image of the Map Game exercise. From the left initial screen, during the test,
final score.
In table 2 it is summarized the rehabilitation cognitive and physical areas
interested.
Table 2. Comparative table with respect to the pattern of neuropsychological functions.
REABILITAITON AREAS ( REABILITATIVE
TARGET-NEROCOGNITIVE FUNCTION INVOLVED )
Spatial orientation
Visual perception (space)
Motion control
Force control
CORRISPONDENCE WITH TRIAL PROPOSED BY
THE STUDENTS ( MODALITY OF INTERACTION
WITH THE FP )
Capacity to collocate yourself in a interactive-map that in
function of the enlargement level, give you a different point
of view
Capacity to recognize place and environment by observing
maps and images
Total time to pass each level, number of error for each
level, mean distance to the objective point, standard
deviation from the objective point
Mean force during the residence time for each level,
standard deviation of the force
3.2. Find the Cheese
This exergame requires that the user guide the mouse through the intermediate cheeses
(the ones in piece) till the final (whole) cheese avoiding the cats. The complexity of the
labyrint is function of the selected level. During the dragging operation no force is
required but when the mouse reach the cheese a green circle shape appear and the user
need to press since a limit set in the option menu to eat the cheese.
There are some different sound and visual feedback during the execution:
•
the mouse become red and a sound is emitted if hit the wall;
•
the mouse become red a sound is emitted and a screen shoot compare if hit the
cat;
•
if the user eat the final cheese but has not yet completed all the intermediate
ones a sound is emitted;
if the level is correctly completed a rewarding sound is emitted and a final
congratulation screen appear.
•
One important feature in this exergame is the personalization of the level. The
rehabilitator can create the labyrinth in function of the patient kind and level of
disability. For example for left hemiparesis the rehabilitator could concentrate the
exercise effort in the left side of the screen thus fostering the user attention in the
affected side.
Figure 5. Find The Cheese screen shoot. Initial screen with the option; start the test; during eating step.
Table 3. Comparative table with respect to the pattern of neuropsychological functions.
REABILITAITON AREAS ( REABILITATIVE
TARGET-NEROCOGNITIVE FUNCTION
INVOLVED )
Visual and spatial perception
Pianification
Motion control
Force Control
CORRISPONDENCE WITH TRIAL PROPOSED BY
THE STUDENTS ( MODALITY OF INTERACTION
WITH THE FP )
Patient need to learn where can move the mouse and where
there are limits (walls and cats). Then he need to touch the
screen with accuracy to avoid the obstacle and reach the
target
The patient need to find the rigth path to reach the target
(cheese)
To avoid the obstacle the patient need to make movement
with a certain level of accuracy. The following parameters
are measured and stored: total time, path compared to the
optimal one, number of collision with the wall, number of
collision with cats, number of cheese eat respect to the
total
When the cheese is reached, the patient need to press
enough to eat it. The threshold is settable. The following
parameters are measured and stored: mean force for each
level, standard deviation of the force, time to eat each
cheese
4. Discussion and Preliminary Results
The effectiveness of the rehabilitation process integrated (motor and cognitive) will be
studied through an experimental design. A group of patients will be undergoing the
integrated cognitive-physical rehabilitation paths, while a control group will be treated
with traditional rehabilitation methods. Improvements autonomy of both groups of
patients will be assessed through standardized tests and scales.
References
[1] Richard H.Y. So*, K.P. Wong, S.L. Yuen, J. Tang, H. Yeung and J. Liu, Virtual reality gaming for
rehabilitation: an evaluation study with physio- and occupational therapists. VRCAI Proceeding, ACM
New York, NY, USA, 2011
[2] Holden, MK (2005). Virtual environments for motor rehabilitation: review. Cyberpsychol &Behav.
8(3):187-211; pp 212-9.
[3] Seongkook Heo, Geehyuk Lee, Force gestures: augmented touch screen gestures using normal and
tangential force, Annual conference on Human factors in computing systems, 2011.
[4] M. Kirchner, M. Confalonieri, A. Paludet, F. Degasperi, M. Da Lio, M. De Cecco, A Joint Force-Position
Measurement System For Accessibility Quantification, Accessibility Reaching Everywhere –AEGIS
International Conference – Brussels, Belgium 28-30 November 2011.
[5] M. Confalonieri, G. Guandalini, M. Da Lio, M. De Cecco, Force and touch make video games ‘serious’
for dexterity rehabilitation, pHealth 2012, 9th International Conference on Wearable Micro and Nano
technologies for Personalized Health, Porto, PT, 26th-28th June, 2012.
[6] Cotman CW, Berchtold NC, Exercise: a behavioral intervention to enhance brain health and plasticity,
Trends Neurosci., Vol 25, pp 295–301, 2002.
[7] Fabel K, Tam B, Kaufer D, Baiker A, Simmons N, Kuo CJ, Palmer TD, VEGF is necessary for exerciseinduced adult hippocampal neurogenesis, Eur. J. Neurosci., vol 18, pp 2803–2812, 2003.
[8] Fordyce DE, Wehner JM, Physical activity enhances spatial learning performance with an associated
alteration in hippocampal protein kinase C activity in C57BL/6 and DBA/2 mice, Brain Res, 1993,
[9] Kramer AF, Hahn S, Cohen NJ, Banich MT, McAuley E, Harrison CR, Chason J, Vakil E, Bardell L,
Boileau RA, Colcombe A, Ageing, fitness and neurocognitive function, Nature, 1999.
[10] Laurin D, Verreault R, Lindsay J, MacPherson K, Rockwood K, Physical activity and risk of cognitive
impairment and dementia in elderly persons, Arch Neurol, 2001.
[11] Radak Z, Sasvari M, Nyakas C, Taylor AW, Ohno H, Nakamoto H, Goto S, Regular training modulates
the accumulation of reactive carbonyl derivatives in mitochondrial and cytosolic fractions of rat
skeletal muscle, Arch. Biochem. Biophys., vol 383, pp 114–118, 2000.
[12] Radak Z, Kaneko T, Tahara S, Nakamoto H, Pucsok J, Sasvari M, Nyakas C, Goto S, Regular exercise
improves cognitive function and decreases oxidative damage in rat brain, Neurochem. Int., 38, 17–23,
2001.
[13] Angevaren M, Aufdemkampe G, Verhaar HJJ, Aleman A, Vanhees L. Physical activity and enhanced
fitness to improve cognitive function in older people without known cognitive impairment. Cochrane
Database of Systematic Reviews, Issue 2. Art. No.: CD005381, 2008.
[14] Cumming TB, Tyedin K, Churilov L, Morris ME, Bernhardt J. The effect of physical activity on
cognitive function after stroke: a systematic review. International Psychogeriatrics; 24(4): 557-567,
2012.
- Devine JM, Zafonte RD. Physical exercise and cognitive recovery in acquired brain injury: A
review of the literature. PM&R;1(6):560-575, 2009.
[15] Foscarin, S., Rossi, F., and Carulli, D.. Influence of the environment on adult CNS plasticity and repair.
Cell Tissue Res. 349, 161-167, 2012.
- Hanssen, H. et al. Translating the use of an enriched
environment poststroke from bench to bedside: study design and protocol used to test the feasibility of
environmental enrichment on stroke patients in rehabilitation. Int J Stroke. 7(6):521-6, 2012.
[16] Kempermann G; Fabel K; Ehninger D et al.. Why and how physical activity promotes experienceinduced brain plasticity. Front Neurosci;4:189, 2010.
[17] Pereira AC, Huddleston DE, Brickman AM, Sosunov AA, Hen R, et al. An in vivo correlate of
exercise-induced neurogenesis in the adult dentate gyrus. Proc Natl Acad Sci USA 104: 5638–5643,
2007.
- Pluncevic J. Influence of the physical activity on the cognitive functions with people depending
on their age. Med Arh.; 66(4): 271-5, 2012.
[18] Sahay A, Scobie KN, Hill AS, et al. Increasing adult hippocampal neurogenesis is sufficient to improve
pattern separation. Nature. Apr 28; 472(7344): 466-70, 2011.
[19] Van Praag, H.; Shubert, T.; Zhao, C.; Gage, F.. Exercise enhances learning and hippocampal
neurogenesis in aged mice. Journal of Neuroscience25(38): 8680-8685, 2005.
[20] Eriksson PS, Perfilieva E, Bjork-Eriksson T, et al. Neurogenesis in the adult human hippocampus.
Nature Med; 4: 1313-1317, 1998.
[21] Jianmin Yang et al., Neuronal release of proBDNF, Nature Neuroscience, Jan 2009.
[22] Cope N., Hall K.: Head injury rehabilitation: benefits of early intervention. Arch Phys Med Rehab,
63: 433-37, 1982.
[23] Blackerby W.F.: Intensity of rehabilitation and length of stay. Brain Injury; 4: 167-173, 1990.
[24] Lombardi F., De Tanti A., Boldrini P., Perino C., Taricco M.: The effectiveness of sensory stimulation
program in patients with severe brain injury (Protocol). The Cochrane Library, Issue 2, 1999.
[25] Mazzucchi, A. La riabilitazione neuropsicologica - Premesse teoriche e applicazioni cliniche, Elsevier,
2012.