A hybrid-Brain Computer Interface for control of a reaching and
grasping neuroprosthesis
M. Rohm 1, G. R. Müller-Putz 2, A. Kreilinger 2, A. von Ascheberg 3, R. Rupp 1
Heidelberg University Hospital, Spinal Cord Injury center, Heidelberg, Germany
2
Graz University of Technology, Institute for Knowledge Discovery, Graz, Austria
3
Otto Bock GmbH, Duderstadt, Germany
1
[email protected]
Abstract
Neuroprostheses using Functional Electrical Stimulation (FES) have proven their clinical relevance for restoration of
grasping in subjects with a cervical spinal cord injury (SCI). Current developments aim at the improvement not only of
the grasping but also of the elbow function. In very high-level lesioned patients only very limited residual functions can
be utilized for control of a reaching and grasping neuroprosthesis. Therefore a novel control interface based on a hybrid
approach is introduced, which is comprised of a shoulder position sensor and a Brain-Computer Interface (BCI). An analog control of the grasp or the elbow position is achieved by the measurement of the shoulder position, whereas the
switching between elbow and hand is accomplished by the BCI based on motor imageries. The hardware of the reaching
and grasping neuroprosthesis consists of a lower and upper arm orthosis combined with an electrically lockable / delockable elbow joint that prevents excessive muscle fatigue. Additionally, an multichannel FES device using surface electrodes has been integrated into the orthosis, which itself can be individually adapted in size and length and can be upgraded with a wrist orthosis. First tests of the feasibility of the system have been performed in SCI subjects.
1
Introduction
The European integrated project TOBI (Tools for BrainComputer Interaction) aims at the development of practical
technology for brain-computer interaction that will improve the quality of life of disabled people and the effectiveness of rehabilitation [1]. The main focus of TOBI is
the transfer of Brain-Computer Interfaces (BCIs) out of the
laboratory into real life situations. The goal of the TOBI
application workpackage “motor substitution” in particular
is the restoration of the grasp and elbow movements in
high-level spinal cord injured (SCI) individuals and in
stroke survivors. Today the only possibility for a significant improvement of a restricted or lost grasp function is
the application of Functional Electrical Stimulation (FES)
[2]. However, current FES methods are only applicable if
shoulder and elbow functions are preserved to such an extent, that the hand can be actively positioned in space [3].
If these functions are impaired, an elbow flexion / extension movement can be generated by a stimulation of the
corresponding innervated muscles of the upper arm. However, the joint position can be maintained only for a short
period of time. This can be ascribed to rapid fatigue due to
the non-physiologic synchronous activation of the nerve
fibers through FES.
Moreover, in high lesioned tetraplegic subjects only a few
residual motor functions are preserved that can be used for
control of assistive devices (ADs) in particular neuroprostheses. In a pioneering work it has been shown that a motor-imagery based brain-switch can be used to control
grasp neuroprostheses either with surface as well as implanted electrodes [4][5].
Based on the promising results of these experiments the
aim of this work is to develop a sophisticated control concept incorporating conventional user interfaces together
with a BCI.
2
Components of the hybrid BCI
The novel control concept combines conventional assistive
devices (ADs) operated by some residual muscular functions with BCI technology. Since brain and muscle-based
interactions happen simultaneously this approach is called
hybrid-BCI (h-BCI).
Brain-computer interfaces (BCIs) are technical systems
that provide a direct connection between the human brain
and a computer [6]. Such systems are able to detect
thought-modulated changes in electrophysiological brain
activity and transform such changes into control signals.
Brain signals can be recorded by placing electrodes on the
scalp (electroencephalogram, EEG), on the cortex or in the
brain. A common control strategy is the imagination of
movements (motor imagery, MI). MI results in measurable
changes in oscillatory components in ongoing EEG over
sensorimotor areas. The non-invasive Graz-BCI used in
this work is based on EEG and MI.
Before using a MI based BCI, a subject has to be trained
with the following training procedure: For the setup of an
asynchronous BCI, synchronous BCI training without
feedback has to be performed in a first step [7]. Based on a
priori information about the timing of the cues, a classifier
can be initialized from the data acquired. During subsequent asynchronous training sessions (without cues) the
parameters of the subject specific classifier are refined.
With the use of this training paradigm, it is possible in
most users to achieve classification accuracies sufficient
for setup of a brain-switch as part of the h-BCI concept for
control of the neuroprosthesis.
For generation of an analog control signals a shoulder position sensor known from established grasp neuroprostheses [4] has been used as a conventional AD.
3
Neuroprosthesis for reaching
and grasping
For implementation and evaluation of the novel user interface a dedicated hardware platform has been realized. Its
main components are an upper and lower arm orthosis
equipped with an in flexion direction self-locking, electrically delockable elbow joint and self-adhesive gel FES
electrodes in combination with a multi-channel electrical
stimulation device (“Motionstim”).
3.1
The elbow joint
The joint of the orthosis is based on a commercially available joint used in a gait phase controlled knee orthosis [8],
which has been equipped with a proprietary toothed wheel
in order to achieve a finer resolution of locking (locking
steps of 8°). The maximum loading torque of the joint exceeds 100 Nm and is therefore more than sufficient for
stabilising the elbow joint.
The joint locks itself only in flexion direction. It can be
electrically delocked to allow extension movements by
digital activation of a pulse-width-modulated (PWM) solenoid driver, which has been developed exclusively for this
application. A flexion moment is needed before the joint
can be delocked, which minimizes the likelihood for a
sudden, unexpected falling down of the lower arm.
Additionally, it is equipped with a reed-relay for providing
feedback of its locking status and a potentiometer has been
integrated for measurement of the absolute elbow angle
without the need for referencing. In figure 1 the orthosis
and the elbow joint is depicted.
3.2
The programmable, eight-channel electrical stimulator
“Motionstim” (Krauth&Timmermann, Hamburg, Germany) is used together with non-invasive surface electrodes for application of the FES and for implementation
of the control tasks.
Several software tools for the “Motionstim” have been developed in C, including a graphics based setup of the
pulsewidth maps for generation of different grasp patterns.
Furthermore, a modularly expandable, standardized serial
communication protocol has been implemented, which
forms the basis for integration of the “Motionstim” into the
hybrid control concept.
In its simplest form of control several input signals – e.g.
from a shoulder position sensor, a myoelectric sensor or a
BCI – are connected independently to the input channels of
the “Motionstim” for autonomous use. In a more sophisticated control concept, these signals are fused by an external computer, which generates high-level control commands and sends them to the “Motionstim” using the standardized serial protocol. In figure 2 a connection diagram
between the electrical components of the current version of
the orthosis is depicted.
For support of the elbow flexion in case of a very weak,
partially denervated biceps muscle two different antigravity-support mechanisms including a spring and a force
deflection have been designed and implemented.
External
PC & BCI
Shoulder joystick
Galvanic decoupling unit
„Motionstim“ stimulator
Serial interface
Elbow joint
PWM
Solenoid
driver
Lock
status
Stimulatuion channels
Figure 2 Overview of the components of the reaching and
grasping neuroprosthesis
4.
Figure 1 The adjustable orthosis (left) and its lockable elbow joint (right)
The „Motionstim“ stimulator
The hybrid-BCI control scheme
In a first scenario, a MI-BCI that employs the imagery of
movements of hands and feet is combined with the analog
signals of a shoulder position sensor. By protraction or elevation (user-dependent) of the shoulder the user can control the degree of elbow flexion / extension or the degree
of hand opening / closing. This is achieved by adjusting
the pulsewidths of constant current stimulation impulses
according to the analog signal originating from the shoulder joystick. This control approach originates from the invasive Freehand system [9], where a similar way of control
(without BCI) has been used.
The routing of the analog signal to the control of the elbow
or the hand and the access to a pause-state is determined
by a digital command signal provided by the BCI (compare figure 3).
channels for the grasp generation are hold at the current
level. If a user returns to the control of the grasp, she/he is
first asked to move the shoulder joystick to the position
where she/he formerly has locked the command signal.
This prevents an unwanted release of grasped objects. To
ensure a correct alignment to the former positions of the
joystick users are guided by an acoustic signal.
4
Figure 3: The basic control concept of the hybrid-BCI
In Figure 4 a state chart of three different states of the
grasping and reaching neuroprosthesis and the corresponding commands for switching between these states is depicted.
Feasibility tests in SCI subjects
After the assembly of an initial prototype of the reaching
and grasping neuroprosthesis first sessions in high-level
spinal cord injured patients were performed.
In order to use the overall system successfully, a FES and
BCI training has to be accomplished first. The FES training is important to achieve a sufficient strength and endurance of the stimulated muscles. The BCI training is
necessary to improve the classification accuracy and
herewith to enable the patient to control the neuroprosthesis.
Usually the FES training takes up to 8 month. It has to be
performed 6 hours/week, which can be done with the
"Motionstim" stimulator with its original firmware provided by the manufacturer. The advantage of the “Motionstim” is that it can be used as a stand-alone device for
training or in combination with the orthosis for FES by
simply switching between the standard and the proprietary firmware, which can be done by the users themselves.
Additionally, an intensive BCI training with the patient
was conducted for three days at his home. At the end of
the training, the patient was able to control a two-class
MI-BCI with an average classification accuracy of 80%
which has been sufficient for a first test run with the neuroprosthesis.
The major components of the orthosis prototype can be
seen in figure 5.
Figure 4 Simplified flowchart of control states of the the
reaching and grasping neuroprosthesis
After an initial calibration of the shoulder joystick the system enters the pause-state without stimulation and an
unlocked elbow joint of the orthosis. In order to control the
elbow, the user emits a “toggle pause”-command to enter
the “Arm-active-mode”. Now, the user has to move the
shoulder joystick into a neutral position, the stimulation of
the elbow flexor muscles is turned on and the joint is delocked by activation of the solenoid driver (if it is not
unlocked already). If the flexion moment is insufficient,
the user may increase the stimulation pulsewidths and
herewith the generated flexion force by simply moving his
shoulder more forward.
A user initiated selection of the state for control of the
hand automatically locks the elbow joint. The analog signal from the shoulder joystick is then used for control of
the opening or closing of hand. If a user switches back to
elbow control, the pulsewidth values of the stimulation
Anti-gravity support
PWM solenoid driver
FES electrodes Wrist support
Lockable joint
Angle sensor
Adjustable bars
Figure 5 The application of the orthosis in a SCI subject
Prior to the first feasibility test the mechanical components of the orthosis prototype were properly fitted to the
patient’s anatomy and the positions of the FES electrodes.
In a second step the pulsewidth maps of the stimulation
patterns were adapted to the patient’s individual muscular
status. Then the electrode cap for measuring EEG data
was applied to the patient's head and the "Motionstim"
device was connected to the control PC of the BCI via the
serial link to receive commands from the BCI. In a last
step the shoulder position sensor was attached to the patient's contralateral shoulder.
During the experiment the patient was able to autonomously control the degree of hand opening / closing and
the angle of the elbow. He was successfully able to switch
between elbow and grasp control by imagination of a foot
movement. With 1. the introduction of a dwell time of 2
seconds and 2. the prerequisite of a stable shoulder position at the time point of switching the rate of false positives was minimized to almost zero. Therefore the feasibility of the proposed control concepts has been successfully shown.
However, due to a limited range of motion in the PIP and
DIP joints of the hands it was not possible to achieve a
sufficient lateral grasp pattern. This pattern will hopefully
be achieved in the future by continuation of the stimulation training in combination with a hand splint and/or local injection of Botulinumtoxin for relaxation of the spastic finger flexors.
During the first test sessions questionnaires of the CATOR
taxonomy [11] have been performed with the two participating SCI individuals to acquire baseline data about their
current situation related to overall quality of life and their
satisfaction with their current assistive technology. The
baseline data will be used in the future to evaluate and improve the impact of the reaching and grasping neuroprothesis.
5
Conclusions and outlook
Within the application workpackage “motor substitution”
of the European project TOBI a reaching and grasping
neuroprosthesis on the basis of FES and a lockable orthosis
has been realized. The novel control concept of a hybrid
BCI incorporating conventional user interfaces together
with a BCI has been implemented and its feasibility successfully tested in two high-level spinal cord injured subjects. Based on the results of the first functionality tests
and the users’ feedback have led to possible improvements
and extensions either of the hardware components as well
as of the application software. This includes the development of a more sophisticated anti-gravity-support of the
elbow flexion, the use of lighter materials, the remanufacturing of the actuating solenoid of the elbow joint due to
optimization of the current consumption and the integration of a synchronizing hand and finger orthosis.
Double sided, self-adhesive FES stimulation electrodes
may simplify the handling of the system if used together
with a textile inlay within the orthosis that contains the
cables and the electrode contacting plates.
The intense work on software development, system integration and refinement of the hybrid control concept will
continue. As a first step towards the realization of a natural control for a hand/arm neuroprosthetic device, a study
with healthy subjects investigating the possibility for
separation of executed hand and elbow movements in the
EEG has been conducted. The participants reached classification accuracies of about 75%, which is a very promising starting point for future research on fully brain controlled reaching and grasping neuroprostheses [10].
6
References
[1] Homepage of the TOBI-Project, www.tobiproject.org
[2] Hentz, V.; Le Clercq, C.: Surgical Rehabilitation of
the Upper Limb in Tetraplegia. W. B. Saunders Ltd.,
2002.
[3] Rupp, R.; Gerner, H.J.: Neuroprosthetics of the upper extremity – Clinical application in spinal cord
injury and challenges for the future. Acta Neurochirurgica, 97, Supplement 1 (2007), S. 419-426
[4] Pfurtscheller, G.; Müller, G.R.; Pfurtscheller, J.;
Gerner, H.J.; Rupp, R.: Thought-control of functional electrical stimualtion to restore hand grasp in
a patient with tetraplegia. Neurosci. Lett. 351
(2003), S. 33–36
[5] Müller-Putz, G.R.; Scherer, R.; Pfurtscheller, G.;
Rupp, R.: EEG-based neuroprosthesis control: A
step towards clinical practice. Neurosci. Lett. 382
(2005), S. 169-174
[6] Wolpaw, J.R.; Birbaumer, N.; McFarland, D.J.;
Pfurtscheller, G.; Vaughan, T.M.: Brain-computer
interfaces for communication and control. Clin Neurophysiol. 113 (2002), S. 767–791
[7] Müller-Putz, G.R.; Scherer, R.; Pfurtscheller, G.;
Rupp, R.: Brain-Computer Interfaces for control of
neuroprostheses: From synchronous to asynchronous
mode of operation. Biomedizinische Technik 51
(2006), S. 57-63
[8] http://www.ottobock.de/cps/rde/xbcr/ob_de_de/
im_646d237_d_e-mag.pdf
[9] Keith, M.W.; Hoyen, H.: Indications and future directions for upper limb neuroprostheses in tetraplegic patients: A review. Hand Clin. 18 (2002), S.
519–528, 2002.
[10] Müller-Putz G.R.: Towards natural arm control:
classification of hand and elbow movements. Proceedings of the TOBI Workshop (2010, Graz
[11] Jutai, J.W.; Fuhrer, M.J.; Demers, L.; Scherer, M.J.;
DeRuyter, F.: Toward a taxonomy of assistive technology device outcomes. Am J Phys Med Rehabil.
84 (2005), S. 294–302
Acknowledgements: The authors would like to thank the
SCI individuals, who participated in the feasibility experiments. The presented work is supported by the European ICT Programme Project TOBI (contract no. FP7224631). This paper only reflects the authors' views and
funding agencies are not liable for any use that may be
made of the information contained herein.