J Neurosurg 106:495–500, 2007
Electrocorticographically controlled brain–computer
interfaces using motor and sensory imagery in
patients with temporary subdural electrode implants
Report of four cases
ELIZABETH A. FELTON, M.S.,1 J. ADAM WILSON, M.S.,1 JUSTIN C. WILLIAMS, PH.D.,1,2
AND P. CHARLES GARELL, M.D.1,2,3
Departments of 1Biomedical Engineering and 2Neurological Surgery, University of
Wisconsin–Madison; and 3Department of Neurological Surgery, William S. Middleton
Memorial VA Medical Center, Madison, Wisconsin
PBrain–computer interface (BCI) technology can offer individuals with severe motor disabilities greater independence and a higher quality of life. The BCI systems take recorded brain signals and translate them into real-time actions,
for improved communication, movement, or perception. Four patient participants with a clinical need for intracranial
electrocorticography (ECoG) participated in this study. The participants were trained over multiple sessions to use motor and/or auditory imagery to modulate their brain signals in order to control the movement of a computer cursor. Participants with electrodes over motor and/or sensory areas were able to achieve cursor control over 2 to 7 days of training. These findings indicate that sensory and other brain areas not previously considered ideal for ECoG-based control
can provide additional channels of control that may be useful for a motor BCI.
KEY WORDS • brain–computer interface • electrocorticography • epilepsy surgery •
subdural electrode
interface technology may provide
increased functionality and independence to individuals with severe disabilities. A BCI is a communication system that does not depend on the brain’s normal
input/output pathways for peripheral nerves, muscles, and
sensory organs.11 Instead, a direct connection is made between a computer and signals recorded from the brain. Our
work is focused on a BCI that will allow patients with severe motor disabilities to interact more independently with
their environment. Specifically, recorded brain signals are
translated into real-time actions, which can be used for a
variety of applications. Examples of BCI applications include giving a person with severe motor disabilities the
ability to answer simple questions quickly, perform word
processing, and adjust environmental variables (for example, lights, temperature, or television) by controlling the
movement of a cursor on a computer screen. Other future
applications may include mentally operating a robot, wheelchair, or neuroprosthetic limb.
The BCI studies performed in animals have led to hope
that the technology will work in humans.1,8,9 The majority of
human BCI studies have used either noninvasive, scalp-
B
RAIN–COMPUTER
Abbreviations used in this paper: BCI = brain–computer interface; ECoG = electrocorticography; EEG = electroencephalography.
J. Neurosurg. / Volume 106 / March, 2007
based EEG electrodes to record brain activity or invasive
intracortical microelectrodes to record single-unit neuronal
activity.4,5,12,14 Although both techniques have shown promise in humans, widespread use of the technology has not occurred due to technical limitations. Using scalp-based EEG
electrodes is safer than using implanted electrodes, but there
are disadvantages to the former method, including low spatial resolution, narrow bandwidth, susceptibility to signal
contamination by artifacts, and low signal-to-noise ratio in
comparison with more invasive techniques. It is unknown
whether implanted microelectrodes will remained stable for
long-term (years), multichannel recording. In ECoG a neural signal is recorded via disclike electrodes embedded in a
flat strip or grid that is placed on the subdural surface of the
cerebral cortex. The few BCI studies using ECoG indicate
that it may provide the optimal balance between signal quality, spatial resolution, temporal resolution, and invasiveness.2,3,6,10 For example, Leuthardt et al.6 demonstrated that
the use of ECoG can enable users to control a computer cursor in one dimension quickly and accurately.
Patients who have temporary subdural electrode implants
represent an excellent study population for evaluating the
feasibility of using ECoG signals to control a BCI. However, the placement of these electrodes is determined by
clinical need, and thus electrode location can vary greatly
between patients. This variety in electrode placement is
495
E. A. Felton et al.
challenging for the creation of a controlled, reproducible
study. Nevertheless, it is useful to have the opportunity to
evaluate multiple brain areas for potential use as BCI control channels. If ECoG control is possible at multiple brain
areas, patients with damage to, or reorganization of, the motor cortex can still benefit from BCIs by using nonmotor
areas. Although motor imagery for BCI control is logical to
make the translation between imagery and action more fluid, using other brain areas and types of imagery may provide equally accurate and reliable outputs. In addition, over
time individuals using BCIs may stop using imagery and
begin to simply think about the desired action itself (such as
thinking about moving a cursor on a computer screen instead of thinking about moving an arm to move the cursor).
Clinical Material and Methods
Patient Population
Four neurosurgical patient participants with a clinical
need for intracranial monitoring or stimulation mapping underwent temporary subdural electrode implantation. All
participants gave informed consent for participation in this
study. Table 1 summarizes the brief medical background
and the electrode locations for each participant. Approval
for this study was obtained from the University of Wisconsin–Madison Health Sciences Institutional Review Board
and the Madison VA Medical Center Research and Development Committee.
Recording Methods
The ECoG signals used for clinical monitoring were split
so that both clinical and research objectives proceeded simultaneously (Fig. 1).
Procedure for BCI Testing
The methodology used here is similar to methodologies
developed for EEG-based BCI systems, which are based on
using motor imagery to modulate the m and b rhythms of
the EEG signal. Study participants first went through a
screening procedure during which two different visual cues
were presented on a computer screen for 2 seconds each in
random order for a total of 3 minutes. They were instructed
to relax during one of the cues and to perform a specific
type of motor or auditory imagery during the other. Recordings from all electrodes were made during the screening
session, and all the recordings were assessed in the offline
analysis. The same motor and auditory imagery tasks were
used in screening all of the patients. Motor imagery tasks involved hand, foot, or tongue movement, and auditory imagery tasks involved familiar sounds, such as a mobile phone,
song, or voice of a relative. The screening data were processed in the frequency domain using autoregressive spectral analysis to calculate and plot the averaged spectra for
the baseline (relaxation) and active (imagery) responses. If
a high correlation (r2 . 0.3) was found for the active response, the participant was trained to use imagery to selfmodulate that signal component. Figure 2 shows examples
of screening data where motor and auditory imagery result
in a decrease and an increase in power, respectively, at a
specific frequency band.
496
TABLE 1
Demographic features and clinical characteristics*
Case Age (yrs),
No.
Sex
1
35, F
2
43, M
3
18, F
4
60, F
Reason for ECoG
epilepsy w/ rt ant
temp lobe lesion
epilepsy w/ lt temp
lobe tumor
Prior Op
none
Subdural
Electrode
Locations
rt temp lobe
lt temp lobe tumor lt temp lobe
resection 20 yrs
prior
epilepsy w/ lt temp none
lt perisylvian
lobe mass
region
medically intractable none
rt primary
facial pain
motor cortex
* All four participants were right handed. Abbreviations: ant = anterior;
temp = temporal.
Based on the screening data, frequency bands measuring 3 to 5 Hz in width (for example, a 3-Hz band could be
12–15 Hz) of individual candidate electrodes were assigned
a preferred dimension for cursor movement (horizontal or
vertical). After these parameters were established, the participants attempted to perform one-degree-of-freedom tasks
in which they controlled the vertical movement of a computer cursor by using motor and/or auditory imagery to selfmodulate their ECoG signals (Fig. 3). Depending on the
individual participant’s performance, the initial parameters
could be changed to allow stable task performance. Occasionally such adjustments were necessary because of changes in signal quality from the electrodes or differences in the
responses obtained from screening data at subsequent testing sessions. Although small changes were made as needed,
they were minimized in an attempt to keep the parameters
constant throughout the sessions, including when the task
difficulty was increased.
At the start of each trial, a square cursor appeared at the
center of the left edge of the screen and moved horizontally at a fixed speed toward the right edge where a target was
located. The participant was instructed to use imagery to
control the cursor’s vertical movement in order to hit the
target. In a two-target task, the size of each target was one
half the height of the screen and the targets appeared one at
a time in a random order. The number of targets was as great
as eight, with each target’s height being one eighth that of
the screen. Each test set was composed of 10 to 20 trials,
and during each trial the power content of the chosen signal(s) was measured continuously; the signal magnitude
was the independent variable in a linear equation that controlled cursor motion in real time. The participants received
visual feedback while they learned how to voluntarily modulate their brain activity in order to complete cursor control
tasks. Accuracy (percentage of correct targets hit), precision
(size of targets), task skill acquisition over time, and response to environmental perturbations were used to evaluate performance. For all experiments, BCI2000 software
(Wadsworth Center) was used.7
As shown in Summary of Cases, each participant in this
study used a unique control strategy necessitated primarily
by electrode location and secondarily by personal preference. This fact illustrates the inherent flexibility of the brain
to adapt to new tasks. Results are shown in Table 2.
J. Neurosurg. / Volume 106 / March, 2007
Electrocorticographic brain–computer interfaces
FIG. 1. Diagram of the ECoG recording system. Signals from the implanted subdural electrodes go to a clinical EEG
unit for monitoring and also to research amplifiers for BCI research.
Summary of Cases
Case 1
Screening data analysis showed that the motor imagery
of blinking produced an increase in power at 12 to 15 Hz
over three electrodes near the sylvian fissure (identified as
black circles in Fig. 4A). Auditory imagery produced a decrease in power over the same frequency range on the same
FIG. 2. Graphs demonstrating screening analysis output. The left panels show the change in power during imagery (solid
line) compared with baseline (dashed line). The right panels show the r2 over the same frequency spectrum. A: Output
of a single electrode in Case 4, Channel 14 (also see Fig. 4C), during face movement imagery and at baseline (rest). B:
Output of a single electrode in Case 3, Channel 9 (also see Fig. 4B), during auditory imagery and at baseline (rest).
J. Neurosurg. / Volume 106 / March, 2007
497
E. A. Felton et al.
FIG. 3. Diagrams showing examples of a two-target (left) and a four-target (right) task. The cursor appears at the center of the left side of the screen at the start of the trial and moves to the right at a constant rate. The participant controls the
vertical movement of the cursor by self-modulating his or her ECoG signals.
channels. These findings demonstrated that this participant
had the potential to modulate brain activity in one brain region using two different modalities, motor and auditory.
The selected electrodes were configured so that motor and
auditory imagery controlled the up and down directions of
cursor movement, respectively.
of cursor movement. The participant was also able to use
auditory imagery for one direction of cursor movement in
separate trials.
Case 4
This case was unique in that the electrode grid was directly over the motor cortex (Fig. 4C). The four electrodes used
for cursor control were chosen based on a response to facial
motor imagery, but instead of being given specific instructions for cursor control, the participant was simply told to
try to find a strategy that worked. The participant tried several different strategies and received continuous visual feedback of cursor movement. The participant found that arm
and tongue movement imagery worked best to make the
cursor move up and down, respectively.
Case 2
Six electrodes (one at 20–25 Hz, one at 25–30 Hz, two at
65–70 Hz, and two at 90–95 Hz) were chosen for cursor
control based on analysis of motor imagery screening tasks.
The participant was instructed to use motor imagery for the
up direction and to rest for the down direction of cursor
movement. However, within the 1st day of testing, this participant indicated that it was possible to cause the cursor
to move by just thinking about cursor movement instead
of using imagery. Frequency analysis of data from the testing tasks showed an increase in the power in all frequency
bands on the selected channels when the participant concentrated on moving the cursor up.
Discussion
The cases reported here demonstrate that study participants can learn to control a cursor in one dimension by selfmodulating ECoG signals. This can be achieved with minimal training (~ 45 minutes per day for 2–7 days). Previous
studies using EEG have shown that it can take 2 to 3 weeks
(with two or three 40-minute sessions per week) to obtain
comparable cursor control.13 Electrocorticography may be
superior to EEG for cursor control because of the reduced
need for training, but this advantage needs to be balanced by
the increase in risk due to the surgical procedure. What is
notable is that ECoG control has the potential to be com-
Case 3
Screening data showed that motor imagery of tongue and
right arm movement produced a change in activity in three
electrodes on the perisylvian grid (indicated as white circles
in Fig. 4B). In addition, one electrode (depicted as a gray
circle in Fig. 4B) showed a response to auditory imagery.
This participant was able to use the two types of motor imagery simultaneously to control the up and down directions
TABLE 2
Overall performance of each participant over 2 to 7 days*
2 Targets
Case Days of No. of
No. Testing Trials
1
2
3
4
2
4
7
3
54
75
702
192
3 Targets
No. of
Accuracy (%) Trials
70.4
70.7
72.8
68.8
54
21
349
176
4 Targets
Accuracy (%)
No. of
Trials
42.6
100.0
61.0
60.8
––
69
84
––
6 Targets
No. of
Accuracy (%) Trials
––
79.7
72.6
––
––
143
––
––
8 Targets
Accuracy (%)
No. of
Trials
Accuracy (%)
––
69.2
––
––
––
272
––
––
––
84.6
––
––
* Random movement would result in an accuracy rate of 50% for a two-target task, 33.3% for a three-target task, 25% for a four-target task, and so forth. –– = not applicable.
498
J. Neurosurg. / Volume 106 / March, 2007
Electrocorticographic brain–computer interfaces
technology to include those with damage to, or reorganization of, the motor cortex. Sensory areas can also be used as
independent control channels in conjunction with motor areas to increase the available dimensions of control. Finally,
using multiple control modalities simultaneously also provides an opportunity to compare brain plasticity of sensory
and motor areas.
In interpreting the results of this study, one should bear in
mind that the electrode grids were placed at different locations in different participants and were placed over tissue
that may not have been entirely normal due to underlying
disease. Performance was also affected by many other factors, including participant discomfort (due to surgery), the
presence of visitors, various distractions, and day-to-day
changes in signal quality. Although some parameter changes were necessary to stabilize performance, we attempted to
standardize the environmental and testing conditions to reduce sources of error within our control. As more patients
participate in the ongoing study, further conclusions can be
made regarding the optimal electrode placement and type of
imagery for BCI control.
Conclusions
Four participants were able to self-modulate their ECoG
signals effectively by using motor, auditory, and/or selfdirected perceptual imagery in a closed-loop design. These
findings indicate that sensory and other brain areas not previously considered ideal for ECoG-based control can provide additional channels of control that could prove useful
for a motor BCI. More advanced control with increased accuracy, precision, and degrees of freedom is the next challenge in ECoG-based BCI research.
Disclaimer
The BCI2000 software is freely distributed by the Wadsworth
Center in Albany, New York. None of the authors has a financial interest in BCI2000, nor did any receive financial support from the
Wadsworth Center during the preparation of this paper.
Acknowledgments
FIG. 4. Schematics showing the location of subdural electrode
grids in three cases. A: Case 1. Location of an 8 3 8 electrode grid
over the right temporal lobe. The electrode channels used for cursor
control are indicated in black. B: Case 3. Location of a 5 3 6 electrode grid over the left perisylvian region. The electrode channels
used for motor imagery cursor control are white and the electrode
used for auditory imagery cursor control is gray. C: Case 4. Location of an 8 3 8 electrode grid schematic over the right motor cortex. The electrode channels used for cursor control are white.
petitive with more invasive control methods that have been
reported to date.
Intuitively, signals from the motor cortex would provide
optimal control for “motor” output; however, this study has
shown that other brain regions, including sensory areas,
may be equally capable of producing signals that can be
self-modulated for BCI control. Using sensory areas expands the population of patients who can benefit from BCI
J. Neurosurg. / Volume 106 / March, 2007
We are grateful to the Wadsworth Center in Albany, New York,
for technical and software assistance, and especially thank Gerwin
Schalk, M.S., Robert Radwin, Ph.D., Michael Neelon, Ph.D., Hans
Bakken, M.D., and the patients who participated in this study.
References
1. Carmena JM, Lebedev MA, Crist RE, O’Doherty JE, Santucci
DM, Dimitrov DF, et al: Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol 1:E42,
2003
2. Felton EA, Wilson JA, Radwin RG, Williams JC, Garell PC: Electrocorticogram-controlled brain-computer interfaces in patients
with temporary subdural electrode implants. Neurosurgery 57:
425, 2005
3. Huggins JE, Levine SP, Fessler JA, Sowers WM, Pfurtscheller G,
Graimann B, et al: Electrocorticogram as the basis for a direct
brain interface: opportunities for improved detection accuracy, in
First International IEEE EMBS Conference on Neural Engineering. Capri Island, Italy, 2003, pp 587–590
4. Kennedy PR, Bakay RA: Restoration of neural output from a
499
E. A. Felton et al.
5.
6.
7.
8.
9.
10.
11.
paralyzed patient by a direct brain connection. Neuroreport 9:
1707–1711, 1998
Kennedy PR, Bakay RA, Moore MM, Adams K, Goldwaithe J:
Direct control of a computer from the human central nervous system. IEEE Trans Rehabil Eng 8:198–202, 2000
Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW:
A brain-computer interface using electrocorticographic signals in
humans. J Neural Eng 1:63–71, 2004
Schalk G, McFarland DJ, Hinterberger T, Birbaumer N, Wolpaw
JR: BCI2000: a general-purpose brain-computer interface (BCI)
system. IEEE Trans Biomed Eng 51:1034–1043, 2004
Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP: Instant neural control of a movement signal. Nature 416:
141–142, 2002
Taylor DM, Tillery SI, Schwartz AB: Direct cortical control of 3D
neuroprosthetic devices. Science 296:1829–1832, 2002
Wilson JA, Felton EA, Garell PC, Schalk G, Williams JC: ECoG
factors underlying multimodal control of a brain-computer interface. IEEE Trans Neural Syst Rehabil Eng 14:246–250, 2006
Wolpaw JR, Birbaumer N, Heetderks WJ, McFarland DJ, Peckham PH, Schalk G, et al: Brain-computer interface technology: a
500
review of the first international meeting. IEEE Trans Rehabil
Eng 8:164–173, 2000
12. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G,
Vaughan TM: Brain-computer interfaces for communication and
control. Clin Neurophysiol 113:767–791, 2002
13. Wolpaw JR, McFarland DJ, Vaughan TM, Schalk G: The Wadsworth Center brain-computer interface (BCI) research and development program. IEEE Trans Neural Syst Rehabil Eng 11:
204–207, 2003
14. Zerris VA, Donoghue JD, Hochberg LR, O’Rourke DK, Chiocca
EA: Braingate: turning thought into action—first experience with
a human neuromotor prosthesis. Neurosurgery 57:425, 2005
Manuscript submitted October 24, 2005.
Accepted June 27, 2006.
Address reprint requests to: Justin C. Williams, Ph.D., Department of Biomedical Engineering, University of Wisconsin–Madison, 1550 Engineering Drive, Madison, Wisconsin 53706-1608.
email: [email protected].
J. Neurosurg. / Volume 106 / March, 2007