Nordic Journal of Art and Research
ISSN: 1893-2479
www.artandresearch.info
In from the margins - enabling people with
disabilities to learn and create music
Tim Anderson1
Inclusive Music2
Abstract: People with the most severe physical disabilities, for example who are only able to move their
eyes, are at most risk of being left in the margins of society. They can communicate via 'eyegaze' text
systems, but the challenge is to design an analogous music system to enable them to learn about,
explore, and create music, so as to communicate and connect with others in a more universal way. The
combination of eyegaze and music is a new area, so the design methodology relies on incorporating
eyegaze features into an existing music system designed for switch-control. The design of this usercentred system has evolved over 20 years, with the continual involvement of disabled users and
teachers. The resulting prototype eyegaze music system is designed to facilitate exploration and
creative work without continual help to operate it, and was successfully tested by eyegaze users. This
is the genesis of an instrument to enable anyone - even with the most severe disability - to participate
and share in creative activities, and connect them more with society.
Keywords: didactics, explorative,
learning, eyegaze, music, software,
AAC, user-centred,
emancipative, interaction, peers.
1
Inclusive Music, E-mail: [email protected]
2
www.inclusivemusic.org.uk
http://dx.doi.org/10.7577/information.v4i2.1552
1
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Introduction
This article focuses on people who have great difficulty taking part in the kind of music-making and
learning which are normal opportunities or activities for most people in society. It focuses on potential
students who are at the limit of physical disability, and cannot control any part of the body except their
eye direction (often termed 'locked in'). Such people can too easily be hidden in the margins of
mainstream human culture, and giving them tools to enable them to explore and learn about music can
help them back. Creating a musical work can allow them to connect with people who hear it, and let
them communicate on an equal level with other people, and help them bring their voice 'in from the
margins'.
If people can control their eye movement, then 'eyegaze’ systems built into a computer screen can
detect this and move a cursor to match where they are looking. The screen can display a set of graphic
items (eg 'buttons'), and they can select one by 'dwelling' on it - ie looking at it for a set time. Such
systems are now commonly used to enable people to type words and communicate ('AAC' 3) using only
eye movement.
Having been asked to develop new software features to provide a way for such people to create or
learn music using eye-control, the designer therefore had to ask the question 'How does such a system
need to work to enable people to learn, to explore and work with music independently using eyecontrol?'. This article explores the issues in designing such a system, to enable people using eyegaze to
participate in music education, and describes some initial outcomes with eyegaze users.
We first briefly look at existing eyegaze AAC systems, then at the issues around designing a similar
system for music. We also find that there is almost no available body of expertise or knowledge for
building an eyegaze music system. Thus, we need to examine other options, specifically whether we can
utilise the experience which has derived from the development of the accessible music software 'EScape'. This is focused on enabling disabled switch-users to work with, and explore music making both by composing or by live performing. If need be, a student can operate everything using just one
physical movement to operate a single switch.
E-Scape has been created via user-centred design over almost 20 years of observation of, and
consultation with, users and teachers, during which a number of specialist features have been added and
refined, and much knowledge of user interface and operational issues gained by the designer. We then
look at how these features could be adapted to enable a viable working prototype to be developed as the
first stage of eyegaze music system development, enabling the first users to operate it successfully and
provide essential feedback.
Art didactics
We first need to consider how a severely disabled student can be integrated and brought into music
education, and given a chance to learn. Music is a subject where interaction forms an integral part, with
people playing and listening together. Those learning theories which emphasise the importance of
participation and interaction in the learning process, and of the relational aspect of teaching, are therefore
of particular relevance to music education. From this perspective, one can say that learning,
communication and participation are core concepts that have a close relationship with each other
(Skogdal 2014b).
3
AAC is an acronym for 'Augmentative and Alternative Communication'.
InFormation
2
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
In the area of didactics, one aspect is a consideration of the ways a teacher and student (and their
peers) can relate within the learning process. These can span a range of teaching models, from traditional
'showing and telling' at one extreme, to unguided 'learning through experiencing', or 'learning by
exploration' at the other.
The former involves an expert teacher showing and telling information, with the student then going
away (a typical higher-education 'lecture'). This is the extreme; the model would usually also be
accompanied by some exercises set by the teacher, for the student to practice and hone the topic they
have been shown, and test if they have absorbed it. Music in particular has an extensive history of
teaching methods of this kind which focus on the 'master' filling up the student with his or her
knowledge. This is a tradition which has excluded many people from music education (Ruud 1996).
There are still many teachers working with conventional instruments who find it hard to embrace or
adapt to the newer explorative teaching methods. However, it must be said that in recent years teaching
has slowly started changing through adapting a wider variety of methods, and adopting plans and
regulations aiming towards a more inclusive educational system.
In contrast, 'learning by exploration' (we can use the term 'explorative learning') focuses on the
student learning about a topic by exploring and working within it. The idea of learning by participating
in activity is over a century old. "Learn to do by knowing and to know by doing" (Dewey and McClellan
1889).
Students can think they know something, but until they test it by trying it out in action they can't
rely on it as 'real knowledge'. The activities will usually not be just random 'play' - a teacher can advise
and gently guide students during their exploration. Most importantly, the teacher can plan the set of
activities (or even specific tasks) with some educational goals in mind. But the focus is on students
trying things out themselves, and 'learning by doing'.
At any learning stage, there will be a range of tasks of varying difficulty. Students will always be
able do some easier things independently without help, and other more advanced things when assisted
by someone with more knowledge (Daniels 1995). If only given tasks at the easier end of the range, then
students are able to work independently without any help, but they learn nothing new. It is only when
stretched to try harder tasks which do require some help that they can develop their knowledge and
skills. Vygotsky calls this range the 'zone of proximal development', and a good teacher should plan
activities that encompass it (Karpov & Haywood 1998). Students may at first rely on help from others,
but can gradually evolve towards working more independently as they gain knowledge, and a good
teacher should gradually withdraw support as a student achieves success (Hammond & Gibbons 2001).
The aim ideally is for the student to achieve results, and create something while working and
learning. To have 'results' is especially important for a disabled student, as the working effort is far more
laborious and physically tiring, so achievement and self-satisfaction are important to maintain
motivation. Concerts and performances with disabled musicians have shown the importance not only of
the process, but also the need to challenge people to do a good job, even when they have other difficulties
in their daily life (Svendsen 2014).
Whatever their starting point, everyone has the potential to evolve their experience and increase their
knowledge (Imsen 2005). With support, learners are able to extend their current level of knowledge
(Cole 2001). This support for development and learning is sometimes termed scaffolding (Rasmussen
2001). Some have criticised this term, as the metaphor might be interpreted as the teacher telling students
what they should learn (Stone 1998). But a correctly judged supportive scaffolding can show what the
student is able to achieve in the future, "What they can ask" (Vygotsky et al. 1978). Having tried doing
InFormation
3
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
something with support, the student now knows what is possible, and what to ask for help with if the
support is reduced.
However, to participate in any learning activities, all students need appropriate 'tools' (whether a
pen, a computer, or a flute) which they can physically operate. For example, when learning to compose
or play a musical instrument, a big part of the process is exploring and practising with composing tool
or instrument, and to do this students need to be able to work independently.
Another important facet of learning is participation in a group. As well as working with a teacher,
Dewey considers that communication and activity together with others is important in order to transfer
and share knowledge. (Dewey 2010). Learning in the company of others, especially interaction with
peers in a social context, is thought to bring many educational benefits, as well as enhancing children's
self-esteem and motivating them to tackle new tasks. (Damon 1984).
So the concept of students learning more by working with peers who have more knowledge than
they could alone is well-established. Peers can be 'scaffolding' support for a learner, but for the latter to
really be a 'peer' the relationship needs to be mutual - ie sometimes the disabled student should be able
to support their peers, not always the other way round. This will only be possible if a disabled student's
music system can provide them with the necessary support.
Until recently there haven’t really been any tools or systems available which can enable people who
can only move their eyes to work with music. Because participation hasn’t been thought of as a
possibility, these people haven’t been taking part in music education or musical activities, and no one
has considered how they could be included. But new opportunities developing from music technology
and computer access technology are gradually changing the perception and reality of what is possible,
and enable us to now envisage the provision of appropriate tools for these students via a computer-based
music system.
The system should ideally aim to provide some of the support 'scaffolding' that disabled students
need, so they can operate on a more equal level and can interact with teachers and peers in a far more
normal manner. The teacher is not then having to continually work for the student (for example to write
things down for them, or operate a computer), but can collaborate with them in discussing or performing
the activity. The system should provide consistent support, but the aim is to gradually reduce the level
as the student learns more. Thus, the system should also aim to promote "... a methodology that
emphasises customising learning" (Skogdal 2014a). This can provide a flexible framework which
enables a teacher to not only set tasks and activity environments of varying difficulty, but also tailor the
level of support, eg to 'remove scaffolding' at points. These are ambitious aims - for the system to not
only provide support for disabled students to participate at all in music eduction, but also enable
explorative learning by varying the level of support and challenge to maximise each student's learning
opportunities.
Design methodology
We now consider the issue of researching how to provide a system which provides such 'scaffolding'
support for an eyegaze student, and also empowers them to work it themselves. As the evolution of eyecontrolled music systems is at a very early stage, little or no literature or experience exists. Thus, the
basis of the research and development of this eyegaze music system has, of necessity, been almost
entirely built on experience from existing users and developers of eyegaze AAC systems, the design of
a proven switch-controlled music system, plus prototype testing with eyegaze users and people working
with them.
InFormation
4
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Even literature about 'assistive music technology' 4 in general is very limited. Hence, during the
process it has been especially important to keep in touch with people working within, and experienced
in the same field - to root the work in practise on the ground, and to discuss observations and design
ideas. These often comprise scattered individuals or small groups in various niches, (eg York, Tromsø,
Glasgow, Manchester, Rotterdam, Cork, Dublin).
Observation and feedback, both from users and teachers working with them is especially vital for
eyegaze, as very few designers can replicate the experience of operating an eyegaze system, and each
user is different. It is very hard to design for eyegaze users, and determine the best ways of doing
operations or placing items on the screen, without observing users who are genuinely dependent on
using their eyes.
This kind of design based on user involvement and testing is often termed 'user-centred design' - the
designer analyses user needs and abilities, foresees how a user will use the system, and designs features
accordingly. However, it is hard for designers to intuitively understand what any user experiences - let
alone a disabled one - so testing in the real world is vital to assess the validity of the designer's
assumptions.
It must be made clear that we are here talking about 'user-centred' design, not 'user-led' or 'userdriven'. There are some criticisms of 'user-led' design, for abdicating responsibility and expecting users
to know what they need. Although users can well flag up problems, they may not always know the best
solution. "Design is about addressing user needs, not just listening to user demands." (Kitson 2011). In
user-centred design then, designers should not just listen to users' ideas, although these are important,
but also observe how they work and conceive improvements to help with specific issues which are noted.
Being user-centered means understanding a problem and the users, analyzing user behavior
and listening to their wants, then translating this into needs that drive a creative solution to the
problem. Being user-led or user-driven, on the other hand, means responding to user feedback
without applying the filters of analysis and translation. (Kitson 2011).
So we both want, and need, to base research on the involvement, and observation, of eyegaze users who
have the ability and motivation to work with music. However, this does require an initial prototype
system being in place for people to try out. As suitable eyegaze users willing or wanting to try out the
system are geographically widespread, and have limited time and energy, it was especially important
that the first system to be designed and built - before any user contact - was as usable and capable as
possible, otherwise the process would take a long time to progress.
The question is then of how we create a prototype eyegaze music system good enough to trial by
users, when there is little or no experience or literature available on the subject. We have plenty of
knowledge of workable eyegaze AAC systems, with practical proof in the shape of competing
commercial systems, which have broadly similar user interfaces and features, and can be seen to be
effective for people to use.
What we do have in the music area, is research done over many years to develop a disabled-access
music system called 'E-Scape', which is now proven in use. So can we use this information to make an
eyegaze music system? In the following sections, we will describe existing eyegaze systems for textentry, the differences in requirements to support working with music, and some relevant features of EScape which assist switch-users.
4
Assistive music technology (AMT) is a general term for the use of custom or standard electronic hardware,
usually linked to software, to create systems to assist disabled people to play music.
InFormation
5
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
If we can identify the requirements for an eyegaze-controlled music system, and the differences
from, and similarities to switch-control, we can determine the changes needed, or additional features
which should be applied to E-Scape to 'convert it' for eyegaze use. We could thus effectively merge
features of the two types of system to have the best chance of having a viable first prototype for eyegaze
user trials.
Eyegaze systems for communication
Over the last 15 years, eyegaze systems have become more sophisticated, usable, and inexpensive, and
it should now be feasible (in the developed world) for anyone in this situation to have one and learn to
use it.
The first stage of the research process involved examining existing AAC eyegaze systems for
entering text, which will inform the design of the equivalent music system. It should be noted that when
using such a system to write text, the task is relatively simple - we are not doing complex layouts and
tables, mainly just writing words. The main 'top level' screen typically displays text items which can be
groups of letters, 'numbers', or 'punctuation'. If users look at and 'dwell' on a group, they are next
presented with a further screen which displays items in that group (eg individual letters or numbers)
from which they can select one. This is then added to a word or sentence they are building up.
Figure 1. A typical eyegaze AAC screen to enter text
Various additional features help reduce the amount of work needed, for example:
(a) if you enter a full-stop (period), the system automatically inserts a space, and makes the next letter
capitalised;
(b) after you have finished a word, a space will automatically be inserted before the next word;
(c) most importantly, there is a 'prediction' system, as used in smart-phones - a dictionary of words, so
that after typing just 3 or 4 letters, you are presented with a list of words beginning with those letters,
from which you can choose. Thus typing a 10 letter word could only actually involve 3 or 4 selections
rather than 10. Some systems also 'understand' grammar, and in some cases can predict and provide a
list of words which could naturally follow the previous one.
But even without these aids, compared with music (see below) the operations the user may want to
do are relatively limited; the majority of users will just be entering text, ie choosing letters, numbers,
words or punctuation in turn. Any editing which they might do will still be at a relatively simple level,
InFormation
6
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
such as moving through the text selecting words, deleting, copying, moving, or pasting them, or perhaps
choosing text style or size.
Working with music
Eyegaze AAC systems, as described above, are in widespread use and work effectively for the job they
do. What are the challenges in transferring their design to work with music?
Let us briefly look at the kinds of activities a student might need to do to explore music-making.
Firstly, the structure of music is far more complex than text, and the operations needed to be done to
enter and edit it are also more varied and involved than word processing. For example, a piece of music
will typically have many tracks ('parts'), each usually having a different instrument. Each track will
contain many notes, each one of which has to be entered, and has various aspects, eg a position in time,
pitch, length and loudness. Notes can overlap other notes or start with them (ie a chord), or be grouped
in other ways. It should also be obvious that we can't use text-style 'prediction' without curtailing a
composer's freedom to structure the music any way they want - there is not usually one 'correct' next
note!
There are then many ways we can edit notes and tracks. As for letters and words in a text editor, we
need to select one or more notes as a block, then copy, delete or paste them. But in addition we need to
be able to do many other operations, for example: change notes' position in time, their pitch, length, or
loudness; choose instrument sounds; add, move or delete parts. We also want to be able to play back the
music we have put in, sometimes while in the process of editing, in various ways and from various time
positions.
The E-Scape switch-accessible music system
We now present some of the history and rationale behind the research and development of the switchaccessible music system 'E-Scape'. This has been built over many years using methods informed by
'emancipatory research', and has been the starting point for this eyegaze music project and the inspiration
for its methodology. Emancipatory research involves researchers being focused on helping disabled
people to empower themselves. "The placing of control in the hands of the researched, not the
researcher" (Oliver 1992). "Researchers must put their knowledge and skills at the disposal of disabled
people” (Barnes 1992).
User participation has been the guideline for this research work over the last 20 or more years,
developing a system which aims to empower people with various disabilities, and enable them to learn
and work with music. Because every user is different, the design of the system has been (and still is)
continually evolving to cater for individual needs, and enable as many people as possible to create music
independently.
Although people with disabilities haven’t been directly involved in the detailed design work, they
have been involved in every other part of the process. The concept has all the time been to develop a
system which can adapt music-making (or in other words adapt society) to the disabled person, rather
than the person having to adapt to how most people compose or perform music. The aim has been, and
is, to improve the life and opportunities of people with severe disabilities.
Thus, E-Scape has developed as a necessarily user-focused or 'user-centred' system. We say
'necessarily', because the target users of the system have very limited scope to adapt to the way a system
works; the system needs to adapt to the way they need to work. Disabled users simply don't possess
InFormation
7
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
much 'ability overhead', where the user copes with sub-optimal features or inefficiencies in a system.
The fact that the developer has been in constant touch with users throughout the development cycle has
not only enabled adaptations to be tailored for each specific user, but also delivered to them on a fast
rotation to enable fresh feedback to be received. This development is thus a prime example of usercentred 'emancipatory' methodology.
Testing and observation has been undertaken over a 20 year period (and is still ongoing), with many
dozens of end users, from 18 months old to late middle-aged adults, from several different countries.
Information has been collected both from direct feedback from users and teachers, and from
observations of people using the system in different situations and stages of development. This degree
of user participation, and continual development means the system is now quite refined for a range of
disabled users. The number and diversity of testers has also resulted in a wide knowledge of the variation
of adaptions which are needed to provide an optimum system for people with disabilities. However, the
research has at no stage hitherto included eyegaze users.
Adapting E-Scape for eyegaze music
We now present the research results and the evolution of the prototype 'E-Scape eyegaze' system. We
can see that E-Scape's user-centred design with extensive observation over time of a wide variety of
users, as well as advice from teachers and helpers has led to the present switch-operated system which
is now well-proven. The question was then as to how to design an eyegaze-operated music system, with
the limited information and experience available. As described earlier, for such a complex system to
evolve we have to rely on observation of people using it, but for this process to work the initial prototype
needs to be capable enough for people to use it to do something useful and interesting in the first
instance.
So, how do the systems compare, and what changes will need to be made to E-Scape? An
examination of both systems showed that many of its specialist features that support independent use
with switches could be transferred to facilitate eyegaze use, and required little or no structural change.
So by adding a few new features, plus some adaptation of existing ones, we might produce a prototype
system with enough functionality to be tested by real eyegaze users.
Key features we know are needed in eyegaze AAC systems include linked screens of buttons, one
of which can be selected by dwelling the cursor in it. Each button performs an action, eg typing a letter
into the text, or opening another screen of buttons. There are various settings such as variable dwell
time, changeable colours and text size, etc.
Happily, E-Scape's top-level user interface is already quite similar; it has a structure of menus which
appear one at a time, with each item performing a musical action or opening another linked menu. This
matches the hierarchy of screens of buttons used in eyegaze AAC systems. So the menus simply needed
expanding to fill the screen, plus making the layout of menu items more flexible, with variable numbers
of columns and rows of items. NB. E-Scape menu items can already speak, for use by non-readers.
E-Scape's menus also have a large 'header' section with a question, prompt or other information,
which can provide a place where the user can look without making anything happen (ie no 'dwell'
function there). Each user required different sizes and positions for this rest area - for example some
needed it on top, some on the right.
InFormation
8
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Figure 2. 'Add notes' menu in eyegaze mode (with header on right)
Some users need larger buttons to be able to dwell on them successfully, so have fewer options at once.
Figure 3. 'Add notes' menu, header at top, with just 4 items
There are many other settings, allowing the system to be tailored to suit each user best.
As described above, the structure and order of context-dependent menus following each other
guides the user, and also reduces the number of scanning and selection operations. This goes some way
to compensate for the fact that there is no possibility of 'text prediction' to save work. Such features are
important for eyegaze users, who typically find working with eye direction very fatiguing, arguably
InFormation
9
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
more so than many switch-users; it is far harder to keep your eyes pointing somewhere particular to
'rest', than just relaxing and not touching your switch. The accuracy you need to look at and maintain
your direction in order to 'dwell' and select something is also hard to achieve without tiring.
Some additional features were needed when performing an editing procedure in the music ('score')
window when no menu is open, for example stepping through a series of notes. In this situation we need
a way to trigger each action like a switch-user would. Happily, the score window already has a large
area at the bottom right where information is shown. This can be used as a button to dwell on (or just
move to) to act as a switch for 'next action', eg to select the next note. The bottom left 'play button' can
also be used as a 'stop' button during operations. All that was needed were some label changes, and
options to make each button taller or wider, to suit the user.
Figure 4. E-Scape eyegaze - Score window
Various key features from E-Scape were also able to be used almost unchanged to help reduce the
physical and cognitive work-load for the eyegaze user. These include menus, guidance, customisation,
and auditioning.
Menus
Each menu (ie screen of buttons) fills the screen, and only presents the choices which are appropriate in
that situation, depending on the activity being undertaken. For example, after opening the main (toplevel) menu, the user might choose 'Edit', which then opens the Edit menu, with various options to edit
the selected notes.
InFormation
10
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Figure 5. 'Normal' Edit menu
But if no notes were selected, the same menu looks rather different.
Figure 6. Edit menu - if no notes are selected
Menus have a large 'rest' panel, which can be looked at without causing anything to happen. This panel
can contain a 'question' or instruction, which may be phrased in verbose or conversational language. The
question can also be altered to suit different circumstances and/or contain information about the current
situation, or stage of the operation. (This exact structure was present in the switch-operated system).
InFormation
11
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Figure 7. Verbose menu asking a question after putting in a note
All this has been found to reduce confusion in users, and enable guidance or explanation to be given
when needed.
Guidance
Much design effort has gone into guiding the user through activities, via a linked series of operations.
This design has evolved from watching people at various stages of experience using the system, and
noting what they find hard to do, and/or understand.
The aim is to guide users through operations and reduce the number of dwell selections needed,
both to reduce physical effort and lead them through the process. Users still need to make decisions at
each stage as to what they want to do, but the laboriousness is reduced. During an operation, they are
presented with a series of context-dependent menus within a framework aimed at completing the task.
After each choice or stage of the procedure, they are automatically presented with the next appropriate
menu. A menu choice might perform a musical operation, after which (if appropriate) it might
automatically return to the previous menu, or re-show the same menu. Menus may alter their contents
depending on the previous action, and only show the options which are relevant.
A good example of this is copying and pasting. We are used to doing this in documents, and the
same can be done in music with notes being copied then pasted somewhere else. However, observation
of users illustrated that the process was problematic, as it does involve a number of stages. We need to
describe these in a little detail, as this issue is important to grasp.
First we must select the set of notes to copy. Next we have to go back to the 'Copy' menu, and select
the 'Copy' button, but nothing then seems to happen, which confused users - remember that most will
have no experience of editing text, even if they have been writing with AAC. Thus we had to explain
that the notes have been copied 'into a hidden storage place'. Then we want to paste the notes, but before
we can do this we need to choose the location we want to paste at. Again we must get to the 'Copy'
menu, then select the 'Move cursor' button, then get to the position we want. Finally, we need to go once
more to the 'Copy' menu, and select 'Paste'.
InFormation
12
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Using eyegaze (or indeed switches), each of these stages can involve a long-winded process, and it
was noted that users - especially those newer to music-making - could easily get lost as to where they
were, and what the next step was. They often needed continual verbal guidance and prompting of what
to do next. So could the system itself help them with similar guidance to make the operation simpler,
easier to operate and less confusing?
An 'automated' procedure for copying was designed5, which guides users through the process by
prompting them at each stage: First they are asked to select the notes to copy, and are guided through
that process. After this, the notes are automatically 'copied' (no menu operation needed), and users are
then presented with a final menu with options to choose where to paste the notes. When they get to the
position they want and finally choose 'OK', the notes are automatically pasted in.
At each stage they are given a menu to confirm they are happy, or want to go back or forward one
step (users quite often made this mistake), or start again. This menu keeps reappearing until they choose
'OK', and means users have the chance to correct a mistake within the process, and don't have to go back
to the top-level menu until they've finished.
This is just one example of a 'guided procedure', developed with user participation over a period of
time. Not all are as involved as this, but observations have shown that users are helped by using such
procedures. People are able to operate the system themselves who might otherwise 'get lost' doing it step
by step as described earlier.
But equally important is that such procedures drastically reduce the number of selection ('dwell')
operations needed. It must be remembered that operating with eyegaze is tiring and requires
concentration, and that reducing laboriousness can help people who couldn't otherwise work physically
for long enough, enabling them to explore more freely and achieve and learn more.
Customisation
A teacher is able to 'customise' each menu to remove items not relevant to the musical activity being
undertaken, or reduce the level of support 'scaffolding'. In addition, by removing or adding choices the
teacher can effectively guide students through a different set of possibilities, to tailor their pathway of
exploration or introduce new topics or terminology. Thus different activity sessions can have different
themes depending on the options made available or not.
Music auditioning
During some note editing operations (eg after each change of pitch or position), music in the area you
are editing can be automatically played back for you. This greatly reduces physical effort - in most music
systems, playing back a specific fragment of music around a note after you have edited it is often
surprisingly laborious (Anderson 2002). Some editing operations also repeat, with feedback provided
each time the note changes.
An example of how this greatly helps users (especially beginners) is when they are adding a new
note to their piece. Eyegaze-users need to do this by 'Step Entry' - they choose the time where the note
will start and its length, then choose the pitch. For beginners, the easiest way to choose the pitch is 'by
ear' using another example of a 'guided procedure': A new note is added and initially has the same pitch
as the previous note, but can then repeatedly change by one step, each time playing back how the music
now sounds. Students can stop the process when they hear the pitch they want. Hopefully, as they learn
5
The design was as ever, repeatedly modified from observation of, and feedback from users trying it.
InFormation
13
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
more about music, students can choose a pitch and octave for a new note using menus ('C' , 'D#' etc).
Another example is that in some menus (eg for choosing an instrument, or note length), as each button
is looked at, a small fragment of music is played to illustrate how it would sound with that option.
All this means that a teacher needs to be less involved in the minutiae of operating the system;
students can work at their own pace, and the teacher can work more as a guide or adviser on the musical
decisions to be taken. Switch-users in London for example, who had only been able to play simple tunes
prepared by a teacher, were able to have weekly composition lessons with a tutor, and produce a CD of
pieces they have composed themselves.
Figure 8. Leon operating E-Scape, with composition tutor
InFormation
14
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Figure 9. Leon's finished piece
The E-Scape eyegaze prototype in use
As described above, the relevant E-Scape features were already suitable in general ('eyegaze ready'), so
were able to be modified relatively easily for eyegaze users. The two key changes were that (i) menus
fill the screen to provide a grid of buttons which can then be selected by 'dwelling'; (ii) during editing
and playing operations in the score screen, the switch actions for 'next' and 'finish' are performed by
looking into the two areas at the bottom of the screen.
Before testing with real eyegaze users, we needed to make sure it worked well enough, so trials were
carried out by the designer and colleagues, using the 'camera mouse' application, which tracks the face
to move the cursor in a similar way to eyegaze. Tests confirmed that all the previously switch-controlled
musical operations could be carried out just by moving the cursor, ie by moving the head.
First user trials - Richard Bennett
During this time we also visited our first user, Richard, at the Royal Hospital for Neuro-disability,
Putney. This eyegaze music project actually came about because of him - the author received an enquiry
in 2013 from the hospital to see if we could help one of their 'locked in' patients, Richard 6, who was
very keen to learn and play music. He was happy to commission new work, if nothing was available.
Nothing was. Several months of development work ensued, as described above, and the first visit to
Richard was more about sorting out his equipment for music-making, and testing what he could do.
6
www.rhn.org.uk/_u/_files/pdf/RHNnews-aug-2013.pdf
InFormation
15
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Nevertheless, Richard was able to select items on screen and enter some notes, but concentrating on
the screen and where he was looking proved very tiring. His helper suggested it would be useful if she
had quick access using the eyegaze touch-screen - if she could see what Richard was trying to select,
she could press an item immediately for him without needing to wait for the dwell time. This facility
was added for the next session, and indeed proved most useful. This is a good example of user-centred
(or even in this case 'user-led'!) design. It is also useful for a teacher to be able to show a student
something quickly, using keyboard shortcuts, eg 'what it would sound like if we make this note higher,
or louder, or longer'. The principle should be that a specialist system should be as usable as possible for
all users.
But people all felt confident that the system was going to work for him, and a further session was
scheduled. So in March 2014 the first real trial of E-Scape eyegaze took place, and Richard was quite
quickly able to enter and edit a melody successfully. The prototype, backed by all the previous years of
developments, was deemed a success - it was usable, and feedback was very positive.
Figure 10. Richard entering music for the first time via E-Scape Eyegaze
By the end of the session Richard was very tired, but we mentioned that he could also play music live,
and would he like to try it? Suddenly Richard was less tired!
E-Scape has several ways of performing music live using the mouse, trackpad or touchscreen, and
as there is an option to 'not press the mouse button', it should also work straight away to perform live
using eyegaze. There is no 'dwell' function - you play something as soon as you look at it, which does
take some practice, but there are various options to help.
Richard tried two ways of playing. Firstly he tried playing phrases by looking at them, and when he
looked at a different phrase the previous one was set to cut off which made for a cleaner performance.
When we zoomed in enough so there were not too many phrases on screen, and had spaces between
InFormation
16
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
them, he was able to pick out and play different phrases quite well, allowing for having had no practice
at all.
Next, he tried playing individual notes by looking along a track. Any note or chord which crosses
the place he looks at will play, but will stop as he looks ‘off it’ (into some empty space). In both cases,
Richard's vertical eye position was ignored, giving him an easier task - only looking left to right.
A video https://www.youtube.com/watch?v=BE8BMa97mAY&feature=youtu.be shows Richard
trying out performing like this - remember this is his very first time. In this example, Richard is currently
playing the dark blue notes in the middle, by looking approximately in the centre of the screen.
Figure 11. E-Scape score being performed live by Richard using eyegaze
Richard was quite excited, and played and had fun for another 20 minutes; then he really was exhausted.
There are many further developments planned in the performing area, for example to let players
change track themselves - which changes which set of notes they are playing. One way would be to
dwell in the bottom right panel, to change to the next track down.
First user trials - Bram Harrison
Bram's aim was to compose his own jingles for his radio show7, and the BBC Performing Arts Fund
sponsored a project to find a system for him to compose music using eyegaze with an expert tutor /
technologist. Attempting to use his eyegaze system to operate a variety of mainstream music software
had not worked for him - it was just too difficult for him to control.
7
Bram Harrison http://www.phonic.fm/2014/superman-dj
InFormation
17
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Figure 12. Bram Harrison wants to write his own music links for his radio show
The author was asked to help at a late stage, and was able to provide the system developed for Richard,
along with improvements resulting from his initial trial feedback earlier in the year. In summer 2014, a
system was delivered to Bram, with his tutor setting up and reporting observations and ideas.
Despite some issues with his computer, when the system was working both Bram and the tutor felt
it to be a success, as he could actually work it to compose a number of short pieces, and was later able
to work on his own.
Figure13. Bram using eyegaze to operate E-Scape and write a melody
InFormation
18
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
In the video https://vimeo.com/112689731 we see how Bram navigates the menus to choose an
instrument and note length. He then enters notes and changes their pitch, creates chords, and plays back
the music he’s written so far. Seeing him working shows how physically demanding it is.
Figure 14. Film of Bram at an early stage, composing with E-Scape
Several features proved their worth in both sessions: The ability to 'customise' each menu proved
invaluable - some menus had too many items, and gave users too many choices, or simply made each
button too small to easily dwell on. The automatic auditioning and repeated procedures while editing
also proved most useful to reduce work.
The feedback and additional ideas from Bram's session - both from him and his tutor - were very
useful, and led to some additional improvements and features to improve usability. This is an example
where an expert in the area can provide useful ideas to the designer, in a way a normal user might not.
This feedback resulted in many improvements, but three major ones were 'auto-repeat' of operations, a
'pause dwell' button in menus, and splitting some menus into two pages.
Auto-repeat
When doing repeating editing operations where the music plays back each time something changes (such
as moving or transposing a note), users still needed to dwell on the bottom panel every time they wanted
to change by one step. So, for example, if they wanted to change by 7 steps, this added up to a lot of
effort.
The new 'auto-repeat' option makes such successive repeated actions happen automatically, ie there
is no need for users to dwell somewhere to make it keep happening. They can just 'rest' (ie can look at
most of the music screen without triggering anything), and listen to the music as it changes, only needing
to take action (ie look at the 'STOP' button') when they want to finish. Users are then presented with a
menu with options to start again, or go on or back one step.
InFormation
19
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Figure 15. Confirm or continue menu after stopping an operation
Other procedures, for example selecting a block of notes, can be made far less laborious in the same
way - users just have to wait while each note in turn is selected, and only act when they want to stop. It
is still time-consuming, but eyegaze users often do have time just not the energy.
Pause dwell
A standard feature of eyegaze AAC systems is a 'pause' button, which when triggered will de-activate
the 'dwell' function temporarily. All buttons then do nothing if looked at, apart from the 'unpause' button
– so the user can turn dwell back on. Pausing dwell enables the user to look round the grid of buttons
without triggering something to happen, and can also relax for a moment. This feature was already
planned for E-Scape phase 2 development, but its importance was highlighted by observing users
navigating a menu screen which gives musical feedback for each item.
One example is the menu to choose an instrument: as soon as an item is looked at, an excerpt of the
music the user is working on is played, using that instrument. But if the eyegaze user has a short dwell
time set, then there won't be time to hear all the music before the button activates. Thus the 'pause dwell'
button was very much needed, and consequently brought well up the priority list.
Split menus
Some menus needed all the choices to be shown, so could not be 'customised', but still had too many
items on them. They were therefore split into 'pages', with a button to swap between them. For example,
the menu for choosing an instrument had 16 groups of sounds, and thus 18 items on screen, which made
them too small to dwell on. This menu was therefore split into two banks which can be swapped between,
thereby making it usable.
InFormation
20
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Figure 16. Choosing an instrument category in 'bank A'
Future work
From observations of what users struggle with, and what they most commonly want to do, the aim in
future is to increase creative possibilities while further reducing laboriousness, for example: (i)
Designing better ways of selecting notes or positions within the score display, eg by dwelling in the
approximate position, then zooming in and dwelling more accurately, with the option of a final menu to
go backwards or forward a tiny step. (ii) Covering up irrelevant items on the score display to make it
clearer what users are currently doing. For example, when selecting notes in one track, other tracks could
be covered up and/or made smaller. (iii) Having a library of phrases or motifs, which can be auditioned
then selected to insert and edit. This means that users will have musical material already available to be
worked with, rather than having to start with single notes. (iv) Creating a 'compositional variations'
generator, which lets users create new sets of phases from an original one, according to simple rules
they can select or alter. (v) Increasing and refining the range of possibilities for performing using
eyegaze.
Examples of performing developments under way, or planned in this area include: Making it easier
to play through a melody from left to right, by ignoring any movement leftwards, until the user has
played (looked at) the last note; Enabling eyegaze users themselves to edit buttons in the performing
screen; Playing phrases by looking at them in any music track, not just in one track; Dwelling on a
phrase to 'cue it' while another one is playing, so the phrase will play as soon as the previous one finishes.
Two further improvements to general eyegaze functionality were also suggested by Rolltalk8 in
Oslo. Firstly, a circular 'dwell timer' display should show in the centre of each button, rather than a
moving bar. It would get smaller to show the passing of time, which helps users keep their eyes on a
button and not 'slip off'. Secondly, if a user's gaze slips into the next button for just an instant then comes
back again, then the dwell timer in the first button should resume from where it was (rather than starting
8
http://www.rolltalk.com/
InFormation
21
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
again). The system is thus recognising that the user's intention was to stay on the first button. But if the
user really does want to select the second button, and dwells on it for more than a certain (user-settable)
time, then it will become active, and its dwell timer jumps into action (to include the time already spent
there).
Conclusion
One of the main aims in designing this music system is to expand the range of students who can have
an 'instrument' to work with music. Thus, a key focus has always been to support exploratory learning
by enabling eyegaze users to operate it themselves. Several features in E-Scape eyegaze were designed
to assist in this, and although there is plenty of scope for further refinement, two features in particular
have proved their worth in trials so far. Firstly, user- guidance through certain procedures makes many
editing and note-entry procedures simpler and less laborious to carry out. Although students still need
to make decisions, the system understands the activity they are undertaking and presents a structured
pathway through appropriate sets of choices for that activity.
Secondly, automatic auditory feedback lets students immediately hear the audible result of changes they
have just made, and enables them to start working and composing almost immediately 'by ear', just by
listening and making choices by 'trial and error'. Thus, a beginner without any music theory knowledge
can explore and experiment with music, and achieve results and motivation right away.
Although dozens of further additions and improvements can be envisaged (and many are planned),
the people who have worked with it, and observed its use so far feel that the goal of unaided use has
been broadly achieved. Richard and Bram were able to start working and exploring very quickly, and a
student in Ireland is now using the system to compose music. Although short of time, Bram managed to
create some musical phrases to use - and was able to 'own' more of his radio show. Richard has now
engaged a regular tutor and developed his interest in film music; he can load pieces and explore them.
He is now planning to try more performing.
So the system can guide (via 'support scaffolding') beginner students to provide interest and
achievement. Although the ideal is always for students to learn more, the fact that they can start and
achieve something without such knowledge greatly helps motivation and enthusiasm. Thus, E-Scape
eyegaze seems to be an effective tool for what might be termed 'emancipatory learning', where the
disabled student is 'emancipated' from relying on a helper or teacher to do things for them. Students with
any degree of physical disability can thus operate and make decisions themselves within a guided
framework, without needing continual instructions to operate the system. The helper can join in, advise
or help on occasion, but they do not need to continually do everything to operate the student's music
system. However, the focus on students working and learning by doing in no way makes a teacher
redundant.
A teacher is still needed, as most eyegaze students will have little experience of operating computers
at anything more than a basic level, but more importantly won't have had the chance to 'play around'
with music at an earlier age (eg plucking a guitar, hitting a chime bar, touching piano keys, clapping or
singing) which non-disabled children usually enjoy. In addition, the act of composing is not intuitive the system can certainly help to guide users through some musical procedures, but students still need to
decide at a higher level what procedure they want to do. How do they create musical building blocks?
How do they then develop a piece from those? Students will need the help of a teacher to guide and set
up activities for them to explore, or work through.
InFormation
22
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
So, teachers are still very much wanted and needed, but they are able to focus on discussion of
musical concepts or tasks with the student, and also plan didactic activities. If creating a piece of new
music, the teacher is more able to take the role of guiding and suggesting, not 'telling' - there is less of a
'right answer' than if, say, doing a theory test. What the teacher does not need to do is operate the
computer; the student is more in change of that themself; they are in control, and can (if they wish!)
ignore the teacher's suggestions. This has been observed quite frequently, with the disabled student
making a certain musical decision 'against advice', often with interesting results. The student is far more
'master of their own destiny' and has a demonstrable pride and ownership of their artistic creations.
E-Scape eyegaze has the advantage that it is effectively a 'new instrument', with less tradition of
'conventional' teaching methods. So what the teacher can and - according to the learning theories
adduced earlier - should do, is customise the explorative learning environment to engender optimum
learning outcomes. Again, E-Scape eyegaze has two key features which facilitate this.
Firstly, a teacher is able to tailor the 'exploratory framework' used by students, by selecting or
limiting the options available to them when they are working. This lets students explore and make their
own choices, but makes them operate within an area that the teacher has planned. As students progress,
the available options can be changed in various ways, eg to focus the activities on a different topic, or
to expand the range of exploration possible.
Secondly, a teacher can customise the 'support scaffolding' provided to the student, by selectively
or progressively removing some of the guidance features. This may involve removing some or all of the
'guided' options from menus leaving only the more basic operations. With less guidance on offer,
students then have to think harder about what they are doing, and in what order - but conversely have
more freedom. The teacher can also change other support options, for example to use more music
terminology, or alter some procedures to move the student gradually away from 'trial by ear' to more
considered choices from menu selections using musical knowledge gained.
Thus, the system can hopefully enable students to progress in learning as they work or explore, while
still achieving creative results. But the fact that the system can be used without help means there are
opportunities for the student to be given homework, or practice something out of lesson time, or work
with friends to make music. The student can play and contribute on more equal terms with their peers,
or could even support them. As was stated at the beginning, this kind of interaction is a key part of both
music making, and learning.
So, it is to be hoped that the development of access to music-making via eye-control can open up
possibilities for people presently living their lives on the margins of society. They need to be able to
participate in music education in a way that they can truly take part, and in due course create and present
their own music. Recently, there are signs that eye-control systems are being investigated by a wider
range of people who want to try new ways of interacting with computers, so eyegaze users may end up
guiding other people, and at the least working and playing with them as peers.
It is exciting to experience the musical expression of people whose voices are rarely heard, and
might change the way we look at people who have hitherto had few possibilities to take part in music or
express themselves creatively.
InFormation
23
Volume 4, No 2 (2015)
T. Anderson: “In from the margins”
Author Presentation
For the last 25 years, Dr Tim Anderson has been focused on developing music systems which are
accessible for disabled people. He is now an independent consultant, working with schools, councils and
(since 2004) the SKUG centre in Tromsø, Norway. Activities include creating systems and teacher
resources, training, and creating or adapting music software and sensors. In the 90s he started developing
the E-Scape switch-operated composition and performance system for teachers and disabled users. He
also adapts mainstream music software for disabled use, first seen on the RTE TV documentary 'In From
The Margins'. In 2013, Anderson was requested by The Royal Hospital for Neuro-disability, London to
add integrated eyegaze control, and after development this system is now also in use in Ireland and SW
England.
References
Anderson, T. (2002). Composing and Performing with switches, using specialised or adapted music
software. Proc. International Conference on Assistive Technology, 2002.
Barnes, C. (1992). Qualitative Research: valuable or irrelevant. Disability, Handicap and Society. 7
(2).
Damon, W. (1984). Peer education: The untapped potential. In Journal of Applied Developmental
Psychology. 5 (4).
Daniels, H. (1995). Pedagogic practstinices, tacit knowledge and discursive discrimination: Bernstein
and post-Vygotskian research. In British Journal of Sociology of Education, 1995.
Dewey, J. (2010). Democracy and Education. Boston: MobileReference.com.
Dewey, J. and McClellan, J. (1889). Applied psychology:An Introduction to the Principles and
Practice of Education. Boston: Educational Publishing Company.
Eisner, E. W. (2001). The Role of the Arts in Educating the Whole Child. GIA Reader. 12 (3).
Eisner, E. W. (2002). What can education learn from the Arts about the practice of education?. The
John Dewey Lecture for 2002, Stanford University.
http://www.infed.org/biblio/eisner_arts_and_the_practice_of_education.htm
Hammond, J. and Gibbons, P. (2001). What is scaffolding?. In Scaffolding: Teaching and learning in
language and literacy education. Sydney: Primary English Teaching Association.
Imsen, G. (2005). Pupils' world: introduction to educational psychology. Oslo: University Press.
Karpov, Y. and Haywood, H. (1998). Two ways to elaborate Vygotsky's concept of mediation.
American Psychologist. 53 (1).
Kitson, L. (2011). User-Led Does Not Equal User-Centered. UX Magazine. Article No. 634.
https://uxmag.com/articles/user-led-does-not-equal-user-centered
Oliver, M. and Zarb, G. (1992): Personal Assistance Schemes in Greenwich: An Evaluation London:
University of Greenwich.
Oliver, M. (1997) Emancipatory Research: Realistic goal or impossible dream?. In Doing Disability
Research. Leeds: The Disability Press.
Rasmussen, J. (2001). The importance of communication in teaching: A systems-theory approach to
the scaffolding metaphor. Journal of Curriculum Studies. 33 (5).
Ruud, E. (1996). Music and values: essays on music pedagogy. Oslo: University Press.
Skogdal, E. (2014a). Inclusive music education - from cradle to professional. In Inclusive practice good experiences from kindergarten, school and leisure. Oslo: University Press.
Skogdal, S. (2014b). Inclusion and participation for all. In Inclusive practice - good experiences from
kindergarten, school and leisure. Oslo: University Press.
Stone, C. (1998). The Metaphor of Scaffolding: Its Utility for the Field of Learning Disabilities.
Journal of Learning Disabilities. 31 (4).
Svendsen, L. (2014). A free and good life: About Duchenne Muscular Dystrophy. In Practice: Good
experiences from kindergarten, school and leisure. Oslo: University Press.
Vygotsky, L., Cole, M., John-Steiner, V., Scribner, S. and Souberman, E. (1978). Mind in society: the
development of higher psychological processes. Cambridge, Mass: Harvard University Press.
InFormation
24
Volume 4, No 2 (2015)