ht. J. Bio-Medical Computing, 17 (1985) 177-184
Elsevier Scientific Publishers Ireland Ltd.
177
AN INTERACTIVEAID FOR EXPRESSIVECOMMUNICATION
WITH
PICTOCRAPHICSYMBOLS
M.C. FAIRHURST and C. STEPHANIDIS
Electronics Laboratories, Universityof Kent, Canterbury, Kent CT2 7NT (U.K.)
(Received May 2nd 1985)
An individual who is unable to acquire skills in natural language communication may frequently
find a pictographic symbolic language to be an appropriate alternative means of expression. This
paper discusses some underlying features of the communication process and describes a computerbased augmentative aid for communication with pictographic linguistic units, which provides flexibility for user-definition of symbols, inter-symbol mapping and multi-mode expression. The system
is consequently a potentially very powerful tool for the development and utilisation of communication processes among groups of users who do not necessarily share access to a common linguistic
knowledge base.
1. Introduction
As part of a programme for the development of a system to facilitate symbol communication, work is in progress on an integrated approach to the computer manipulation of pictographic languages. This paper will consider the underlying features and
implementation
of a system which allows symbol definition by a user, facilitates communication through the ‘transmission’ (e.g. display) of defined symbols, and enables
translation of symbolic forms in a variety of ways.
A particular feature of the system is that the flexibility offered at the user-computer
interface provides the potential for a very rich pattern of inter-personal interaction,
such as in the mapping from a locally-defined or otherwise constrained vocabulary of
linguistic elements to a more universal basis of expression.
2. A Descriptive Notation for Communication Ptocesses
In order to describe the mechanisms of the communication
process, particularly in
relation to the use of communication
aids, it is useful to develop a notation which,
although not comprehensive, may be helpful in identifying important areas for discussion.
Let {A} be a set of activities available to a particular
communicator.
0020-7101/85/$03.30
0 1985 Elsevier Scientific Publishers Ireland Ltd.
Printed and Published in Ireland
The set would
MC. Fairhurst, C, Stephanidis
178
generally be highly personal and would range from direct vocalisation to operation
of the simplest switch.
Let {C} be a set of ‘communication
units’. It is not necessary to define this term too
precisely at this stage, except to say that a communication unit could be represented
of a
by a simple word label (e.g. ‘home’, ‘dog’, ‘help’) or could a representation
rather broader concept.
Let {S} be a set of symbols used for communication
which, as will be seen, could be
spoken, written, or expressed through some other medium.
It is assumed that any set {X} has elements
{XtXa..
. . .Xi
. . . . .}
Inter-personal
communication
operates on the assumption that, in general, it is
possible to form an association between an element from the set {S} and an element
from the set {C} though this may require some explicit teaching/learning experience.
This process may be described as:
or, in general
where {X*} implies a subset of {X}
The process of communication
consists, in an elementary descriptive form, of selecting an action from the set {A} to convey a concept ‘from {C} by means of activating
or ‘transmitting’ a symbol from {S}
Here, in the notation given,
or; more generally,
{A*} + {S”} --f {C*}
Two further points should be noted. Firstly, it will normally be necessary to develop
the relationship
Pictogruphic symbols
119
Secondly, it is important to realise that an element from {C) may be communicated
by symbols from more than one {S} set ({S’), {S”}. . . .) corresponding to different
modes of expression (e.g. spoken, written, etc.). This gives rise to processes such as
and so on.
A set {S} may exist for many different modes of expression, and the mode available to an individual, in general, determines the nature of the set {A). For example, if
some manual sign language is used (Stokoe, 1972; Walker, 1980) then the set {A} is
rich and extensive in terms of motor functioning, while a natural speech medium would
be even richer, but generating activities at an entirely different level.
3. Communication and Communication Aids
Problems of communication
may conveniently be analysed within the descriptive
framework developed by Kylen (1982), who breaks down the communication
process into a series of stages, allowing the identification
of specific areas of difficulty
which can often be localised to a small number of links in the chain. One particularly
significant problem area for the language dysfunctioning
communicator
lies in the
choice of an appropriate expressive medium and this, in turn, is to a large extent dictated by the nature of the set {A} for the individual. Manual signing, for example, is
clearly inappropriate for an individual with poor control over limb movement.
A common strategy inherent in the design of many currently available communication aids is to try to match the set {A} for an individual to an apparently inappropriate
set of symbols {S} by providing a linking or matching interface. The MAC system
(Peon, 1982), for example, seeks to provide access to regular natural language symbols
through the operation of a simple and restricted-action
switching device used as a
menu-selector.
For many individuals a pictographic symbolic communication language {S} is appropriate, and an example of a commonly encountered
system is Blissymbolics (Bliss,
1965). Previous work (Fairhurst and Bonaventura,
1983; Fairhurst and Stephanidis,
1984; Fairhurst and Maynard, 1979) has concentrated on the specification of a system
intended to promote acquisition of pictographic symbols and, more specifically, which
has been designed to allow mappings of the form
where (A) is a set of activities private to an individual
and for which no {A} * {S} link
180
MC. Fairhurst, C. Stephanidis
exists, taking {S} to be a symbol set of ‘universal’ or semi-universal significance. The
II indicates the intervention of the aid in the expressive chain.
This paper will describe the design and implementation
of a system which operates in a mode employing pictographic symbolic communication
for an individual for
whom the set {A} is very restricted. This corresponds to a typical situation where, for
example, a Bliss-like language is in use, where the communicator
simply points to a
symbol on a display board or accesses a pictographic communication
unit in some
other simple way. The principal aims of the system are the following:
(1) To facilitate activities of the form
and hence
(2) To provide sufficient flexibility for the unrestricted definition of a symbol
vocabulary suitable for a range of purposes.
(3) To provide a means of mapping from one symbol set to another. This requires
some discussion, but can be described as
(4) To provide, at least in a rudimentary way, the potential
translation between communication
modes, such that
for some restricted
{S’)I-+ w
It is helpful to indicate some typical activities which illustrate
areas addressed by the implementation
of the system.
4. Requirements for a Symbolic Communication
a range of problem
Aid
This discussion will draw a distinction between ‘universal’ and ‘private’ symbols,
and it is necessary to clarify these terms. A ‘universal’ symbol will be taken to represent
a standard&d
symbol for some widely agreed symbol set semantically defined according to an accepted convention. The Blissymbolic language, for example, conforms to
this difinition. Conversely, a ‘private’ symbol is one which has an accepted semantic
connotation
among a specific community of users on a purely local basis. The user
community could be defined in terms of a single child-parent or child-teacher relationship or, more commonly, might include a significant number of symbol users based at
a particular centre or institution.
Fictographic symbols
181
Communication
which involves pictographically
symbolic elements can range from
a simple “message selection” protocol, where a user selects a symbol indicative of
some particular message from a restricted range of possibilities, to activities involving
the manipulation
of very sophisticated and linguistically subtle communication
units
such as is required by Blissymbolics.
It is useful to consider some specific examples of possible communication
vocabularies (see Fig. 1) which illustrate the variability in complexity of ideas conveyed in
situations which might be envisaged.
Figure 1A illustrates a symbolic ‘message’ selection board which uses arbitrarily
defined (perhaps typically a mixture of private and universal) symbols. Symbols such
as these might represent, as a single communication
unit, a relatively complex idea
relating to a restricted range of available, and generally quite fundamental, activities.
Figure 1B shows a (subset of) a rather richer pattern of pictographic symbols
(essentially private in nature) which could perhaps have been found effective to convey
a range of expressions(needs,
feelings, ideas, etc.) and which might generally be expected
to be iconographic in nature. Such a set of symbols might evolve in the home environment or in a local community where a need for symbolic expression arises, but where
a standardised language (such as Blissymbolics) is unavailable or where the subtleties
of such a language preclude access by the potential user group.
Figure IC shows a more familiar subset of a universal symbolic communication
language (Blissymbolics in this case). It is assumed that, in all cases, communication
consists in pointing to (or selecting in some other simple way) one - or a string of symbol(s) to convey the desired message.
An aid for expressive communication
which could accommodate a range of communication protocols including those discussed in relation to Fig. 1 would have (at least)
two principal prerequisites. First of all, it would clearly need to provide an appropriate
methodology of interaction such that ‘information transmission’ is facilitated, that is,
a pictographic symbol is selected and the corresponding information it represents
conveyed. Secondly, it is necessary that the symbols to be used are user-definable,
such that the constraints in the symbolic forms adopted are minimised. The following
section describes the implementation
of a system which meets these requirements and,
in addition, provides a framework for both inter-symbol and inter-mode translation.
5. A Prototype System for Symbolic Communication
The system is implemented on a BBC microcomputer
linked to a single disc drive,
a STAR Microterminals
Concept Keyboard, and a speech synthesiser unit (this is
optional, and various types are under consideration, though the principle of operation
is independent of precise implementation).
For development purposes the system has
also incorporated a 6502 second processor, though sideways RAM/ROM is an alternative in an ‘on-site’ version.
The system operates in one of two modes, corresponding to SYMBOL COMMUNICATION (the system functioning as a communication
medium), or SYMBOL CREA-
182
MC. Fairhurst, C. Stephanidis
SYMBOL
In_-..f
MEANING
t
10 Y
cl
(A)
E
CHANGE
I 'M
MY POSITION
HUNGRY
PLAY
TAKE ME
MUSIC
HOME
BASIC MESSAGE SELECTION
TREE
t-t
(B)
A "PRIVATE" SYMBOL SET
I
HAPPY
I
I
J
I
1
it) A "UNIVERSAL" SYMBOL SET
Fig. 1. Examples of pictographic symbol sets
Pictographic symbols
183
TION (where a teacher creates interactively a set of pictographic symbols and intersymbol/inter-mode
transformations
appropriate for a specific user).
(a) Symbol communication
In this mode the user may select one of twenty-five possible symbol libraries (either
provided by the system designer or user-generated), each containing up to one hundred
symbols. Each symbol is defined as a two-dimensional pictographic representation and
is associated with two alternative modes of expression - an English text equivalent
and, if desired, by a synthesised spoken utterance.
Each library is associated with a Concept Keyboard overlay, and communication
occurs by means of touch-activating
a symbol on the overlay, which is then added to
a displayed symbol string on the VDU screen, with simultaneous generation of the
alternative mode expressions. For display, the screen is divided into four graphics
windows to accommodate
the symbols and associated text, together with one text
window below to display instructions to the user where necessary. Symbols are written
within the defined window.
(b) Symbol creation
In this highly interactive mode, the user is able to perform three functions: (i) Specify the ‘form of the pictogram which defines a required symbol. (ii) Provide an English
text string to be associated with a specified symbol. (iii) Construct a phonetic representation of a required spoken utterance (to be generated by the speech synthesiser)
for association with the visual symbol.
A particularly important feature of this mode is that, specifically in respect of functions (i) and (ii), full visual and aural feedback is available to enable highly interactive
real-time modifications and adjustments to be made to the specified visual/vocal forms,
so as to optimise the representation subsequently stored for future use.
The pictograms are constructed by combining primitives (elemental graphical units)
to form a composite whole. The system provides a comprehensive set of primitives
(identified from a Concept Keyboard overlay) which can be called up and positioned
anywhere within a graphics window under control of four cursor keys. In this way,
two-dimensional
pictograms of any complexity may be formed. Associated text may
likewise be positioned within the display window, and the spoken representation
is
constructed
(again interactively)
by phoneme combination.
Rudimentary
editing
functions are provided and unlimited interaction is allowed until a satisfactory result
is achieved, whereupon the symbol and associated translated modes are stored for
subsequent use.
It is clear that a judicious choice of pictograms on the keyboard overlay, pictograms
displayed on the screen, text (natural language expression) and vocally articulated
representation,
allow the flexibility to achieve the range of objectives defined in $4.
The system is user-oriented and completely flexible and requires no special expertise in computing on the part of either teacher or communicator,
as extensive prompts
184
M.C. Fairhurst, C. Stephanidis
provide most of the instructions
required for rapid acquisition of operating skills.
Alternative forms of symbol definition mechanisms are currently under consideration,
but would be accommodated within the general operational framework described.
6. Conclusions
This paper has described an analytical framework within which the processes of
communication
may be characterised. The discussion suggests some broad guidelines
for the design of augmentative aids for expressive communication
and has lead to the
implementation
of a prototype system for pictographic symbol communication, with an
emphasis on flexibility to allow inter-symbol and inter-mode mapping.
Such a system is primarily intended to facilitate communication
among users who
may not have access to a common linguistic knowledge base, but also has the potential
for application as an aid for teaching the acquisition of a variety of linguistic skills.
References
Stokoe, W.C., 1912, Semiotics and human sign languages, Mouton
Walker, M., 1980, Revised Makaton Vocabulary, Makaton Vocabulary Development fioject, 3rd
edn.
Kylen, G., 1982, Non-verbal communication and mental handicaps, Proc. DISTECH 82, Brighton,
(Sponsored by Spastics Society).
Poon, P., 1982, MAC- a writetyper, ACE, VoL 1, No. 3, Special Education Edition.
Bliss, C.K., 1965, Semantography - Blissymbolics, Semantography Publications.
Fairhurst, M.C. and Bonaventura, M.P., 1983, A user-oriented design for a microprocessor-based
symbol communication aid, in The Computer as an Aid for those with Special Needs, Sheffield.
Fairhurst, MC. and Stephanidis, C., 1984, User-computer interaction in hand-eye coordination
tasks, in The Computer as an Aid for those with Special Needs, Sheffield.
Fairhurst, M.C. and Maynard, C.A., 1979, Application of pattern recognition methodology to the
design of an electronic aid for coordination therapy, J. Biomed. Eng., 1,257-262.