Foundations of representation: Where do graphical symbol systems come from?
Simon Garrod1, Nicolas Fay2, John Lee3, Jon Oberlander3 & Tracy MacLeod1
1
University of Glasgow, Scotland
University of Western Australia, Australia
3
University of Edinburgh, Scotland
2
Keywords: Graphical representation, Symbols, Icons, Signs, Communication
Corresponding author:
Professor Simon Garrod
Department of Psychology
University of Glasgow
56, Hillhead Street
Glasgow, G12 8QT
United Kingdom
Email: [email protected]
Tel: +44 141 3305033
Fax: +44 141 3304606
-1-
Abstract
It has been suggested that iconic graphical signs evolve into symbolic graphical signs
through repeated usage. We report a series of interactive graphical communication
experiments using a ‘pictionary’ task to establish the conditions under which the
evolution occurs. Experiment 1 rules out a simple repetition based account in favour of
an account that requires feedback and interaction between communicators. Experiment 2
shows how the degree of interaction affects the evolution of signs according a process of
grounding. Experiment 3 confirms the prediction that those not involved directly in the
interaction have trouble interpreting the graphical signs produced in Experiment 1. On
the basis of these results we argue that icons evolve into symbols as a consequence of the
systematic shift in the locus of information from the sign to the users’ memory of the
sign’s usage supported by an interactive grounding process.
-2-
“Symbols grow. They come into being by development out of other signs, particularly
from icons, or from mixed signs partaking of the nature of icons and symbols” Peirce,
1893.
1 Introduction
From the cave paintings of Lascau to the earliest pictographic writing systems of the
ancient Egyptians, early human settlements are marked by the use of simple graphical
representations. Even today, graphical representation plays an important part in various
sign systems, such as road traffic signs or the signs on a computer interface. In some
cases they serve as direct iconic representations of objects (e.g., the trash can icon on
many computer interfaces), but in other cases they serve as more abstract symbolic
representations (e.g., the red triangle motif used in traffic signs to indicate warning). Like
Pierce some argue that symbolic graphical representations evolved from earlier iconic
representations. For example, the Assyrian symbolic writing system evolved from the
iconic pictographic system of early Cuneiform via early Babylonian (Tversky, 1995). A
similar observation can be made about the evolution of Chinese characters (See Figure 1).
It has also been suggested that modern symbolic traffic signs incorporating the red
triangle motif evolved from an earlier traffic sign that indicated danger (Krampen, 1983).
Yet little is known about how or why icons evolve into symbols. This paper addresses the
question of how graphical signs become symbolic by investigating the conditions under
which graphical representations evolve during the course of a graphical communication
task.
------- Insert Figure 1 about here --------
-3-
Consider the following experimental set up. Pairs of participants are asked to
communicate concepts to each other using only graphical means. The set up is similar to
the popular parlour game ‘Pictionary’, except that the concepts are drawn from a fixed
list which is known to both participants. One participant (the matcher) has to identify the
concept drawn on a whiteboard by the other participant (the director) from a set of 16
alternatives. These include groups of easily confused concepts such as: art gallery,
museum, parliament, theatre; Robert de Niro, Clint Eastwood, Arnold Schwarzenegger.
The participants are able to make some minimal assumptions about each other’s shared
cultural background (e.g., they might expect the other to know that Schwarzenegger has
large muscles), but have relatively few shared resources for using the communication
system. During the experiment, they carry out the task through a number of blocks, each
consisting of twelve of the same 16 items, chosen arbitrarily, so the items regularly recur,
though at random intervals.
We will show that when participants carry out this task under most but not all conditions,
three things happen to their drawings as illustrated by the sequence in Figure 2. As the
game proceeds from block to block the drawings change: (1) they become graphically
simpler (e.g., they contain fewer lines, fine details are lost etc.); (2) they become more
schematic or abstract as representations of the concepts they stand for; and (3) when
director’s alternate with each other their drawings increasingly converge with those of
their partner.
-----insert Figure 2 about here -----
Block 1
Block 2
-4-
Block 3
Block 4
Block 5
Block 6
What proves to be critical for the development of the graphical representations in these
three ways is the degree to which the participants can interact graphically with each other
during the experiment. Our basic argument will be that interactive graphical
communication enables participants to develop symbolic representations from what
started out as primarily iconic representations through a “grounding process” similar to
that proposed for interactive spoken communication (Clark & Brennan, 1991).
The experiments we report use a communication task because signs are essentially
communicative devices. However, understanding how graphical signs represent objects is
fundamental to understanding the nature of graphical symbol systems as cognitive
devices (Barsalou, 1999) and how graphical representations develop has a bearing on the
evolution of conceptual systems in general (DeLoache, 2004; Karmiloff-Smith, 1990).
First, we develop a framework for analyzing different kinds of graphical representation
based on C.S.Peirce’s account of signs, using ideas from information theory. Then we
review what is known about the development during early childhood of the use and
interpretation of graphical representations. This leads to our main hypotheses: first, that
communicative interaction is vital for iconic graphical representations to evolve into
symbolic representations and second, that the evolution involves a shift in the locus of the
information from the graphical sign to the mental representation required for successful
interaction. In particular we will argue that during the evolution from iconic to symbolic
graphical representation structural complexity migrates from the sign to memory
representations in sign users. The remainder of the paper reports three graphical
-5-
communication experiments which test these predictions.
2 Classifying representations as icons, indices and symbols
According to Peirce, a sign is a relation among three elements: the sign, the object being
signified and what he called the interpretant or understanding of the sign (Peirce, 193158; see also Greenlee, 1973; Krampen, 1983). For example, the trash icon on a computer
screen is a sign that relates to an object (i.e., the memory deletion buffer of the computer
system) and has an interpretant (i.e., the understanding that it represents where to drag
and drop another icon in order to delete that file). For Peirce it is the presence of all three
elements that defines signification. But he distinguished between three kinds of sign,
which he called icon, index and symbol. The three kinds of sign differ in terms of how the
sign relates to its object. Icons exist by virtue of a salient perceptual/structural
resemblance between sign and object (e.g., the picture of the trash can in the computer
icon resembles a trash can itself). Most pictorial representations are therefore icons,
because they usually reflect the appearance of the signified object. By contrast, indices
relate to their objects in a causal way. For example, the level of mercury in a barometer is
an index of atmospheric pressure; it does not resemble atmospheric pressure but it is
causally linked to it. Similarly, smoke is an index of fire. Finally, symbols relate to their
objects, neither in terms of resemblance nor causality, but purely on the basis of habit or
convention. Thus, a symbol has to have been habitually or conventionally used to signify
its object in order to operate as a sign for that object. Unlike icons and indices, symbols
only bear an arbitrary relation to their objects. For example, both the words dog and
chien act as symbols for canine creatures, but there is neither a resemblance between the
sounds of the words and the sounds made by canine creatures nor is there any
relationship between the sounds in the two words themselves.
Peirce’s distinction between three kinds of sign is intuitively appealing. However, when it
comes to classifying graphical productions of the sort shown in Figure 2, it often seems
that the distinction between iconic and symbolic graphical signs is not binary, but rather a
matter of degree. For example, the depiction of ‘cartoon’ on the left of Figure 2 is clearly
-6-
iconic because it resembles a cartoon character, as does the depiction in the middle.
However, the later depiction on the right of Figure 2 has become so schematic that it no
longer resembles the cartoon character and so becomes more a symbol for ‘cartoon’. In
other words, its interpretation now depends to a greater degree on the fact that the part of
the drawing (i.e., the part resembling the top of the cartoon character’s ears) has been
habitually used to signify its object ‘cartoon’ over the course of the “pictionary” task and
so is now sufficient to bring the object to the interpreter’s mind. Nevertheless the
depiction still retains something of its original iconicity because it is still possible to see
how it resembles a part of the cartoon character. So, like Krampen (1983) and to a certain
extent Peirce (1893) himself we treat the distinction between iconic and symbolic
representations as graded. Similar arguments can be applied to the distinction between
index and icon. For example, a photograph is usually taken to be an index of its subject,
because the pattern of pixels on the photographic paper is caused by the light reflecting
from the subject. However, the photo is also iconic to the extent that it resembles the
subject. So we shall treat signs as exhibiting different degrees of iconicity, symbolicity
and indexicality.
One way of characterising the difference between iconicity, symbolicity and indexicality
is in terms of the knowledge required to produce or interpret signs. Because icons signify
through resemblance to their object they only depend upon perceptual knowledge. By
contrast, indices depend on world knowledge about the causal chain that relates the index
to its object. Finally, symbols only depend on communicative associations between signs
and their objects. Now, if it is true that a given sign may be simultaneously iconic,
indexical and/or symbolic for a given individual, it follows that various kinds of
knowledge about a sign—perceptual, causal, communicative —can co-exist.
We can also characterise the difference between icons and symbols in information
theoretic terms by looking at how information is distributed between the sign itself and
the user’s knowledge of how the sign has been used in the past. Because iconic signs
depend on a structure mapping between sign and object they carry information in their
graphical structure. If a sign for an object maps some of the structure of the object, then
-7-
we can see that the structure in the sign is carrying information. And this sign is
functioning iconically. Other things being equal the more structured the sign, the more
iconic the sign is. On the other hand, when symbols are used complexity (or information)
resides in the users’ knowledge of previous uses of the symbol. To this extent a symbol
only requires enough structure to be recognisable as that symbol distinct from any other.
Hence, symbols can become graphically simpler than icons.
We can now express key differences between icons and symbols of the kind seen in the
‘pictionary’ task. Over an extended sequence of interactions, we hypothesise there is a
shift of the locus of information in the interaction from the sign itself to the
communicators’ representations of the sign’s usage. If there is no structure in the
interlocutors’ representations of how the sign is used, then all the information must be
carried by the sign. But as structure builds up elsewhere, then signs can become less
complex.
At the beginning of the ‘pictionary’ experiment communicators need to be able to
recognise the object from the sign and they can only do this through the visual
resemblance between sign and object. So the sign functions iconically. Sometime later in
the interaction, the same concept needs to be communicated. Now, a much-reduced
drawing serves to secure recognition. What differs is the shared experience of the
interaction that the participants now possess. In particular, recognition of a sign will
involve checking the current sign against a memory representation built up from a variety
of previous recognition events, classified as involving variants of the same sign. To the
extent that resemblance matters at all, it is resemblance between sign and sign, not
between sign and object. Depending on the precise conditions of interaction, particular
aspects or features of the original iconic drawing will have been accentuated and others
diminished or removed. Thus, a very simple sign can be used very precisely to
communicate something very specific and at any level of complexity. Under such
circumstances the sign is functioning symbolically. The more structured the interaction
history behind the sign, the more symbolic it can become.
-8-
Notice that this means that a sign can be simultaneously iconic and symbolic, if the sign
contains much structure, and so does the interaction history. But, while redundancy in a
communication system is useful (particularly in noisy conditions), efficiency
considerations will tend to replace one kind of structure with another. The iconic to
symbolic transition thus involves a transfer of structure (or complexity) away from the
sign.
We predict that indices should undergo a similar transformation. Whereas an icon
visually resembles its object, an index is only causally related to its object; but there still
has to be a visual representation of the index, and to begin with this will represent
features of the index iconically. For example, to signify ‘fire’ with the index ‘smoke’ we
need an iconic depiction of smoke. Via interaction, the iconic component of any indexical
sign should become simpler and thereby increasingly symbolic. In what follows, we will
rarely distinguish such index-based icons from icons in general. It is worth noting,
however, that Deacon (1997) argues for a reinterpretation of index, which covers cases
we might consider genuinely symbolic. For now, we note that we do not follow Deacon’s
usage, but we will return to discuss relations between our account and his in the final
section of this paper.
To sum up, the Peircean notions of icon, symbol and index are useful tools in
approaching the topic of representation in general, and graphical representation in
particular. Thinking about where structure (and hence, information) lies within an
interaction helps us see how signs can function in more than one way at once, and that a
sign may function one way at one time, and another way later on. We have suggested that
if an interaction is structured in some way, the burden of information-carrying will shift
from individual signs into history. As a consequence what began as a graphically
complex icon can evolve into a graphically simpler and more abstract symbol. But if this
does happen in the way we have supposed, what mechanism underlies it? Which kinds of
interaction encourage the evolution of icons and indices into symbols? The experiments
we report later directly address these questions. Before reporting those experiments we
-9-
give a brief review of the psychological literature on graphical representation which has
mainly been concerned with developmental issues.
2. Development of graphical representations
Research on the development of graphical representations has addressed 3 main
questions. First, the perceptual question of how an infant comes to recognize what a
picture is of; second, the cognitive question of when the infant starts to treat a picture as a
distinct representation of its object, and third, the communicative question of when the
child starts to be able to treat graphical representations (in the form of drawings) as
conveying information about objects even when there is relatively little resemblance
between drawing and object.
In relation to the perceptual question, dishabituation studies confirm that infants as young
as 3 months old have no trouble recognizing a photograph of their mother as such
(Barrera & Maurer, 1981). This and other related findings indicate that recognizing the
perceptual relationship between a two-dimensional picture and its object does not seem to
have to be learned, but rather is built into the perceptual system(Ittelson, 1996). In fact,
slightly older infants (around 9 months) will sometimes manipulate and explore a
photograph as if it were the object itself (DeLoache, Pierroutsakos, Uttal, Rosengren, &
Gottlieb, 1998). This last finding suggests that initially infants do not treat pictures as
distinct representations (i.e., signs in Peirce’s sense), rather they treat them as an
alternative means of perceiving objects. The cognitive ability to appreciate that a picture
is a distinct representation only emerges later at around 18 months when children respond
by looking for and pointing to the pictured object rather than interacting with the picture
itself (Pierroutsakos & DeLoache, 2003). De Loache (2004)attributes the older child’s
ability to treat graphical depictions as sign-like representations to their emerging capacity
for dual representation. In other words, it depends on the child’s being able to maintain
two separate representations one for the picture and another for its object. She argues that
this explains why young children (2-3 year olds) have trouble using a picture or physical
model to interact with the world depicted if the model itself is made too salient. The more
- 10 -
that the child interacts with the model the less they are able to use it to interpret what is
going on in the situation. Zaitchik (1990) also found that 4 year olds often treat pictures
as if they are causally related to their objects, imagining that changes in the object will be
reflected in changes in a previously recorded photograph of the object. Interestingly, this
last finding suggests that the preschoolers sometimes treat photographs as indexical
rather than iconic signs.
However, the research reviewed so far does not address how or why abstract symbolic
graphical representations emerge. As we argued above iconic representations differ from
symbolic representations in terms of how much they resemble their objects. Bloom and
Markson (1998) investigated the extent to which a child’s understanding of a drawing
depends upon the degree to which it resembles its object. They had 3-4 year olds draw
pictures of confusable objects (e.g., a lollipop and a balloon) and then required them to
name the drawings after a delay. Despite the fact that the drawings were quite insufficient
to identify the objects on the basis of their likenesses (i.e., the pictures were easily
confusable) both the 3 and the 4 year olds were surprisingly good at naming the pictures
correctly. Furthermore, when the experimenter subsequently queried the children they
were adamant that the picture was of the object that they had intended to draw rather than
the alternative object that the picture also resembled. Bloom and Markson concluded that
the children (particularly the 4 year olds) considered drawings as representations of what
they were originally intended to depict rather than representations of what the drawing
happened to resemble. This experiment does not directly address what underlies this
more symbolic understanding of pictures, but together with the findings reviewed in the
previous paragraph indicates the increasing development during the preschool years of an
understanding of depictions as distinct objects intended to represent other objects – in
other words as Piercian signs.
A clue as to what might drive the development of this representational ability comes from
a study by Callaghan(1999). She had 2-4 year olds first draw simple objects (e.g., a plain
ball and a spiked ball) in a free drawing task and then subsequently draw them again as
part of a simple communication game with the experimenter. She compared the two sets
- 11 -
of drawings in terms of how many features of the objects each drawing represented and
found that the older children’s (3-4 years) drawings improved markedly when they were
employed as part of the communication task. A subsequent experiment ruled out practice
as an explanation for the improvement and showed that negative feedback (i.e., the
experimenter indicating failure to understand the drawing) also produced more effective
drawings from the older children. Although this study does not address development of
the use of symbolic as opposed to iconic graphical representation it does underline the
importance of the communicative context and communicative feedback in refining the
older children’s use of drawings as signs.
In the developmental literature graphical representations are treated as symbols whether
or not the representation resembles its object (See e.g., DeLoache, 2004). But this
obscures the important question raised at the beginning of the paper of where abstract
symbolic graphical representations come from. The interpretation of iconic
representations is straightforward because of the perceptual resemblance between
representation and object. However, this is not the case for symbolic representations. If
symbolic representations evolve from their iconic or indexical precursors understanding
this process of evolution would give a straightforward explanation of how symbols arise.
3. Hypotheses about how iconic and indexical representations evolve into
symbols
In the Introduction, we made some preliminary observations about how the drawings in
the ‘pictionary’ task change when repeated over the course of the experiment. Basically,
they seem to become graphically simpler and more schematic with repeated use, evolving
from a clear iconic representation of their object to a more abstract symbolic
representation. What is responsible for this shift?
We consider two hypotheses: (1) Simplification and refinement through repeated
production of the sign; and (2) Simplification and refinement through grounding.
According to the first hypothesis, the simplification of the drawings comes about through
- 12 -
repeated production. The basic idea is that directors (i.e., those drawing the pictures) will
increasingly refine their drawings on the basis of their prior experience and come to
depict only the ‘important’ aspects of the object in the drawing. In other words, as the
task proceeds the director will discover how to produce the most efficient sign – the one
which is most unique and diagnostic. Tversky (1995) offers a version of this hypothesis
to explain the increased simplification in written signs as they evolve from pictographic
to more abstract writing systems.1 Notice that for simplification-by-repetition, feedback
from the interpreter of the sign is not required.
However, for simplification-by-grounding, feedback of some kind is required. According
to this hypothesis graphical simplification occurs because the interpreter can indicate
when the drawing is sufficiently clear to identify the object and so ground that sign.
Successful grounding then helps to fix the sign-object interpretation in both the director’s
and the matcher’s memories. Notice that when there is no grounding or it is unsuccessful
(e.g., when a matcher queries a drawing) then the sign-object interpretation is less likely
to be fixed in memory. With repeated use of the sign (i.e., the drawing) increasingly less
information will be needed to recover the sign-object interpretation. The drawer can then
increasingly simplify the sign so long as it can be grounded on each occasion of use. To
the extent that simplification through grounding depends upon feedback, we can further
hypothesise that the rate and degree of simplification will depend upon the quality of the
communicative feedback that is given.
Opportunity for giving feedback can be manipulated in various ways in the ‘pictionary’
task. First, one can have situations in which there is the same director and the same
matcher throughout the experiment, where the matcher is allowed to offer graphical
feedback but not required to depict the object themselves. For example, the matcher can
indicate when they have ‘understood’ the drawing by producing a tick next to the
drawing or can annotate the director’s drawing in an attempt to check their interpretation.
Second, one can have a situation in which both communicators act as director or matcher
at different points in the experiment. So, both at some time depict the same object. This
means that a matcher who subsequently draws a concept in a similar fashion to that of the
- 13 -
director can demonstrate his or her understanding of the drawing and both participants
can increasingly reduce the information in their pictures on the basis of this reciprocal
understanding of the sign-object interpretation. This process should eventually lead to
sign-object interpretations that represent what is mutually intelligible to the two
participants. When both participants act as director we predict two outcomes; (1)
participants should become better at identifying the drawings, and (2) their respective
drawings should start to converge. It is also possible to further manipulate the quality of
the interactive feedback by allowing both director and matcher to be co-present during
drawing process as opposed to separating them during drawing. The experiments reported
here manipulated the quality of the feedback that could be given in all these ways.
4. Experiments
To test the two basic hypotheses, the first experiment compared graphical communication
under three conditions. First, there was a Single Director with no Feedback (SD-F)
condition, in which one participant repeatedly drew items but for an imaginary audience.
This tests whether repeated production alone is sufficient to produce refinement and
simplification. Second, there was a Single Director with Feedback (SD+F) condition, in
which there were two participants present: one who drew (the director) and one who
matched (the matcher). This condition allowed for simple feedback and so tests the basic
grounding hypothesis. And finally, there was a Double Director with Feedback (DD+F)
condition, in which the role of director and matcher alternated between the two
participants between blocks. This meant that both participants acted as directors and
matchers over the course of the experiment. The DD+F condition tests the above
predictions that reciprocal production should improve communicative success and the
prediction that the two directors’ drawings should converge.
In the second experiment we explore the Double Director situation further by varying the
degree of interaction available to the two participants. We contrasted a DD+F condition
of Experiment 1 with a condition in which both director and matcher were co-present
during the “pictionary” task affording a higher degree of interaction [Double Director Copresent (DDC) condition]. And, in the third and final experiment, we aim to establish
- 14 -
whether non-participants (‘overseers’) have difficulty in understanding graphical signs,
compared to the participants.
4.1 Experiment 1
The experimental task was the graphical equivalent of a verbal referential communication
task. Like the game ‘Pictionary’, participants were required to depict various concepts, in
such a way that a partner could identify them. Again, like ‘Pictionary’, participants were
not allowed to speak or use letters in their drawings. Unlike ‘Pictionary’, concepts were
drawn from a list of 16 items, which were known to both participants. Concepts included
easily confused groups such as drama, soap opera, theatre, Clint Eastwood and Robert
DeNiro (see Figure 3 for a complete listing).
The director, or drawer, was instructed to depict the first 12 items from an ordered list (12
targets plus 4 distracters) such that the matcher could identify each drawing from their
unordered list.
Places
Theatre
Art gallery
Museum
Parliament
---Insert Figure 3 about here ---People
Programmes
Objects
Robert De Niro
Drama
Television
Arnold
Soap opera
Computer monitor
Schwarzenegger
Clint Eastwood
Cartoon
Microwave
Abstract
Loud
Homesick
Poverty
4.1.1 Design and procedure
The experiment had three conditions. In two conditions (SD-F, SD+F) there was a single
director throughout the experiment and each director was paired with one matcher.
However, in the SD-F condition the matcher only saw and identified the drawings once
the director had finished drawing all the items over a series of four blocks, so there was
no matcher feedback. In the SD+F condition the director and matcher were separated by a
screen while the drawing was taking place. However, after each drawing was completed
- 15 -
the matcher was taken around the screen to identify it. At this point the matcher was able
to give graphical feedback. Matchers were instructed that they could annotate or modify
the director’s drawing or simply indicate that they had identified the concept by marking
it with a tick. The director could then modify the drawing and if necessary receive further
graphical feedback until the matcher was satisfied that he or she had interpreted the
picture correctly. In the third DD+F condition participants alternated roles (director and
matcher) from block to block. Otherwise the instructions were identical to that in the
SD+F condition. Contrasting task performance (i.e., accuracy of identification of the
drawings by the matcher) in SD+F versus DD+F conditions tests whether reciprocal
production and comprehension of drawings improves mutual understanding, as predicted
above. It also makes it possible to investigate both graphical refinement and convergence
in drawings from the two communicators (we leave analysis of convergence until after
reporting Experiment 2.)
Participants went through 4 blocks in the SD-F condition. In Block 1 the single directors
drew 12 of the 16 target items in isolation. In Block 2 they again drew 12 of the 16 items
(in a different random order) in isolation. They did this twice more (Blocks 3&4). Thus,
each participant drew a total of 48 items, with each of the 12 target items recurring four
times. Participants in the SD+F condition also drew 12 of the 16 items but across six
blocks, so that each item recurred six times during the experiment. Unlike participants in
the SD-F condition after each drawing was completed the matcher was allowed to give
feedback (see above). In the DD+F condition participants again drew 12 of the 16 items
(presented in a different random order for each block) over six blocks, but in this
condition the director and matcher alternated across blocks. So by the end of the
experiment each participant had drawn the 12 items three times as director and matched
12 drawings of the same items as matcher. Drawing took place on a standard whiteboard
using white board marker pens, with each participant using a different coloured pen. The
different colours were used to aid in the analysis of the feedback. The completed
drawings were recorded on digital camera for subsequent analysis.
- 16 -
4.1.2 Subjects
Twenty-four undergraduate participants took part in the SD-F condition. Twelve of them
acted as directors and the other 12 matchers subsequently identified the drawings after all
had been drawn. Thirty-two undergraduates (working in pairs) took part in the SD+F
condition. Another 32 undergraduates (working in pairs) took part in the DD+F
condition. Each participant was paid £5.
4.1.3 Data analysis
Graphical Complexity
To quantify graphical complexity we adopted the Perimetric Complexity (PC) measure
first defined by Attneave and Arnoult (1956) and subsequently developed by Pelli, Burns,
Farrell and Moore (2003). (PC = Inside +outside Perimeter2/Ink Area.) This measure is
obtained by first identifying the inside and outside perimeter of the graphic by removing
all pixels not on an edge, squaring the result and then dividing it by the ink area (i.e., all
the black pixels in the graphic). Perimetric Complexity has been identified as a scale-free
measure of the graphical information in a picture and Pelli et al. have convincingly
demonstrated that the inverse of the PC of a graphic predicts the efficiency with which
observers can identify the graphic from a learned set of competitors. For example, they
show that the PC of a letter in a particular font is inversely correlated with the efficiency
of identifying the letter in that font when masked with Gaussian noise (log efficiency
correlates with log complexity with r = -0.90).
-------- insert Fig. 4 about here -----(a)
- 17 -
Block 1
Block 5
Perimetric complexity =
Perimetric complexity =
1388
650
(b)
Block 1
Block 5
Perimetric complexity =
Perimetric complexity =
877
892
(c)
Block 1
Block 5
Perimetric complexity =
Perimetric complexity =
467
504
- 18 -
To illustrate the relationship between PC and perceived complexity examples of drawings
together with their perimetric complexity are shown in Fig. 4. In general the PC measure
does a good job of capturing intuitive judgments of graphical complexity (see 4a). Taking
a subset of the drawings (268 drawings taken from the first and last drawings of pairs in
the DD+F condition), we had a naïve participant rank the members of each pair of
drawings in terms of perceived graphical complexity and then compared those rankings
with the rankings produced by Pelli et. al.’s (2003) PC measure (continuous measure
transformed to rank order). There was 82% agreement between the two sets of rankings
(Kappa = 0.64). Examining the 18% cases where there was a discrepancy in the rank
ordering suggested that the human observer treated graphics with filled areas as more
complex than those with unfilled areas (see Fig. 4c for an example of such a discrepancy
due to the filled sunglasses) and graphics with more distinct meaningful elements as more
complex than those with less distinct meaningful elements (see Fig. 4b). In other words,
when a picture has two distinct meaningful elements such as the man and the roulette
table in Figure 4b(left) it is judged more complex than when it only has one meaningful
element as when there is only the roulette table in Figure 4b (right). Such semantic
complexity is not reflected in the pure PC measure. Because we are interested in
establishing only the structural complexity of the graphics as opposed to their semantic
complexity (see Final Discussion for a detailed discussion of this), we take PC as an
objective measure of the graphical complexity of the drawings.
- 19 -
Feedback
The drawings from each trial in the SD+F and DD+F conditions were analysed for
evidence of graphical feedback. (It was possible to identify instances of feedback from
the different coloured inks used by directors and matchers in the task). The feedback was
then classified according to whether it was minimal (a tick), or more interactive
(involving one or more rounds of annotation from both director and matcher). This data
was used to measure the extent of feedback across different conditions
4.1.3 Results
Overall task performance
For the two feedback conditions (SD+F, DD+F) identification accuracy was based on the
performance of the communicators themselves. For the no feedback SD-F condition
identification accuracy was based on the performance of the second group of 12 matchers
who were paired with a particular director and given the sequence of drawings produced
by that director. Figure 5 shows the identification rates (as proportion correct) for the SDF, SD+F and DD+F conditions across blocks.
---- Insert Figure 5 -----
Identification Accuracy
(proportion)
1.00
0.95
0.90
0.85
SD-F
SD+F
DD+F
0.80
0.75
0.70
0.65
0.60
0.55
0.50
1
2
3
4
Block
- 20 -
5
6
As can be seen from the figure, identification accuracy improved across blocks in all
three conditions. However, overall identification rates were lower in the SD-F condition
than in the SD+F or DD+F conditions. Finally, unlike the SD+F condition, where
identification accuracy plateaus with a slight fall after block 4, in the DD+F condition
matchers maintain identification performance across the final 2 blocks of the experiment.
Analysis of Variance (ANOVA) on the error proportions confirms these observations.
Both by-subject (F1) and by-item (F2) analyses were conducted. F1 analyses used a 3 X
4/6 mixed design ANOVA, treating Block (1-4 or 1-6) as a within subject factor and
Condition (SD-F, SD+F and DD+F) as between. F2 analyses were more powerful,
treating both Block and Condition as within subject factors. For all effects reported in this
paper p<0.05 unless otherwise stated.
The first set of analyses, conducted over blocks 1 to 4, produced a main effect of Block
[F1(3, 123)=14.79; F2(3, 45)= 6.16] and a main effect of Condition in the by-items
analysis [F2(2,30)=6.19] but not in the less powerful by-subjects analysis(p>0.1). There
was no reliable interaction between Condition and Block. Further analysis of the
Condition effect revealed that there was reliable difference by-items between SD-F and
DD+F [F2(1,15)=9.71] and marginally also between SD-F and SD+F [F2(1,15)=4.23,
p=.057].
A separate analysis was conducted across the six blocks of the SD+F and DD+F
conditions. There was a reliable effect of Block [F1(5, 150)=8.82; F2(5, 75)=3.04] and of
Condition in the by-items analysis [F2(1, 15)=8.98], but no interaction (Fs<1). However,
tests of simple effects showed that prior to block 6 there was no difference in
identification rate between the SD+F and DD+F conditions (Fs<2.14). However, after
this point there were reliable differences between the conditions [Block 6. F1(1,
180)=3.77; F2(1, 15)=6.18]. These analyses confirm that the main effect of Condition
between SD+F and DD+F was caused by the greater identification accuracy for the
DD+F communicators compared with the SD+F communicators at Block 6.
- 21 -
Graphical Complexity
Figure 6 gives examples of sequences of drawings from the SD-F and the SD+F
conditions. It is clear from these examples that the refinement in drawings across trials is
quite different for the two cases. This difference can be noted in the PC measures shown
in Figure 7 [Mean scores were calculated after removal of scores 2.5 standard deviations
(2% of data) from the condition mean and replaced with scores corresponding to the
mean plus or minus 2.5 standard deviations.]
------insert Figure 6 about here -------
Block 1 (SD-F)
Block 2 (SD-F)
Block 3 (SD-F)
Block 1 (SD+F)
Block 2 (SD+F)
Block 3 (SD+F)
Block 4 (SD+F)
Block 5 (SD+F)
Block 6 (SD+F)
- 22 -
Block 4 (SD-F)
Block 1 (DD+F)
Block 2 (DD+F)
Block 3 (DD+F)
Block 4 (DD+F)
Block 5 (DD+F)
Block 6 (DD+F)
As can be seen from the figure, in contrast to the SD-F condition where graphical
complexity increases across blocks of the experiment, in the SD+F and DD+F conditions
graphical complexity decreases. Furthermore, graphical complexity starts at, and
decreases at, a similar rate in these last two conditions. Tests by ANOVA validate these
observations.
---------- insert Figure 7 about here --------
- 23 -
3000
Complexity
2500
2000
SD-F
SD+F
DD+F
1500
1000
500
0
1
2
3
4
5
6
Block
Again ANOVA was conducted both by subject (F1) and by item (F2). F1 analyses treated
Block as a within subject factor and Condition as between. F2 analyses treated both
Block and Condition as within subject factors. The by-items analysis (across the four
blocks of each condition) produced a main effect of Block [F2(3, 45)=3.68] with a
marginally significant effect also in the by-subjects analysis [F1(3, 123)=2.08, p=.10].
There was also an effect of Condition [F1(2, 41)=10.46; F2(2, 30)=109.84]. More
importantly, there was a reliable interaction between Block and Condition [F1(6,
123)=11.27; F2(6, 90)=43.96]. This was due to the increasing complexity of graphical
representations in the SD-F condition [F1(3, 123)=8.28; F2(3, 45)=18.38] compared with
their decreasing complexity in the SD+F condition [F1(3, 123)=9.50; F2(3, 45)=15.56]
and the DD+F condition [F1(3, 123)=6.82; F2(3, 45)=10.91]. There was no difference in
the complexity pattern between the SD+F and DD+F conditions (ts<1.65).
Feedback
Figure 8 shows an example of a complex sequence of graphical feedback from a matcher
- 24 -
and director in the first block of the DD+F condition. The director is trying to portray
Robert De Niro (picture 1 in the sequence). In picture 2 the matcher graphically queries
the depiction, asking something along the lines 'Is this a drawing of someone who can be
seen on television?'. The director then emphasizes who is being depicted by circling the
head of the 'to be identified' representation and adds a star to indicate that this person is
indeed a film/television 'star' (picture 3). The matcher then queries this depiction by
drawing a question mark and an arrow linking the question mark to the 'to be identified
picture’ (picture 4). This indicates that the depiction is inadequate for identification
purposes. Next, by drawing a circle with an arrow and using additional arrows to link this
to his/her depiction, the director indicates that his/her depiction is of a male (picture 5).
But this is still insufficient and the matcher has to draw a gun with a question mark,
asking whether the person depicted is 'an action hero' (picture 6). Finally, the director
replies by drawing three figures (the competing action figures used in the task) and
indicating with arrows that it is the mid-sized action hero. This appears to resolve the
drawing ambiguity, as the matcher makes his/her selection.
-----Insert Figure 8 about here-----
Block 1 Picture 1
Block 1 Picture 5
Block 1 Picture 2
Block 1 Picture 6
- 25 -
Block 1 Picture 3
Block 1 Picture 7
Block 1 Picture 4
In fact this kind of complex repair process proved to be extremely rare. Figure 9 shows
the proportion of trials with more than minimal feedback (i.e., involving more than just a
tick) across all blocks for the SD+F and the DD+F conditions. For the first block only
17% of trials in the SD+F and 7% in the DD+F were accompanied with more than
minimal feedback. By the third block the percentages were less than 1% in either
condition. So on most occasions matchers were happy to offer the minimal tick against
the initial depiction to indicate recognition.
Feedback Trials (proportion)
----- Insert Figure 9 about here -----0.25
0.20
0.15
SD+F
DD+F
0.10
0.05
0.00
1
2
3
4
5
6
Block
The proportion of trials with more than minimal feedback were entered into a 2-way
ANOVA with Condition (SD+F, DD+F) and Block (Games 1-6) as factors. The bysubjects analysis treated Condition as a between subject factor and Block as within,
whereas the by-items analysis treated both Condition and Block as within-subjects
factors. The ANOVAs showed a main effect for Block [F1(5, 130) = 17.25; F2(5,75) =
9.23] and a main effect of Condition in the by-items analysis [F2(1,15) = 7.07] but not by
subjects. These main effects were qualified by an interaction between Block and
Condition [F1(5,130) = 2.99; F2(5,75) = 10.44]. The interaction reflected the fact that
there was more above-minimal feedback in the first Block for the SD+F condition than
the DD+F [F1(1,180) = 16.7; F2(1, 15) = 33.95], but no difference in subsequent Blocks
- 26 -
(all Fs < 1). For all subsequent games there was no difference between the two
conditions.
Presumably matchers in the SD+F condition are aware that they will not have an
opportunity to draw themselves so they more readily engage the drawer when the referent
of the drawing is unclear. This stands in contrast to those in the DD+F condition who are
afforded the opportunity to A) indicate understanding by adopting the previously used
representation or B) offer an alternative, potentially more effective representation.
4.1.4 Discussion
The results from Experiment 1 allow us to rule out a simplification through repeated
production explanation for the transformation of the graphical signs in the ‘pictionary’
task. In fact, the complexity results from the SD-F condition indicate that drawings
become increasingly complex with repeated production in the absence of any feedback
from matchers (compare the example drawings in Fig 5). This result is reminiscent of
findings in referential communication tasks. When speakers have to repeatedly describe
the same picture to an imaginary audience their descriptions often become longer and
more complex over trials (Krauss & Weinheimer, 1967; Hupet and Chantraine, 1992).
One speculation is that directors in the absence of feedback think of more features of the
objects/concepts to be communicated on each occasion and include these additional
features in each successive representation.
By contrast, the overall pattern of results is consistent with the grounding explanation
whereby feedback enables matchers and directors to ground each representation and
commit it to memory. On each subsequent occasion the director can then produce a
reduced representation sufficient to elicit the prior use of that representation which is then
grounded again through feedback. Finally, the results from the performance measure do
point to a difference between the effect of comprehension-based feedback alone and the
effect of reciprocal production and comprehension. Participants in the DD+F condition
continued to improve and/or maintain their performance across the whole six blocks of
the experiment, whereas those in the SD+F condition became poorer after the fifth block.
Although not reliably different the graphical complexity measures are numerically lower
- 27 -
for the DD+F than SD+F drawings. This means that the improved performance on the
task was not at the expense of increased complexity in the drawings, if anything quite the
opposite.
Experiment 1 demonstrates that communicative feedback is required to support the
increased simplification of graphical signs in the ‘pictionary’ situation, as predicted by
the grounding hypothesis. Interestingly, in the vast majority of cases a simple mark of
recognition is sufficient for this to happen. More complex feedback was extremely rare
(particularly in the DD+F condition). The results also support the hypothesis that
reciprocal production and comprehension enhances the mutual intelligibility of the
grounded graphical signs. The second experiment was designed to manipulate the quality
of interactive feedback available to participants and establish the extent to which quality
of feedback affects the rate of simplification of graphical signs.
4.2 Experiment 2
4.2.1 Design and procedure
Experiment 2 had two conditions corresponding to different levels of interaction for the
Double Director arrangement in Experiment 1. In the concurrent feedback (CF)
condition, graphical interaction took place in real time, partners standing side-by-side at
the whiteboard. In the non-concurrent feedback (NCF) condition, a visual barrier
separated the partners during drawing, but after each drawing episode was finished they
were able to interact graphically (as in the SD+F and DD+F conditions in Expt.1).
Whereas the concurrent feedback condition simulates face-to-face conversation in which
feedback can accompany the speaker’s production, the non-concurrent condition
simulates e-mail communication in that participants can only exchange graphics once
they have completed them. In both CF and NCF conditions participants alternated
between acting as director and acting as matcher as for the DD+F condition in
Experiment 1. In all other respects the procedure was the same as that for DD+F
condition of Experiment 1.
- 28 -
4.2.2 Subjects
The participants were 56 undergraduates, each of whom was randomly assigned to
concurrent or non-concurrent feedback conditions. 12 pairs participated in the CF
condition and16 in the NCF condition. Members of each pair had never met each other
prior to the experiment and each participant was paid £5 for taking part.
4.2.3 Results
Overall performance
As can be seen from Figure 10, identification rates (proportion of items correctly
identified by matchers) in the CF and NCF conditions were comparable. In both cases,
identification rates improved across Blocks.
--------- insert Figure 10 about here -------
Identification Accuracy
(proportion)
1.00
0.95
0.90
0.85
0.80
NCF
CF
0.75
0.70
0.65
0.60
0.55
0.50
1
2
3
4
5
6
Block
Proportion scores were entered into a 2 x 6 ANOVAs carried out by subject (F1) and by
item (F2). The by-subject ANOVA used a mixed design, treating Type of Feedback (CF,
NCF) as a between subject factor and Block (1 to 6) as within, whereas by-item tests
treated both factors as within. Analyses showed a main effect of Block [F1(5, 130) =
11.59; F2(5, 75) = 13.46] but no effect of Type of Feedback, or interaction between Block
and Type of Feedback (Fs < 1.87). Thus, participants’ ability to identify the referent of
their partner’s drawing was not influenced by the nature of the feedback.
- 29 -
Graphical Complexity.
To illustrate the typical changes in representational form across blocks we provide an
example from CF conditions in Figure 11.
------- insert Figure 11 about here ------
Block 1 (CF)
Block 2 (CF)
Block 3 (CF)
Block 4 (CF)
Block 5 (CF)
Block 6 (CF)
The first drawings from each participant take an iconic form, resembling the concepts
they depict. However, over the course of the experiment directors strip unnecessary
information from the initial representation, leaving only the salient aspects of the image.
Note also, that as a result of the interaction participants’ drawings converge; contrast the
first drawings by Director1 and Director 2 of computer monitor (i.e., the first two
drawings in Figure 11) with the last drawings by the same participants (i.e., the last two
drawings in Figure 11).
- 30 -
The mean PC scores for the drawings were calculated after replacing scores more than
2.5 standard deviations from the condition mean (1.92% of data). Figure 12 shows the
result across Blocks in the CF and NCF conditions (for comparison measures of
complexity for the SD-F condition of Experiment1 are also shown).
-------- insert Figure 12 about here ------3000
Complexity
2500
2000
SD-F
NCF
CF
1500
1000
500
0
1
2
4
3
5
6
Block
The figure shows that the complexity of drawings decreases more rapidly, and more
smoothly in the CF condition than the NCF condition. These observations are
corroborated by ANOVA.
Complexity scores were entered into 2 x 6 ANOVA. By-subject analysis used a mixed
design, treating Type of Feedback (CF, NCF) as a between subject factor and Block (1 to
6) as within, whereas by-item analysis treated both factors as within. There were main
effects of Type of Feedback [F1(1, 26)=2.85;F2(1,15)=16.7] and Block[F1(5,190)=16.66;
F2(5,75)=40.35]. These effects were qualified by a reliable interaction in the by-items
analysis [F2(5,75)=5.78].
Tests of simple effects show that images produced in the CF
condition increasingly become simpler than those in the NCF condition [Type of
Feedback at Block 4, F1(1,156)=3.47, p=.06; F2(1,15)=6.66; at Block 5, F1(1,156)=2.56,
p=.1; F2(1,15)=5.40; at Block 6, F1(1,156)=4.9;F2(1,15)=10.59].
- 31 -
Feedback
As in Experiment 1 there were very few trials in which there was more than minimal
feedback given. Even in the first block only around7% trials in the NCF and the CF
conditions were successfully negotiated with just a tick of recognition. After this more
than minimal feedback was given on less than 1% of occasions. Figure 13 shows the
proportion of trials with different degrees of feedback for the CF and NCF conditions of
the experiment. It is interesting to note that there is only minimal feedback given in either
condition. Thus the increased rate of simplification of the drawings in the CF condition as
compared to the NCF can only be attributed to the improved nature of the feedback (i.e.,
that it can be given during the drawing process) rather than to the amount or complexity
of the feedback given. Because the proportions of more than minimal feedback trials was
so low ANOVA was considered inappropriate for this data.
------- insert Figure 13 about here -------
Proportion Interaction Trials
0.25
0.20
0.15
CF
NCF
0.10
0.05
0.00
1
2
3
4
Block
- 32 -
5
6
Graphical Convergence
We examined graphical convergence in the drawings produced in the CF and NCF
conditions. For each pair, the drawings for each item (e.g. drama, as drawn by each
subject in the pair) were placed in three categories: early (block 1 and 2 images), middle
(block 3 and 4 images) and late (block 5 and 6 images). Two independent judges then
rated each pair of images (presented in random order) for similarity on a ten-point scale
(1= dissimilar to 10 = very similar). The inter-coder agreement was high with a
correlation of r= 0.85 between the similarity judgements for each pair of drawings. The
mean similarity judgments for early, middle and late drawings from the pairs are shown
in Figure 14.
------ Insert Figure 14 about here -----
Convergence Rating (0-9)
8
7
6
NCF
CF
5
4
3
1&2
3&4
5&6
Drawings
As can be seen from the figure graphical convergence increased across blocks for both
the CF and NCF conditions with no apparent difference between the two conditions.
These data were entered into ANOVAs. The by-subjects ANOVA was a mixed design
with Block (early, middle, late) as a within subject factor and condition as between
subject factor. For the by-items analysis both Block and Condition were treated as withinsubject factors. The results confirm our observations yielding a main effect for Block
- 33 -
[F1(2, 52)=31.1; F2(2,30)=45.4]. No other effects or interactions approached significance
(All Fs<1).
Thus, as is the case in verbal communication, in which conversational partners entrain
one another’s verbal expressions, partners’ graphical expressions converge through
interaction (Garrod & Anderson, 1987; Brennan & Clark 1996). Interestingly the degree
of convergence was not influenced by the quality of feedback.
4.2.4 Discussion
The results from Experiment 2 extend the findings from Experiment 1 in two ways. First,
comparison of the relative reduction in graphical complexity of drawings indicates that
the more direct the feedback between participants the greater the tendency to reduce the
complexity of the drawings over the course of the experiment. This increased graphical
refinement occurs without any loss in identification accuracy, because there was no
difference in the overall performance of the two groups. The second finding is that the
drawings of each director in both CF and NCF conditions increasingly converge.
These results are interestingly comparable to those found in other referential
communication tasks (Hupet & Chantraine, 1992; Clark & Wilkes-Gibbs, 1986; Krauss
& Weinheimer, 1967). In the same way that referential descriptions become increasingly
shortened in the presence of listener feedback graphical signs become increasingly simple
and in line with our earlier argument they become more symbolic. Similarly, in the
absence of any listener feedback referential descriptions become more complex with
repetition (Hupet & Chantraine, 1992) and so do graphical signs. Furthermore, in line
with our earlier arguments they retain their iconicity. We attribute this pattern to a
general interactive communicative mechanism that is in common to the two situations.
As communicators ground the signs successfully they can increasingly leave out nonessential information. Notice that such collaborative reduction in the sign’s complexity
depends upon the prior history of use of that sign by those communicators. In the
referential communication literature the importance of this shared history of interaction
was shown by comparing an ‘overhearer’s’ ability to understand the referential
description with that of a communicative participant. Schober and Clark (1989)
- 34 -
demonstrated that ‘overhearers’ performed systematically more poorly than
conversational partners when it came to identifying the intended referent of a description.
The third experiment was designed to establish whether ‘overseers’ would have the same
problem with graphical signs produced in the ‘pictionary’ experiments as ‘overhearers’
do with referential descriptions.
4.3 Experiment 3
In this experiment ‘overseers’ were given complete sets of drawings taken from those
produced by the participants in the SD+F and DD+F conditions of Experiment 1. The
drawings were presented in the order in which they had been produced during the original
experiment and ‘overseers’ were required to identify each drawing in relation to the list
of alternatives presented to the original Matchers in Experiment 1. There were two
groups of ‘overseers’ that corresponded to Schober and Clark’s (1989) early and late
‘overhearers’. The early ‘overseers’ saw all the drawings produced starting with block 1
and moving through to block 6, whereas the late ‘overseers’ were only exposed to
drawings produced in the final 3 blocks (blocks 4-6). The prediction is that both early and
late ‘overseers’ will perform consistently less well than the original matchers, because
they cannot engage in any interaction with the original director to help clarify the
meaning of the drawing. Furthermore, late ‘overseers’ should perform consistently less
well than early ‘overseers’. This is because they do not have access to the full
communicative history of that drawing. As a consequence they will be in a less
favourable position to establish its symbolic interpretation than the early ‘overseers’.
4.3.1 Method
28 complete sets of drawings were sampled from 14 pairs of players in the SD+F and 14
pairs of players in DD+F conditions of Experiment 1. Each set formed a stimulus set for a
different participant. 28 participants took part in the early overseer with 14 in the SD+F
condition and 14 in the DD+F condition. Each participant identified a set of images
generated across all blocks (1-6) from one pair of either SD+F or DD+F drawers. In the
- 35 -
late overseer condition, 28 different participants did the same for drawings from blocks 46. Again each participant identified only one set of images within the conditions. In both
conditions, participants were given a table of items, including four distracter items, and
were instructed to choose which object they thought that the picture represented.
4.3.2 Results
In addition to the original identification rates (taken from the SD+F/DD+F conditions of
Experiment 1), early ‘overseer’ (SDO1 for the SD+F ‘overseers’ and DDO1 for the
DD+F ‘overseers’) and late ‘overseer’ (SDO2 and DDO2) identification rates are plotted
in Figure 15. As hypothesised, ‘overseers’ performed consistently less well than the
original matchers across all blocks. Identification accuracy was better in the DD+F
condition than for both early/late ‘overseers’ (DDO1/DDO2) early ‘overseers’ were
better at identifying the items than late ‘overseers’. Similarly, Identification accuracy was
better for the SD+F condition than for both early and late ‘overseers’ (SD01, SD02) and
early ‘overseers’ were better than late ‘overseers’.
- 36 -
Identification accuracy across conditions
Accuracy (proportion)
1
0.9
0.8
SD+F
0.7
DD+F
SD01
0.6
DD01
SD02
0.5
DD02
0.4
0.3
0.2
1
2
3
4
5
6
Block
---- insert Figure 15 about here ----ANOVAs were carried out to confirm the observed differences. Because each participant
had a different set of the original stimuli (i.e., all items in the population of drawings
sampled were assigned to different subjects on a one-to-one basis) only by-subjects
ANOVAs are appropriate (Clark, 1973). The first ANOVA had one between subject
factor Condition (SD+F/DD+F from Experiment 1 and early ‘overseer’ SDO1/DDO1
from Experiment 3) and one within factor Block (Blocks 1-6).
The analysis showed significant main effects of Condition [F(3,52)=21.9] and Block
[F(5,260)=18.3], but no interaction (F < 1). Planned pair-wise comparisons show that
accuracy for the original matchers in the SD+F condition was significantly better than for
the early ‘overseers’ in the same condition (SDO1) [F(3,52)=34]. Accuracy of the
original DD+F matchers was also significantly higher than for the early ‘overseer’
(DDO1) [F(3,52)=28]. Overall, ‘overseers’ of the more interactive condition (DDO1)
were not significantly more accurate than those of the less interactive condition (SDO1)
(F< 2).
- 37 -
The second ANOVA analysed blocks 4-6 of the original and early overseer identification
(SD+F/DD+F/SDO1/DDO1) and late overseer identification (SDO2/DDO2). In this 3
(blocks) X 6 (conditions) ANOVA the design treated Blocks as within and Condition as
between. There was a significant main effect of Condition [F(5,78)=17.7] with no main
effect of Block or interaction between the two (Fs<1.4).
Planned pair-wise comparisons show that the accuracy of original SD+F matchers was
significantly greater than for early overseers (SDO1) [F(5,78)=18.9]. In addition, the
accuracy for early overseers (SDO1) was significantly greater than for that of late
overseers (SDO2) [F(5,78)=4.5]. Similarly, the accuracy across blocks 4-6 of the original
DD+F condition was significantly greater than for the late overseers (DDO2)
[F(5,78)=39.5]. And, the accuracy of early overseers (DDO1) was significantly greater
than late overseers (DDO2) [F(5,78)=5.0].
4.3.3 Discussion
As hypothesised, both early and late overseer groups showed similar patterns of
identification accuracy to that of the original matchers only at significantly lower
accuracy rates. This is in line with Schober and Clark’s (1989) finding that ‘overhearers’
who did not participate in the interaction were significantly less accurate in identifying
referents than original interlocutors. For early ‘overseers’ identification improved across
blocks 1-3 in both the SD and DD conditions. As with the original matchers, this
improvement reached a plateau in the SD condition at block 3, thereafter only increasing
for the more interactive DD condition. Late ‘overseers’, not exposed to the original
interaction and refinement of the drawings, found identifying later symbolic drawings
significantly more difficult than both original matchers and early overseers.
The results from the overseer experiment support the hypothesis that the drawings are
refined as a result of feedback given in the original SD+F and DD+F situations. Without
being able to engage in the interaction itself ‘overseers’ are much poorer at interpreting
the drawings than the original participants. This is exactly as predicted by our hypothesis
that the locus of structure moves from the sign into the communicators’ memory
- 38 -
representations as a consequence of grounding the signs during the course of the
interaction. In this respect performance in the graphical communication task is strikingly
similar to that observed in referential communication tasks using speech. This raises the
question of whether the general simplification and convergence of pictorial
representations is driven by the same interactive constraints that apply in the language
cases.
5 Final Discussion
This paper set out to investigate the origin of graphical symbols. It is straightforward to
explain how iconic graphical representations arise because their interpretation is based on
perceptual resemblance to their objects. It is not so obvious how graphical symbols arise
because their interpretation depends upon prior understanding of the arbitrary
relationships between the sign and its object. However, if it can be shown that graphical
symbols systematically evolve from the repeated use of iconic/indexical signs, then this
would account for the problem. Although such an evolution is suggested by the historical
development of many symbolic writing systems, which start out as iconic representations,
it is not clear precisely what drives the evolution.
We argued for an information theoretic account of the basis of iconicity and symbolicity
of signs; whereas iconic signs carry information in their structure and hence are
structurally complex, symbolic signs rely on information in the communicators’ memory
representations arising from the prior use of the signs to signify their particular objects.
The key question was what mechanisms underlie the systematic evolution from icon to
symbol: is it simply repeated production, or does it require interaction and feedback
between sign users?
The results from the experiments clearly show that this evolution depends upon
interaction between communicators. Experiment 1 showed that in the SD-F condition in
which there was no interaction between director and matcher the drawings became
increasingly complex with repetition and retained their iconic character, whereas with
even minimal opportunity for feedback in the SD+F condition the drawings became
- 39 -
increasingly simple and more symbolic in character. Furthermore, the experiment showed
that when both communicators are able to produce the drawings (DD+F condition) the
communication becomes more effective. Matchers become better at identifying the
drawings while the drawings become at least as simple if not slightly simpler than in the
SD+F condition. Experiment 2 showed that the rate at which the drawings become
simpler and more abstract is affected by the quality of the interactive feedback allowed
by the task. This experiment also demonstrated that the drawings produced by each
communicator increasingly converge with those of their partner. Experiment 3
demonstrated that interpreting the drawings appropriately depended on being party to the
interaction that drives their simplification and abstraction. ‘Overseers’ not engaged in the
original communication process perform consistently less well at identifying the
drawings than the original participants. According to our argument in Section 1, this
process of simplification and abstraction reflects a transformation from iconic graphical
signs to symbolic graphical signs.
So in summary, these graphical communication studies offer an insight into the evolution
of graphical symbols from iconic and indexical signs. Also they are consistent with the
information theoretic account of the basic nature of signs. In this account iconic (and
indexical) signs are inherently more complex than symbolic signs because they need to
reflect the structure of their objects in the structure of the sign itself according to the
structure mapping principle. Symbolic signs only require enough structure to enable users
to identify that sign from its competitors in the set. However, this brings us to an aspect
of signification apparent in many of the drawings produced in the ‘pictionary’
experiments that we have not considered thus far, which introduces the problem of
compositionality and systematicity of signs.
------ insert Figure 16 about here -----
- 40 -
Block 1
Block 2
Block 3
Block 4
Block 5
Block 6
Compositionality and systematicity of signs
Consider the series of drawings for ‘art gallery’ shown in Figure 16, which were
produced by a pair in Experiment 1. Like many of the graphical depictions encountered
in the ‘pictionary’ experiments, all of the examples are semantically complex. In other
words, they contain distinct graphical elements each with its own semantic interpretation,
which in turn combine with the interpretations of the other elements to produce an
interpretation of the whole. Each drawing contains three different elements: (1) an iconic
depiction of a building; (2) a pair of diverging lines similar to a ‘less than’ symbol; and,
(3) an iconic depiction of a painting. The interpretation of the whole drawing is derived
from interpreting the ‘less than’ symbol as indicating that the object signified on its left
contains the object(s) signified on its right: hence, it is a building that contains paintings.
With repeated use, the drawings become graphically simpler, and the iconic components
depicting the building and the painting become more symbolic, but nevertheless the
whole sign retains its semantic complexity. It is perhaps worth emphasising that semantic
complexity need not be retained in every case: in another series of drawings which
- 41 -
incorporated the ‘less than’ sign to signify containment, the whole sign eventually
evolved until this was the only element left.
Now, compare two hypothetical symbol systems: one containing semantic complexity
like this, and the other without it. As argued in the introduction we can move structure
from individual signs into representations that reflect the history of interaction. We can
also put it into the structure of the coding system. But we can detach the graphical
complexity of a single sign (its lines and details) from its semantic complexity. The latter
corresponds to the extent to which it contains independent subcomponents. When these
subcomponents occur across multiple signs, each sign associated with a different object,
there is structure within the coding scheme. When a subcomponent only occurs in signs
associated with a family of related objects, then the coding scheme’s structure
corresponds to (indeed, structure-maps) at least some of the structure within the domain
of objects. This is very different from the structure mapping found in icons: there, subparts of drawings correspond to sub-parts of the objects they resemble. Here, sub-parts of
drawings need not correspond to sub-parts of objects (and the overall drawing need not
resemble the overall object). Rather, sub-parts of drawings correspond to taxonomic subparts of the domain of objects as a whole. For instance, in the art gallery example, the
building sub-component could turn out to be associated with all and only the institutions
in the domain of objects.
Thus, a system of signs can exhibit structure that constitutes a kind of systematicity, and
signs within that system can have compositional structure. Their overall meaning will
depend on their components, and their mode of combination. In the art gallery example,
part of the drawing is the ‘less than’ symbol, a connective which helps the knowledgeable
viewer calculate the overall meaning of the drawing from the more iconic elements which
it links. It seems plausible that natural systems of signs can mix systematic with nonsystematic signs: and systematic signs can contain both iconic and symbolic subcomponents. But the more structure there is in the coding system, the more prevalent
compositionality will be. Such systematicity has advantages for new users of the sign
system. Learners are more likely to guess correctly how to associate objects with signs
- 42 -
they have never used before, thanks to structure in the domain, coupled with the elements
the new signs may have in common with other signs that they have encountered.
Systematicity of this kind has a likely effect on the discriminability of symbols from one
another. Where there is no semantic complexity or systematicity, users of a set of
symbols benefit if each symbol is perceptually discriminable from other members of the
set. Thus, one might expect the symbols to evolve to be globally discriminable from one
another, and hence resemble one another less and less. However, where there is
systematicity, subsets of symbols (which correspond to a subset of related domain
objects) will continue to resemble one another to some extent. At the same time, these
subsets will still tend to be discriminable from other subsets. And within a subset, more
local differences will help locally discriminate related symbols from one another; there
will thus be a collection of non-arbitrary perceptual similarities and differences between
sets of symbols.
Our account of the evolution of sets of icons into sets of symbols, and of sets of symbols
into sets of systematic symbols has some direct parallels in Deacon’s (1997) Peircean
account of signs. Obviously, there are some terminological differences, and Hurford
(1998) argues forcefully that Deacon’s terminology is non-standard. As noted in the
introduction, where we would use the term ‘symbol’, Deacon reserves the term ‘index’
for a sign regularly associated with an object. And while we consider a set of such
associations to be a kind of symbol system, Deacon additionally requires the properties
we have associated with systematicity (p99). It is also true that Deacon discusses symbol
systems in terms of compositionality and predication, whereas we have only discussed
systematicity and reference. But in spite of these differences, there are several parallels.
For instance, he too emphasises how perceptual categorisation, memory, and prediction
must interact to lead to fully developed symbol systems (p78); and he emphasises that the
set of relations between signs can come to resemble the set of relations between objects
(p88).
- 43 -
There remain some deeper differences. First, as we have just noted, even in a set of signs
without systematicity, there can be relationships between the signs, via global
discriminability; and so it seems reasonable to see these sets of signs as being symbolic in
a traditional way. More importantly, we also expect local discriminability when a set of
signs contains members exhibiting semantic complexity. What is particularly notable is
that, in practice, these symbols retain sub-parts which have iconic properties. In Deacon’s
discussion, this issue does not arise. The reason for this is that, although he argues that
symbolic relations depend on lower-level indexical and iconic relations, when he
discusses symbolic systems, he assumes that an object is (indexically) picked out by a
purely arbitrary sign. The assumption of arbitrariness drives a distinct wedge between
sets of icons and sets of symbols. Here, we have emphasised that sets of symbols may
evolve from sets of icons; and hence, members of a set of symbols may possess a
significant degree of residual iconicity. It means that whether a system of signs is
compositional (or systematic) or not, and whether its signs retain iconicity or not, are
orthogonal questions.
The idea that systematicity and iconicity vary independently fits with a number of other
theories and observations. For instance, Barsalou’s (1999) perceptual symbol systems
theory makes room for mental representations composed of parts which perceptually
resemble objects. And in natural language, there has been continuing interest in the extent
to which supposedly arbitrary compositional systems possess iconic features. For
instance, structure in syntax and discourse often iconically represents temporal structure
or task structure (Haiman 1985). More recently, Shillcock et al. (in submission) have
argued that resemblances between words in the English lexicon are non-arbitrary, so that
perceptual similarities in phonological form can be a guide to underlying similarities in
meaning, and vice versa. And, work on American Sign Language (ASL) has argued that
such non-arbitrary “richly grounded” forms can co-exist with arbitrary forms (Macken et
al. 1993). Furthermore, ASL can be compared with homesign languages, where deaf
individuals in non-signing households develop their own communication systems (eg
Goldin-Meadow and Mylander 1998). Reviewing such work, Morford (1996:166-7)
- 44 -
suggests that iconicity is retained when “symbols are used with individuals unfamiliar
with the system”.
Thus, we advocate further investigation of the ways in which icons can evolve into
symbols, the constraints on simplification-by-grounding, and the ways in which symbol
systems can sustain residual iconicity. Previous work has studied naturally-occurring
systems of signs; on the basis of our experience, we would argue that the ‘pictionary’
paradigm we have exploited here provides an excellent vehicle for further, controlled
investigation of the evolution of representations.
References
Attneave, F., & Arnoult, M. D. (1956). The quantitative study of shape and pattrn
perception. Psychological Bulletin, 53, 452-471.
Barrera, M. E., & Maurer, D. (1981). Recognition of mother's photographed face by three
month old. Child Development, 52, 714-716.
Barsalou, L.W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22,
577-609.
- 45 -
Bloom, P., & Markson, L. (1998). Intention and analogy in children's naming of pictorial
representations. Psychological Science, 9, 200-204.
Brennan, S. E., & Clark, H. H. (1996). Conceptual pacts and lexical choice in
conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition,
22, 1482-1493.
Callaghan, T. C. (1999). Early understanding and production of graphic symbols. Child
Development, 70, 1314-1324.
Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language
statistics in psychological research. Journal of Verbal Learning and Verbal
Behavior, 12, 335-359.
Clark, H H (1993) Arenas of Language Use. University of Chicago Press.
Clark, H. H., and Brennan, S. A. (1991). Grounding in communication. In L.B. Resnick,
J.M. Levine, & S.D. Teasley (Eds.). Perspectives on socially shared cognition.
Washington: APA Books.
Clark, H. H., & Schaefer, E. F. (1987). Collaborating on contributions to conversation.
Language and Cognitive Processes, 2, 19-41.
Clark, H. H., and Wilkes-Gibbs, D. (1986). Referring as a collaborative process.
Cognition, 22, 1-39. Reprinted in P. R. Cohen, J. L. Morgan, & M. E. Pollack
(Eds.), Intentions in Communication. Cambridge: MIT Press, 1990.
Deacon, T.W (1997) The Symbolic Species: The Co-Evolution of Language and the
Brain. W.W. Norton & Company.
DeLoache, J. S. (2004). Becoming symbol-minded. Trends in Cognitive Sciences, 8, 6670.
DeLoache, J. S., Pierroutsakos, S. L., Uttal, D. H., Rosengren, K. S., & Gottlieb, A.
(1998). Grasping the nature of pictures. Psychological Science, 9, 205-210.
Garrod, S., & Anderson, A. (1987). Saying what you mean in dialogue: a study in
conceptual and semantic co-ordination. Cognition, 27,181-218.
Garrod, S., & Doherty, G. (1994) Conversation, coordination and convention: An
empirical investigation of how groups establish linguistic conventions. Cognition,
53, 181-215.
- 46 -
Goldin-Meadow, S., and Mylander, C. (1998) Spontaneous sign systems created by deaf
children in two cultures. Nature, 391, 279–281.
Greenlee, D. (1973) Peirce’s Concept of Sign. The Hague & Paris: Mouton.
Haiman, J. (1985) Iconicity in Syntax (ed.). John Benjamin.
Hupet, M., and Chantraine, Y. (1992). Changes in repeated references: Collaboration or
repetition effects? Journal of Psycholinguistic Research, 21, 485-496.
Hurford, J.R. (1998) Review of ‘The Symbolic Species’ by Terrence Deacon. Times
Literary Supplement, October 23rd, 1998, p34.
Ittelson, W. H. (1996). Visual perception of markings. Psychonomic Bulletin and Review,
3, 171-187.
Karmiloff-Smith, A. (1990). Constraints on representational change: Evidence from
children's drawing. Cognition, 34, 57-83.
Krampen, M. (1983) Icons of the Road. Semiotica 43: 1-204.
Krauss, R. M. and Weinheimer, S. (1964). Changes in reference phrases as a function of
frequency of usage in social interaction: A preliminary study. Psychonomic
Science, 1, 113-114.
Krauss, R. M., & Weinheimer, S. (1967). Effects of referent similarity and
communication mode on verbal encoding. Journal of Verbal Learning and Verbal
Behavior, 6, 359-363.
Krauss, R. M., & Weinheimer, S. (1967). Concurrent feedback, confirmation and the
encoding of referents in verbal communications; Effects of referent similarity and
communication mode on verbal encoding. Journal of Personality and Social
Psychology, 4, 343-346.
Macken, E., Perry, J. and Haas, C. (1993) Richly grounded symbols in ASL. Sign
Language Studies, 81, 375–394.
Morford, J. P. (1996) Insights into language from the study of gesture: A review of
research on the gestural communication of non-signing deaf people. Language
and Communication, 16,165-178.
Pelli, D. G., Burns, C. W., Farrell, B., & Moore, D.C. (2003). Identifying letters. Vision
Research. (Accepted with minor revisions, full citation not yet available.)
- 47 -
Peirce, C. S (1931-58) Collected Papers of Charles Sanders Peirce, 8 vols. Edited by
Charles Hartshorne, Paul Weiss, and Arthur Burks. Harvard University Press,
Cambridge, Massachusetts.
Pierroutsakos, S. L., & DeLoache, J. S. (2003). Infant's manual exploration of pictured
objects varying in realism. Infancy, 4, 141-156.
Schober, M. F., & Clark, H. H. (1989). Understanding by addressees and overhearers.
Cognitive Psychology, 21, 211-232.
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical
Journal, 27, 379-423, 623-656.
Shillcock, R.C., Kirby, S., McDonald, S. & Brew,C. (in submission to Psychological
Review). Exploring systematicity in the mental lexicon.
Tversky, B. (1995). Cognitive origins of graphic conventions. In F. T. Marchese (Ed.).
Understanding images. (pp. 29-53). New York: Springer-Verlag.
Zaitchik, D. (1990). When representations conflict with reality: The preschooler's
problem with false beliefs and "false" photographs. Cognition, 35, 41-68.
- 48 -
Figure captions
Figure 1. Evolution of the Chinese characters for ‘woman’ and ‘gate’ illustrating the shift
from detailed iconic representations to abstract symbolic representations.
Figure 2. Six increasingly schematic depictions of cartoon from Experiment 1.
Figure 3. The set of concepts that directors communicated to matchers (used in
Experiments 1-4).
Figure 4. Examples of first and last drawings from pairs in the DD+F condition of
Experiment 1 together with their Perimetric Complexity scores. In 4a the Perimetric
Complexity scores agree with graphical complexity judgements from a human rater. In
4b and 4c they do not (see text).
Figure 5. Proportion of correct identification of drawings by matchers as a function of
experiment block for Experiment 1.
Figure 6. An example of the drawing of “art gallery” over the 4 blocks of the SD-F
condition and the 6 blocks of the SD+F condition and of “De Niro” over the 6 blocks of
the DD+F condition from Experiment1.
Figure 7. Perimetric Complexity of items drawn in the SD-F, SD+F and DD+F
conditions of Experiment 1.
Figure 8. Examples of the 7 rounds of feedback needed to identify a drawing of “De
Niro” from the first block of a pair in the DD+F condition of Experiment 1.
Figure 9. Proportion of trials with more than minimal feedback in the SD+F and DD+F
conditions of Experiment 1.
Figure 10. Proportion of items correctly identified by matchers in the NCF and CF
- 49 -
conditions across the 6 Blocks of Experiment 2.
Figure 11. Examples of drawings of “Computer Monitor” over 6 Blocks in from the CF
condition of Experiment 2,
Figure 12. Perimetric Complexity of items in the NCF and CF conditions of Experiment
2 (the complexity scores for the SD-F condition of Experiment 1 are shown for
comparison).
Figure 13. Proportion of more than minimal feedback given in the NCF and CF
conditions of Experiment 2.
Figure 14. Similarity ratings from the two judges for early (rounds 1&2), middle (rounds
3&4) and late (rounds 5&6) drawings produced in the NCF and CF conditions of
Experiment 2.
Figure 15. Identification accuracy for original matchers in the SD+F and DD+F
conditions of Experiment 1 compared with that of early ‘overseers’ (SD01/DDO1) and
late ‘overseers’ (SD02/DD02) from Experiment 3.
Figure 16. Depictions of art gallery from a pair in Experiment 1 illustrating increasing
simplicity without reduction in semantic complexity.
- 50 -
- 51 -
Block 1
Block 2
Block 3
Block 4
Block 5
Block 6
- 52 -
- 53 -
Identification Accuracy
(proportion)
1.00
0.95
0.90
0.85
SD-F
SD+F
DD+F
0.80
0.75
0.70
0.65
0.60
0.55
0.50
1
2
3
4
Block
- 54 -
5
6
Block 1 (SD-F)
Block 2 (SD-F)
Block 3 (SD-F)
Block 1 (SD+F)
Block 2 (SD+F)
Block 3 (SD+F)
Block 4 (SD+F)
Block 5 (SD+F)
Block 6 (SD+F)
Block 1 (DD+F)
Block 2 (DD+F)
Block 3 (DD+F)
- 55 -
Block 4 (SD-F)
Block 4 (DD+F)
Block 5 (DD+F)
- 56 -
Block 6 (DD+F)
3000
Complexity
2500
2000
SD-F
SD+F
DD+F
1500
1000
500
0
1
2
3
4
Block
- 57 -
5
6
Block 1 Picture 1
Block 1 Picture 5
Block 1 Picture 2
Block 1 Picture 6
- 58 -
Block 1 Picture 3
Block 1 Picture 7
Block 1 Picture 4
Feedback Trials (proportion)
0.25
0.20
0.15
SD+F
DD+F
0.10
0.05
0.00
1
2
3
4
Block
- 59 -
5
6
Identification Accuracy
(proportion)
1.00
0.95
0.90
0.85
0.80
NCF
CF
0.75
0.70
0.65
0.60
0.55
0.50
1
2
3
4
Block
- 60 -
5
6
Block 1 (CF)
Block 2 (CF)
Block 3 (CF)
Block 4 (CF)
Block 5 (CF)
Block 6 (CF)
- 61 -
3000
Complexity
2500
2000
SD-F
NCF
CF
1500
1000
500
0
1
2
3
4
Block
- 62 -
5
6
Proportion Interaction Trials
0.25
0.20
0.15
CF
NCF
0.10
0.05
0.00
1
2
3
4
Block
- 63 -
5
6
Convergence Rating (0-9)
8
7
6
NCF
CF
5
4
3
1&2
3&4
Drawings
- 64 -
5&6
Identification accuracy across conditions
Accuracy (proportion)
1
0.9
0.8
SD+F
0.7
DD+F
SD01
0.6
DD01
SD02
0.5
DD02
0.4
0.3
0.2
1
2
3
4
Block
- 65 -
5
6
Block 1
Block 2
Block 3
Block 4
Block 5
Block 6
- 66 -
Endnotes
1
Tversky includes additional factors in her production driven account such as the changes
in writing media (e.g., from stone to papyrus to hand held clay tablets) that also attended
the changes in the writing systems that she talks about.
- 67 -