Mapping Emotions in Multimodal Texts
Carminda Silvestre
Abstract
Taking as my point of departure the assumption that schools and universities must incorporate new tools to empower citizens and the need to incorporate new concepts and theories of
language as an alternative way of making sense of the world, I shall focus my attention on the
role of the teacher in constructing materials to develop students’ abilities in multiliteracies.
In order to show how written and visual language interact I shall analyse a book
written for children to show how patterns of different semiotic systems are co-articulated in
meaning-making in multimodal texts.
The purpose of the present paper is to: (i) give some evidence that Social Semiotics
and the theoretical framework of Systemic Functional Grammar are essential tools to
account for the co-articulation of the different semiotic systems used in our everyday life; (ii)
map paths of development in the literacy of reading images. Within these overall purposes,
the following specific objectives were defined: a) to identify how the image locates entities in
space; b) to identify the co-relation of the two semiotic systems of image and written language; c) to map emotions in images; d) to identify the co-relations of emotions in
multimodal texts.1
1. Introduction
The realm of the visual is increasingly controlling people’s lives through the use of webcams,
cameras in shopping centres, highways, schools, houses and all forms of visualisation from
television and the Internet to other aspects of people’s everyday lives. The amount of information that people sort out through watching and listening does not necessarily mean that
they understand what they see. Western societies are increasingly assuming a visual turn and
the role of new technologies is part of our students’ tools to get acquainted to new knowledge.
Despite this new environment, schools and universities have underestimated this reality as regards literacy.
Changes in the landscape of communication, as a result of new technologies, have
integrated modes of communication other than verbal language. Nowadays, spoken and
written language are still central modes of communication in people’s daily lives and the
educational systems are no exception. In the course of time, verbal language in its printed
form became the privileged way of carrying information, but in the last decades printed
verbal language has turned into a co-mode, creating meaning together with visual data. The
printed word in the newspaper or the magazine co-articulates with photos and other images;
Nina Nørgaard (ed.) 2008. Systemic Functional Linguistics in Use.
Odense Working Papers in Language and Communication vol. 29
(ISSN 0906-7612, ISBN: 978-87-90923-47-1)
school books and academic books are no longer written texts only, but multimodal texts
where graphs, charts, drawings and images co-occur in meaning-making.
Taking this socio-cultural context as its point of departure, and following Kress and
van Leeuwen’s (1996: 15) concern about the needs to integrate new competencies in schools
and universities and to provide a visual literacy, this paper claims that a better understanding
of the complex interplay of different elements that constitute multimodal texts (written,
spoken, visual, and other modes) is needed to help us make sense of society. We all know the
impact that images have on our values, beliefs, opinions and behaviours and their increasing
territory as texts. This way, attention must be given to the visual as well as the linguistic
meaning conveyed by multimodal texts in order to develop students’ abilities in multiliteracies and also empower them through critical reading.
Having this framework as my initialassumption, I claim that a theory of language
which respects individuals as language producers together with the appropriation of the concept of multiliteracy should be integrated in schools and universities in order to provide the
required emerging competencies. This paper reflects on the nature of verbal written language
and images and the way the systemic functional theory of language can be used to analyse
multimodal texts. In order to show how these different systems interact, I shall analyse a book
written for children so as to bring some evidence of how identities are placed in space and
how the mapping of emotions take place in the two semiotic systems.
The analysis is based on the book entitled O livro que só queria ser lido ‘The book
who only wanted to be read’, written by José Jorge Letria, a famous Portuguese writer for
children, and illustrated (oil on paper) by Daniel Silva. It consists of forty pages whose
targeted readers are children aged from 8 years old.
This is the story of a sad book which used to have a lot of readers and be fashionable,
but has ended up in a sort of loneliness and solitude on a shelf, nourishing a dream of being
read again, which means to be loved again. In fact, ‘to be read’ is its mission, its purpose for
living. The book is presented here as humanized. In its solitude it has the company of a typewriter which has also been condemned to loneliness by the growing importance of the personal computer in people’s lives. Together, the book and the typewriter have found some strategies to overcome sadness of being forgotten.
The main purposes of the present paper are to (i) give some evidence that Social
Semiotics and the theoretical framework of Systemic Functional Grammar (SFG) are essential
736
Systemic Functional Linguistics in Use, OWPLC 29, 2008
tools for analysing the co-articulation of the different semiotic systems used in our everyday
life; (ii) map paths of development in multiliteracy of reading images. Within these overall
purposes, the following specific objectives were defined: a) to identify how the image locates
entities in space; b) to identify the co-relation of the two semiotic systems (image and written
language); c) to map emotions in images; d) to identify the co-relations of emotions in multimodal texts.
2. Theoretical framework
The theoretical approach informing this study is that of Social Semiotics (Hodge & Kress
2006 [1988]; Kress & Van Leeuwen 1999 [1996]; Kress 1997, 2003) and Halliday’s (1994
[1985]) systemic functional theory of language whose view of language as a social semiotic
has been extended and applied to other semiotic resources by many followers.
Following Kress and Van Leeuwen (1999 [1996], 2001), Kress (2003), Iedema (2003),
Baldry (2000, 2006), O’Halloran (2004), to name just a few, this study focuses on the linguistic type of enquiry according to which language use is no longer theorized as an isolated
phenomenon. The theoretical framework is used to analyse how the various semiotic resources are (re)semiotised and interact with each other to make meaning.
As mentioned above, due to the changing world of new technologies, the concept of
literacy in its traditional sense is no longer satisfactory to face our everyday needs. The ability
to read and write as the traditional definition of literacy was identified in the 60s at a world
congress of Ministers of Education as being an insufficient concept; UNESCO argued that it
should be regarded as a way of preparing man for a social, civic and economic role that goes
beyond the limits of that rudimentary training (McArthur 1998: 357). From pre-schooling up
to higher education, materials have been developed which focus on spoken and written
language. Whenever teachers use audiovisual aids such as television, films or any other
media, these are explored from the traditional concept of language as support: it conveys
information, a message. In spite of this, languages have adapted to these new needs, and
expressions such as “computer literacy”, “entrepreneurial literacy”, “media literacy”, “visual
literacy”, among others, have extended the semantic domain of the concept. However, these
phrases lead to the need of knowing the basics of those areas of application. The inclusion of
visual, aural, spatial, gestural and linguistic modes of meaning in print and their articulation
are basic needs to the ever-increasing emergence of multimodal texts.
Systemic Functional Linguistics in Use, OWPLC 29, 2008
737
Before going any further, an account of what multimodality urges. This term is used to
refer to the diversity of semiotic resources of various kinds that co-occur, interplay and are
entangled in the work of textual meaning-making. The concept of text, however, is not used
in the traditional sense, but follows Halliday’s perspective of a unit of meaning materialised
by a chunk of language that is actually used for purposes of communication in a context of
situation:
We can define text, in the simplest way perhaps, by saying that it is language that is
functional. By functional, we simply mean language that is doing some job in some
context, as opposed to isolated words or sentences that I might put on the blackboard
[…]. So any instance of living language that is playing some part in a context of
situation, we shall call a text. It may be either spoken or written, or indeed in any
other medium of expression that we like to think of. (Halliday 1989: 10)
As Halliday points out, texts are not limited to the written mode of language. Texts may be
created by other semiotic resources, namely images. They can even be created without written
verbal language as is the case of the Benetton advertisements which consist of visual images
only.
Language is one of the modes through which thoughts, ideas and feelings are
represented in a culture. Images are another mode through which thoughts, ideas and feelings
are represented in the same culture. In fact, language is the privileged mode through which
we make sense of the world, yet images also represent concepts, ideas or feelings. Thus,
language is not a self-contained system of communication, but rather requires reference to
other systems to make sense of the world, namely body language and images, among many
others.
Following Kress and van Leeuwen (1999 [1996]), it is assumed that images are not
merely illustrations of texts, as they are traditionally understood, but they are texts or parts of
texts; they have their own grammar of constructing meanings, and therefore, they should be
understood as a meaning-making process and product in our semiotic system: language. The
choices people make to communicate convey meaning. Following this idea, we can explore
the way people choose images to co-articulate writing to produce texts and construct
meaning. Images are also structured, following some patterns of the written mode. I shall
focus on how the elements are compositionally brought together in the analysis.
In Halliday’s theory of language, the highly generalized functions of the linguistic
system are referred to as metafunctions. These three distinct types of meaning are classified
738
Systemic Functional Linguistics in Use, OWPLC 29, 2008
as ideational, interpersonal and textual. The ideational metafunction is concerned with construing experience, that is, language as a theory of reality, as a resource for reflecting the
world. The interpersonal metafunction is concerned with enacting interpersonal relations
through language, language as a praxis of intersubjectivity, as a resource for interacting with
others. The textual metafunction is concerned with organizing ideational and interpersonal
meaning as discourse, as meaning that is contextualized and shared; it is the ongoing creation
of a semiotic realm of reality seen as a process (discourse) or as a product (text). Since my
focus is representation, the related metafunction is the ideational.
The ideational semantic resources construe the world that is around us and inside us.
The phenomena of our experience are construed as units of meaning that can be ranked into
hierarchies and organized into networks of semantic types. The units of meaning are
structured as configurations of functions (roles) at different ranks in the hierarchy (Halliday &
Matthiessen 2006 [1999] :11). Based on this view of the metafunctions as different types of
meaning I shall explore, for example, how a clause construes experience by categorizing and
configuring it as a figure.
3. Analysis
To answer the research questions “How does image locate entities in space?”, “How do
patterns of the two semiotic systems intersect to produce meaning?”, “How are emotions
mapped in images?”, “How are the representations of the two semiotic systems co-related in
the multimodal text?” I shall be looking at two analytical tools (i) the location of entities in
space and (ii) mapping emotions as representation.
3.1 Locating entities in space
The title of the book O livro que só queria ser lido ‘The book who only wanted to be read’ is
an ontological metaphor (THE BOOK IS A LIVING BEING with feelings). Lakoff and
Johnson (1980) name this type of metaphor ‘personification’. As a living being, the book has
emotions that are explored and this metaphor pays tribute to the book and to reading, which
conveys the essential truth that books are part of our lives and our quest for knowledge.
The way people locate objects with respect to one another involves the recognition of
some kind of asymmetrical relation between the object people want to locate and the object
with respect to which it is located. There are choices that must be made. Asymmetrical
Systemic Functional Linguistics in Use, OWPLC 29, 2008
739
relations may be established with respect to position, order, distance, size, or other types. The
entity, the represented participant of the story (the book) is displayed on the page as a
potential sign, which together with other signs can provide access to potential meanings by
specifying possible reading paths. The semiotic resources used are images and printed words.
The book is not the only represented participant of the story, but it is the most important
participant. As such, it is placed on the cover, assuming its level of importance.
The layout of the cover materialises the textual metafunction. The printed page is a
visual spatial unit of textual organisation. In this full-page visual display (the cover), shown
in Figure 1, two semiotic systems work together to produce the overall meaning. Visual
prominence is given to the book located on the left-hand side of the first shelf.
Figure 1: cover (visual and title)
Considering the information structure, this spatial arrangement of the book and its relation to
the shelves shows that the book is represented as Given, i.e., known, taken for granted in both
the image and the title, fulfilling the left-right order. It is the most important element in the
visual. The same is true for the title, as follows:
740
Systemic Functional Linguistics in Use, OWPLC 29, 2008
(Port.) O livro que
só queria
ser lido
GIVEN (The book)
Table 1: Title of the book
(Engl.) [The book who
only wanted
to be read] (my translation)
NEW (who only wanted to be read)
Attention is concentrated on this first element, the book. This information (Given) is usually
found at the beginning of a clause. The other information (New) is the focus of the message,
what this book is about: que só queria ser lido ‘who only wanted to be read’. The placement
of the printed words follows the structural left-right order, but graphically chooses the topdown hierarchical order, following the “vertical elongation”, as Kress and van Leeuwen
(1999 [1996]) put it, where the most important goes on top.
Secondly, the fact that the book is positioned on the first shelf and there are other
shelves that follow the one it is placed on, suggesting a hierarchical construal of the image,
resembling the orientation of the page as a top-down order. In fact, the location of the identity
is suggested by the verbal text co-patterned with the image and mentioned on page 7 that the
book found its place on a high shelf of the bookcase next to the desk, where the computer is
now king and sovereign. Era o caso deste livro, que encontrara o seu pouso certo numa
prateleira alta de uma estante colocada ao lado da secretária, onde agora era rei e senhor o
computador. The reader will perceive a similarity of visual patterning in the image and the
written language. However, the spatial placement of the book is verbally expressed in an
ampler context than the visual arrangement of location. Focusing the analysis on this specific
topic, the deleted depiction of the image seems to be a semiotic element where reduction for
purposes of spatial arrangement is at issue. To sum up, spatial arrangement of entities may be
described linguistically in a number of ways, each expressing the speaker’s or the writer’s
choices. The same is true for images. The construal of the spatial arrangement is a choice the
illustrator makes in order to (re)present this semiotic part of the text in a process of
resemiotization and of meaning production. Iedema (2003) considers that this concept is more
consistent with multimodal texts in a twofold way (i) it traces how semiotics are translated
from one into the other as social processes (ii) it asks why these semiotics rather than others
are mobilized to do certain things at certain times.
The book is not the only represented participant in the story. It is the most important,
but there is a typewriter that suffers from the same problem, being relegated to the second
rank now that the personal computer is now king and sovereign.
Systemic Functional Linguistics in Use, OWPLC 29, 2008
741
Language is a means of representing the world. As mentioned before, this is the
ideational metafunction, the clause as representation. Looking more closely at the linguistic
representation of Table 2, there are two entities involved in the process. In SFG terms,
relational processes are those which encode meanings of being. This type of process involves
establishing a relationship between two terms – in this case, the two represented entities ‘the
book’ and ‘the typewriter’. The relation established between these two entities is one of
possession. This is encoded through the process. ‘The book’ is the Carrier, the possessor,
‘had’ is the process, it is a common attributive possessive verb, and ‘the typewriter’ is the
Attribute, the possessed.
(Port.)
A única companhia com que o livro podia
contar era a de uma velha máquina de
escrever que já tivera, naquela casa, a sua
época e a sua utilidade.
(Eng.)
[the only companion that the book could
count on with was the old typewriter that
had already had its time and usefulness in
that house.] (my translation)
Table 2: The book had the typewriter as a companion
According to Halliday and Matthiessen (2006 [1999]: 52) a figure is a representation of
experience in the form of a configuration, consisting of a process, participants taking part in
this process and associated circumstances. They add that there are indefinitely many kinds of
process in the non-semiotic world, but these are construed semiotically, according to the way
in which they configure participants into a small number of process types – being, doing,
742
Systemic Functional Linguistics in Use, OWPLC 29, 2008
sensing, and saying. In fact, the book, an anthropomorphized non-human, is mostly represented as Senser and Carrier where emotions and relations are displayed.
On the other hand, the visual representation of the two represented participants
follows the written mode as can be observed in Table 2. “Horizontal elongation” (Kress &
Van Leeuwen 1996) is materialized where the book is presented as “Given”, as information
that is already familiar to the reader and serves as a point of departure for the message,
whereas the typewriter is presented as “New”, as information not yet known to the reader, and
deserving his or her special attention, this role of affective possessed.
Thus, a possible spatial relation in terms of position between a book and a typewriter is
that of the book being on the shelf and the typewriter is on the table. Following both the position
and order relation, the book is placed on the left-hand side of the shelf: it has a Senser position
whereas the typewriter is placed on the right. When specifying the kind of spatial relation between these two entities (book and typewriter), the book is perceived as having a more important role in the image. The visual structuring is then organised in ways similar to the written
structure to create meaningful propositions by means of visual syntax. Thus, when comparing
the two semiotic media of creating meaning, both follow the left-right orientation of the sentence.
Typically typewriters are bigger than books, satisfying the asymmetry condition of size.
The spatial arrangements of these two entities are here presented as fairly symmetrical in terms
of size, following, I would suggest, their relational configuration of friendship symmetry.
3.2 Mapping emotions in images
As mentioned in the first part of this article, this is a forty page story about a book, O livro
que só queria ser lido ‘The book who only wanted to be read’. The story begins with ‘Once
upon a time there was a sad book.’. In fact, this is a story about affections. The title presents
the book as a Senser. He [the book] feels sad, alone and neglected by a family who does not
read it anymore and together with a typewriter, he makes some strategies to forget their loneliness. The first thirty-eight pages of the book convey this atmosphere. It is only on page thirty-nine that the happy ending occurs. Looking at the different images of the book, the same
atmosphere is perceived. Feelings are polarized between sadness and happiness. Most of the
emotional occurrences are centred on negative feelings such as sadness, solitude and loneliness, both linguistically and through the imagery. In spite of that, the story ends well. When
Systemic Functional Linguistics in Use, OWPLC 29, 2008
743
looking at Figure1, Table 2 and Table 4 we can guess how the book ends: it has a happy ending.
Focussing this part of the analysis on Attitude (Martin 2008), I shall look at the mapping of emotions in images and try to identify the co-relations of emotions in multimodal texts.
Martin (2008) argues that Affect deals with resources for construing emotional reactions. It is
concerned with registering positive and negative feelings, that is, whether we feel sad or
happy, confident or anxious. These are located both linguistically and through images.
Regarding body language as a mode of communication, and understanding it as
discourse, i.e. as social practice, the configuration of some properties (arms, legs and eyes
projected to the earth; top down position) shown in Table 3 makes us perceive the images as
a book expressing sadness. The different parts of the body of the book are perceived as
consistent with certain traits (gestures) that grouped together give a certain configuration of
properties, which have been instantiated through frequency and time in our perceptions of the
world. These properties constitute patterns of behaviour adopted by the community and are
therefore part of our own practices (Hasan 2004: 18). The role of orientational metaphors is
an important tool to instantiate Attitude in images.
Parts of the body
(image)
Parts of the body
(description)
Verbal language:
Attitude
Eyes: looking down
Era uma vez um livro triste.
[Once upon a time there was a
sad book.]
Shoulders/arms: Down
Sentiu-se ainda mais só, triste
e esquecido o pobre livro.
[(It) felt even more lonesome,
sad and abandoned the poor
book.]
Legs: Down
Porque me sinto muito só e
triste…
[Because I feel very lonesome
and sad…]
Table 3: The representation of sadness
Orientational metaphors display a high relevance of spatial and physical orientation, which is
triggered by the image schemata that underlie them. Typical examples are HAPPY IS UP and
744
Systemic Functional Linguistics in Use, OWPLC 29, 2008
SAD IS DOWN. The prepositions “up” and “down”, for example, are well-known examples
that provide a vast cluster of associations around them, relating strong bias in favour of “up”
and opposite ideas in favour of “down”. Table 4, on the other hand, provides certain traits that
convey the opposite meanings.
Looking at the last picture of the book, Table 4, people can guess the happy ending.
The book is open. Legs and arms seem to express reward and glory. Emotions are near the
state of happiness. The eyes, wide open, are looking up.
(Port.) E se houvesse na casa uma tabela dos
mais procurados, como há nas livrarias, o
livro teria ficado várias semanas seguidas no
primeiro lugar, sem rival à vista. E bem
merecia esse momento de reconhecimento e
consagração.
(Eng.) ‘And if, at home, there was a “top 10”
table as we find in bookshops, the book would
have stayed several weeks in the first place,
without any competitor. Moreover, it would
certainly deserve that moment of reward and
glory.’ (my translation)
Table 4: The last picture of the book (reward and glory)
The different parts of the body of the book, shown in Table 5 are perceived as consistent with
certain traits (gestures) that, grouped together, give a certain configuration of properties of the
representation of positive and high value emotions. As mentioned before, these have been
instantiated through frequency and time in our perceptions of the world. Such properties
constitute patterns of behaviour adopted by the community and are therefore part of our own
practices, as the pictures seem to express.
Systemic Functional Linguistics in Use, OWPLC 29, 2008
745
Parts of the body
(image)
Parts of the body
(description)
Verbal language
Eyes: looking up or straight up
(Port.) E bem merecia esse
momento de reconhecimento
e consagração.
Shoulders/arms: up
(Eng.) It [the book] would
certainly deserve that moment
of reward and glory.
Legs: crossed straight up
Table 5: The representation of reward and glory [happiness]
It is the intersection and co-articulation of the two semiotic systems that contribute to the
production of meaning-making. The patterns of body language identified in Table 5 are a
resemiotization of verbal language that the illustrator chose for the representation of this high
value emotion.
Hypothesizing about possible connections between semiotic visual features and written
language I wonder, for example, whether body language features materialised through
metaphors are complementary to Appraisal and whether they constitute a system on their own.
4. Final remarks
Aware of the importance of the power of media, of images and of graphs, to name only a few
resources, in today’s society as influencing people how to think and what to believe, it is
crucial to empower students with tools to think alternatively by using language as a social
practice.
746
Systemic Functional Linguistics in Use, OWPLC 29, 2008
The theoretical framework of Social Semiotics and SFG are specifically geared to
relating different semiotic resources that are part of multimodal texts. They provide tools to
afford literacy in reading images and insights about the co-articulation of the different
semiotic systems used in our everyday life.
The two entities, writer and illustrator, have created the book The Book that only
wanted to be read as a universe of meaning production together with their readers. The book
privileges the written language medium. Not everything that is realized in verbal language is
realized by means of images as has been demonstrated above. In the section ‘locating entities
in space’ the deleted depiction of the image seemed to be a semiotic element where reduction
for purposes of spatial arrangement was used. However, certain similarities of visual patterning in the image and the written semiotic systems were identified, namely the spatial arrangements of entities. Considering the information structure, the spatial arrangement of the
identities in space showed that both the written and the visual semiotic resources followed the
top-down and left-right orientation. As regards the section ‘mapping emotions in images’,
emotions were (re)presented in a polarized way. The abstract entities ‘sadness’ and ‘happiness’ were represented in the written resource. The Affect, i.e. how the book felt, was resemiotized by means of gestual elements that were compositionally brought together as metaphors, entailing certain configurations of properties of the representation of negative and positive emotions. The images encode a universe of emotions felt by a humanised book that are
polarized from sadness to happiness, or sort of happiness, as its achievements were accomplished.
Carminda Silvestre, Adjunct Professor
Language Science Department
Polythecnic Institute of Leiria, School of Technology and Management, Portugal
[email protected]
Note
1
I would like to thank José Jorge Letria, the writer, and Daniel Silva, the illustrator, for permission to
use copyright material.
Systemic Functional Linguistics in Use, OWPLC 29, 2008
747
References
Baldry, A. (ed.) (2000). Multimodality and Multimediality in the Distance Learning Age.
Campobasso, Italy: Palladino Editore.
Baldry, A. & Thibault, P. J. (2006). Multimodal Transcription and Text Analysis. London:
Equinox.
Hall, S. (ed.) (1997). Representation: Cultural Representations and Signifying Practices.
London: Sage Publications.
Halliday, M.A.K. (1989). “Part A” in M.A.K. Halliday & Ruqaiya Hasan (eds.) Language,
Context and Text: Aspects of language in a social-semiotic perspective. Oxford:
Oxford University Press: 1-49.
Halliday, M.A.K. (1994 [1985]). An Introduction to Functional Grammar. London: Arnold.
Halliday, M.A.K. & Matthiessen, C. (2006 [1999]). Construing Experience Through
Meaning: A Language-based Approach to Cognition. London: Continuum.
Hasan, R. (2004). “Analysing Discursive Variation” in Lynne Young & Claire Harrison (eds.)
Systemic Functional Linguistics and Critical Discourse Analysis. London: Continuum: 15-52.
Iedema, R. (2003). “Multimodality, resemiotization: extending the analysis of discourse as a
multi-semiotic practice”. Visual Communication, 2(1): 29-57.
Kress, G. & Van Leeuwen, T. (1999 [1996]). Reading Images: The Grammar of Visual
Design. London: Routledge.
Kress, G. & Van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of
Contemporary Communication. London: Arnold.
Kress, G. (1997). Before writing: Rethinking the paths to literacy. London: Routledge.
Kress, G. (2003). Literacy in the New Media Age. London: Routledge.
Lakoff, G. & Johnson, M. (2002[1980]). Metáforas da vida cotidiana. São Paulo: EDUC.
Letria, José Jorge (2006). O livro que só queria ser lido. Lisboa: Texto Editores.
Martin, J. (2008). Appraisal. Lisbon Workshop.
McArthur, T. (1998). Oxford Concise Companion to the English Language. Oxford: Oxford
University Press.
O’Halloran, K. (ed.) (2004). Multimodal Discourse Analysis: Systemic Functional Perspectives. London: Continuum.
748
Systemic Functional Linguistics in Use, OWPLC 29, 2008