Gestural Imagery in the Service of Musical Imagery
Rolf Inge Godøy
Section for musicology, University of Oslo
P.O.Box 1017 Blindern, N-0315 Norway
[email protected]
tel. (+47)22854064, fax. (+47)22854763
Abstract. There seem to be strong links between gestural imagery and musical
imagery, and it is suggested that gestural imagery can be instrumental in triggering and sustaining mental images of musical sound. Gestural images are
seen as integral to most experiences of music, and several practical and theoretical musical disciplines could profit from focusing on these gestural images.
Research in support of this is reviewed, and some topics for future research are
presented.
1 Introduction
The topic of this paper is how gestural imagery, meaning imagining or mentally simulating various gestures, can trigger, sustain, and enhance images of musical sound in
our minds. This is part of a long-term research project, Motor-mimesis, which aims at
enhancing our means for thinking musical sound in various tasks (composing, arranging, improvisation, performance, analysis, music education, etc.) through mental
images of sound-associated actions, including both sound-producing actions such as
hitting, stroking, plucking, bowing, blowing, etc., and other sound-related body
movements such as dancing and various kinds of sound tracing gestures.
In the call for papers prior to a conference on musical imagery some years back,
we tentatively defined musical imagery as 'our mental capacity for imagining musical
sound in the absence of a directly audible sound source, meaning that we can recall
and re-experience or even invent new musical sound through our 'inner ear'.' [1]. It
could now be useful to propose a similar definition here of gestural imagery as 'our
mental capacity for imagining gestures without seeing them or actually carrying them
out, meaning that we can recall and re-experience or even invent new gestures
through our 'inner eye' and inner sense of movement and effort.' It must be emphasized that imagery is not just a matter of daydreaming or arm-chair contemplation, but
is in fact the very basis for thinking and feeling: Memory, and hence, imagery, is at
work in all perception and cognition, and, as pointed out already by the phenomenologists at the end of the 19th century [2], there simply would be no perception and
cognition at all without mental images of past and expected (future) events.
Although there has been a number of studies of musical imagery during the last
decades [3], the most intriguing, and also most practically oriented question is in my
opinion that of what actually triggers and sustains images of musical sound in our
minds. This question of what is the 'engine' of musical imagery has oddly enough
A. Camurri and G. Volpe (Eds.): GW 2003, LNAI 2915, pp. 55–62, 2004.
© Springer-Verlag Berlin Heidelberg 2004
56
Rolf Inge Godøy
been given little attention by researchers, however, the prime suspect for this triggering agent which emerges from various studies of musical imagery, is actually gestural
imagery. In other words: Images of gestures seem to be efficient in evoking images
of sounds. The challenge now is to substantiate this claim, both by providing evidence from available research reports on the neurophysiological and cognitive bases
for the gesture-sound interaction in imagery, and to design schemes for practical applications which can demonstrate that imagining gestures is actually an efficient strategy for evoking lucid and vivid images of musical sound in various music related
tasks.
2 Gesture-Sound Links in Perception and Cognition
Many musicians are familiar with the experience that recall of music is facilitated by
mimicking the sound-producing movements or other kinds of movements associated
with a musical work, e.g. moving hands and fingers as if actually playing when recalling a piano piece, playing ’air guitar’ or ’air drum’ when recalling a song, making
dance movements when recalling a dance tune, etc. In other words, there can hardly
be any doubt that gestures are effective in triggering images of musical sound, something which has in fact been documented through a series of experiments by Mariko
Mikumo [4], and something which has been remarkably depicted by David Sudnow
in his introspective account of jazz improvisation [5].
Generally, imagery research in any domain, be that visual, auditive, motor, etc.,
has inherently significant methodological problems because it is difficult to study
what goes on within the mind, i.e. we can not place an 'observer' in people’s minds to
register what is going on when people imagine various scenes, gestures and sounds.
However, the last couple of decades has seen the emergence of some clever methods
for capturing what is assumed to go on in processes of imagery, such as measuring
reaction times, and effects of mental preparation and rehearsal, for various tasks [6].
In addition to such indirect methods, advances in methods for non-invasive neurophysiological observation, such as fMRI, PET and ERP [7], have given us more information on the workings of mental imagery. One significant finding here is that of
'functional equivalence', meaning the close resemblance between the neuronal apparatus involved in actual actions and/or perceptions and in the imagined actions and/or
perceptions. One possible interpretation of this could be that imagery does share
many features with actual experience, hence, that studying the links between gestures
and sound in actual experience could also give us clues as to the links between gestures and sound in imagery. Also, another interesting finding here is that certain motor areas of the brain seem to be activated when imagining sound, hence that action
imagery is activated concurrently with tasks of musical imagery.
Neurophysiological research will in the near future no doubt come up with interesting findings on gesture-sound links in imagery. However there is also a considerable amount of existing audition research which suggests close links between gesture
and sound in perception and cognition. This research could be summarized as 'ecological' in the sense that its basic paradigm is to understand human audition as embedded in evolutionary constraints, hence, as trying to understand human audition as a
holistic and cross-modal phenomenon where the different sense-modalities cooperate
in order to extract meaning from what we hear in our environment. There are many
Gestural Imagery in the Service of Musical Imagery
57
instances of this in the domain of Auditory Scene Analysis [8], and just to mention
one classic example, the so-called ’McGurk effect’ demonstrates that we may be
tricked when seeing a certain sound-producing gesture to believe that we have heard
something else than we actually have been presented with in the acoustic signal [9].
However, the most persistent project to explore and exploit gesture-sound links is
that of the so-called ’motor theory’ of perception [10]. Going back more than three
decades, the advocates of this theory have claimed that the various features in the signal alone are not sufficient to account for perception and discrimination of language
sounds, and that the listener actually mentally mimics the sound-producing gestures
when trying to make sense of what is spoken, projecting gestural images on to the
sound input in a top-down manner. The motor theory thus claims that perception involves a gestural simulation in the mind of the listener, and that learning to understand a language is actually a process of learning to imagine the sound-producing
gestures of that particular language. The motor theory has been controversial. However, the above-mentioned advances in neurophysiological research is now increasingly giving support for this theory [11].
Ideas from motor theory are fundamental here, not only for exploring gesturemusic links, but also for structuring our images of musical sound as gesture units (see
section 5 below). In this connection, research into the role of gestures in language is
also relevant [12], as this indicates how gestures not only amplify certain rhetorical
elements of speech, but suggests that gestures may have been the evolutionary basis
for spoken language [13], and may be instrumental in actually generating utterances
[14], i.e. that gestures in speech may play a cognitive role similar to the role of gestures in music. As to this last point, there is of course also a fast growing body of
research into practical applications of gesture control in music [15], and experiences
here can no doubt tell us more about the links between gestural imagery and musical
imagery as well.
3 Separating Gestures and Sound
Although the classical separation of sense-modalities such as vision, hearing, balance,
etc. may now seem questionable from what is known about the cooperation of many
channels of sensory input in the brain [16], it will for strategic reasons be useful to
conceptually separate gestures and sound here [17]. This means separating images of
what we do from images of the effects of what we do, or separating the silent choreography of sound-producing and sound-accompanying actions from sonorous images.
This means furthermore that we separate ecological knowledge of actions that we all
have accumulated since our births (or even from a pre-natal stage), i.e. knowledge of
action-trajectories and effort in stroking, blowing, hitting, kicking, etc., from ecological knowledge of resonant features of whatever sounding body (strings, tubes, vocal
tract, etc.) and environment (rooms, ambient sounds, transducing medium, etc.) that
we similarly have experienced by being in the world [18, 19].
First of all, this separation of gestures and resultant sound reveals quite clearly that
what we often (or in most cases) think of as the meaning or content of music is actually a matter of gestural images: Images of effort, velocity, contours, trajectories, gait,
etc., could all be understood as gestural phenomena, as gestural images transmitted by
sound and ’decoded’ in listening back to a gestural language as in ballet or pantomime
58
Rolf Inge Godøy
[20]. For example, when listening to the ferocious beating of drums, it is probably
almost impossible to avoid having images of an equally ferocious activity of hands
and mallets hitting the drum membranes, and conversely, listening to a slow, quiet,
and contemplative piece of music would probably evoke images of slow and smooth
sound-generating gestures. For disciplines such as musical aesthetics and musical
semiotics, a focus on gesture could be very useful to characterize musical meaning.
In our context here, such a shift of focus reveals that gestural scripts are deeply embedded in what we think of as musical imagery, making the ’pure’ sound a kind of
’residue’ which is left over once we remove gestural images. The consequences of
this is that a cultivation of gestural images is not only legitimate, but can have some
very attractive features in practical applications as well (see section 5 below).
Secondly, this separation of gesture and sound may encourage us to understand
music as organized not only by traditional principles for pitch, harmony, consonancedissonance, motives, melodies, thematic development, etc. but equally well by principles for motor cognition, i.e. action-units or action-gestalts, motor programmes,
motor-hierarchies, coarticulation, etc., as I shall briefly present in the next section.
There can be no doubt that human motor cognition has a number of constraints on
music making, e.g. need for alternating between effort and rest, need to conserve
energy, thresholds for velocity, phase transitions with changes in tempi [21], etc., and
that human motor cognition exerts schematic constraints on melodic, rhythmic and
textural grouping [22, 23].
Thirdly, separating gesture and sound opens up the fascinating domain of motor
imagery [24], hence, opens up for thinking musical imagery as simulation of gestures.
This in turn opens up for all the advantages inherent in the dynamics of motor imagery, such as imagining actions at variable resolution (mentally fast running through
a sequence of actions or slowly replaying all the details) and variable acuity (vague,
approximate, sketch-like recollections of actions and effort or very precise, salient
images of trajectories). Musical sound can not (in principle) unfold faster than what it
does without distorting its features, but motor programmes may create compressed
overview images of gestures in the form of scripts or lists of highlights, similar to
trailers in the promotion of films where the point is to pack as many salient scenes as
possible (often the most spectacular and/or violent) into a 30 second advertisement. In
fact, scores in common practice western notation are partially such gesture scripts,
allowing for rapid running through or ’at a glance’ rough impression of what the music
is like (making some suspicious conductors prefer a quick glance at a score rather
than spending time on actually listening through a work).
This separation of gesture and musical sound prepares the ground for talking about
mental simulation as the substance of musical imagery, meaning that musical imagery
is not a kind of abstract representational or propositional system, but is a matter of reenacting from a first-person, egocentric perspective what we have perceived in the
world. This simulation view of cognition [16, 25] seems fortunately now to be gaining plausibility, and means that whatever kind of music we imagine, and regardless
our level of expertise and/or the complexity of the music, there is always the possibility of mentally mimicking gestures we think belong to the music (although perhaps
only in a low-acuity, sketchy manner).
Gestural Imagery in the Service of Musical Imagery
59
4 Motor Cognition
Accepting that gestural images are integral to musical imagery, it could be useful to
have a brief look at some principles of motor cognition [26]. There are physiological
constraints on human action which are obvious in musical situations (e.g. singers,
brass and woodwind players need to breathe, velocity of bow, hand and finger movements are limited within a certain range, etc.), but most of all, there are some cognitive constraints and principles of organization which are highly relevant for gestures
in music, hence also for gestural imagery in music.
The most important concept here is that of motor programmes. A motor programme is a mental image of an action or a sequence of actions, such as opening a
door, sitting down, walking to work, or hitting a drum, playing a tune on the flute, or
playing a long piece on the piano. A motor programme is thus a kind of script for
how something is to be done, and this script may, as mentioned above, have a variable resolution, e.g. I can very quickly envisage the walk to my place of work, or I
can go through the entire trajectory quite slowly, envisaging every little part of the
trajectory and how my feet will be moving at any point of time. This variable resolution is interesting for musical purposes, as it shows how it is possible to quickly run
through the gestural script of some music, or to go through it very slowly, focusing on
every single detail.
Also, motor programmes are flexible and allow for alternatives when needed. For
instance, I can push open the door with my shoulder or my foot when I am carrying
something in my hands, or to take a more musical example, I can play a melody on
the piano with my left hand if I have injured my right hand. This possibility of alternative executions is called motor equivalence, and is important for music cognition as
it may be a model for understanding how alternative renderings of similar musical
ideas are possible. The idea of motor equivalence could allow for generalizations of
musical gestures, e.g. a rapid glissando on a harp would be quite similar to a rapidly
played series of tones on a piano, although in detail the actions would be rather different in these two cases.
Furthermore, motor programmes are hierarchical in the sense that they provide
overviews of entire actions, i.e. provide executive control, yet at the same time allow
for control of detail through ’sub-routines’. With practice and increased level of skill
in various tasks, there is a tendency to group actions into larger units, i.e. into chunks,
making detail actions automatic and attention to these details unnecessary. These
action-based groupings are significant in musical contexts, and it can be shown that
rhythmical groupings often follow these principles of action-chunking [22, 23]. Furthermore, chunkings of action seem also to have a basis in physiological optimalization, meaning that with practice, there is usually a tendency towards maximum efficiency or energy conservation, resembling a mass-spring model with
alternations between peaks effort and phases of relaxation.
Related to chunking of musical sound on the basis of physiological and cognitive
optimalization, is the phenomenon of coarticulation, meaning that several otherwise
separate actions are fused or embedded into larger scale action-units. The concept of
coarticulation was probably first used in linguistics to denote the preparation of the
next phoneme by shaping the vocal apparatus in advance, but it has now also been
applied to other areas of human movement, such as when my arm is moving towards
an object to pick it up, my fingers simultaneously move to anticipate the optimal way
60
Rolf Inge Godøy
of grasping the object [27]. The occurrences of coarticulation in music are numerous,
as in phrasing, articulation, and even in intonation, and naturally occurring coarticulation in vocal and instrumental music is in fact that which makes naturally sounding
melodic phrases on sampled wind, brass and string instruments practically impossible.
5 Advantages and Applications of Gestural Imagery
A focus on gestural imagery in music will have many advantages. As I have tried to
suggest, the gestural component is a significant component of musical listening in
general, and deserves much more attention as a carrier of meaning in studies of musical aesthetics, semiotics, analysis, performance, music education, etc., and it could be
tempting to speak of a ’gestural hermeneutics’ in musical thinking, meaning gesture as
the mediating element between music and other modes of thought. Furthermore, gestural imagery could also shed light on enigmatic issues in music cognition and aesthetics such as temporality and experience of musical form through the possibility of
thinking gestures at various levels of resolution as gesture scripts. Actually, a cultivation of gestural imagery could be seen as a contribution to phenomenological approaches to music theory, related to the project of Pierre Schaeffer [28], with mental
gestural tracing or sketching of the shapes of sound objects as an extension of
Schaeffers typo-morphological characterizations. Furthermore, gestural images may
also help us understand categorization in music because gestures can be easily recognized yet at the same time be variation-tolerant in detail, meaning there can be variant
versions of a gesture such as in hitting a drum, yet the gesture retains the general
feature of a sudden, ballistic movement. This possibility of categorizations by actions
was suggested by Elanor Rosch and co-workers [29], and is similarly reflected in the
motor theory of perception [10].
Finally, gestural imagery can also have some more specific, practical applications,
and now just to mention some:
• Thinking melodic contours as gestures, hence, as holistic entities and not as collections of pitches, and thus enhance our understanding of melodic phenomena.
• Thinking musical textures as complex, multi-dimensional patterns of gestures, all
seen from the egocentric, first-person perspective, and thus help us grasp the workings of different texture types.
• Thinking orchestration (and score reading) as similarly complex, multidimensional patterns of gestures, however with the addition of the resultant sound
from these gestures, generating more lucid predictions of how scores are going to
sound.
• Providing a better basis for music education in general, and ear-training in particular, by specifically cultivating the musical imagery triggering capabilities of gestural imagery.
• Helping to understand and master rhythmical grouping, expressivity, and phrasing
in musical performance by mental practice of gesture chunks.
• Helping to understand the mental and motor processes of improvisation, i.e. how it
is possible to simultaneously have images of the immediate past and images of
what is to come in the immediate future, by simultaneously thinking different gestural scripts.
Gestural Imagery in the Service of Musical Imagery
61
6 Conclusion and Ideas for Further Research
Altogether, there is reasonable grounds for assuming that there are quite close links
between gestural images and images of musical sound, and it is easy to see how these
links could be exploited in a number of theoretical and more practical challenges in
music. Needless to say, there are a large number of unanswered questions here, and
in the overseeable future, research within the following domains could be useful to
substantiate our knowledge of the links between gestural and musical imagery:
• Compiling relevant data from existing observational (non-invasive) neurophysiological studies. There is a fast growing body of research here, and increasingly
so, there seems to be much interest in the subjects of motor imagery and mental
simulation.
• Gathering data on performance gestures, meaning that we need to know more
about both the physiological and cognitive constraints and schemata which are at
work in performance gestures, as well as developing suitable representations for
this data.
• Physical model synthesis with incremental changes of the excitatory gestures followed by listening judgements, i.e. an ’analysis by synthesis’ approach to studying
the relationship between gestures and timbral qualities.
• Various kinds of experiments on the links between gestural and musical memory,
e.g. along the lines of [4], and on the priming and mental practice effects of gestural imagery in practical tasks such as orchestration, improvisation and performance.
All through this gathering of data from various sources, the unifying principle here
should be that of mental simulation, i.e. that of understanding perception, cognition
and imagery in music as an active, embodied re-enactment of the gestures we believe
belong to musical sound.
References
1. Godøy, R. I., Jørgensen, H. (eds.): Musical Imagery. Swets and Zeitlinger, Lisse (Holland)
(2001) p. ix
2. Husserl, E.: Vorlesungen zur Phänomenologie des inneren Zeitbewusstseins. Max Nimeyer Verlag, Tübingen (1980)
3. For an overview, see Godøy, R. I., Jørgensen, H. (eds.): Musical Imagery. Swets and Zeitlinger, Lisse (Holland) (2001)
4. Mikumo, M.: Encoding Strategies for Pitch Information. Japanese Psychological Monographs No. 27 (The Japanese Psychological Association) (1998)
5. Sudnow, D.: Ways of the Hand. Harvard University Press, Cambridge, Mass. (1978)
6. Kosslyn, S. M.: Image and Brain. The MIT Press, Cambridge, Mass. (1994)
7. Janata, P.: Neurophysiological Mechanisms Underlying Auditory Image Formation in Music. In: Godøy, R. I., Jørgensen. H. (eds.): Musical Imagery. Swets and Zeitlinger, Lisse
(Holland) (2001) pp. 27-42
8. Bregman, A. S.: Auditory Scene Analysis. The MIT Press, Cambridge, Mass. & London
(1990)
9. McGurk, H., MacDonald, J.: Hearing Lips and Seeing Voices. Nature, 264 (1976) 746-748
62
Rolf Inge Godøy
10. Liberman, A. M., Mattingly, I. G.: The Motor Theory of Speech Perception Revised. Cognition, 21 (1985) 1-36
11. Fadiga, L., Craighero, L., Buccino, G., Rizzolatti, G.: Speech Listening Specifically
Modulates the Excitability of Tongue Muscles: a TMS Study. European Journal of Neuroscience, Vol. 15 (2002) pp. 399-402
12. McNeill, D.: Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press, Chicago, IL (1992)
13. Rizzolatti, G., Arbib, M. A.: Language Within Our Grasp. Trends in Neuroscience Vol. 21
(1998) 188–194
14. Kita, S.: How representational gestures help speaking. In: McNeill, D. (ed.): Language and
Gesture. Cambridge University Press, Cambridge (2000) pp. 162-185
15. Cadoz, C., Wanderley, M.: Gesture-Music. In: Wanderley, M. M., Battier, M. (eds.):
Trends in Gestural Control of Music (CD-ROM). IRCAM, Paris (2001) pp. 28-65
16. Berthoz, A.: Le sens du mouvement. Odile Jacob, Paris (1997)
17. Godøy, R. I.: Imagined Action, Excitation, and Resonance. In: Godøy, R. I., Jørgensen. H.
(eds.): Musical Imagery. Swets and Zeitlinger, Lisse (Holland) (2001) pp. 239-252
18. Freed, D. J.: Auditory Correlates of Perceived Mallet Hardness for a Set of Recorded Percussive Sound Events. Journal of the Acoustical Society of America, 87 (1990) 311-322
19. Handel, S.: Timbre Perception and Auditory Object Identification. In: Moore, B. C. (ed.):
Hearing. Academic Press, San Diego (1995) pp. 425-461
20. Laban, R.: The Mastery of Movement (fourth edition by Lisa Ullmann). Northcote House
Publishers Ltd., Plymouth (1980)
21. McAngus Todd, N. P., O'Boyle, D. J., and Lee, C. S.: A Sensory-Motor Theory of
Rhythm, Time Perception and Beat Induction. Journal of New Music Research,Vol. 28,
No. 1 (1999) pp. 5-28
22. Godøy, R. I.: Knowledge in Music Theory by Shapes of Musical Objects and SoundProducing Actions. In M. Leman (ed.): Music, Gestalt, and Computing. Springer Verlag,
Berlin (1997) pp. 106-110
23. Godøy, R. I.: Cross-modality and Conceptual Shapes and Spaces in Music Theory. In I.
Zannos (ed.): Music and Signs. ASCO Art & Science, Bratislava (1999) pp. 85-98
24. Jeannerod., M.: Mental Imagery in the Motor Context. Neuropsychologia, Vol. 33, No. 11
(1995) 1419-1432
25. Dokic, J. & Proust, J. (eds.): Simulation and Knowledge of Action. John Benjamins Publishing Company, Amsterdam/Philadelphia (2002)
26. Rosenbaum, D. A.: Human Motor Control. Academic Press, Inc, San Diego (1991)
27. Rosenbaum, D. A., Loukopoulos, L. D., Meulenbroek, R. G. J., Vaughan, J., Engelbrecht,
S. E.: Planning Reaches by Evaluating Stored Postures. Psychological Review, Vol 102,
No. 1 (1995) 28-47
28. Schaeffer, P.: Traité des objets musicaux. Éditions du Seuil, Paris (1966)
29. Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., Boyes-Braem, P.: Basic Objects in
Natural Categories. Cognitive Psychology, 8 (1976) 382-436