Course book for COGS 180: Gesture Bodo Winter Ch. 2 — Introductory semiotics and ways of classifying gestures To understand gesture, we first have to understand some of the basic principles of meaning-­‐making. Semiotics is the study of signs and sign processes. Semiotics should not be confused with semantics, the study of linguistic meaning. For example, semantics might be concerned with how the word bark means both “dog sound” and “outer surface of a tree”. Semiotics, on the other hand, is the most abstract way one can study signs, focusing on the general ways with which systems—including non-­‐human systems—communicate. To give you an example, imagine yourself standing on the top of a hill and seeing smoke emanate from a far-­‐
away forest. Seeing smoke allows you to infer that there must be a fire. Thus, the smoke is a natural sign of fire. Semiotics is often studied by philosophers and it is used by scientists from other disciplines when talking about sign systems1. The smoke-­‐fire example actually highlights why semiotics is so useful for so many fields—it deals with information and meaning at its most abstract level. When approached from this abstract perspective, human meaning-­‐making through gesture and speech is a subset of this study. The world is full of signs, and only a few of them are linguistic or gestural. Nevertheless, starting a book on gesture studies with semiotics is useful to understand how gesture fits in most generally with other meaning-­‐making processes. Semiotics is a large field—we will mostly use it to distinguish between different kinds of signs. Saying that something is a “sign” is saying that some token or event “stands for” or “represents” something else. Thus, a sign is something that is inherently relational, it connects a form (such as a word or a gesture) with an intended or inferred meaning. Based on the influential work of the philosopher and 1 There are only few programs where one can actually get a full degree in semiotics, but these are rare and hard to find. 1 mathematician Charles Sanders Peirce, researchers commonly distinguish between at least three types of signs: 1.
Index 2.
Icon 3.
Symbol Many other ways of categorizing signs have been developed, but the basic Peircean tripartite distinction will suffice for the present purposes. So what are “indices”, “icons” and “symbols”? An index (plural “indices” but often just “indexical signs”) is a sign which identifies the object referred to by an actual connection to the object. The smoke-­‐fire example above is actually a good example of an index: The smoke is an index of fire, since fire produces smoke, so the two are causally connected. Smoke and fire are moreover spatially connected, since smoke isn’t just distributed randomly across the globe, but it is focused on a region of space close to the fire from which the smoke originates. For example, you could follow a trail of smoke to find the fire source. In gesture, the prototypical indexical sign is a pointing gesture. For example, you might ask a friend at a dinner table “could you hand me the salt?” and simultaneously point to the location of the salt. This pointing gesture makes a visible connection between your index finger (and the imaginary line that extends from it) to the salt. The pointing gesture is a sign that indexes the cup; it identifies the cup among several other potential things one could refer to. Quite similar to a pointing gesture, an arrow is also an indexical sign: An arrow is a sign that directs the viewer’s attention to a specific object or direction. Finally, indexical signs also exist in language. For example, the word “here” is indexical. When you say, “I am here”, the “here” refers to the present spatial position of the speaker, and the word “here” is spatially and causally connected to your current position. 2 An icon is a sign that identifies the object referred to through resemblance. A portrait of somebody is an icon—it identifies the person portrayed through resemblance. The portrait is not the actual person, but it looks like the person identified. Words, too, can be iconic. For example, the word “bang” sounds a bit like the sound concept this word refers to, and so do words such as “beep”, “squealing”, “hissing” and “muffled”. Gestures are frequently iconic as well. A speaker might say “and there was this gigantic pile of candy” while moving her arms outward, away from her body. In this case, the spatial distance between the arms resembles the width of the pile—and because the relationship between gesture distance and object width is one of resemblance, this is an iconic gesture. Finally, symbols are probably the most well-­‐known examples of signs. One way to explain symbols is by saying what they are not: Symbols do not, like indexical signs, identify an object through a spatial or causal connection. And symbols do not resemble the object they refer to. Most words are, in fact, “symbols”. Take, for example, the English word “dog”, compared to the German “Hund” and the French word “chien”—three different symbols with three different sound structures for the same meaning. You would have to know German or French to understand what the respective words mean since the words do not necessarily “resemble” dogs (i.e., there is nothing intrinsically canine-­‐sounding about the word “dog”), and the words are not indexical of dogs either. Thus, in order to understand a symbol, you need to know the conventions of the language. Symbols are thus “inherited” through tradition: A “dog” only means “dog” because we as speakers of the English language collectively agree on this meaning. What are some other examples of symbols? The letters of the English language (the “Latin alphabet”) are abstract symbols that do not look like what they sound. A “t” or “m” bears no resemblance to the sounds denoted by those symbols. As another example, consider the Google Chrome logo: 3 Is there anything about this logo that bears a resemblance to the intended meaning, i.e., a browser used for surfing the web? No—instead, the Chrome logo symbolizes the browser in a conventionalized fashion: You have to know that this is the Chrome logo in order for you to recognize its meaning. As a final example—and one that is a bit more complicated—consider the flag of Chicago: Now ask yourself this: Without knowing that this is the flag of Chicago, would you be able to infer the meaning of “Chicago” from the shapes, the colors, and the arrangement of the flag? Probably not. This shows that the flag is mostly symbolic. However, interestingly, the flag also has iconic elements. The blue stripes represent the Chicago River and the Great Canal. The color of blue as well as the elongated shape clearly iconically represents a river—however, this iconicity is not necessarily detectable unless you know it. Similarly, the four stars are part symbolic and part iconic. They represent the Great Chicago Fire, the World’s Columbian Exposition, the Century of Progress Exhibition and Fort Dearborn. You have to know this in order to interpret what the stars mean (hence, they are symbolic), but at least one of the starts is also partly iconic: The redness of the star resembles the redness of fire as in the Great Chicago Fire. The flag illustrates an important point: Although it is useful to distinguish between index, icon and symbol, in actual practice, most given signs combine indexical, iconic and symbolic elements—most signs in human communication are composite signals (Clark, 1996: 161). The flag of Chicago, for example, combines 4 symbolic with iconic elements. In his book Using language, Herbert Clark talks about the three different types of signs as methods of signaling, rather than as types of signs. This is a much more dynamic perspective—rather than thinking of indices, icons and symbols as static entities, we view them as processes that can be used together in meaning-­‐making practice. The following break-­‐down (adapted from Clark, 1996) is useful to review the three processes of meaning-­‐making: Sign Method created Cognitive process Example for signaling involved the meaning “dog” index indicating locating entities pointing to a dog icon demonstrating imagining using one’s hand to mimic appearances the shape of a dog activating rules saying “dog” symbol describing-­‐as Clark talks about the method that gives rise to an indexical sign as “indicating”, since you are indicating an object to your interlocutor. Constructing an icon, on the other hand, is a form of “demonstrating”. Finally, Clark talks about symbols as arising from “describing-­‐as”. When a speaker looks at a dog and says “dog”, she is identifying the animal in front of her as a specific instance of the general category dog, which only works against the backdrop of a conventionalized system of word-­‐meaning relations (“rules”). Each of the three processes involves different cognitive processes. Indexical signs require the direction of (usually visual) attention to a specific location. Iconic signs require the listener or viewer to imagine the appearance of the object being depicted, and finally, understanding a symbol requires accessing knowledge. In the case of words, this means accessing linguistic knowledge, in such cases as the symbol for the Chrome browser, this means accessing non-­‐linguistic knowledge—
crucially, both linguistic and non-­‐linguistic forms of knowledge that underlie symbolic semiotics rely on conventions that have been established in a community. 5 So far, everything in this chapter dealt with signs in general—this is important. All signs can be indexical, iconic or symbolic. Semiotics is not only about linguistic or gestural indices, icons or symbols. However, the tripartite distinction of methods of signaling does turn out to be really useful for talking about gestures. Applying Peirce to gesture, we may distinguish between at least three types of gesture: 1.
Indexical gestures Other names: pointing gestures; deictic gestures 2.
Iconic gestures Other names: representational gestures, depictive gestures 3.
Symbolic gestures Other names: emblems, quotable gestures, narrow gloss gestures; Italianate gestures The most prototypical indexical gestures are pointing gestures. Pointing is a process by which speakers signal to each other specific locations, they identify points in space. Within the broader communicative contexts, however, pointing gestures do much more than just identifying particular spatial positions: If you ask me “Where is the salt?” and I point to the drawer where I believe the salt is, my pointing gesture is in fact not just identifying the position of the salt for you, it is also quite literally an answer to your question. Thus, pointing gestures are imbued with communicative significance and additional meaning that goes beyond their mere spatial functions. Pointing is a super-­‐interesting gestural process that is actually quite complex. There are many different forms of pointing (e.g., we may point with our head, eyes, with our index finger, with the full hand etc.) and pointing varies quite considerably between cultures. For example, when Japanese speakers tell a friend to come over, 6 they might perform a gesture like this, with the hand bent upside down and the fingers moving toward the speaker: http://blog.tripwolf.com/en/blog/wp-­‐content/uploads/2011/06/Photo-­‐52-­‐500x375.jpg This is quite unlike how people in the US or Europe would usually wave a friend over, which would probably more likely be done with the palm facing upward. This example is just one of many one could have chosen to illustrate cross-­‐
cultural differences in something as seemingly trivial as pointing. Later in this course, we will have an entire lecture (and chapter) dedicated to pointing gestures. Iconic gestures, or sometimes called “representational” gestures come in many forms. In fact, there is no set list of iconic gestures—iconicity is better thought of as an open-­‐ended process. Speakers can iconically gesture about many different things. For example, in Ch. 1 we briefly discussed a gesture representing a handgun. Here’s two ways this gesture could look like: 2 2 http://thehoustonfreethinkers.com/hft/wp-­‐content/uploads/2013/12/finger-­‐gun-­‐student-­‐suspended.jpg http://wac.450f.edgecastcdn.net/80450F/q103albany.com/files/2014/03/Gun-­‐Hand-­‐Craig-­‐Jewell.-­‐630x473.jpg 7 There are several things to notice about these two gestures. First, they both resemble the basic shape of a gun, with an elongated gun shaft and the thumb presumably indicating the visor or safety lock. Now, importantly, even though both hand shapes resemble a handgun (which makes them iconic), they don’t look at all like actual guns. Any iconic gesture only resembles a part or some parts of the actual object being referred to. Of course, the color of the handgun is not represented by this gesture, nor is its texture. Details of the shape are omitted—a hand cannot simply mold itself into a shape that allows for all the complexities of a real handgun. Moreover, even the size of the handgun is probably not accurate—it is quite unlikely that the handgun has exactly the size or shape of the speaker’s hand. Iconic gestures are like sketches of the objects they represent, they lack detail. Iconic gestures are characterized by what Herbert Clark calls selective depiction—only certain parts of the object are represented. Then, notice that the two forms (left one and right one) differ in whether the middle finger is involved or not. This serves to highlight that there is considerable freedom in how gesture can be used to convey the meaning “handgun”—when it comes to resembling the gun shaft, both one or two fingers will do, although this contrast might matter if the size of the gun shaft is relevant within a given conversation. What this shows is that speakers have freedom in what aspects of an object they want to convey (its shape? its height? its speed of motion?), and there is a whole range of iconic gestures that can “get the job done”, no conventionalized pre-­‐
made list of gestures. However, notice furthermore that there probably is an element of conventionalization even in this gesture. The thumb being extended is the conventionalized way of representing a handgun in the US. So, this gesture is iconic, as well as also symbolic. Finally, what about symbolic gestures? These have a special name in the gesture literature, they are so-­‐called emblems. Here are some examples of emblems: 8 When most people think about gestures, they probably think first about emblems (cf. Susan Goldin-­‐Meadow, 2003). This is because emblems are highly salient when they are used, and they are also highly memorable because they are complete units. Emblems in many ways are like words in that they can be quoted, and there are often verbal labels, so called “glosses”, for these gestures: You probably immediately know what I mean when I talk about the “peace sign”, the “ok sign”, and the “thumbs up” and “thumbs down” gestures. You probably also know how a “Vulcan greeting” looks like (the gesture Spock famously makes in Star Trek), and you might also know how a “Hitler greeting” was performed in Nazi Germany (“Hitlergruss” in German). The fact that there are labels for these emblems also highlights their autonomous status, which is different particularly from iconic gestures. Consider somebody who performs an iconic gesture signifying a basket ball. The same gesture could also be used to signal a golf ball, a yo-­‐yo, a planet or a Christmas ball. When the Vulcan greeting is performed, it means just one thing. The glossable and quotable nature of these gestures is also why emblems are sometimes called “quotable gestures” or “narrow gloss gestures”. Notice that emblems are also symbolic gestures, because they primarily rely on convention. This also means that emblems differ the most between different cultures. Some emblems are widely known thanks to the media and cross-­‐cultural contact, such as the “ok sign” or the “peace sign”. However, these gestures are far from universal—somebody in the Amazonian rainforest who has never seen a movie will not be able to interpret the “peace sign” correctly. In fact, to him or her it might look like a pair of scissors or animal ears. 9 Again, notice the fact that many emblems have an iconic element in them. For example, the “Victory sign” performed by Winston Churchill in the picture above (to the right of Batman and Simon Cowell) iconically represents a “V”, the first letter of the word “victory”. Thus, even in this highly abstract emblematic gesture, there is an element of iconicity. Re-­‐iterating Clark: Most gestures combine multiple methods of signaling, with indexicality, iconicity and symbolicity working in tandem. I should note here that emblems will be the gesture type that is least important for the rest of this book—despite the fact that it is what laymen are most likely to think of when they hear “gesture”. There are several reasons for this: First, although there are still many things to be discovered about emblems (such as how they are used in discourse, where they are processed in the brain etc.), they are comparatively well understood—iconic gestures turn out to be much more complex. This complexity has, in part, to do with the fact that iconic gestures form an open-­‐
ended system. Every iconic gesture is somewhat unique, and it is not at all clear how people interpret them. Emblems form a rather restricted part of the gestural repertoire: They occur relatively less frequently (although no real data exists on the frequency of gesture types), and they form a small set of conventionalized signs. Iconic gestures are produced almost all the time when people talk about objects, and, as we will see later, they are even used in the context of abstract entities such as time, similarity and number. The preceding discussion of gesture types was structured around Peircean semiotics and we identified three gesture types—indexical gestures, iconic gestures and symbolic gestures (“emblems”)—that correspond to the three methods of signaling. Now, this is a very broad way of categorizing gestures, and different and more fine-­‐grained classification systems have been discussed. Pretty much all of them contain at least these three types, but other things have been suggested. For example, the famous gesture researcher McNeill also has an additional category of beat gestures. These gestures are thought to just “underline” the rhythm of speech, adding emphasis. They are defined by their absence of 10 identifiable meaning: A beat gesture is simply waving the hands in the rhythm of speech to go along with the particular phrasing and timing in the vocal modality. It is useful to know the word beat gesture and one can definitely say that some gestures have beat-­‐like qualities in that they seem to mostly just underline speech—
however, as a category, a beat gesture is not very well-­‐defined. In actual practice, almost all gestures (including indexical, iconic and sometimes symbolic gestures) are closely time-­‐locked with speech, so all gesturing can be considered to have a beat-­‐like, rhythmic quality. Moreover, supposed “beat” gestures are quite difficult to classify in individual cases: “Having meaning” is a property that allows for considerable wiggle room. Consider, for example, a public speaker waving his hands in the rhythm of speech simply with a fist, or with a precision grip (index finger and thumb held together, as if grasping a pellet). The precision grip might add emphasis and make the speaker appear sharp and “to the point” (Lempert, 2011). The fist might signal force and conviction. So, the same “beat-­‐like” rhythmic movement pattern of the hands can actually have meaning—and it is difficult to make a distinction between meaningful and meaningless gestures. Nevertheless, it is useful to know the term “beat gesture”, since some gestures are so schematic in their meaning that they seem to primarily signal the rhythm of speech. One can furthermore make a subdivision among iconic gestures, namely, one can distinguish between literal gestures and metaphorical gestures. You probably would not think that a metaphor can be “metaphorical”, because the term metaphor in common usage evokes poetry and literature, or what some people consider fanciful language. However, in cognitive science, the term metaphor has a much broader meaning. Following George Lakoff and Mark Johnson’s highly influential Conceptual Metaphor Theory (Lakoff & Johnson, 1980), we can characterize metaphors as conceptual mappings between different mental domains. For example, the concrete domain of SPACE is mapped onto the relatively abstract domain of TIME, as in linguistic expressions such as “Christmas is coming” (where Christmas isn’t literally coming in a spatial sense) or “Halloween is just ahead of us” (where Halloween isn’t literally “ahead”). Crucially, because Lakoff and Johnson (1980) define metaphor to be a matter of thought, rather than just language, they 11 allow for non-­‐linguistic realizations of conceptual metaphor. A calendar, for example, is a non-­‐linguistic representation of time that is essentially metaphorical in Lakoff and Johnson’s sense because the concrete domain of space is used to represent the abstract domain of time, i.e., in terms of a linear ordering of time points. Because metaphors may be non-­‐verbal, gestures can be metaphorical as well. For example, the following is a still image of a speaker gesturing about time. The example is taken from Esther Walker and Kensy Cooperrider’s paper The continuity of metaphor: Evidence from temporal gestures. The stills are taken from a television show where the speaker gestures backward with his left hand while saying “where I had been” (referring to the past) and simultaneously forward and to the right while saying “looking to the future”. Metaphorical gestures are often subsumed under the “iconic gesture” category. From on perspective, this makes sense, since these gestures are clearly “depictive” in some fashion. For example, the gesture above creates an iconic link between gesture space and imagined conceptual space, i.e., the movement to the right of the hands resembles the movement to the right along an imagined time line. However, the iconic link between gesture and meaning is of a more abstract kind. In a way, a metaphorical gesture involves two interpretational steps: First, from the gesture iconically to a spatial mental image. Second, from the spatial mental image 12 to the abstract domain—in this case time. Metaphorical gestures thus go beyond “literal” iconic gestures in representing not concrete ideas but abstract ideas that are one step removed from concrete iconicity. Because of this, one may also consider metaphorical gestures its own category. Another useful category is that of pragmatic gestures. Pragmatics is the field of linguistics that deals with how meaning relates to context, including the local spatial and social context. When it comes to gestures, pragmatic gestures can be understood as those gestures that do not directly represent a concrete object as such, but rather, they are used to negotiate and manage interpersonal interactions. For example, somebody might stretch her arm toward a speaker with the palm facing downward, indicating that the speaker should stop talking. Conversely, resting one’s arms may signal that a speaker considers her turn to be completed. So far, this chapter has inundated you with different categories of gestures. You might have gotten the impression that all that gesture researchers do is assign discrete labels to gestures, essentially pigeonholing them to belong to a specific category in an either/or fashion. And indeed, categories are useful in science because they help us talk about recurring phenomena and nicely sort out similarities and differences between different behaviors. However, from time to time one has to acknowledge that categories can also be unhelpful, and that gesture is, in fact, quite continuous. Indeed, it has been said multiple times that gesture is highly multifunctional, i.e., the same gesture can simultaneously have different functions, which often means that it can also be of different categories. For example, pragmatic gestures are also partially metaphorical, and they clearly have an iconic component as well. Moreover, when embedded in the speech stream, a pragmatic gesture may also be a beat gesture. In some studies a particular type of gesture is in focus, so a pragmatic gesture might be classified as “pragmatic” and an iconic gesture as “non-­‐pragmatic”. In another analysis with a different focus, both gestures could be classified as “iconic”. The categories identified in this chapter are really just broad labels, descriptive tools that help us gesture scientists talk about recurring patterns of gesture—the categories do not directly correspond to 13 reality and may be shifted or interpreted differently depending on the goals of one’s study. 14