You Give Me The Creeps!
The Uncanny Valley and Oculesics
MASTER THESIS
A thesis submitted in partial fulfillment of the
requirements for the degree of
“Master of Arts in Arts and Design”
MultiMediaArt
University of Applied Sciences Salzburg
Author:
Ilonka van Houwelingen
Supervisors:
FH-Prof. Dr. Michael Manfé (1st supervisor with regard to contents)
DI (FH) Michael Großauer
Salzburg, 15 July 2016
(2nd supervisor with regard to contents)
Acknowledgements
Acknowledgements
Writing this master thesis has been a long road and there are several people who helped me
on this way. First, I would like to thank my two supervisors FH-Prof. Dr. Michael Manfé and DI
(FH) Michael Großauer from the University of Applied Science Salzburg, for their input, time
and positive feedback. Secondly, I thank my friend Sebastian Geschke for his time to read
everything, but especially for his patient every time I talked about my thesis. This cannot have
been very interesting anymore after so many times. Next, I would like to thank my brother
Marco van Houwelingen for his time and effort to read my thesis and his useful feedback. Also
I would like to thank my parents, family and friends for all their support and interest about the
progress of my work.
And the most important person, I would like to thank my girlfriend Magdalena Birkl, without
you there would not have been a master thesis. Thank you for pushing me and believing in
my ability to finish it.
2|Page
Eidesstattliche Erklärung
Eidesstattliche Erklärung
Hiermit versichere ich, Ilonka van Houwelingen, geboren am 15. April 1985 in Bennekom, die
Niederlande, dass ich die Grundsätze wissenschaftlichen Arbeitens nach bestem Wissen und
Gewissen eingehalten habe und die vorliegende Masterthesis von mir selbstständig verfasst
wurde. Zur Erstellung wurden von mir keine anderen als die angegebenen Quellen und
Hilfsmittel verwendet.
Ich versichere, dass ich die Masterthesis weder im In- noch Ausland bisher in irgendeiner
Form als Prüfungsarbeit vorgelegt habe und dass diese Arbeit mit der den Begutachtern
vorgelegten Arbeit übereinstimmt.
Salzburg, den 15.07.2016
____________________________
Ilonka van Houwelingen
1210627040
_________________________
Matrikelnummer
3|Page
Kurzfassung
Kurzfassung
Vor- und Zuname:
Ilonka van Houwelingen
Institution:
Fachhochschule Salzburg
Studiengang:
MultiMediaArt
Titel:
You Give Me The Creeps! The Uncanny Valley and Oculesics
Begutachter (1):
FH-Prof. Dr. Michael Manfé
Begutachter (2):
DI (FH) Michael Großauer
Schlagwörter:
1. Uncanny Valley
2. Oculesics
3. Gesichtsausdrücke der Emotionen
Wenn ein Computer-Grafik (CG) Charakter in seiner Erscheinung zu nah an das Aussehen
eines echten Menschen herankommt, kann das beim ZuschauerIn ein unheimliches Gefühl
auslösen. Dieses Phänomen wurde das erste Mal durch Masahiro Mori beschrieben und heißt
das Uncanny Valley. In einigen Filmen haben die Schöpfer versucht die Charaktere wie echte
Menschen aussehen zu lassen. Diese Charaktere haben im Anschluss schlechte Kritiken
bekommen, die mit dem Uncanny Valley verbunden werden können. Insbesondere die Augen
haben negative Aufmerksamkeit bekommen. Es gibt mehr als eine Theorie über das Uncanny
Valley und warum das Phänomen auftritt. Der Begriff unheimlich (uncanny) ist das erste Mal
beschrieben als intelektuelle Unsicherheit und ein Geheimnis, das im verborgenen bleiben
sollte aber hervorgetreten ist in verschiedenen Essays von Jentsch (1906/1997) und Freud
(1919) am Anfang des 20 Jahrhunderts. Weitere Theorien über das Uncanny Valley basieren
auf der Angst ersetzt zu werden, die Angst vor inteligenten Maschinen und die natürliche
menschliche Angst vor Gefahr und Tod. Um den Einfluss der Augen von CG-Charaktern
besser zu verstehen, beschäftigt sich diese Arbeit mit Oculesics, die Rolle der Augen in
nonverbaler Kommunikation. Dafür werden die Gesichtsausdrücke der Emotionen mit Hilfe
des Facial Action Coding Systems von Ekman und Friesen untersucht. Ebenfalls werden die
Bewegungen der Augen, die Pupillenerweiterung und das Augenzwinkern diskutiert.
Abschließend werden die Unterschiede in Oculesics zwischen Charakteren der Filme The
Polar Express (2004) und The Adventures of Tintin: The Secret of the Unicorn (2011)
diskutiert.
4|Page
Abstract
Abstract
First and last name:
Ilonka van Houwelingen
Institution:
University of Applied Sciences Salzburg
Course of Studies:
MultiMediaArt
Title:
You Give Me The Creeps! The Uncanny Valley and Oculesics
Supervisor (1):
FH-Prof. Dr. Michael Manfé
Supervisor (2):
DI (FH) Michael Großauer
Keywords:
1. Uncanny Valley
2. Oculesics
3. Facial Expressions of Emotions
When a Computer Graphic (CG) character’s appearance becomes to near to a real human
being, this can give the perceiver a feeling of uneasy or uncanny. This phenomenon is first
described by Masahiro Mori and it is called the uncanny valley. In some movies the creators
have tried to make the characters near human-like. These characters have received negative
reviews referring to the uncanny valley. Especially the eyes are getting negative attention.
There is more than one theory about why the uncanny valley phenomenon occurs by people.
The term uncanny is first described as intellectual uncertainty and something secret that
emerges but should have stayed concealed in the different essays by Jentsch (1906/1997)
and Freud (1919). Other theories about the uncanny valley are based on the fear of being
replaced, the fear of intelligent machines and the human’s natural fear of danger and death.
To see what the influences of the eyes of a CG-character are, this thesis studies oculesics,
the role of the eyes in nonverbal communication. Here, we look at the facial expression of
emotion with references to the Facial Action Coding System of Ekman and Friesen. Also the
movements of the eyes, pupil dilation and blinking rate are discussed. In the end we compare
the differences in oculesics between the characters in the movies The Polar Express (2004)
and The Adventures of Tintin: The Secret of the Unicorn (2011).
5|Page
Table of Contents
Table of Contents
1 Introduction ...................................................................................................................... 10
2
1.1
Research Interest ................................................................................................. 10
1.2
Field of Research and Research Question ............................................................ 10
1.4
Structure ............................................................................................................... 11
1.5
Notes on Reading ................................................................................................. 12
Understanding the Uncanny Valley .............................................................................. 13
2.1
Creepiness ........................................................................................................... 13
2.2
Ernst Jentsch and Sigmund Freud ........................................................................ 14
2.3
The Uncanny Valley by Masahiro Mori .................................................................. 18
2.3.1
The Effects of Movement ............................................................................... 22
2.3.2
Individual Responses ..................................................................................... 22
2.3.3
Examples of the Uncanny Valley.................................................................... 24
2.4
3
Theories about the Uncanny Valley ...................................................................... 28
2.4.1
Doppleganger ................................................................................................ 28
2.4.2
Expectation Violation ..................................................................................... 29
2.4.3
Mixed Human Identity and Cognitive Dissonance .......................................... 30
2.4.4
Evolutionary Aesthetics .................................................................................. 30
2.4.5
Theory of Disgust ........................................................................................... 31
2.4.6
Terror Management and Subconscious ......................................................... 32
2.4.7
Lack of Empathy ............................................................................................ 33
2.4.8
Mind Perception ............................................................................................. 35
2.5
The Eyes and the Uncanny Valley ........................................................................ 36
2.6
Summary .............................................................................................................. 39
Understanding Oculesics ............................................................................................. 40
3.1
The Anatomy of Facial Muscles and the Eyes....................................................... 40
3.1.1
Facial Muscles in the Upperpart of the Face .................................................. 40
6|Page
Table of Contents
3.1.2
3.2
Nonverbal Communication .................................................................................... 43
3.3
Emotions............................................................................................................... 44
3.4
Facial Expressions of Emotions ............................................................................ 44
3.4.1
Action Units in the Upperpart of the Face....................................................... 45
3.4.2
Anger ............................................................................................................. 48
3.4.3
Disgust........................................................................................................... 49
3.4.4
Happiness ...................................................................................................... 50
3.4.5
Fear ............................................................................................................... 51
3.4.6
Surprise ......................................................................................................... 52
3.4.7
Sadness......................................................................................................... 53
3.4.8
Contempt ....................................................................................................... 54
3.5
Pupil Dilation ......................................................................................................... 56
3.6
Blinking Rate......................................................................................................... 57
3.7
Saccades and Fixations ........................................................................................ 57
3.8
Gaze in Object Related Actions ............................................................................ 58
3.9
Gaze in Human Interaction ................................................................................... 59
3.10
4
The Anatomy of the Eye ................................................................................ 42
Summary ........................................................................................................... 62
Case Studies ............................................................................................................... 64
4.1
The Polar Express ................................................................................................ 65
4.2
The Adventures of Tintin ....................................................................................... 66
4.3
Emotions............................................................................................................... 68
4.3.1
Anger ............................................................................................................. 69
4.3.2
Happiness ...................................................................................................... 71
4.3.3
Fear ............................................................................................................... 72
4.3.4
Surprise ......................................................................................................... 75
4.4
Eye Movements in Actions and Human Interaction ............................................... 76
4.5
Pupil Dilation ......................................................................................................... 79
7|Page
Table of Contents
5
6
4.6
Startle reflex.......................................................................................................... 79
4.7
Summary .............................................................................................................. 80
Conclusion ................................................................................................................... 81
5.1
Discussion ............................................................................................................ 81
5.2
Summary of the Thesis ......................................................................................... 84
Bibliography ................................................................................................................. 86
List of Figures ..................................................................................................................... 95
List of Tables ...................................................................................................................... 98
8|Page
List of Abbreviations
List of Abbreviations
Abbreviation
Explanation
3D
Three-Dimensional
AU
Action Unit
CG
Computer Graphic
Disney
Walt Disney Animation Studios
FACS
Facial Action Coding System
Pixar
Pixar Animation Studios
The Adventures of Tintin
The Adventures of Tintin: The Secret of the
Unicorn
TMT
Terror Management Theory
9|Page
1 Introduction
1
Introduction
When I was about 6 or 7 years old, I can remember that I had one of my first experience with
the uncanny valley. Of course at that time, I did not realize that it was the uncanny valley. I
was watching The Thunderbirds on the television and though I did not quite know what it was,
the dolls made me feel uneasy, and something was off about them. But why did they, and still
do, make me feel so uneasy? Why are they giving me the creeps?
1.1
Research Interest
Though The Thunderbirds are already a long time ago, there are still characters in movies that
can make the viewer feel uneasy. The remarkable thing is that these character do not meant
to give this uncanny feeling; they are supposed to be the heroic ones. But these characters
have a near-human appearance which causes these feelings. For a Computer Graphic (CG)
artist, working in the film or gaming industry, this is an important phenomenon. Nobody wants
to create a protagonist hero character that the public should like, but which is perceived as
creepy. That way the audience cannot connect with the character. This phenomenon of creepy
near-human characters is called the uncanny valley. Any part of the appearance of a character
can cause a character to fall into the uncanny valley but what stands out in most negative
reviews about these films with near-human characters, are the eyes. In normal life the eyes
play a main role in our nonverbal communication. This is also the case when we look at a
near-human CG-character.
1.2
Field of Research and Research Question
Currently, there still is not a human-realistic CG-character that has managed to cross the
uncanny valley. There are a handful of theories why the uncanny valley phenomenon occurs
but not one is proven to be the right one. A character falling into the uncanny valley can be
caused by many parts of the appearance of a character. For example the speech that does
not fit the look of a character, the body movements are stiff and unnatural or the skin might
not look like real skin. People and near-human CG-characters with abnormal eyes are
perceived as creepy or uncanny (Gray & Wegner, 2012). Therefore problems with the eyes of
a character are directly noticed by the audience. There is not a lot of research done about the
eyes of characters, though this gets the most negative reviews.
Oculesics is the study of the role of the eyes in nonverbal communication. But there is not
much information about this subject. The term can be found in a few books but there is not
one book that focus solely on this subject. Therefore it is hard for an artist, specialised in three10 | P a g e
1 Introduction
dimensional (3D) computer animation, to gather all the information necessary to create the
perfect eyes for a character.
The goal of this thesis is to provide an overview about how the eyes of a near human CGcharacter affect our sympathy towards the character. Why do they give us a feeling of
creepiness, what can the movement of the eyes of a character tell us? How far did the
animation industry get with applying oculesics in films that have received negative reviews
related to the uncanny valley? Therefore the research question focuses on the contribution of
oculesics to the problem of the uncanny valley:
How can the knowledge of Oculesics enhance the design and animation of a nearhuman character to help overcoming the uncanny valley?
It is not a purpose of this thesis to come up with an answer if it is possible to cross the uncanny
valley or to decide which existing theory is responsible for the uncanny valley. This is because
there are many aspects which have influence on the fact if a character falls into the uncanny
valley or not. Therefore it will be impossible to answer this question only based on oculesics.
It might as well be never possible to cross the uncanny valley, but until we can get to the point,
where a CG-character is not distinguishable from a real human being, we will never be sure
about it.
1.4
Structure
In order to get a complete overview of the term uncanny valley, the second chapter discusses
this theme. First, by reviewing what creepiness and uncanny is. Therefore we will look at the
two essays of Jentsch and Freud in the early 20th century. This is followed by discussing the
term uncanny valley, which is first mentioned by Masahiro Mori in 1970. We will look at
individual responses to this phenomenon and examples of characters that have falling into the
uncanny valley. We continue with the different theories about why the uncanny valley
phenomenon occurs. The chapter ends with an overview of different researches that have
been done to the effect of the eyes as well as the upperpart of the face and the uncanny valley.
The third chapter has its focus on oculesics. Here we will discuss the anatomy of the facial
muscles and the eyeball. This is followed by an introduction about nonverbal communication
and emotions. We will look further at the seven universally recognized facial expressions of
emotions. After this the eye pattern of saccades and fixations is discussed, followed by pupil
dilation, blinking rate and gaze direction in nonverbal communication.
11 | P a g e
1 Introduction
With the information of the second and third chapter we will discuss the use of oculesics in the
movies The Polar Express (2004) and The Adventures of Tintin: The Secret of the Unicorn
(2011). We will compare the facial expressions of the characters and their eye movement
behavior in mutual gaze. The findings will be further discussed in the last chapter of this thesis.
1.5
Notes on Reading
The problem of the uncanny valley is not only due to problems with the eyes of a character.
Also things like body movement, skin colour and speech can contribute to a character falling
into the uncanny valley. Nevertheless, the problem with the eyes is still something that recurs
in almost every review of a film with human-realistic Computer Graphic characters. Therefore
this thesis will focus on the eyes and the upperpart of the face. This, to help other 3D artists
and researchers on the way of overcoming the uncanny valley.
12 | P a g e
2 Understanding the Uncanny Valley
2
Understanding the Uncanny Valley
“Tintin looks simultaneously too-human and not human at all, his face weirdly fetal,
his eyes glassy and vacant instead of bursting with animated life.” (Buchanan, 2011)
This quote comes from the website Vulture where Kyle Buchanan wrote a review about the
animation film The Adventures of Tintin: The Secret of the Unicorn (2011). In other reviews as
well, the Computer Graphic (CG) character of Tintin is described as “creepy” and “unsettling”
(Snyder, 2011). An opinion that is shared by others (Pinkerton, 2011; Tinwell, 2015, p. 111;
Rom, 2011).
A human-like CG-character in animation films or in games can give the viewer a creepy
feeling. This is due to the phenomenon of the uncanny valley. But why is this happening? This
is an important question for a 3D artists in the film and gaming industry. You don’t want your
public to have an uncanny feeling when they look at your protagonist, when you actually want
them to feel empathy with your character (MacDorman & Entezari, 2015, p. 142). The goal of
this thesis is to discover what the effect of the eyes of a character is, in giving a creepy feeling
to the one who is looking at a human-like character like Tintin. Before discussing oculesics,
(“The study of the communicative role of the eyes in nonverbal communication” (Chandler & Munday,
2011)) in the next chapter, this chapter will explain the uncanny valley. First, the meaning of
uncanny will be explored, where the essays of Ernst Jentsch and Sigmund Freud in the early
20th century will serve as a resource. The chapter continues with the concept of the uncanny
valley, first described by Masahiro Mori in 1970. Subsequently, theories about the
explanations of why the uncanny valley exist are discussed. At the end, this chapter will look
at the studies which have been done in exploring the relationship between the eyes (in some
cases combined with the upperpart of the face) and the uncanny valley.
2.1
Creepiness
To survive in life we need fear. This is what kept our ancestors alive. For example the fear of
dangerous animals or the fear of enemies. It made the people alert on the possible danger to
get wounded or killed. These days we still need fear to survive, there is a great possibility that
you will die if you feel no fear. It prevents you from falling off a cliff, get run over in the streets
or it will make you flee when there is a gun pointed at you. Though these events can be
terrifying, they are not creepy. So what exactly is creepy? (McAndrew & Koehnke, 2016, pp.
10-11; LoBue & Rakison, 2013, p. 285)
13 | P a g e
2 Understanding the Uncanny Valley
There is the fear of things or events that are not going to kill you and where there is no obvious
danger. These things can give you a feeling of unease or in other words the creeps. Having
the creeps is something most people will recognize. It is a universal feeling and people can
experience it in everyday life situations. For example walking through a dark street and
thinking someone is directly behind you. Other examples are people who get an uneasy feeling
at a masked ball or people who get the creeps from clowns (Jentsch, 1906/1997, p. 10;
McAndrew & Koehnke, 2016, p. 12; Freud, 1919, p. 236).
This feeling of creepiness can give a physical reaction of feeling cold or chilly (McAndrew &
Koehnke, 2016, p. 11; Leander, et al., 2012, p. 772). But what is this feeling and where does
it come from? Remarkably, creepiness has not been studied a lot (McAndrew & Koehnke,
2016, p. 10). The word creepy is an informal form of the word uncanny. Therefore, in the next
part we will look at the first essays written about the term uncanny.
2.2
Ernst Jentsch and Sigmund Freud
Almost a century ago, in 1919, the psychoanalyst Sigmund Freud published an essay with the
title The Uncanny (Freud, 1919). In this essay he refers to the only ‘literature’ he has found on
the subject, which is the in 1906 written essay On the Psychology of the Uncanny by the
German psychiatrist Ernst Jentsch (Jentsch, 1906/1997).
To better understand why an uncanny feeling occurs and what it causes, both Jentsch and
Freud explored the word uncanny in their essays. The original essays of Freud and Jentsch
were written in German. Therefore, we will look at the German translation of the word uncanny,
which is the word unheimlich. The part un- of the word unheimlich means that the meaning of
the word is the opposite of the word heimlich. English translations of the word heimlich are
familiar, native and belonging to the house (Freud, 1919, p. 222; Tinwell, 2015, p. 4). To bring
this back to unheimlich, Freud declares that unheimlich is therefore the opposite of what is
familiar. He also thinks it is possible that something that is unfamiliar and new is unheimlich
but this does not mean that everything that is unfamiliar and new is unheimlich. Something
that is unheimlich can become heimlich and familiar but something that is heimlich and familiar
cannot become unheimlich, unless there is something added that is new and unfamiliar that
makes it unheimlich again (Freud, 1919, pp. 220-221).
Remarkable, native and familiar are not the only meanings of the German word heimlich. The
second meaning of the word can be translated as secret or hidden. Freud explains why the
two different meanings of the word are so interesting: “on the one hand it means what is familiar
14 | P a g e
2 Understanding the Uncanny Valley
and agreeable, and on the other, what is concealed and kept out of sight” (Freud, 1919, pp. 224-225;
Tinwell, 2015, p. 4)
Although the word unheimlich is mostly used as the opposite of familiar, native and homely, it
also corresponds to the second translation of the word heimlich and thereby to everything that
is hidden or secret. Or as how Freud describes it, unheimlich, and therefore the English
translation uncanny as well, stands for the disclosure of everything that should have stayed
hidden or secret (Freud, 1919, p. 225; Tinwell, 2015, p. 4).
As been mentioned before, the essay of Ernst Jentsch was the only other literature Freud
found about the theme of the uncanny. Jentsch describes unheimlich as feeling someone can
have when he or she is “not quite “at home” or “at ease” in the situation concerned”. (Jentsch,
1906/1997, p. 8) He also describes the feeling as a “disturbing and uncomfortable feeling that may
escalate to a haunting, shocking and ghastly experience” (Tinwell, 2015, p. 2; Jentsch, 1906/1997).
If one encounters an object or a person, but one is not sure what the object is or if the person
is really a human being, the person will get an uncanny feeling. It could be possible that the
object is in fact not the object which it was thought to be or the human being might appeared
to be, using Jentsch his word, an automatic which is lifeless and animated. But one is not sure
about it. According the Jentsch this intellectual uncertainty is the cause of the uncanny feeling.
(Jentsch, 1906/1997, p. 8; Tinwell, 2015, p. 2) Jentsch describes it as following:
“The unpleasant impression is well known that readily arises in many people when
they visit collections of wax figures, […] In semi-darkness it is often especially
difficult to distinguish a life-size wax or similar figure from a human person. For
many sensitive souls, such a figure also has the ability to retain its unpleasantness
after the individual has taken a decision as to whether it is animate or not.” (Jentsch,
1906/1997, p. 12)
Intellectual uncertainty is used in storytelling to create an uncanny feeling as well. Jentsch
(1906/1997, p. 13) writes in his essay:
“In storytelling, one of the most reliable artistic devices for producing uncanny
effects easily is to leave the reader in uncertainty as to whether he has a human
person or rather an automaton before him in the case of a particular character.”
Jentsch discusses one of the works of E.T.A Hoffmann as one of the examples where the
intellectual uncertainty gives the readers an uncanny feeling. In the story The Sandman
(Hoffmann, 1885), the protagonist Nathanael has some creepy experiences. The story starts
with Nathanael recalling a trauma from his youth. He describes the evenings where the
15 | P a g e
2 Understanding the Uncanny Valley
mysterious ‘Sandman’ visits his father. His sister’s nanny told him that the Sandman comes at
night to bad children who will not go to sleep, throws sand in their eyes so they will jump out
and then takes the eyes to the half-moon, where he feeds them to his on hook-beaked
children.
At one night when the Sandman is about to come, Nathanael hides in his father’s room to see
who this mysterious Sandman is. He discovers it is the lawyer Coppelius. Nathanael is caught
in his hiding place and as a punishment, Coppelius wants to burn his eyes out with fire embers,
but his father prevent this from happening.
A year later, Nathanael’s father dies in an explosion when Coppelius is visiting him. Coppelius
disappears that night and the story goes on with Nathanael, who is a student now, telling about
the visit he had from the barometer-seller Coppola. Nathanael beliefs it is Coppelius. When
Nathanael does not want to buy a barometer, the man says that he also has got “foine oyes as
well – foine oyes!” (“Fine eyes as well, fine eyes!”) (Hoffmann, 1885, p. 197). This scares
Nathaneal as he does not understand what he means by that. How can he have eyes to sell?
But it turns out Coppola has spyglass which he calls ‘eyes’. Nathanael buys a spyglass which
he start to use to look at the daughter, Olimpia, of his professor who happen to live right across
the street from him.
Nathanael falls in love with Olimpia at a ball, organized by his professor. But not everyone is
amazed by Olimpia. His friend Siegmund talks with Nathanael about Olimpia:
“Yet it’s very strange that several of us have formed pretty much the same opinion
about Olimpia. We think she is — you won’t take it ill, brother? — that she is
singularly statuesque and soulless. Her figure is regular, and so are her features,
that can’t be gainsaid; and if her eyes were not so utterly devoid of life, I may say,
of the power of vision, she might pass for a beauty. She is strangely measured in
her movements, they all seem as if they were dependent upon some wound-up
clockwork. Her playing and singing have the disagreeably perfect, but insensitive
timing of a singing machine, and her dancing is the same. We felt quite afraid of this
Olimpia, and did not like to have anything to do with her; she seemed to us to be
only acting like a living creature, and as if there was some secret at the bottom of it
all.” (Hoffmann, 1885, p. 205)
In the time that follows, Nathanael visits Olimpia but he has only one-way conversation with
her. She only replays with “Ah ah” on his poems and stories. On the day he wants to ask her
to marry him, he finds the professor and Coppola fighting over the eyeless body of Olimpia.
16 | P a g e
2 Understanding the Uncanny Valley
Nathanael goes mad and he tries to strangle the professor as Coppola flies with the body of
Olimpia.
After a recovery time Nathanael visits a tower in his home town. When he uses his spyglass
again, he is again driven to madness. In the crowd that is drawn by the commotion also
Coppelius appears. Nathanael sees him and cries “Foine oyes! Foine eyes!” (“Fine eyes! Fine
eyes!”) (Hoffmann, 1885, p. 215) and then falls to his death. Coppelius disappears once again
(Hoffmann, 1885).
According to Jentsch, the uncanniness of the story comes from the intellectual uncertainty the
reader experience with the question whether or not Olimpia is a human being (Jentsch,
1906/1997, p. 13). Though Freud agrees with Jentsch that Hoffmann uses the intellectual
uncertainty of the appearances of Olympia to create a certain uncanniness in the story, he
does not agree with him that Olimpia is the main theme which is responsible for the
uncanniness. In Freud’s opinion, the uncanny factor in the story of the Sandman is the idea of
losing one’s eyes. He declares that from psychoanalytic experience it is known that losing or
injuring one’s eyes is a dreadful fear for children as well as for adults (Freud, 1919, pp. 226231).
In the story of the Sandman we can see that not everyone experience the same uncanniness
with the same events. In the passage where Nathanael friend, Siegmund, talks to him about
Olimpia it is clear that Nathanael’s friends experience an uncanny feeling by the appearance
of Olimpia: “We felt quite afraid of this Olimpia, and did not like to have anything to do with her;”
(Hoffmann, 1885, p. 205). But Nathanael does not experience this feeling, he falls in love with
her. This example shows that not everybody experience uncanny feelings in the same amount
(Jentsch, 1906/1997, p. 10; McAndrew & Koehnke, 2016).
As been mentioned before, Freud did not agree with Jentsch that the main problem about
uncanniness is the intellectual uncertainty. Freud thinks this uncanny feeling occurs when
certain events, things or situations recurrence. He gives an example of a situation where he
ends up in the same street that makes him feel uncomfortable. By taking some bystreets he
tries to get away from the street. After three times ending up in this same street, he declares
his feeling about the situation is what can be called as uncanny. In this example the situation
repeated itself again and again. An uncanny sensation can also occur when we have a
repressed fear that we are suddenly reminded of at some point, when we interact with a
strange or uncanny object or event. In other words, a revelation of the repressed (Freud, 1919,
pp. 237-241) or, how the definition from Freud is declared by Tinwell (2015, p. 5):
17 | P a g e
2 Understanding the Uncanny Valley
“Importantly, the uncanny exists not only as a revelation of what may be hidden in
others but also what may be repressed or hidden in oneself.”
So to summarize, according to Jentsch, the uncanny appears if there is some kind of
intellectual uncertainty. Freud thinks the feeling comes from the secrets inside of us. Clear is
that, uncanny has to do with ambiguity. We do not know if we should fear something and this
may come from, for example, of not knowing what the motives of another person are or not
knowing why an event is repeated. McAndrew and Koehnke say:
“ It is our belief that creepiness is anxiety aroused by the ambiguity of whether there
is something to fear or not and/or by the ambiguity of the precise nature of the threat
(e.g., sexual, physical violence, contamination, etc) that might be present.”
(McAndrew & Koehnke, 2016, p. 10)
And they follow with the conclusion:
“So, it is not the clear presence of danger that makes us feel creepy, but the
uncertainty of whether danger is present or not.” (McAndrew & Koehnke, 2016, p.
11)
In the story of the Sandman the theme of the eyes are an important part of the story. It is
remarkable that in the passage of Siegmund to Nathanael he also speaks about the person
Olimpia being soulless and her eyes that are devoid of life. This is an example of how important
the eyes of one person are to another person. The uncanny feeling deviating eyes can give a
person is the one of main themes of this thesis. It will be more discussed in the next parts of
the thesis.
2.3
The Uncanny Valley by Masahiro Mori
After dealing with the concept of uncanny, this part goes on with explaining the Uncanny
Valley. Although the uncanny valley phenomenon was first written about in 1970, Jentsch
already describes the phenomenon in his 1906 essay:
“A doll which closes and opens its eyes by itself, or a small automatic toy, will cause
no notable sensation of this kind, while on the other hand, for example, the life-size
machines that perform complicated tasks, blow trumpets, dance and so forth, very
easily give one a feeling of unease. The finer the mechanism and the truer to nature
the formal reproduction, the more strongly will the special effect also make its
appearance.” (Jentsch, 1906/1997, p. 12)
18 | P a g e
2 Understanding the Uncanny Valley
Fifty-three years after Jentsch wrote his essay, the first industrial robot arm was created in
1959. The designers of robots soon began to develop more human-like robots with, for
example, skin and hair to cover up the mechanical parts. These kind of robots were called
android robots (Tinwell, 2015, p. 6). Then, in 1970 the Japanese roboticist Masahiro Mori
noticed that the more a robot was given humanlike characteristics, the more it increased the
sense of familiarity until a point on which it became uncanny to the observer. He calls this
graph the uncanny valley or bukimi no tani in Japanese. He describes a hypothetical curve of
appearance versus shinwakan for how a human perceives a robot. Shinwakan is until then a
non-existing word in the Japanese language. There is not just one word to translate it in
English. Words that are used for shinwakan are familiarity, rapport, comfort level and likeability
(Ho & MacDorman, 2010, p. 1508; MacDorman, et al., 2009, p. 695; Bartneck, et al., 2009, p.
270).
For the negative of Shinwakan, Mori used the Japanese word bukimi. The direct translation of
bukimi is in fact not uncanny but its meaning is more to be translated as weird, ominous or
eerie. Synonyms of uncanny include unfamiliar, eerie, strange, bizarre, abnormal, alien,
creepy, spine tingling, inducing goose bumps, freakish, ghastly, and horrible. (MacDorman &
Ishiguro, 2006, p. 312; Ho & MacDorman, 2010, p. 1508).
Figure 1 is the graph Mori published in his essay. At the start of the curve there is the industrial
robot, which is designed to overtake tasks from humans. An industrial robot has hardly no
comparison with a human being. Its functionality is only based on the desires of what it should
be able to do according to the task it should fulfil. Therefore its appearance is not disturbing
in our human minds (Mori, 2012, p. 98; Tinwell, 2015, p. 7).
From here the curves rises to the point of where you can find, the next example of Mori, a toy
humanoid robot. These robots have a face, a torso, arms and legs. They are not designed for
functionality, like industrial robots, but for children to enjoy there familiarity. These humanoid
robots are approaching the peak before the valley on Mori’s curve (Mori, 2012, p. 98).
19 | P a g e
2 Understanding the Uncanny Valley
Figure 1:
The Uncanny Valley graph by Masahiro Mori (Tinwell, et al., 2011b, p. 328).
On the right side of the valley, and the end of this curve, we find a healthy person. But between
the first peak and this other side we can find the objects and creatures that have a more human
like appearance then the toy humanoid robot, but still do not have the appearance of a healthy
human being. They might have hair, fake skin and try to express emotions with their faces. As
they appear to be human we also expect them to behave like a human. But when this is not
the case, we get suspicious and get an uneasy feeling about them. Therefore, these robots
fall right down into the uncanny valley (Tinwell, 2015, p. 8). Mori gives the example of a
prosthetic hand that looks very realistic, but when shaking the hand the familiarity disappears
because of the cold and fake tissue, which we do not perceive as human skin (Mori, 2012, pp.
98-99).
“Recently prosthetic hands have improved greatly, and we cannot distinguish them
from real hands at a glance. Some prosthetic hands attempt to simulate veins,
muscles, tendons, finger nails, and finger prints, and their color resembles human
pigmentation. So maybe the prosthetic arm has achieved a degree of human
verisimilitude on par with false teeth. But this kind of prosthetic hand is too real and
20 | P a g e
2 Understanding the Uncanny Valley
when we notice it is prosthetic, we have a sense of strangeness. So if we shake the
hand, we are surprised by the lack of soft tissue and cold temperature. In this case,
there is no longer a sense of familiarity. It is uncanny.” (MacDorman & Ishiguro,
2006, p. 303).
The uncanny valley does not only occur with android robots but also with other realistic human
figures like dolls and near-human CG-characters in films and games (MacDorman & Entezari,
2015, pp. 141-142; Seyama & Nagayama, 2007, p. 337).
The essay of Mori did not have any scientific prove of the uncanny valley existence, it was a
report of his experiences. Although the essay of Mori was written in 1970, most of the studies
on this topic have been done in the last ten years. Not every research has proven the existence
of the uncanny valley. Bartneck and his colleagues cannot confirm the existence of the
uncanny valley after their experiment where participants interact with either a person or his
android doppelganger (Bartneck, et al., 2009). Though most of the empirical researches did
find evidence the uncanny valley exist (Burleigh, et al., 2013; MacDorman & Ishiguro, 2006;
Seyama & Nagayama, 2007; Tinwell, et al., 2011a; Tinwell, 2015). The uncanny valley is a
complex phenomenon and the question if it may ever be possible to cross the uncanny valley
with robots and CG-characters is still unsolved (Tinwell, 2015, pp. 183-184).
As been mentioned before, the Japanese word shinwakan has no direct English translation.
Therefore some studies have tried to come up with indices for the axis of the uncanny valley
graph that Mori published in 1970. Bartneck and his colleagues (Bartneck, et al., 2009) used
the word likeability as a translation to measure shinwakan. As already stated, this is the
research that did not confirm the existence of the uncanny valley. But after this study the
colleagues Ho and MacDorman did not agree with this outcome. In their opinion the index
likeability was not well chosen. The reason they give for this is that “likeability is virtually
synonymous with interpersonal warmth” (Ho & MacDorman, 2010, p. 1508). They determine, the
negative of warmth is cold and that is not the same as Mori’s negative, bukimi, which is
translated as eeriness and unfamiliar. They conclude:
“A person can be cold and disagreeable without being eerie – at least not eerie in
the way that an artificial human being is eerie.” (Ho & MacDorman, 2010, p. 1508)
According to Ho and MacDorman the not correctly chosen indices are the reason for the
negative outcome in the research of Bartneck et al (2009). In their opinion it should have been
more focused on the emotions that causes eeriness, like fear, anxiety and disgust. They
started a new research to find better suitable indices for the uncanny valley than Bartneck et
al. (2009) did. The result of this study was that the indices attractiveness, eeriness,
21 | P a g e
2 Understanding the Uncanny Valley
humanness and warmth were more suitable as indices for shinwakan than likeability is. (Ho
& MacDorman, 2010, pp. 1508-1517)
2.3.1
The Effects of Movement
In his article of the uncanny valley Mori (Mori, 1970) says: “Movement is generally a sign of life.”
As is shown in figure 1 of the uncanny valley graph, Masahiro Mori has created two curves.
The solid line indicates the uncanny valley when the object is not moving and the dotted line
represents the familiarity versus the appearance in terms of a moving object. In relation to the
non-moving curve, the curve of a moving object or creature has a bigger peak and a deeper
valley (Mori, 2012, pp. 99-100). An example is a dead human being. It still has a lot of
similarities in its appearance with a living human being but most people do feel uneasy when
looking at a corps. If the corps would start to move, this would terrified us (Kaba, 2013, p. 190).
When a human interacts with an android or humanlike CG-character, the human expect the
android or humanlike CG-character to behave correspondingly to their human-like
appearance, it should act like a human. If their sound or movement differs from what is
expected the human perceiver will be alerted because he or she is sensing strange behavior.
The uncanny feeling will be increased by this strange behavior (Tinwell, 2015, p. 29; Kaba,
2013, p. 190).
2.3.2
Individual Responses
Some of the researchers who have investigated the uncanny valley, have wondered if the
uncanny valley is a universal or an individual response. As been noticed in chapter 2.2 of this
thesis, not everyone experiences an uncanny feeling with the same degree of unease. It
appears that some people have a more uncanny feeling with robots and CG-characters than
other people. And researchers believe that also animals can experience the uncanny valley.
This part will be a short review of these researches.
2.3.2.1
Ages
In an experiment with a child android (named Repliee R1) the researchers Minato et al. (2004)
found out that one-year-old babies were attracted by the android but children in the ages
between three and five years were afraid of the robot and did not want to look at it. Because
the babies’ idea of how a human should react and behave is still underdeveloped, they
perceive the android as human. In contrast to that, adults have more experience with robots
as they had more time to get used to them. Therefore they have less expectations of the robot
to fit their idea of human behavior. They know the robot is not a human being and this tempers
22 | P a g e
2 Understanding the Uncanny Valley
their uncanny feeling. But young children are trying to match their idea of human behavior to
fit the robots behavior and it does not compare. Because of this the three till five year old
children got scared (MacDorman & Ishiguro, 2006, pp. 308-309; Tinwell, 2015, pp. 38-39).
2.3.2.2
Animals
The uncanny valley does not only appear between humans and near-human stimuli. A study
from Steckenfinger and Ghazanfar (2009) has demonstrated that also macaque monkeys
experience this uncanniness when they look at 3D modelled faces of other monkeys. In the
research they used images of real faces, realistic CG-faces and unrealistic CG-faces of a
macaque monkey. The result was that the monkeys preferred to look longer at the real faces
and unrealistic CG-faces. According to the researches this demonstrates the existence of the
uncanny valley phenomenon with monkeys. Though, the investigators could not conclude, if
the monkey did experience emotions of fear or disgust. For this more research is needed
(Steckenfinger & Ghazanfar, 2009, pp. 18362-18364).
2.3.2.3
Sensitivity to the Uncanny Valley
In 2006 MacDorman and Ishiguro thought that the uncanny valley is less of a problem in
Japan. They suggest this based of the fact that in the Japanese culture, robots are more
integrated in normal life and therefore the people are more used to them (MacDorman &
Ishiguro, 2006, p. 306; Ho & MacDorman, 2010, p. 1516). But according to the research of
Desthephe et al. (2015) this is not the case. In this research they have tested how French
people versus Japanese people felt towards robots. The conclusion of this research is that the
French people had warmer feelings and felt saver with robots than the Japanese people do
(Destephe, et al., 2015). This result was a confirmation of a cross-cultural research on the
attitude towards robots, done by Bartneck et al. in (2007). They found out that US participants
had the most positive attitude towards robots. They think that the Japanese culture is less fond
of robots because they realize the shortcomings of the robots better as they are more
integrated in the Japanese culture (Bartneck, et al., 2007, p. 225). Next to that, people who
play a lot of video games and/or use 3D software are not accepting CG-characters more than
people who have less experience with video games or 3D software. This is based on the paper
of Tinwell and Grimshaw from 2009 (Tinwell & Grimshaw, 2009; Tinwell, 2015, p. 183).
23 | P a g e
2 Understanding the Uncanny Valley
2.3.3
Examples of the Uncanny Valley
Mori suggest in his essay that designers should focus on keeping their robots on the first peak
before the valley and thereby preventing their robot design to fall into the uncanny valley. The
same applies to CG-characters (Seyama & Nagayama, 2007, p. 337). Animation studios like
Walt Disney Animation Studios (Disney) and Pixar Animation Studios (Pixar) are keeping their
characters on the safe side of the valley by giving their characters not a near-human but more
cartoonish appearance (Nakashima, 2011). Examples of this are Riley in the by Pixar
animated film Inside Out (see figure 2) (Inside Out, 2015) and the princesses Elsa and Anna
from Frozen by Disney (see figure 3) (Frozen, 2013).
Figure 2:
The character Riley from Pixar's Inside Out (left)
Available at:
(https://i.ytimg.com/vi/t4okAfKCwRk/maxresdefault.jpg).
Figure 3:
The characters Elsa and Anna from Disney's Frozen (right)
Available at:
(http://images6.fanpop.com/image/photos/39100000/Elsa-andAnna-frozen-39135031-1218-1858.jpg).
Pixar made one of the first animations short films with a character that ended up in the uncanny
valley. This was the short animation film Tin Toy (Tin Toy, 1988). Though the film was the first
CG animation short film to win an Oscar, the film industry was starting to take the uncanny
valley phenomenon seriously after all the negative reactions from the audience. These were
towards the baby character Billy (see figure 4). After this short film, Pixar did not released
another film with a near-human character. They have stayed on the safe side of the uncanny
valley (Barfield, 2015, p. 232).
In 2001 the film Final Fantasy: The Spirits Within (Final Fantasy: The Spirits WIthin, 2001)
came out (see figure 5). This was the first film which really got into trouble due to the uncanny
valley phenomenon. The creators expected the film to be a revolutionary success because it
was the first full CG film to use human-like CG-characters. But exactly that was the problem
24 | P a g e
2 Understanding the Uncanny Valley
with the movie. The characters were a big disappointment to the audience (Tinwell, 2015, p.
11). Tinwell declares:
“The main protagonist character, Doctor Aki Ross, was designed to be regarded as
an empathetic, human-like character, yet her jerky movements and unnatural and
emotionally limited facial expression evoked an adverse reaction from the audience.
Instead of being perceived as attractive and likeable, she was regarded as creepy
and strange.” (Tinwell, 2015, p. 11)
Figure 4:
The character Baby Billy from the short
film Tin Toy by Pixar.
Available at:
(http://simplymaya.com/articles/uncannyvalley/images/uncanny_01.jpg).
Figure 5:
Final Fantasy: The Spirits Within. The
character Doctor Aki Ross falls into the
uncanny valley.
Available at: (http://4.bp.blogspot.com/WckWWM7pvo8/UN8dmIKcpUI/AAAAAAAA
Aic/fcWEB49EPE8/s1600/fftsw.jpg)
In 2004 the animation film The Polar Express of Warner Bros. Entertainment Inc. (The Polar
Express, 2004) (see figure 6), as well as the animation film The Incredibles of Disney and
Pixar (The Incredibles, 2004) (see figure 7), were released. Both of the films had human-like
characters in the story but only the characters of The Polar Express were described as
“disturbing” and “creepy” (Kaba, 2013, p. 191; Clinton, 2004; Tinwell, 2015, p. 12). The
difference was: The characters of The Incredibles had a more cartoonish plastic appearance
and the characters of The Polar Express were created as realistic human beings (Kaba, 2013,
p. 191). Kaba describes the problem of The Polar Express as following in his article (2013, pp.
191-192):
“The dull expression in the eyes, the tightness in the skin and the pausing in the
motions prevented the audience from enjoying the film and made it impossible to
convey the desired feeling effectively. In the beginning of the film, when the child
character is in his bed, everything may seem fine until when he opens his eyes. The
way he opens his eyes, gets up, and looks puzzled when getting out of the door just
pulls the audience to a non-normal world and may feel creepy.”
25 | P a g e
2 Understanding the Uncanny Valley
Figure 6:
Three of the main characters of The Polar Express (left) (The Polar Express, 2004).
Figure 7:
The main characters from the film The Incredibles (right)
Available at: (http://mickeymousepictures.com/wp-content/uploads/2015/08/The-Incredibles-1.jpg).
Also Disney did not manage to stay on the right side of the uncanny valley with every animation
film they have made. According to all the negative reactions, the in 2009 released animation
film Mars Needs Moms has fallen right down into the valley. (Nakashima, 2011) And as been
discussed in the intro of this chapter, the film The Adventures of Tintin: Secret of the Unicorn
(The Adventures of Tintin: The Secret of the Unicorn, 2011) did get a lot of negative reactions
on the main character Tintin (see figure 8) (Pinkerton, 2011; Tinwell, 2015, p. 111; Rom, 2011;
Snyder, 2011; Kaba, 2013, pp. 192-193). The character of Captain Haddock (see figure 8)
was found less uncanny than Tintin. Kaba thinks this is because the appearance of Captain
Haddock is “between a caricature and a real man” due to his big nose (Kaba, 2013, p. 193).
Figure 8:
The characters Tintin (right) and Captain Haddock (left) in The Adventures of Tintin: The
Secret of the Unicorn (The Adventures of Tintin: The Secret of the Unicorn, 2011).
26 | P a g e
2 Understanding the Uncanny Valley
3.3.1
Motion Capture
The animations of the characters in the movies Final Fantasy: The Spirit Within, The Polar
Express, Mars Needs Moms and The Adventures of Tintin: Secret of the Unicorn are all
created by using the technique motion capture. With this technique an actor or actress has
markers all over his or her body (see figure 9). The markers are positioned so when they are
tracked by one or multiple cameras, they give the information about how the actor or actress
is moving. This information can be transferred to a CG-character causing it to have the
identical movement of the actor or actress (Tinwell, 2015, p. 10). Frequently, animators will
use keyframe animation to clean up the motion capture data. This is due to problems with the
tracking or the animation still does not look quite right. In the human face there are 43 muscles
which can create a facial expression with a very small movement. This must be capture by
high-definition cameras and still the data can be wrong. Motion capture is definitely not a
guarantee for a perfect animation (Tinwell, 2015, p. 18; Menache, 2011, pp. 50-52).
Figure 9:
2.3.3.2
Tom Hanks as motion capture actor for The Polar Express (left) and his CG-character in the same
position (right) (Bredow, et al., 2005).
Realistic but not Uncanny
There are characters in movies who are not cartoonish CG but realistic CG and they ‘play’
along real human beings in the movie but they are not negative connected with the uncanny
valley. An example is Davy Jones, a character in the film Pirates of the Caribbean: At World’s
End, with tentacles growing out of his face. Though Davy Jones is a complete CG-character,
his appearance in the movie was believable because he was supposed to be creepy. This
also applies for Gollum in The Lord of the Rings trilogy. His appearance is not human-like
enough to lose his believability as a character (MacDorman, et al., 2009, p. 695; Kaba, 2013,
p. 192). The makers of the movie Avatar also managed to keep their characters out of the
uncanny valley. In the movie the animated Na’vi people play alongside of real human actors
and actresses but because of their stretched human proportions and blue skin they are simply
not human enough to give the perceiver a creepy feeling. (Kaba, 2013, p. 192)
27 | P a g e
2 Understanding the Uncanny Valley
In chapter 4 of this thesis we will analyse some of these examples to see how the movements
of the eyes influences the characters.
2.4
Theories about the Uncanny Valley
In his essay, Mori thinks that the uncanny feeling we experience when we are seeing robots,
might occur to warn us for possible danger.
“The sense of eeriness is probably a form of instinct that protects us from proximal,
rather than distal, sources of danger. Proximal sources of danger include corpses,
members of different species, and other entities we can closely approach. Distal
sources of danger include windstorms and floods” (Mori, 2012, p. 100; Tinwell,
2015, p. 101).
Humans are highly sensitive to the imperfections of androids, both in how they look and how
they move (MacDorman & Ishiguro, 2006, p. 308). But after years of research on the uncanny
valley there still is not an exact explanation why this phenomenon exist (Seyama & Nagayama,
2007, p. 338). The next part will be an overview of the different experiments and theories about
the uncanny valley and what possible explanations there are of what is causing this
phenomenon. Most of the research on the uncanny valley has been done in the last ten years.
Some of the research and experiments are done on the subject of robots and the uncanny
valley. But because the uncanny valley phenomenon is also perceived with near-human CGcharacters, these researches can help a 3D artist as well in understanding the uncanny valley
and why an uncanny feeling occurs with the audience when they are looking at near-human
characters (Seyama & Nagayama, 2007, p. 337). In chapter 4 these theories will be combined
with the findings of chapter 3 Oculesics.
2.4.1
Doppleganger
An android can be seen as a doppelganger of one. The Japanese engineer and professor
Hiroshi Ishiguro has created an android with his own appearance. This android has got the
name Geminoid HI-1 (Bartneck, et al., 2009, p. 270). In his essay about the uncanny, Freud
has written about the theme of the doppelganger, which is extensively studied by his colleague
Otto Rank. In German the theme is called der Doppelgänger and this can be translated in
English as the double or doppelganger. The theme of the double is retrievable as shadow,
mirror, reflections the soul and ghost. It is a cultural theme that is closely connected with the
uncanny and death. An old superstition of German and Austrian culture is that one will die if
the light of a candle, which has been lit at Christmas- or New Year’s Eve, does not cast a
shadow of a person or a shadow with no head. In other cultures one will die if been stepped
28 | P a g e
2 Understanding the Uncanny Valley
into his or her shadow. As well as with death, the double is often connected with the bad and
evil. The double theme can be found in different forms of art like literature (Rank, 1925, pp.
68-69; Živković, 2000, pp. 122-124; St-Germain, 2010, pp. 1-3; Freud, 1919, pp. 234-236). An
example of an evil double can be found in The strange case of Dr. Jekyll and Mr. Hyde, a story
by Stevenson (1886/2015). In this story Dr. Jekyll managed to separate the good and the evil
in himself by drinking a self-made poison. His good form is represented by Dr. Jekyll and his
evil form is Mr. Hyde. Being in the form of Mr. Hyde, he commits crimes which he would never
had done as Dr. Jekyll. As the story develops Dr. Jekyll increasingly loses the control over Mr.
Hyde who ends up murdering someone. He kills himself after he realizes that he cannot
controle Mr. Hyde anymore, because he has run out of ingredients to make the antidote to
become Dr. Jekyll. It is only clear in the end of the story that Dr. Jekyll and Mr. Hyde are the
same person (St-Germain, 2010, p. 1; Stevenson, 1886/2015).
In the story of the Sandman, the doppelganger is as well a theme (St-Germain, 2010, p. 5).
This is the character of the lawyer Coppelius and his double, the Italian barometer seller
Coppola. These doppelgangers are both evil. One is connected with the death of Nathanael’s
father and the other drives Nathanael to madness when he has removed Olimpia’s eyes. The
question if Coppelius and Coppola are the same person is never really answered in the story
but Freud is certain they are (Freud, 1919, p. 230): “For the conclusion of the story makes it quite
clear that Coppola the optician really is the lawyer Coppelius and also, therefore, the Sand-Man.” In
both the stories of The Sandman and in The strange case of Dr. Jekyll and Mr. Hyde, the
intellectual uncertainty is found whether the persons are really the same. This contributes to
the uncanniness in the stories (Jentsch, 1906/1997, p. 13; Schweigert, 2010, pp. 27-28).
Not only the connection with bad, evil and death are so uncanny about a doppelganger-robot,
but also being replaced by an android is what can result in fear for the robot. It is a scary
thought that one day they can have the same intelligence as a human being (Ho, et al., 2008,
p. 170).
2.4.2
Expectation Violation
People have an idea or expectations of how a person should react and behave in a situation.
An android robot falling into the uncanny valley may have to do with these expectations
(MacDorman & Ishiguro, 2006, p. 301). Androids do not satisfy this expectations because they
are still not able to respond to the person as a normal living human would respond. They make
believe to have a human brain but due to their behavior they clearly have not. Thereby the
androids violates the expectations of how someone should behave in interaction (MacDorman
& Ishiguro, 2006, p. 309; Tinwell, 2015, p. 8; MacDorman & Entezari, 2015, p. 144).
29 | P a g e
2 Understanding the Uncanny Valley
2.4.3
Mixed Human Identity and Cognitive Dissonance
Ramey (2005) thinks that the uncanny valley appears because two categories (‘human’ and
‘robot’) which are normally separate from one other, now are mixed together. The problem
with this is that the perceiver cannot make up his mind whether the character in front of him
or her is a human or a robot. Especially if one of the categories is ‘human’ cause a conflict
with the perceiver about his or hers own humanity and identity (Tinwell, et al., 2011b, p. 329;
MacDorman & Ishiguro, 2006, p. 310). Ramey says:
“Uncanny valleys can arise in any cognitive act that links qualitatively different
categories by quantitative metrics that call into question the originally differentiated
categories. This effect can be especially pronounced when one of those categories
is one’s self or one’s humanity. From a phenomenological standpoint, humanlike
robots may force one to confront one’s own being by creating intermediate
conceptualizations that are neither human nor robot.” (Ramey, 2005)
Ramey’s theory is the base of the explanation of Plantec in 2007. He explains the uncanny
valley due to the theory of cognitive dissonance. What happens is that the perceiver’s
subconscious mind knows it is not a real human he or she is looking at and therefore it is not
excepted as human being, even if the perceiver knows it is a virtual character who is trying to
play a human part. Subconsciously, the perceivers will feel fooled by the CG-character. In
other words, it is a conflict between two or more opposing beliefs, emotions and feelings a
person can have at the same time (Plantec, 2007).
2.4.4
Evolutionary Aesthetics
Another explanation of the uncanny valley is the fact that androids deviate from the norms of
physical beauty (MacDorman & Ishiguro, 2006, p. 310). Most people are attractive to a person
of the other sex. Typically, when we are attracted to someone, this is because we think this
person is more fertile than another person. Therefore we can say that beauty is an indicator
to how successful this person will or can be in reproducing itself. Youth is a significant feature
of beauty, as well as vitality, skin quality and the facial and body proportions. Youth and vitality
can been seen in how a person moves. Another important feature of beauty is bilateral
symmetry which indicates the person is healthy. To most woman, a man with a high waist-tohip ratio is considered the most attractive. For most man this is the other way around when
they look at a woman. They like woman to have a low waist-to-hip ratio and a body mass index
that is mid-level. Also woman with large breast are a sign of fertility (MacDorman & Ishiguro,
2006, pp. 310-311; MacDorman & Entezari, 2015, p. 148).
30 | P a g e
2 Understanding the Uncanny Valley
Thus, the norms of beauty are universal and relate to the physical appearance of a person
and their movements. Beauty stands in close relation with the fitness, fertility and health of a
person. In the process of selecting or rejecting a mate, we judge a person to be attractive or
unattractive, otherwise said, it is a binary decision between attraction and aversion. According
to this, MacDorman and Ishiguro (2006, p. 311) conclude:
“A robot is uncanny when subconscious processing produces sensations that motivate the
beholder to reject it. The uncanny valley may be a symptom of the need to exaggerate
differences along the boundaries of such classes as male versus female, fertile versus infertile,
child versus adult, living versus dead, human versus nonhuman, and familiar versus unknown.”
(MacDorman & Ishiguro, 2006, p. 311)
2.4.5
Theory of Disgust
It is also possible that the natural defence mechanism of disgust is involved in the
phenomenon of the uncanny valley. Rozin et al. (Rozin & Fallon, 1987; Rozin, et al., 2008)
describes disgust as a natural defence mechanism which should avoid that the body gets
infected with disease. This defence mechanism becomes stronger if another organism is
genetically more related to the human being. This is because it is more presumable that this
other mechanism is carrying bacteria or viruses that can harm the human being. This is also
a reason why we find one person more attractive than the other person. We still feel the
pressure which our ancestors also felt, to reproduce our self with a mate who maximize the
probability of a fit and healthy offspring. Because we want to mate with this other human,
which we find attractive, we do not feel any disgust for this person. In other words, we do not
feel any disgust for organisms, which genes are different than a human being and we don't
feel any disgust for a human we want to mate with. But others we might perceive as eerie,
because they are diseased or we think they have bad genes (MacDorman & Ishiguro, 2006,
p. 312; MacDorman & Entezari, 2015, p. 148).
As been written in chapter 2.3, disgust is not one of the synonyms of uncanny or eeriness. But
they may have the same function in keeping us safe from diseases and bad genes
(MacDorman & Ishiguro, 2006, p. 312). In his essay on the uncanny valley Mori thinks about
the relationship between uncanny and death:
“And when we die, we fall into the trough of the uncanny valley. Our body becomes
cold, our color changes, and movement ceases. Therefore, our impression of death
can be explained by the movement from the second peak to the uncanny valley…
We might be happy this line is into the still valley of a corpse and not that of the
living dead! I think this explains the mystery of the uncanny valley: Why do we
31 | P a g e
2 Understanding the Uncanny Valley
humans have such a feeling of strangeness? Is this necessary? I have not yet
considered it deeply, but it may be important to our self-preservation.” (Mori, 1970)
This brings us to the next part, where we will look at the possible explanation of the uncanny
valley and what its relation with death is.
2.4.6
Terror Management and Subconscious
The next theory about why the uncanny valley appears is about the fear of death. Human
robots may occur uncanny to us human beings because they remind us of our own mortality
(MacDorman & Ishiguro, 2006, p. 328). Naturally, like other living beings, the purpose of a
human being is to reproduce itself and prevent to die before it is able to reproduce itself. But
unlike any other species, the human being knows that death is inevitable and someday dying
happen to him or her (MacDorman & Ishiguro, 2006, p. 329).
“It is awesome to be alive and know it, and dreadful to recognize that death is one's
inevitable fate as an ambulatory assemblage of blood, tissue, and guts, inherently
no more significant or enduring than a barnacle, a beetle, or a bell pepper.”
(Solomon, et al., 2000, p. 200)
Because of this, a human being will try to avoid situations and circumstances that can lead to
the death of this human (MacDorman & Ishiguro, 2006, p. 329).
The theory of how humans manage their fear of death has been supported by over 80
experiments (Solomon, et al., 2000, p. 200) and is known as the Terror Management Theory
(TMT) (MacDorman & Ishiguro, 2006, p. 328; Solomon, et al., 2000, p. 200; Pyszczynski, et
al., 1999, p. 835). It is based on the book The Denial of Death by Ernest Becker (Becker,
1973). The TMT is a dual-process model. Distinct defensive processes will start to work when
a person has conscious or subconscious thoughts about death. The TMT distinguishes
between distal defensive processes and proximal defensive processes. Proximal defences
suppress the thoughts of death or weaken the terrifying meaning of it by rationalization.
Examples of this defensive process are: starting to think about a different subject or rationalize
thinking like “My grandmother lived to be 90” (MacDorman & Ishiguro, 2006, p. 329).
A human being can also have subconscious thoughts about death. When these thoughts
occur, distal terror management defences processes will start to mitigate the anxiety and
debilitating terror. This is managed by the development of the faith in cultural worldviews and
self-esteem. The cultural worldview reduces the fear of death. Although cultures are different
in what they belief, through concepts as the soul and the afterlife (e.g. by giving literal or
symbolic explanations about what happens after dying) (MacDorman & Ishiguro, 2006, p. 329)
32 | P a g e
2 Understanding the Uncanny Valley
they share the same claims that “the universe is meaningful, orderly and immortality is attainable.”
(Solomon, et al., 2000, p. 201) By being part of a culture and through accomplishments like
writing a book, gaining fame, having children, we try to outlive ourselves. Hereby the selfesteem of a human being increases and the person will feel more valuable (Solomon, et al.,
2000, p. 201). When the subconscious thoughts develop up to the point where they enter the
consciousness, the proximal terror management defences start to work (Pyszczynski, et al.,
1999, p. 835).
Other than mechanical-looking humanoid robots, androids are robots that are designed to be
indistinguishable from humans. The question that rises from the TMT in combination with
these androids is, whether they give us an eerie feeling because they remind us of our own
death and mortality and that therefore terror management defence processes are starting to
work? For example, an android which is not moving like a normal human being, may look dead
to us. Conscious or subconscious, this could remind us of death and thus the terror
management defence processes are influencing our attitude about the android (MacDorman
& Ishiguro, 2006, p. 330).
To see how TMT and the uncanny valley related to another, K.F. MacDorman and H. Ishiguro
(2006) did an experiment. The goal of this experiment was to test if the appearance of an
uncanny android elicits the same defence mechanism that the reminders of death do. In this
experiment they showed an image of an uncanny-looking android to one group of participants
and an images of a young Asian woman to the control group. After this the participants were
questioned. These questions were designed to see how or if their distal terror management
defense would react by choosing the answers that supported their worldview the most
(MacDorman & Ishiguro, 2006, pp. 336-337).
MacDorman and Ishiguro (2006) discovered with this experiment that the group which has
been shown the android, relative to the control group, preferred information that supported
their worldview.
2.4.7
Lack of Empathy
Part of what makes us human, is the fact that we can feel and show empathy towards another
person. Empathy is the ability to recognize an emotional state another person is in and also
the ability to respond to this emotional state with an appropriated reaction. A person can read
another person’s emotional state by the facial expression, speech and movement of this other
person. If we can understand what another person is feeling, we can respond to that with
compassion and sympathy. Because of the importance of this ability to feel and show empathy
33 | P a g e
2 Understanding the Uncanny Valley
for another human being, Dr. Angela Tinwell of the University of Bolton, comes to the
hypothesis that the lack of empathy in a character towards us is a possible explanation of why
the uncanny valley exist (Tinwell, 2015, p. 100). When we cannot sense any empathy toward
us from a human-like character, this could remind us of, or warn us for, a dangerous
personality. Thus concludes Tinwell (2015, p. 103):
“Therefore, perception of a lack of empathy in a character may suggest to the viewer
that the character cannot experience the affective state of the viewer (experience
emotional empathy), understand (cannot process cognitively) the thoughts (or
feelings) of the viewer or demonstrate compassion toward them.” (Tinwell, 2015, p.
103)
Psychopathy is an antisocial personality disorder. Someone who has this mental disorder has
problems with the ability to feel empathy for another person. Commonly, most of the
psychopaths are male. Psychopaths are self-centered, deceptive, have shallow emotions and
show a lack of empathy. They will try to get their way by manipulation, charm or violence.
(Tinwell, 2015, p. 113)
Tinwell thinks that if we are using the scale of Ho and MacDorman (Ho & MacDorman, 2010)
(See 2.3) to measure how uncanny a character is, we are actually measuring whether the
characters shows traits of psychopathic. (Tinwell, 2015, p. 105) She also proposes:
“That the essence or cause of the Uncanny Valley lied in the perception of a lack of
empathy in a human-like agent, which may suggest psychopathic tendencies in that
character” (Tinwell, 2015, p. 103)
There is an automatic reflex a human has when he or she is confronted with a fighting or
surprising sound or sight. This reflex communicates the shock we are feeling and the emotive
state we are in by widen our eyes which, our pupils dilate and we lift our eyebrows. This is
called the startle reflex and happens together with our body leaning back or jumping up and
our blood pressure increases (Tinwell, 2015, p. 106; Ekman & Friesen, 1985, p. 1416).
If a person with psychopathy is confronted with a shocking or frightening situation, he or she
does not show the startle reflex in able to communicate the emotional state of their mind at
this time. Given this fact, it is for everyone who sees someone else respond to fear or surprise,
visible when this person is psychopathic due to the lack of the startle reflex (Tinwell, 2015, p.
106).
The conclusion of the research of Tinwell, et al. in 2011 is that a human-like virtual character
with no movement in his upperpart of the face is concealed the most uncanny, especially when
34 | P a g e
2 Understanding the Uncanny Valley
the human-like character is showing the emotions fear and surprise. According to Tinwell et
al. (2011a) the uncanny feeling is due to the unsettling moment when someone comes across
a psychopathic person. The feeling of uncanniness can be triggered because we sense the
possible danger that we might be confronted with a psychopathic person (Tinwell, et al.,
2011a; Tinwell, 2015, p. 106).
In 2013 Tinwell and her colleagues (Tinwell, et al., 2013) did another experiment to see if it
confirmed the results from 2011. In the research of 2011 the research team only used male
stimuli. One of the purposes of the new research was to see which particular negative
personality traits may evoke perception of the uncanny (Tinwell, 2015, p. 107). It appeared to
be that the negative personality traits that can also be found in psychopathy, gave the
participant the most uncanny feeling. For example personality traits as being angry, having a
cold personality and being dominant (Tinwell, 2015, pp. 107-108). In the research of 2013, the
new team also tested how participants react to human-like females. As in 2011, the research
of 2013 had three stimuli for the male. These were a human actor, a 3D model of a humanlike male with movement in the whole face and a 3D model of a human-like male face who
had no movement in the upperpart of the face. For the female stimuli they used a human
female actress and 3D models of a human-like female face. Here as well they have used one
3D model with movement in the whole face and one with lack of movement in the upperpart
of the face. The results of this research was that the female stimuli were rated less uncanny
than the male stimuli (Tinwell, 2015, p. 110; Tinwell, et al., 2013, p. 1623).
Due to the lack of the startle reflex perceivers can feel a possible danger that the character
might be psychopathic. But Tinwell comes with a further explanation of why we may reject a
character with no startle reflex. The CG-character is not able to communicate his emotions as
how a perceiver may expect this. Therefore the perceiver fails to understand them and feel
any empathy for this character. Tinwell concludes:
“As in real life, we do not want to be rejected or misunderstood, and we may be
concerned that a human-like character that is incapable of feeling more vulnerable
negative emotions may reject us for doing so! As a solution, we reject that character
first to prevent him or her from making us feel any (more) uncomfortable.” (Tinwell,
2015, pp. 119-120)
2.4.8
Mind Perception
As described above, the experiments and the explanations of the uncanny valley of
MacDorman & Ishiguro (2006), Tinwell et al. (2011) and Tinwell et al. (2013) focus on the
physical appearance of the robots and CG-characters. But Gray and Wegner (2012, p. 126)
35 | P a g e
2 Understanding the Uncanny Valley
think it is more about what the observer perceives when he or she sees a robot with humanlike
features. He or she will also expect it to have a mind. It can be said that it is disturbing to think
an android has a mind like a human. In their research Gray and Wegner ask themselves what
kind of minds enhanced the feeling of unease in the opinion of the perceiver. An older research
of Gray (Gray, et al., 2007):“suggests that people perceive mind along the two independent
dimensions of agency and experience” (Gray & Wegner, 2012, p. 126; Gray, et al., 2007, p. 619).
“Agency is the capacity to do, to plan and exert self-control; experience is the
capacity to feel and to sense. Adult humans are seen to possess both agency and
experience, and some have suggested that agency is the essential feature of
human minds because it separates us from animals.” (Gray & Wegner, 2012, p.
126)
In the research of Gray, Gray and Wegner (2007) about experience and agency, they did an
experiment to test what people expect from different subject cases. One of the outcomes from
this study was that a baby was expected the have experience but no agency, a robot only a
bit agency but no experience and an adult human to have both agency and experience (Gray,
et al., 2007, p. 619).
In 2012 Gray and Wegner did experiments with participants to find out what they found
disturbing about a non-human robot and a supercomputer. Was this agency, experience or
both? They thought, the most disturbing is something that is not human but can experience
things like it is a human. For example things like hunger, fear and pain. In these experiments
the participant had to rate statements about a human-like robot, a supercomputer and a
human man. The subjects all had different states, which were described as ‘normal’, ‘withagency’ and ‘with-experience’. Examples of the statements are: “This robot has the capacity to
feel pain” or “This robot has the capacity to plan actions” (Gray & Wegner, 2012, p. 126). The
outcomes of the research confirmed the hypothesis of the authors. Robots and other machines
who have the capability to experience emotions, feel and sense things, are perceived as to be
more uncanny than a robot or other machine that does not have this capability. On the other
hand, humans who appear to do not have the capability to experience, make other people feel
uneasy. A person without agency does not give another person an uncanny feeling (Gray &
Wegner, 2012). Schein and Gray conclude this in one sentence: “Robots are creepy when they
seem to feel” (Schein & Gray, 2015, p. 174).
2.5
The Eyes and the Uncanny Valley
After the possible theories of why the uncanny valley exists, it is now time to look at some
researches that have investigated the relationship of the eyes (and in some cases in
36 | P a g e
2 Understanding the Uncanny Valley
combination with animation of the upperpart of the face) and the uncanny valley. This part of
the thesis will be a review of the researches of these combined themes.
In a horror movie, altering the eyes of a character can make that character even more
horrifying to the audience. For example, zombies have white eyes and the eyes of vampires
are red. Another example of a horrifying character is the character Pale Man (see figure 10)
in the Academy Award winning film Pan’s Labyrinth, directed by Guillermo del Toro (2006).
Pale Man is a character who has no eyes in his head but instead he has his eyes in the palms
of his hands. Changing the eyes of a character easily makes the character creepy (Schein &
Gray, 2015, p. 173).
Figure 10: Pale Man from the movie Pan's Labyrinth from
Guillermo del Toro.
Available at:
(http://41.media.tumblr.com/tumblr_m4o4punTWq1
qj22x9o1_1280.jpg)
The uncanny valley and the eyes of a character has been studied before by several
researches (Schein & Gray, 2015; MacDorman, et al., 2009; Tinwell, et al., 2011a; Tinwell, et
al., 2013). In the study of Seyama and Nagayama (2007) they found out that a character gives
a more uncanny feeling to the perceiver when the eyes get more abnormal. This experiment
was done by morphing artificial and real human faces. Participants had to rate the scale of
pleasantness of the image shown. In the first part of the experiment they morphed the eyes
and the face of a human and a CG-character. Here the uncanny valley effect was perceived
most when the eyes and face degree of realism was the largest apart. For example, the
unadjusted eyes of the CG-character are placed in the face of a human. Also, in the next part
of the experiment, they scaled up the eyes comparing to a face of the stimuli and here as well
the uncanny effect appeared. (Schein & Gray, 2015, p. 173; Seyama & Nagayama, 2007)
Centuries ago Shakespeare already noticed the importance of the eyes in living beings. His
quote “The eyes are the windows to the soul” (Schein & Gray, 2015, p. 174) is well known and
37 | P a g e
2 Understanding the Uncanny Valley
indicates the important role the eyes have in showing ones identity and emotions. We rely on
what we perceive when we look into someone’s eyes to decide if a person is a real person or
if he or she is an automatic (Schein & Gray, 2015, p. 174). Schein and Gray did a research to
see if a face with no eyes is more uncanny than a face with no nose. The experiment was
done by having participants rated a headshot of a man. In some of the pictures the eyes or
nose were cut out of the picture. Like the researchers predicted, the participants found the
picture with the eyes cut out more uncanny than the unedited picture or the one with the nose
cut out. Also the eye-less picture was “more likely to be missing the capacity for emotional
experience of pain and fear […] and more likely to be soulless” (Schein & Gray, 2015, p. 175).
In the researches of Tinwell et al. (Tinwell, et al., 2011a; Tinwell, et al., 2013) it is shown that
the lack of animation in the upperpart of the face in CG-characters is increasing the uncanny
valley effect. As has been previously described in this thesis (see chapter 2.4.7), Tinwell
beliefs this is due to the missing of the startled reaction which is also missing in mental ill
humans, who are psychopathic. In this research they studied how the lack of upperpart facial
movement in the upperpart of the face effected the ‘uncanniness’ of the character. The
upperpart facial movement includes the eyelids, eyebrows and forehead. A second goal of the
research was to see if the quantity of the uncanniness would vary between different emotions.
They investigated “six emotions: anger, disgust, fear, happiness, sadness and surprise in addition to
a neutral expression” (Tinwell, et al., 2011a, p. 741). These emotions where communicated to
the participants of the research with a male human actor and two 3D models of a male human
face, one with the whole face animated when showing the emotions and one without the
upperpart of the face animated. The conclusion of this research was that the 3D model with
lack of movement in the upperpart of the face was considered the most uncanny to the
participants. Also, the emotions fear, sadness and to a lesser extent disgust and surprise
where increasing the uncanniness of a character. (Tinwell, et al., 2011a, p. 748)
In the follow-up research of Tinwell et al. (2013) they tested how the participants would react
on female characters as well. The stimuli used were again a picture of an actor and actress,
a male and female CG-character with and without upperpart facial movement. The result of
this research was that the participants had the most uncanny feeling with the stimuli without
the upperpart facial movement but the female characters where overall perceived as less
uncanny that the male characters (Tinwell, et al., 2013).
38 | P a g e
2 Understanding the Uncanny Valley
2.6
Summary
This chapter has discussed the concept of the uncanny valley. In the first two decades of the
20th century, Jentsch and Freud have both written an essay about the theme uncanny. Their
conclusions where that intellectual uncertainty and secrets that are revealed but should have
stayed hidden are what can give a person an uncanny feeling.
Mori has proposed the uncanny valley graph. With this graph he explains that the more a
character or object becomes human, after one point it becomes unsettling and will give the
viewer an uncanny feeling. Researchers are not certain why this phenomenon exists. There
are some theories about it. It might be because of the natural fear of death humans have or
due to the fact that we think the near-human characters or robots cannot feel any empathy
with us.
As we can see in this last part of the chapter, not a lot of research has been done to the
influence the eyes have on the uncanny valley. This is quite strange because of all the negative
reactions there are about film and game characters and their eyes. As been written before,
the uncanny valley is not a problem of only the eyes but it is something that is mentioned in a
lot of the reviews. Also there has not been done a lot of research about the uncanny valley
with animated stimuli. In almost every research the stimuli used were still images. This is
remarkable because Mori predicts in his graph that motion will increase the uncanniness of a
near-human character or robot. In the next chapter we will look at oculesics, the body language
of the eyes.
“It isn’t that clear what causes it but some people think that it is all due to the eyes.
Our brains are so well trained to know what a human is supposed to look like that
something as small as a twitch in the eye or the wrong dilation of the pupil can make
it all seem strange.” (Menache, 2011, p. 51)
39 | P a g e
3 Understanding Oculesics
3
Understanding Oculesics
As been discussed in chapter 2, the facial expression of the CG-character can be
unconvincing to the perceiver. This can result in a creepy feeling which the perceiver
experiences. As we have seen in chapter 2.5, the eyes of a character contribute to the problem
of the uncanny valley. In this chapter we will look at oculesics. This term is not used very often.
The book A Dictionary of Media and Communication defines oculesics as “The study of the
communicative role of the eyes in nonverbal communication” (Chandler & Munday, 2011).
Richmond, McCroskey and Payne defines it as following (1991, p. 14): “The study of the
communicative aspects of the eye behavior is known as “oculesics”.” Because it is not clearly defined
what parts of the eyes are studied in oculesics, this chapter will cover the movements of the
facial expressions in the upperpart of the face, pupil dilation, blinking rate, the eye movement
pattern saccades and fixations and the eye movements in object related action and human
interaction. Important is to note that the focus of this chapter lies on the movements of the
eyes and the upperpart of the face, like the eyebrows and forehead. The information of this
chapter will be used for the analysis in the next chapter were we will discuss what the
association is between the eyes of a character and the uncanny valley. But first we will start
off with an overview of the muscle anatomy and the anatomy of the eyeball as these terms will
be used throughout the whole chapter.
3.1
The Anatomy of Facial Muscles and the Eyes
This part of the thesis gives a short overview of the anatomy of the muscles of the face and
the important parts of the eye. To this information will be referred to in the rest of this chapter.
3.1.1
Facial Muscles in the Upperpart of the Face
The muscles of the face, which can create facial expressions, are thin and flat muscles. The
muscles are attached to either, the skull and the skin of the face, or between to two points of
the skin. Hereby, when the muscles contract, the skin moves, builds wrinkles and furrows
whereby a facial expression occurs. There are, for example, muscles that can lift or bring down
the eyebrows and muscles that can dilate the eyes. In elderly people the skin is not as flexible
as the skin of a younger person, therefore the wrinkles of an elderly person stay longer visible
(Kahle, et al., 1984, p. 312).
On the forehead of the face the muscle with the name frontalis is responsible for lifting the
eyebrows. When this muscle contracts, the skin on the forehead is drawn together which
results in horizontal wrinkles on the forehead. The frontalis is connected with the muscle that
40 | P a g e
3 Understanding Oculesics
orbits the eye. This one is called the orbicularis ocluli. The part on the outside of this muscle
is called the pars orbitalis and it is responsible for closing the eyes when a person is squeezing
them tight. The inner part of the orbicularis ocluli, the pars palpebralis, is connected to the
eyelids and its main function is to let the eyelids blink (see figure 11) (Kahle, et al., 1984, pp.
312-314). The antagonist of the pars palpebralis is the levator palpebrae superioris muscle.
This one raises the upper eyelid. This muscle is not completely underneath the skin of the
face but is on the one side attached to the upper eyelid and on the other side to the back of
the eye where all the eye muscles come together. These other eye muscles control the rotation
of the eye (Faller & Schünke, 2012, p. 640; Ekman, et al., 2002, p. 24).
The corrugator supercilii is the muscle that draws the eyebrows together and the depressor
supercilii pulls them downwards. The procerus muscle is connected to the skin of the forehead
and the bridge of the nose. It brings the inner eyebrows down. Contracting these three muscles
results in a frown between the eyebrows. The muscles that runs down both sides of the nose
is called the levator labii superioris alaeque nasi. This muscle pulls the upper lip upwards,
opens the nostrils and squeezes the skin of the nose together which will wrinkle the skin on
the nose (Kahle, et al., 1984, p. 314).
Figure 11: The facial muscles of the human face.
Available at:
(https://montereybayholistic.files.wordpress.com/2013/01/facial-muscles.jpg)
(Modified by the author)
41 | P a g e
3 Understanding Oculesics
There are two more muscles that are worth mentioning to support the information in the
following chapters. These muscles are located in the lower part of the face. The first one is
responsible for giving the mount a smile-shape. The name of this muscle is the zygomaticus
major and it pulls the corner of the mouth outwards and upwards. The second one is called
the buccinator and this one tightens the corner of the mouth and pulls it inwards (Kahle, et al.,
1984, p. 316).
3.1.2
The Anatomy of the Eye
In this chapter we will refer to the anatomy of the eye. Therefore, here follows an overview of
the main parts of the eye.
Figure 12: A side view of the human eye.
Available at: (http://www.arizonaeyeconsultants.com/wpcontent/uploads/2015/09/EyeAnatomy600x540.jpg)
(Modified by the author)
The side view of the eye can be seen in figure 12. When we look at a scene, we can only see
things when there is a light source. The light rays of this source are reflected by the objects in
the scene. When a reflected light ray reaches the eye, we are able to see. The process of light
entering the eye is as following. When the light ray reaches the eye, it first passes through the
pupil of the eye. The size of the pupil is controlled by the iris. Depending on the amount of light
that the eye reaches, the iris enlarge or shrinks the opening. This is controlled by the muscles
under the iris. The pupil is perceived as black because there is no light reflected from the
surface in the back of the eye. This surface is the retina. This surface of the retina is covered
with receptor cells which are sensitive for light intensity and colours. These are processing the
light and send the light impulses through the optic nerve to the brain. Here it will be processed
into an image. The lens of the eye is located between the pupil and the retina. Its goal is to
produce visually sharp images on the retina, depending on the distance from the eye to the
object. (Faller & Schünke, 2012, pp. 622-629).
42 | P a g e
3 Understanding Oculesics
As can be seen in figure 12, in a straight line directly behind the pupil and the lens, there is a
small depression in the retina. This is called the fovea and the light rays that fall on this part
of the retina forms the brightest and sharpest part of our vision (Faller & Schünke, 2012, pp.
629-630).
3.2
Nonverbal Communication
This part will be a global introduction about nonverbal communication before we discuss
oculesics extensively. Nonverbal communication reveals much more of the emotions of a
person than what they say when they speak:
“Nonverbal behavior provides valuable information when we can’t trust what we are
told in words, either because the speaker is purposefully trying to deceive us, or
because he has blocked or repressed information we want” (Ekman & Friesen,
1969, p. 52)
The book Nonverbal Communication in Human Interaction of Knapp and Hall (1992) starts
with a story about the horse Hans, nickname Clever Hans, and his owner Herr von Osten in
1900. Von Osten thought he had learned Clever Hans mathematics. Von Osten asked Clever
Hans to solve a math problem and the horse would respond with tapping the correct number
on the ground with his front hoof. Hans was a big success and soon he got the attention of
several researches. They wanted to find out if it was really true that Clever Hans could count.
But after some tests they found out that Hans could not answer a question when the
questioners did not know the answer. For example when two researchers separately gave
Clever Hans a number and he had to add them together. This, without the researchers
knowing the number of each other. It turned out that Hans was reading the body language of
the people who knew the answers. These people would relax their body tension, raise their
eyebrows or widen their nostrils when the right answer was reached. This was the signal for
Hans to stop tapping with his hoof (Knapp & Hall, 1992, pp. 3-4). Knapp and Hoof explain
what we can learn from this story:
“The story of Clever Hans makes two important points regarding the role of
nonverbal behavior in human encounters: (1) while we are in the presence of
another person, we are constantly giving signals about our attitudes, feelings and
personality; and (2) other may become particularly adept at sensing and interpreting
these signals.” (Knapp & Hall, 1992, p. 4)
Not only these nonverbal signals but also the lack of these will be notice by other people.
Someone who cannot see expected signals will wonder why the nonverbal communication is
43 | P a g e
3 Understanding Oculesics
missing and might interpret the wrong reason for this. For example, a doctor who is trying to
be professional by not showing his emotions can be read by the patient as not interested or
withholding information (Knapp & Hall, 1992, pp. 4-5).
3.3
Emotions
We have learned about emotions in our childhood. When a child is crying a parent may ask
why it is sad. Hereby the child learns what the emotion sadness is. Shiota and Kalat mention:
“Ordinarily, when we talk about emotions, we refer to internal feelings as well as observable behaviors,
and we often invoke feelings as explanations for behaviors (“Don’t mind him, he’s just cranky today”)”
(Shiota & Kalat, 2012, p. 3). But how can an emotion be defined? It is hard to give an exact
definition of the word emotion. Joseph LeDoux says about it: “Unfortunately, one of the most
significant things ever said about emotion may be that everyone knows what it is until they are asked
to define it.” (LeDoux, 1996, p. 23). There is no clear definition of the word emotion, but between
the different formulations found in the literature, there are similarities. First of all, emotions
have a function, otherwise said, they are useful. “Our tendency to feel emotions has evolved
because in past generations, those who experienced emotions were more likely to survive, reproduce,
and become our ancestors.” (Shiota & Kalat, 2012, p. 4) An emotion will help us to take action.
An example of this is trying to escape when we feel fear. (Shiota & Kalat, 2012, p. 4) The
second similarity is that a person experience an emotion as a reaction to an external stimulus.
This means that an emotion is always about something. For example, someone is happy, sad
or angry about something (Shiota & Kalat, 2012, p. 4; Solomon, 2008, p. 12). Then the last
similarity: there are four aspects an emotion has. These are, cognition, feeling, physiological
changes and behavior. For example, someone gets good news (cognition), is feeling happy
about this (feeling), gets excited (physiological changes) and starts to smile (behavior) (Shiota
& Kalat, 2012, p. 5).
3.4
Facial Expressions of Emotions
The facial expressions of people have interested researchers for a long time. It began with
The Expressions of the Emotions in Man and Animals by Charles Darwin (1890/2009). He
thought that the expression of emotions on the face of humans are universal and Darwin used
information from informants in different countries across the world to prove this theory (Keltner
& Ekman, 2000, p. 236). He also suggest that even someone who tries to hide his emotions
for another person, the facial expression shows how he or she really feels. (Tinwell, 2015, p.
79)
44 | P a g e
3 Understanding Oculesics
Dr. Paul Ekman is a psychologist who has been researching nonverbal communication and is
specialized in facial expressions. Darwin inspired him to expand the research of universal
expressions. In the late sixties Ekman has travelled to Papua New Guinea to research the
facial expressions of an isolated and primitive tribe. Because the people of this tribe had no
influences of the outer world, he could conclude that there are facial expressions which are
universal identical (Ekman, n.d.).
According to Ekman, primarily emotions are displayed by the facial expressions, the rest of
the body contributes to express the arousal level. For example a clenched fist when someone
is angry (Ekman & Friesen, 1967, pp. 712-722; Knapp & Hall, 1992, p. 23). More than speech
and body movement, facial expressions are an important communication tool. (Tinwell, 2015,
p. 76)
“Very often we do manage to sense the inner states of others even though they try
to hide them. We feel sadness behind a faked smile, or bad intentions behind
seemingly generous actions. How do we do it? How do we manage to feel what is
concealed?” (Keysers, 2011, p. 9)
Through several experiments Ekman distinguished six facial expressions of emotions that are
universally recognized. These were: anger, disgust, fear, happiness, sadness and surprise
(Ekman, 1977, p. 98; Tinwell, 2015, p. 72). In 1999 Ekman published a new list which
contained a seventh universal recognized facial expression. This is the emotion contempt
(Ekman, 1999, p. 55).
In 1978, Ekman and his colleague Wallace V. Friesen developed a framework of all the
muscles movements which are involved when a person shows different facial expressions.
This is called the Facial Action Coding System (FACS) (Ekman, et al., 2002; Tinwell, 2015, p.
72). The facial muscles can work separately or combined with other facial muscles to create
expressions. The movements of these muscle, separately or combined, are referred to as
Action Units (AU). All the AUs have a different number indicating a facial movement. (Tinwell,
2015, p. 75). There are 44 unique AUs in the system. (Ekman & Friesen, 1982, p. 241; Ekman
& Rosenberg, 2005, p. 13) The FACS is used as a guide and has been integrated in different
3D facial animation modelling software. (Tinwell, 2015, p. 72)
3.4.1
Action Units in the Upperpart of the Face
For the facial movement of the seven basic emotions, there are three AU’s distinguished for
the eyebrows. Due to the movement of the eyebrows, these AUs influence the forehead as
well. One of the muscles which is responsible for creating these AU’s is the frontalis. This
45 | P a g e
3 Understanding Oculesics
muscle can contract on the inside, on the outside or on both sides. When the inside of the
muscle contracts we can see the AU1, also known as the Inner Brow Raiser. The muscle
contraction of the outside of the frontalis, is known as the Outer Brow Raiser and has the name
AU2. Both the AU1 and AU2 causes wrinkles on the forehead because the eyebrows are
pulled upwards and therefore the skin on the forehead is drawn together. The muscles which
are responsible for the frown between the eyebrows, are the corrugator supercilii, the
depressor supercilii and the procerus. When these contract we can see AU4, the Brow
Lowerer. Hereby the eyebrows are lowered but also pulled together. The frown visible with
AU4 causes one or more vertical or 45 degree angled wrinkles between the eyebrows. It
narrows the eye aperture as well (Ekman, et al., 2002, p. 17).
AU9 is an Action Unit that is visible in the lower part of the face as well as in the upperpart of
the face. This AU is called the Nose Wrinkler. The muscle that is responsible for this AU, the
levator labii superioris alaeque nasi, runs on both sides of the nose. When it is contracted the
skin of the nose is drawn together. It also pulls up the upper lip and it dilates the nostrils. In
the upperpart of the face between the eyes horizontal wrinkles appear. It may as well pull the
eyebrows lower and this narrows the aperture of the eyes (Ekman, et al., 2002, p. 39).
Now it is time to look at the Action Units that are visible when the different parts of the
orbicularis ocluli, the muscle orbiting the eye, contracts. There are three Action Units visible
in the basic emotions due to this muscle. As been written in chapter 3.1 of this chapter, the
outer part of the orbicularis ocluli is called the pars orbitalis. Contracting this part of the muscle
will result in AU6, the Cheek Raiser and Lid Compressor. This Action Unit is characterize by
the skin around the eye that is pulled towards the eye which narrows the eye aperture. This
happens because the cheeks are raised and this is pushing the skin underneath the eye
upwards. Also the upper eyelid is lowered, which narrows the eye aperture from above. In a
small extent it can lower the eyebrows. In the outer corner of the eye aperture, wrinkles appear
which are known as crow’s feet (Ekman, et al., 2002, p. 31). The AU7, the Lid Tightener, is
due to the contraction of the inner part of the orbicularis ocluli, which is called the pars
palpebralis. This will tighten the eyes and narrow the eye aperture. It is possible that the lower
eyelid is raised more than the upper eyelid is lowered. This causes a bulge beneath the eye
(Ekman, et al., 2002, p. 28).
Instead of narrowing the eye aperture it is also possible the eye aperture is dilated. This
happens when the levator palpebrea superioris pulls the upper eyelid upwards. This results in
sclera exposure, which means that the white of the eye above the iris is visible. Also the lower
46 | P a g e
3 Understanding Oculesics
eyelid raises a small amount. This is the AU5, which is called the Upper Lid Raiser (Ekman,
et al., 2002, p. 24).
As been described in chapter 3.1, there is a muscle, the zygomaticus major, that pulls the
corners of the mount out and upwards. This is the AU12, the Lip Corner Puller and will be
used in explaining the emotion happiness and contempt in chapter 3.4.5 and chapter 3.4.8.
The other AU in the lower part of the face is the AU14, the Dimpler. This one tightens the
corners of the mouth and pulls those inwards (Ekman, et al., 2002, p. 148). This Action Unit
will also be further mentioned in the emotion contempt in chapter 3.4.8.
Most people can express these AUs voluntary which give the impression of feeling an emotion.
But in some of the facial expressions, there are one or more action that most people find
difficult to preform voluntary. This means they are only visible when the facial expression is
involuntary (Tinwell, 2015, pp. 91-92). This happens for example in a truthfully smile which will
be further explained in the chapter 3.4.5
As been written before, Ekman has distinguished seven basic universal recognized emotions.
These are the emotions anger, disgust, fear, happiness, sadness, surprise and contempt. The
facial expressions of the basic emotions are the same for people for different cultures, who
spoke different languages and had different backgrounds and educations. “Our innate ability to
recognize and respond to another’s facial expression serves as a primordial survival technique”
(Tinwell, 2015, p. 73) Therefore, we are constantly reading the emotions of other people, as
well as in groups as in one-on-one conversations. If, for example, we see the emotion of
disgust on the face of someone else, we will be alerted to find what this person is disgusted
about and we will try to avoid the reason of his or hers disgust emotion. This is to protect
ourselves from threatening situations that can, for example, infect or contaminate us. Another
example is the facial expression of anger on someone’s face. The person who recognizes this
emotions on the face of someone else, can decide if he or she should make themselves ready
to fight with or flee from this angry person (Tinwell, 2015, p. 74).
Each of the seven basic emotions are part of a family. A family of emotions share the same
characteristics which distinguished one family from another. (Ekman, 1992, p. 172) Let us now
discuss the seven emotions and which AUs they are using for the facial expression around
the eyes.
47 | P a g e
3 Understanding Oculesics
3.4.2
Anger
As mentioned before, for a human it is important to see if another person is angry. It is one of
the most important facial expressions we want to recognize in another human being. This is
for our own safety and it is a part of our survival instinct. Someone can get angry when there
is an obstacle that need to be overcome to reach a certain goal. But what if, for example,
Person A is the obstacle to Person B’s goal? In this situation Person A probably wants to
recognize the anger in Person’s B facial expression. This way Person A knows what can be
expected and can make him or herself ready to fight person B or to flight from person B
(Tinwell, 2015, pp. 74-89; Lemerise & Dodge, 2008, p. 730).
Figure 13: The facial expressions of a neutral face, anger, disgust and happiness (from left to right).
(Ekman, et al., 2002, pp. 45-93)
There is not just one expression for anger. The family of anger contains over 60 expressions
of anger (Ekman, 1993, p. 386). But what all the anger expressions in the family have in
common is that the eyebrows are lowered and pulled together. This results in an intense frown
and vertical or almost vertical wrinkles between the eyebrows. The Action Unit creating this
frown is the AU4, the Brow Lowerer. In the facial expression of anger we can in some cases
see the AU5, the Upper Lid Raiser, which opens the eyes. In other cases the eyes are more
closed, caused by the AU7, the Lid Tightener. (Ekman, 1992, p. 172; Tinwell, 2015, p. 90;
Matsumoto, et al., 2008, p. 213)
Remarkably, in the research of Tinwell et al. (2011a) (see also chapter 2.5), the stimuli with
lack of animation in the upperpart of the face was not rated significant more uncanny as the
one with upperpart facial movement in the case of the facial expression of anger. This was
unexpected because Tinwell thought the delay of realizing a person is angry, and therefore a
delayed warning of possible danger, was expected to be more uncanny. Tinwell conclude that
because pressed lips are also an authentic sign of anger, this might be the reason why the
character without upperpart facial movement was not rated as uncanny as the one with
complete facial movement. The emotion was distinguished by the mouth and the perceiver did
48 | P a g e
3 Understanding Oculesics
not need to see the region around the eyes to recognize the anger (Tinwell, et al., 2011a, p.
748; Tinwell, 2015, p. 90).
3.4.3
Disgust
Just like anger, it is important to recognize the emotion of disgust in another’s facial expression
as well. Knowing someone is disgusted about something will help us in preventing being
potentially contagious or infected. It is an important warning emotion to help another person
stay clean and healthy (Tinwell, 2015, pp. 87-88). Darwin defines disgust as following:
“Something revolting, primarily in relation to the sense of taste, as actually perceived
or vividly imagined; and secondarily to anything which causes a similar feeling,
through the sense of smell, touch and even of eyesight” (Darwin, 1890/2009, p. 265)
Disgust is a reaction on something negative. But not every negative event provokes the
emotion disgust. There are nine elicitors that can make someone feel disgusted about. These
are “food, body products, animals, sexual behaviors, contact with death or corpses, violations of the
exterior envelope of the body (including gore and deformity), poor hygiene, interpersonal contamination
(contact with unsavory human beings), and certain moral offenses” (Rozin, et al., 2008, p. 757).
The facial expression of disgust can be recognized by the wrinkles which occurs above the
nose (see figure 13). This is due to the muscle that runs along both side of the nose.
Contracting this muscle will pull up the upper lip and the nostrils are dilating. This causes the
skin of the nose to compress whereby the wrinkle at the top of the nose arises (Tinwell, 2015,
p. 88) In the Facial Expression Coding System, this is the AU9. This Action Unit only appears
by this basic emotion and is therefore very distinctive for the emotion disgust (Ekman, et al.,
2002, p. 93; Matsumoto, et al., 2008, p. 213).
In the research of Tinwell et al. in (2011a) (see chapter 2.5) was concluded that the facial
expression of disgust, with lack of movement in the upperpart of the face, was perceived as
uncomfortable by the participants. Tinwell thinks this is because it is unclear to the viewer what
the CG-character is trying to communicate when the wrinkles above the nose are not visible.
Because of this uncertainty of what was meant by the CG-character, this resulted in an
uncanny feeling for the participants. (Tinwell, et al., 2011a, p. 747; Tinwell, 2015, p. 88) Tinwell
suggest that the skin resolution of a CG-character may be part of the problem if the audience
is feeling uneasy with a character showing the facial expression of disgust. If the wrinkle up
the nose cannot be rendered realistic enough because of a low resolution or a limited render
time, the facial expression will be hard to recognize by the viewers. She concludes:
49 | P a g e
3 Understanding Oculesics
“Without a clear representation of this most distinctive nose wrinkle movement
involved in disgust, the character’s expression appeared abnormal and bizarre, thus
exaggerating the uncanny.” (Tinwell, 2015, p. 88)
3.4.4
Happiness
A smile is a facial expression of happiness, which can also be recalled as joy (see figure 13).
A smile is easy to recognize but can also be confusing. People can easily be misled by a smile.
This is because a smiling person is not always feeling happy when he or she is smiling. When
a person is smiling this can be a real or a false smile (Tinwell, 2015, pp. 91-92). Real smiles
are those smiles that are shown when a person truly feels joy. There are many reason why a
person has a real smile. This can be for example because the person is relieved from tension
or pain, is amused or enjoys the company of another person. (Ekman & Friesen, 1982, p. 242)
In the book, Harry Potter and the Goblet of Fire, the character Igor Karkaroff is described by
J.K. Rowling as “his smile did not extend to his eyes” (Rowling, 2000, p. 247). With this sentence,
Rowling expresses a false smile in the character. A person can deliberately show a false smile
to convince another person he or she is happy. With a false smile, people try to cover up their
negative emotions. This seems to be culture dependent. For example, the facial expression
for the emotions fear and disgust are the same when Japanese and Americans are alone, but
in social situations the Japanese tried more than the Americans to cover up these feelings by
smiling. (Ekman & Friesen, 1982, p. 239)
The most remarkable part of the face to recognize if a person is smiling, is the mouth of this
person. The muscles (zygomaticus major muscles) from the corners of the mouth to the
cheekbones are contracted. This will pull the corners of the mouth upwards and to the side of
the face. (Tinwell, 2015, p. 91) It is the AU12 by Ekman and Friesen’s Facial Action Coding
System. This Action Unit can easily be controlled by a person. When a person is voluntarily
contracting this muscle and thereby give the impression that a person is happy, the person is
showing a false smile with his or hers facial expression. (Tinwell, 2015, pp. 91-92) But a real
facial expression of happiness can be distinguish by the upperpart facial movement and the
movement in the cheek of the person. This cheek movement is known as the AU6 of the FACS
and is called the Cheek Raiser, which raises the cheeks of a person. The muscle responsible
for this Cheek Raiser is the orbicularis oculi muscle. The AU6 will also narrow the eye aperture,
gathering the skin around the eye whereby crow’s feet wrinkles appear on the outside of the
eye aperture. The contraction of this muscle happens involuntarily, which means, the person
cannot control this movement. (Ekman & Friesen, 1982, pp. 242-246; Tinwell, 2015, p. 92;
Ekman, et al., 1990, p. 342) These real smiles are called Duchenne smiles because it was the
50 | P a g e
3 Understanding Oculesics
researcher Guillaume Duchenne who first wrote about this differences between real and false
smiles. This is how Duchenne in (1862/1990) describes them:
“The emotion of frank joy is expressed on the face by the combined contraction of
the zygomaticus major muscle and the orbicularis oculi. The first obeys the will but
the second is only put in play by the sweet emotions of the soul; the […] fake joy,
the deceitful laugh, cannot provoke the contraction of this latter muscle. [….] The
muscle [around the eye] does not obey the will; it is only brought into play by true
feeling, by an agreeable emotion. Its inertia, in smiling, unmasks a false friend”
(Duchenne de Boulogne, 1862/1990, pp. 126-128)
Compared to the other basic facial expressions, the participants in the research of Tinwell et
al. (2011a) (see chapter 2.5) found the facial expression of happiness the least uncanny when
there was no movement in the upperpart of the CG-character’s face. Remarkable was the fact
that the CG-character with movement in the upperpart of the face was rated the least familiar
and human-like compared with the other emotions. This was unexpected because happiness
is a positive emotion and therefore it is not needed as warning for survival. Tinwell believes
this outcome is because the participant may have thought they were, despite the animation in
the upperpart of the face, confronted with a false smiling person (Tinwell, 2015, pp. 91-92;
Tinwell, et al., 2011a, p. 747).
3.4.5
Fear
The next facial expression that helps us to survive in the world is that of fear (see figure 14).
This, as well as anger and disgust, warns us of danger if we notice it on the facial expressions
of others. If we see fear on someone’s face, we are alerted that we might be in danger
ourselves. Therefore, we will immediately search for the cause of this fear so we have the
longest possible time to protect ourselves against or escape the threat (Tinwell, 2015, pp. 8183).
The question is, how can we distinguish fear from the other basic emotions? The combination
of AU1 + AU2 + AU4 is a sign that the person who is showing these AUs is experiencing fear.
As been described in chapter 3.4.1, the AU1 + AU2 are raisers of the eyebrows and AU4 pulls
the eyebrows downwards. This might seem contradictory, however the result of this
combination are raised eyebrows but not as high as in the case of only AU1 + AU2 together.
The eyebrows are at the same time pull together but not so much in AU4 only. Because of this
pulling downwards and raising at the same time, the shape of the eyebrow will be more
flattened, whereby generally the inner and outer corners of the eyebrow are at the same level.
The eyebrows of some people show a small downward curve between the inner corner and
51 | P a g e
3 Understanding Oculesics
the middle of the eyebrow. This is due to the downwards pull of AU4 (Ekman, et al., 2002, p.
56). Simultaneous with this eyebrow combination of AU1 + AU2 + AU4, the AU5 appears.
Hereby the eye aperture dilates due to the upwards pulling of the upper eyelid. This results in
a wider visual field which allows the person to see more and obtain more information what
might be needed to respond to the fearful situation (Ekman, 1992, p. 178). Fear and surprise
are hard to distinguish from one another (Tinwell, 2015, p. 81). This will be further discussed
in chapter 3.4.6 Surprise.
Tinwell et al. (2011a) (see chapter 2.5) revealed that the emotion fear on a CG-characters
face with lack of animation in the upperpart, was rated the most uncanny by the participants.
Tinwell suggest that the reason for this is because the expression fear only requires small
movements in the face. Then, when also the movement of the upperpart of the face is missing,
this could remind the perceiver of a corps. This sudden reminder of death is what might trigger
the uncanny feeling (Tinwell, et al., 2011a, pp. 746-747; Tinwell, 2015, p. 83). Also the
uncertainty if another person is trying to communicate the emotion fear with us, might be
something what we find uncanny. In this case we are not sure how to react and if we should
fear something, what is unknown to us, as well. (Tinwell, 2015, p. 84)
Figure 14: The facial expressions of fear (left) and surprise (right)
(Ekman, et al., 2002, pp. 60-66).
3.4.6
Surprise
The facial expression of surprise is often mistaken by the facial expression that is shown with
the emotion fear. This is because fear and surprise have similar facial movements. The eye
aperture is dilated and the eyebrows are raised. In everyday life it is easier to distinguish the
facial expression of surprise from fear. This is due to the events that are happening at the time
someone shows the emotions (Tinwell, 2015, p. 85; Matsumoto, et al., 2008, p. 213). The
emotion surprise is experienced when a discovery is made or when there is a violation of
expectancy (Matsumoto, et al., 2008, pp. 315-316).
52 | P a g e
3 Understanding Oculesics
Now we will review the AUs visible in the emotion surprise (see figure 14). This is a
combination of the Inner Brow Raiser, the AU1 and the Outer Brow Raiser, the AU2. The
difference between fear and surprise is the absence of the AU4 in surprise (Tinwell, et al.,
2011a, p. 743; Matsumoto, et al., 2008, p. 213). As well as in fear, in the facial expression of
surprise, the AU5 is visible. This raises the eyelids which dilates the eye aperture and normally
the sclera, the white of the eye, above the iris is visible. Due to the raising of the eyebrows
and the eyelids, the eye apertures are dilated, which results in an extended field of vision. This
allows the person to gather more visual information (Ekman, 1992, p. 178).
The quantity of uncanniness did increase significant for the CG-character with and without
facial movement compared with the footage of the actor when the facial expression of surprise
was shown in the research of Tinwell et al. in (2011a) (see chapter 2.5). This was unexpected
according to Tinwell because the facial expression of surprise is not very essential for
someone’s survival in the world. Therefore it has no accurate need to recognize this emotion
in another person and to be unsure about the meaning of the facial expression. Tinwell
believes that the uncanny feeling is increased when a facial expression, that helps distinguish
survive situations, is not easily recognized (Tinwell, et al., 2011a, p. 742). But because the
facial expression of fear and surprise are often hard to distinguish from one another when the
situation is not clear to the perceiver, this might be the reason why the CG-character was so
uncanny to the participant, when the upperpart of the face had no movement. They might have
thought it was the emotion fear that they saw in the CG-character (Tinwell, 2015, p. 85).
3.4.7
Sadness
Sadness is an emotion that can last between a few seconds till several hours and is associated
with the realization of a permanent loss. Sadness has the function that we can take time to
consider the situation and think about our goals and plans. Showing sadness as a facial
expression will tell other people how a person is feeling and therefore another person is able
to response to the sad person with sympathy and support. It helps the people in their social
behavior. (Bonanno, et al., 2008, pp. 798-799)
As the other seven emotions, sadness is universal. Every person in the world who is showing
a sad facial expression uses the same muscles and shows the same visible movements in the
face (see figure 15). (Bonanno, et al., 2008, p. 797) The facial expression of sadness can be
recognized in the upperpart of the face by the slanted eyebrows. Hereby the inner part of the
eyebrows are raised higher than the outer parts forming an oblique line. With some people the
eyebrows are a bit curved, showing an S-shape respectively mirrored S-shape. The Action
Units which are responsible for the eyebrows are the AU1 and AU4. This creates a frown and
53 | P a g e
3 Understanding Oculesics
wrinkles on the centre of the forehead. The AU1 pulls the eyebrows up and AU4 pulls them
together. The muscle corrugator supercilii pulls the middle of the eyebrow downwards with the
result of the eyebrows forming the oblique line (Ekman, et al., 2002, p. 51; Tinwell, 2015, p.
87).
The research of Tinwell et al. in (2011a) (see chapter 2.5) revealed that the CG-characters
who showed sadness as a facial expression where rated the most familiar and human-like in
both the CG-character with and without the upperpart facial movement. Still, the CG-character
without movement in the upperpart of the face was much more uncanny to the participants.
The movement of the mouth may be significant enough to recognize the expression as
sadness. This is the conclusion Tinwell has for the fact that the participants rated the CGcharacters as the least strange. She also thinks that because the emotion is recognized lightly
on the mouth, the participant could easily felt sympathy for the characters (Tinwell, et al.,
2011a, pp. 746-747; Tinwell, 2015, pp. 86-87).
Figure 15: The facial expressions of the emotions sadness (left) and contempt (right)
(Ekman, et al., 2002, p. 51)
3.4.8
Contempt
The first list of basic emotions that Ekman published did not contain the emotion contempt. In
1999, Ekman (1999) published a new list of basic emotions on which contempt was added.
This emotion is the aversion of something that is conceived as having a lower status. When
this emotion is provoked by another human being, the person experiencing this emotion
judges the other person as inferior or unworthy human being (Fischer, 2011, p. 77).
The facial expression of contempt is mainly visible in the lower part of the face (see figure 15).
These are the AU12 and AU14 and these are unilateral. This means that only one corner of
the mouth is pulled upwards and is pressed together. If AU12 is strong, the skin underneath
the eye bags and the eye aperture can be narrowed. But in the case of contempt, this
54 | P a g e
3 Understanding Oculesics
contraction is not so strong, resulting in a minimum of the movement around the eyes
(Matsumoto, et al., 2008, p. 213).
In the research of Tinwell et al. in (2011a) (see chapter 2.5), the emotion contempt in CGcharacters is not researched. This might be because the facial expression of contempt does
not have any movement in the upperpart of the face, the eyes stay neutral.
The next table is an overview of all the Action Units described in this chapter. Here is displayed
by which muscles the Action Units are created and with what emotion they occur.
Action
Unit
Name
Muscles
In the facial
expression of
1
Inner Brow
Raiser
frontalis (Inner part)
Fear, Surprise, Sadness
2
Outer Brow
Raiser
frontalis (Outer part)
Fear, Surprise
4
Brow Lowerer
corrugator supercilii,
depressor supercilii and
procerus
Anger, Fear, Sadness
5
Upper Lid
Raiser
levator palpebrea
superioris
Fear, Surprise,
sometimes Anger
6
Cheek Raiser
and Lid
Compressor
orbicularis ocluli, pars
orbitalis
Happiness
7
Lid Tightener
orbicularis ocluli, pars
palpebralis
sometimes anger
9
Nose Wrinkler
levator labii superioris
alaeque nasi
Disgust
12
Lip Corner
Puller
zygomaticus major
Happiness,
Contempt (unilateral)
14
Dimpler
Buccinator
Contempt (unilateral)
Table 1:
An overview of the Action Units, involved muscles and in which facial expressions they appear
(Ekman, et al., 2002) (Author’s table).
55 | P a g e
3 Understanding Oculesics
3.5
Pupil Dilation
As been described in chapter 3.1, the pupil is the hole in the iris of the eye. It is the part where
the light travels through before it reaches the retina surface. The main function of the size, or
in other words the dilation of the pupil, is to regulate the amount of light that can enter the eye
and reach the retina surface. If there is a lot of light entering the eye, the iris will make the
pupil smaller by enlarging itself. This will protect the retina for the exposure of too much light.
In the case where there is hardly any or no light, the iris will shrink itself whereby the pupil
dilates and therefore more light is able to reaches the retina. This way the possibility of visual
information is increased (Knapp & Hall, 1992, p. 311; Faller & Schünke, 2012, pp. 625-626).
But the pupil of the eye does not only changes its size because of the light conditions it is
confronted with. The dilation of a pupil can be due to emotional states as well (Knapp & Hall,
1992, p. 311). Eckhard Hess was an American psychologist and he studied the reaction of
pupils. He states “When we say that someone’s eyes are soft, hard, beady, cold or warm, we are in
most instances referring only to a certain aspect of that person’s eyes: the size of the pupils” (Hess,
1975, p. 110). Hess thought that the dilation of pupils can be related with attitudes. A positive
attitude will shrink the iris and therefore the pupil will dilate, were with a negative attitude the
opposite will happen (Knapp & Hall, 1992, p. 311). Hess concluded: “The changes in emotions
and mental activity revealed by changes in pupil size are clearly associated with changes in attitude”
(Hess, 1975, p. 110). An example of change in attitude is when a person sees something he
or she is interested in or excited about. These are positive attitudes and hereby the pupils
dilates. This source of interest or excitement can as well be another person (Knapp & Hall,
1992, pp. 311-312). According to the research of Hess, et al., the pupils of heterosexual males
dilated when they were looking at pictures of females. This, in contrast to homosexual men,
whereby the pupils dilated when they were looking at pictures of other men (Hess, et al., 1965).
Not only being excited or interesting about something will dilate the pupils. Dilation may also
happen when a person tries to solve an intellectual task. The harder a task is the further the
pupil will dilate. It will stay dilated as long as the person is dealing with the problem until the
answer is reached or the person quits solving the problem (Fong, 2012). Chapman et al.
(1999) discovered that in the case a person is experiencing pain, the pupils of this person will
dilate as well. The pupil is fully dilated after 1.25 seconds. Other stimuli that influence the
dilation of the pupil are, for example, loud noises or drugs (Knapp & Hall, 1992, p. 313;
Chapman, et al., 1999, p. 44).
In his paper, Hess (1975) describes that the size of the pupil plays a part in nonverbal
communication. This happens on an unconscious level. Pupil dilation can be provoked by the
56 | P a g e
3 Understanding Oculesics
dilated pupils of someone else. The research of Hess concluded that when a man, who
identifies as a heterosexual male, sees a pictures of a woman with dilated pupils, his pupils
will also dilate. This does not happen when he sees the same picture of the woman with
manipulated small pupils. Remarkably, woman prefer pictures of other woman with small
pupils over picture of woman with large pupils (Hess, 1975, pp. 110-112). If person A is
sexually attacked to person B, the pupils of person A will dilate. This can be found attractive
by person B, whereby the pupils of person B will dilate as well. This explains why romantic
scenes are often preferred with dimmed or candle light; it dilates the pupils which is more
attractive. (Knapp & Hall, 1992, pp. 313-314).
3.6
Blinking Rate
The pars palpebralis of the orbicularis ocluli, the muscle orbiting the eye, is responsible for the
blinking of the eyelids. The main purpose of normal blinking is to lubricated, keep the eye
clean and to protect it from potential hazard which can damaged the eye. A healthy adult will
bink his eyelids about six to ten times every minute. When a person is very concentrated on a
task or focused on an object in his or her environment, this amount could decrease. In the
case a person experience different states of anxiety, it might be this person starts to blink
more. This is because this person is unconsciously try to “cut of reality” (Knapp & Hall, 1992, p.
296).
3.7
Saccades and Fixations
Since 1980 it became possible to track the eyes of a person when they were doing activities
as walking and driving (Land, 2006, p. 296). The same pattern of eye movement can be found
through the whole animal world. Also in humans our eye movements follow the saccade and
fixate strategy. When we overview a scene and we perceive all the objects within this scene,
we might think that our eyes are slowly panning around, allowing us to get all the information.
We are not aware of our body, head and eye movements and therefore we have this illusion
of the eyes that pan around. In fact, our eyes are alternating between saccades and the
moment where our eyes fixate on an object. “Our eyes do not wander at all: They jump around!”
(Land, 1999, p. 341) A saccade is a brief fast eye movement between the moments the gaze
is centred on a point of interest. Land describes them as following:
“Saccades are the fast movements that redirect the eye to a new part of the
surroundings, and fixations are the intervals between saccades in which gaze is
held almost stationary.” (Land, 2006, p. 297)
57 | P a g e
3 Understanding Oculesics
Saccades can be as frequent as up to three times per second and the rotational movement of
the eye can reach speeds of 700 degrees per second (Land, 1999, p. 341; Land, 2006, p.
297). Then after a saccade comes a fixation. In this period our view is almost stationary and
last circa 300 milliseconds or longer. During a fixation we gather information, then, during a
saccade to switch to another point of interest, we are effectively blind. So why do we need
these fast movements and these longer fixations? There are two reasons for this. The first
reason is because the region of most acute vision the fovea, is only a very small region. The
fovea is a small depression in the centre of the retina which provides the vision that we see
the sharpest. The angular diameter of this area is between 0.3° and 2° and it is only about a
1/4000th of the whole retinal surface. The further away from the foveal centre light is capture
on the retina surface, the worse the resolution of the image we can see becomes. To see
something in detail we need to focus on this by moving the fovea to the centre of the target.
Hereby the light of the object we want to see falls directly on the fovea. This needs to happen
as fast as possible because due to the fast movement of the saccades we effectively cannot
see anything. This is because of the combination of motion blur and that our brain active
suppresses the image (Land, 2006, p. 297). The second reason for the saccade and fixation
strategy is that we gather the visual information we want from a point of interest during the
fixation. This is not a fast process. It takes about 20 milliseconds for a receptor cells to respond
fully to a new lighting situation (Land, 2006, p. 297).
3.8
Gaze in Object Related Actions
When we are doing a task, like driving, sports or making a sandwich, our gaze is mainly fixated
on areas that are relevant to complete the task, the rest of the areas are mostly ignored. Our
gaze is tightly timed to see the right thing in the right moment. This is called the just-in-time
strategy whereby the necessary visual information is acquired by a fraction of a second before
it is required in the task. (Hayhoe & Ballard, 2005, p. 189; Land & Hayhoe, 2001, p. 3560).
The fixations in a task can be classified in four different categories. These are locating,
directing, guiding and checking. Land and Hayhoe (2001) explain this with the example of
making a cup of tea. It starts with locating the objects that might be used for tea making. Also
during the task there are moments were the gaze is fixated on objects that are needed in a
later moment for making the cup of tea. When we make hand movements to grab an object,
the gaze will be focused on the object we want to reach. The visual information helps in
directing where the hand should go. For example, when we reach out to the kettle, the gaze
will not be on the hand but on the kettle. Guiding is more complex than directing. In guiding
there is more than one object involved. If we want to put the lid on the kettle, the gaze will
58 | P a g e
3 Understanding Oculesics
alternate between the objects by a number of fixations. With checking the gaze will be
repeatedly at, or for a longer time fixated on, a curtain location. For example, to see if there is
enough water in the kettle or if the water is already boiling (Land & Hayhoe, 2001, p. 3563).
“In both tea-making and sandwich-making […] we were impressed by the way gaze
moved from one task-relevant object to the next, ignoring all other objects that were
not involved in the activity.” (Land, 2006, p. 318)
When we are looking from on direction to a new direction, not only the eyes turn, but also the
head and trunk rotate to get the gaze on the right target. Land explains this movement as
following:
“What combination of eye head and trunk movements occurs depends in part on
the size of the gaze rotation required. In the horizontal plane the eyes can rotate up
to a maximum of about ± 50°, although head movements typically accompany them
for gaze rotations larger than 10°. The neck can turn the head through about ± 90°.
Thus, for gaze rotations greater than 140° the trunk has to become involved, and
with freely moving subjects, especially during locomotion, trunk turns can occur with
gaze rotations of any amplitude. The eyes rotate faster than the head, which in turn
rotates faster than the body. This means that there are potential coordination
problems: for example if the eyes reach their target before the head rotation is
complete they must counter-rotate as the head catches up, and mechanisms are
required to ensure that this can occur without gaze direction being perturbed.
Similar considerations apply when part of the rotation is carried out by trunk
rotation.” (Land, 2006, p. 320)
3.9
Gaze in Human Interaction
“We associate various eye movements with a wide range of human expressions:
downward glances are associated with modesty; wide eyes may be associated with
frankness, wonder, naiveté, or terror; raised upper eyelids along with contraction of
the orbicularis may mean displeasure; generally immobile facial muscles with rather
constant stare are frequently associated with coldness; eyes rolled upward may be
associated with fatigue or suggestion that another’s behavior is a bit weird.” (Knapp
& Hall, 1992, p. 295)
Where we look is established by our society. For example, there are eye-related norms of how
to behave in public. Like, it is not accepted to stare at a stranger or to look at a woman’s
breast.
59 | P a g e
3 Understanding Oculesics
Human interaction starts with making contact to another person. We can talk to someone but
if this person does not know someone is trying to communicate with him or her it is, of course,
difficult to respond. If we want to communicate with someone else, the first thing we will try in
order to get their attention is to make eye contact with this other person. We want to signal the
other person that the communication channel is open (Knapp & Hall, 1992, pp. 298-299).
The situation where two people are looking at each other, for example when they talk, is
referred to as mutual gaze. It is the eye contact between two people. This does not mean that
they are directly looking each other in the eyes. Looking at each other’s face is part of the
mutual gaze. Mostly they will look at the area around the eyes. (Knapp & Hall, 1992, p. 297;
Richmond, et al., 1991, p. 85). As been described in chapter 3.7, when we are looking the
gaze moves around. When we are looking each other in the eyes, this is actually a sequence
of saccades and fixations. The gaze jumps around from different parts around the eyes of the
other persons. In a research with an eye tracking camera, they found eighteen different points
a person was looking at when this person was listening to somebody else and was trying to
keep a steady gaze. If someone is constantly focussing on one point in the face, this is
perceived as staring, which is uncomfortable (Knapp & Hall, 1992, p. 297).
Gazing at the other person in a normal interaction lasts between five to fifteen seconds. When
we listen to someone, we look more at the other person then when we speak. By glancing at
the facial expressions of the speaker, the listener tries figure out if the nonverbal
communication fits the spoken words. The speaker on the other hand, looks at the listener to
see if he or she is still interested in what the speaker is saying (Knapp & Hall, 1992, pp. 298299; Lewis, 2012, p. 83). Direct eye contact between two people will last for about three
seconds. After this, normally one or both of the persons feels the urge to break the direct eye
contact. (Lewis, 2012, p. 70).
The gazing behavior can varying in different situations. With our gaze we can send signals to
other persons. For example, if we are in a conflict with another person, we will have a longer
direct eye contact. This is because breaking the eye contact in such a situation signals being
defeated or feeling fear (Lewis, 2012, p. 75). A person who is not looking at the other person
when he or she is talking or listening is perceived as uninterested, or being secretive. (Lewis,
2012, pp. 75-76)
When a person is thinking and talking, the movement of the eyes show what type of
information is thought of and how this information is being processed at that moment. This
next overview of eye movements are seen at right-handed and most left-handed people, but
60 | P a g e
3 Understanding Oculesics
for some left-handed people, these movements are reversed (Lewis, 2012, pp. 85-87; Knapp
& Hall, 1992, pp. 300-301).
Looking up and to the right: The person visualises an image or event he or she has never
seen. This eye movement can indicate that a person who is supposed to tell the truth, is in
fact lying.
Looking up and to the left: This eye movement is made when a person is recalling an event
that he or she has seen.
Looking level and to the right: The difference between looking right and upwards and looking
right and level, is in the last case, the person is imaging a sound and not a visual event or
image.
Looking level and to the left: Here, a person recalls a sound that is stored in his or her memory.
Looking downward and to the right: The person access emotions or sensations of the body.
Looking downward and to the left: In this case the person has an internal conversation with
his or her self.
Unfocused eyes: When the eyes are unfocused, the person is visualizing an event. This can
be real or imagined. The direction of the unfocused eyes can be in all the position that has
been described here above.
Closed eyes: When someone tries to recall a certain smell or taste, he or she will usually close
the eyes. But underneath the eyelids the eyes can still be moving in any of the above described
ways.
Lewis (2012, pp. 70-71) comes with the next statements about gazing:

“People generally look away as they begin to speak, then turn their gaze
back to their listeners.

While speaking, people alternate between gazing at and gazing away.

There is more direct gaze when people like each other and cooperate.

People make less eye contact when they dislike each other or disagree.

An unwavering gaze can signal dominance and threat.
61 | P a g e
3 Understanding Oculesics

Gaze avoidance can be a cue for submission.

A direct gaze when accompanied by a smile with body thrust forward is a
sign of trust between two individuals.

Gaze direction clearly shows others where our attention lies.”
When we do not wish to communicate with someone, we will avoid to make eye contact with
them; this is the easiest way to avoid interaction. This is called gaze aversion (Richmond, et
al., 1991, p. 85). For example, a student in class, who does not know the answer to a teachers
question, will avoid to make eye contact with the teacher, hoping he does not need to answer.
He is nonverbally saying that the communication channel is closed. When we are engaging a
stranger we will only look at the person to register him or her. In the words of Knapp and Hall:
“When confronting unknown others, we typically recognize them with a brief glance, but this initial
glance is followed by the avoidance of gaze unless further contact is desired.” (Knapp & Hall, 1992,
p. 299) Gaze omission is not the same as gaze aversion. The difference is, with gaze aversion
one person intentionally avoids they eye contact because he or she does not wish to
communicate where with gaze omission there is unintentionally no eye contact. This may be
because one of the persons does not realize someone is trying to communicate with them
(Richmond, et al., 1991, pp. 85-86).
3.10
Summary
Oculesics is “The study of the communicative role the eyes have in nonverbal communication”
(Chandler & Munday, 2011). More than body movements, our facial expressions can
communicate the emotions we are experiencing. There are seven basic facial expressions
that are universally recognized. These are anger, disgust, fear, happiness, sadness, surprise
and contempt. The facial expressions can be distinguished by different, in most cases
combinations of, Action Units of the Facial Action Coding System, created by Ekman and
Friesen (Ekman, et al., 2002). In this chapter we have discussed the seven basic facial
expressions and how these can be distinguished by the Action Units.
The dilation of pupils are due to the light that is entering the eyes and to the attitude of the
person. Pupils dilates when a person is excited or interested in the object or person he or she
is looking at, during mentally difficult tasks and when the person is experiencing pain. A person
normally blinks about six to ten times a minute.
The eyes jump around due to the saccade and fixation strategy. In object related tasks the
eyes have a locating, directing, guiding and checking function to complete the task. By the
62 | P a g e
3 Understanding Oculesics
just-in-time strategy, the gaze is on an object just before the information of this object is
required for the next step in the task.
With eye contact we signal another person that we want to communicate. The easiest way to
not communicate or to stop the communication with a person is to avoid and break the eye
contact.
In human interaction, mutual gaze is the gazing behavior between two people. The length of
the gaze in the face of the other last between five to fifteen seconds. Hereby the gaze jumps
around over the area of, and around the eyes. When there is direct eye contact, one or both
persons will feel the urge to break the eye contact after three seconds. Speakers tend to look
shorter at the listeners than the other way around. The position of the eyes in interaction are
related to the information that is being processed and if this information is real or imagined.
In the next chapter we will analyse two CG-animation films which both have received negative
reviews about the eyes of the characters. The analysis is used to see how oculesics is used
in these films and what the differences are between them and what the part of the eyes is in
being associated with the uncanny valley.
63 | P a g e
4 Case Studies
4
Case Studies
In this chapter we will analyse different scenes from The Polar Express and The adventures
of Tintin: the Secret of the Unicorn (Abbreviated to The Adventures of Tintin). Both these films
have received negative reviews about the eyes of the character. The next quote is from a
review on the website CNN.com about The Polar Express: “To quote an old cliche, the eyes are
the windows to the soul -- so these characters look soul dead.” (Clinton, 2004). Also a review of The
Adventures of Tintin refers to the eyes: “Tintin looks simultaneously too-human and not human at
all, his face weirdly fetal, his eyes glassy and vacant instead of bursting with animated life.” (Buchanan,
2011).
The decision to analyse these two films are based on several reasons. To start off, they are
both full CG-animation feature films with characters that represent real human beings. They
are not cartoonish but are created with normal human proportions. Thereby, both these films
used motion capture for the animation of the characters. But, of course, there are more films
who have these two reasons. As been written before, the goal of this analyse is to see what
the influence of the movement of the eyes is on the possibility that a character ends up in the
uncanny valley and how far the animation industry has come in overcoming the uncanny
valley.
To analyse the progress that has been made in the industry, it makes sense to discuss one of
the first feature films that fell into the uncanny valley and one of the last. The last feature film
is The Adventures of Tintin. This film will be used in the analysis as last benchmark of where
the development of facial expressions in full CG-character stands.
The first two films who had uncanny valley related criticism, were Final Fantasy: The Spirits
Within and The Polar Express. Because Final Fantasy was the first full CG-animation feature
film with near-humanlike characters, the phenomenon of the uncanny valley was not that well
known. Also the motion capture technique was used for the body movements and partly for
the facial expressions. This means that the facial expressions were also animated by hand
(Park, 2001). Therefore in this analysis we will use The Polar Express, because facial motion
capture was used here and they knew what the uncanny valley phenomenon could do to the
characters.
This chapter will start off with the introductions and the plots of the two movies, to give an
impression about the production and where the analysed scenes fit into the story. This is
followed by the analysis of four of the universal recognized facial expressions. The reason
why not all the seven basic facial expressions are analysed will be declared later in this
64 | P a g e
4 Case Studies
chapter. Next, we will look at the eye movements of the characters. Here we look at the pattern
of saccades and fixations, the blinking rate and the eye behavior in mutual gaze. This chapter
will finish with the discussion of the startle reflex in the character with the main focus on the
character of Tintin. In the analysis, scenes from both films will be discussed and compared to
one another.
4.1
The Polar Express
The Polar Express is a full CG animation film which has been released in November 2004. It
was written, directed and produced by Robert Zemeckis. The film is based on the book, The
Polar Express of Chris Van Allsburg and Sony Pictures Imageworks was the responsible
animation studio. The character animation is created with the motion capture technique. Tom
Hanks was one of the main actors who was preforming as motion capture actor. His
movements were used for six of the characters including the protagonist Boy, the Conductor
and Santa Claus (see figure 9 in chapter 2) (Bredow, et al., 2005).
The motion capturing of the eyes and region around the eyes gave some problems in the
production of the movie. Brewdow, et al. explain:
“Attempts were made to capture the eyelids and the movement of muscles
surrounding the eye. Markers were placed on the eyelids but when the eyes
opened, the lids tucked up under the fat pad below the brow and the markers
disappeared. The tracking system could not solve for the missing markers and the
end result was noisy and unreliable data in the critical region of the eyes.” (Bredow,
et al., 2005, p. 34)
They decided to base the animations between the cheeks and eyebrows on the movements
of the eyeball. About the eye movements they declare:
“Every effort was made to emulate the natural behavior of eyes in animation. The
eyes are constantly in motion. This is because the retina only has a small region of
high acuity about the size of a thumbnail. To explore an area, the eyes are
constantly moving over a scene in short intermittent bursts called saccades. Even
the brief moments of fixation include a certain level of noise that had to be animated
in each of the close-up shots.” (Bredow, et al., 2005, p. 34)
Plot: The Polar Express
On Christmas Eve, the narrator and protagonist, a young Boy of whom the name is not
revealed, doubts his believe in Santa Claus. While lying in his bed, he hears a train in front of
his house. Curious, he runs outside and sees the Polar Express Train. The Conductor tells
65 | P a g e
4 Case Studies
him the train is going to the North Pole and he asks the Boy to come along on this journey.
The Boy hesitates shortly but decides to come along. On the train are more children, one of
them the protagonist Girl. The next boy that enters the train is a little boy named Billy, who
never in his life had a visit from Santa Claus. The Girl loses her ticket and is taken to the roof
of the train by the Conductor. The Boy finds the lost ticket and goes after them. He finds the
Girl alone in the locomotive of the train. The two drivers of the train are at this moment on the
very front of the train to replace the head light. After they see something on the tracks they
yell to the Boy and Girl to stop the train. The Boy pulls the brakes and the train stops just in
time. A group of reindeers is blocking the rails. After the track is cleared, the train goes on.
Then a locking pin comes loose, which makes it impossible for the train to slow down. At this
moment the Conductor, the Boy and the Girl are at the very front of the train. Driving through
the hills, the train changes in a rollercoaster and after a terrifying ride the train lands on a
frozen lake. The train tracks are cover by the ice which causes the train to slide.
Luckily they reach the tracks on the other side of the lake. Then they finally reach the North
Pole. All the children get off the train except for the Boy, the Girl and Billy. The wagon they
are in is disconnected and by another rollercoaster ride they end up in the working area of
Santa Claus’ elves. Here Billy finds his Christmas present. After being reunited with the other
children, they meet Santa Claus, who chooses the child that gets the first Christmas present
of the year. In this moment a bell comes lose from Santa Claus’ sledge and the Boy caches
this bell but he cannot hear the sound of it. After saying he believes in Santa Claus he can
hear the bell ring. Then the Boy is chosen by Santa Claus to get the first present and he
chooses the bell as his present. Santa Claus leaves with his sledge and the children are going
home with the train. Here the Boy notice that he lost his bell. After being home and back in his
bed he is woken up by his little sister to open the Christmas presents. In a small present he
finds his lost bell. The narrator explains that though his parents and later his friends and sister
could not hear the bell anymore, he always could.
4.2
The Adventures of Tintin
The movie The Adventures of Tintin: The Secret of the Unicorn (2011) is directed by Steven
Spielberg and produced by Peter Jackson. The film is based on the comics of The Adventures
of Tintin by the Belgian Hergé. As The Polar Express, this film is also created with the motion
capture technique. The actors for the motion capture are Jamie Bell as Tintin, Andy Serkis as
Captain Haddock and Daniel Craig as Ivan Ivanovitch Sakharine. The realization of the film
has been done by the visual effects company WETA Digital. The process for the motion
capturing of the face is known as image-based facial performance capture. The actors wear a
66 | P a g e
4 Case Studies
helmet with a mounted camera that is pointed at the face. This allows to even record the
smallest facial movement (see figure 16) (Paramount Pictures & Columbia Pictures, 2011, p.
12).
Figure 16: With the camera mounted on a helmet, the facial expression of
actor Jamie Bell (Tintin) are capture with the help of the trackers
on his face.
Available at:
(http://cdn.moviestillsdb.com/sm/00d97618c9745b074a6f488b0
9b1a369/the-adventures-of-tintin-the-secret-of-the-unicorn.jpg)
Plot: The Adventures of Tintin
Tintin is a journalist who buys a model of the Unicorn, a triple-masted sailing ship. He is directly
approached by Ivan Ivanovitch Sakharine who wants to buy the model from him. But Tintin
refuses to sell the ship and takes it home. In a little accident involving his dog Snowy and a
cat, the mast of the ship breaks off. A small paper scroll in a metal tube falls out of the mast
but this is first unnoticed by Tintin. In order to get more information about the Unicorn, he goes
to the maritime library to do some research. Here he read that the secret of the Unicorn can
only be solved by a true Haddock. After he comes home he discovers that somebody has
stolen the model. Tintin suspects that Sakharine has stolen the ship and therefore he goes to
his house. Here he discovers that there is a second model of the Unicorn.
When he comes home again, he sees that his whole apartment has been turned upside down
like somebody was looking for something. With the help of Snowy, he finds the small tube and
discovers a piece of paper in it.
The next day, Tintin is being kidnapped by the men of Sakharine and is taken on a ship. There
he gets locked inside a room but managed to escape and meet Captain Haddock, the original
and also captured captain of the ship. Together they managed to escape the ship on a small
lifeboat. During their escape, Tintin discovers that there is a third model of the Unicorn and
67 | P a g e
4 Case Studies
that this is what Sakharine is after. The three models all contain a part of the paper clue to find
the treasure of the Unicorn.
After their escape in the lifeboat, Tintin tries to find out what Captain Haddock knows about
the Unicorn. But due to his drinking problem he cannot remember anything. Then, they are
attacked by the men of Sakharine who are flying in a water plane. Tintin managed to shoot
down the plane, take it over and fly to the shore, where they end up in the desert. Here it is
that Captain Haddock remembers the story of his ancestor, the Captain of the Unicorn. He
tells Tintin that the ship was attacked by the pirate Red Rackham and his crew. Captain
Haddock realizes that Sakharine is the ancestor of Rackham.
Together they travel to the place where the third model is. Sakharine is here as well and he
manage to get all the three clues. But after a long chase, Tintin manage to get the three pieces
of paper and Captain Haddock defeats Sakharine. When they put the pieces together, the
coordinates of the treasure are revealed. These lead to the house of the ancestor of Captain
Haddock.
In the house of his ancestor, Captain Haddock manage to open a globe where part of the
treasure is hidden in. On the bottom of the globe Tintin finds the coordinates of the wreck of
the real Unicorn. The secrets have been solved…
Because it is proposed that the appearance of Captain Haddock might not have fallen in the
uncanny valley because he has a more cartoonish appearance due to his big nose, he will not
be used in the following analysis (Kaba, 2013, p. 193). The reviews of this movie were
especially negative for the appearance of Tintin. Therefore we will focus, with one exception,
on examples of his appearance in the analysis.
4.3
Emotions
Based on the research of Ekman, there are seven universally recognized facial expression of
emotions. These anger, disgust, fear, happiness, sadness, surprise and contempt. The facial
expressions can be recognized by the Action Units defined in the Facial Action Coding System
(See chapter 3) by Ekman and Friesen (2002). Because the facial expression of contempt has
no Action Units in the upperpart of the face, we will not discuss it in this analysis.
Unfortunately, Tintin does not show the emotions of disgust and sadness in The Adventures
of Tintin. Captain Haddock does experience the emotion sadness briefly but his appearance
is unsuitable for this analysis. Therefore both the sadness and disgust will be excluded from
68 | P a g e
4 Case Studies
the analysis of the facial expressions and we will concentrated on the other remaining four
facial expressions: Anger, fear, happiness and surprise.
4.3.1
Anger
In an angry facial expression we will see the Action Unit 4. The Brow Lowerer pulls the
eyebrows downwards and together. An intense frown and vertical or 45 angled wrinkles will
appear between the eyebrows due to contraction of the muscle corrugator supercilii. In some
cases of anger, we can see the AU5, the Upper Lid Raiser and in other cases the AU7, the
Lid Tightener. These Action Units can dilate or compress the eye aperture (Ekman, et al.,
2002, pp. 17-22; Kahle, et al., 1984, p. 312).
The Polar Express
There are two moments where the Conductor could show an angry facial expression. After
the Boy has pulled the emergency brake for the first time to let Billy get on the train, the
Conductor acts by shouting, as being angry, but here there is no facial expression of anger.
In this case he might have been more irritated than angry with the Boy for pulling the
emergency brake.
Figure 17: The facial expression of anger on the face of the Conductor in The Polar Express (The Polar Express,
2004).
At the moment the Boy has stopped the train again because the tracks are blocked by a group
of reindeers, the Conductor gets angry at the Boy for stopping the train for the second time
and causing more delay. The Conductor wants to get to the North Pole on time. In an overview
shot of the train and the tracks with the reindeers, we can already hear the angry voice of the
Conductor, who is coming to the front of the train. In the next shot we see him face to face at
a profile angled shot with the Boy. As we can see in figure 17 the eyebrow of the Conductor
is pulled downwards and a frown is visible above the nose. This part of AU4 is recognizable.
69 | P a g e
4 Case Studies
Because we see the Conductor only at a profile angle, it is hard to distinguish if there are any
vertical or 45 angled wrinkles. When we look at the eyelids of the Conductor, we can see that
the lower eye lid is relative high raised. This can be because of the AU7, which tightens the
eyelid whereby the lower eyelid covers more of the eyeball than it would normally do. Because
the upper eyelid is not that far lowered, in this case we can conclude this is the AU5, which
raises the upper eye lid whereby the lower eyelid raises a little as well. The sclera above the
iris, which is normally exposed in AU5 is not visible because the AU4 pulls the skin beneath
the eyebrow downwards, which narrows the eye aperture.
The Adventures of Tintin
Tintin shoots down the water airplane from where he, Captain Haddock and Snowy has been
shot at. He swims to the landed airplane and point his gun on the two pilots. In this moment
he shouts at the pilots that they should put their hands in the air. At the point where he shouts
“Now” we can see his angry facial expression (see figure 18). The upper eyelids are pulled
upwards by AU5. This can be distinguish because the upper eyelid disappears beneath the
skin above the eye. Because of AU4 that pulls the eyebrows downwards, the sclera above the
iris is not visible. This is due to the skin underneath the eyebrows that is also lowered with the
eyebrows. We can see that the eyebrows are pulled together by the two almost vertical
wrinkles between the eyebrows.
Figure 18: Tintin's facial expression of the emotion anger (The Adventures of Tintin: The Secret of the Unicorn,
2011).
In both the films the facial expression of anger is clearly visible. Both characters show the
AU4, we can see the eyebrows are being pulled downwards. Though, in both the expressions
this could have been slightly stronger. In the face of Tintin we can see the almost vertical
wrinkles between the eyebrows but when AU4 was stronger we could see them more clearly
which would have emphasized the anger in the expression more. The same applies to the
70 | P a g e
4 Case Studies
Conductor, a visible wrinkle would have made the anger more believable. In this case of the
Conductor it is hard to say if he really is angry or if he just pretends being angry.
4.3.2
Happiness
The facial expression of happiness is recognized by the smile-shaped mouth. This is due to
the AU12. If a person wants to fake a smile when he or she is not feeling happiness, they will
do this with AU12. But if a person is smiling because he or she is experiencing the emotion
happiness, then also the AU6 will be visible. This means that the pars orbitalis of the orbicularis
ocluli muscles contracts. This will be visible in the outer corners of the eye aperture. Here
crow’s feet wrinkles will appear (Ekman, et al., 2002, pp. 31-178; Kahle, et al., 1984, p. 314).
The Polar Express
In this scene of the Polar Express, Billy finds his first Christmas present ever. He is smiling
because this is, of course, a happy moment in his life. As we can see in figure 19, Billy’s mouth
is smile-shaped but the crow’s feet wrinkles are not visible in the corners of his eye aperture.
Therefore the feeling of joy is not communicated by his eyes. As well as in the Harry Potter
character Karkaroff we can say here: “his smile did not extend to his eyes” (Rowling, 2000, p.
247).
Figure 19: The character Billy (right) is smiling but does not show any signs of crow's feet on the outside of this
eye aperture (The Polar Express, 2004).
The Adventures of Tintin
A real smile is hard to find in the facial expression of Tintin and when he laughs, this is most
of the time not captured by the camera. After Tintin and Captain Haddock has managed to get
all the papers clues and they see that the marks on the papers together form the coordinates
to the treasure he does laugh, even if it is not a big smile. In figure 20, we can see the
71 | P a g e
4 Case Studies
screenshot of this moment. Tintin mouth is smile shaped. His eye apertures are slightly
narrowed and there is a light crow’s feet wrinkle visible on the outer side of the eye aperture.
Figure 20: The facial expression of happiness on the face of Tintin (left). A close-up of his left eye with minimal
visible crow's feet on the outside of the eye aperture (The Adventures of Tintin: The Secret of the
Unicorn, 2011).
This is caused by AU6. To make this facial expression more extensive, these could have been
deeper to make them more noticeable.
As been mentioned before, the real smile can be recognized by presence of AU6. The crow’s
feet wrinkles on the outer side of the eye aperture are very determinative for a real smile.
Though, these wrinkles occur in the facial expression of Tintin, they are not very clear. But if
we look at the eyes of Billy in The Polar Express we can conclude Tintin’s expression is much
better. On the other hand, this is not that hard to accomplish because Billy does not show any
sign of AU6. Therefore his smile looks fake and the feeling of true happiness is not
communicated to the audience.
4.3.3
Fear
The facial expression of fear shows the combination of the AU’s 1 + 2 + 4. This means that
the eyebrows are raised but not as far as in the combination of AU1 and AU2. The brows are
pulled together and more flattened to a straight or the inner parts of the eyebrows have a small
curve. Horizontal wrinkles will appear on the forehead due to the contraction of the frontalis
muscle. The AU5 is as well a part of the fear facial expression. Here the upper eyelids are
raised and the sclera about the iris becomes visible (Ekman, et al., 2002, pp. 17-24; Faller &
Schünke, 2012, p. 640).
72 | P a g e
4 Case Studies
The Polar Express
The scene where the characters experience the most fear, is where the train changes into a
rollercoaster due to the stuck handle, which makes it impossible to slow down the train (see
figure 21). Here the Boy, the Girl and the Conductor are on the very front of the train.
When the train goes almost vertically down the tracks, we can propose that this is a fearful
moment for the two children and the Conductor. We will discuss the facial expressions of the
Boy and the Girl. This, because the facial expression of the Conductor are mostly concealed
by his head. In the Girl’s facial expression there is only a small amount of the combination of
AU1 + AU2 + AU4 visible. The eyebrows are raised a small amount and the little curve the
eyebrows make at the inner parts shows that AU4 pulls them downwards. Also AU5 is very
minimal present. In this kind of fear it is expected AUs for the facial expression where more
visible. Right now, the upperpart of the face does not communicate the amount of fear she is
in.
Figure 21: The Girl (left) and the Boy (right) show the facial expression of fear in The Polar Express (The Polar
Express, 2004).
The Boys facial expression is much more expressive. The eyebrows are raised much higher
and are also more pulled together. The inner and outer parts of the eyebrows at the same
height. The pulling downwards of AU4 on the middle part of the eyebrow could have been
slightly more so the eyebrows would be more flattened. What is missing in this facial
expression, and in the Girls as well, are horizontal wrinkles. Also, AU5 could have been more
present as well. This way the eye aperture would have dilated more and this emphasizes the
fear in the facial expression.
73 | P a g e
4 Case Studies
The Adventures of Tintin
At the moment Tintin walks back to his apartment, one of Sakharine’s men tells him he has
got a delivery. When Tintin turns around to look at this box, Sakharine’s man grabs him from
behind and pushes a cloth in Tintin’s face. In this short moment Tintin shows a facial
expression of fear. His upper eyelids are raised exposing the sclera above the iris, which
indicates AU5. His eyebrows are raised, indicating AU1 and AU2, causing some horizontal
wrinkles on his forehead. The small dip in the inner part of the eyebrows shows that they are
pulled down by AU4. The only thing which is not visible is the vertical wrinkles between the
eyebrows, showing that these are pulled together. We can see the same facial expression of
Tintin when he is chased by a dog. In this case we can distinguish that the AU4 pulls the
eyebrows together. Here is not a deep wrinkle but we can see the skin bags together between
the eyebrows (see figure 22).
Figure 22: Action Units 1, 2 and 5 are visible in Tintin's facial expression of fear. On the left, the moment Tintin
gets grabbed from behind and on the right the moment Tintin is chased by a dog (The Adventures of
Tintin: The Secret of the Unicorn, 2011).
The facial expression of Tintin could have been slightly more presented in the first case. It is
possible that in this case he was experiencing fear as well as surprise. Also, as described in
chapter 3.4.7, the facial expressions of surprise and fear are very similar. This can explain
why the eyebrows are not pulled together that much as could have been expected in the facial
expression of fear. In the moment he is chased by a dog, we know that he is not surprised
because he knows, of course, that he is being chased. Therefore we can conclude that here
only the facial expression of fear should be clearly distinguishable. We can see that this facial
expression is much more expressive than that of the Boy. The most striking difference is the
lack of wrinkles or even a minimal tension of the skin on the forehead of both the Boy and the
Girl in the Polar Express. This is in the facial expression of Tintin clearly visible.
74 | P a g e
4 Case Studies
4.3.4
Surprise
The inner and outer parts of the eyebrows are raised when the frontalis muscle is contracted
by the combination of AU1 and AU2. This causes horizontal wrinkles on the forehead because
the skin is drawn together. Also the eye aperture is dilated by the raising of the upper eyelid.
This is the AU5 of the FACS. The dilated eye apertures gives the person the possibility to
gather more visual information (Ekman, et al., 2002, pp. 20-24; Kahle, et al., 1984, p. 312;
Faller & Schünke, 2012, p. 640).
The Polar Express
Just before the train reaches the North Pole, the Conductor, the Boy and the Girl are standing
on the back balcony of the last wagon. Here the Conductor points out to them that the lights
in the distance are the North Pole. Especially the Boy is very surprised by this news (see figure
23). This can be distinguish by the raised eyebrows and the slightly dilated eye aperture
caused by AU5. In this facial expression we can see the combination of AU1 + AU2. There is
no tension between the eyebrows indicating that AU4 is visible as well, therefore we know this
is not the facial expression of fear. But what is missing in this facial expression, as well as it
was in the facial expression of fear with the same character, is the lack of horizontal wrinkles
caused by the combination of AU1 and AU2.
Figure 23: The Boy (right) shows the facial expression of surprise in The Polar Express
(The Polar Express, 2004).
The Adventures of Tintin
Tintin is several times surprised in the film. For example when he finds the model of the
Unicorn. But just as often the case is with his happy facial expression, there are objects
between his face and the camera, which makes his facial expression hard to see. Though
75 | P a g e
4 Case Studies
there is one time when his facial expression is clearly visible in the shot. This is the moment
he finds out where the third model of the Unicorn is (see figure 24).
In his facial expression we can see wide open eyes. This is a sign the AU5 pulls the upper
eyelid upwards. Clearly visible is the sclera above the iris. The whole eyebrows are raised due
to AU1 and AU2. The skin on the forehead is compressed together causing clear horizontal
wrinkles. This is the exact facial expression how it is described by Ekman and Friesen.
Figure 24: Tintin's facial expression of surprise (The Adventures of Tintin: The Secret of the Unicorn, 2011).
Comparing the facial expression of surprise of both of the characters, we can conclude that
the one of Tintin is exactly how it is described and this of the Boy is clearly missing the
horizontal wrinkles. This is the main difference between the two films. In The Polar Express
has been hardly any movement in the upperpart of the forehead in the facial expressions of
the emotions whereas in Tintin these are present to contribute to the expressions of his
feelings.
4.4
Eye Movements in Actions and Human Interaction
The Polar Express
Though Brewdow, et al. claim that the eyes of the characters are moving all the time, this is in
many shots not visible. In the scene, which we can see in figure 25, the Boy has just entered
the train and he has taken his seat. On the other side of the aisle sits the Girl. She starts to
talk with him but not only is she staring at him by not showing any saccades in her gaze, the
position of where she looks is not in the direction that is expected. Instead of looking him in
the face, her gaze is pointed in the direction of his navel. This can be seen in following images.
Figure 25 (left) is the last frame of where the shot of the Boy ends and figure 25 (right) is the
76 | P a g e
4 Case Studies
frame where the shot of the Girl starts. In the rest of the shot of the Girl, which has a duration
of 58 frames (approximately 2.4 seconds), her head slightly moves but her gaze stays at the
same point.
In mutual gaze it is normal to look at another person between five and fifteen seconds before
looking away. Irrespective of the wrong position of the gaze, it is therefore plausible that she
looks at him the whole length of the 58 frames of this shot. However, it is unexpected that she
stares at him without showing any saccades and fixation patterns in her eye movement. Also,
the eyes of the Conductor and of Santa Claus do not show any saccades and the stay fixated
on the same place during mutual gaze. On the other hand, the eyes of the Boy does show the
pattern of saccades and fixations some of the time.
Figure 25: A conversation between the Boy (left) and the Girl (right) in The Polar Express. Especially the Girl has
an unfocussed, wrongly placed gaze with a lack of saccades (The Polar Express, 2004).
Because of the short facial shots in the polar express, it is hard to analyse the blinking of the
persons. All the characters do blink in the movie but some of them only a few times. This, in
combination with the lack of saccades, the characters give the impression that they stare a
lot. This is especially with the Conductor, Billy and Santa Claus. In these characters we cannot
see a lot of blinking in the close-up shot where we can see their eyes.
The Adventures of Tintin
In this movie the eye movements are very realistic. There is a continue pattern of saccades
and fixations. This is as well in the human interaction and also when the character looks at an
object. In the mutual gaze the characters look each other in the face. Here the eyes are not
focussed all the time at one position and the eyes make the small and fast movement like
saccades. The direct gaze is timed like it is expected, whereby the gaze is moved away from
the face after a few seconds. When the character talks, he does not keep looking at the face
of his interlocutor the whole time he speaks. It is a reciprocity between looking away from the
face and back again for nonverbal feedback of the other character. In this example we see by
the position of his Sakharine’s eyes that he is recalling an image he has seen before as he is
77 | P a g e
4 Case Studies
looking upwards and to the left. We know that Sakharine’s is right-handed because shortly
before this shot happens he wants to write a cheque to buy the model of Tintin. In this shot he
is talking about the Haddock family and is probably recalling an image of Captain Haddock
who he detains on his ship at the same moment in time.
Because of the image-based facial performance capture that was used during the shooting of
the film, the blinking of the characters in The Adventures of Tintin are highly realistic. They are
the same blinks the motion captures actors have made. This is all due to the facial camera
that was mounted on the helmet and pointed to the face of the actor. This makes a big
difference when it is compared to the characters of The Polar Express. In The Adventures of
Tintin there are no staring eyes or long times without blinking which also gives the impression
of staring.
Figure 26: Screenshots from a mutual gaze scene in The Adventures of Tintin (The Adventures of Tintin: The
Secret of the Unicorn, 2011).
78 | P a g e
4 Case Studies
4.5
Pupil Dilation
Throughout the films we cannot see any pupil dilation in the characters. In the close-ups, which
can been seen in figure 27, the pupils of the Boy (top row) and Tintin (bottom row) stay the
same size in several shots. Even when they get excited or in different light circumstances.
Because there is no compare we cannot examine and judge if this has any influence on the
credibility of the characters.
Figure 27: The pupil sizes of the Boy in The Polar Express (top row) and of Tintin (bottom row) thoughout the
movies (The Polar Express, 2004; The Adventures of Tintin: The Secret of the Unicorn, 2011).
4.6
Startle reflex
Tinwell thinks that the character of Tintin did received negative reviews because he is missing
the startle reflex (see chapter 2.4.7). According to Tinwell, Tintin does not show any facial
reaction in the upperpart of the face when there are sounds like gunshots and explosions. She
declares that the audience expects Tintin to show a reaction when he is shocked or scared by
these events, but there was no startle reflex in his face. The audience wondered if he is really
scared or shocked by these life-threatening events happening to him. With the startled reaction
missing in his face, the audience may have thought that Tintin was a cold and antisocial
person. “Viewers questioned whether Tintin was really scared and shocked” (Tinwell, 2015, p. 112).
Therefore it is difficult to feel any empathy for him and this character becomes less believable.
(Tinwell, 2015, pp. 111-112)
It is not clear where Tinwell has based this conclusion on. In the film Tintin is confronted with
two explosions and four situations where there was gun-firing. In none of these events we can
see Tintins face at the exact moment the sound is heard. But during one of the firefights, Tintin
bumps into one of Sakharine’s men. In this exact moment we can see the startle reflex in
79 | P a g e
4 Case Studies
Tintin’s facial expression. Another moment where we can see this reflex is when Tintin is in
the house of Sakharine and finds the second model. At this moment he hears a sound behind
him.
With these images we can conclude that Tintin does in fact show a startle reflex to unexpected
noises and sights.
Figure 28: The startle reflex of Tintin (The Adventures of Tintin: The Secret of the Unicorn, 2011).
4.7
Summary
In this chapter we have analysed the movements of the eyes and the upperpart of the face of
characters from the films The Polar Express and The Adventures of Tintin. This has been
done with the knowledge of oculesics and the Facial Action Coding System of Ekman and
Friesen. The four facial expressions that have been analysed are those of the emotions anger,
fear, happiness and surprise. Also the eye pattern movements of saccades and fixations
characters has been analysed. With this analysis we can conclude that there is a difference
between the two films were the facial expressions and eye movements in The Adventures of
Tintin are significant better than those of The Polar Express.
80 | P a g e
5 Conclusion
5
Conclusion
In this chapter the findings of the analysis of the two films will be discussed further. We will
combine this with the knowledge of the previous chapters. At the end of this chapter there will
be a summary of the whole thesis.
5.1
Discussion
From the analysis in chapter 4 it can be concluded, that the facial expressions and the
movements of the eyes are significant different between in The Polar Express and The
Adventures of Tintin. Due to the image-based facial performance capture the performance of
the actors can accurately be transmitted on the CG-Character.
The analysis makes it clear why the characters of The Polar Express fall into the uncanny
valley. The lack of saccades and fixations, staring eyes, uncompleted facial expressions of
emotions and unfocussed, wrong positioned gaze are giving the characters a creepy and
unnatural appearance.
What is outstanding is the use of saccades and fixation pattern in the eye movements of Tintin
in The Adventures of Tintin. As what have could been seen in the analysis of The Polar
Express, this was mainly responsible for the creepy staring faces. Compared to this, the eye
movements of the characters in The Adventures of Tintin are completely natural. Here,
everything fits together according to oculesics. The only thing that could not be found in the
film was the dilation of the pupils. It is possible that this is the reason for the negative reviews
about Tintin's lifeless eyes. However, this can only be concluded after investigating characters
with pupil dilation.
The declaration of Tinwell, that the character Tintin is giving the audience an uncanny feeling
because of the lack of the startle reflex when being confronted with loud sounds or unexpected
sights could not be confirmed. In the scenes of the movie that where viewed in order to
research this statement, the startle reflex was visible in the facial expression of Tintin.
If we only look at their eye movements and facial expressions, the following image is an
estimation where the characters of both films would end up on Mori’s hypothetical graph. The
characters of The Polar Express are at the bottom of the valley. The Boy’s facial expressions
and eye movements where slightly better than the rest due to the fact that he sometimes
showed the saccades and fixation pattern. The Conductor would be found at approximately
the same height as Billy. The Girl was the worst of the characters due to her unfocussed eyes
81 | P a g e
5 Conclusion
and staring gaze. Therefore she is at the bottom of the valley. Captain Haddock of The
Adventure of Tintin is found before the valley. His big nose might have saved him from the
valley. Because we have not analyzed his eyes, it is not possible to say where he would have
end up on the uncanny valley graph. We can find Tintin on the other side of the valley but not
at the top. His eye movements where very realistic but not all of his facial expressions were
evident enough. Also the lack of pupil dilation holds him from the top of the valley.
Figure 29: The uncanny valley graph by Mori with the characters of The Polar Express and The Advenures of
Tintin. (Tinwell, et al., 2011b, p. 328; The Polar Express, 2004; The Adventures of Tintin: The Secret
of the Unicorn, 2011) (Modified by the author)
To come back to the research question of this thesis:
How can the knowledge of Oculesics enhance the design and animation of a near
human character to help overcoming the uncanny valley?
The fear of the unknown, of threats, of being replaced or being overruled and the fear of death,
are theories why the phenomenon of the uncanny valley exists. This thesis has tried to make
clear which important part oculesics has in overcoming the uncanny valley. But not just
oculesics, also the facial expressions and the movement in the forehead of the characters are
82 | P a g e
5 Conclusion
extremely important. Even the slightest off-movement will be noticed by the audience. This
has been highlighted by the difference of the two movies. Where the character of The Polar
Express where compared to zombies, the reviews of Tintin were milder. The pattern of
saccades and fixations and the Action Units of the Facial Action Coding System of Ekman and
Friesen can help the animation industry further on its way to the CG-character in overcoming
the uncanny valley.
The following points should be considered after this research. Not for everyone the effect of
the uncanny valley is the same and it is still not clear what the exact reason is for the existence
of the phenomenon uncanny valley. It is plausible that there is not just one explanation or
theory. Likely, it is a combination of the different theories discussed in this thesis.
Though the findings of this thesis are focussed on the eyes of a near-human CG-character,
this does not mean that with perfectly applied oculesics and facial expression a character can
overcome the uncanny valley. Wrong eyes and facial expressions do contribute to the
phenomenon but to overcome the uncanny valley, the whole picture should fit together. This
means the style of the character, the voice and the rest of the body movements. Because we
are so used to other human beings, every little mistake in the animation of a character can
cause it to fall in the uncanny valley. In the case of Tintin, the problem might also be that he
sometimes takes cartoonish postures. These are based on the postures of Tintin in the comic
books. Whether or not this contributes to the style of the film, it might sometimes seem
unnatural to the audience.
Another possible problem of the film is that there are not many emotions in the first place.
There is a lot of action and very few time for the audience to bind with Tintin as hero. Also in
the moments were it is probable that he is feeling strong emotions as fear, his face is not in
visible in the shots. Mostly his facial expression is more concentrated on his action. Therefore
the audience might not be able to feel an emotional connection or feel empathy with him.
The question if Tintin has crossed the uncanny valley is therefore very hard to answer. Based
on the reviews, apparently there are people who think Tintin has something creepy and has
fallen into the uncanny valley. The fact that there are people who think this, indicates that there
is still something not quite right about the character. But, based on the analysis, Tintin is
definitely on the way up on the other side of the uncanny valley. Some facial expressions can
be slightly stronger and there should be more research to the influence of the pupil dilation in
a character.
83 | P a g e
5 Conclusion
There are rumours that there will be a new film of The Adventures of Tintin. This movie will
get the title The Adventures of Tintin: Prisoners of the Sun. There is not a fix released date for
this movie and the status of the production is still unclear. The rumours on the internet say,
the movie again is directed and produced by Steven Spielberg and Peter Jackson. Possible
release date are indicated for Christmas 2016 or in the summer of 2017. This will, of course,
be very interesting for animation industry. The question is, did they learn from the criticism of
the first movie and will they be able to get Tintin completely on the other side of the uncanny
valley? (IMDb, 2016; Jagernauth, 2016)
This thesis has tried to give an overview of what the uncanny valley is and what the influences
of the eyes are on a character falling into the uncanny valley. At the same time it is an overview
for those who animate the facial expressions of near-human characters and a designation of
how important the eyes are to other human beings. It would be advisable for animators to
focus on the upperpart of face of the character, because a lot of nonverbal communication
happens in this area of the face. The question if there will ever be a near-human character
who will overcome the uncanny valley can only be answered when we get there. Maybe it will
be a new Adventure of Tintin.
5.2
Summary of the Thesis
“The uncanny is ghostly. It is concerned with the strange, weird and mysterious with
a flickering sense (but not conviction) of something supernatural. The uncanny
involves feelings of uncertainty, in particular regarding the reality of who one is and
what is being experienced.” (Royal, 2003, p. 1)
This Thesis has started off with the exploring of what creepiness and uncanny is. The first
essays about uncanny were those of Jentsch and Freud. They came to the conclusion that
the uncanny is caused by intellectual uncertainty and secrets that come out but should have
stayed concealed. The story of The Sandman has shown us a first case of the uncanny valley.
This phenomenon has been discussed next. The uncanny valley is a hypothetical graph first
proposed by the roboticist Masahiro Mori. He had noticed if a robots design got to humanlike,
at some point the robot appearance was perceived as uncanny.
Not only in robots can we find the phenomenon of the uncanny valley. This is as well noticed
in near-human CG-characters. Examples of this are the films are Final Fantasy: The Spirits
Within and The Polar Express.
There are different theories of why the uncanny valley phenomenon appears. There are
theories that believe that we have a fear of being replaced by our doppelganger. Other theories
84 | P a g e
5 Conclusion
include the fear of threats and death. Tinwell thinks the uncanny valley appears because the
near-human character might be showing signs of being psychopathic. Tinwell has also done
research to the lack of animation in the upperpart of the face when a stimuli is showing a facial
expression.
Oculesics is “The study of the communicative role of the eyes in nonverbal communication” (Chandler
& Munday, 2011). Chapter 3 has been an overview of the different aspects of oculesics. Here
the seven universally recognized facial expressions of emotions are reviewed. These are:
anger, disgust, fear, happiness, sadness, surprise and contempt. For each of these facial
expressions the Action Units according to the Facial Action Coding System of Ekman and
Friesen have been discussed. With these Action Units it is possible to distinguish the different
facial expression from one another.
As we have seen in the analysis of the two films The Polar Express and The Adventures of
Tintin: The Secret of the Unicorn, the eyes and upperpart of the face influencing the
appearance of the characters. With the knowledge of the Facial Action Coding System of
Ekman and Friesen, we learned how the seven universally recognized facial expressions can
be distinguish in the face of a character. Then we looked at the other areas of oculesics, like
gaze direction, pupil dilation, the pattern of saccades and fixations. With this information we
have seen what went wrong in the faces of the characters in the Polar Express. The missing
of saccades and the blinking of the eyes gave the impression of staring and unfocussed eyes.
This is an enormous difference with the movie of Tintin, where the eyes are continue moving
and correctly focused. This is due to the developed motion capture technique where in The
Adventures of Tintin, the face was separately recorded with an on the face pointed camera.
A remarkable difference between the films is also the animation of the forehead of the
character. In The Polar Express there was hardly any movement or wrinkles in the upperpart
of the face. This, compared to the facial expressions of Tintin, where in every facial expression
the expected wrinkles were visible.
We can end this thesis with the conclusion that oculesics has an important part in overcoming
the uncanny valley. The proper use of the Facial Action Coding Systems in the animation of
near-human CG-characters will help the facial expression of the character to get realistic as
well as the pattern of saccades and fixations. This is something that should always be notable
in the animations. The eyes and upperpart facial movement should be realistic otherwise this
will be noticed by the audience. Maybe then, the next near-human CG-character will not give
us the creeps.
85 | P a g e
6 Bibliography
6
Bibliography
Barfield, W., 2015. Cyber-Humans: Our Future with Machines. Göttingen(Germany):
Copernicus.
Bartneck, C., Kanda, T., Ishiguro, H. & Hagita, N., 2009. My Robotic Doppelgänger – A Critical
Look at the Uncanny Valley. The 18th IEEE International Symposium on Robot and Human
Interactive Communication Toyama, Japan,, pp. 269-276.
Bartneck, C., Suzuki, T., Kanda, T. & Nomura, T., 2007. The influence of people’s culture and
prior experiences with Aibo on their attitude towards robots. The Journal of Human-Centred
Systems, 21(1-2), pp. 217-230.
Becker, E., 1973. The Denial of Death. New York: The Free Press.
Bonanno, G. A., Goorin, L. & Coifman, K. G., 2008. Sadness and Grief. In: M. Lewis, J. M.
Haviland-Jones & L. Feldman Barrett, eds. Handbook of Emotions. 3rd ed. New York: The
Guilford Press, pp. 797-810.
Bredow, R. et al., 2005. From Mocap to Movie: The Polar Express, Los Angeles: Sony Pictures
Imageworks.
Buchanan, K., 2011. The Biggest Problem With the Tintin Movie Might Be Tintin Himself.
[Online]
Available
at:
http://www.vulture.com/2011/07/the_biggest_problem_with_the_t.html
[Accessed 24 April 2016].
Burleigh, T. J., Schoenherr, J. R. & Lacroix, G. L., 2013. Does the uncanny valley exist? An
empirical test of the relationship between eeriness and the human likeness of digitally created
faces. Computers in Human Behavior, 29(3), pp. 759-771.
Chandler, D. & Munday, R., 2011. A Dictionary of Media and Communication. Oxford: Oxford
University Press.
Chapman, R. C. et al., 1999. Phasic pupil dilation response to noxious stimulation in normal
volunteers: Relationship to brain evoked potentials and pain report. Psychophysiology, 36(1),
pp. 44-52.
86 | P a g e
6 Bibliography
Clinton,
P.,
Available
at:
2004.
Review:
'Polar
Express'
a
creepy
ride.
[Online]
http://edition.cnn.com/2004/SHOWBIZ/Movies/11/10/review.polar.express/
[Accessed 14 May 2016].
Darwin, C., 1890/2009. The Expressions of the Emotions in Man and Animals. Cambridge:
Cambridge University Press.
Destephe, M. et al., 2015. Walking in the uncanny valley: importance of the attractiveness on
the
acceptance
Available
of
at:
a
robot
as
a
working
partner.
[Online]
http://journal.frontiersin.org/article/10.3389/fpsyg.2015.00204/full
[Accessed 07 July 2016].
Duchenne de Boulogne, G. -B., 1862/1990. The Mechanism of Human Facial Expression.
Cambridge: Cambridge University Press.
Ekman, P., 1977. Facial Expression. In: Nonverbal Behavior and Communication. New
Jersey: Lawrence Erlbaum Association, pp. 97-126.
Ekman, P., 1992. An Argument for Basic Emotions. Cognition and Emotion, 6(3/4), pp. 169200.
Ekman, P., 1993. Facial Expression and Emotion. American Psychologist, 48(4), pp. 384-392.
Ekman, P., 1999. Basic Emotions. In: Handbook of Cognition and Emotion. Chichester: John
Wiley & Sons Ltd., pp. 45-60.
Ekman, P. & Friesen, W. V., 1967. Head and Body cues in the Judgment of Emotion: A
Reformulation. Perceptual and Motor Skills, Volume 24, pp. 711-724.
Ekman, P. & Friesen, W. V., 1969. The Repertoire of Nonverbal Behavior: Catergories,
Origins, Usage, and Coding. Semiotica, 1(1), pp. 49-98.
Ekman, P. & Friesen, W. V., 1982. Felt, False, and Miserable Smiles. Journal of Nonverbal
Behavior, 6(4), pp. 238-252.
Ekman, P. & Friesen, W. V., 1985. Is the Startle Reaction an Emotion?. Journal of Personality
and Social Psychology, 49(5), pp. 1416-1426.
87 | P a g e
6 Bibliography
Ekman, P., Friesen, W. V. & Davidson, R. J., 1990. The Duchenne Smile: Emotional
Expression and Brain Physiology II. Journal of Personality and Social Psychology, 58(2), pp.
342-353.
Ekman, P., Friesen, W. V. & Hager, J. C., 2002. Facial Action Coding System: The Manual,
[CD-ROM] Salt Lake City: A Human Face.
Ekman,
P.,
Available
n.d.
at:
About
Paul
Ekman.
[Online]
http://www.paulekman.com/paul-ekman/
[Accessed 15 Juni 2016].
Ekman, P. & Rosenberg, E. L., 2005. What the Face Reveals: Basic and Applied Studies of
Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford: Oxford
University Press.
Faller, A. & Schünke, M., 2012. Der Körper des Menschen. Stuttgart: Georg Thieme Verlag.
Final Fantasy: The Spirits WIthin. 2001. [Film] Directed by Hironobu Sakaguchi. USA: Square
Pictures.
Fischer, A., 2011. Contempt: A Hot Feeling Hidden under a Cold Jacket. In: Re-Constucting
Emotional Spaces: From Experience to Regulation. Prague: Prague College of Psychosocial
Studies Press, pp. 77-88.
Fong, J., 2012. Eye-Opener: Why Do Pupils Dilate in Response to Emotional States?. [Online]
Available
at:
http://www.scientificamerican.com/article/eye-opener-why-do-pupils-dialate/
[Accessed 11 July 2016].
Freud, S., 1919. The 'Uncanny'. The Standard Edition of the Complete Psychological Works
of Sigmund Freud, Volume XVII (1917-1919): An Infantile Neurosis and Other Works, pp. 217256.
Frozen. 2013. [Film] Directed by Chris Buck, Jennifer Lee. USA: Walt Disney Animation
Studios.
Gray, H. M., Gray, K. & Wegner, D. M., 2007. Dimensions of Mind Perception. Science, p.
619.
88 | P a g e
6 Bibliography
Gray, K. & Wegner, D. M., 2012. Feeling robots and human zombies: Mind perception and the
uncanny valley. Cognition, Volume 125, pp. 125-130.
Hayhoe, M. & Ballard, D., 2005. Eye movements in natural behavior. TRENDS in Cognitive
Sciences, April, 9(4), pp. 188-194.
Hess, E. H., 1975. The Role of Pupil Size in Communication. Scientific American, 233(5), pp.
110-112, 116-119.
Hess, E. H., Seltzer, A. L. & Shlien, J. M., 1965. Pupil response of hetero- and homosexual
males to pictures of men and women: A pilot study. Journal of Abnormal Psychology, 70(3),
pp. 165-168.
Ho, C.-C. & MacDorman, K. F., 2010. Revisiting the uncanny valley theory: Developing and
validating an alternative to the Godspeed indices. Computers in Human Behavior, 26(6), pp.
1508-1518.
Ho, C.-C., MacDorman, K. F. & Dwi Pramono, Z., 2008. Human Emotion and the Uncanny
Valley: A GLM, MDS, and Isomap Analysis of Robot Video Ratings. Human-Robot Interaction
(HRI), 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction, pp. 169176.
Hoffmann, E. T. A., 1885. Weird Tales - The Sandman. New York: Charles Scribner's Sons.
IMDb,
2016.
The
Adventures
Available
of
Tintin:
at:
Prisoners
of
the
Sun.
[Online]
http://www.imdb.com/title/tt1037226/
[Accessed 05 July 2016].
Inside Out. 2015. [Film] Directed by Pete Docter, Ronnie Del Carmen. USA: Pixar Animation
Studios.
Jagernauth, K., 2016. Peter Jackson Working On A Secret Project For Steven Spielberg’s
Amblin
Available
Entertainment.
at:
[Online]
http://theplaylist.net/peter-jackson-working-secret-project-steven-spielbergs-
amblin-entertainment-20160630/
[Accessed 05 July 2016].
Jentsch, E., 1906/1997. On the psychology of the uncanny. Angelaki, 2(1), pp. 7-16.
89 | P a g e
6 Bibliography
Kaba, F., 2013. Hyper-Realistic Characters and the Existence of the Uncanny Valley in
Animation Films. International Review of Social Sciences and Humanities, 4(2), pp. 188-195.
Kahle, W., Leonhardt, H. & Platzer, W., 1984. Taschenatlas der Anatomie. Stuttgart: Georg
Thieme Verlag.
Keltner, D. & Ekman, P., 2000. Facial Expressions of Emotions. In: M. Lewis & J. M. HavilandJones, eds. Handbook of emotions. 2nd ed. New York: Guilford Publications, Inc., pp. 236249.
Keysers, C., 2011. The Empathic Brain, [E-book] s.l.: Social Brain Press.
Knapp, M. L. & Hall, A. J., 1992. Nonverbal Communication in Human Interaction. 3rd ed.
Orlando: Ted Buchholz.
Land, M. F., 1999. Motion and vision: why animals move their eyes. Journal of Comparative
Physiology, Volume 185, pp. 341-352.
Land, M. F., 2006. Eye movements and the control of actions in everyday life. Progress in
Retinal and Eye Research, Volume 25, pp. 296-324.
Land, M. F. & Hayhoe, M., 2001. In what ways do eye movements contribute to everyday
activities?. Vision Research, Volume 41, pp. 3559-3565.
Leander, N. P., Chartrand, T. L. & Bargh, J. A., 2012. You Give Me the Chills: Embodied
Reactions to Inappropriate Amounts of Behavioral Mimicry. Psychological Science, 23(7), pp.
772-779.
LeDoux, J., 1996. The Emotional Brain: The Mysterious Underpinnings of Emotional Life. New
York: Touchstone.
Lemerise, E. A. & Dodge, K. A., 2008. The Development of Anger and Hostile Interactions. In:
M. Lewis, J. M. Haviland-Jones & L. Feldman Barrett, eds. Handbook of Emotions. 3rd ed.
New York: The Guilford Press, pp. 730-741.
Lewis, H., 2012. Body Language : A Guide for Professionals. 3rd ed. Thousand Oaks: SAGE
Publications India Pvt Ltd.
LoBue, V. & Rakison, D. H., 2013. What we fear most: A developmental advantage for threatrelevant stimuli. Developmental Review, Volume 33, pp. 285-303.
90 | P a g e
6 Bibliography
MacDorman, K. F. & Entezari, S. O., 2015. Individual differences predict sensitivity to the
uncanny valley. Interaction Studies, 16(2), pp. 141-172.
MacDorman, K. F., Green, R. D., Ho, C.-C. & Koch, C. T., 2009. Too real for comfort? Uncanny
responses to computer generated faces. Computers in Human Behavior, Volume 25, pp. 695710.
MacDorman, K. F. & Ishiguro, H., 2006. The uncanny advantage of using androids in cognitive
and social science research. Interaction Studies 7:3, pp. 297-337.
Matsumoto, D. et al., 2008. Facial Expressions of Emotion. In: M. Lewis, J. M. Haviland-Jones
& L. Feldman Barrett, eds. Handbook of Emotions. 3rd ed. New York-London: THE
GUILFORD PRESS, pp. 211-234.
McAndrew, F. T. & Koehnke, S. S., 2016. On the Nature of creepiness. New Ideas in
Psychology, Volume 43, pp. 10-15.
Menache, A., 2011. Understanding Motion Capture for Computer Animation. 2nd ed.
Burlington: Morgan Kaufmann.
Minato, T., Shimada, M., Ishiguro, H. & Itakura, S., 2004. Development of an Android Robot
for Studying Human-Robot Interaction. In: Innovations in Applied Artificial Intelligence.
Heidelberg: Springer, pp. 424-434.
Mori, M., 1970. Bukimi no tani [the uncanny valley] (K. F. MacDorman & T. Minato, Trans.).
Energy, 7(4), pp. 33-35.
Mori, M., 2012. The Uncanny Valley (K. F. MacDorman & Norri Kageki, Trans.). IEEE Robotics
& Automation Magazine, pp. 98-100.
Nakashima, R., 2011. Too real means too creepy in new Disney animation. [Online]
Available
at:
http://usatoday30.usatoday.com/tech/news/2011-04-04-creepy-
animation_N.htm
[Accessed 11 May 2016].
Pan's Labyrinth. 2006. [Film] Directed by Guillermo Del Toro. Spain/Mexico: Tequila Gang,
Estudios Picasso.
91 | P a g e
6 Bibliography
Paramount Pictures & Columbia Pictures, 2011. The Adventures of Tintin: The Secret of the
Unicorn - Production Notes, [PDF] s.l.: Paramount Pictures and Columbia Pictures.
Park, J. E., 2001. Behind the Scenes on 'Final Fantasy: The Spirits Within'. [Online]
Available at: http://www.awn.com/animationworld/behind-scenes-final-fantasy-spirits-within
[Accessed 07 July 2016].
Pinkerton, N., 2011. The Adventures of Tintin: Spielberg's 3-D Detective Is a Little Creepy.
[Online]
Available
at:
http://www.seattleweekly.com/2011-12-21/film/the-adventures-of-tintin-
spielberg-s-3-d-detective-is-a-little-creepy/
[Accessed 24 April 2016].
Plantec,
P.,
Available
2007.
at:
Crossing
the
Great
Uncanny
Valley.
[Online]
http://www.awn.com/vfxworld/crossing-great-uncanny-valley
[Accessed 04 June 2016].
Pyszczynski, T., Greenberg, J. & Solomon, S., 1999. A Dual-Process Model of Defense
Against Conscious and Unconscious Death-Related Thoughts An Extension of Terror
Management Theory. Psychological Review, 106(4), pp. 835-845.
Ramey, C. H., 2005. The uncanny valley of similarities concerning abortion, baldness, heaps
of
Available
sand,
and
at:
humanlike
robots.
[Online]
http://www.androidscience.com/theuncannyvalley/cfp.html
[Accessed 04 Juni 2016].
Rank, O., 1925. Der Doppelgänger. Leipzig/Wien/Zürich: Internationaler Psychoanalytischer
Verlag.
Richmond, V. P., McMcroskey, J. C. & Payne, S. K., 1991. Nonverbal Behavior in
Interpersonal Relations. Englewood Cliffs: Prentice-Hall, Inc.
Rom, S., 2011. Review: The Adventures of Tintin: The Secret of the Unicorn. [Online]
Available at: http://www.trespassmag.com/review-the-adventures-of-tintin-the-secret-of-theunicorn/
[Accessed 24 April 2016].
Rowling, J. K., 2000. Harry Potter and the Goblet of Fire. New York: Scholastic Press.
92 | P a g e
6 Bibliography
Royal, N., 2003. The Uncanny. New York: Routledge.
Rozin, P. & Fallon, A. E., 1987. A Perspective on Disgust. Psychological Review, 94(1), pp.
23-41.
Rozin, P., Haidt, J. & McCauley, C. R., 2008. Disgust. In: M. Lewis, J. M. Haviland-Jones & L.
Feldman Barrett, eds. Handbook Of Emotions. 3rd ed. New York: The Guilford Press, pp. 757776.
Schein, C. & Gray, K., 2015. The eyes are the window to the uncanny valley: Mind perception,
autism and missing souls. Interaction Studies, 16(2), pp. 173-179.
Schweigert, J., 2010. The Uncanny: The Double as a Literary Convention, Brno, Czech
Republic: Masaryk University Faculty of Arts - English Language and Literature.
Seyama, J. & Nagayama, R. S., 2007. The Uncanny Valley: Effect of Realism on the
Impression of Artificial Human Faces. Presence, pp. 337-351.
Shiota, M. N. & Kalat, J. W., 2012. Emotions. 2nd ed. Belmont: Wadsworth.
Snyder, D. D., 2011. 'Tintin' and the Curious Case of the Dead Eyes. [Online]
Available
at:
http://www.theatlantic.com/entertainment/archive/2011/12/tintin-and-the-
curious-case-of-the-dead-eyes/250432/
[Accessed 24 April 2016].
Solomon, R. C., 2008. The Philosophy of Emotions. In: M. Lewis, J. M. Haviland-Jones & L.
Feldman Barrett, eds. Handbook of Emotions. 3rd ed. New York: The Guilford Press, pp. 316.
Solomon, S., Greenberg, J. & Pyszczynski, T., 2000. Pride and Prejudice: Fear of Death and
Social Behavior. Current Directions in Psychological Science, 9(6), pp. 200-204.
Steckenfinger, S. A. & Ghazanfar, A. A., 2009. Monkey visual behavior falls into the uncanny
valley. Pinas, 27 October, 106(43), pp. 18362-18366.
Stevenson, R. L., 1886/2015. The Strange Case of Dr Jekyll and Mr Hyde, [E-Book] Adelaide:
The
University
of
Adelaide.
Available at: https://ebooks.adelaide.edu.au/s/stevenson/robert_louis/s848dj/
93 | P a g e
6 Bibliography
St-Germain, P., 2010. Doubles and doppelgangers: religious meaning for the young and old.
[Online]
Available
at:
http://oic.uqam.ca/en/publications/doubles-and-doppelgangers-religious-
meaning-for-the-young-and-old
[Accessed 2016 May 22].
The Adventures of Tintin: The Secret of the Unicorn. 2011. [Film] Directed by Steven
Spielberg. USA/New Zealand: Nickelodeon Movies,.
The Incredibles. 2004. [Film] Directed by Brad Bird. USA: Walt Disney Pictures, Pixar
Animation Studios.
The Polar Express. 2004. [Film] Directed by Robert Zemeckis. USA: Castle Rock
Entertainment.
Tin Toy. 1988. [Film] Directed by John Lasseter. USA: Pixar Animation Studios.
Tinwell, A., 2015. The Uncanny Valley in Games & Animation. Boca Raton: CRC Press.
Tinwell, A., Abdel Nabi, D. & Charlton, J. P., 2013. Perception of psychopathy and the
Uncanny Valley in virtual characters. Computers in Human Behavior, Volume 29, pp. 16171625.
Tinwell, A. & Grimshaw, M., 2009. Survival Horror Games – An Uncanny Modality. Montreal,
Canada, University of Bolton, pp. 1-26.
Tinwell, A., Grimshaw, M., Abdel Nabi, D. & Williams, A., 2011a. Facial Expression of Emotion
and Perception of the Uncanny Valley in Virtual Characters. Computers in Human Behavior,
Volume 27, pp. 741-749.
Tinwell, A., Grimshaw, M. & Williams, A., 2011b. The Uncanny Wall. International Journal of
Arts and Technology, 4(3), pp. 326-341.
Živković, M., 2000. The Double as the "Unseen" of Culture: Toward a Definition of
Doppelganger. Facta Universitatis - Linguistics and Literature, 2(7), pp. 121-128.
94 | P a g e
List of Figures
List of Figures
Figure 1:
The Uncanny Valley graph by Masahiro Mori (Tinwell, et al., 2011b, p. 328). ... 20
Figure 2:
The character Riley from Pixar's Inside Out (left).
Available at: (https://i.ytimg.com/vi/t4okAfKCwRk/maxresdefault.jpg). .............. 24
Figure 3:
The characters Elsa and Anna from Disney's Frozen (right).
Available at: (http://images6.fanpop.com/image/photos/39100000/Elsa-andAnna-frozen-39135031-1218-1858.jpg). ........................................................... 24
Figure 4:
The character Baby Billy from the short film Tin Toy by Pixar.
Available at: (http://simplymaya.com/articles/uncannyvalley/images/uncanny_01.jpg). ........................................................................ 25
Figure 5:
Final Fantasy: The Spirits Within. The character Doctor Aki Ross falls into the
uncanny valley.
Available at: (http://4.bp.blogspot.com/-WckWWM7pvo8/
UN8dmIKcpUI/AAAAAAAAAic/fcWEB49EPE8/s1600/fftsw.jpg). ....................... 25
Figure 6:
Three of the main characters of The Polar Express (The Polar Express, 2004). 26
Figure 7:
The main characters from the film The Incredibles (right)
Available at: (http://mickeymousepictures.com/wp-content/uploads/2015/08/TheIncredibles-1.jpg). ............................................................................................. 26
Figure 8:
The characters Tintin (right) and Captain Haddock (left) in The Adventures of
Tintin: The Secret of the Unicorn (The Adventures of Tintin: The Secret of the
Unicorn, 2011). ................................................................................................. 26
Figure 9:
Tom Hanks as motion capture actor for The Polar Express (left) and his CGcharacter in the same position (right) (Bredow, et al., 2005). ............................ 27
Figure 10: Pale Man from the movie Pan's Labyrinth from Guillermo del Toro
Available at:
(http://41.media.tumblr.com/tumblr_m4o4punTWq1qj22x9o1_1280.jpg) ........... 37
Figure 11: The facial muscles of the human face
Available at: (https://montereybayholistic.files.wordpress.com/2013/01/facialmuscles.jpg)
(Modified by the author). ................................................................................... 41
95 | P a g e
List of Figures
Figure 12: A side view of the human eye.
Available at: (http://www.arizonaeyeconsultants.com/wpcontent/uploads/2015/09/EyeAnatomy600x540.jpg)
(Modified by the author) .................................................................................... 42
Figure 13: The facial expressions of a neutral face, anger, disgust and happiness (from left
to right) (Ekman, et al., 2002, pp. 45-93) ........................................................... 48
Figure 14: The facial expressions of fear (left) and surprise (right) (Ekman, et al., 2002, pp.
60-66). .............................................................................................................. 52
Figure 15: The facial expressions of the emotions sadness (left) and contempt (right)
(Ekman, et al., 2002, p. 51) ............................................................................... 54
Figure 16: With the camera mounted on a helmet, the facial expression of actor Jamie Bell
(Tintin) are capture with the help of the trackers on his face.
Available at:
(http://cdn.moviestillsdb.com/sm/00d97618c9745b074a6f488b09b1a369/theadventures-of-tintin-the-secret-of-the-unicorn.jpg) ............................................ 67
Figure 17: The facial expression of anger on the face of the Conductor in The Polar Express
(The Polar Express, 2004). ............................................................................... 69
Figure 18: Tintin's facial expression of the emotion anger (The Adventures of Tintin: The
Secret of the Unicorn, 2011). ............................................................................ 70
Figure 19: The character Billy (right) is smiling but does not show any signs of crow's feet
on the outside of this eye aperture (The Polar Express, 2004). ......................... 71
Figure 20: The facial expression of happiness on the face of Tintin (left). A close-up of his
left eye with minimal visible crow's feet on the outside of the eye aperture (The
Adventures of Tintin: The Secret of the Unicorn, 2011). .................................... 72
Figure 21: The Girl (left) and the Boy (right) show the facial expression of fear in The Polar
Express (The Polar Express, 2004). ................................................................. 73
Figure 22: Action Units 1, 2 and 5 are visible in Tintin's facial expression of fear. On the left,
the moment Tintin gets grabbed from behind and on the right the moment Tintin
is chased by a dog
(The Adventures of Tintin: The Secret of the Unicorn, 2011). ............................ 74
Figure 23: The Boy (right) shows the facial expression of surprise in The Polar Express
(The Polar Express, 2004). ............................................................................... 75
96 | P a g e
List of Figures
Figure 24: Tintin's facial expression of surprise (The Adventures of Tintin: The Secret of the
Unicorn, 2011). ................................................................................................. 76
Figure 25: A conversation between the Boy (left) and the Girl (right) in The Polar Express.
Especially the Girl has an unfocussed, wrongly placed gaze with a lack of
saccades (The Polar Express, 2004). ............................................................... 77
Figure 26: Screenshots from a mutual gaze scene in The Adventures of Tintin (The
Adventures of Tintin: The Secret of the Unicorn, 2011). .................................... 78
Figure 27: The pupil sizes of the Boy in The Polar Express (top row) and of Tintin (bottom
row) thoughout the movies (The Polar Express, 2004; The Adventures of Tintin:
The Secret of the Unicorn, 2011). ..................................................................... 79
Figure 28: The startle reflex of Tintin (The Adventures of Tintin: The Secret of the Unicorn,
2011). ............................................................................................................... 80
Figure 29: The uncanny valley graph by Mori with the characters of The Polar Express and
The Advenures of Tintin. (Tinwell, et al., 2011b, p. 328; The Polar Express, 2004;
The Adventures of Tintin: The Secret of the Unicorn, 2011)
(Modified by the author) .................................................................................... 82
97 | P a g e
List of Tables
List of Tables
Table 1:
An overview of the Action Units, involved muscles and in which facial
expressions they appear
(Ekman, et al., 2002) (Author’s table). ............................................................... 55
98 | P a g e