What Expression Could Be Found More Quickly? It
Depends on Facial Identities
Hang Zhang1,2, Yuming Xuan1, and Xiaolan Fu1
1 State Key Laboratory of Brain and Cognitive Science,
Institute of Psychology, Chinese Academy of Sciences,
Beijing 100101, China
{zhangh, xuanym, fuxl}@psych.ac.cn
2 Graduate School of the Chinese Academy of Sciences,
Beijing 100049, China
Abstract. Visual search task was used to explore the role of facial identity in the
processing of facial expression. Participants were asked to search for a happy or
sad face in a crowd of emotional face pictures. Expression search was more
quickly and accurate when all the faces in a display belonged to one identity than
two identities. This suggested the interference of identity variance on expression
recognition. At the same time the search speed for a certain expression also depended on the number of facial identities. When faces in a display belonged to
one identity, a sad face among happy faces could be found more quickly than a
happy face among sad faces; otherwise, when faces in a display belonged to two
identities, a happy face could be found more quickly than a sad face.
1 Introduction
Recognizing facial expressions is a fundamental and important ability people possess
to understand emotions of other people and to communicate with others efficiently. It
is also becoming a crucial element for the new generation of human-computer interfaces, such as vision-based or camera-based interfaces, which require the computer to
identify the user’s emotional status and change its behavior accordingly to provide
better support [11]. Understanding the mechanism of human’s recognition of facial
expressions could be useful and helpful to make computers achieve matchable or
acceptable performance.
Human faces contain a series of semantic information besides expression, such as
identity, gender, age and race, and a natural question is whether the extraction and
coding of certain kinds of information, especially, expression and identity, are independent of each other. The answer relies on two aspects. The first one is the physical
separability of facial information. A principle component analysis of the pixel intensities of facial expression pictures revealed that different information is responsible for
expression coding and identity coding [3]. For example, eye width and jaw drop were
important expression components, while face width was an important identity component. Another study suggested that high-frequency components of a face might be
important for the recognition of facial expression [14]. But these are not doubtless
evidences for the separation of expression and identity information, but only imply
that people may have different biases towards the two processes. The other aspect is
J. Tao, T. Tan, and R.W. Picard (Eds.): ACII 2005, LNCS 3784, pp. 195 – 201, 2005.
© Springer-Verlag Berlin Heidelberg 2005
196
H. Zhang, Y. Xuan, and X. Fu
whether there are independent systems in the brain respectively for the processing of
expression and identity. "Fusiform face area" (FFA), i.e., the middle lateral fusiform
gyrus, a region responding more to faces than to other object categories, which contributed to distinguishing new faces from old ones [13], might be the area for identity
processing. In contrast, expression processing was thought to relate with the superior
temporal sulcus and the amygdala [8]. But it was shown that the FFA was also sensitive to variations in expression even for identity processing or passive viewing of
faces [7]. Therefore, whether expression recognition and identity recognition are
parallel or interactive is still a question in debate.
Most behavioral evidences for the interaction between the processing of expression
and identity came from studies using Garner’s task [1][6]. Participants were asked to
do speeded classification according either to identity or to expression. When the classification task was on one dimension, the other dimension was correlated, constant, or
orthogonal. The reaction time difference between these three conditions for expression classification reflected the interference of identity variance on the processing of
expression. Is this effect robust enough to be replicated with other paradigms? The
present study aimed to use visual search paradigm to explore whether and how facial
identity would affect the processing of facial expression.
The basic logic of our study is as follows: If searching for an expression among
faces of more than one identity is more difficult than searching among faces of a single identity, it implies that automatic identity activation could exert an effect on expression processing.
In a visual search task, participants may be required to search for a target with a
predefined property or a property distinct from other items on a certain dimension.
Search is usually more efficient in the former condition than in the latter condition,
because a predetermined target allows people to expect what it looks like and have
more top-down guidance accordingly [16]. In our study the effect of target certainty
on search was also examined to evaluate to what extent expression search is modulated by top-down information.
In previous studies about facial expression perception, a famous but debatable finding was the search asymmetry between different expressions. Negative expressions
were observed to be found more promptly than positive expressions in some studies,
but this result failed to be replicated in other studies [10][12]. But items to be
searched in these experiments were mainly schematic faces or faces of one identity at
a time. In the present study, identity of faces was manipulated to investigate its possible effects on negative-face advantage in visual search.
2 Method
2.1 Participants
Eleven paid undergraduate students took part in our experiment. Among them, eight
were males, three were females, and all had normal or corrected-to-normal vision.
What Expression Could Be Found More Quickly? It Depends on Facial Identities
197
2.2 Stimuli and Apparatus
Stimuli were black-and-white pictures of two female faces (A & B, Fig. 1) with happy
or sad expressions. Seven independent raters categorized expressions of about 20 digital photographs into happiness, anger, fear, sadness, disgust, surprise or neutral expression. The four stimuli pictures selected from the photographs had happy or sad expressions agreed by at least half of the raters, and had no irrelevant salient features, and
were polished with Photoshop 7.0 to diminish their differences in image attributes.
Fig. 1. Pictures of faces used in the experiment
Stimuli were presented on black background in the center of a 17 inch monitor.
Matlab PsychToolbox [2] was used to control the experiment and record responses.
Participants were seated about 65 cm from the monitor.
Stimuli were composed of several faces, each of which extended 1.9 deg in width
and 2.6 deg in height. They scattered randomly in an imaginary four-by-three grid
covering 9.5 × 9.5 deg in the center of the screen. Each unit of the grid was 2.4 × 3.2
deg and an individual face could appear anywhere in its display unit on a random
basis.
2.3 Procedure
Each trial started with a centered "+" fixation point for 1 s, and after it disappeared,
four, six or eight happy or sad faces was displayed until a response was made or for a
maximum duration of 3 s. The interval between trials varied from 1 s to 1.8 s.
There were three kinds of tasks. In Searching-for-Happy-Faces (SHF) blocks, participants were told that the displays were a happy face among a crowd of sad faces, or
all sad faces, and they had to decide the presence of happy face. In Searching-for-SadFaces (SSF) blocks, participants were told that the displays were a sad face among a
crowd of happy faces, or all happy faces, and they had to decide the presence of sad
face. In Searching-for-Different-Faces (SDF) blocks, participants were told that there
might be a happy face among a crowd of sad faces, or all sad faces, or a sad face
among a crowd of happy faces, or all happy faces, and they had to decide if there was
a face of different expression than others. The numbers of target-present trials and
target-absent trials were equal. Half of the participants were instructed to press the "z"
key with left index finger for "yes" response and "/" with right index finger for "no"
response, and the other half were told the reverse. They were asked to respond as
promptly and accurately as possible. Accuracy and reaction time were recorded for
experimental trials.
198
H. Zhang, Y. Xuan, and X. Fu
2.4 Design
In each task, faces could belong to female A only, or female B only, or half faces
were A and the other were B. Multiplied with three kinds of tasks, there were nine
conditions altogether. The displays in each condition had the same possibility to consist of four, six, or eight items.
Each of blocks of SHF or SSF with A’s or B’s faces had 6 practice and 60 experimental trials. Blocks of SHF or SSF with A and B’s faces, and blocks of SDF with
A’s or B’s faces had 12 practice and 120 experimental trials each. The condition of
SDF in A and B’s faces was split into two blocks, both had 12 practice and 120 experimental trials. Participants were tested through all the ten blocks in one of four
counterbalance orders.
There were five independent variables: target state (present vs. absent), number of
items (four, six, or eight), target certainty (one certain expression vs. one of the two
expressions), target expression (happy vs. sad), and number of identities (one identity
vs. two identities).
3 Results
All the participants completed the experiment and the lowest accuracy was 0.89. Accuracy and reaction time for correct responses were analyzed respectively.
3.1 Reaction Time
A repeated-measures ANOVA of reaction time was conducted on the five independent variables. The main effects of target state, number of items, target certainty, and
number of identities were significant. As expected, responses were faster for targetpresent conditions (1476 ms) than for target-absent conditions (1794 ms), F(1, 10) =
227.3, p < 0.001. There were significant differences among four-item conditions
(1434 ms), six-item conditions (1674 ms), and eight-item conditions (1828 ms),
F(2, 9) = 99.3, p < 0.001. Effects of target certainty and number of identities on
searching performance were shown in table 1. Searching was faster when target was a
certain expression (1517 ms) than when target was an uncertain distinct expression
(1770 ms), F(1, 10) = 62.8, p < 0.001. Expression search was done more quickly in
one person’s face crowds (1519 ms) than in two persons’ face crowds (1769 ms), F(2,
9) = 99.3, p < 0.001, suggesting that the irrelevant dimension, facial identity, could
influence expression processing.
Table 1. Mean reaction time (ms) and percentage correct (%) under two conditions of number
of identities (one vs. two) by two conditions of target certainty (certain vs. uncertain)
Target Certainty
Certain
Uncertain
Reaction Time
Number of Identities
One
Two
1401
1637
1640
1908
Percentage Correct
Number of Identities
One
Two
95.45
91.86
93.11
87.50
What Expression Could Be Found More Quickly? It Depends on Facial Identities
199
The interaction between target state and number of items was significant, F(2, 9) =
54.6, p < 0.001,which indicated search slope difference under target-present and target-absent conditions. In the four conditions shown in table 1, target-present slopes
ranged from 55 to 63 ms/item, target-absent slopes from 123 to 152 ms/item, and
slope ratios (target-absent to target-present) from 2.25 to 2.48, suggesting patterns of
typical serial and self-terminating search [15].
Fig. 2. Slopes of reaction time for target-present conditions (a) and target-absent conditions (b)
An interesting result was the three-way interaction of number of items by target
expression by number of identities, F(2, 9) = 8.52, p = 0.008. Since "no" response
might involve some special strategy unlike "yes" response, separate analyses were
made on slopes under target-present and target-absent conditions. Main results were
shown in Fig. 2. When there was a target face in the display, the interaction between
number of identities and target expression was significant, F(1, 10) = 8.99, p = 0.013.
A sad face among happy faces would be more quickly found than a happy face among
sad faces if all faces in a display belonged to one person; but if two persons’ faces
were involved simultaneously in a search, a happy face would be found more quickly.
When target was absent, there was no statistically significant interaction, but the trend
was similar.
3.2 Percentage Correct
A repeated-measures ANOVA of percentage correct was conducted on the five independent variables. The main effects of target state, number of items, target certainty,
and number of identities were significant, and the directions of difference was consistent with those of reaction time data, i.e., the conditions under which participants had
faster responses were also the conditions under which more accurate responses were
made, thus excluding the possibility of speed-accuracy tradeoff. Main results of percentage correct were given in table 1. There was also no interaction between target
certainty and number of identities.
200
H. Zhang, Y. Xuan, and X. Fu
4 Discussion
Consistent with earlier studies [10][12], expression search in our experiment is serial
and self-terminating, which means participants indeed searched for expressions and so
the accompanying effects can’t be simply attributed to some salient feature irrelevant
to facial expression. As expected, the reaction time and accuracy data show that
search is faster and more accurate when the identity of faces keeps constant than
when it varies, indicating that identity information can’t be fully neglected in the
coding of facial expression. The present result from visual search adds positive evidences to the hypothesis that the processing of expression may be interfered with the
processing of identity.
Target certainty also has effects on the efficiency of expression search. Knowing
the target is a happy or sad face facilitates the search, compared with the conditions in
which target expression is uncertain and may vary from trial to trial. The number of
identities and target certainty has no interaction, implying the effects of identities on
expression recognition are more likely mediated by a bottom-up mechanism but not
top-down expectations.
The new finding of the present study is that what expressions could be easily found
depends on the number of identities involved in the searching. When all the faces are
of a single identity, searching for a sad face in a crowd of happy faces is faster than
searching for a happy face in a crowd of sad faces. This result is consistent with previous studies demonstrating faces with negative expressions capture more attention
[4][5]. But this is only one side of the coin. When two identities are involved, the
happy face among sad faces is more promptly found than the sad face among happy
faces. There was evidence that the expression of happiness is more consistent between
different people than the expression of sadness is [9]. Supposed this is the case in our
study, in a display involved two identities, happy faces should be more easily identified because of their higher similarity. When the effect of high similarity of happiness
expression outweighs the capturing-attention effect of negative expressions, a reverse
result appears.
5 Conclusion
In this study visual search task was used to investigate the possible influence of facial
identity on facial expression. Switching from faces of one person to faces of another
person has cost on the processing of expression. And the number of identities for
expression search affects which expression, happiness or sadness, could be searched
faster.
Acknowledgements
This research was supported in part by grants from 973 Program of Chinese Ministry
of Science and Technology (2002CB312103), from the National Natural Science
Foundation of China (60433030 and 30270466), and from the Chinese Academy of
Sciences (0302037).
What Expression Could Be Found More Quickly? It Depends on Facial Identities
201
References
1. Baudouin, J.-Y., Martin, F., Tiberghien, G., Verlut, I., Franck, N.: Selective Attention to
Facial Emotion and Identity in Schizophrenia. Neuropsychologia. 40 (2002) 503-511
2. Brainard, D.H.: The Psychophysics Toolbox. Spat. Vis. 10 (1997) 433–436
3. Calder, A.J., Burton, A.M., Miller, P., Young, A.W., Akamatsu, S.: A Principal Component Analysis of Facial Expressions. Vision Res. 41 (2001) 1179-1208
4. Eastwood, J.D., Smilek, D., Merikle, P.: Negative Facial Expression Captures Attention
and Disrupts Performance. Perception & Psychophysics. 65 (2003) 352-358
5. Fenske, M.J., Eastwood, J.D.: Modulation of Focused Attention by Faces Expressing
Emotion: Evidence From Flanker Tasks. Emotion. 3 (2003) 327-343
6. Ganel, T., Goshen-Gottstein, Y.: Effects of Familiarity on the Perceptual Integrality of the
Identity and Expression of Faces: The Parallel-Route Hypothesis Revisited. Journal of
Experimental Psychology: Human Perception and Performance. 30 (2004) 583-597
7. Ganel, T., Valyear, K.F., Goshen-Gottstein, Y., Goodale, M.A.: The Involvement of the
"Fusiform Face Area” in Processing Facial Expression. Neuropsychologia. In Press
8. Haxby, J.V., Hoffman, E.A, Gobbini, M.I.: The Distributed Human Neural System for
Face Perception. Trends Cogn. Sci. 4 (2000) 223-233
9. Kohler, C.G., Turner, T., Stolar, N.M., Bilker, W.B., Brensinger, C.M., Gur, R.E., et al.:
Differences in Facial Expressions of Four Universal Emotions. Psychiatry Research. 128
(2004) 235-244
10. Nothdurft, H.C.: Faces and Facial Expressions Do Not Pop Out. Perception. 22 (1993)
1287-1298
11. Porta, M.: Vision-Based User Interfaces: Methods and Applications. Int. J. HumanComputer Studies. 57 (2002) 27-73
12. Purcell, D.G., Stewart, A.L., Skov, R. B.: It Takes a Confounded Face to Pop Out of a
Crowd. Perception. 25 (1996) 1091-1108
13. Rossion, B., Schiltz, C., Crommelinck, M.: The Functionally Defined Right Occipital and
Fusiform "Face Areas" Discriminate Novel from Visually Familiar Faces. NeuroImage. 19
(2003) 877-883
14. Tsurusawa, R., Goto, Y., Mitsudome, A., Tobimatsu, S.: An Important Role for Highspatial Frequencies in Recognition of Facial Expressions. Int. Congress Series. 1278
(2005) 53-56.
15. Wolfe, J.M.: What Can 1 Million Trials Tell Us About Visual Search? Psychol. Sci. 9
(1998) 33-39
16. Wolfe, J.M., Butcher, S.J., Lee, C., Hyle, M.: Changing Your Mind: On the Contributions
of Top-Down and Bottom-Up Guidance in Visual Search for Feature Singletons. Journal
of Experimental Psychology: Human Perception and Performance. 29 (2003) 483-502