DISCUSSION PAPER SERIES
IZA DP No. 2764
Implicit Discrimination in Hiring: Real World Evidence
Dan-Olof Rooth
April 2007
Forschungsinstitut
zur Zukunft der Arbeit
Institute for the Study
of Labor
Implicit Discrimination in Hiring:
Real World Evidence
Dan-Olof Rooth
Kalmar University,
CReAM and IZA
Discussion Paper No. 2764
April 2007
IZA
P.O. Box 7240
53072 Bonn
Germany
Phone: +49-228-3894-0
Fax: +49-228-3894-180
E-mail: [email protected]
Any opinions expressed here are those of the author(s) and not those of the institute. Research
disseminated by IZA may include views on policy, but the institute itself takes no institutional policy
positions.
The Institute for the Study of Labor (IZA) in Bonn is a local and virtual international research center
and a place of communication between science, politics and business. IZA is an independent nonprofit
company supported by Deutsche Post World Net. The center is associated with the University of Bonn
and offers a stimulating research environment through its research networks, research support, and
visitors and doctoral programs. IZA engages in (i) original and internationally competitive research in
all fields of labor economics, (ii) development of policy concepts, and (iii) dissemination of research
results and concepts to the interested public.
IZA Discussion Papers often represent preliminary work and are circulated to encourage discussion.
Citation of such a paper should account for its provisional character. A revised version may be
available directly from the author.
IZA Discussion Paper No. 2764
April 2007
ABSTRACT
Implicit Discrimination in Hiring: Real World Evidence*
This is the first study providing evidence of a new form of discrimination, implicit
discrimination, acting in real economic life. In a two-stage field experiment we first measure
the difference in callbacks for interview for applicants with Arab/Muslim sounding names
compared to applicants with Swedish sounding names using the correspondence testing
methodology. In the second stage of the experiment we measure, for a sample of the
recruiters involved, their explicit and implicit attitudes/performance stereotypes by the means
of explicit questions and the implicit association test (IAT). We find (i) only weak correlations
between explicit attitudes/performance stereotypes and implicit performance stereotypes but
(ii) a strong and statistically significant negative correlation between the implicit performance
stereotypes and the callback rate for an interview for applicants with Arab/Muslim sounding
names, but not for applicants with Swedish sounding names. These results indicate that
implicit discrimination acts differently compared to explicit discrimination and that it is an
important determinant of the hiring process.
JEL Classification:
Keywords:
J64, J71
implicit attitudes and stereotypes, discrimination, situation testing,
exit from unemployment
Corresponding author:
Dan-Olof Rooth
Kalmar University College
391 82 Kalmar
Sweden
E-mail: [email protected]
*
I thank Jens Agerström, Per Johansson, Olof Åslund, participants at seminars in Kalmar and at IFAU
(Uppsala) fpr valuable comments and helpful suggestions. Magnus Carlsson, Rickard Carlsson, Klara
Johansson, and Terese Johansson provided excellent research assistance. A research grant from the
Swedish Council for Working Life and Social Research and another research grant from the Institute
for Labour Market Policy Evaluation (IFAU) are gratefully acknowledged.
*
+
3) /
*
38
*
+
3
/
1
2
*
3
;
<
1
=
+
3 >??(23
*
+
+
@
A
@
A
3>
B
@=
A
;
*
*
*
+
1
;
4
C
>??D23 4
3
*
1
3 /
2
$
*
1>??D2
3
+
3
E
F
)
5
>
5
<
C
"
E
C
1)GG'2
+
+
3
1>??)2
+ 1)GGG23
1
<
E
!
3
1>??(2
1
E
>??>23 4
+
E
B
5
"
H
<#
3
+
3 !
1>??(2
) DD> E
5
+
3
4
1
"42
)GG?
;
I
1)GGJ23K
"
,
1>??(2
*
.
"4
5
"
<#
3 4
1>??D2
"
1
2
<#
3
*
3'
"
)GK
3 "
"4
"
<#
5
+
3=
3
4
3 5
+
K
"
*
;
<
"4
*
3 6
;
'
>
*
+
3
E
3
E
3
2
35
K
<
'
3 5
D
+
1
"4
3 "
2
*
*
+
+
H
*
3 4
3
2. A model of implicit and explicit discrimination in hiring
3 5
"
<#
L
3
3
"
<#
5
C
1>??D2
3
+
<
+
3
*
3
Assume that the probability to receive a callback for a job interview and ethnic
discrimination have the following relationship for individual i belonging to ethnic group
j:
(1)
- 1
= 12 = β + δ ; j=Arab/Muslim or native Swede.
where X is a vector of characteristics in the job application, which by construction of the
experiment is the same for both ethnic groups,
is the return to those characteristics for
3
ethnic group j, while
is a measure of the degree of ethnic discrimination against group
j, expected to be zero against native Swedes.
In economics several forms of ethnic discrimination have been modelled (see the
survey by Altonji and Blank, 1999). For our purposes we focus on two of those,
preference based discrimination and statistical discrimination, both of which can be
regarded as explicit forms of discrimination.5 As mentioned in the introduction the aim of
this study is to test whether also implicit forms of discrimination are important in the
hiring situation. Hence, we can rewrite Equation 1 as:
(2)
= 12
- 1
=
β +δ
+δ
+δ
where the three discrimination terms express explicit preference discrimination, explicit
statistical discrimination and implicit discrimination, respectively.
One might wonder if it is possible to divide also the implicit discrimination measure
into preferences/attitudes and stereotypes as is done for the explicit discrimination
measure. 6
Agerström et al (2007)
implicit
attitude score and the implicit stereotype score have a correlation of above 0.5 indicating
that they measure, if not exactly the same, but then at least highly related constructs.&
H
3
4
*
"
#
D
>??D
/
>??&
E
/
35
+
1)GGJ2
3
&
4
"4
3
"4
"4
*
@
"4
A
@
H
A3
+
+
3
4
5
+
+
3 4
I
+
<
+
3 4
1
<
2
+
+
3
4
+
"
1
7
#
>???
5
8+
>??K23 5
*
/
"
3
5
<#
34
H
5
"
<#
5
3
4
"43
4
*
pplicants had identical human capital within occupations and
34
were on average 25-30 years old, had two to four years of work experience in the same
occupation as the job applied for and had obtained their education in the same type of
school, but at different locations. Through the schooling information it is signalled that
the Arab/Muslim applicant is born in Sweden, but has a non-native Swedish name.
Further, the application consisted of a quite general biography on the first page and a
detailed CV of education and work experience on the second page.
!
1>??(2
5
8
"
) DD> E
4
")
*3
) ?K?
D>>
3 C
>KG
5
"
>)(
<#
&&
3 4
+
5
"
<#
3
5
!" #
4
"4
"
>??&
+
3
7
%
>??(3
"4
+
1>??(2
;
"4
1 M?3D23
4
;
*
*
*
E
34
E
34
*
*
+
36
E
3
3 6
E
+
@
A
H
<
"
<#
3
+
<
K?? 58
KJ
1
*
KK
23 4
*
35
3 "
<
4
(>G
)3 /
KG>
3
*
)GK
+
"43
<
5
<"
<
<
2
*
@
@
1
A3 "
A
3 4
6
*
3(
4
$%
)
&
5
+
+
"4
3
*
3=
<
+
3 6
*
1
E
2
3
3 6
N
+
"4 1
F
2
3
3J
+
"4
3
8
3
F
"4
"
,
1
1>??(23 8
2
4
">
(
/
*3 4
*
@
E
J(<D3 5
4
4
J
#
"43
+
D
A
+
J(
"43 4
<
H
3
3
3
H
7
3
@
A
@
A
5
1
+
23
#
4
H
+
3 /
*
Feeling thermometer they were asked to rate their positive or
negative feelings on a ten-point scale (1 = very negative feelings, 10 = very positive
feelings) toward Arab-Muslim minority men and native Swedish men, and then a
difference between the two scales was calculated. This question is identical to the one
used by Greenwald et al. (1998) and Nosek et al. (2005).
Second, in the Hiring preference rating participants had to choose which group they
prefer when hiring people. Rather than being directly related to the IAT, hiring
preferences are supposed to measure actual discrimination. The employers/recruiters had
to choose one of the following five alternatives: “When hiring staff, I strongly prefer
Arab-Muslim men (in Sweden) to native Swedish men”, “When hiring staff, I moderately
prefer Arab-Muslim men (in Sweden) to native Swedish men”, “When hiring staff, I
prefer Arab-Muslim men (in Sweden) and native Swedish men equally much”, “When
hiring staff, I moderately prefer native Swedish men to Arab-Muslim men (in Sweden)”,
and “When hiring staff, I strongly prefer native Swedish men to Arab-Muslim men (in
Sweden)”. The participants’ responses were coded from -2 to +2, with 0 as an
intermediate point reflecting no preference when hiring staff.
Third, in the Performance stereotype rating participants choose which of the two
groups in question they consider to be more productive at work. The response alternatives
were “Arab-Muslim men (in Sweden) are much more productive at work than native
Swedish men”,” Arab-Muslim men (in Sweden) are slightly more productive at work
than native Swedish men”, “Arab-Muslim men (in Sweden) and Swedish men are equally
productive at work”, “Swedish men are slightly more productive at work than ArabMuslim men”, “native Swedish men are much more productive at work than ArabMuslim men”. Again, the participants’ responses were coded from -2 to +2, with 0 as an
8
intermediate point reflecting neutrality. 4
H
)
5
@
A
3
*
1D'O2
1'DO2
E
E
5
1((
4
H
"
*
<#
2
>3
"
*
3 5
"
<#
H
3
4
' "
>
!"
4
"
4
1 "42
3
s a computer
based test designed to specifically measure individual differences in relative associations
between two concepts. It was first introduced by Greenwald et al. (1998) and has since
become a widely used measure, particularly in sensitive areas such as attitudes and
stereotypes toward social groups.9
The test is illustrated by the Arab-Muslim performance stereotype IAT found in
"
,
. In this computer
3 1>??(2
based version participants first classify, as fast as possible, Swedish and Arab/Muslim
sounding names appearing in the middle of the screen according to the (target) category
to which they belong, “Arab-Muslim men” or “Swedish men”. In the next part of the IAT
the participant now instead classifies words that are found being associated with high and
low work productivity. These include words such as “lazy”, “slow”, “efficient” and
“hard-working”. In the third step names and high/low work productivity words appear at
G
5
9<<
3
3
"43
9
random. The intuitive idea is that it will be easier, and hence, go faster, to classify names
and words that are compatible than those that are incompatible. The IAT measures every
latency in response to the presented stimulus. For example, when two categories are
“easily” associated in terms of their nominal features (Arab names + low work
productivity and Swedish names + high work productivity) the participant classifies the
stimuli much faster and with fewer errors than when they are not associated. A total of
sixty stimuli are presented for the compatible and compatible part, respectively. The
difference in response latencies, or rather a recalculation of this difference called
Greenwald’s D (as opposed to Cohen’s d), between the compatible and incompatible
parts is known as the IAT effect or the IAT score.
I
1>??K >??&2
"4
?3)D
*
?3)D ?3KD
"4
?3KD ?3&?
?3&?
"
3)? 4
"4
)GK
E
@ F A
@
A
"
3
1
/
<#
"4
?3)D
)23
"4
"4
F
?3'
3 'G
?3'
"43))
( )
4
*
3 3
"4
*
*
5
"
+
<#
E
4
K3 6
?3)D
)? 4
?3K)
!
;
3
))
4
?3'
H
"
<#
3 C
3
10
"4
@
*
A
*
3 4
*
?3K
*
3
"
#
"4
*
4
4
*
3)>
K
F
*
*
3 3
5
>3 4
< *
+
5
"
<#
1
23 4
+
5
+
4
"
'" 'C '!
"
<#
'
3
F
3
<#
'"
@
< *
"
4
4
+
A
'C
+
<#
< *
3
4
'!
F
*
*
4
'
+
@
3 /
5
A
< *
3
"
*
3
4
'"
+
)GK
)>
*
1#
"23
*
1 P?3>2
"
,
1>??(23
11
!
1>??(2
*
+
*
>?
3 4
@
A
+
3)K
#
C
8
3 4
#
3
C
+
#
!
3
#
8
8H
>
*
>
<
3 4
?3?K
#
F
!
83
3 /
*
1
#
!
23 4
<
1
<
2
"
/
K
<#
3)'
*
3)D I
*
"
<#
3 /
*
*
1
23
)K
*
*
*
@
A
3
)'
4
"4
I
1>??D2
)D
?3KD
?3K'
?3&
"4
<
3
*
3
12
&
)?
34
3)&
H
36
1 2
3
H
@
A
< *
5
+
"
<#
3
6
#
!
"
3
+
*
3 4
1
"
<#
#
823
5
"
/
!
<#
3
*
3
*
F
3
*
+
#
,
Non-linear effects of the implicit discrimination measure have so far been neglected.
When introducing a cubic of the standardized IAT score into the regression model C, for
the full as well as the conditional sample, we find that the correlation is much stronger for
values of the IAT effect above 0.6, i.e. for strong negative attitudes, see Figure 2 and 3. A
related issue is whether there exists a significant interaction effect between the implicit
and explicit discrimination measures. However, such an interaction effect is not easily
)&
"
*
+
3
13
interpreted with the “continuous” explicit and implicit discrimination measures.
Therefore we have constructed binary explicit and implicit “equivalents”.
4
4
'"
< *
4
'C3 4
"4
34
?3'
*
F
<
5
"
<#
"
F
34
<#
<
3
"
"
#
<#
&(
E
34
4
*
'"
3
6
*
#
/
4
'!3 4
3 4
<
*
<
*
+
*
<
<
3
4
*
1
#
23 /
*
*
"
<#
3 4
+
< *
))
14
*
"
<#
34
6
H
4
3
'
+
5
< *
#
3 /
8
3 "
*
+
34
F
*
3 4
*
*
3 /
+
)(
*
1
4
-
23
'" 'C '!
.
'
&
F
1
2
<
1 2
3 4
*
)GK
"
5
+
KJ&
36
<
*
3"
4
)3
4
*
<
;
34
4
D 1#
*
"2
"
)(
4
*
*
"
#
3
15
5
1#
C
3"
!2
+
*
36
1#
2
+
3
*
*
H
+
3
*
3 4
>(&
*
3
4
/
D
#
Since the explicit discrimination measures are collected by interviewing recruiters it is
likely that some are reluctant to reveal their true attitudes, for instance, for political
correctness reasons. Such measurement errors will bias the estimates of the explicit
discrimination measures. Also, the empirical analysis reveals that the estimates for the
explicit measures are unstable across specifications, indicating measurement problems. A
well known strategy is then to find an instrumental variable and estimate IV 2SLS, which
helps in deleting the measurement error. This is achieved by having a second measure of
explicit preferences that is not perfectly correlated with the first one. Such an
instrumental variable exists for the explicit attitude as well as for the explicit stereotype
measure. In the case of the explicit attitude measures there are two questions being asked.
Hence, the feeling thermometer will be used as an instrument for the hiring preference.
For the explicit stereotype measure the IAT measure will be used as an instrument.
6
Q >575
*
4
&3 I
3
*
4
'
D
*
3
16
(
4
+
*
*
36
"4
"
<
<#
34
E
#
"
<#
F
5
3 "
*
*
E
"
<#
3 4
*
E
"
+ 3
<#
.
1>??D2
"
<#
3 6
*
E
3
E
@
A
+
N
34
"
*
*
+
"4
3 /
*
3
*
*
35
+
3
E
35
"4
3
8
+
3 #
17
strategy as ours are needed before fully accepting implicit discrimination as a new form
of labor market discrimination.
18
)
"
,
%3 !
6 +-
"
"
E %3
"
K)'KBK>DG3
.
C
3
+
1
!
3 1>??(2 @8
5
5
3 1)GGG2
2
B8
A
3
I
7
38
#
+ 3
3"
3
3 1>??D2 ”Shifts in Attitudes and Labor Market Discrimination:
3
Swedish Experiences after 9-11”, Journal of Population Economics 18 (4), pp. 602-629.
C
#3 !
9
3
#
53 1>??D2 @=
A "
8
"
GD 1>2
3 G'
GJ3
!
5
#3
7
!
5
#3
7
I
#
#
8
+
3 1>??&2 @8
0
8*
8
+
3 1>??(2 @8
0
8*
"3 # I
I
"3 =
"
I
"3 =
"
0
3
5
94
A $"
A
F %3 1)GGJ23 #
"
! )'&'B)'J?3
5
+
%3
I8 4
7
3 C
6
+
$
3
0
A
H
C
7
%3 1)GGJ2 @
3 )?) )&3
+
4
=3 1>??&2 @!
3
C
%3 1)GG'2 @C
# $ % J' 1D29))(' ))G'3
)>
3
+ C3
C
E # 3 1>??K2 @0
4 9 3"
5
"
JD
3 )G( >)&
+ C3
4 9!
3
-R>>J) $"3
%
#
A3
+ A "
A
-
S3 1>??)2 @
A "
+ C3
I
9 Q3 6
5
F =3 18 2
3 DG )?>3 =
S +9 I
$ &
=
# $ %
E #3 =
+
" 1>??(2 @0
15 / 2
'
"
3
5+
97
33
#
A
C3
(
19
7
C
"3 1>???2 )
1
+ 2 =
&
+,
4
3
Nosek, B. (2005) ”Moderators of the relationship between implicit and explicit
evaluation”, Journal of Experimental Psychology, vol 134, pp. 565-584.
Nosek, B., Greenwald, A. and Banaji, M. (2007) ”The Implicit Association test at Age 7:
A Methodological and Conceptual Review”, %3 C
18 32 "
$
-3 "3
A *
%3 1>??>2 @/
))>
3
$
"
-
3 >&D >G>3 -
%3
#
G?1K2
8+
-
3
*
3 /'J? /D)J3
+
%3 1>??K2 T0
5
3 8
+
)& 1'2
3 (J(BJ)'3
-3 1>??D2 @8
!
3 DDK D&>3
T
C
A
94
%
"
20
0
/
)3 4
"4
5
1
2
"
3
40
30
20
10
Mean = 0,3841
Std. Dev. = 0,34083
N = 193
0
-0,50
0,00
0,50
1,00
IAT Effect (D measure)
/
>3 =
"4
3/
)GK
3
Linear
Cubic
0,0
-0,1
-0,2
-0,3
-0,4
.00
.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Standardised IAT effect
+
94
1 "4
F
3"
M?3&23 4
?3?>&
H
3
F
"4
>
+
?3?K)
"4
H
4
'"
34
21
/
K3 =
"4
3!
)K&
3
Linear
0,00
Cubic
-0,20
-0,40
-0,60
-0,80
-1,00
.00
.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Standardised IAT effect
+
94
1 "4
F
3"
M?3&23 4
?3??)
H
3
F
"4
>
+
?3?D)
"4
H
4
'"
34
22
" 4
)3 -
*
&
3
!"
1
-
15 2
1
5
"
C
=
"
+
4
>?'
DG
>>)
) ?)'
) 'GJ
/
O2
D& 1K?2
)J 1K)2
&> 1K>2
D( 1)G2
)GK 1>&2
)J'
DG
)G>
>G'
(>G
*
K3& 1>3?2
>3G 1)3K2
K3> 1)3D2
)(3K 1)(3'2
<
1) 'GJ2
5
1
@
A
+
"4
3 /
@
@
A3
4
>3 8*
3 )GK
!
5
;
23 5
34
5
3
3
1K?O2
A
3
"
"
"
!
!
+
94
4
)
3
23
4
*
K3 !
*
"
3 )GK
#
$ %&
' ()*
$ %- !.
$ %
- !.
$ %1 !.
/ !
$"%
1 !.
/ !
$ %2 /
!
'
$ %
2 /
!
Note: * p<0.05. 4
#
"4
3
"
+ ",
$/
$/
0 %
0 %
'
$/
4
0 %
)
+
+
+
+
+
+
",
,
,
,
+ ,
+ ,
+ ,
+
+
,
+ ,
+ ,
+ ,
+ ,
+
+
+ ,
+ ,
+ ,
+ ,
+ ",
+ ,
+
+
+
+
,
,
,
,
+ ,
+ ,
+
+
+
+ ",
+ ,
+ ,
+
+
+ ,
+ ,
+ ,
+ ,
3
24
Table 4A. The correlation between the callback rate for interview and the implicit and explicit attitude and stereotype measures.
Percentage points. Full data of 193 observations.
3
. !
//
!
#
4 5
!
!. ! ) #
3 7
)
& !7 7
-
+
$ +
%
/
/
!
).
+
$ +
%
'
+ ,,,
$ + %
+
$ + %
+ "
$ + %
/
#
/
:
< .
9 /
+
,
%
'
! /
5
7 //
2 1
2
4
0
9
;
7
91
+
+
$ +
,
%
+
$ +
,
%
+
$ +
;
9
+ ",,,
$ + " %
+ ",,
$ + %
+
$ + %
;
;
+ ,,
$ + " %
%
+ "
$ + "%
+
$ +
+
$ +
+
$ +
6
,
%
+ ,,
$ + " %
+ ,,
$ + " %
+ ,,,
$ + " %
+ ,,
$ + %
+
$ + %
+ ,,,
$ + " %
+ ",,
$ + %
+ "
$ + %
;
;
+
$ + " %
%
+ "
$ + %
+
$ +
+
$ +
%
+ ,,,
$ + %
+
$ + %
+
$ + %
;
;
9
;
+ ,,,
$ + %
+
$ + %
+ ""
$ + %
+ ,,,
$ + %
+ "
$ + %
+
$ + %
+ ,,,
$ + %
+ "
$ + %
+
$ + "%
;
9
;
;
;
;
%
%
;
;
38
54"4" G3 4
&
)?
34
8
%
1 2
)GK
!
+
$ +
!
3
9
+
$ +
6
!.
1 !.
2
7
8
3 7
)
!
6 !7
3"
4
*
)
*
#
C3
34
)
D
)K&
?3?J
?3D'
3
25
4
'C3 4
3-
+
)GK
3/
*
3
3
. !
//
!
#
4 5
!
!. ! ) #
!
3 7
)
$/
-
0 %
!.
/
/
!
$/
!
9
:
< .
9 /
+
/
'
! /
7 //
$/
0 %
+ ,,,
$ + %
+
$ + %
+
$ + "%
91
+
2 1
2
4
0
9
;
7
;
9
+ ",
$ + %
+ "
$ + %
+
$ +
+
$ +
+ ,
$ + "%
%
+
$ +
+ ,,,
$ + " %
+ ,,
$ + %
+ ,
$ + %
,
%
+
$ + "%
+ ,,,
$ + " %
+ ,,
$ + %
+ ,
$ + %
+ ,,,
$ + " %
+ ,,
$ + %
+ ,
$ + %
;
;
%
+
$ +
%
+ ",,,
$ + %
+
$ + %
+ "
$ + %
;
;
1 2
9
;
+ ",
$ + %
+ ,,,
$ + %
+
$ + %
+
$ + %
;
9
%
+ ,,,
$ + %
+ "
$ + %
+
$ + %
;
;
;
;
+ "
$ + %
+ "
$ + %
+ ,,,
$ + %
+
$ + %
+
$ + %
;
;
38
3"
?3D'
,,
%
6
+ "
$ + %
+
$ +
+
$ +
;
;
54"4" G3 4
?3?J
,
%
0 %
'
5
+
$ +
%
8
+ ",
$ + %
/
#
+
$ +
6
0 %
3
).
8
+
$ + " %
$/
1 !.
2
+
3 7
)
*
4
)
34
34
3
26
Table 4C. Interaction effects. Percentage points.
3
. !
//
!
#
4 5
!
!.
3 7
6
-
=
&5 7
6 !7
3 7
6
1
!
!
-
=
1
!"
$/
-
0 %
!.
1 !.
2
+
$/
/
/
+
$ +
!
$/
!
0 %
+
+
$ +
,
%
+
$ +
%
'
$/
+
$ +
7 //
;
+
$ +
%
;
+
$ +
+ ,
$ + "%
%
+
$ +
,,
%
+
$ +
,
%
+ ,,
$ + " %
+
$ +
%
0 %
+
$ +
! /
+ "
$ + %
0 %
#
:
9
,
%
%
+
$ +
+
$ +
;
;
A
3"
4
+
$ +
%
+
$ +
+
$ +
%
+
$ +
%
;
+
$ +
%
;
,
%
%
;
%
%
;
/
91
+
2 1
2
1 2
38
@
54"4" G3 4
D
)
3
)
*
'
J
34
)GK
)K&
?3>'
?3JJ
34
3
27
4
' 34
3=
+
5
*
3-
3
3
. !
//
!
#
4 5
!
!.
-
&5 7
6 !7
!
!
3 7
3 7
$"
& !7 7
-
7
!.
/
%
+
$ +
+ "
$ + %
%
+ " ,,
$ + %
1 !.
2
+
$ +
/
!
!
'
+
$ + "%
+
$ +
+
$ +
+
$ +
+
$ +
%
+
$ +
+
$ +
%
+
$ + "%
%
%
%
%
!"
$/
-
:
9
+
0 %
!.
$/
1 !.
/
2
!
/
! /
+ "
$ + %
!
$/
0 %
+
$ +
$/
0 %
7 //
+
$ +
;
;
+
$ +
,,
%
0 %
'
+
$ + "%
%
;
%
+
$ +
%
+ "
$ + %
+
$ + %
+ "
$ + %
;
+
$ +
;
;
;
%
+
$ +
+
$ +
%
%
;
/
+
91
2 1 2
+
+
3"
4
1 2
38
@
*
4
)
3
)
A
*
34
'
)K&
?3?J
?3D'
54"4" G3
)GK
D
J
34
3
28
4
D3
+
N-
*
3 7
)
) # >3
!
(
@
9
+
5
8*
6 !7
8
+ ,,
$ + " %
+ "
$ + %
3 7
)
6
+
$ +
%
+
$ +
%
+ ,,,
$ + " %
!
8
+
$ +
6
,
%
+
$ +
,,,
%
+
$ +
? ) # ,(
9
;
9
;
9
;
9
;
? ) #,
9
9
;
;
9
9
;
;
! 7 /
4 / !
/7
#
*
3
! 7 // !
5 #'
>
!
!
!
"
"
%
"
/
9 1
2 1
2
1 2
3 4
34
X23 4
34
"
1- 1!
*
"
<#
<#
+
+U)2U V "
V
+
54"4" G3 4
V W"
4
)
XV 8*
<
*
3
V W"
29
4
&3 #
*
3 Q >5753 3
. !
//
!
3
#
4 5
!
!. ! ) # !
-
3 7
)
7
+
$ +
8
%
+
$ +
.
9
/
+
! /
!
! /
9 1
7 //
2 1
.
;
9
.
2
+"
+
$ +
;
;
8
%
+
$ +
;
9
+ "
$ + %
;
9
;
;
+
+
3 4
"
!
4
'C
+
6
%
%
1 2
#
2
+ "
$ + %
;
9
;
;
!
+
$ +
%
'
:
(
3 7
)
6
3 4
4
*
)
33
;
;
+
Q
54"4" G3 4
#
C
*
1
*
30
Appendix
4
")3 "
6 #
A #
9 +
$ %
6
/
!
*
$
!7
! %
8 !
!
2
*
)
! !
9
*
!7 '
*
$ !. . %
6 !
! 5 4
B
! 5 4
3
7
&
!
*
+
9
94
0
7
9 +
$ %
)
! !
9 +
$ %
"
7
*
!
9 +
$ %
: ! ' &5 7
!7 !. !
!
7
9 +
$"%
: ! ' ) #
!7 !. !
!
7
9 +
$ %
"
""
"
"
""
"
"
"
C""
!
C
"
) #
&5 7
!7 !.
!7 !.
!
!
B
$ ? "%>$ % $ ? %>$ % $&5 7 %>$) # %
+
+
+
+"
+ "
+
+ "
+
+
+
+
+
+
+
+
+
+
+
+
+
+ "
+
+ "
+
+
+
+
+
+
+ "
+
+
+
+
+
+
+
+
+"
χ2
+
+
+ ,,,
+ ,,,
+
+ ,,,
+ ,,
+ ,,,
+ ,,,
"+ ,,
+ ,,,
+ ,,,
+ ,,,
1>??(23
@C
&3&K 1
9
(!
4
2
H
K3J' 1
A
23 1 2 0
χ2
1D2 U 1&23 4
3
31
4
">3
+
N
<
"4
3
4
) # >3
6 #
4
!
3
9
#
/
&5 7
+"
+"
/
!.
+"
+
'
5
+
+"
+
+
#+ +
+
+
+
+ "
+ "
+
+
+ "
%
+
+
. %
+
+ "
+
+
+
+
+ "
+
+
+
+
+
+
+
+
$ + %
+ " ,
$ + %
+
$ + %
+
$ + %
+
$ + %
+
$ + %
+ " ,
$ + %
+
$ + %
+
$ + %
+ "
$ + %
+
$ + %
+ "
$ + %
3
8
4
!
! 5
4
7
!
!
2
*
$
*
$ !.
*
$
6
!
!7 '
%
! !.
6
)
! !
B
! 5
4
9
9
+
!
+
&
6 !
9
+
! #
2
3
! /
!
2
6 #
1?<)
/
+
$ + %
+ ",
$ + %
+
$ + %
+
$ + %
"
9 4
3 4
*
+
1
"4
F
23
2
3
32