Why do Airport Security Screeners Sometimes

Why do Airport Security Screeners Sometimes Fail in
Covert Tests?
Adrian Schwaninger
University of Applied Sciences Northwestern Switzerland
School of Applied Psychology, Institute Human in Complex Systems
4600 Olten, Switzerland
and
Center for Adaptive Security Research and Applications (CASRA)
Thurgauerstr. 39
8050 Zurich, Switzerland
covert tests. Several of the cited studies were funded by the
European Commission Leonardo da Vinci Program (VIA
Project, see www.viaproject.eu for details).
Abstract- This paper discusses covert testing data and key
human factors. First, different reasons why airport security
screeners sometimes fail to detect threats when covert tests are
conducted at airports are discussed . Key human factors are
identified and analyzed with regard to covert test results. It is
explained that pre-employment selection and training are
indispensable prerequisites for good operational performance of
airport security screeners. However, they alone cannot guarantee
that screeners react appropriately when tested with real threat
items at a security checkpoint. Humans often fail if something
happens that they do not expect. Therefore, it is recommended to
conduct covert tests on a regular basis so that screeners become
prepared and get trained to react appropriately. A recently
conducted study on covert tests provides empirical evidence in
favor of this view.
II.
For x-ray image interpretation, [I] has specified three
image-based factors which affect human threat detection
performance significantly: View difficulty, superposition, and
bag complexity. As can be seen in Figure 1, objects are more
difficult to recognize when depicted from an unusual viewpoint
(effect of viewpoint). When superimposed by other objects,
detection is impaired as well (effect of superposition). Finally,
objects become more difficult to detect, when there are many
other objects in the bag which distract visual attention (effect of
bag complexity). This has important implications for covert
tests. For a reliable comparison of results obtained from
different airports, companies, or screeners, it is important to
know whether covert tests are comparable in terms of view
difficulty, superposition, and bag complexity. [2] have
developed image measurements to determine the degree of
viewpoint difficulty, superposition and bag complexity
automatically (see also [3]). This model is very powerful and
the most recent formulae [4] provided the basis for a study
conducted in collaboration with QinetiQ at 5 European airports
in order to investigate the relative importance of image based
factors (including bag size) and computer-based training [5].
Keywords- Aviation security human factors; covert tests;
selection and pre-employment assessment; training
I.
ABILITIES AND APTITUDES
INTRODUCTION
Covert tests have sometimes shown deficiencies at airport
security checkpoints. Despite technological factors, there are
several key human factors which influence covert test results:
Abilities and aptitudes
Training
Motivation, attention and supervision
There are large differences between people regarding their
visual abilities to cope with image-based factors [6]. Therefore,
it is important to use a reliable, valid and standardized object
recognition test in pre-employment assessment to select
Expectations regarding covert tests
In the following sections each of these factors is discussed
to better understand why security officers sometimes fail at
'--------:---::' View Difficulty
Superposition
Bag Complexity
Figure I. Image-based factors view difficulty , superpos ition, and bag complexity.
978-1-4244-4170-9/09/$25.00 ©2009 IEEE
41
screeners who have the abilities needed to cope with viewpoint,
superposition and bag complexity. For example , [7] could
show that screeners selected with the X-Ray Object
Recognition Test (X-Ray aRT) performed significantly better
in an x-ray image interpretation test already after one year than
other screeners who worked at the airport for several years but
were not selected using this test (Figure 2).
1.00 , - -0.95
g
0.85
E
0.80
ctc
o
&l
0.75
~
~
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-----,
,- - - - - - - - - - - - - - - - - - - - -
~ 0.90 1 -
III
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
looks quite different in the x-ray image than in reality . This is
one reason why many threat items are difficult to recognize
without training . A second reason is that several objects are not
known from everyday experience, which accounts at least for
the self defense gas spray depicted in Figure 3c and the
improvised explosive device (lED) depicted in Figure 3d.
Additionally, some threat objects may bear resemblance to
harmless objects when seen in an x-ray image . For exampl e the
switch-blade knife depicted in Figure 3b resembles a pen or a
laser pointer.
-
1
1
Another problem is impaired recognrtion resulting from
viewpoint changes (e.g., [10]; [12]). If an object is depicted
from an unusual viewpoint, it becomes difficult to recognize.
This is illustrated in Figure 4. Most people have already seen
each of the three objects in real life. However, if seen from an
unusual viewpoint like at the top, most people have great
difficulties in recognizing the images without training . These
examples illustrate the importance of initial and recurrent
training to achieve and maintain a sufficient level of x-ray
image interpretation competency.
0.70
0.65
0.60
0.55
0.50
l t d ith lXR
ORT
l t d ithXR
ORT
Figu re 2. Detection performan ce in an x-ra y image inter pretatio n test (XRay Prohibited Items Test) of screeners who we re sele cted w ith the X-Ray
ORT within pre-employment assessment (right) and tho se who were not
selec ted with the X-Ray ORT (left),
III.
Several studies in the last years have provided converging
evidence showing that computer-based training which is
adaptive and features a large threat image library (e.g. X-Ray
Tutor) is a very effective tool for increasing x-ray image
interpretation competency (e.g., [13]; [14]; [15]). As mentioned
above , in a study conducted in collaboration with QinetiQ at 5
European airport s the relative importance of image based
factors and training for threat detection performance of human
operators in airport security x-ray screening was investigated
[5]. Again , computer-based training turned out to be a key
driver to improving threat detection performance in x-ray
TRAI NING OF AIRPORT S EC URITY SCR EENERS
For the x-ray screening task, training is very important.
Visual cognition and object recognition studies have shown
that an object can only be recognized if it is similar to
something you have seen before ([9]; [10], [II]). The
consequences for x-ray screening are illustrated in Figure 3.
Each of the depicted bags contains a threat object and each one
b
a
d
Fig ure 3. Different types of threat item s in x-ra y images o f pas senger bags. a Electric shock dev ice, b switch blade knife , c sel f
de fense gas spray "G uardian Angel " , d improvised explosive device (l ED).
42
at the security checkpoint. However, more research is needed
to determine the exact role of supervision for operational
performance and what good supervision actually means.
screening and seemed to mediate the effects of some image
based factors. In fact, amount of hours of computer-based
training (with X-Ray Tutor) could predict detection
performance almost as well as all image-based factors together.
a
b
V.
c
As explained above, with TIP, screeners are exposed to
improvised explosive devices, guns, knives and other threats
during the routine baggage screening operation. When they
detect a threat, they immediately press a key on the x-ray
machine (the "TIP key"). Then screeners receive immediate
feedback on whether they have detected or missed a TIP. This
technology is very valuable to increase motivation and
attention. However, there are also risks associated with TIP
when not used properly.
Figure 4. Effect of viewpo int on recogn ition.
The following true story was observed about one year ago
at a security checkpoint of a European airport. It was a busy
day. There was a long waiting line and passengers were
stressed. Screeners got stressed, too and tried to do their best to
decide as quickly as possible for each x-ray image of a
passenger bag whether it is OK, or whether it needs to be handsearched. Suddenly, a screener noticed that an x-ray image of a
passenger bag seemed to contain an improvised explosive
device (lED). Because the screener was familiar with TIP and
he had seen similar fictional threat items many times since TIP
had been activated at this airport, the screener immediately
pressed the TIP key . However, this time the feedback message
was "No TIP was projected". The screener knew that this
means that he should send the bag to manual search because
theoreticalIy it could be a real bomb. However, the waiting line
at the checkpoint was quite long, passengers were stressed, and
the screener realized that this time there was no TIP. After
working at the airport for many years, the screener was well
aware that the probability of a real bomb is extremely low . As
he explained later, his thoughts were "Alright, that can happen,
it just looked like a bomb, next time I will press the button only
if I am realIy sure there is a TIP" and the bag was not sent to
manual search. A couple of minutes later a supervisor told the
screener that he had just failed at a covert test conducted by an
EU official.
Training is a very important factor for achi eving and
maintaining good operational performance, which has also
been shown by examining how adaptive CBT results in large
increases of threat image projection (TIP) performance [16]. As
will be discussed below, training is also an important
prerequisite for good operational performance in covert tests of
x-ray screening operations.
For other tasks , like for example pat-down or manual bag
search , training might be less important than other factors such
as motivation to conduct the task thoroughly (see below).
However, more research is needed to investigate the
importance of training for non x-ray security screening tasks.
IV.
EXPECTATIONS REGARDING COVERT TESTS
MOTIVATION, A TTENTIO N AND S UPERVISION
Let's assume a security company conducts a reliable, valid
and standardized pre-employment assessment procedure to
select the right people for airport security screening. Let's
assume further that this security company conducts highly
effective initial and recurrent computer-based training of x-ray
image interpretation. If the screeners are not motivated and as a
consequence do not pay attention to what happens on the x-ray
screening monitor they will still miss threats in x-ray images.
Thus, although pre-employment selection and training are
indispensable
prerequisites
for
good
operational
performance of airport security screeners, they alone
cannot guarantee good operational performance. In addition
to pre-employment selection and training, measures must be
taken to keep motivation and attention of screeners at a
sufficient level. One possibility is to use threat image
projection (TIP). This technology of state-of-the art x-ray
screening systems allows projecting fictional threat items
(FTls) into x-ray images of real passenger bags during the
routine baggage screening operation. This way, screeners are
exposed to different types of threat objects, which can help
increasing attention and motivation of screeners substantially.
Psychology telIs us that humans often fail when something
happens that they do not expect. Examples are wrong reactions
when there is a fire alarm , failures of reacting correctly when
somebody has a heart attack or another type of accident, as well
as many other examples. When screeners do not expect that
they could be tested with real threat objects in a covert test at
the security checkpoint, there is a substantial risk of failure.
Therefore, it is quite important that covert tests are conducted
on a periodical basis. With TIP it is possible to test whether
screeners can detect threats in x-ray images. With covert tests it
can be examined whether they also react appropriately after
detecting a threat item in an x-ray image. Thus , covert tests and
TIP are rather complementary than interchangeable tools to
enhance operational performance. This is especially important
if TIP is used with a small threat image library and as a result,
screeners become eagle eyed for TIP images but might miss
real threats if no covert tests are conducted.
Unfortunately, for obvious reasons, such a technology does
not exist for other tasks such as for example manual bag search
or pat-down search. As will be further elaborated below, this is
another reason why covert tests are important and have a
training value. In addition, a good screener supervisor could
help increasing operational performance by carefully
supervising screeners and giving them feedback while working
As mentioned above, there are several other tasks at a
security checkpoint in addition to x-ray screening where there
43
is no technology like TIP available. For these tasks the
importance of covert tests as means of training and quality
control are even more important as will be shown in the next
section.
VI.
•
•
AN EXAMPLE OF A SCIENTIFIC STUDY ON COVERT
TESTS
•
•
Until today there have been many studies investigating the
importance of pre-employment assessment, computer-based
training and TIP in x-ray screening. In contrast, the area of
covert testing has received much less attention. Recently, [17]
have conducted a study on covert tests at one large European
airport over the period of fourteen calendar months. The key
findings can be summarized as follows:
•
Ideally, such questions would be investigated in close
collaboration between scientists, appropriate authorities and
airport security providers. This would be very beneficial to
better understand and provide a solid basis for the enhancement
of current airport security controls.
Covert tests are valuable for quality control and risk
analysis purposes and also to train screeners to react
appropriately in dangerous situations.
Performance at covert tests increased significantly over
the full period of fourteen months.
VIII.
Threats hidden in carry-on baggage were detected best,
which according to the authors, is not surprising since at
this airport, screeners conduct frequent recurrent
computer based training since 2005.
SUMMARY AND CONCLUSIONS
A. Abilities and aptitudes
There are large differences between people regarding their
ability to become a good screener. For x-ray screening, people
need to have the ability to cope with image-based factors such
as viewpoint, superposition and bag complexity. This can be
tested using an object recognition test in pre-employment
assessment. For other tasks, more research is needed to
determine which are specific abilities/aptitudes that can be
tested reliably in pre-employment assessment.
Compared to the detection rates observed in x-ray
screening there was a loss in detection when threat items
were carried on the body.
There were substantial differences regarding different
types of threat objects. IEDs were detected best and
knives were detected worst. Although this result might
seem at first surprising, in fact it is not, as pointed out by
the authors, because IEDs and fuses were always hidden
in carry-on baggage and screeners detected those items
best because they had received adaptive computer-based
training for cabin baggage screening since early 2005.
B.
Training
For x-ray screening, initial and recurrent training is
essential to learn and keep up to date regarding threat objects
and what they look like in x-ray images. For other tasks it is
currently assumed that initial training suffices. However, this
should be further investigated.
The time taken to conduct pat-down search was an
important variable, indicating that it is crucial that the
security officers take enough time for this control task.
C. Motivation, attention and supervision
If the screeners are not motivated and as a consequence do
not pay attention to what happens on the x-ray screening
monitor they will still miss threats in x-ray images. Thus,
although pre-employment selection and training are
indispensible
prerequisites
for
good
operational
performance of airport security screeners, they alone
cannot guarantee good operational performance. Means for
increasing attention and motivation are TIP (for x-ray
screening), good supervision (for all tasks) and other factors.
More research is needed in this regard.
The authors suggest that by regular covert testing, a certain
effect of training and awareness occurs. This is illustrated by
the fact that when security officers were tested the second time,
in almost all cases, they performed much better. Moreover,
"some [screeners] have reported that until the experience of
failure, they were not aware of the difficulty and importance of
a thorough pat-down search. As a result, they afterwards
conducted much more thorough controls than before" [5], p. 5.
The authors point out that it is unclear whether the same
results would be obtained at other airports which have different
procedures, training programs and employees. Nevertheless, it
seems clear that this study provides good evidence for the view
that covert tests have a training value and are important to
make sure screeners react appropriately when confronted with a
real threat.
VII.
How many covert tests are needed to achieve the optimal
cost / benefit ratio?
How many covert tests are needed for a reliable
measurement of performance of an airport / a security
company / a checkpoint, etc.
Which abilities / aptitudes are relevant for covert test
results in non x-ray related tasks?
How can we define a standardized method for conducting
covert tests so that results for different countries / airports
/ companies, etc. become comparable?
Do covert test results correlate with TIP data?
D. Why covert tests seem to be very valuable
Humans often fail if something happens that they do not
expect. When screeners do not expect that they could be tested
with real threat objects in a covert test at the security
checkpoint, there is a substantial risk of failure. With TIP it is
possible to test whether screeners can detect threats in x-ray
images. With covert tests it can be examined whether they also
react appropriately after detecting a threat item in an x-ray
QUESTIONS TO BE ADRESSED IN FUTURE STUDIES
Several questions remain to be addressed in future studies:
44
image. Moreover, when TIP is operational, screeners become
exposed to threat items on a regular basis. It is then even more
important, that covert tests are conducted on a regular basis to
ensure that screeners react appropriately when facing a real
threat. Thus, covert tests and TIP are rather complementary
than interchangeable. A technology such as TIP is not available
for all other non x-ray related tasks at a security checkpoint.
For those tasks, covert tests are important for quality control
p~rposes. Finally, covert tests seem to have a training value,
SInce people get better when tested repeatedly.
International Conference on Research in Air Transportation, ICRAT
2006, Belgrade, Serbia and Montenegro, June 24-28, 2006, 393-397.
Schwaninger, A. (2003b). Evaluation and selection of airport security
screeners. AIRPORT, 02/2003, 14-15.
[2]
Schwaninger, A., Michel, S., & Bolfing, A. (2007). A statistical
approach for image difficulty estimation in x-ray screening using image
measurements. Proceedings of the 4th Symposium on Applied
[13] Ghylin, K.M., Drury, C.G., & Schwaninger, A. (2006). Two-component
model of security inspection: application and findings. 16th World
Congress of Ergonomics, lEA 2006, Maastricht, The Netherlands, July,
10-14,2006.
Bolfing, A., Halbherr, T., & Schwaninger, A. (2008). How image based
factors and human factors contribute to threat detection performance in
x-ray aviation security screening. HCI and Usability for Education and
Work, Lecture Notes in Computer Science, 5298,419-438.
Bolfing, A., & Schwaninger, A. (2007). Measurement formulae for
image-based factors in x-ray imagery. VICOREG Technical Report,
November 26,2007.
[5]
Schwaninger, A., Bolting, A., Halbherr, T., Helman, S., Belyavin, A., &
Hay L. (2008). The impact of image based factors and training on threat
detection performance in X-ray screening. Proceedings of the 3rd
[14] Michel, S., de Ruiter, J. C., Hogervorst, Koller, S.M., M., Moerland, R.,
& Schwaninger, A. (2007). Computer-based training increases efficiency
in x-ray image interpretation by aviation security screeners. Proceedings
of the 41st Carnahan Conference on Security Technology, Ottawa,
October 8-11, 2007.
[15] Koller, S.M., Hardmeier, D., Michel, S., & Schwaninger, A. (2008).
Investigating training, transfer, and viewpoint effects resulting from
recurrent CBT of x-ray image interpretation. Journal of Transportation
Security 1(2),81-106.
International Conference on Research in Air Transportation, ICRAT
2008, Fairfax, Virginia, USA, June 1-4,2008,317-324.
[6]
[7]
Graf, M., Schwaninger, A., Wallraven, C., & Biilthoff, H.H. (2002).
Psychophysical results from
experiments on recognition &
categorisation. Information Society Technologies (1ST) programme,
Cognitive Vision Systems - CogVis (IST-2000-29375).
[12] Koller, S., & Schwaninger, A. (2006). Assessing X-ray image
interpretation competency of airport security screeners. Proceedings of
the 2nd International Conference on Research in Air Transportation,
ICRAT 2006, Belgrade, Serbia and Montenegro, June 24-28,2006,399402.
123-130.
[4]
[9]
[11] Schwaninger, A. (2005a). Objekterkennung und Signaldetektion. In: B.
Kersten (Ed.), Praxisfelder der Wahrnehmungspsychologie (pp. 108132). Bern: Huber.
Perception in Graphics and Visualization, ACM Press, New York, USA,
[3]
Hardmeier, D., & Schwaninger, A. (2008a). Selection and preemployment assessment of aviation security screeners, Report of VIA
Project Work Package 5, available at http://viaproject.eu/results.php.
[10] Schwaninger, A. (2004b). Computer based training: a powerful tool to
the enhancement of human factors. Aviation Security International,
FEB/2004, 31-36.
REFERENCES
[1]
[8]
[16] Schwaninger, A., Hofer, F., & Wetter, O.E. (2007). Adaptive computerbased training increases on the job performance of x-ray screeners.
Schwaninger, A., Hardmeier, D., & Hofer, F. (2005). Aviation security
screeners visual abilities & visual knowledge measurement. IEEE
Aerospace and Electronic Systems, 20(6), 29-35.
Proceedings of the 41st Carnahan Conference on Security Technology,
Ottawa, October 8-11, 2007.
[17] Wetter, O.E., Hardmeier, D., & Hofer, F. (2008). Covert testing at
airports: exploring methodology and results. Proceedings of the 42nd
IEEE International Carnahan Conference on Security Technology,
October 13-16,2008, Prague, Czech Republic.
Hardmeier, D., Hofer, F., & Schwaninger, A. (2006a). Increased
detection performance in airport security screening using the X-Ray
ORT as pre-employment assessment tool. Proceedings of the 2nd
45