IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
Reasoning About Threats: From Observables
to Situation Assessment
1
2
Gertjan J. Burghouts and Jan-Willem Marck
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Abstract—We propose a mechanism to assess threats that are
based on observables. Observables are properties of persons, i.e.,
their behavior and interaction with other persons and objects. We
consider observables that can be extracted from sensor signals and
intelligence. In this paper, we discuss situation assessment that is
based on observables for threat assessment. In the experiments, the
assessment is evaluated for scenarios that are relevant to antiterrorism and crowd control. The experiments are performed within
an evaluation framework, where the setup is such that conclusions
can be drawn concerning: 1) the accuracy and robustness of an
architecture to assess situations with respect to threats; and 2) the
architecture’s dependence of the underlying observables in terms
of their false positive and negative rates. One of the interesting conclusions is that discriminative assessment of threatening situations
can be achieved by combining generic observables. Situations can
be assessed with a precision of 90% at a false positive and negative
rate of 15% using only eight learning examples. In a real-world
experiment at a large train station, we have classified various types
of crowd dynamics. Using simple video features of shape and motion, we have proposed a scheme to translate such features into
observables that can be classified by a conditional random field
(CRF). The implemented CRF shows to classify successfully the
crowd dynamics up to 80% accuracy.
27
28
29
Index Terms—Architecture, evaluation framework, information
processing, observables, situation understanding, threat recognition.
I. INTRODUCTION
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Q1
1
ITUATION understanding is relevant to the field of security (i.e., robbery and vandalism), public safety (i.e.,
aggression and riots), and health care (i.e., incidents with
elderly people). Technology is making its entrance in solutions
for these domains. More recently, research institutes and technology providers have focused also on the domain of antiterrorism, aiming for solutions that detect threats in an early stage [1].
To detecting a threat at an early stage is important, as it enables
security professionals to mitigate the situation. In this paper,
we focus on technology to recognize the stages that build up to
potential threats.
Our objective is a system to alert for potential threats. To
that end, the system assesses the current situation in order to
S
Manuscript received December 30, 2009; revised September 20, 2010 and
January 25, 2011; accepted March 19, 2011. This paper was recommended by
Associate Editor J. Tang.
The authors are with The Netherlands Organization for Applied Scientific
Research (TNO) Observation Systems, The Hague 2597 AK, The Netherlands
(e-mail: [email protected]; [email protected]).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TSMCC.2011.2135344
Fig. 1. Our framework: outputs of various sensors (JDL level 0) are processed
to the higher abstraction level of observables (JDL level 1). Examples of observables are taken from our earlier work, e.g., trajectory analysis, pose estimation,
color of clothing, carrying an attribute, behavior recognition, and group dynamics [3]. These observables provide information to the situation estimator (JDL
level 2). Specific situations are associated with threats. Situation assessment is
the focus of this paper.
determine whether a threat may occur. Our approach is to assess
the situation from multiple object assessments. These two layers
are adopted from the well-known JDL data fusion model and
its extensions [2]. Object assessments are based on sensors and
intelligence. This is illustrated in Fig. 1.
The output of sensors requires automatic processing up to the
level of meaningful and robust information, which we will refer
to as observables. Intelligence is also taken as an observable.
The observables are object assessments and range from camera
observables (i.e., person carries some object) to geo-observables
(i.e, the person has a short interaction with another person) to
intelligence observables (the suspect is a person with a gray
shirt and black pants). Based on the observables, the situation
is assessed. For instance, the observables that a suspect person
hands over an object to another person who is mixing in the
crowd would result in the situation assessment that somebody
is carrying a suspect object.
In this paper, we discuss the methods to exploit observables
for threat assessment. We motivate our choices for specific pattern recognition methods. In the experiments, the added value
of the implemented method is demonstrated for scenarios that
are relevant to antiterrorism and crowd control. The experiments
are performed within an evaluation framework, where the setup
is such that conclusions can be drawn concerning: 1) the performance and robustness of an architecture to assess threats; and
2) the architecture’s dependence of the underlying observables
in terms of their false positive and negative rates.
1094-6977/$26.00 © 2011 IEEE
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
2
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
A. Previous Work
Many observables have been introduced, which range from
detecting specific objects such as cars [4] to detecting aggression
[5], from generic features of human walking patterns [6] or
highly specific human acts such as shaking hands and kissing [7].
In this paper, we focus on properties and behavior of humans that
can be detected and recognized in video. The goal of our work
is to assess situations beyond the level of isolated behaviors,
which is based on multiple observables.
Previously, other researchers have also considered situation
assessment that is based on multiple observables, e.g., the combination of visual and audio features to improve the detection
of aggression [8]. With an appropriate combination scheme, the
discriminative power of situation assessment can be improved
by using multiple observables. Situation assessment is a challenge, as the situations under investigation can range from instantaneous (e.g., one person gives an object to another person)
to longer periods (e.g., a pickpocket is waiting the best moment
to steal someone’s belongings). In recent literature, researchers
tend to focus on either instantaneous events [9] or on complex,
long-term person–person or person–object interactions [10]. In
this paper, we propose a generic scheme to model and classify
both types of situations.
In this paper, we consider only situations that are known
a priori. This requires examples from which a system can learn.
Interestingly, recent work has shown that it is possible to detect
abnormal situations at run time [11]. Such methods are not
within the scope of this paper.
Another observation from reviewing recent work is that scientific communities have released descriptions and datasets of
behaviors of crowds and individuals. These datasets are very interesting and include recordings of single observables to experiment with, e.g., the PETS and TRECVID benchmarks. However,
such communities have not yet focused on short- and long-term
sequential scenarios. As a consequence, no scenarios are available for antiterrorism and crowd-control applications.
B. Contributions in this Paper
The contribution in this paper is fourfold.
1) Relevant scenarios for antiterrorism and crowd control.
We discuss how expert knowledge of incidents can be
used. This is crucial, as for the particular incidents related
to terrorism, escalation of crowd behavior, and suspect
people and objects, few or no data will be available.
2) A tool to generate a multitude of variations of the scenarios. The variations include statistical and systematic error
sources.
3) A framework to evaluate both 1) the performance of the
situation assessment as a whole and 2) its dependence of
underlying observables. The experiments are performed
by the usage of a conditional random field (CRF). The
outcomes of the evaluation give insight in 1) whether
more (complementary) observables are required to assess
situations successfully; and 2) whether the robustness of
observables needs to be improved in order to be useful for
situation assessment.
4) Given the scenarios and their variations and the framework, we evaluate a system for situation assessment that
was implemented for this paper. The system is described in
Section III that is based on a CRF that takes observables as
input and estimates situations as output. We demonstrate
in Section IV that discriminative power can be achieved
by combining generic observables.
This paper is organized as follows. In Section II, we discuss the scenarios and the tool to generate their variations.
In Section III, we propose the CRF that learns the best relation between observables and the assessment of situations. In
Section IV, the experimental setup is discussed. In Section V,
the CRF is evaluated for the scenarios and their variations. In
addition, the CRF is evaluated against a real-world dataset. Section VI concludes this paper.
140
II. SCENARIOS
141
In this section, we define scenarios to which our system (see
Section III) is applied and on which the experiments will be
based (see Section IV). A scenario is interpreted as a sequence
of states. The state is a description of the situation at a particular
time. That is, “a group of people is agitated.” Note that an
expert may be consulted to associate each state with a level of
escalation or alarm, to bridge the gap to the decision maker. For
each subsequent period, a number of observables are observed.
Observables are observed object properties. That is, “a person
enters the scene.” A scenario is specified by the sequence of
states, their duration, and the observables that are observed for
each subsequent period.
By the previous definition, the observables may vary from
time to time. Different sets of observables may be observed for
the same state at different periods. There is no direct coupling
between states and observables. In Section III, we select a pattern recognition method to learn the best probabilistic relation
between states and observables. In Section IV, we evaluate how
well this learned relation predicts the states for unseen variations
of the scenarios.
Observables may be chosen such that they relate to specific
persons or groups of persons that are of particular interest. That
is, “a suspect person enters the scene” or even more specific
“John enters the scene.” In this paper, we consider observables
that are generic, e.g., “a person enters the scene,” “a suspect
person is in the scene,” etc. Interestingly, in Section IV, we
demonstrate that discriminative power can be achieved by combining generic observables.
142
126
127
128
129
130
131
132
133
134
135
136
137
138
139
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
A. Observables
170
We have selected the following observables: empty square,
one person, two-five persons, many people, people flux +, people flux −, sudden large people flux, tracks toward hotspot,
tracks in hotspot, group formation, group moves, group on collision, individual is avoided by other people, individual’s head
orientation varies, individual carries object, individual has wild
gestures, somebody wears clothes related to a suspect person.
The observables have been adopted from interviews with security professionals and from a training in The Netherlands
171
172
173
174
175
176
177
178
179
BURGHOUTS AND MARCK: REASONING ABOUT THREATS: FROM OBSERVABLES TO SITUATION ASSESSMENT
182
(police academy) on “search, detect, and react” (based on an
Israeli security training). The observables are illustrated in
Fig. 1.
183
B. Antiterrorism and Crowd-Control Scenarios
184
Five scenarios for antiterrorism and crowd control are proposed: square is revolting, aggressive political speaker, extreme
demonstration, neutral, fight between a few people.
The situation at a given time is formalized by one of the
following states: neutral, group with agitator, other people are
avoiding the situation, group is coordinating something, group
is agitated, group riot.
The scenarios are defined by the format: {[state1 , [observable1,1 ,
observable1,2 , . . . ,
observable1,N ],
length1 ],
. . . , [stateT . . .]}. The variable “length” refers to the length of
the period of the current state. The number of observables may
vary per period. These formal specifications will be illustrated
at the end of this section.
First, we present the five scenarios in natural language.
1) Square is revolting: market square, many people moving randomly, two people are loitering, group assembles
just outside the market square, one group member is agitated, the group starts moving, the loitering people join
the group, another group starts to move, one of the group
members carries something, people are avoiding the two
groups, the two groups confront each other, fight, people
fleeing.
2) Aggressive political speaker: somebody start to speak in
the midst of people, people are moving around, people
start to listen, people are walking away.
3) Extreme demonstration: people are avoiding the group that
is demonstrating, the atmosphere is tense, little riot, the
demonstrating group is moving.
4) Fight between a few people: some people are agitated, one
troublemaker is starting a fight, small fight, people are
avoiding the fight, fight stops.
5) Default: neutral situation, people moving, many people
enter buildings, a small group is walking by, square is
getting more crowded, another group walks by.
180
181
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
C. Scenario Generation Including Statistical and Systematic
Errors
The scenarios are transformed into datasets. The inputs are
the observables (i.e, binary values) and the ground truth is the
state per timestep. Four types of noise can be added to the
observables.
1) Statistical errors: random false positives and negatives.
2) Systematic errors: fade in/out: false positives are added in
front of active observables, or at the end.
3) Systematic errors: ambiguity: false positives are added
for a particular observable, and false negatives are added
randomly.
4) Systematic errors: clutter: for each state, a state-dependent
clutter effect is generated, which results in more false
positives and negatives when more clutter is apparent.
3
Fig. 2. Example of increasing statistical errors for the scenario “square is
revolting.” On the horizontal axis, the timesteps are shown. The vertical axis
indicates the observables (each row represents one observable). (a) FPR =
FNR = 0%. (b) FPR = FNR = 10%. (c) FPR = FNR = 20%.
Fig. 3. Example of nontrivial relation between (a) observables and (b) states.
The relation between observables and states is not clear and probabilistic in
nature. Learning the probabilistic relation between observables and states is the
objective of this section and the experiments.
All types of noise will be expressed in terms of false positive
ratio (FPR) and false negative ratio (FNR) in the experiments.
Fig. 2 illustrates the effect of various FPRs and FNRs on the
datasets.
236
III. FROM OBSERVABLES TO SITUATION ASSESSMENT
237
Recall from the previous section, the observables may be
noisy. In this section, we exploit a CRF to learn the best probabilistic relation between states and observables.
To assess the current state of the situation, the observables are
simply considered altogether. The situation assessment is based
on a “bag of observables.” More specifically, in this paper, we
interpret situation assessment as a mapping problem from the
“bag of observables” to a situation state.
The observables are perceptual concepts. The situation states
are implicit concepts that are not directly observed from the data.
Fig. 3 makes this clear. There is no obvious relation between
the observables and states. The situation state can be interpreted
as a hidden parameter of the observables. This makes the state
prediction nontrivial. In Section V, we evaluate whether the
proposed CRF is a good technique to model the probabilistic
relations between observables and situation states.
238
233
234
235
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
A. Conditional Random Field
254
A CRF [12] is a type of discriminative probabilistic model
that is most often used to label or parse the sequential data, such
as natural language text or biological sequences. CRFs are a
more general form of hidden Markov models (HMM, [13]) and
relaxes the Markov assumption over the sequential data that is
characteristic for the HMM. This allows a CRF model to use
information from time separated data [12].
We made the observation that situations in real life evolve
in certain sequential orders. The evolution over time of the
scenario states resembles the kind of probabilistic sequences
255
256
257
258
259
260
261
262
263
4
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
Fig. 4. Example of a CRF. (a) Weights for observables to current state potential. (b) Weights of the previous state to current state potential. (c) Cumulative
probabilities. (d) Estimated state and ground truth.
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
of natural language. These sequences can be modeled by an
undirected graphical model in which each vertex represents a
state. The edges between the vertices can be understood as a
dependence between states. There are two main ways to model
such graphical dependences: HMM and CRF.
We choose the CRF over the HMM, because it has two advantages that are important to our problem. Both advantages are
due to the fact that the CRF model is a discriminative model
instead of generative model [14].
A discriminative model models the conditional probability
P (Y |X). A generative model is more widely applicable as it
can be used to calculate the joint probability P (X, Y ) and all
marginals of the joint, for example, P (X) or P (Y ). A discriminative model is limited to predicting state Y given certain
sequence of observations X. When fitting the model of a generative model, one approximates both P (X) and P (Y |X) instead
of merely P (Y |X) in the case of the discriminative model. This
results in a number of advantages.
1) A discriminative model is independent of P (X). This
means that to estimate P (X) for rarely occurring states
is not a problem. This results in better accuracy for rare
scenarios.
2) It can be trained in one setting and tested in a different setting with another P (X), since it is independent of P (X)
and dependences within P (X).
3) A discriminative model has better predictive performance
of P (Y ), since it is trained on classification of P (Y |X),
instead of modeling the joint probability. It does not need
additional effort to fit data with P (X).
Usage of the CRF consists of two phases: the data-fitting
phase and the inference phase. In the data-fitting phase, the
model parameters are fitted to a training dataset. The best fit is
the maximum of the (log) likelihood of these parameters that are
given the training dataset. The likelihood is a multidimensional
concave function. The optimum of this function gives the best fit
to the parameters. In effect, two matrices are learned, which are
used by the CRF to classify newly observed observables into the
next state: one to capture the dependences between observables
and states, and the other to capture the dependences from the
previous state to the next state. The observable-state matrix also
captures the relative importance of each observable for each
state.
In the inference phase, the learned model is used to estimate
P (Y ) from a given X. When estimating P (Y ), the forward
backward algorithm [13] is used to refine the estimation of
P (Y ). The forward backward algorithm finds the most probable
path in time over the states in P (Y ) [13] and smooths the
probabilities of P (Y ).
An example of a CRF is illustrated in Fig. 4. This example is used in Section V-B, where the precise states and
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
BURGHOUTS AND MARCK: REASONING ABOUT THREATS: FROM OBSERVABLES TO SITUATION ASSESSMENT
325
observables (in a real-world experiment) are further explained.
For the next illustration, the naming of states and observables is
not yet important; here, we want to discuss the underlying mechanism. We illustrate the weighting and potentials of observables
and the estimation of states. The weights of the observables
are shown in Fig. 4(a). The weights of the observables in the
previous state are shown in Fig. 4(b). Together, they influence
the estimation of the current state. Positive weights excite the
corresponding state, whereas negative weights inhibit that state.
This is exemplified by the cumulative probabilities in Fig. 4(c).
The state with highest probability becomes the estimated state
[see Fig. 4(d)].
326
IV. EXPERIMENTAL SETUP
327
328
We model the situation state as the hidden state of the CRF,
and the observables are the observations of the hidden state.
329
A. Conditional Random Field Evaluation Objectives
330
341
We evaluate how well the relation between states and observables can be modeled by a CRF. We are interested in the
following properties of the CRF.
1) Discriminative power: How well does the CRF predict the
situation states for unseen variations of the scenarios? This
is an interesting question, as the observables are generic
and, hence, not tailored to the specific scenarios and their
states.
2) Robustness: How sensitive is the CRF with respect to
statistical and systematic errors?
3) Capacity to generalize: How many examples of a given
set of scenarios are required to obtain a robust CRF?
342
B. Evaluation Tool
343
349
To test the discriminative power of the CRF for situations, we
adapted the Schmidt MATLAB CRF implementation [15]. We
implemented a scenario generator to generate the scenarios (see
Section II-B). In this tool, we also added the capability to include
the four noise modes (see Section II-C) to the observables of
the scenarios in order to measure the performance of the CRF
approach against various noise models.
350
C. Datasets
351
We test each noise variable to increase amounts of noise. The
datasets are generated from the scenarios in Section II and are,
subsequently, contaminated with increasing levels of noise. The
types of noise are statistical (random false positives/negatives)
and systematic (fade ins/outs, ambiguity, and clutter). All types
of noise will be expressed in terms of false positive and negative
ratios in the experiments.
314
315
316
317
318
319
320
321
322
323
324
331
332
333
334
335
336
337
338
339
340
344
345
346
347
348
352
353
354
355
356
357
358
D. Cross Validation
359
For each noise setting, 48 repetitions were generated.
Each repetition contains the entire set of five scenarios (see
Section II-B). For each noise parameter setting, a 12-fold cross
validation will be performed. Each test fold contains four repetitions and is tested against a training set that contains N randomly
360
361
362
363
5
Fig. 5. Results for the experiment with 16, 8, 4, 2, and 1 scenario variations
to learn from. Results are shown for nine fixed samples of FPR and FNR. For
each FPR/FNR sample, we indicate how the CRF classification accuracy drops
as a consequence of fewer learning examples. This is shown by each bar, where
the accuracy drops from red (almost 100%) to green (just above 50%). This
effect is worse for higher FPR/FNR (e.g., the bar in the upright corner). A good
balance between number of examples and performance of the CRF is obtained
for eight examples (see second from the left of each colored bar).
selected repetitions out of the 48 scenarios that are generated
for the current parameter setting. The averaged results from the
folds are the CRF performance results for the selected parameter
setting.
367
E. Training the Conditional Random Field
368
The CRF will be trained at each test fold on the randomly
selected training set. The data fitting, i.e, the optimizing of the
loglikelihood, is performed by the L-BFGS optimization algorithm [16]. This quasi-Newton optimization algorithm is a
standard to optimize large multidimensional optimization problems such as a CRF [17]. The optimization ends when the average error of the training set is smaller than 10−5 or after 500
iterations.
369
376
F. Evaluation Measures
377
The results of the CRF are presented as confusion matrices.
Following conventions, each confusion matrix contains on the
vertical axis the ground truth situation states and on the horizontal axis the predicted states by the CRF. Hence, a diagonal
with high prediction accuracies is aimed for.
We expect that for higher FPRs and FNRs, the predicted states
become unreliable, which will cause the confusion matrices
to have lower values on the diagonal. Each increasing state
index indicates an increasing alarm level. In the case of errors
(i.e., wrongly predicted states), we hope that the errors are not
distributed among all states but remain confined to subsequent
states, such that the perturbing effect on wrongly predicted alarm
levels remains limited.
378
364
365
366
370
371
372
373
374
375
379
380
381
382
383
384
385
386
387
388
389
390
6
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
Fig. 6. Situation assessment: results. Experiment: only one scenario variation to learn from, for various types of noise (see captions directly under the figure) and
the degree of noise (see false positive and negative ratios). Confusion matrices for each of the six situation states are displayed for various FPRs and FNRs. Up to
FPR or FNR of 15%, the CRF is surprisingly accurate to assess situations. (a) Statistical errors. (b) Systematic: fade ins. (c) Systematic: fade outs. (d) Systematic:
fade ins and outs. (e) Systematic: ambiguity. (f) Systematic: clutter.
BURGHOUTS AND MARCK: REASONING ABOUT THREATS: FROM OBSERVABLES TO SITUATION ASSESSMENT
391
V. RESULTS
392
395
First, we focus on experiments on synthetic data where we
can control the sources and amount of noise (see Section V-A).
Second, we show experimental results on real-world data (see
Section V-B).
396
A. Synthetic Data
397
The results in this section are used to evaluate the capacity
to generalize (see Section V-A1), the discriminative power (see
Section V-A2), and robustness (see Section V-A2).
1) Effect of the Size of the Training Set: Our first objective is
to find a good balance between the number of learning examples
to train the CRF and the performance of the CRF. If the number
of learning examples is low, then fewer examples have to be
collected, and training takes less time. However, with fewer
learning examples, the performance of the CRF will be lower.
To evaluate this tradeoff in detail, we investigate the results of
the CRF approach to increase FPR and FNR levels.
Fig. 5 shows the accuracy of the CRF for various sizes of the
training set, which has been decreased from 16 to smaller sets
of 8, 4, 2, and 1 training example(s). The figure shows at each
FPR (horizontal coordinate) and FNR (vertical) a bar of values,
one for each size of the training set. One value represents the
CRF’s accuracy of the situation assessment, where the accuracy
has been averaged equally over the situation states.
With 16 examples and errors up to 30%, the average accuracy
of the CRF is 81%. With fewer errors, up to 15%, the accuracy is
94%. With only one example, the accuracy drops, respectively,
to 50% (was 81%) and 63% (was 94%). A good tradeoff seems to
be eight examples, which achieves only slightly lower accuracy
(i.e., 4% lower than with 16 examples) while using only half
of the examples. Situations can be assessed with a precision of
90% at a false positive and negative rate of 15% using only eight
learning examples.
2) Effect of Statistical and Systematic Errors on the CRF:
Our second objective is to find out where the errors of the CRF
(wrongly classified states) occur and how they are distributed
over the states. To that end, we need a more detailed figure.
Rather than to display a single value that represents the accuracy
as averaged over the states, we display the confusion matrix for a
given size of the training set. As an extreme test case, we display
results for a training set of one example only. In addition, the
results are split into the various noise sources that we identified
earlier, each produces one subfigure [see Fig. 6(a)–(f)]. The
captions indicate the error sources: There are four error sources,
where the fade in/out source is separated into three related but
different cases, which results in six subfigures in total. The line
around each of the confusion matrices indicates the average
predication accuracy, which has been weighted equally over all
states. Fig. 6 altogether is meant to give a detailed insight in
how error sources and the levels of FPR and FNR deteriorate
the results.
Up to FPR or FNR of 15% for any type of noise, the CRF
is surprisingly accurate to assess situations. The accuracy starts
to deteriorate above 15% FPR or FNR or both. Interestingly,
the CRF is not very sensitive to fading effects or ambiguity (we
393
394
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
7
Fig. 7. Video descriptors to capture the crowd dynamics in a compact way.
This figure shows 12 out of the 104 descriptors.
TABLE 1
RESULTS FOR THE CLASSIFICATION OF CROWD DYNAMICS BY A CRF
introduced correlations) within the observables. On the contrary,
the CRF is sensitive to random noise and to clutter (state-specific
levels of random noise). The clutter case is most difficult, as the
noise is different for each state. As a consequence, the noise
increases and decreases over time. The CRF is least robust to
noise levels that vary over time.
451
B. Real-World Data
452
In this section, we consider an experiment that is based on
real data. The experiment is relevant for security and safety
applications. We have recorded and annotated 4-h video data
at the second largest train station in The Netherlands. The
video data includes many and diverse behavioral patterns of
the crowd, of which a limited set have been classified by a CRF.
For this real-world experiment, we have adopted state-of-the-art
video features that capture local structure and temporal dynamics. Given these features, we adopt a quantization scheme to
translate the many and high-dimensional video features into a
fixed number of low-dimensional observables. Based on these
spatiotemporal observables, various crowd dynamics are classified by the CRF. The goal is to classify three types of crowd
dynamics: “normal” (10–100 people), “abandoned” (0–10 people), “rush” (many people moving, to go to or leave the tram
platforms). In the recorded video, these three event classes are
correspondingly labeled by hand.
1) Describing the Crowd: Video Features: To describe the
crowd, we aim to capture the temporal dynamics in the crowd
(the general movement of, e.g., heads, shoulders, and legs). We
tailor existing video features for this task. We start with optical flow [18], which is computed at each pixel in the video
images, that yields at each pixel a magnitude and orientation.
We summarize these flows in spatial bins to reduce the amount
of data. We choose 104 fixed bins that are distributed equally
453
446
447
448
449
450
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
8
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
Fig. 8. Examples of errors for each type of crowd dynamics, where the CRF misclassified the state of the crowd (between parentheses). (a) “Normal” (“Rush”).
(b) “Abandoned” (“Normal”). (c) “Rush” (“Normal”).
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
over the image. The 104 spatial bins are obtained as follows.
The video is of HD quality, therefore each image frame is
1080 × 1440. We aim to sample approximately every 100 pixels. Discounting the border areas, the grid becomes 8 × 13,
which yields 104 spatial bins. Next, in each spatial bin, the flow
vectors are binned into eight orientations and four magnitudes.
This approach is inspired by other work, e.g., scale-invariant
feature transform [19], where discriminative yet compact image features were extracted from images by summarizing many
pixel values into few orientations and magnitudes. For adequate
continuous weighting of the optical flows, the contribution of
each flow vector to each bin is determined by a kernel-based
weight. With our approach, 104 × 8 × 6, i.e., 104 descriptors of
length 48 are obtained per video frame. This procedure captures
the dynamics of the crowd in a compact way. An illustration is
depicted in Fig. 7.
2) Transforming the Video Features Into Observables: We
experimentally established that the CRF is able to learn the three
types of crowd dynamics when the input is bound at maximum
40 observables. Recall that the video features yield 104 × 48 =
4992 values, i.e., much higher than the maximum of 40 observables. Hence, a translation is needed that reduces the video
features into approximately 40 observables. To do so, we adopt
a method of quantization: visual codebooks [20]. In a visual
codebook, the descriptors are assigned to a smaller selection
of representative descriptors (which are also called primitives).
This yields a histogram, of which the length is determined by
the number of primitives. We choose eight primitives, that result
in a histogram of length 8. Next, we discretize each bin (of the
eight-bin histogram) into four levels, to obtain 8 × 4 binary
values. Recall that the CRF is especially suited to learn from
binary values. In this translation procedure, the 104 multivalued
video features are translated into a 32 binary values. These 32
binary values are the observables, on which the CRF is trained
and tested.
3) Results: Classification of Crowd Dynamics: All examples
of the three classes are taken from the 4-h videos. These examples are subsampled at 2 frames/s. This is a choice to reduce
the amount of stored data and a limitation posed by processing
speed. We argue that 2 frames/s is sufficiently fast for awareness on crowd dynamics, yet the variation between samples is
significant. For each frame, the video features are computed and
translated into observables. Together with the classification of
observables into a next state, this procedure can be done at a
sampling rate of 2 frames/s. The trained CRF, with its weights
for the observables and state transitions, has been shown in
Fig. 4 and was discussed in Section III.
For the training of the CRF, the transitions between states
are essential. In our real-world dataset (i.e., the examples of
the classes), we found that the number of transitions is limited.
This poses a challenge to make distinct yet rich training and test
sets. We choose to solve this as follows. We perform a fourfold
cross validation. We subsample every Nth encoded video frame
to be included in the Nth-fold. This way, the folds are disjunct
and different enough as they are at minimum 2 s apart (given
the sampling of 2 frames/s and fourfolds). That is, the scene is
changing rapidly. Going from one video frame to the next, with
2 s in between, gives an enormous variation in the images. People
have walked significantly, new people have entered, other people
have exited the scene, and people have got closer or further apart
from other people. Following conventions, we train on onefold
and test on another. This experiment is repeated four times.
The results are summarized by the confusion matrix in
Table I. Interestingly, the CRF classifies some states that are
more accurate than others. The CRF has shown to successfully
classify the crowd dynamics up to 80% accuracy.
Examples of errors are shown in Fig. 8. An error that occurs
regularly is a badly timed state transition of the estimation.
Usually, the transition is to the right state but is too early or too
late. This is illustrated in Fig. 8(b) and (c). Sometimes there are
“true misclassifications;” this is illustrated in Fig. 8(a).
VI. CONCLUSION
We have proposed scenarios that are relevant for antiterrorism
and crowd control. Given the scenarios, and their variations
including various kinds of errors, we have assessed situations
that are based on observables with the objective to recognize
threats.
We have chosen the CRF as a method to learn probabilistically the relation between observables and states (as the hidden
parameter). A good size for the training set was shown to be
eight examples for each of the five scenarios, which achieves
only slightly lower accuracy (i.e., 4%) than with 16 examples
while using only half of the examples. Situations can be assessed
with a precision of 90% at a false positive and negative rate of
15% using only eight learning examples.
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
BURGHOUTS AND MARCK: REASONING ABOUT THREATS: FROM OBSERVABLES TO SITUATION ASSESSMENT
580
We have investigated the extreme test case of using a training
set of only one example to find out where the CRF becomes
inaccurate. Up to FPR or FNR of 15% for any type of noise
(random, fade in/out, ambiguity, and clutter), the CRF is surprisingly accurate to assess situations. The accuracy starts to
deteriorate above 15% FPR or FNR or both. The CRF is not
very sensitive to fading effects or ambiguity but is sensitive to
random noise and to clutter (i.e., state-specific levels of random
noise).
In a real-world experiment at a large train station, we have
classified various types of crowd dynamics. Using simple video
features of shape and motion, we have proposed a scheme to
translate such features into observables that can be classified by
a CRF. The CRF has shown to successfully classify the crowd
dynamics up to 80% accuracy.
Overall, we conclude this paper with the interesting finding
that discriminative power can be achieved by combining multiple, generic, and simple observables.
581
ACKNOWLEDGMENT
582
585
The authors would like to thank Dr. R. den Hollander to
provide the camera-based observables and their characteristics.
They are also grateful to Dr. K. Schutte for useful discussions
on the inference of the situation that is based on observables.
586
REFERENCES
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
[1] M. M. Kokar, “Situation awareness: Issues and challenges,” in Proc. Int.
Conf. Inf. Fusion, 2004, pp. 533–534.
[2] J. Llinas, C. Bowman, G. Rogova, and A. Steinberg, “Revisiting the JDL
data fusion model II,” in Proc. Int. Conf. Inf. Fusion, 2004, pp. 1218–1230.
[3] G. J. Burghouts, B. Broek, B. G. Alefs, E. den Breejen, and K. Schutte,
“Automated indicators for behavior interpretation,” in Proc. Int. Conf.
Crime Detect. Prevent., 2007, pp. 1–6.
[4] S. Agarwal, A. Awan, and D. Roth, “Learning to detect objects in images
via a sparse, part-based representation,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 26, no. 11, pp. 1475–1490, Nov. 2004.
[5] A. Datta, “Person-on-person violence detection in video data,” in Proc.
Int. Conf. Pattern Recognit., 2002, pp. 433–438.
[6] D. M. Gavrila, “The visual analysis of human movement: A survey,”
Comput. Vis. Image Understand., vol. 73, no. 1, pp. 82–98, 1999.
[7] I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, “Learning realistic
human actions from movies,” in Proc. IEEE Int. Conf. Comput. Vis. Pattern
Recognit., 2008, pp. 1–8.
[8] W. Zajdel, D. Krijnders, T. Andringa, and D. Gavrila, “CASSANDRA:
Audio-video sensor fusion for aggression detection,” in Proc. Int. Conf.
Adv. Video Signal Based Surveillance, 2007, pp. 200–205.
[9] L. Duan, D. Xu, I. W. Tsang, and J. Luo, “Visual event recognition in
videos by learning from web data,” in Proc. IEEE Int. Conf. Comput. Vis.
Pattern Recognit., 2010, pp. 1959–1966.
[10] D. Küttel, M. Breitenstein, L. van Gool, and V. Ferrari, “What’s going on?
discovering spatio-temporal dependencies in dynamic scenes,” in Proc.
IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2010, pp. 1951–1958.
[11] T. Xiang and S. Gong, “Video behaviour profiling for anomaly detection,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 5, pp. 893–908, May
2008.
[12] A. McCallum and C. Sutton. (2006). “An introduction to conditional random fields for relational learning,” in Introduction to Statistical Relational Learning, L. Getoor and B. Taskar, Eds. Cambridge, MA: MIT Press [Online]. Available: http://www.cs.umass.edu/
mccallum/papers/crf-tutorial.pdf
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
583
584
9
[13] L. Rabiner, “A tutorial on hidden Markov models and selected applications
in speech recognition,” Proc. IEEE, vol. 77, no. 2, pp. 257–286, Feb. 1989.
[14] I. Ulusoy and C. Bishop, “Generative versus discriminative methods for
object recognition,” in Proc.IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, vol. 2, pp. 258–265.
[15] M. Schmidt. (2008). “CRF toolkit in MATLAB,” [Online]. Available:
http://people.cs.ubc.ca/schmidtm/Software/
[16] D. C. Liu and J. Nocedal, “On the limited memory BFGS method for large
scale optimization,” Math. Program., vol. 45, no. 3, pp. 503–528, 1989.
[17] F. Sha and O. Pereira, “Shallow parsing with conditional random
fields,” in Proc. HLT-NAACL, 2003, pp. 213–220 [Online]. Available:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.10.9849
[18] B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artif. Intell.,
vol. 17, nos. 1–3, pp. 185–203, 1981.
[19] D. G. Lowe, “Object recognition from local scale-invariant features,” in
Proc. Int. Conf. Comput. Vis., 1999, pp. 1–8.
[20] B. T. F. Jurie, “Creating efficient codebooks for visual recognition,” in
Proc. Int. Conf. Comput. Vis., 2005, pp. 604–610.
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
Gertjan J. Burghouts received the Ph.D. degree
from the University of Amsterdam, Amsterdam, The
Netherlands, in 2007 on the topic of visual recognition of objects and their motion, in realistic scenes
with varying conditions.
He is currently a Lead Research Scientist of Visual Pattern Recognition with The Netherlands Organization for Applied Scientific Research (TNO), The
Hague, The Netherlands. He studied artificial intelligence at the University of Twente during 1997–2002
with a specialization in pattern analysis and human–
machine interaction. Since 2007, he has been the Principal Investigator of automated understanding of human behavior based on sensory perception. He is
the Principal Investigator of a DARPA project named CORTEX (2.3M), about
recognition of events and behaviors. He has written papers on this topic in
internationally renowned journals, e.g., the IEEE TRANSACTIONS ON IMAGE
PROCESSING, the Computer Vision and Image Understanding, the International
Journal of Computer Vision, and International Conference on Crime Detection
and Prevention. His work has been cited mote than 200 times since 2005.
Dr. Burghouts received an award from the Netherlands Association of Engineers for the best innovative project in 2007.
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
Jan-Willem Marck received the M.Sc. degree in artificial intelligence from the University of Groningen,
Groningen, The Netherlands, in 2006, with a specialization in autonomous systems.
He is currently a Research Scientist of Artificial Intelligence with the Department of Distributed Sensor
Systems, The Netherlands Organization for Applied
Scientific Research (TNO), The Hague, The Netherlands. Since 2006, he has been a Research Scientist
on various topics, e.g.,sensor information fusion, relevance of information, human–machine interaction
and human behavior classification based on state estimation methods using sensory data. He has (co-)authored a number of papers and managed projects on
these topics.
661
662
663
664
665
666
667
668
669
670
671
672
673
674 Q2
675
676
677
678
QUERIES
Q1: Author: Please verify the affiliation of the authors as typeset.
Q2. Author: Please verify the current location of author “Jan-Willen Marck.”
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
Reasoning About Threats: From Observables
to Situation Assessment
1
2
Gertjan J. Burghouts and Jan-Willem Marck
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Abstract—We propose a mechanism to assess threats that are
based on observables. Observables are properties of persons, i.e.,
their behavior and interaction with other persons and objects. We
consider observables that can be extracted from sensor signals and
intelligence. In this paper, we discuss situation assessment that is
based on observables for threat assessment. In the experiments, the
assessment is evaluated for scenarios that are relevant to antiterrorism and crowd control. The experiments are performed within
an evaluation framework, where the setup is such that conclusions
can be drawn concerning: 1) the accuracy and robustness of an
architecture to assess situations with respect to threats; and 2) the
architecture’s dependence of the underlying observables in terms
of their false positive and negative rates. One of the interesting conclusions is that discriminative assessment of threatening situations
can be achieved by combining generic observables. Situations can
be assessed with a precision of 90% at a false positive and negative
rate of 15% using only eight learning examples. In a real-world
experiment at a large train station, we have classified various types
of crowd dynamics. Using simple video features of shape and motion, we have proposed a scheme to translate such features into
observables that can be classified by a conditional random field
(CRF). The implemented CRF shows to classify successfully the
crowd dynamics up to 80% accuracy.
27
28
29
Index Terms—Architecture, evaluation framework, information
processing, observables, situation understanding, threat recognition.
I. INTRODUCTION
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Q1
1
ITUATION understanding is relevant to the field of security (i.e., robbery and vandalism), public safety (i.e.,
aggression and riots), and health care (i.e., incidents with
elderly people). Technology is making its entrance in solutions
for these domains. More recently, research institutes and technology providers have focused also on the domain of antiterrorism, aiming for solutions that detect threats in an early stage [1].
To detecting a threat at an early stage is important, as it enables
security professionals to mitigate the situation. In this paper,
we focus on technology to recognize the stages that build up to
potential threats.
Our objective is a system to alert for potential threats. To
that end, the system assesses the current situation in order to
S
Manuscript received December 30, 2009; revised September 20, 2010 and
January 25, 2011; accepted March 19, 2011. This paper was recommended by
Associate Editor J. Tang.
The authors are with The Netherlands Organization for Applied Scientific
Research (TNO) Observation Systems, The Hague 2597 AK, The Netherlands
(e-mail: [email protected]; [email protected]).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TSMCC.2011.2135344
Fig. 1. Our framework: outputs of various sensors (JDL level 0) are processed
to the higher abstraction level of observables (JDL level 1). Examples of observables are taken from our earlier work, e.g., trajectory analysis, pose estimation,
color of clothing, carrying an attribute, behavior recognition, and group dynamics [3]. These observables provide information to the situation estimator (JDL
level 2). Specific situations are associated with threats. Situation assessment is
the focus of this paper.
determine whether a threat may occur. Our approach is to assess
the situation from multiple object assessments. These two layers
are adopted from the well-known JDL data fusion model and
its extensions [2]. Object assessments are based on sensors and
intelligence. This is illustrated in Fig. 1.
The output of sensors requires automatic processing up to the
level of meaningful and robust information, which we will refer
to as observables. Intelligence is also taken as an observable.
The observables are object assessments and range from camera
observables (i.e., person carries some object) to geo-observables
(i.e, the person has a short interaction with another person) to
intelligence observables (the suspect is a person with a gray
shirt and black pants). Based on the observables, the situation
is assessed. For instance, the observables that a suspect person
hands over an object to another person who is mixing in the
crowd would result in the situation assessment that somebody
is carrying a suspect object.
In this paper, we discuss the methods to exploit observables
for threat assessment. We motivate our choices for specific pattern recognition methods. In the experiments, the added value
of the implemented method is demonstrated for scenarios that
are relevant to antiterrorism and crowd control. The experiments
are performed within an evaluation framework, where the setup
is such that conclusions can be drawn concerning: 1) the performance and robustness of an architecture to assess threats; and
2) the architecture’s dependence of the underlying observables
in terms of their false positive and negative rates.
1094-6977/$26.00 © 2011 IEEE
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
2
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
A. Previous Work
Many observables have been introduced, which range from
detecting specific objects such as cars [4] to detecting aggression
[5], from generic features of human walking patterns [6] or
highly specific human acts such as shaking hands and kissing [7].
In this paper, we focus on properties and behavior of humans that
can be detected and recognized in video. The goal of our work
is to assess situations beyond the level of isolated behaviors,
which is based on multiple observables.
Previously, other researchers have also considered situation
assessment that is based on multiple observables, e.g., the combination of visual and audio features to improve the detection
of aggression [8]. With an appropriate combination scheme, the
discriminative power of situation assessment can be improved
by using multiple observables. Situation assessment is a challenge, as the situations under investigation can range from instantaneous (e.g., one person gives an object to another person)
to longer periods (e.g., a pickpocket is waiting the best moment
to steal someone’s belongings). In recent literature, researchers
tend to focus on either instantaneous events [9] or on complex,
long-term person–person or person–object interactions [10]. In
this paper, we propose a generic scheme to model and classify
both types of situations.
In this paper, we consider only situations that are known
a priori. This requires examples from which a system can learn.
Interestingly, recent work has shown that it is possible to detect
abnormal situations at run time [11]. Such methods are not
within the scope of this paper.
Another observation from reviewing recent work is that scientific communities have released descriptions and datasets of
behaviors of crowds and individuals. These datasets are very interesting and include recordings of single observables to experiment with, e.g., the PETS and TRECVID benchmarks. However,
such communities have not yet focused on short- and long-term
sequential scenarios. As a consequence, no scenarios are available for antiterrorism and crowd-control applications.
B. Contributions in this Paper
The contribution in this paper is fourfold.
1) Relevant scenarios for antiterrorism and crowd control.
We discuss how expert knowledge of incidents can be
used. This is crucial, as for the particular incidents related
to terrorism, escalation of crowd behavior, and suspect
people and objects, few or no data will be available.
2) A tool to generate a multitude of variations of the scenarios. The variations include statistical and systematic error
sources.
3) A framework to evaluate both 1) the performance of the
situation assessment as a whole and 2) its dependence of
underlying observables. The experiments are performed
by the usage of a conditional random field (CRF). The
outcomes of the evaluation give insight in 1) whether
more (complementary) observables are required to assess
situations successfully; and 2) whether the robustness of
observables needs to be improved in order to be useful for
situation assessment.
4) Given the scenarios and their variations and the framework, we evaluate a system for situation assessment that
was implemented for this paper. The system is described in
Section III that is based on a CRF that takes observables as
input and estimates situations as output. We demonstrate
in Section IV that discriminative power can be achieved
by combining generic observables.
This paper is organized as follows. In Section II, we discuss the scenarios and the tool to generate their variations.
In Section III, we propose the CRF that learns the best relation between observables and the assessment of situations. In
Section IV, the experimental setup is discussed. In Section V,
the CRF is evaluated for the scenarios and their variations. In
addition, the CRF is evaluated against a real-world dataset. Section VI concludes this paper.
140
II. SCENARIOS
141
In this section, we define scenarios to which our system (see
Section III) is applied and on which the experiments will be
based (see Section IV). A scenario is interpreted as a sequence
of states. The state is a description of the situation at a particular
time. That is, “a group of people is agitated.” Note that an
expert may be consulted to associate each state with a level of
escalation or alarm, to bridge the gap to the decision maker. For
each subsequent period, a number of observables are observed.
Observables are observed object properties. That is, “a person
enters the scene.” A scenario is specified by the sequence of
states, their duration, and the observables that are observed for
each subsequent period.
By the previous definition, the observables may vary from
time to time. Different sets of observables may be observed for
the same state at different periods. There is no direct coupling
between states and observables. In Section III, we select a pattern recognition method to learn the best probabilistic relation
between states and observables. In Section IV, we evaluate how
well this learned relation predicts the states for unseen variations
of the scenarios.
Observables may be chosen such that they relate to specific
persons or groups of persons that are of particular interest. That
is, “a suspect person enters the scene” or even more specific
“John enters the scene.” In this paper, we consider observables
that are generic, e.g., “a person enters the scene,” “a suspect
person is in the scene,” etc. Interestingly, in Section IV, we
demonstrate that discriminative power can be achieved by combining generic observables.
142
126
127
128
129
130
131
132
133
134
135
136
137
138
139
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
A. Observables
170
We have selected the following observables: empty square,
one person, two-five persons, many people, people flux +, people flux −, sudden large people flux, tracks toward hotspot,
tracks in hotspot, group formation, group moves, group on collision, individual is avoided by other people, individual’s head
orientation varies, individual carries object, individual has wild
gestures, somebody wears clothes related to a suspect person.
The observables have been adopted from interviews with security professionals and from a training in The Netherlands
171
172
173
174
175
176
177
178
179
BURGHOUTS AND MARCK: REASONING ABOUT THREATS: FROM OBSERVABLES TO SITUATION ASSESSMENT
182
(police academy) on “search, detect, and react” (based on an
Israeli security training). The observables are illustrated in
Fig. 1.
183
B. Antiterrorism and Crowd-Control Scenarios
184
Five scenarios for antiterrorism and crowd control are proposed: square is revolting, aggressive political speaker, extreme
demonstration, neutral, fight between a few people.
The situation at a given time is formalized by one of the
following states: neutral, group with agitator, other people are
avoiding the situation, group is coordinating something, group
is agitated, group riot.
The scenarios are defined by the format: {[state1 , [observable1,1 ,
observable1,2 , . . . ,
observable1,N ],
length1 ],
. . . , [stateT . . .]}. The variable “length” refers to the length of
the period of the current state. The number of observables may
vary per period. These formal specifications will be illustrated
at the end of this section.
First, we present the five scenarios in natural language.
1) Square is revolting: market square, many people moving randomly, two people are loitering, group assembles
just outside the market square, one group member is agitated, the group starts moving, the loitering people join
the group, another group starts to move, one of the group
members carries something, people are avoiding the two
groups, the two groups confront each other, fight, people
fleeing.
2) Aggressive political speaker: somebody start to speak in
the midst of people, people are moving around, people
start to listen, people are walking away.
3) Extreme demonstration: people are avoiding the group that
is demonstrating, the atmosphere is tense, little riot, the
demonstrating group is moving.
4) Fight between a few people: some people are agitated, one
troublemaker is starting a fight, small fight, people are
avoiding the fight, fight stops.
5) Default: neutral situation, people moving, many people
enter buildings, a small group is walking by, square is
getting more crowded, another group walks by.
180
181
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
C. Scenario Generation Including Statistical and Systematic
Errors
The scenarios are transformed into datasets. The inputs are
the observables (i.e, binary values) and the ground truth is the
state per timestep. Four types of noise can be added to the
observables.
1) Statistical errors: random false positives and negatives.
2) Systematic errors: fade in/out: false positives are added in
front of active observables, or at the end.
3) Systematic errors: ambiguity: false positives are added
for a particular observable, and false negatives are added
randomly.
4) Systematic errors: clutter: for each state, a state-dependent
clutter effect is generated, which results in more false
positives and negatives when more clutter is apparent.
3
Fig. 2. Example of increasing statistical errors for the scenario “square is
revolting.” On the horizontal axis, the timesteps are shown. The vertical axis
indicates the observables (each row represents one observable). (a) FPR =
FNR = 0%. (b) FPR = FNR = 10%. (c) FPR = FNR = 20%.
Fig. 3. Example of nontrivial relation between (a) observables and (b) states.
The relation between observables and states is not clear and probabilistic in
nature. Learning the probabilistic relation between observables and states is the
objective of this section and the experiments.
All types of noise will be expressed in terms of false positive
ratio (FPR) and false negative ratio (FNR) in the experiments.
Fig. 2 illustrates the effect of various FPRs and FNRs on the
datasets.
236
III. FROM OBSERVABLES TO SITUATION ASSESSMENT
237
Recall from the previous section, the observables may be
noisy. In this section, we exploit a CRF to learn the best probabilistic relation between states and observables.
To assess the current state of the situation, the observables are
simply considered altogether. The situation assessment is based
on a “bag of observables.” More specifically, in this paper, we
interpret situation assessment as a mapping problem from the
“bag of observables” to a situation state.
The observables are perceptual concepts. The situation states
are implicit concepts that are not directly observed from the data.
Fig. 3 makes this clear. There is no obvious relation between
the observables and states. The situation state can be interpreted
as a hidden parameter of the observables. This makes the state
prediction nontrivial. In Section V, we evaluate whether the
proposed CRF is a good technique to model the probabilistic
relations between observables and situation states.
238
233
234
235
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
A. Conditional Random Field
254
A CRF [12] is a type of discriminative probabilistic model
that is most often used to label or parse the sequential data, such
as natural language text or biological sequences. CRFs are a
more general form of hidden Markov models (HMM, [13]) and
relaxes the Markov assumption over the sequential data that is
characteristic for the HMM. This allows a CRF model to use
information from time separated data [12].
We made the observation that situations in real life evolve
in certain sequential orders. The evolution over time of the
scenario states resembles the kind of probabilistic sequences
255
256
257
258
259
260
261
262
263
4
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
Fig. 4. Example of a CRF. (a) Weights for observables to current state potential. (b) Weights of the previous state to current state potential. (c) Cumulative
probabilities. (d) Estimated state and ground truth.
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
of natural language. These sequences can be modeled by an
undirected graphical model in which each vertex represents a
state. The edges between the vertices can be understood as a
dependence between states. There are two main ways to model
such graphical dependences: HMM and CRF.
We choose the CRF over the HMM, because it has two advantages that are important to our problem. Both advantages are
due to the fact that the CRF model is a discriminative model
instead of generative model [14].
A discriminative model models the conditional probability
P (Y |X). A generative model is more widely applicable as it
can be used to calculate the joint probability P (X, Y ) and all
marginals of the joint, for example, P (X) or P (Y ). A discriminative model is limited to predicting state Y given certain
sequence of observations X. When fitting the model of a generative model, one approximates both P (X) and P (Y |X) instead
of merely P (Y |X) in the case of the discriminative model. This
results in a number of advantages.
1) A discriminative model is independent of P (X). This
means that to estimate P (X) for rarely occurring states
is not a problem. This results in better accuracy for rare
scenarios.
2) It can be trained in one setting and tested in a different setting with another P (X), since it is independent of P (X)
and dependences within P (X).
3) A discriminative model has better predictive performance
of P (Y ), since it is trained on classification of P (Y |X),
instead of modeling the joint probability. It does not need
additional effort to fit data with P (X).
Usage of the CRF consists of two phases: the data-fitting
phase and the inference phase. In the data-fitting phase, the
model parameters are fitted to a training dataset. The best fit is
the maximum of the (log) likelihood of these parameters that are
given the training dataset. The likelihood is a multidimensional
concave function. The optimum of this function gives the best fit
to the parameters. In effect, two matrices are learned, which are
used by the CRF to classify newly observed observables into the
next state: one to capture the dependences between observables
and states, and the other to capture the dependences from the
previous state to the next state. The observable-state matrix also
captures the relative importance of each observable for each
state.
In the inference phase, the learned model is used to estimate
P (Y ) from a given X. When estimating P (Y ), the forward
backward algorithm [13] is used to refine the estimation of
P (Y ). The forward backward algorithm finds the most probable
path in time over the states in P (Y ) [13] and smooths the
probabilities of P (Y ).
An example of a CRF is illustrated in Fig. 4. This example is used in Section V-B, where the precise states and
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
BURGHOUTS AND MARCK: REASONING ABOUT THREATS: FROM OBSERVABLES TO SITUATION ASSESSMENT
325
observables (in a real-world experiment) are further explained.
For the next illustration, the naming of states and observables is
not yet important; here, we want to discuss the underlying mechanism. We illustrate the weighting and potentials of observables
and the estimation of states. The weights of the observables
are shown in Fig. 4(a). The weights of the observables in the
previous state are shown in Fig. 4(b). Together, they influence
the estimation of the current state. Positive weights excite the
corresponding state, whereas negative weights inhibit that state.
This is exemplified by the cumulative probabilities in Fig. 4(c).
The state with highest probability becomes the estimated state
[see Fig. 4(d)].
326
IV. EXPERIMENTAL SETUP
327
328
We model the situation state as the hidden state of the CRF,
and the observables are the observations of the hidden state.
329
A. Conditional Random Field Evaluation Objectives
330
341
We evaluate how well the relation between states and observables can be modeled by a CRF. We are interested in the
following properties of the CRF.
1) Discriminative power: How well does the CRF predict the
situation states for unseen variations of the scenarios? This
is an interesting question, as the observables are generic
and, hence, not tailored to the specific scenarios and their
states.
2) Robustness: How sensitive is the CRF with respect to
statistical and systematic errors?
3) Capacity to generalize: How many examples of a given
set of scenarios are required to obtain a robust CRF?
342
B. Evaluation Tool
343
349
To test the discriminative power of the CRF for situations, we
adapted the Schmidt MATLAB CRF implementation [15]. We
implemented a scenario generator to generate the scenarios (see
Section II-B). In this tool, we also added the capability to include
the four noise modes (see Section II-C) to the observables of
the scenarios in order to measure the performance of the CRF
approach against various noise models.
350
C. Datasets
351
We test each noise variable to increase amounts of noise. The
datasets are generated from the scenarios in Section II and are,
subsequently, contaminated with increasing levels of noise. The
types of noise are statistical (random false positives/negatives)
and systematic (fade ins/outs, ambiguity, and clutter). All types
of noise will be expressed in terms of false positive and negative
ratios in the experiments.
314
315
316
317
318
319
320
321
322
323
324
331
332
333
334
335
336
337
338
339
340
344
345
346
347
348
352
353
354
355
356
357
358
D. Cross Validation
359
For each noise setting, 48 repetitions were generated.
Each repetition contains the entire set of five scenarios (see
Section II-B). For each noise parameter setting, a 12-fold cross
validation will be performed. Each test fold contains four repetitions and is tested against a training set that contains N randomly
360
361
362
363
5
Fig. 5. Results for the experiment with 16, 8, 4, 2, and 1 scenario variations
to learn from. Results are shown for nine fixed samples of FPR and FNR. For
each FPR/FNR sample, we indicate how the CRF classification accuracy drops
as a consequence of fewer learning examples. This is shown by each bar, where
the accuracy drops from red (almost 100%) to green (just above 50%). This
effect is worse for higher FPR/FNR (e.g., the bar in the upright corner). A good
balance between number of examples and performance of the CRF is obtained
for eight examples (see second from the left of each colored bar).
selected repetitions out of the 48 scenarios that are generated
for the current parameter setting. The averaged results from the
folds are the CRF performance results for the selected parameter
setting.
367
E. Training the Conditional Random Field
368
The CRF will be trained at each test fold on the randomly
selected training set. The data fitting, i.e, the optimizing of the
loglikelihood, is performed by the L-BFGS optimization algorithm [16]. This quasi-Newton optimization algorithm is a
standard to optimize large multidimensional optimization problems such as a CRF [17]. The optimization ends when the average error of the training set is smaller than 10−5 or after 500
iterations.
369
376
F. Evaluation Measures
377
The results of the CRF are presented as confusion matrices.
Following conventions, each confusion matrix contains on the
vertical axis the ground truth situation states and on the horizontal axis the predicted states by the CRF. Hence, a diagonal
with high prediction accuracies is aimed for.
We expect that for higher FPRs and FNRs, the predicted states
become unreliable, which will cause the confusion matrices
to have lower values on the diagonal. Each increasing state
index indicates an increasing alarm level. In the case of errors
(i.e., wrongly predicted states), we hope that the errors are not
distributed among all states but remain confined to subsequent
states, such that the perturbing effect on wrongly predicted alarm
levels remains limited.
378
364
365
366
370
371
372
373
374
375
379
380
381
382
383
384
385
386
387
388
389
390
6
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
Fig. 6. Situation assessment: results. Experiment: only one scenario variation to learn from, for various types of noise (see captions directly under the figure) and
the degree of noise (see false positive and negative ratios). Confusion matrices for each of the six situation states are displayed for various FPRs and FNRs. Up to
FPR or FNR of 15%, the CRF is surprisingly accurate to assess situations. (a) Statistical errors. (b) Systematic: fade ins. (c) Systematic: fade outs. (d) Systematic:
fade ins and outs. (e) Systematic: ambiguity. (f) Systematic: clutter.
BURGHOUTS AND MARCK: REASONING ABOUT THREATS: FROM OBSERVABLES TO SITUATION ASSESSMENT
391
V. RESULTS
392
395
First, we focus on experiments on synthetic data where we
can control the sources and amount of noise (see Section V-A).
Second, we show experimental results on real-world data (see
Section V-B).
396
A. Synthetic Data
397
The results in this section are used to evaluate the capacity
to generalize (see Section V-A1), the discriminative power (see
Section V-A2), and robustness (see Section V-A2).
1) Effect of the Size of the Training Set: Our first objective is
to find a good balance between the number of learning examples
to train the CRF and the performance of the CRF. If the number
of learning examples is low, then fewer examples have to be
collected, and training takes less time. However, with fewer
learning examples, the performance of the CRF will be lower.
To evaluate this tradeoff in detail, we investigate the results of
the CRF approach to increase FPR and FNR levels.
Fig. 5 shows the accuracy of the CRF for various sizes of the
training set, which has been decreased from 16 to smaller sets
of 8, 4, 2, and 1 training example(s). The figure shows at each
FPR (horizontal coordinate) and FNR (vertical) a bar of values,
one for each size of the training set. One value represents the
CRF’s accuracy of the situation assessment, where the accuracy
has been averaged equally over the situation states.
With 16 examples and errors up to 30%, the average accuracy
of the CRF is 81%. With fewer errors, up to 15%, the accuracy is
94%. With only one example, the accuracy drops, respectively,
to 50% (was 81%) and 63% (was 94%). A good tradeoff seems to
be eight examples, which achieves only slightly lower accuracy
(i.e., 4% lower than with 16 examples) while using only half
of the examples. Situations can be assessed with a precision of
90% at a false positive and negative rate of 15% using only eight
learning examples.
2) Effect of Statistical and Systematic Errors on the CRF:
Our second objective is to find out where the errors of the CRF
(wrongly classified states) occur and how they are distributed
over the states. To that end, we need a more detailed figure.
Rather than to display a single value that represents the accuracy
as averaged over the states, we display the confusion matrix for a
given size of the training set. As an extreme test case, we display
results for a training set of one example only. In addition, the
results are split into the various noise sources that we identified
earlier, each produces one subfigure [see Fig. 6(a)–(f)]. The
captions indicate the error sources: There are four error sources,
where the fade in/out source is separated into three related but
different cases, which results in six subfigures in total. The line
around each of the confusion matrices indicates the average
predication accuracy, which has been weighted equally over all
states. Fig. 6 altogether is meant to give a detailed insight in
how error sources and the levels of FPR and FNR deteriorate
the results.
Up to FPR or FNR of 15% for any type of noise, the CRF
is surprisingly accurate to assess situations. The accuracy starts
to deteriorate above 15% FPR or FNR or both. Interestingly,
the CRF is not very sensitive to fading effects or ambiguity (we
393
394
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
7
Fig. 7. Video descriptors to capture the crowd dynamics in a compact way.
This figure shows 12 out of the 104 descriptors.
TABLE 1
RESULTS FOR THE CLASSIFICATION OF CROWD DYNAMICS BY A CRF
introduced correlations) within the observables. On the contrary,
the CRF is sensitive to random noise and to clutter (state-specific
levels of random noise). The clutter case is most difficult, as the
noise is different for each state. As a consequence, the noise
increases and decreases over time. The CRF is least robust to
noise levels that vary over time.
451
B. Real-World Data
452
In this section, we consider an experiment that is based on
real data. The experiment is relevant for security and safety
applications. We have recorded and annotated 4-h video data
at the second largest train station in The Netherlands. The
video data includes many and diverse behavioral patterns of
the crowd, of which a limited set have been classified by a CRF.
For this real-world experiment, we have adopted state-of-the-art
video features that capture local structure and temporal dynamics. Given these features, we adopt a quantization scheme to
translate the many and high-dimensional video features into a
fixed number of low-dimensional observables. Based on these
spatiotemporal observables, various crowd dynamics are classified by the CRF. The goal is to classify three types of crowd
dynamics: “normal” (10–100 people), “abandoned” (0–10 people), “rush” (many people moving, to go to or leave the tram
platforms). In the recorded video, these three event classes are
correspondingly labeled by hand.
1) Describing the Crowd: Video Features: To describe the
crowd, we aim to capture the temporal dynamics in the crowd
(the general movement of, e.g., heads, shoulders, and legs). We
tailor existing video features for this task. We start with optical flow [18], which is computed at each pixel in the video
images, that yields at each pixel a magnitude and orientation.
We summarize these flows in spatial bins to reduce the amount
of data. We choose 104 fixed bins that are distributed equally
453
446
447
448
449
450
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
8
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS
Fig. 8. Examples of errors for each type of crowd dynamics, where the CRF misclassified the state of the crowd (between parentheses). (a) “Normal” (“Rush”).
(b) “Abandoned” (“Normal”). (c) “Rush” (“Normal”).
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
over the image. The 104 spatial bins are obtained as follows.
The video is of HD quality, therefore each image frame is
1080 × 1440. We aim to sample approximately every 100 pixels. Discounting the border areas, the grid becomes 8 × 13,
which yields 104 spatial bins. Next, in each spatial bin, the flow
vectors are binned into eight orientations and four magnitudes.
This approach is inspired by other work, e.g., scale-invariant
feature transform [19], where discriminative yet compact image features were extracted from images by summarizing many
pixel values into few orientations and magnitudes. For adequate
continuous weighting of the optical flows, the contribution of
each flow vector to each bin is determined by a kernel-based
weight. With our approach, 104 × 8 × 6, i.e., 104 descriptors of
length 48 are obtained per video frame. This procedure captures
the dynamics of the crowd in a compact way. An illustration is
depicted in Fig. 7.
2) Transforming the Video Features Into Observables: We
experimentally established that the CRF is able to learn the three
types of crowd dynamics when the input is bound at maximum
40 observables. Recall that the video features yield 104 × 48 =
4992 values, i.e., much higher than the maximum of 40 observables. Hence, a translation is needed that reduces the video
features into approximately 40 observables. To do so, we adopt
a method of quantization: visual codebooks [20]. In a visual
codebook, the descriptors are assigned to a smaller selection
of representative descriptors (which are also called primitives).
This yields a histogram, of which the length is determined by
the number of primitives. We choose eight primitives, that result
in a histogram of length 8. Next, we discretize each bin (of the
eight-bin histogram) into four levels, to obtain 8 × 4 binary
values. Recall that the CRF is especially suited to learn from
binary values. In this translation procedure, the 104 multivalued
video features are translated into a 32 binary values. These 32
binary values are the observables, on which the CRF is trained
and tested.
3) Results: Classification of Crowd Dynamics: All examples
of the three classes are taken from the 4-h videos. These examples are subsampled at 2 frames/s. This is a choice to reduce
the amount of stored data and a limitation posed by processing
speed. We argue that 2 frames/s is sufficiently fast for awareness on crowd dynamics, yet the variation between samples is
significant. For each frame, the video features are computed and
translated into observables. Together with the classification of
observables into a next state, this procedure can be done at a
sampling rate of 2 frames/s. The trained CRF, with its weights
for the observables and state transitions, has been shown in
Fig. 4 and was discussed in Section III.
For the training of the CRF, the transitions between states
are essential. In our real-world dataset (i.e., the examples of
the classes), we found that the number of transitions is limited.
This poses a challenge to make distinct yet rich training and test
sets. We choose to solve this as follows. We perform a fourfold
cross validation. We subsample every Nth encoded video frame
to be included in the Nth-fold. This way, the folds are disjunct
and different enough as they are at minimum 2 s apart (given
the sampling of 2 frames/s and fourfolds). That is, the scene is
changing rapidly. Going from one video frame to the next, with
2 s in between, gives an enormous variation in the images. People
have walked significantly, new people have entered, other people
have exited the scene, and people have got closer or further apart
from other people. Following conventions, we train on onefold
and test on another. This experiment is repeated four times.
The results are summarized by the confusion matrix in
Table I. Interestingly, the CRF classifies some states that are
more accurate than others. The CRF has shown to successfully
classify the crowd dynamics up to 80% accuracy.
Examples of errors are shown in Fig. 8. An error that occurs
regularly is a badly timed state transition of the estimation.
Usually, the transition is to the right state but is too early or too
late. This is illustrated in Fig. 8(b) and (c). Sometimes there are
“true misclassifications;” this is illustrated in Fig. 8(a).
VI. CONCLUSION
We have proposed scenarios that are relevant for antiterrorism
and crowd control. Given the scenarios, and their variations
including various kinds of errors, we have assessed situations
that are based on observables with the objective to recognize
threats.
We have chosen the CRF as a method to learn probabilistically the relation between observables and states (as the hidden
parameter). A good size for the training set was shown to be
eight examples for each of the five scenarios, which achieves
only slightly lower accuracy (i.e., 4%) than with 16 examples
while using only half of the examples. Situations can be assessed
with a precision of 90% at a false positive and negative rate of
15% using only eight learning examples.
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
BURGHOUTS AND MARCK: REASONING ABOUT THREATS: FROM OBSERVABLES TO SITUATION ASSESSMENT
580
We have investigated the extreme test case of using a training
set of only one example to find out where the CRF becomes
inaccurate. Up to FPR or FNR of 15% for any type of noise
(random, fade in/out, ambiguity, and clutter), the CRF is surprisingly accurate to assess situations. The accuracy starts to
deteriorate above 15% FPR or FNR or both. The CRF is not
very sensitive to fading effects or ambiguity but is sensitive to
random noise and to clutter (i.e., state-specific levels of random
noise).
In a real-world experiment at a large train station, we have
classified various types of crowd dynamics. Using simple video
features of shape and motion, we have proposed a scheme to
translate such features into observables that can be classified by
a CRF. The CRF has shown to successfully classify the crowd
dynamics up to 80% accuracy.
Overall, we conclude this paper with the interesting finding
that discriminative power can be achieved by combining multiple, generic, and simple observables.
581
ACKNOWLEDGMENT
582
585
The authors would like to thank Dr. R. den Hollander to
provide the camera-based observables and their characteristics.
They are also grateful to Dr. K. Schutte for useful discussions
on the inference of the situation that is based on observables.
586
REFERENCES
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
[1] M. M. Kokar, “Situation awareness: Issues and challenges,” in Proc. Int.
Conf. Inf. Fusion, 2004, pp. 533–534.
[2] J. Llinas, C. Bowman, G. Rogova, and A. Steinberg, “Revisiting the JDL
data fusion model II,” in Proc. Int. Conf. Inf. Fusion, 2004, pp. 1218–1230.
[3] G. J. Burghouts, B. Broek, B. G. Alefs, E. den Breejen, and K. Schutte,
“Automated indicators for behavior interpretation,” in Proc. Int. Conf.
Crime Detect. Prevent., 2007, pp. 1–6.
[4] S. Agarwal, A. Awan, and D. Roth, “Learning to detect objects in images
via a sparse, part-based representation,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 26, no. 11, pp. 1475–1490, Nov. 2004.
[5] A. Datta, “Person-on-person violence detection in video data,” in Proc.
Int. Conf. Pattern Recognit., 2002, pp. 433–438.
[6] D. M. Gavrila, “The visual analysis of human movement: A survey,”
Comput. Vis. Image Understand., vol. 73, no. 1, pp. 82–98, 1999.
[7] I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, “Learning realistic
human actions from movies,” in Proc. IEEE Int. Conf. Comput. Vis. Pattern
Recognit., 2008, pp. 1–8.
[8] W. Zajdel, D. Krijnders, T. Andringa, and D. Gavrila, “CASSANDRA:
Audio-video sensor fusion for aggression detection,” in Proc. Int. Conf.
Adv. Video Signal Based Surveillance, 2007, pp. 200–205.
[9] L. Duan, D. Xu, I. W. Tsang, and J. Luo, “Visual event recognition in
videos by learning from web data,” in Proc. IEEE Int. Conf. Comput. Vis.
Pattern Recognit., 2010, pp. 1959–1966.
[10] D. Küttel, M. Breitenstein, L. van Gool, and V. Ferrari, “What’s going on?
discovering spatio-temporal dependencies in dynamic scenes,” in Proc.
IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2010, pp. 1951–1958.
[11] T. Xiang and S. Gong, “Video behaviour profiling for anomaly detection,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 5, pp. 893–908, May
2008.
[12] A. McCallum and C. Sutton. (2006). “An introduction to conditional random fields for relational learning,” in Introduction to Statistical Relational Learning, L. Getoor and B. Taskar, Eds. Cambridge, MA: MIT Press [Online]. Available: http://www.cs.umass.edu/
mccallum/papers/crf-tutorial.pdf
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
583
584
9
[13] L. Rabiner, “A tutorial on hidden Markov models and selected applications
in speech recognition,” Proc. IEEE, vol. 77, no. 2, pp. 257–286, Feb. 1989.
[14] I. Ulusoy and C. Bishop, “Generative versus discriminative methods for
object recognition,” in Proc.IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, vol. 2, pp. 258–265.
[15] M. Schmidt. (2008). “CRF toolkit in MATLAB,” [Online]. Available:
http://people.cs.ubc.ca/schmidtm/Software/
[16] D. C. Liu and J. Nocedal, “On the limited memory BFGS method for large
scale optimization,” Math. Program., vol. 45, no. 3, pp. 503–528, 1989.
[17] F. Sha and O. Pereira, “Shallow parsing with conditional random
fields,” in Proc. HLT-NAACL, 2003, pp. 213–220 [Online]. Available:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.10.9849
[18] B. K. P. Horn and B. G. Schunck, “Determining optical flow,” Artif. Intell.,
vol. 17, nos. 1–3, pp. 185–203, 1981.
[19] D. G. Lowe, “Object recognition from local scale-invariant features,” in
Proc. Int. Conf. Comput. Vis., 1999, pp. 1–8.
[20] B. T. F. Jurie, “Creating efficient codebooks for visual recognition,” in
Proc. Int. Conf. Comput. Vis., 2005, pp. 604–610.
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
Gertjan J. Burghouts received the Ph.D. degree
from the University of Amsterdam, Amsterdam, The
Netherlands, in 2007 on the topic of visual recognition of objects and their motion, in realistic scenes
with varying conditions.
He is currently a Lead Research Scientist of Visual Pattern Recognition with The Netherlands Organization for Applied Scientific Research (TNO), The
Hague, The Netherlands. He studied artificial intelligence at the University of Twente during 1997–2002
with a specialization in pattern analysis and human–
machine interaction. Since 2007, he has been the Principal Investigator of automated understanding of human behavior based on sensory perception. He is
the Principal Investigator of a DARPA project named CORTEX (2.3M), about
recognition of events and behaviors. He has written papers on this topic in
internationally renowned journals, e.g., the IEEE TRANSACTIONS ON IMAGE
PROCESSING, the Computer Vision and Image Understanding, the International
Journal of Computer Vision, and International Conference on Crime Detection
and Prevention. His work has been cited mote than 200 times since 2005.
Dr. Burghouts received an award from the Netherlands Association of Engineers for the best innovative project in 2007.
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
Jan-Willem Marck received the M.Sc. degree in artificial intelligence from the University of Groningen,
Groningen, The Netherlands, in 2006, with a specialization in autonomous systems.
He is currently a Research Scientist of Artificial Intelligence with the Department of Distributed Sensor
Systems, The Netherlands Organization for Applied
Scientific Research (TNO), The Hague, The Netherlands. Since 2006, he has been a Research Scientist
on various topics, e.g.,sensor information fusion, relevance of information, human–machine interaction
and human behavior classification based on state estimation methods using sensory data. He has (co-)authored a number of papers and managed projects on
these topics.
661
662
663
664
665
666
667
668
669
670
671
672
673
674 Q2
675
676
677
678
QUERIES
Q1: Author: Please verify the affiliation of the authors as typeset.
Q2. Author: Please verify the current location of author “Jan-Willen Marck.”