Arbitrary Trajectories Tracking using Multiple Model Based Particle Filtering in
Infrared Image Sequence
Mukesh A. Zaveri S. N. Merchant Uday B. Desai
SPANN Lab, Electrical Engineering Dept., IIT Bombay - 400076.
Email address : [mazaveri,merchant,ubdesai]@ee.iitb.ac.in
Abstract
Particle filtering is being investigated extensively due to
its important feature of target tracking based on nonlinear and non-Gaussian model. It tracks a trajectory with
a known model at a given time. It means that particle filter tracks an arbitrary trajectory only if the time instant
when trajectory switches from one model to another model
is known apriori. Because of this reason particle filter is
not able to track any arbitrary trajectory where transition
from one model to another model is not known. For real
world application, trajectory is always random in nature
and may follow more than one model. In this paper we propose a novel method, which overcomes the above problem.
In the proposed method an interacting multiple model based
approach is used along with particle filtering, which automates the model selection process for tracking an arbitrary
trajectory. We have utilized nearest neighbor (NN) method
for data association, which is fast and easy to implement.
In the proposed approach, there is no need to have apriori
information about the exact model that a target may follow.
1. Introduction
The standard Kalman filter gives an optimal estimate
with linear and Gaussian model assumption. For nonlinear
case, one typically uses the extended Kalman filter (EKF)
[18]. EKF requires Hessian and Jacobian matrices to be
evaluated and it may lead to divergence. The unscented
Kalman filter (UKF) [16] is being used to preserve the nonlinearity of the model by properly chosen sigma points. It
also assumes state to be a Gaussian distributed. In recent
times, for nonlinear and non-Gaussian models, particle filtering has been proposed as an alternative to the extended
Kalman filter [12, 10, 8, 11, 5]. Basic philosophy behind
tracking using Bayesian approach is to propagate and update the probabilistic density function of the state to be
tracked. Basic concept in particle filter is to represent a
pdf of the state by a set of random samples, called particles [2, 13]. Each particle is assigned a weight, known as
importance weight. As the number of particle is very large,
it represents true pdf of the state. Using these particles and
weights, it is possible to estimate any moment of pdf of a
state vector. Comparison of particle filter with other nonlinear filters can be found in [1].
Particle filtering has been extended to multiple target
tracking and different methods have been proposed for this
problem [4, 3, 17]. In [4] a stochastic simulation Bayesian
method has been proposed for multitarget tracking but simulation and results are depicted for single target only. In
[3] the data association problem is treated as an incomplete data problem. It treats observation to track assignment as missing data and Gibbs sampler method is used to
estimate the assignment probability. Particles are sampled
from the probability density function (pdf) representing the
combined state of all the targets in [3]. In multiple target
scenario the number of state parameters varies from target
to target. Moreover the computational complexity of this
method increases exponentially as number of measurements
increase, and number of targets to be tracked increase. Estimating the joint probability distribution of the state of all
targets makes the problem intractable in practice. Particle
filter along with JPDAF for multiple target tracking is described in [6]. It avoids iteration as required in Gibbs sampler method used in [3]. But JPDAF requires to evaluate all
possible data association events, which is again time consuming.
Particle filter, of course, needs the knowledge of the
model to track a target; but more importantly it needs to
know the time instant when trajectory switches from one
model to another model. Now, if the target movement is
random, then the trajectory formed by the target is arbitrary
and there is no apriori knowledge about which model to use
at a given time and when to switch. In such a situation, particle filter suffers from the degeneracy problem, and the pdf
of the state collapses. To track an arbitrary trajectory, it is
incumbent to use a multiple model based approach, namely,
IMM filtering. In this paper we propose a novel method
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04)
0-7695-2108-8/04 $ 20.00 © 2004 IEEE
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 3, 2008 at 01:33 from IEEE Xplore. Restrictions apply.
that works with multiple nonlinear or non-Gaussian state
space models to track arbitrary trajectories. It is important
to note that the proposed approach does not require any
apriori information about the exact models which targets
may follow for particle filtering. We performed the simulations where trajectories are generated using B-spline function and tracked successfully using the proposed method. In
the proposed method an interacting multiple model (IMM)
[9] based approach is used along with particle filtering,
which automates the model selection process. In the proposed approach, a nearest neighbor (NN) method is used
for data association. NN method is fast and easy to implement in comparison with joint probabilistic data association
filter and multiple hypothesis tracking methods [18]. The
proposed method is able to track multiple trajectories in the
presence of dense clutter, and does not require the apriori
knowledge of the time when the trajectory switches form
one model to another.
2. Multiple Model Based Particle Filtering
In this section, the problem is described in multimodel framework to track both maneuvering and nonmaneuvering targets. The algorithm is divided into three
major parts. First part is the data association. As soon as
observation set is available, using nearest neighbor method
an observation is assigned. It is followed by the update
state, which evaluates particle weights and updates target
state for each model. In the last step, the model probability is evaluated and target state is predicted for each model.
Let, Y and X denote the observation process and the state
process respectively. Y t is a set of all observation set for
time 1 ≤ t, where t is current time. Y (t) and X(t) represent the realization of observation process and state process at time t, respectively. At time t, a vector of measurements is received, Y (t) = (yt1 , . . . .ytmt ), where mt
represents the number of measurements received. Similarly,
X(t) = (xt (1), . . . , xt (Nt )).
Here, Nt is the total number of targets at time instant t and
xt (s) (1 ≤ s ≤ Nt ) represents the combined state estimate
for target s. xm
t (s) is the state estimate of target s due to
model m at time t, where 1 ≤ m ≤ M and M is the total
number of models used to track a particular target.
It is important to note that the proposed method does not
need to have any apriori information about the exact dynamic models which target may follow at a given time. This
approach is completely different compared to conventional
particle filtering. The conventional particle filters needs the
knowledge of the model to track a target. For each model
the probability density function is approximated by a set
of samples, called particles. Each particle is assigned a
weight, known as importance weight. For every model, particle weights [2, 13] are evaluated at each time instant inde-
pendently. If the trajectory does not follow any model at a
given time instant its probability density function may collapse or all importance weights may have negligible value
for respective particles. At this time instant, particles are
initialized using the mix state vector given by the IMM filtering method and hence, it is possible to follow an arbitrary trajectory. IMM filtering mixes the state vector from
different models using model probabilities. When a trajectory switches from one model to another, particle weights
have marginal values if it matches a model and hence, it is
reflected in model probability. Mix state vector takes care
of the likelihood of a model for a given trajectory. Model
probability is calculated using an observation assigned by
nearest neighbor method. Nearest neighbor (NN) method
is used for data association because it is easy to implement
and computationally efficient. NN method is based on minimum statistical distance. For our proposed method, innovation error is used as a statistical distance. NN method is
implemented using Munkres’ optimal data assignment algorithm [14]. Inclusion of IMM based approach allows us
to track an arbitrary trajectory with different models. The
basics of particle filter is described very briefly in the following section.
2.1. Basics of Particle Filter
For particle filter, a set of weighted particles are drawn
from the posterior pdf. The pdf can be approximated using
discrete sums in place of integrals as follows:
p(xt |Y1:t ) =
N
1 δ i (dxt )
N i=1 xt
(1)
where Y1:t = {y1 , y2 , . . . , yt } is a set of measurements
up to time t, yt is a measurement available at time t and
xit (1 ≤ i ≤ N ) represents ith sample drawn from pdf at
time t. Here, N is the total number of samples used to represent pdf and δxit is the Dirac delta function. Based on this
approximation, any moment can be evaluated [7]. It can be
written as
N
E(gt (xt )) =
gt (xt )p(xt |Y1:t )dxt ≈ N1 i=1 gt (xit )
(2)
The particles xit are assumed to be independent and identically distributed. As N → ∞ estimation converges to its
true value [13]. Generally it is difficult to sample from the
posterior pdf. But it is easy to sample from the proposal
distribution function q(xt |Y1:t ). There are various method
for sampling from the proposal function. Sequential importance sampling (SIS) is one these technique. Each particle
is weighted by an importance weight and it is given by,
wt (xt ) =
p(yt |xt )p(xt |Y1:t−1 )
q(xt |Y1:t )
=
p(yt |xt )p(xt |xt−1 )
q(xt |Y1:t )
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04)
0-7695-2108-8/04 $ 20.00 © 2004 IEEE
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 3, 2008 at 01:33 from IEEE Xplore. Restrictions apply.
(3)
The proposal function should be chosen to minimize the
variance of the importance weights [2, 13]. The most popular choice for proposal function [12] is
q(xt |xt−1 , yt ) = p(xt |xt−1 )
(4)
The problem with the above choice is that the most recent
measurement is not incorporated but it is very easy to implement. This simplifies the evaluation of weights wt (xt )
and written as
wt (xt ) = p(yt |xt )
(5)
and it can be shown that expectation in (2) can be written
as,
N
i
i
(6)
E(gt (xt )) =
i=1 gt (xt )w̃t (xt )
i
w
where w̃ti = N t
j=1
wtj
is normalized weight. The major
problem with the above technique is that the variance of the
importance weights increases over time. It results into degeneracy phenomenon. To overcome this degeneracy problem a resampling is performed to eliminate the particles
with low weights and multiply particles with high weights.
There are number of resampling methods: sampling importance resampling (SIR), residual resampling and minimum
variance sampling. In our proposed method, residual resampling method is used because it is computationally less
expensive and the variance is small than that given by SIR
method. With this problem formulation, the proposed algorithm to track arbitrary trajectories using multiple model
based particle filtering is as follows.
2.2. Proposed PF IMM Algorithm
The proposed PF IMM algorithm is as follows:
1. Initialize particles for each model m(1 ≤ m ≤ M )
by drawing samples xi (i = 1, . . . , N ) from the prior
pm (x0 ) for each target s(1 ≤ s ≤ Nt ). Initialize
model probability and transition probability. Here, Nt
represents the total number targets at time t.
2. For time t = 1, 2, . . .
(a) Nearest neighbor based observation to track fusion:
• For each target s(1 ≤ s ≤ Nt )
i. Evaluate statistical distance for an observation falling inside the validation region
formed using combined predicted state vector with respect to each model m
T (−1)m
dist = (yt − h(xm
(yt − h(xm
t )) R
t ))
where Rm is covariance matrix for observation noise.
ii. For a given target s; select a minimum statistical distance dist found among the models. A matrix is formed using this minimum
statistical distance. Rows of a matrix indicate targets and columns represent the observations. It is followed by Munkres’ optimal data assignment algorithm [14]. The
assumption made in the Munkres’ algorithm
is that only one observation is assigned to a
single track. It assigns an observation to a
track based on minimizing total sum of all
statistical distances.
iii. Using this observation evaluate likelihood
of each model with respect to this observation i.e. for target s for each model
m(1 ≤ m ≤ M ) evaluate
Ls (m) = pm (yt |xt )
This measurement is used for updating weights.
(b) Update weights:
• For each target s(1 ≤ s ≤ Nt ) for each model
m(1 ≤ m ≤ M ) using wti = pm (yt |xit ). In the
N
j
case of degeneracy, i.e.
j=1 wt equals zero,
then go to the next model otherwise perform the
following steps for that model
i
w
i. Normalize weights: w̃ti = N t
j=1
wtj
ii. Perform Residual Resampling to obtain N particles distributed according to
pm (xt |Y1:t ). Set wti = w̃ti = N1
iii. Obtain estimation E(gt (xm
t )) for model m
(in our case mean of a state) by
N
1
mi
E(xm
t )= N
i=1 xt
(c) Propagate particles:
• For each target s(1 ≤ s ≤ Nt )
i. Update model probability m = 1, . . . , M :
m
µm
t|t−1 L
where
µm
t = µi
Li
i t|t−1
M
i
µm
i=1 ξim µt−1
t|t−1 =
ii. Combined
update for target s:
state
M
m
xt (s) = m=1 xm
t|t µt
iii. Mix state
for model m:
initialization
M
x0m
= i=1 xit|t µi|m
s
where µi|m = ξim µit /µm
t+1|t and
M
m
i
µt+1|t = i=1 ξim µt
N
If j=1 wtj equals zero for model m then
initialize the particles with mix state x0m
s
otherwise go to the next step.
iv. Predict particles:
For each model m(1 ≤ m ≤ M ); draw a
process noise sample vti from a pdf p(vt ) and
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04)
0-7695-2108-8/04 $ 20.00 © 2004 IEEE
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 3, 2008 at 01:33 from IEEE Xplore. Restrictions apply.
propagate particle xi i = 1, . . . , N by
xit+1 = f (xit ) + vti
v. Predicted state for model m:
N
1
mi
E(xm
)
=
t+1
i=1 xt+1
N
vi. Combined state
for target s:
M prediction
m
xt+1 (s) = m=1 xm
t+1 µt
3. Simulation Results
Synthetic IR images were generated using real time temperature data[15]. For simulation, the generated frame size
is 1024 × 256 and very high target movement of ±20 pixels per frame. Maneuvering trajectories are generated using
B-Spline function. It is important to note that these generated trajectories do not follow any specific model. In our
simulations, we have used constant acceleration (CA) and
Singers’ maneuver model (SMM) for IMM. For all trajectories, filters are initialized using positions of the targets in
the first two frames. Two hundred particles are used for all
simulations.
Figures 1 and 2 depict the results of tracking in the presence of different clutter level using the proposed algorithm
for ir44 and ir50 clips respectively. In these figures true
trajectories are represented by solid line and predicted trajectories are depicted by dotted line with the same color.
The model probability plots for target 1 are shown in figures 3-(a) for ir44 clip with 0.05% clutter and 3-(b) for ir50
clip with 0.02% clutter respectively. Due to space limit the
tracked trajectory for ir49 clip with 0.03% clutter and the
combined mean prediction error plot for different trajectories are not shown here.
Table A: Mean Prediction Error in Position
using combined target state.
Traj.
ir44 clip
without clutter with 0.05% clutter
1
0.9375
0.9375
2
1.0915
1.0915
Traj.
ir49 clip
without clutter with 0.03%clutter
1
0.7689
1.0393
2
0.8447
0.7956
Traj.
ir50 clip
without clutter with 0.02%clutter
1
0.6749
0.7662
2
0.6298
0.8587
Table A depicts mean prediction error in position for each
trajectory in different clips without clutter and with clutter.
The key point in the proposed method is that during tracking
the time instant when transition from one model to another
model takes place is not known and is random in nature, and
there is no apriori information about the model that target
obeys. From our extensive simulations, we have also noted
that the application of a single model, either CA or SMM,
with particle filtering for tracking fails.
4. Conclusion
From simulation results it is concluded that in absence
of any apriori information about the exact dynamic models which targets may follow at a given time, our proposed
method is able to track multiple arbitrary target trajectories
in the presence of dense clutter. Only two filters, namely,
CA and SMM filters were used in IMM mode to track random movement of targets.
References
[1] A. Farina, B. Ristic and D. Benvenuti. Tracking a Ballistic Target: Comparison of Several Nonlinear Filters.
IEEE Transactions on Aerospace and Electronics Systems,
38(3):854–867, July 2002.
[2] Arnaud Doucet et. al.
On Sequential Monte
Carlo Sampling Methods for Bayesian Filtering.
Technical Report CUED/F-INFENG/TR 310:
http://citeseer.nj.nec.com/doucet00sequential.html, 1998.
[3] C. Hue, J-P. Le Cadre and P. Pérez. Tracking Multiple Objects with Particle Filtering. IEEE Transactions on
Aerospace and Electronics Systems, 38(3):791–812, July
2002.
[4] D. Avitzour. Stochastic simulation Bayesian approach to
multitarget tracking. IEE Proc. - Radar, Sonar Navig.,
142(2):41–44, April 1995.
[5] David J. Bullock and John S. Zelek. Real-Time Tracking
for Visual Interface Applications in Cluttered and Occluding Situations. Proc. of 15th International Conf. on Vision
Interface, 2002.
[6] Dirk Schulz, Wolfram Burgard et. al. Tracking Multiple
Moving Objects with a Mobile Robot. Proc. of IEEE Computer Society conf. on Computer Vision and Pattern Recognition (CVPR), 2001.
[7] J. E. Handschin and D. Q. Mayne. Monte Carlo techniques
to estimate the conditional expectation in multi-sate nonlinear filtering. International Journal of Control, 9(5):547–
559, 1969.
[8] R. Karlsson and N. Bergman. Auxiliary Particle Filters for
Tracking a Maneuvering Target. Proc. of 39th IEEE conf. on
Decision and Control, pages 3891–3895, Dec. 2000.
[9] X. R. Li and Y. Zhang. Numerically Roubst Implementation of Multiple-Model Algorithms. IEEE Transactions
on Aerospace and Electronic Systems, 36(1):266–277, Jan.
2000.
[10] Neil Gordon. A Hybrid Bootstrap Filter for Target Tracking in Clutter. IEEE Trans. on Aerospace and Electronics
Systems, 33(1):353–358, Jan. 1997.
[11] Neil Gordon and Simon Maskell. Efficient Particle Filters
for Joint Tracking and Classification. In SPIE - Signal and
Data Processing of Small Targets, volume 4728, April 2002.
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04)
0-7695-2108-8/04 $ 20.00 © 2004 IEEE
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 3, 2008 at 01:33 from IEEE Xplore. Restrictions apply.
Figure 1. Tracked trajectories at frame number 57 - ir44 clip (0.05% clutter).
Figure 2. Tracked trajectories at frame number 44 - ir50 clip (0.02% clutter).
(a)
(b)
Figure 3. Model Probability plot for target 1 (a) for ir44 clip (b) for ir50 clip.
[12] N.J. Gordon, D.J. Salmond and A.F.M. Smith. Novel approach to nonlinear/non-Gaussian Bayesian state estimation.
IEE Proceedings-F, 140(2):107–113, April 1993.
[13] Rudolph van der Merwe et. al. THE UNSCENTED PARTICLE FILTER. www.ece.ogi.edu/ rvdmerwe - Technical
Report CUED/F-INFENG TR 380, Aug. 2000.
[14] Samuel S. Blackman. Multiple-Target Tracking with Radar
Applications. Artech House, Inc., Boston, 1986.
[15] Shankar T. More, Avinash A. Pandit, S.N. Merchant and
U.B. Desai. Synthetic IR Scene Simulation of Air-borne
Targets. In Proceedings of 3rd Conference ICVGIP 2002,
pages 108–113, Ahmedabad, India, Dec. 2002.
[16] Simon J. Julier. A Skewed Approach to Filtering. In Proceeding of SPIE, Signal and Data Processing of Small Targets, volume 3373, pages 271–282, April 1998.
[17] Simon Maskell, Malcom Rollason and Neil Gordon. Efficient Particle Filtering for Multiple Target Tracking with
Application to Tracking in Structured Images. SPIE - Signal
and Data Processing of Small Targets, 4728, April 2002.
[18] Y. Bar-shalom and T. E. Fortmann. Tracking and Data Association. Academic Press, 1989.
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04)
0-7695-2108-8/04 $ 20.00 © 2004 IEEE
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on December 3, 2008 at 01:33 from IEEE Xplore. Restrictions apply.