A Target Tracking System based on Radar and
Image Fusion
Zhiqiang Hou
Chongzhao Han
Institute of Automation
School of Electronics and
Information Engineering
Xi'an Jiaotong University
Xi'an
710049
P.R. China
[email protected]
Institute of Automation
School of Electronics and
Information Engineering
Xi'an Jiaotong University
Xi'an
710049
P.R. China
[email protected]
Abstract: A target tracking system based on radar and
image information fusion is studied in this paper, the idea
of " feedback + guide " is presented. The fusion
information coming from fusion center is fed to the local
sensors as their initial value. The distance element and
angle elements are used to guide image sensor to track
target. Angle elements are used to guide radar to do the
same thing. This system really combines the merit of radar
and image sensor. Simulations show that this system is
effective.
Keywords: target tracking; fusion; radar; image
1
because it is very complicated and need a broad
knowledge background.
Aim at this problem, radar and image fusion tracking
system based on one and the same carrier is proposed.
Based on the advanced tracking algorithms and the fusion
strategy, the idea of " feedback + guide" is presented in
this paper. The fusion information coming from fusion
center is fed to the local sensor as initial value, the
distance element and the angle elements are used to guide
image sensor to track target, the angle elements are used to
guide radar to do the same thing. This system really
combines the advantages of radar and image sensor.
Simulations show that this system is very effective.
Introduction:
2
There are many problems only using a single sensor to
detect, recognize and track target. For instance, radar can
detect object very quickly, and can use mature algorithm
to track the detected object, meanwhile, radar can give a
highly accurate distance measure. But radar has a weak
ability to recognize the object and easily to give a false
alarm. On the other hand, because of an active-sensor,
radar easily makes its carrier discovered by the enemy.
Image sensor, such as infrared sensor or visible light CCD,
is not sensitive and quick enough to detect the target, and
almost doesn't have the ability to measure range. Only the
knowledge about object's scales has been known. But
image sensor can give the object shape information, so it is
easy to recognize the object. At the same time, image
sensor can give a accurate azimuth and elevation angle
about target, it is also a passive-sensor, so it is more
covered than radar. More attention has been paid on how
to fuse these two sensors using each merits in tracking
area. But this problem is not solved entirely so far,
System structure
2.1
System work condition
The system consists of radar and image sensor
(infrared or CCD), radar can detect a remote distance
object, but is poor to detect a close object, but image
sensor is reverse. The data accepted by two sensors are
fused in their superposition area; figure1 is a sketch map
about superposition area called fusion area.
Radar blind area
Radar
Fusion area
Camera
a
b
Image blind area
c
d
Figure1. Radar and image sensor's fusion area
(a-c is image sensor's work area , c-d is image sensor's blind area
a-b is radar's blind area, b-d is radar's workarea, b-c is fusion area.)
1426
How to choose the tracking coordinates is very
important in tracking system, because the tracking
coordinates have a direct effect on tracking precision and
tracking stability. In this paper, inertial Cartesian
coordinates with the origin at the carrier's centroid are
adopted.
In order to make fusion simple, radar and image sensor
have the same sampling time.
2.2
System structure and work flow
System structure is given in figure2. Radar subsystem
deals with the data in the data-level, then makes the
feature-level vectors and finally sends them to fusion
center. Fusion center is showed by broken line frame in
figure2, it includes three parts: Measurement Data
Association, Measurement Data Fusion and Tracking
Filter. Fusion center decides whether it needs to start
image sensor or not. Once needs to start it, fusion center
can send the related information to image sensor
subsystem to make it works. Image sensor subsystem, the
same as radar, deals with the data in the pixel-level, then
forms the feature-level vectors and sends them to fusion
center. Fusion center fuses the data in the feature-level and
sends the results to decision center.
Radar
Image
Measurement Data Association
Measurement Data Fusion
Tracking Filter
adjusts the camera's direction according to the azimuthelevation measure, and then searches the object and
locates it. Image sensor obtains the accurate azimuthelevation measure about the object, and at the same time
recognizes and tracks the object to finally send
information to fusion center. Meanwhile, radar can work
intermittently to take cover its carrier. While radar is
working, the system fuses the information coming from
radar and image sensor, while radar isn't working, the
system tracks the object only by using image sensor. Once
image sensor loses the object, fusion center starts radar
immediately to detect and locate the object, and directs
image sensor to catch the object quickly again.
(3) When the object comes near and is in radar blind
area, the system works only by image sensor.
3
Radar work state
After radar detects target, it picks up the target's
position and creates a track. Through preperforming, then
the new track is associated with the existing data, the
associated data is used to update track so that the
prediction gate for the target's next position is built. If the
new track is not associated with the existing data, a new
track will be started. If existing track is not associated with
an only track for many times, this track will be ended.
In order to track target precisely, a target-moving
model should be built first. It is very important to track
target whether the model is precise or not. Current input
statistic model is adopted in this paper, this model is
presented based on current statistics model. The author
thinks that the mean of the maneuvering target
acceleration should be the sum of predicted acceleration
and statistical disturbing input of acceleration, so it is fitter
for target maneuvering.
If sampling period is T, target state equation is
X (k + 1) = F (k ) X (k ) + U (k )(a~ + ∆a~) + W (k )
Decision Center
Figure2. System structure and work flow
At first, radar searches target because of a good ability
to detect it, once radar detects an object, fusion center
decides whether the object is in fusion area or not,
according to the distance measure,
(1) If the object is in the radar's work area but is not in
the fusion area, the system works only by radar.
(2) If the object is in the fusion area, fusion center can
send the distance measure and the azimuth-elevation
measure obtained by radar to image sensor. Image sensor
adjusts focus according to the distance measure and
(1)
Where, X ( k ) = [ xˆ k x&ˆ k &xˆ&k ]T , F is state transition matrix,
U is input matrix, a~ and ∆a~ are respectively the predicted
acceleration and the disturbing input of acceleration at
previous time. W (k ) is a zero-mean white noise process
with covariance σ w2 = 2ασ a2 q σ a2 is the covariance of
the acceleration, q is a constant matrix being relative to α
and T[1]. To estimate ∆a~ , refer to reference [2].
Supposed that the radar’s measures under the polar
coordinates
(r ,θ ,ϕ ) , measure equation is
Y (k ) = h(k , X (k )) + V (k )
1427
(2)
 r (k ) 


Where, Y (k ) = θ (k ) ,


ϕ (k )
1
1 1
= +
f
u v


x2 + y2 + z 2

y
h(k , X (k )) = 
arctan

x

z
arctan 2
x + y2 + z2




,




V (k ) is a zero-mean Gaussian measurement noise with
covariance R (k ) .
Extending above 1-D state equation to 3-D space, the
target state estimation can be obtained by using EKF.
4
Image sensor work state
The function of image sensor is to send the accurate
azimuth-elevation measure and the recognizing result
about the object to fusion center, at the same time, to give
the object direction trace in order to fuse with the trace
obtained by radar. Image sensor gets image using a
changeable focus and ahead looking camera.
4.1
Azimuth-elevation measure
At first, image sensor accepts the information from
fusion center; the information includes the object's
distance measure and directions measure obtained by
radar.
According to the object's distance measure, image
subsystem adjusts the camera's focus on the object.
Formula is following:
(3)
Where f is camera's focus, u is the object distance
measure and v is image distance measure.
The camera's moving direction is decided by the
direction measure from radar. The camera coordinate is
showed in figure3. The object's azimuth-elevation measure
decided in figure4.In figure3, X-axis is vertical direction,
Z-axis is optical center, Y-axis is vertical to X-Z plane,
and this coordinate is built based on the carrier. In figure4,
Z'-axis is the actual camera optical center, α is the azimuth
angle and β is the elevation angle. These two angles are
coarse measure, the fine angles are decided according to
the object's location in image.
4.2
Image tracking
Image tracking subsystem searches relative area in
accordance with the orientation information coming from
radar, detects, recognizes and tracks target, then gives an
accurate orientation information and sends it to fusion
center. If image tracking subsystem doesn't find target in
relative area in accordance with the orientation
information coming from radar, it sends this information
to fusion center, which can judge that the correspondent
radar signal is a false alarm.
4.2.1
Target detection
When the image background is simple, such as the sky
or the sea, the peak method can be used to detect the
object, the maximum gray-scale value points can be seen
as the object. These points can be extracted by using
thresholding, the formula is following
f ( x, y ) ≥ Threshold
f ( x, y ) < Threshold
1,
f ( x, y ) = 
0,
O
Y
Y
a
Z
Z
X
X
Z'
Figure3. The camera coordinate
Figure4. The object's azimuth-elevation
1428
(4)
Where Threshold is used to judge which pixels are
target and which pixels are backgrounds.
When the background is complicated, such as the
ground, the frame subtraction can be used to extract the
object from background, but it needs attention that the
sensor's carrier is moving, so the image background must
be registered before using the frame subtraction.
The above mentioned methods don't need human's
intervention generally. A more direct method is to detect
the object by human.
4.2.2
Object recognition and tracking
Human can also perform object recognition, if a person
detects the object, he can recognize the object while
locking it. Using machine to recognize the object is
developing now, especially recognizing the object image.
Because the object image is a 2-D projection in the image
planar coming from 3-D object, a intuition method is
reconstructing the 3-D object from many 2-D projections
and then recognized. This method is very difficult and has
a high computational cost. A practicality method is by
using a supervised neural network, the inputs to train the
neural network are many different 2-D projections of the
object, BP is one of the general algorithms. There is
another effective machine method, having known the
object's a priori knowledge, this method uses deformable
template to recognize the object[3].
The traditional correlation method[4] can be used to
track the locked target. A more robust tracking method is
active contour model or snakes[5]. If the image background
is simple, the centroid method can be used to track target.
The above methods can be used to track target under many
conditions, and each method would be invalid under some
conditions, but it is very infrequent that every method is
invalid under some conditions. So a more general method
is all the above methods working together, named
multiple-model method[6].
5
Then, the angle measures coming from two sensors are
fused and this fused angle measures and radar's distance
measure are used for system tracking filter.
According to minimizing the mean-square error
(MSE), azimuth and elevation can be fused as following
equations
σ 12θ θ 2 + σ 22θ θ1
σ θ21 + σ θ22
(6)
ϕ=
σ 12ϕ ϕ 2 + σ 22ϕ ϕ 1
σ 12ϕ + σ 22ϕ
(7)
Where, 1 means radar and 2 means image sensor,
σ θ2
and σ ϕ2 are azimuth and elevation measure MSE,
respectively.
At last, the fused filter results are fed back to radar and
image sensor as their respective next initialization value,
figure2 shows the work flow.
6
Experiment results and analysis
6.1
Experiment condition
On the assumption that target moves with equality
velocity and beeline, initial position is (0, 0, 0), initial
velocity is (300, 50, 20) (unit: m/s), radar distance
measure noise intensity is σ
elevation
angle
r
measure
= 100 m , azimuth and
noise
intensities
are
σ θ = 20 mrad and σ ϕ = 20 mrad respectively. Image
sensor azimuth and elevation angle measure noise
intensities
are
σ θ = 2mrad
and
σ ϕ = 2 mrad
respectively. Two sensors have the same sampling time, a
Fusion rules
Monte Carlo simulation with N=50 runs was carried out.
Firstly, the statistical distance between image sensor
and radar's observation vectors is a standard to judge
whether the target captured by image sensor and radar is
the same one or not. Image sensor doesn't provide the
distance measure, here, only the angle measures (azimuth
and elevation angles) are used.
If there are two observation vectors X and Y, the
statistical distance between them is
d 2 = AT S −1 A
θ=
(5)
Where, A=X-Y, is the difference between two
observation vectors.
6.2
Experiment results
According to the above condition, using our method to
simulate, the experiment results are shown in the
following figures. Figure5 ~ figure7 are compared curves,
among them, red curves are acceleration, velocity and
position estimation error absolute values in X-axis
direction when radar tracks solely, and blue curves are the
corresponding when radar and image sensor measures are
fused. Figure8~figure10 are compared acceleration,
velocity and position estimation error absolute value in Xaxis direction when radar has feedback and hasn’t
1429
Figure 5. Acceleration estimation error
Figure 6. Velocity estimation error
Figure 7. Position estimation error
1430
Figure 8. Acceleration estimation error
Figure 9. Velocity estimation error
Figure10. Position estimation error
1431
feedback, respectively, and doesn't include image's
information. Among figure8~10, red curves are with no
feedback and blue curves are with feedback.
6.3
Experiment analysis
As show above, estimation precision coming from
radar and image sensor measures fused, is higher than
single sensor's. Feeding fused results to every sensor can
greatly improve each sensor's tracking performance.
7
Conclusion
In this paper, a radar and image sensor fusion tracking
system on the same platform is presented and built, this
system really combines the merit of radar and image
sensor. The idea of "feedback + guide" greatly improved
system's tracking performance and is worthy to be applied
widely in practice.
reference
[1] Zhou Hongren, Jin Zhongliang, Wang Peide,
Tracking of Maneuvering Targets, Beijing: National
Defence Industry Press, pp: 134-153, 1991
[2] Feng Xinxi ,Liu Zuoliang, Shi Lei, Maneuvering
target tracking using real-time input estimation, Proc. of
CIE Int. Conf. of Radar, Beijing, pp:731-734, 1996
[3] Marie-Pierre Dubuisson Jolly, Sridhar Lakshmana,
Anil K. Jain , Vehicle segmentation and classification
using deformable templates, IEEE. Trans. PAMI, Vol:18,
No.3, pp:293~308, Mar.1996
[4] R.C.Gonzalez, P. Wintz, Digital Image Processing,
Addison-Wesley Publishing Company,Inc., 1977
[5] M. Kass, A. Witkin, and D. Terzopoulos, Snakes:
Active contour models, Int. J. Comput. Vis., vol. 1, pp.
321–331, 1987.
[6] CPT Benjamin Reischer, Target tracking
methodologics present and future, Workshop on Image
Trackers and Autonomous Acquisition Application for
Missile Guidance, RA, Alabama, 19-20, Nov. 1979
1432