Robust Horizontal Line Detection and Tracking in Occluded
Environment for Infrared Cameras
Sungho Kim1 , Soon Kwon2 , and Byungin Choi3
1 LED-IT Fusion Technology Research Center and Department of Electronic Engineering,
Yeungnam University, Gyeongsan, Gyeongbuk, Korea
2 Daegu Gyeongbuk Institute of Science and Technology, Daegu, Korea
3 Electro-Optics Laboratory, Samsung Thales Company, Yongin, Korea
Abstract— Detecting a horizontal line in an infrared image
is an important component of automatic surveillance applications such as detecting ships, missiles on the horizon,
unmanned aerial vehicle control, flight navigation, and port
security. Most of the existing solutions for the problems only
use single image to detect horizon line. Although this results
in good accuracy for some images, it often fails to detect
horizons in foggy, occluded environments. In this paper,
we propose a novel horizon detection and tracking method
that is robust to sensor vibrations and occlusions using an
infrared camera for 24 hour running. An initial horizon
is detected by a sensor geometry-based method for the
robustness. Local horizon optimization and tracking produce
stable horizons in occluded environments. The experimental
results validate the feasibility of the proposed method in real
infrared images.
Keywords: IRST, horizontal line, detection, tracking, clutter
1. INTRODUCTION
It is important to detect horizon for a number of applications such as sea-based infrared search and track (IRST)
[1], vision-guided flight stability and control for micro air
vehicles [2], surveillance for coast security [3]. Fig. 1 shows
an infrared image example of sea-based environment for
horizon detection.
One of the previous approaches using only image processing methods performed very well in some cases [4]. Liu
et al. presented an improved linear fit method, in which an
effective preprocessing step is employed and those points
fitted by line are reconfirmed [5]. Nevertheless, the improved
linear fit method was proved poor applicability in cloud or
sea clutter background. Having analyzed the weakness of
the improved linear fit method, Yang et al. put forward
a variance weighted information entropy (VWIE) based
algorithm, but this algorithm is not fit for the complicated
background infrared images [6]. Wen et al. proposed an
Otsuaŕs
˛ threshold method, and it uses morphological opening
and closing to smooth the segmented image but can only
process the simple background infrared images [7]. Another
method combining machine learning methods with morphol-
Sky line
Horion/coast line
Fig. 1: Example of infrared image and location of horizon/cost line.
ogy based operations, the Hough Transform and Expectation
Maximization function to separate pixel distributions [8].
However, their performance in identifying a horizon suffers when images are complicated with clutter such as a very
cloudy or foggy environment, an uneven horizon line, and
varying lighting conditions. In addition, horizon detection
can be fragile if there is strong occlusion by islands or some
targets. In this paper, we present a robust horizon detection
and tracking method in infrared sequences by introduction
hybrid horizon initialization and outlier identification-based
optimal tracking. In Section II, we introduce the overall
horizon detection system. Geometry-based horizon initialization method is explained in Section III and optimal horizon
tracking method is presented in Section IV. In In Section
V, various performance evaluations and experimental results
are explained by using real infrared sequences. We conclude
and discuss this paper in Section VI.
2. Overview of the proposed system
The proposed system consists of three components: sensor
line of sight (LOS), horizon prediction, and horizon optimization in infrared video as shown in Fig. 2.
We consider a sea-based IRST system as a test environment. Infrared search and track (IRST) systems are
Sensor LOS
- Height ( h )
- Elevation ( α )
Provided by IRST system
Signal Processing
Horizon Prediction
- Geometric analysis
. Horizon position (H prior )
. Horizon tilt ( θ
)
prior
Horizon Optimization
- Local horizon search
. Horizon position ( H opt )
. Horizon tilt ( θ opt )
Fig. 2: Proposed horizon initialization and optimization in
infrared sequences.
developed for autonomous searching, detection, acquisition,
tracking and designation of potential incoming targets [9],
[10]. Related research was actively conducted in the late
1980’s. In these applications, targets including missiles
and ships are typically small and they appear in the sea
background. An infrared camera is mounted on the top
of a ship, which provides sensor height (h) and elevation
(α) information. Based on these pose information, we can
predict horizon location using geometric analysis. In the optimization block, occluded horizon is detected by RANSAC
(RANdom SAmple Consensus) algorithm followed by local
optimization. Horizon tracking is conducted on the inlier
horizon with local search. Horizon is initialized statistically
to adapt environmental changes.
3. Geometry-based horizon initialization
3.1 Geometrical model of horizon prediction
In the sea-based IRST systems, infrared images consist
of sky, horizon and sea regions. The horizontal region has a
strong boundary line that divides heterogeneous backgrounds
such as sky and sea region. The sea surface region contains
many sun-glints and the ship targets are close to a sensor. So,
an image segmentation scheme is necessary for a successful
detection system. Image based region segmentation can be
made possible by using clustering algorithms. However,
this approach is unstable to environmental changes. The
horizontal lines can be ambiguous when there is a strong sea
fog. So, we use more stable approach based on the geometric
analysis using the sensor pose information.
As the pose information of an IRST sensor is recorded in
the image header, we can estimate the horizontal line. The
horizontal information is very important as it can provide a
region segmentation cue. If we assume that an IR camera
has height (h), elevation angle (α, assume 0◦ for easy
analysis), and earth radius (R) then, we can depict the
geometric relations as shown in Fig. 3 (a). The projected
horizontal line in any image can be found by calculating
the angle (θH ) as in equation (1). In fact, a real IRST
sensor can change the elevation angle, which changes the
location of the horizontal line in the image domain. If the
elevation angle of a camera is given as α and the field of
view (FOV) of the sensor is given as β then, the angle
of a sky region (θsky ) is determined by the equation (2).
If elevation angle (α) is smaller than θH − β/2 then, the
sensor can observe only the sea region. So, the angle of the
sky region (θsky ) is 0. Similarly, we can also analyze other
cases. The angle of the sea region (θsea ) is determined as,
θsea = β − θsky . As the sky-sea region segmentation ratio
is determined by tanθsea /tanθsky , the final horizontal line
(Hprior ) is calculated by using equation (3). If we assume
the image height is 1,280 pixels, vertical field of view is
20◦ then, the sensor height is 20m, the elevation angle is
5◦ , then the prediction horizontal line (Hprior ) is located at
974 pixels as shown in Fig. 4.
θH = −cos−1


θsky =

(
R
R+h
0
β
α − θH + β/2
Hprior = ImageHeight ∗
)
if α < θH − β/2
if α > θH + β/2
else
tanθsky
tanθsky + tanθsea
(1)
(2)
(3)
3.2 Analysis of horizon prediction error
In ideal cases, the horizon prediction using above method
can be accurate. However, there are several noise factors
such as uncertainty of the sensor height caused by waves,
uncertainty of the sensor elevation and roll angle after after
mechanical stabilization. In the first case, we consider the
noise of sensor height as 0 − 10m. 10m can be possible
if there is strong hurricane. Fig. 5(a) shows the horizon
prediction error or offset [pixel] according to the noise of
sensor height. We can predict maximum 2 pixel offset of
horizon location if the sensor height noise is 10m.
In the second case, we consider the noise of sensor elevation angle. Normal mechanical stabilization can provide the
angle error of ±0.005◦ , which cause horizontal offset of 0.3
pixel. If we assume the noise as ±0.5◦ by considering 100
times stability margin, the horizon offset can be predicted
(a)
Fig. 3: Geometry of sea-based IRST system. (a) Relationship
between sensor height and horizontal line, (b) camera geometry with the field of view and elevation angle (α = 0), (c)
approximated position of horizontal line when the elevation
angle is α.
(b)
Fig. 5: Noise analysis of horizon prediction error: (a) Horizon offset caused by sensor height noise, (b) horizon offset
caused by sensor elevation noise.
as shown in Fig. 5(b). The maximal horizontal offset can be
± pixels. According to the results of noise analysis, sensor
elevation noise is more critical than the sensor height noise.
There can be additional sensor noise of roll stabilization.
In normal roll stabilization error of 0.005◦ , there is almost
no horizontal tilt (θprior ). If we consider the stabilization
error of 0.5◦ , the maximal horizontal tilt is 10 pixels in
1280 × 1080 image.
By summarizing above noise analysis, we can conclude
that the maximal horizon offset boundary is ±30 pixels
including horizontal tilt.
4. Optimal horizon tracking
Fig. 4: Synthetic horizon prediction using geometric analysis
of sensor pose.
From the previous sensor LOS, we can predict horizontal
location with pre-defined search boundary. The next step
is optimal horizon tracking in video sequence as shown in
Fig. 6. Given an input frame, horixels (horizontal pixels)
are extracted using column directional gradient and max
Input frame
Horixel Extraction
- Column directional gradient filtering within
local search space (use sampling interval)
- Locating horixel (horizon pixel) by max
gradient
Inlier index
Initialization mode
Inlier Detection
Tracking mode
- RANSAC (RANdom SAmple Consensus)
. Robust to outliers
. Identifying inlier index
Total Least Square Optimization
- SVD-based line fitting for inlier index
. Closed form solution
. Fast and stable
Fig. 6: Horizon optimization and tracking flow in infrared
sequence.
selection. Then, inlier horixels are identified using robust
line fitting method of RANSAC [11]. The important role
of RANSAC is to find inlier indices of true horixels.
Based on the inlier index, total least square optimization
can detect final horizon stably. Since inlier horixels are
identified through the process, horizon tracking is conducted
using horixel extraction and optimization. Inlier detection
block is activated in the beginning and statistically to adapt
environmental changes.
Extracted horixels
Horizon prediction
using sensor LOS
Fig. 7: Example of the predicted horizon and detected
horixels.
4.1 Horixel extraction
Given a predicted horizon as shown in Fig. 7(dotted blue
line), a search boundary is set. Then, sampling interval is
defined to reduce the computational complexity. For each
sample position, column direction gradient filter is conducted
using derivative of Gaussian kernel. Then horixels close to
a predicted horizon are extracted by max selection. Fig.
7(dotted black line) shows the extracted horixels.
outlier
inlier
4.2 Inlier detection using RANSAC
In a sea environment, horizon is frequently occluded by
islands, coasts, and cloud. So, we need a robust horizon
estimation method such as RANSAC. Basically, RANSAC
algorithm picks two horixels and predict horizon line. Then,
it checks line fitting and inliers. After a number of iterations,
horizon line parameter is selected that has largest inliers.
Fig. 8 shows the inlier detection results using a RANSAC
method. Note that inliers and outliers are classified almost
correctly. The inlier indices are used in the optimization of
line fitting and horizon tracking.
Fig. 8: Example of inlier horixels found by a RANSAC.
Optimized
horizontal line
Set1: Occluded by cloud
Set 2: Occluded by near island
Set3: Occluded by near/remote island
Set4: Occluded by near coast
Prediction
Initialization
Optimization
Fig. 9: Example of horizontal line optimization in occluded
environments.
4.3 SVD-based optimization and tracking
The last step is to refine horizon parameters using total
least square fitting given a set of inlier horixels. The fitting
process is as follows. First, we normalize inlier horixels and
then conduct a singular value decomposition (SVD) [12].
Horizon direction is selected by an eigenvector with the
smallest eigenvalue. Figure 9 shows the horizon optimization
results for an image occluded by near island and remote
island. Horizontal area is enlarged to show the results.
Horizon tracking is done by the horixel extraction and SVDbased optimization with the inlier indices. RANSAC-based
initialization is activated statistically.
5. Experimental results
We prepared four kinds of test sequences as shown in
Fig. 10 to validate the robustness of the proposed method.
The Set 1 is remote sea images occluded by strong cloud.
Horizons in Detected horizon is decided as correct if the line
fitting error is the Set 2 are occluded by near island which
occupies 1/3 of the horizon length. Set 3 has a near islands
and a remote island. The last Set 4 has near coast in which
boats and buildings occlude horizons.
A detected horizon is declared as correct detection if a
line fitting error is within 1 pixel in average. The ground
truth of horizon location is prepared by manual inspection.
The original test sets has almost no sensor noise. So, we
add artificial sensor tilt noise by ±0.5◦ and horizon location
noise by ±3.0 pixels generated by uniform for that range.
Table 1 summarizes the overall experimental results. Our
method detected horizons correctly for the noiseless sequence data. In the case of noisy data, only one frame of
Set 4 shows incorrect horizon detection. Fig. 11, 12, 13,
and 14 show the sampled horizontal detection results for the
noise added sequences. Dotted blue lines denotes horizon
prediction by sensor LOS, solid black or white line denotes
optimal horizon, and magenta dots denote inlier horixels
extracted by RANSAC. Note that horizon lines are detected
robustly regardless to occlusion types under sensor noise.
Fig. 10: Composition of the test database.
Table 1: Detection rate (DR) of horizon for the noiseless
data and noisy data.
Test set
Set 1
Set 2
Set 3
Set 4
DR w/o noise [%]
100 (20/20)
100 (35/35)
100 (35/35)
100 (30/30)
DR with noise
100 (20/20)
100 (35/35)
100 (35/35)
97 (29/30)
6. Conclusions
In this paper, we present a robust horizon detection and
tracking method using sensor geometry and optimization.
Through the analysis of sensor geometry, we can predict
the search range of horizon. Inlier indices are found by
RANSAC and these indices are utilized in the SVD-based
line fitting and tracking. Experimental results for the various
infrared sequences validate the robustness of the proposed
method.
Acknowledgement
This research was supported by the DGIST R&D Program
(12-BD-0202) and by a grant-in-aid of Samsung Thales. It
was also supported by the Basic Science Research Program
through the National Research Foundation of Korea (NRF)
funded by the Ministry of Education, Science and Technology (No. 2012-0003252).
Fig. 11: Examples of horizon detection for the noise added
test Set 1.
Fig. 12: Examples of horizon detection for the noise added
test Set 2.
References
[6] L. Yang, Y. Zhou, and L. Chen, “Variance wie based infrared images
processing,” Electron. Lett., vol. 42, no. 15, pp. 338–340, 2006.
[7] P. zhi Wen, Z. lin Shi, and H. bin Yu, “Automatic detection method
of ir small target in complex sea background,” Infrared and Laser
Engineering, vol. 32, no. 6, pp. 590–593, 2003.
[8] S. Todorivic and M. Nechyba, “Sky/ground modelling for autonomous
mav flight,” in ICRA. IEEE, 2003, pp. 4679–4684.
[9] S. B. Campana, “The infrared and electro-optical systems handbook,”
SPIE Optical Engineering Press, vol. 5, no. 4, 1993.
[10] A. N. de Jong, “IRST and perspective,” in Proc. of SPIE, vol. 2552,
1995, pp. 206–213.
[11] R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer
Vision, 2nd ed. Cambridge University Press, ISBN: 0521540518,
2004.
[12] R. Hanson and M. Norris, “Analysis of measurements based on the
singular value decomposition,” SIAM J. Sci. Stat. Comput., vol. 2, pp.
363–373, 1981.
[1] S. Kim and J. Lee, “Scale invariant small target
detection by optimizing signal-to-clutter ratio in heterogeneous
background for infrared search and track,” Pattern Recognition,
vol. 45, no. 1, pp. 393–406, 2012. [Online]. Available:
http://dx.doi.org/10.1016/j.patcog.2011.06.009
[2] S. M. Ettinger, M. C. Nechyba, P. G. Ifju, and M. Waszak, “Visionguided flight stability and control for micro air vehicles,” Advanced
Robotics, vol. 17, no. 7, pp. 617–640, 2003.
[3] S. P. van den Broek, H. Bouma, M. A. Degache, and G. Burghouts,
“Discrimination of classes of ships for aided recognition in a coastal
environment,” in Proc. of SPIE, vol. 7335, 2009, p. 73350W.
[4] T. G. McGee, R. Sengupta, and J. K. Hedrick, “Obstacle detection for
small autonomous aircraft using sky segmentation,” in ICRA. IEEE,
2005, pp. 4679–4684.
[5] S. tao Liu, T. sheng Shen, Y. li Han, and X. dong Zhou, “Research
on locating the horizontal region of ship target,” Infrared and Laser
Engineering, vol. 33, no. 1, pp. 51–53, 2003.
Fig. 13: Examples of horizon detection for the noise added
test Set 3.
Fig. 14: Examples of horizon detection for the noise added
test Set 4.