Road tracing by profile matching and
Kalman filtering
1
George Vosselman and Jurrien de Knecht
1
2
Faculty of Geodetic Engineering, Thijsseweg 11, 2629 JA Delft
E-mail: [email protected]
2
Faculty of Applied Physics, Lorentzweg 1, 2628 CJ Delft
E-mail: [email protected]
Delft University of Technology
The Netherlands
Abstract
Road tracing is a promising technique to increase the efficiency of road mapping. In this paper a
new road tracing algorithm is presented. Road positions are computed by matching the average
grey value profile of a reference road segment with profiles taken from the image. The road
parameters are estimated by the recursive Kalman filter. By utilising the prediction step of the
Kalman filter the road tracer is able to continue following the road despite temporary failures of
the profile matching that are due to road crossings, exits and cars.
1 Introduction
Of all parts in the process of making a map from aerial photographs, the actual mapping
phase is one of the most time consuming and expensive parts. Research, like presented at
this workshop, is therefore increasingly focusing on the development of efficient methods
to extract topographical features like houses and roads from digital aerial photographs. As
fully automatic methods for mapping are still far out of reach, semi-automatic methods for
feature extraction that interact with a human operator are considered to be a good
compromise between the mensuration speed of a computer algorithm and the
interpretation skills of the operator.
This paper deals with road tracing, a semi-automatic approach to measuring roads in
digital aerial images in which a human operator initialises the mapping process by
indicating a small segment of the road to be mapped. After initialisation the continuation
of the road is mapped by the road tracing algorithm (Bajcsy and Tavakoli (1976), Groch
(1982), McKeown and Denlinger (1988), Quam and Strat (1991)).
Another class of road mapping techniques also has to be mentioned. Whereas road tracers
try to extend the initial segment, techniques from this other class use a sparse set of points
distributed along the complete road to be mapped and try to interpolate between the given
points with dynamic programming, relaxation or snakes (Lemmens et al. (1988), Gunst
(1991), Gruen and Li (1994)). We will now return to road tracing.
Most road tracing algorithms consist of two steps: detection of road positions and
concatenation of those positions. The detection step is supported by a prediction based on
the previously detected road segment. Detection is performed by either an edge or line
detection operator (Bajcsy and Tavakoli (1976), McKeown and Denlinger (1988)) or an
analysis of the road profile (cross section) (Groch (1982), McKeown and Denlinger
(1988), Quam and Strat (1991)). In most cases a straightforward connection of the
detected road pixels is used to describe the road.
In this paper we present a new road tracing algorithm based on least squares profile
matching and the Kalman filter. In section 2 we will outline the motivation for this using
least squares matching and the Kalman filter. Section 3 describes the profile matching
while section 4 contains a brief description of the Kalman filter and the model used to
parameterise the shape of the road. In section 5 results of the road tracer are presented.
2 Concepts
2.1 Road characteristics
Road characteristics can be classified in five groups: geometrical, radiometrical,
topological, functional and contextual characteristics (also see Garnesson et al. (1990)).
Examples of these characteristics are:
- Roads are elongated (geometry)
- Roads have a maximum curvature (geometry)
- The road surface usually is homogeneous (radiometry)
- The road surface often has a good contrast with the adjacent areas (radiometry)
- Roads do not stop without a reason (topology)
- Roads intersect and build a network (topology)
- Roads connect cities (functional)
- Higher roads (fly-overs) may cast a shadow (contextual)
- Trees may occlude the road surface, but, on the other hand, an array of trees may also
indicate a road (contextual)
The characteristics describe many features that are used by a human operator to recognise
and map roads. Especially the functional and contextual characteristics require quite some
intelligence in order to exploit them in the image interpretation process. In the developed
semi-automatic road tracer we will not attempt to include these high level vision
components, but instead primarily focus on radiometric and geometric characteristics.
Complex situations (occlusion by trees, interpretation of shadows) will be left to the
operator.
It will be shown that by examining radiometric and geometric properties the road tracer is
capable of detecting indications about the topology of the road network. This capability
potentially can be used to extend the road tracer to a road network tracer.
2.2 Short outline of the road tracer
The operator initialises the road tracer by measuring two points that indicate a short road
segment. Between the two points grey value cross sections are computed at intervals of
one pixel. The model road profile is taken as the average of these cross sections. This
model profile is used as a template in the profile matching.
Based on the indicated road segment an initial estimate is made of the parameters that
describe the road's position and shape. This estimate is used to predict the position of the
first road profile adjacent to the indicated segment. The profile at the predicted position is
matched with the model profile. The result of this match is a shift between the two
profiles. This shift is used by the Kalman filter to update the parameters that describe the
road's position and shape. In the following iterations, the position of the next profile is
predicted, the profile at this position is matched with the model profile, and the road
parameters are updated. The road tracer continues until some break-off criterion (discussed
later) is fulfilled.
2.3 Why least squares profile matching?
The grey value profile is a very characteristic property of a road. It often shows a good
contrast between the road surface and its vicinity. Since adjacent profiles usually have a
very similar shape, profile matching algorithms should be able to detect changes in the
road direction.
Least squares matching is preferred over maximising cross correlation because least
squares matching can estimate the precision of the profile shift. This precision can be used
to evaluate the success of the matching algorithm and moreover is required as input for the
Kalman filter.
Another advantage of least squares matching is the possibility to model the geometric (and
radiometric) transformation between the two profiles. Not only the road position, but also
the road width can be estimated. Thus, when the road width is changing, least squares
matching can obtain good results, while cross correlation will fail. Besides, increasing
road widths in a sequence of profiles can be regarded as an indication for a Y-junction.
2.4 Why Kalman filtering?
The Kalman filter is a recursive procedure to estimate the parameters of a dynamic system
and has found many applications in navigation (Kalman (1960), Gelb (1974)). In the case
of road tracing the parameters to be estimated are the parameters that describe the position
and shape of the road. These parameters are called the state.
Strictly speaking, we do not have a dynamic system since the state is not time-dependent.
However, we will treat the distance along the road as if it were the time variable. In this
way, we can use the recursive estimation procedure.
The Kalman filter consists of two steps: the time update and the measurement update. In
the time update an estimate of the state at time t+dt is made using all observations (i.e.
profile matches) that have been made up to time t. Thus the time update predicts the state
at the future epoch t+dt. In the measurement update the results of the profile match at time
t+dt are combined with the prediction from the time update to obtain an optimal estimate
for the state at time t+dt.
For several reasons the use of the Kalman filter in a road tracing procedure seems very
attractive:
- The Kalman filter has a solid background in statistics and estimation theory. At each
epoch it can produce an optimal estimate of the current road parameters. By using all
available observations in combination with a parameterised road model, the filter result
will be much better than a sequence of adjacent road pixels that are determined
independently.
- The background in statistics enables the Kalman filter to detect and remove outliers in
the observed profile shifts. Thus, if the profile match outputs a very unlikely shift, this
observation is classified as erroneous and will not corrupt the state estimation.
- When following a road the road tracer may encounter situations like intersections and
fly-overs. In those situations the template profile will not match the profile extracted
from the image. Hence, there will be no observation to update the road parameters in
the measurement update. However, by using one or more iterations of the time update
the future states can be estimated by prediction only. After crossing the intersection the
profile matching will succeed again and the time and measurement updates will again
be applied alternately.
- Due to the extrapolation and uncertainty in the time update (so-called system noise) the
variances of the estimated state parameters will increase very rapidly if no observations
are available. This allows us to formulate a break-off criterion for the road tracer: once
the variance of the estimated parameters exceeds some pre-set threshold it is assumed
that (for some reason) the road tracer lost the road.
2.5 Optimal estimate of road position
The described road tracer is optimal in a sense that it produces the least squares estimate of
the road parameters at time t based on all observations at and prior to time t. It does,
however, not make use of observations after time t. Hence, the result of the road tracer can
be improved by inserting all observations along a road segment into a single adjustment
problem. The result will be a road description that is smoother than the one obtained by
the described road tracer. This final adjustment will not be discussed here and has not been
used for the results in this paper.
3 Profile matching
The profile matching compares the model profile with the road profile at the position
predicted by the time update. The differences between the two profiles are modelled by
two geometric (shift and width) and two radiometric (brightness and contrast) parameters.
These parameters are estimated by minimising the square sum of the grey value
differences between the profiles (Ackermann (1983)).
After determining the optimal transformation between the profiles the matching results are
evaluated by three checks:
- The cross correlation coefficient between the grey values of the two profiles after the
transformation is required to be higher than 0.8.
- The estimated values of the geometric and radiometric parameters should be
reasonable. E.g., if the estimated contrast parameter has a high value, say 10, the match
can not be accepted. A contrast value of 10 would mean that the grey value contrast in
the model profile is 10 times the contrast in the profile at the predicted position. A high
contrast value therefore indicates that the latter profile hardly contains any signal and
most likely does not correspond to a part of the road.
- A match is only accepted if the estimated standard deviation of the estimated shift
parameter is below 1 pixel.
If for one of the above reasons the result of the matching is not accepted, the Kalman filter
will not perform a measurement update but instead continue with another time update.
Several consecutive rejections of the profile matching can be used as an indication for a
road junction or the end of the road.
The standard deviation of the estimated profile shift is obtained by propagating the a
posteriori standard deviation of the differences between the grey value profiles and based
on the assumption that the grey value differences have a Gaussian distribution. While
testing the road tracer, it appeared that the estimated standard deviation usually is very
optimistic. For many profiles with disturbances like cars or shadows the assumption of
Gaussian distributed residuals is not met. Instead, the systematic differences often cause a
significant bias in the estimated profile shift. Therefore, the estimated standard deviation
could not be used as input for the Kalman filter. In the Kalman filter a constant standard
deviation of the profile shift of 0.3 pixel was used.
Profiles along a road often change (both gradually and abruptly). Quam and Strat (1991)
therefore suggest to update the model profile by calculating a weighted average of the
initial model profile and the profiles that were successfully matched. In our experiments
this approach led to shifts of the road position within the model profile. Whereas the
model profile was initialised with the road centre in the centre of the profile, the updating
could cause the road centre to shift by several pixels. These shifts were caused by biases in
the shift estimation of the profile matching due to unequal contrast at both roadsides. The
results shown in section 5 were therefore produced with a constant model profile derived
from the segment indicated by the operator.
4 Kalman filtering
For the parameterisation of the road a constant road curvature is assumed. In each position
along the road the row and column coordinate of the road position ( r t , ct ) , the road
direction ( φ t ) and the change in road direction ( φ t ) are estimated. These four parameters
constitute the state vector ( xt ) of the Kalman filter.
In the time update the state vector at time t+dt is predicted by the following equation using
the state vector at time t. Both these state vectors at time t+dt and time t are based on all
observations that have been made up to time t. This is expressed by their indices t+dtt
and tt.

ˆ 

 rˆt|t + dt cos φˆ + φ t|t dt  
 t|t


2


 rˆt+dt|t  

 

 cˆ  

t +dt|t

 

φ t|tˆ  


ˆ
=
=
xˆ t+dt|t 
dt
cˆt|t + dt sin  φ t|t +

ˆ 
2
 φ t+dt|t  



 

φ ˆ 
φˆt|t + φ t|tˆ dt 
 t+dt|t  




ˆ
φ
t|t 

The covariance matrix of the predicted state vector is
T
Pt+dt|t = Φt+dt|t Pt|t Φt+dt|t + Qt+dt
where Φt+dt|t contains the coefficients of the linearized time update equation. The
assumption of a constant road curvature is not quite correct. The deviation of the true road
shape from the shape modelled in the time update equation is known as the system noise.
The resulting uncertainty in the prediction is accounted for by the covariance matrix Qt+dt .
After the time update, a profile is extracted from the image at the predicted position at
time t+dt and perpendicular to the road's direction at that time. The match of this road
profile with the model profile gives an estimated shift s and its variance σ 2s . This shift is
not directly used as an observation in the Kalman filter, but converted to a twodimensional observation vector y t+dt with covariance matrix Rt+dt .
 r t+dt   r t+dt|t - s sin( φ t+dt|t
=
y t+dt = 

 
 ct+dt   ct+dt|t + s cos( φ t+dt|t
)

) 
2

sin ( φ t+dt|t ) sin( φ t+dt|t ) cos( φ t+dt|t ) 


2
Rt+dt = σ s 

2
 sin( φ t+dt|t ) cos( φ t+dt|t )
(
)
φ
cos
t
+
dt
|
t


Note that Rt+dt is singular. This implies that the Kalman filter can only adjust the observed
road position perpendicular to the road direction.
With the above observation the measurement model becomes
 1 0 0 0
E{y t+ dt}= A x t+ dt = 
 x t+ dt
0 1 0 0
This measurement model is used in the measurement update to estimate the state vector at
time t+dt based on the prediction and the observations at time t+dt.
xˆ t+dt|t+dt = xˆ t+dt|t + K t+dt ( yt+dt - A xˆ t+dt|t )
with
-1
T
T
K t+dt = Pt+dt|t A ( Rt+dt + A Pt+dt|t A )
As can be seen from these formula the difference between the observation y t+dt and the
"predicted observation" A xˆ t+dt|t plays an important role in the Kalman filter. Since the
covariance matrix of this residual can be computed, the observation can be submitted to a
statistical test to decide whether it can be used to update the state vector.
5 Road tracing results
Figure 1 shows the results of the road tracer on the blue channel of a colour image of a
complex road junction. The initial road segments measured by the operator are shown in
black. The traced roads are shown in white (good profile matching) and black (rejected
profile matching: the road position is based on the time update only).
Image size:
551 x 527 pixels
Pixel size:
1.6 m
Average road width:
10 pixels
Model profile width:
16 pixels
Min. curvature radius:
160 m
Step size dt:
1 pixel
Fig. 1: Road tracing results for a complex road junction.
Since the road model assumes a constant curvature, the variance of the system noise is
derived from the possible curvature change. The major roads in the scene show a road
curvature with a minimum radius of 160 m or 100 pixels. In the worst case (a straight road
connected to an arc with a radius 100 pixels) the road tracer could encounter a change in
road direction ( φ ) of 1/100 rad. A common approximation of the standard deviation of the
system noise is 1/4 of the maximum error. Thus, the standard deviation of the system noise
in the derivative of the road direction was set to 1/400 rad. The standard deviations of the
system noise of the other state variables were obtained by propagating the noise in the
direction change. In this way, knowledge about the road design was used to estimate the
covariance matrix of the system noise.
The iterations of the road tracer were stopped after the profile matching failed for 10
consecutive profiles and the standard deviation of the predicted road position became
larger than 1 pixel. With this break-off criterion all roads could be traced with exception of
the two major straight roads. For the latter two roads many more predictions were
necessary to cross the fly-overs. For these cases it is useful to have an interaction between
the operator and the road tracer. If the Kalman filter is predicting in the right direction, the
operator can overrule the break-off criterion and let the road tracer continue despite the
high variances of the estimated road positions.
Figure 2 depicts the variance of the road positions of a traced road segment. The two peaks
correspond to the intersecting fly-overs. At these positions the profile matching failed and
the estimation of the road parameters was done by prediction only. The steep increase of
the variances in these areas is caused by the extrapolation and the additive system noise.
Fig. 2: Estimated road position variances along a road segment.
Further results showed that the tuning of the ratio between the variance of the profile
matching and the variance of the system noise is rather sensitive in case of small distances
between the profiles (step size of 1 pixel). As stated before, the estimated variance
obtained from the least squares matching is very optimistic and could not be used in the
Kalman filtering. The low variances make the Kalman filter react vehemently to every
little disturbance and, as a consequence, obstacles like fly-overs can not be crossed if they
are preceded by a small mismatch of the road profiles. On the other hand, if the variances
of the profile shifts are set to a high value, the coefficients of the gain matrix K t+dt may
become very small. Hence, the estimation will be primarily based on the prediction and a
change in road curvature may cause the road tracer to get off the road.
6 Conclusions
In this paper we presented a new road tracing algorithm which combines least squares
matching of grey value profiles with the Kalman filter. Since both techniques have a solid
background in statistics, decisions about the usefulness of the matching results could be
based on statistical tests.
First results showed that the road tracer is able to correctly predict the road path at
positions where the profile matching fails. Thus, the algorithm can cope with intersections,
fly-overs and cars.
The rejected profile matches appeared to be a good indicator for intersecting roads.
Together with the estimation of the road width, they can potentially be used to extend the
road tracing algorithm to an algorithm for mapping road networks.
Literature
Ackermann, F. (1983): High Precision Digital Image Correlation. Proceedings 39th
Photogrammetric Week, Stuttgart, pp. 231-243.
Bajcsy, R. and M. Tavakoli (1976): Computer Recognition of Road from Satellite
Pictures. IEEE Transactions on Systems, Man, and Cybernetics, vol. 6, no. 9, pp. 623-637.
Garnesson, Ph., G. Giraudon, and Montesinos (1990): An Image Analysis System,
Application for Aerial Imagery Interpretation. Proceedings 10th IAPR International
Conference on Pattern Recognition, Atlantic City, June 1990, vol.1, pp. 210-212.
Gelb, A (Ed.) (1974): Applied Optimal Estimation. MIT Press, Cambridge, MA.
Gunst, M. de (1991): Automatic Extraction of Roads from SPOT Images. Technical
Report, Delft University of Technology, 91 p.
Groch, W.D. (1982): Extraction of Line Shaped Objects from Aerial Images using a
Special Operator to Analyze the Profile of Functions. Computer Graphics and Image
Processing, vol. 18, pp. 347-358.
Grün, A. and H. Li (1994): Semi-automatic Road Extraction by Dynamic Programming.
International Archives of Photogrammetry and Remote Sensing, vol. 30, part 3/1, pp. 324332.
Kalman, R.E. (1960): A New Approach to Linear Filtering and Prediction Problems.
ASME Journal of Basic Engineering, vol. 82D, pp. 35-45.
Knecht, J.P. de (1994): Tracing of Roads in Aerial Photographs by Profile Matching and
the Kalman-filter (in Dutch). MSc-Thesis, Faculty of Geodetic Engineering, Delft
University of Technology, 90 p.
Lemmens, M.J.P.M., E.W. Bruel and F. Fennis (1988): Linear Feature Extraction and
Road Recognition from Large Scale Digital Aerial Images. International Archives of
Photogrammetry and Remote Sensing, vol. 27, part B8, pp. 20-31.
McKeown, D.M. and J.L. Denlinger (1988): Cooperative Methods for road tracing in
aerial imagery. IEEE Proceedings on Computer Vision and Pattern Recognition, Ann
Arbor, MI, pp. 662-672.
Quam, L.H. and T.M. Strat (1991): SRI Image Understanding Research in Carto-graphic
Feature Extraction. In: Ebner et al. (ed), Digital Photogrammetric Systems, Wichmann
Verlag, Karlsruhe, pp. 111-122.