DE-INTERLACING OF VIDEO DATA
G. de Haan and E.B. Bellers
Philips Research Laboratories, Television Systems Group,
Prof. Holstlaan 4, 5656 AA Eindhoven, The Netherlands
A new de-interlacing algorithm is proposed, suitable for high-quality icker-free display
of television images, for matrix type of displays,
and as a basis for scan-rate conversions. The algorithm applies motion estimation and compensation
techniques to achieve a high performance for moving and stationary image parts. This paper provides details of the new algorithm and an evaluation
showing the relative performance of the proposal
and a set of recently proposed and/or commercially
available methods.
The paper is organised as follows: Section 2 briey
introduces the evaluated algorithms. Section 3 summarizes the main features of the motion estimator applied in all motion compensated de-interlacing methods. Section 4 introduces the new AR de-interlacer.
An evaluation of alternative methods is presented in
section 5, and we draw our conclusions in section 6.
De-interlacing, motion-compensation,
scan-rate conversion.
The conventional non-motion compensated deinterlacing algorithms currently available in commercial products either apply linear or non-linear
VT-ltering. Most popular examples from these two
categories are the three-tap spatio-temporal median
lter:
Abstract:
2
Keywords:
1
Introduction
De-interlacing techniques
Fo (~
x ; n)
De-interlacing is a basic requirement for video scanning format conversions. Since perfection under all
circumstances is impossible to achieve, many dierent
algorithms to realize a good quality have been proposed. The products currently available on the consumer electronics market, either use linear VerticalTemporal (VT) ltering [10, 11], MEDian ltering
(MED) [12], or in the most advanced product, Motion
Compensated MEDian ltering (mcMED) [13].
In the literature, more advanced motion compensated algorithms have been described, including TimeRecursive (TR) methods [1, 6, 14], and (GST) methods
based on a Generalized Sampling Theorem [2, 3, 15].
This paper evaluates the existing commercially available methods, the alternative methods as described in
the literature cited above, as well as the new, Adaptive Recursive concept (AR). The AR de-interlacer
is a further elaboration of ideas presented in [14].
The algorithm that controls the recursion, particularly,
has been improved. The resulting AR-de-interlacer is
shown to outperform all alternative methods.
8
>
>
>
>
<
=
F (~
x ; n)
; (y
mod 2 = n mod 2)
0
1
F (~
x; n 1)
>
>
MED @ F (~x yu ; n) A ; (otherwise)
>
>
:
F (~
x + yu ; n)
(1)
(where we use F (~x; n) for the luminance value in eld n,
at position ~x = (x; y )t , with t for transpose, Fo (~x; n) the
de-interlaced output signal, MED the median function, and
the unit vector in vertical direction yu = (0; 1)t .),
and the linear VT-lter using samples from two or
three elds:
Fo (~
x ; n)
8
>
<
=
F (~
x; n)
P
>
: k;j F (~x
; (y
2kyu ; n
j)
mod 2 = n mod 2)
Ph(k;j )
k;j
h(k;j )
; (otherwise)
(2)
with k and j integer, and h(k; j ) the FIR lter impulse
1
Solution 2 utilizes the possibility to perfectly reconstruct a signal sampled at 1=n times the Nyquist rate
with n independent sets of samples that describe the
signal. For the de-interlacing problem n = 2, and the
required two sets are the current eld, F (~x; n), and the
~ (~
motion compensated previous eld, F (~x D
x; n); n
1). If the two do not coincide, i.e. the object does not
have an odd vertical motion vector component, the independency constraint is fullled, and the problem can,
theoretically, be solved. The resulting output signal is
dened as:
Fo (~
x ; n) =
8
F (~
x ; n)
; (y mod 2 = n mod 2)
>
>
>
<
P
x (2m + 1)yu ; n
) h(m)+
m F(~
>
>
>
: Pp F ~x D
~ i 2pyu ; n 1 h(p) ; (otherwise)
(5)
~
~
where Di diers from the displacement vector D in
that the y-component is replaced by its integer part,
while h(m) and h(p) depend on the fractional part of
this y-component.
Practical problems are:
response, for which we used (two-elds)1 :
8
1; 8; 8; 1
; (k = 3; 1; 1; 3) ^ (j = 0)
>
>
>
>
<
5; 10; 5 ; (k = 2; 0; 2) ^ (j = 1)
h(k; j ) =
>
>
>
>
: 0
; (otherwise)
(3)
Motion compensation can be straighforwardly added,
e.g. for the median lter [4, 13], using the displacement
t
~ short for: D
~ (~
vector D
x; n) = (Dx (~
x; n); Dy (~
x; n)) :
Fo (~
x; n)
8
>
>
>
>
<
=
F (~
x; n)
; (y
mod 2 = n mod 2)
0
1
~ n 1);
F (~
x D;
>
>
A ; (otherwise)
>
> MED @ F (~x yu ; n) ;
:
F (~
x + yu ; n)
(4)
In general, however, the samples required for the motion compensated de-interlacing do not exist in the
time discrete input signal, e.g. due to non-integer velocities. In the horizontal domain this problem can be
solved with linear Sampling Rate Conversion (SRC)
theory, see e.g. [5], but not in the vertical domain, as
the constraints of the sampling theorem, due to the
interlacing, are not met. Neglecting this fundamental
problem can lead to acceptable results, as appears in
our evaluation.
Improved concepts to partly cope with this problem
have been proposed recently in the literature:
1. A straight extension of the motion vector into earlier pictures until it points (almost) to an existing
pixel [7].
The velocity can have an odd vertical component.
For velocities near the vertical odds, noise may be
enhanced.
A perfect reconstruction requires the use of pixels
from many lines, for which the velocity need not
be constant.
Solution 3 departs from the assumption that it is possible, at some time, to have a perfectly de-interlaced
picture in a memory. Once this is true, this picture is
used to de-interlace the next input eld:
2. The application of a generalized sampling theory
(GST) [2, 3, 14].
Fo (~
x ; n)
3. Recursive de-interlacing of the signal [1, 6, 15].
8
<
~ (~
D
x; n); n
Solution 1 replaces F (~x
1) by F (~x
~ (~
2D
x; n); n 2) if this is closer to an existing pixel on
the eld grid. The spatial error should be smaller than
a small fraction of the pixel distance, otherwise a simple bi-linear interpolation is used as a fall-back option.
Essential is the implicit assumption that the velocity
is constant over a two-eld period. This assumption is
rather often violated, which reduces the practical use
of this method.
:
=
F (~
x ; n)
Fo (~
x
;
~ n
D;
(y mod 2 = n mod 2)
(6)
1) , otherwise
With correct motion vectors this solution is perfect, as
the de-interlaced picture in the memory allows the use
of SRC-theory also in the vertical domain. If this new
de-interlaced eld is written in the memory it can be
used to de-interlace the next incoming eld, etc...
Limitations of this method are:
1 The VT-lter in the evaluation uses the estimated coecients as obtained from measuring the device [10] in a single eld
memory architecture. The coecients of [11] were not available.
2
Propagation of errors due to motion vector inaccuracy and interpolation defects.
Even a perfectly de-interlaced picture can contain
alias in the vertical frequency domain, assuming
the common case of a camera without optical prelter.
tial and/or temporal `prediction vectors' from a 3-D
neighbourhood, and a single updated prediction vector. This implicitly assumes spatial and/or temporal
consistency. The updating process involves update vectors added to either of the spatial prediction vectors.
~,
Assuming blocks of height Y , width X , and center X
~
we dene a candidate set CS (X; n), from which the
block-matcher selects its result vector:
8
9
X
>
>
>
~
~
~
~
; n) + U1 (X; n)); >
>
>
>(D(X
>
Y
>
>
>
>
<
=
X
~
~
~
~
~
CS (X; n) = (D(X
; n) + U2 (X; n)); (8)
>
>
>
>
Y
>
>
>
>
0
>
>
~
~
>
>
; n 1));
:(D(X
;
2Y
In practice, the rst problem is the more serious one,
particularly for nearly odd valued vertical velocities
and/or noisy input signals. To prevent errors from
propagating, in [1] some additional measures are described. Particularly, the median lter is suggested to
realize a protection of the interpolated lines:
Fo (~
x; n)
=
8
>
>
<
0
F (~
x; n)
MED @
>
>
:
; (y mod 2 = n mod 2)
1
F (~
x+~
yu ; n);
F (~
x ~
yu ; n); A ,(otherwise)
~
Fo (~
x D; n 1)
~ 1 (X;
~ n) and U
~ 2 (X;
~ n) are
where the update vectors U
block-alternatingly equal to the zero vector (~0), or
taken from a limited xed integer update set, in our
case:
8
9
< ~0;
=
~ n) = ~
U Si (X;
(9)
yu ;
~
yu ;
~
xu ;
~
xu ;
:2~y ;
2~yu ; 3~xu ;
3~xu ;;
u
(7)
Although further renements are suggested in [1], we
will use this algorithm as the basis four our comparison.
We have concluded in earlier publications that timerecursive de-interlacing according to [1], de-interlacing
based on GST [3], and particularly the synthesis of
both methods [15, 16] are the best methods presently
known. However, even these best methods are imperfect. It is our target to present in this paper an improvement that can be applied in combination with
other methods to suppress the remaining artifacts in
the de-interlaced output signal. In fact, our proposal
can be used to improve any de-interlacing algorithm.
Outperforming the best methods by adding the improvement to one of the poorest algorithms seems to
be most spectacular. Therefore, to show the advantages, we will apply the improvement to simple lineaveraging, and use the best known algorithms as a reference in our evaluation.
3
where, similar to ~yu , we introduce ~xu = (1; 0)t .
To realize sub-pixel accuracy, the update set of equation (9) is extended with fractional update values. An
overall quarter pel resolution is achieved by adding the
following fractional update vectors to the update set:
1
1
1
~ n) =
y ;
~
~
y ; 41 ~
xu ;
~
x ;
(10)
U Sf (X;
4 u
4 u
4 u
Because of the small number of candidate vectors that
have to be evaluated, the method is very ecient. Furthermore, due to the inherent smoothness constraint,
it yields very coherent vector elds that closely correspond to the true-motion of objects.
Description of the applied motion estimator
4
As we expect the quality of any algorithm with motion
compensation to depend heavily on the performance of
the motion estimator, we applied the same high quality estimator [9] in combination with all de-interlacing
methods. This estimator achieves a quarter pixel accuracy, and a close to true-motion vector eld, which
is considered very important for scan rate conversion.
Rather than calculating all possible candidate vectors,
the so-called recursive search block-matcher takes spa-
The Adaptive Recursive (AR)
de-interlacing
The main imperfection of the recursive de-interlacing
algorithm is remaining alias in the output signal. In
an earlier publication [14] we have proposed an improvement that is generally applicable to the various
de-interlacing methods. This improvement was based
on the observation that the alias can be interpreted as
non-stationarities along the motion trajectory. Such
non-stationarities can eectively be reduced with a mo3
Fo (~
x; n)
=
8
k1 (~
x; n)Fi (~
x; n)+
>
>
>
>
< (1 k1 (~x; n))Fo (~x
>
>
>
>
:
k2 (~
x; n)Fi (~
x; n)+
(1 k2 (~x; n))Fo (~x
~ n
D;
1); (y mod 2 = n mod 2)
~ n
D;
1); (otherwise)
D
motion
vector
X X=k2C+(1-k2)(D/2+E/2)
E
A
(11)
where Fi (~x; n) is the output of an initial de-interlacing
method, for which any method can be used. The ltering of pixels on existing lines of the input signal is not
necessarily the same, as k1 and k2 may, and preferrably
do, dier. Using a clip function dened as:
8
0 ; (a < 0)
>
>
>
>
<
1 ; (a > 1)
clip(0; 1; a) =
(12)
>
>
>
>
: a ; (otherwise)
k1
Vertical position
tion compensated temporal recursive lter:
can e.g. be calculated as:
p
k1 = clip 0; 1; C1 Dif f
k2 selected such that:
2|X-C|=|A-D+B-C|
C
B
A,B,and C interpolated on frame grid
Previous output frame
Current input field
Time
Figure 1: Illustration of the AR-de-interlacing algorithm.
to:
k2 (~
x ; n)
k1 (~
x
(13)
= clip(0; 1;
~
yu ; n)
jF
j
~
yu ;n) Fo (~
~
x y
~u D;n
1) +
~
Fi (~
x;n) Fo (~
x D;n
1) +
x
i (~
j
j
)
(16)
where a small constant, , prevents division by zero,
and biases towards identical ltering of neighbouring
pixels if the numerator and the denominator are both
small. As an implication, the temporal recursive ltering of the interpolated pixels depends on the quality of
the initial de-interlacing method. A simple line averaging algorithm will cause stronger temporal ltering
of the interpolated pixels than e.g. an motion compensated median lter. Experimentally, we found that the
dierence in the resulting de-interlacing performance
was small.
The assumption of eq. (15) leads to an adaptation of
temporal recursive ltering on the interpolated line applying dierences measured on the upper neighbouring
original line. Rather than using the upper original line
as a reference, the lower neighbouring line can be used
equally well. For symmetry considerations the average
eect on the two neighbouring original lines seems advantageous. Elaborating this results in a symmetrical
alternative for eq. (16):
where, Dif f is a pixel-dierence, measured on pixels
at the interlaced grid only:
x D
~ (~
Dif f (~
x; n) = Fo (~
x; n); n 1) F (~
x ; n)
(14)
By appropriately choosing C1 , it is possible to tune the
relative importance of the mean square error, that will
generally increase with too strong temporal ltering,
and the consistency along the motion trajectory, which
improves if the ltering is stronger.
Although this tuning seems rather straightforward for
k1 , it is more complicated for k2 . For a similar ltering
of the interpolated pixels, a signicant pixel dierence
cannot be found, as the quality of the input pixels to
this temporal lter depends on the quality of the initial
sequential scan conversion algorithm.
To escape from this fundamental problem, we propose
to tune k2 such that the dierence along the motion
trajectory is identical for vertically neighbouring pixels. This basic assumption leads to the equation:
F (~x; n) F (~x D;
=
~
n
1)
o
o
(15)
F (~x ~y ; n) F (~x ~y
~
D; n 1)
o
u
o
u
k2 (~
x ; n)
clip
=
1
)d
0; 1; 2 j(k1jF(~x(~x~y;n;n
) F
u
i
x+~
yu ;n))dl
u +k1 (~
x
o (~
j
~
D;n
1) +
j+
(17)
with the dierence dl at the position of the lower neighbouring pixel:
Combination of eqs. (11) and (15), and clipping between 0 and 1, results in a calculation of k2 according
dl
4
= Fi (~x
yu ; n)
~
Fo (~
x
~
D
~
yu ; n
1)
(18)
Input
110
Output
MUX
line
average
Frame
Motion
memory
compensation
100
k2
1-k
90
2
80
Motion estimator
70
60
Figure 2: Block diagram showing a possible implementation of the AR-de-interlacer.
50
and the the dierence du at the position of upper neighbouring pixel:
du
= Fi (~x + ~yu ; n)
Fo (~
x
~ +~
D
yu ; n
1)
40
8
<
:
motion trajectory:
MT I
(F (~x
; (y
=
1 X
N
Fo (~
x ; n)
Fo (~
x
~ n
D;
2
1)
(22)
~
x;n
Figure 3 shows the mean{square{errors (M SE ) and
motion{trajectory{inconsistencies (M T I ), as obtained
with the commercially available devices, the methods
described in the referenced literature, and the new ARalgorithm. M SE and M T I are calculated here as an
average over various sequences, identical to the ones
used in the evaluation of [15, 16].
In the category of simple non-motion compensated
methods, the VT lter suered from serious degradation on fast and particularly vertically moving sequences, whereas the the MED lter approach showed
quality loss in material with high spatial frequencies in
the vertical domain.
The category of motion compensated methods showed
that vector error protection means were indispensable.
The lack of it seriously decreases the score of the GSTbased method, whereas the TR-method [1] prots from
its median protection. The dierence showed mainly
in the complex motion sequences.
In all categories, the new AR algorithm shows the best
performance as can be concluded from Figure 3.
mod 2 = n mod 2)
yu ; n) + F (~
x + yu ; n)) ; (otherwise)
(20)
The AR-de-interlacing algorithm is illustrated in Figure 1, while Figure 2 shows a block diagram of a possible implementation.
5
AR
=
F (~
x; n)
1
2
GST
Figure 3: Resulting M SE (dark) and M T I (light).
The various methods are indicated by the abbreviations suggested in the introduction.
(19)
In our evaluation, we applied the control according to
equation (17) for k2 , while xing the value of k1 to one,
i.e. no ltering on the original lines. Generally, compared to k1 dened as in equation (13), this will lead to
somewhat lower M SE gures, and a higher M T I score
[14]. Although this slightly degrades the performance,
it seems fairer for the comparison, as all methods in
the test now leave the input pixels unaltered.
For the initial de-interlacing to generate Fi we applied
simple line-averaging:
Fi (~
x; n)
MED VT mcMED TR
Evaluation
To evaluate the relative performance of the deinterlacing algorithms we applied two criteria that we
used in earlier publications as well [15, 14, 16]. The
rst criterion is a mean-squared-error (M SE ) dened
as:
2
1 X
~ n 1)
M SE =
F (~
x; n) Fo (~
x D;
(21)
N
~
x;n
where N is the total number of pixels, and the second
criterion (M T I ) measures the inconsistency along the
5
6
Conclusion
[6] J.S. Kwon, K. Seo, J. Kim, and Y. Kim. `A motion adaptive de-interlacing method', IEEE Tr. on
Consumer Electronics, 38(3), 1992, pp. 145-150.
In this paper, we outlined some de-interlacing methods that are considered highly relevant, either because
they exist as a commercially available product, or since
they have recently been introduced in the literature as
a high performance method. Additionally, we introduced a new advanced algorithm for scan rate conversion, the Adaptive Recursive de-interlacer. In the
evaluation, we could prove a clear superiority of our
new method over the highly relevant alternatives. Although our concept can be applied to improve any deinterlacing method, we evaluated the improvement of
simple line-averaging. The background of this choice
was our belief that outperforming the currently best
methods by adding an improvement to one of the poorest algorithms is most spectacular.
[7] J.W. Woods and S.C. Han, `Hierarchical Motion
Compensated De-interlacing', Proc. SPIE Visual
Communication and Image Processing VI, Boston,
November 1991.
[8] G. de Haan, P.W.A.C Biezen, H. Huijgen, and
O.A. Ojo, `True Motion Estimation with 3-D Recursive Search Block-Matching', IEEE Tr. on Circuits and Systems for Video Technology, Vol. 3, October 1993, pp. 368-388.
[9] G. de Haan, and P.W.A.C. Biezen, `Sub-pixel motion estimation with 3-D recursive search blockmatching', Signal Processing: Image Communication 6, 1994, pp.229-239.
[10] Preliminary data sheet of Genesis gmVLD8, 8
bit Digital Video Line Doubler, version 2.0, January 1997, available from internet: www.genesisvideo.com.
Acknowledgement
The authors wish to thank Paul Biezen for his contribution to the optimization of the AR de-interlacing
algorithm.
[11] N. Seth-Smith and G. Walker, `Flexible Upconversion for High Quality TV and Multimedia Displays, Proc. Int. Conf. on Consumer Electronics,
ICCE'96, June 1995, Chicago, pp. 338-339.
References
[12] Progressive scan, Zoom, and Noise reduction IC
(PROZONIC), preliminary data sheet of Philips
SAA4990H, October 1996, available from internet:
www.semiconductors.philips.com.
[1] F.M. Wang, D. Anastassiou, and A.N. Netravali,
`Time-recursive deinterlacing for idtv and pyramid
coding', Signal processing: Image Communication
2, pp. 365{374, 1990.
[13] G. de Haan, J. Kettenis, and B. Deloore, `IC for
Motion Compensated 100Hz TV, with a Smooth
Motion Movie-Mode', IEEE Tr. on Consumer Electronics, vol. 42, no. 2, May 1996, pp. 165-174.
[2] P. Delogne, L. Cuverlier, B. Maison, B. Van Caillie,
and L. Vandendorpe, "Improved Interpolation, Motion Estimation, and Compensation for Interlaced
Pictures", IEEE Tr. on Image Processing, Vol. 3,
No. 5, September 1994, pp. 482-491.
[14] G. de Haan and P.W.A.C. Biezen, `Time-recursive
de-interlacing for high-quality television receivers,'
Proc.
[3] L. Vanderdorpe, L. Cuvelier, B. Maison, P. Quelez,
and P. Delogne, `Motion-compensated conversion
from interlaced to progressive formats,' Signal Processing: Image Communication 6, 1994, pp. 193{
211.
of
the
Int.
Workshop
on
HDTV and
the
, Taipei, Taiwan, November
Evolution of Television
1995, pp. 8B25{8B33.
[15] E.B. Bellers and G. de Haan, `Advanced motion
estimation and motion compensated de-interlacing'
Proc. HDTV'96, October 1996, Los Angeles, 3rd
paper of session A2.
[4] G. de Haan and G.F.M. De Poortere, `Method and
apparatus for processing a picture signal', European Patent Application no. EP-A 0 474 28.
[16] E.B. Bellers and G. de Haan, `Advanced
de-interlacing techniques', Proc. ProRISC/IEEE
[5] A.W.M van den Enden and N.A.M. Verhoeckx,
Discrete-time signal processing,
Prentice Hall,
1989, ISBN 0-13-216763-8, pp. 233-.
Workshop on Circuits, Systems and Signal Process-
, Mierlo, The Netherlands, November 1996, pp.
7-17.
ing
6
Gerard de Haan
was born in
Leeuwarden, The Netherlands, on
April 4, 1956. He received the
B.Sc., the M.Sc., and the Ph.D.
degree from Delft University of
Technology in 1977, 1979, and 1992,
respectively. In 1979 he joined the
Philips Research Laboratories in
Eindhoven, where he led several
research projects in the area of image processing, and
participated in a number of European projects. He
has coached students from various universities, and
teaches since 1988 for the Philips Centre for Technical
Training. In 1991/1992, he was a visiting researcher in
the Information Theory Group of Delft University. At
present, he is a Senior Scientist in the group Television
Systems of Philips Research and has a particular
interest in algorithms for motion estimation, scan
rate conversion, and image enhancement. His work
in these areas has resulted in about 35 patents and
patent applications. He is a Senior Member of the
IEEE, and received the rst place of the 1995 ICCE
Outstanding Paper Awards. The Philips 100 Hz TV
with `Natural Motion', based upon his work received
the European Innovation Award of the Year 95/96
from the European Imaging and Sound Association.
Erwin Bellers
was born in Enschede, The Netherlands, on November 14, 1965.
He received the
B.Sc degree from the Technische
Hogeschool Enschede in 1987 and
the M.Sc degree (with distinction)
from the University of Twente in
1993. In December 1993, he joined
Philips Research Laboratories in Eindhoven as a
Research Scientist in the Television Systems Group.
He worked on several video processing projects, with
major interest in video enhancement algorithms. At
present Mr. Bellers contributes to the area of motion
estimation, motion compensation, de-interlacing and
scan rate conversion.
7