Rate-Constrained Multi-Hypothesis
Motion-Compensated Prediction
for Video Coding
Markus Flierl
Thomas Wiegand
Bernd Girod
Telecommunications Laboratory
University of Erlangen-Nuremberg
Erlangen, Germany
Image Processing Department
Heinrich-Hertz-Institute
Berlin, Germany
Information Systems Laboratory
Stanford University
Stanford, CA, USA
[email protected]
[email protected]
[email protected]
Overview
1. Multi-hypothesis motion-compensated prediction with
multiple reference frames
2. Rate-constrained multi-hypothesis motion estimation
3. Integration into a hybrid video coder
4. Experimental results for multiple hypotheses on multiple
reference frames
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
1
Motivation
Multi-hypothesis prediction
for P-Frame coding with one
reference frame
time
current frame
What do we gain when we
combine both concepts?
Long-term memory prediction
for P-frame coding (requires
picture reference)
time
current frame
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
2
Multi-Hypothesis Motion-Compensated Prediction
Multi-hypothesis prediction
for P-frame coding with
two hypotheses and multiple
reference frames
time
current frame
⇒ Each prediction signal (hypothesis) is assigned a motion vector and
picture reference
⇒ Hypotheses are linearly combined with constant scalar weights
⇒ Hypotheses are chosen only from previous decoded frames
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
3
Multi-Hypothesis Motion Estimation
⇒ Improved prediction performance and higher bit-rate due to
more than one hypothesis per block
⇒ A trade-off between prediction performance and bit-rate
(motion vectors and picture references) is necessary
,→ Rate-constrained multi-hypothesis motion estimation
⇒ A full search algorithm for a 2-hypothesis is not practical.
,→ Successive improvement of 2 suboptimal conditional solutions
by an iterative algorithm
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
4
Hypothesis Selection Algorithm
0: Assuming 2 hypotheses (c1, c2), the rate-distortion cost function
°
°2
°
1 °
1
°
+ λ[r(c1) + r(c2)]
j(c1, c2) = °s − c1 − c2°
2
2 °2
is subject to minimization for each original block s, given the Lagrange multiplier
(0) (0)
λ. Set i := 0 and guess 2 initial hypotheses (c1 , c2 ).
1: Minimize the rate-distortion cost function by full search for
(i+1)
a: hypothesis c1
(i)
while fixing hypothesis c2
(i+1)
min j(c1
(i+1)
c1
(i+1)
b: and hypothesis c2
(i)
, c2 )
while fixing the complementary hypothesis.
(i+1)
min j(c1
(i+1)
c2
(i+1)
, c2
)
2: Set i := i + 1 and continue with step 1 as long as the cost function decreases.
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
5
Multi-Hypothesis Modes for H.263
⇒ INTER-Mode
• 1 motion vector and picture
reference per block
s
+
-
Q
ŝ
+
P
• Data for residual encoding
⇒ INTER2H-Mode
• 2 motion vectors and picture
references per block
• Data for residual encoding
0.5
s
+
-
Q
P2
ŝ
+
0.5
P1
⇒ INTER4V2H-Mode
Multi-hypothesis block pattern indicates 1 or 2 hypotheses per 8 × 8 block
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
6
Rate-Constrained Decisions
• Multi-hypothesis prediction improves the prediction signal by
spending more bits for the side-information
• Encoding of the prediction error and its associated bit-rate also
determines the quality of the reconstructed block
⇒ Rate-constrained multi-hypothesis motion estimation independent
of prediction error encoding is an efficient and practical solution
⇒ A rate-constrained decision which also incorporates the encoding
of the prediction error determines whether 1 or 2 hypotheses are
used.
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
7
Previous Results
⇒ Practical video coding schemes should utilize two jointly optimized
hypotheses. (VCIP 2000)
⇒ Theoretical investigations on the efficient number of hypotheses
support this finding. (VCIP 2000)
⇒ Variable block size and multi-hypothesis prediction can be
successfully combined. (VCIP 2000)
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
8
Multiple Hypotheses on Multiple Reference Frames
400
BL
BL+MHP
380
• Fixed block size prediction with and
without MH prediction
10%
360
R [kbit/s]
340
⇒ MH prediction with
1 reference frame
saves 10% of bitrate
19%
320
300
280
260
240
1
2
5
10
number of reference frames M
20
⇒ MH prediction with
20 reference frames
saves 19% of bitrate
Mobile & Calendar (QCIF, 10 fps, 10 s) at 34 dB PSNR
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
9
Multiple Hypotheses on Multiple Reference Frames
400
BL+VBS
BL+VBS+MHP
380
• Variable block size
prediction with and
without MH prediction
360
R [kbit/s]
340
10%
⇒ MH prediction with
1 reference frame
saves 10% of bitrate
320
300
280
21%
260
240
1
2
5
10
number of reference frames M
20
⇒ MH prediction with
20 reference frames
saves 21% of bitrate
Mobile & Calendar (QCIF, 10 fps, 10 s) at 34 dB PSNR
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
10
Multiple Hypotheses on Multiple Reference Frames
90
85
• Fixed and variable
block size prediction
with and without
MH prediction
BL
BL+MHP
BL+VBS
BL+VBS+MHP
11%
R [kbit/s]
80
⇒ MH prediction with
1 reference frame
saves 11% for FBS
and 7% for VBS
prediction
75
70 7%
16%
65
60
55
1
10%
2
5
10
number of reference frames M
20
⇒ MH prediction with
20 reference frames
saves 16% for FBS
and 10% for VBS
prediction
Foreman (QCIF, 10 fps, 10 s) at 34 dB PSNR
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
11
Multiple Hypotheses on Multiple Reference Frames
38
• VBS + MH prediction with 1 and 20
reference frames.
37
36
35
34
⇒ MH prediction with
1 reference frame
gains up to 0.8 dB
PSNR [dB]
33
32
31
30
⇒ MH prediction with
20 reference frame
gains up to 1.6 dB
29
28
27
M= 1, VBS
M= 1, VBS + MHP
M=20, VBS
M=20, VBS + MHP
26
25
24
0
50
100 150 200 250 300 350 400 450 500 550 600
R [kbit/s]
Mobile & Calendar (QCIF, 10 fps, 10 s)
⇒ Efficiency of MH
prediction improves
for a larger number
of reference frames
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
12
Example: Mobile & Calendar
M = 1 without MHP
Sequence at 200 kbit/s and 30 dB
M = 20 with MHP
Sequence at 200 kbit/s and 33 dB
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
13
Conclusions
⇒ Long-term memory enhances the efficiency of multihypothesis motion-compensated prediction for video
coding.
⇒ This observation is independent from the block size
used for prediction.
⇒ The bit-rate savings saturate for 20 reference frames.
Markus Flierl: Rate-Constrained Multi-Hypothesis Motion-Compensated Prediction
14