Stereo Matching with
Color-Weighted Correlation,
Hierarchical Belief Propagation,
and Occlusion Handling
Qingxiong Yang, Student Member, IEEE,
Liang Wang, Student Member, IEEE,
Ruigang Yang, Member, IEEE,
Henrik Stewe´ nius, Member, IEEE,
and David Niste´ r, Member, IEEE
IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE
INTELLIGENCE, VOL. 31, NO. 3, MARCH 2009
Outline
• Introduction
• System Overview
• Methods
•
•
•
•
Initialization
Pixel Classification
Iterative Refinement
Fast-Converging Belief Propagation
• Depth Enhancement
• Experiments
• Conclusion
Introduction
Introduction
• Stereo is one of the most extensively researched
topics in computer vision.
• Energy Minimization framework:
• Graph Cut
• Belief Propagation(BP)
Objective(Contribution)
• To formulate stereo model with careful handling
of:
• Disparity
• Discontinuity
• Occlusion
• Differs from the normal framework in the final
stages of the algorithm
• Outperforms all other algorithms on the average
System Overview
• 1) Initialization
• 1) Initialization
• 1) Initialization
• 2) Pixel Classification
• 3) Iterative Refinement
Initialization
(Block 1)
Initialization
• Input:
• Left Image IL
• Right Image IR
• Output:
• Initial Left Disparity Map DL(0)
• Initial Right Disparity Map DR
• Initial Data Term ED(0)
Image
Color-Weighted
Correlation
Correlation
Volume
CL
CR
Data Term
Initialization
ED(0)
Hierarchical BP
Disparity Map
Initialization
DL(0) DR
Initialization
• Color-weighted Correlation
• To build the Correlation Volume
• Makes the match scores less
sensitive to occlusion boundaries
Image
Color-Weighted
Correlation
Correlation
Volume
CL
CR
Data Term
Initialization
ED(0)
• By using the fact that occlusion
boundaries most often cause
color discontinuities as well
Hierarchical BP
Disparity Map
Initialization
DL(0) DR
Correlation Volume
• Color difference Δxy between pixel x and y
(in the same image)
Ic: Intensity of the color channel c
• The weight of pixel x in the support window of y:
10
21
Color Difference
Spatial Difference
Correlation Volume
• The Correlation Volume[27]:
Pixels in the window
weight
Dissimilarity[1]
weight
•
•
•
•
•
Wx : support window around x
d(yL, yR ) : pixel dissimilarity[1]
xL , yL : pixels in left image IL
xR , yR : corresponding pixels in right image IR
dx : disparity value of pixel XL in IL
dx = arg min CL,x (yL, yR)
x R = x L – dx
y R = y L – dx
Correlation Volume
Disparity Map
Bad Pixel
[1] S. Birchfield and C. Tomasi, “A Pixel Dissimilarity Measure That Is Insensitive to Image
Sampling,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, pp. 401-406, 1998.
[27] K.-J. Yoon and I.-S. Kweon, “Adaptive Support-Weight Approach for Correspondence
Search,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, pp. 650-656, 2006.
Initialization
Image
• Initial Data Term
Color-Weighted
Correlation
• Total energy
= Data Term + Smooth Term
• Computed from Correlation Volume
• Given an iteration index i = 0 here
because it will be iteratively refined
Correlation
Volume
CL
CR
Data Term
Initialization
ED(0)
Hierarchical BP
Disparity Map
Initialization
DL(0) DR
Initial Data Term
• Initial Data Term:
0.2
Correlation
Volume
Average
Correlation X 2
Volume
• Ƞbp : twice the average of correlation volume to exclude
the outliers
Initialization
• hierarchical Belief Propagation
Image
Color-Weighted
Correlation
• Employed with the data term and
the reference image
Correlation
Volume
• Resulting in the initial left and
right disparity maps DL(0) and DR
Data Term
Initialization
Hierarchical BP
Disparity Map
Initialization
DL(0) DR
CL
CR
ED(0)
Pixel
Classification
(Block 2)
Pixel Classification
Input
Output
Pixel Classification
• Mutual Consistency Check
• Requires that the disparity value from the left and
right disparity maps are consistent, i.e.,
• Not Pass : occluded pixel
• Pass : unoccluded pixel
=> Correlation Confidence Measure
Pixel Classification
• Correlation Confidence
• Based on how distinctive the highest peak in a
pixel's correlation profile is
0.04
If > αs stable
Else
unstable
•
•
: the cost for the best disparity value
: the cost for the second best disparity value
dx = arg min CL,x (yL, yR)
Iterative
Refinement
(Block 3)
• Goal: to propagate information from the stable
pixels to the unstable and the occluded pixels
Input
Iteration
Iterative Refinement
• Color Segmentation
• Color segments in IL are extrated by Mean Shift[6]
[6] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach Toward Feature Space
Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, pp. 603-619, 2002.
Iterative Refinement
• Plane Fitting
• Using the disparity values for the stable pixels in each
color segment
• Disparity values are taken from the current hypothesis
for the left disparity map DL(i). (Initial: DL(0))
• The plane-fitted depth map is used as a regularization
for the new disparity estimation.
Iterative Refinement
• Plane Fitting
[10] M.A. Fischler and R.C. Bolles, “Random Sample
Consensus: A Paradigm for Model Fitting with
Applications to Image Analysis and Automated
Cartography,” Comm. ACM, vol. 24, pp. 381-395, 1981.
• Using RANSAC[10]
• Iterates until the plane parameters converge
Iterative Refinement
• Plane Fitting output : D(i)
pf
• The ratio of stable pixels of each segment:
0.7
• If Ratio > ȠS
• Stable pixels: DL(i)
• Unstable, Occluded pixels: D(i)
pf
• If Ratio ≤ ȠS
• All pixels : D(i)
pf
Iteration
Iterative Refinement
• Absolute Difference:
• D(i+1)
L : New Disparity Map
(i) : Plane-fitted Disparity Map
• Dpf
• Data Term:
2.0
0.5
0.05
Belief Propagation
• The core energy minimization of our algorithm
is carried out via the hierarchical BP algorithm.
Total Energy for Pixel X
Data Term
Smooth Term
Max-Product Belief Propagation
• Max-Product BP[25] :
Data Term
Jump Cost
•
: Message vector passed from pixel X to
one of its neighbors Y
[25] Y. Weiss and W. Freeman, “On the Optimality of Solutions of the Max-Product Belief Propagation
Algorithm in Arbitrary Graphs,” IEEE Trans. Information Theory, vol. 2, pp. 732-735, 2001.
Max-Product Belief Propagation
Y
x
Max-Product Belief Propagation
• Jump Cost:
Disparity Difference
of pixel X and its neigbor Y
1
•
•
•
•
•
dx : Disparity of pixel X
d : Disparity of pixel Y (X’s neighbor)
αbp : The number of disparity levels / 8
ρs : 1 – (normalized average color difference)
ρbp : The rate of increase in the cost
Max-Product Belief Propagation
• Total Energy for pixel X:
Data Tem
Smooth Tem
• Finally the label d that minimizes the total Energy
for each pixel is selected.
Hierarchical Belief Propagation
• Standard loopy BP algorithm is too slow.
• Hierarchical BP[9] runs much faster while
maintaining comparable accuracy.
• Works in a coarse-to-fine manner
[9] P.F. Felzenszwalb and D.P. Huttenlocher, “Efficient Belief Propagation for
Early Vision,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol.
1, pp. 261-268, 2004.
Hierarchical Belief Propagation
Coarser
(Level 1)
Finer
(Level 0)
Fast-Converging Belief Propagation
• A large number of iterations is required to guarantee
convergence in a standard BP algorithm.
• Fast-Converging BP effectively removes the
redundant computation.
0.1
• Only updating the pixels that have not yet converged
(value bigger than ȠZ )
Fast-Converging Belief Propagation
Depth
Enhancement
Depth Enhancement
• To reduce the discontinuities caused by the
quantization
• Sub-pixel Estimation algorithm is proposed.
• Cost Function:
Depth Enhancement
• The depth with the minimum of the cost function:
• d: the discrete depth with the minimal cost
• d+: d+1
• d- : d-1
• Replace each value with the average of those values
that are within one disparity over a 9 x 9 window
Experiments
Experiments
Parameter Settings Used Throughout:
Experiments
Parameter Settings Used Throughout:
Experiments
Results on the Middlebury Data Sets with Error Threshold 1
Error%
nonocc : The subset of the nonoccluded pixels
disc : The subset of the pixels near the occluded areas.
all : The subset of the pixels being either nonoccluded or half-occluded
Experiments
Results on the Middlebury Data Sets with Error Threshold 0.5
Color-Weighted Correlation Voume :
Initial Hierarchical BP:
Plane fitting:
Integer-Based Disparity Map:
Depth Enhancement:
Ground Trueh:
Conclusion
Conclusion
• Propose a stereo model based on
•
•
•
•
•
energy minimization
color segmentation
plane fitting
repeated application of hierarchical BP
depth enhancement
• A fast converging BP approach is proposed.
• Preserves the same accuracy as the standard BP
• The runtime is sublinear to the number of iterations.
Conclusion
• The algorithm is currently outperforming the
other algorithms on the Middlebury data sets on
average.
• There’s space for Improvement:
• Only refined the disparity map for the reference
image
• [19] suggests that, by generating a good disparity
map for the right image, the occlusion constraints can
be extracted more accurately.
J. Sun, Y. Li, S.B. Kang, and H.-Y. Shum, “Symmetric Stereo Matching for Occlusion Handling,”
Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 399-406, 2005.