A Non-local Cost Aggregation
Method for Stereo Matching
Qingxiong Yang
City University of Hong Kong
2012 IEEE Conference on
Computer Vision and Pattern Recognition
1
Outilne
• Introduction
• Related Works
• Method
• Experimental Results
• Conclusion
2
Introduction
_________________________
3
Introduction
• Goal : Get fast and accurate disparity map.
• Solution : Non-local cost aggregation + MST
• Advantage : Better in low textures region
Low complexity
4
Related Works
_________________________
5
Related Works
1
2
3
4
• Matching cost computation
• Cost (support) aggregation
• Disparity computation and optimization
• Disparity refinement
[21] D. Scharstein and R. Szeliski.
A taxonomy and evaluation of dense two-frame stereo correspondence algorithms.
International Journal of Computer Vision (IJCV), 47:7–42, 2002.
6
Related Works
Local methods
• 1=>2=>3
• A local support region
with winner take all
• Implicit smoothness
• Fast but inaccurate.
Global methods
• 1(=>2)=>3
• Energy minimization
process
(GC,BP,DP,Cooperative)
• Per-processing
• Explicit smoothness
• Accurate but slow
7
Comparison (Rank in Middleburry)
Real time Non-Real
1
-
2
O
-
Method
-
(no Seg.) Cross-based Aggregation>>Scanline
Optimization
3
O
Mean-shift >> BP>>Self-adapting
4
O
Mean-shift >> Cooperative Optimization
-
-
O
Mean-shift>>Color-weighted>>BP
5
-
6
7
O
(no Seg.) Seed Detection>>Scanline Propagation
8
O
Mean-shift>>BP
9
O
Mean-shift(Region-based)>>B-spline
10
O
Up-sample>>Bilateral Filter>>BP
-
-
12
O
(No Seg.)>>Convex Relaxation>>regularization
13
O
Mean-shift>>Image Warping>>BP
11
-
Reference(1/2)
C. Shi, G. Wang, X. Pei, H. Bei, and X. Lin. High-accuracy stereo
matching based on adaptive ground control points. Submitted to
IEEE TIP 2012
 X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang. On
building an accurate stereo matching system on graphics hardware.
GPUCV 2011.
 A. Klaus, M. Sormann and K. Karner. Segment-based stereo
matching using belief propagation and a self-adapting dissimilarity
measure. ICPR 2006.
 Z. Wang and Z. Zheng. A region based stereo matching algorithm
using cooperative optimization. CVPR 2008.
Anonymous. A dense stereo matching with reliability aggregation and
propagation. CVPR 2012 submission 1170.
 Q. Yang, L. Wang, R. Yang, H. Stewénius, and D. Nistér. Stereo
matching with color-weighted correlation, hierarchical belief
propagation and occlusion handling. IEEE TPAMI 2009
9
Reference(2/2)
 X. Sun, X. Mei, S. Jiao, M. Zhou, and H. Wang. Stereo matching
with reliable disparity propagation. 3DIMPVT 2011.
 L. Xu and J. Jia. Stereo matching: an outlier confidence approach.
ECCV 2008.
 M. Bleyer, C. Rother, and P. Kohli. Surface stereo with soft
segmentation. CVPR 2010.
 Q. Yang, R. Yang, J. Davis, and D. Nistér. Spatial-depth super
resolution for range images. CVPR 2007.
Y. Mizukami, K. Okada, A. Nomura, S. Nakanishi, and K. Tadamura.
Sub-pixel disparity search for binocular stereo vision. ICPR 2012
submission 1439.
 S. Zhu, L. Zhang, and H. Jin. A locally linear regression model for
boundary preserving regularization in stereo matching. ECCV 2012.
 M. Bleyer, M. Gelautz, C. Rother, and C. Rhemann. A stereo
approach that handles the matting problem via image warping. CVPR
2009.
10
Method
_________________________
11
Method
1
• Matching cost computation
－ Bilateral Filter
2
• Cost aggregation
－ MST
3
4
• Disparity computation and optimization
• Disparity refinement
－ Median Filter
12
Bilateral Filter
• Every sample is replaced by a weighted average of its
neighbors.
• These weights reflect two forces
• How close are the neighbor and the center sample
• How similar are the neighbor and the center sample
• Edge-preserving and noise reducing smoothing filter
13
Bilateral Filter

q
p

14
Bilateral Filter
Center Sample : p
Neighborhood : q
15
Bilateral Filter
Total Distance
16
Bilateral Filter
Original image
Gaussian wieght
Bilateral wieght
17
Minimum Spanning Tree
• Kruskal's Algorithm
• Scan all edges increasing weight order, if an edge is
safe, add it to F.
4
B
C
4
A
2
4
E
1
F
2
D
1
3
10
Orginal Graph
5
G
5
6
3
4
I
18
H
2
J
3
PPT By Jonathan Davis
4
B
4
A
E
1
3
10
A
1
D
B
4
C
B
4
D
B
10
J
C
2
E
C
1
F
D
5
H
D
6
J
E
2
G
F
3
G
F
5
I
G
3
I
G
4
J
H
2
I
3
5
G
5
B
1
F
2
D
4
C
2
4
A
6
3
4
I
H
2
J
3
J
19
J
Sort Edges
(in reality they are placed in a priority
queue - not sorted - but sorting them
makes the algorithm easier to visualize)
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
20
J
Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
21
J
Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
22
J
Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
23
J
Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
24
J
Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
25
J
Cycle
Don’t Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
26
J
Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
27
J
Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
28
J
Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
29
J
Cycle
Don’t Add Edge
4
B
4
A
E
1
3
10
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
B
10
5
G
5
D
1
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
J
30
J
Add Edge
4
B
4
A
E
1
C
1
F
C
2
E
E
2
G
H
2
J
F
3
G
G
3
I
I
3
J
A
4
B
B
4
D
B
4
C
G
4
J
F
5
I
D
5
H
D
6
J
B
10
J
1
3
10
5
G
5
D
F
2
D
1
C
2
4
A
6
3
4
I
H
2
J
3
31
Minimum Spanning Tree
4
B
4
C
2
A
E
1
Orginal Graph
4
1
F
4
B
A
2
4
E
1
2
D
C
F
2
D
1
3
10
G
5
G
3
5
6
3
4
I
I
H
2
J
3
H
2
J
3
32
Cost Computation
• Cd(p) : matching cost for pixel p at disparity level d
•
: aggregated cost
-- σS and σR : constants used to adjust the similarity.
33
Cost Aggregation on a Tree Structure
• Weight between p and q
• w(p, q) = | I(p)-I(q)| = image gradient
• Distance between p and q
• D(p, q) = sum of weights of the connected edges
• Similarity between p and q
•
• Aggregated cost
•
＝＞
34
Cost Aggregation on a MST
• Claim 1. Let Tr denote a subtree of a node s and r denote
the root node of Tr, then the supports node s received from
this subtree is the summation of the supports node s
received from r and S(s, r) times the supports node r
received from its subtrees.
• Supports r = 𝒗 𝑺(𝒓, 𝒗)𝑪𝒅 𝒗
• Supports s = 𝑺 𝒔, 𝒓 𝑪𝒅 𝒓 = 𝒗 𝑺(𝒔, 𝒗)𝑪𝒅 𝒗 =
𝒗 𝑺(𝒔, 𝒓)𝑺(𝒔, 𝒗)𝑪𝒅 𝒗 = 𝑺(𝒔, 𝒓) 𝒗 𝑺(𝒔, 𝒗)𝑪𝒅 𝒗
s
r
35
Tr
Cost Aggregation on a MST
• Aggregated cost
=>
•
, if node v is a leaf node
• P(vc) denote parent of nodevc
36
Cost Aggregation on a MST
37
Cost Aggregation on a MST
• Aggregated cost
=>
38
Cost Aggregation on a MST
• Cost aggregation process
• Aggregate the original matching cost Cd from leaf
nodes towards root node using Eqn. (6)
• Aggregate
from root node towards leaf nodes
using Eqn. (7)
• Complexity
• Each level：2 addition/subtraction + 3 multiplication
39
Disparity Refinement
•
• D：the left disparity map
• Unstable：occlusion, lack of texture, specularity
• Median filter overlap
40
Experimental Results
_________________________
41
Experimental Results
• Device： a MacBook Air laptop computer with a 1.8 GHz
Intel Core i7 CPU and 4 GB memory
• Parameter：
• σ = 0.1 (non-local cost aggregation)
• Source : Middlebury http://vision.middlebury.edu/stereo/
HHI database(book arrival)
Microsofy i2i database(Ilkay)
42
Experimental Results
• Time：
• Proposed average runtime ： 90 milliseconds (1.25× slower)
• Unnormalized box filter average runtime ： 72 milliseconds.
• Local guided image filter average runtime ： 960 milliseconds
[7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz.
Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.
[24] P. Viola and M. Jones. Robust real-time face detection.
International Journal of Computer Vision, volume 57, pages 137–154, 2003.
43
44
[7] C.Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz.
Fast cost-volume filtering for visual correspondence and beyond. In CVPR ,2011.
Experimental Results
45
Experimental Results
46
Experimental Results
47
Experimental Results
• Different disparity level
(depth of spanning tree)
Max=7
48
Max=10
Max=14
49
Max=16
Max=20
50
Max=50
Max=75
51
Conclusion
_________________________
52
Conclusion
• Contributions
• Outperform all local cost aggregation methods both in
speed and accuracy.
• Present a near real-time stereo system with accurate
disparity results.
• Future works
• Apply to parallel algorithms
• Refine matching cost estimation
53