Advances in Adaptive Weight
Support Windows
Barry McCullagh,
Keimyung University
Daegu
Presentation Overview

Advances in adaptive support weight
windows:
–
–
–
–
–
Introduce the problems of fixed support weights.
Seminal paper by Yoon and Kweon.
Advances in the past 12-18 months.
Future research.
Questions and answers.
Correlation in stereovision

Correlation windows (support windows) are
used to locate matching pixels in stereo
image pairs.
Window comparison



Traditionally each pixel was given the same
weight.
Comparison performed using SAD, SSD,
NCC or other metric.
Different objects in a window?
Problems
Adaptive weights


First introduced by Yoon and Kweon from
KAIST.
Importance of pixel depends on:
–
–
Color similarity to center pixel.
Distance to the center pixel.
Color similarity

Pixels in the blue boundary are more
important than the other pixels because they
are of similar color.
Euclidean Distance
Examples
Since Yoon and Kweon

Use of adaptive weights has become
popular:
–
–
–
Geodesic support weights
Biologically inspired weighting
Disparity Calibration Systems
Geodesic Support Weights

Local Stereo Matching Using Geodesic
Support Weights.
–

Honsi, Bleyer, Gelautz, Rhemann, ICIP 2009.
Looks at the path to the center pixel.
–
–
Support weight is high if the pixels on the path are
of similar color.
Support weight is low if the pixels on the path are
dissimilar.
Path to the center

Problem identified in this paper:
–
–
Similar colors separated by dissimilar colors might
not belong to the same object.
Not at the same depth.

Original paper: some areas are given
incorrectly high levels of support.
Geodesic Technique


Geodesic technique looks at the path from a
pixel in the window to the center pixel.
Consistent colors indicate the same object
and therefore the same disparity.
Results

Image
Original
Weighting

Geodesic
Weighting

Biologically Inspired Weighting

“Biologically and psychophysically inspired adaptive
support weights algorithm for stereo
correspondence” Lazaros Nalpantidis and Antonios
Gasteratos

Desire to simulate human matching in computers.
Identify techniques humans use and model these in
stereo matching programs.

Biological Matching





Color Matching
Circular Windows
Gestalt Laws
Psychophysically-based weight assignment.
All expanding on the Absolute Difference
calculated between individual pixels at
different disparities.
Color Matching – color space

Color images produce more accurate results than
color.

Different color spaces produce different levels of
accuracy:
–

CIELab better than RGB
However:
–
–
color sensors are RGB.
RGB computations are less demanding.
Color Matching – weighting factors

Human Visual System Weightings:
–
–
–

0.299 Red
0.587 Green
0.114 Blue
Stereovision: Equal weightings
–
Equal amount of information.
Circular Windows


Most approaches use rectangular windows.
Biological model is better approximated
using circular windows.
–
–
Eye is circular
Contribution of neighboring pixels is perfectly
isotropic.
Gestalt Laws

Gestalt – relationships that bond single items
to make a group.
–
–

Makes a pattern instead of parts.
Pattern has different characteristics than the
parts.
These laws are useful to help locate
matching objects in stereovision.
Gestalt Laws

Elements (pixels) making up a group (belong to a
larger object) are governed by the following rules:
–
–
–
–
–
–
–
Proximity: elements that are close to each other.
Similarity: elements similar in an attribute (color, etc).
Continuity: elements that could belong to a smooth larger
feature.
Common fate: elements that exhibit similar behavior.
Closure: elements that could provide closed curves.
Parallelism: elements that seem to be parallel.
Symmetry: elements that exhibit a larger symmetry.
Gestalt Laws for image processing

Three most basic laws can be used to assist
matching:
–
Proximity (or equivalently Distance).
–
Intensity similarity (or equivalently Intensity
dissimilarity).
–
Continuity (or equivalently discontinuity).
Psychophysically-based weight assignment

Assign support weights based on the human
response.

Weber-Fechner law shows the relationship
between perceived change in stimulus and
actual change in stimulus
dS
dp  k
S
S
p  k ln
S0
Combining these laws

Three basic Gestalt laws are combined:
–
Wtotal = Wdist * Wdisc * Wdissim
Performance

Less accurate than the original algorithm
from Yoon and Kweon.

Provides a basis for modeling the human
visual system in computing.
Expanding on Adaptive Weights

“Local stereo matching with adaptive
support-weight, rank transform and disparity
calibration” Zheng Gu, Xianyu Su, Yuankun
Liu and Qican Zhang.

Uses adaptive weight as the starting point
and applies additional techniques to the
result.
Additional Steps

Rank Transform
–

Pixels assigned discrete weights based on
similarity to center pixel.
Disparity Calibration
–
–
Selection of window
Disparity calculation
Rank Transform

Calculates the intensity differences between
the center pixel and pixels in the support
windows. Dl  Il ( x, y )  Il ( x' , y' )
Dr  Ir ( x, y )  Ir ( x' d , y ' )

If corresponding pixels in the two support
windows have different differences, then
those pixels are not included
if ( Dr  Dl )
pixel included
else
pixel not included
Disparity Calibration

Selection of window: applies adaptive
weights but with stronger emphasis on color
similarity.

Recalculation of disparity values based on
those of surrounding pixels
Disparity recalculation


Examine the disparity values for all pixels in
the support window.
Assign the most common disparity to that of
the center pixel.
–
–
Removes outliers
Increase smoothness of the disparity map.
Performance

Performs better than the original adaptive
weight and better than most local methods.
Future Research

Two main goals:
–
improve speed by


–
re-implementing algorithms on parallel architectures.
modifying algorithms to reduce the computational cost.
improve accuracy:


using more complex algorithms.
merging components of different algorithms.
Parallel Architectures

GPUs, CBE, Multi-core CPU
–
–

Powerful, ubiquitous and cost effective.
Efficient implementation of adaptive support
weight algorithms will achieve real-time rates.
Adaptive Support Weight
–
–
implementable on parallel devices.
may be restricted by local memory, available
instructions and order of execution.
Modifying Algorithms

Computation cost is proportional to window
size:
–
–

sliding windows cannot be used in most cases.
ASW often use windows of 35x35 pixels or larger.
Investigate techniques to reduce the window
size while maintaining accuracy:
–
–
hierarchical approaches.
adaptively sized windows.
Modifying Algorithms


Combining and modeling more of the Gestalt
laws.
Varying the importance of these laws.
–
–
Geodesic supports show color is very important.
Are other laws more important?
Merging Algorithms

Adaptive Support Weights forms the basis of
many algorithms.
–
–
e.g. ASW combined with rank transform, modified
to allow use with curvelets.
What other techniques can ASW be combined
with?


As an initial step.
As a final step.
Questions and Answers