EECS 442 – Computer vision
Volumetric stereo
•Definition
•Shape from Contours
•Shape from Shadows
•Voxel coloring
Today’s lecture is a special topic
“Traditional” Stereo
P
p
p’
O1
O2
Goal: estimate the position of P given the observation of
P from two view points
Assumptions: known camera parameters and position (K, R, T)
“Traditional” Stereo
P
p
p’
O1
Subgoals:
1. Solve the correspondence problem
2. Use corresponding observations to triangulate
O2
Volumetric stereo
Scene volume
O1
1. Hypothesis: pick up a point within the volume
2. Project this point into 2 (or more) images
3. Validation: are the observations consistent?
Assumptions: known camera parameters and position (K, R, T)
O2
Consistency based on cues such as:
- Contours/silhouettes
- Shadows
- Colors
Contours are a rich source of
geometric information
Apparent contour
DEFINITION: projection of the locus of points on the surface which separate the visible
and occluded parts on the surface
[sato & cipolla]
apparent
contour
Image
Camera
Silhouettes
silhouette
Image
Camera
Why contours are
interesting visual cues?
™ Provide information in absence of other visual cues
No texture
No shading
Why contours are
interesting visual cues?
™ Relatively easy to detect
Object
Easy contour
segmentation
Image
Camera
How can we
use contours?
Object
Visual cone
Image
Camera
Object
apparent
contour
How can we
use contours?
Object
Image
Camera
Image
object’s
contour
Visual
cone
Image object’s contour
Object
in 2D:
Camera
Image
View point 1
Object
The views are
calibrated
Object
Object estimate
(visual hull)
View point 2
how to perform visual cones intersection?
™ decompose visual cone in polygonal surfaces
(among others: Reed and Allen ‘99)
Using contours/silhouettes in volumetric stereo
also called Space carving
[ Martin and Aggarwal (1983) ]
voxel
Computing visual hull in 2D
Computing visual hull in 2D
Computing visual hull in 2D
Computing visual hull in 2D
Visual hull:
an upper bound estimate
Consistency:
A voxel must be projected into a silhouette in each image
Space Carving
has complexity …
O(N3)
™ Octrees
(Szeliski ‘93)
N
N
N
Complexity reduction: octrees
1
1
1
Complexity reduction: octrees
Subdiving volume in voxels of progressive smaller size
2
2
2
Complexity reduction: octrees
4
4
4
Complexity reduction: 2D example
4 voxels analyzed
Complexity reduction: 2D example
16 voxels analyzed
Complexity reduction: 2D example
52 voxels analyzed
Complexity reduction: 2D example
16x34 voxels analyzed
1+ 4 + 16 + 52 + 34x16 =
617 voxels have been analyzed in total
(rather than 32x32 = 1024)
Advantages of space carving
™ Robust and simple
™ No need to solve for correspondences
Limitations of space carving
™ Accuracy function of
number of views
Not a good
estimate
What else?
Limitations of space carving
™ Concavities
are not modeled
Concavity
For 3D objects:
Are all types of concavities
problematic?
Limitations of space carving
Concavity
™ Concavities
are not modeled
Visual hull
™ Laurentini (1995)
Closest approximation
(hyperbolic regions are ok)
Conservative
Space carving: a classic setup
Camera
Object
Courtesy of seitz & dyer
Turntable
Space carving: a classic setup
Space carving: Experiments
™ 24 poses (15O)
™ voxel size = 2mm
6cm
10cm
Space carving: Experiments
30 cm
™ 24 poses (15O)
™ voxel size = 1mm
Space carving: Conclusions
™ Robust
™ Produce conservative estimates
™ Concavities can be a problem
™ Low-end commercial 3D scanners
blue backdrop
turntable
Contours in the computer
vision literature
Analyzing changes in apparent contours
™ Giblin and Weiss (1987)
™ Cipolla and Blake (1992)
™ Vaillant and Faugeras (1992)
™ Ponce (’92), Zheng(‘94)
™ Furukawa et al. (‘05…)
Picture from of Sato & Cipolla
Volumetric stereo
•Definition
•Shape from Contours
•Shape from Shadows
•Voxel coloring
Self-shadows are visual cues
for shape recovery
Self-shadows indicate
concavities
(no modeled by contours)
Shape from shadow in the literature
™ Shafer & Kanade (’83)
™ Hatzitheodorou & Kender (’89)
™ et al…
Hatzitheodorou & Kender (’89)
™ shape as a 2.5D terrains -- i.e. surface modeled as f(x)
™ accurate shadow detection – i.e. xb and xe
Shape from shadow in the literature
™ Shafer & Kanade (’83)
™ Hatzitheodorou & Kender (’89)
™ et al…
Hatzitheodorou & Kender (’89)
™ shape as a 2.5D terrains -- i.e. surface modeled as f(x)
Using shadows in volumetric stereo
™ accurate shadow detection – i.e. xb and xe
Shadow Carving
Savarese et al ‘01
Self-shadows
+
™ Robust with respect to shadow estimates
™ Object with arbitrary topology (no 2.5D terrains)
Object’s upper bound
Shadow carving: setup
Object
Light
Image
Camera
Let’s define a 2D model…
2D slice model
Object
Light
Image
Camera
2D slice model
Object
Light
From 3D
Image
to 2D
Camera
Object
Image line
Light
Camera
The idea
Object
Object’s
upper
bound
Shadow
Image line
Image
shadow
Camera
Light
The idea
Object
Object’s
upper
bound
Shadow
Image line
Image
shadow
Camera
Light
The idea
Object
Object’s
upper
bound
Shadow
Image line
Image
shadow
Camera
Light
The idea
Object
Object’s
upper
bound
Shadow
Image line
Image
shadow
Camera
Light
The idea
Object
Object’s
upper
bound
Shadow
Image line
Image
shadow
Camera
Light
The idea
Object
Object’s
upper
bound
Shadow
Image line
Image
shadow
Camera
Light
The idea
Reconstruction after 3 iterations
What is known?
Camera
Image
line
^
S
™ Light known
™ Camera known
™ Object unknown
™ Upper bound
object estimate
Object
Light
source
What can we carve?
Camera
Image
line
Light
source
^
S
Image
shadow
Shadow
Object
What can we carve?
Camera
™ shadow cone
™ upper-bound
shadow
™ light cone
Image
line
Light
source
^
S
Image
shadow
™ carvable area
Shadow
Object
What can we carve?
Camera
™ shadow cone
™ upper-bound
shadow
™ light cone
Image
line
Light
source
^
S
Image
shadow
™ carvable area
Shadow
The upper-bound
estimate is refined
Object
What can we carve?
Camera
Image
line
Light
source
^
S
Image
shadow
™ No further volume
can be removed
Shadow
™ Carving process
always conservative
Proof of correctness
Object
Algorithm: step k
Camera
Image
line
Image
Light
source
Image
shadow
Upper-bound
from step k-1
Object
Algorithm: step k
Camera
Image
line
Image
dual image
shadow
Light
source
Image
shadow
?
Upper-bound
from step k-1
Object
Algorithm: step k
Camera
Image
line
Image
dual image
shadow
Light
source
Image
shadow
?
Upper-bound
from step k-1
Consistency:
A voxel must be projected into both image shadow and
dual image shadow
Object
Algorithm: step k
Camera
Image
line
Image
“light” image
shadow
Light
source
Image
shadow
Upper-bound
from step k-1
Complexity?
O(2N3)
Object
Shadow carving: the setup
Camera
Object
Array of
lights
Turntable
Shadow carving: the setup
Camera
Object
Array of
lights
Turntable
Experimental results
Simulating the system with 3D studio Max
- 24 positions
- 4 lights
- 72 positions
- 8 lights
- 24 positions
- 4 lights
- 72 positions
- 8 lights
- 16 positions
- 4 lights
Shadow carving: summary
™ Produces a conservative volume estimate
™ Accuracy depending on view point and
light source number
™ Limitations with specular & low albedo
regions
Volumetric stereo
•Definition
•Shape from Contours
•Shape from Shadows
•Voxel coloring
Voxel Coloring
Seitz & Dyer (‘97)
R. Collins (Space Sweep, ’96)
™ color/photo-consistency
™ Jointly model structure and
appearance
Basic idea
Is this voxel in or out?
View1
View3
View2
Basic idea
View1
View3
View2
Uniqueness
Multiple consistent scenes
Ijcv – seitz & dyer
Uniqueness
Multiple consistent scenes
Ijcv – seitz & dyer
Tractability
M colors
N
N
N
™ Combinatorial number possible assignments!
™ Exhaustive search not feasible
™ Model occlusions!
™ Use visibility constraint
The algorithm
C
Algorithm complexity
™ voxel coloring visits each N3 voxels only once
™ project each voxel into L images
Æ O(L N3)
NOTE: not function of the number of colors
Photoconsistency test
λ = corr (
,
Image 1
If
λ < Thresh Î voxel consistent
)
Image 2
A critical assumption: Lambertian surfaces
I1(p)
I1(p) = I2(p)
I2(p)
Non Lambertian surfaces
Experimental results
Dinosaur
™ 72 k voxels colored
™ 7.6 M voxels tested
™ 7 min to compute on a 250MHz
Experimental results
Flower
™ 70 k voxels colored
™ 7.6 M voxels tested
™ 7 min to compute on a 250MHz
Experimental results
Room + weird people
Voxel coloring: conclusions
Good things:
™ model intrinsic scene colors and texture
™ no assumptions on scene topology
Voxel coloring: conclusions
Good things:
™ model intrinsic scene colors and texture
™ no assumptions on scene topology
Limitations:
™ Constrained camera positions
™ Lambertian assumption
space carving
™ Space carving is a
binary voxel
coloring
out
in
Shadow carving
Camera
Light
source
^
S
no visibility
constraint
needed
Further contributions
™ A Theory of Space Carving
(Kutulakos & Seitz ‘99)
™ Voxel coloring in more general framework
™ No restrictions on camera position
™ Probabilistic Space Carving
(Broadhurst & Cipolla, ICCV 2001)
(Bhotika, Kutulakos et. al, ECCV 2002)