Multiple-View Reconstruction from Scene Knowledge
SYMMETRY & MULTIPLE-VIEW GEOMETRY
• Fundamental types of symmetry
• Equivalent views
• Symmetry based reconstruction
3D Reconstruction from Scene Knowledge
MUTIPLE-VIEW MULTIPLE-OBJECT ALIGNMENT:
• Scale alignment: adjacent cells in a single view
• Scale alignment: same cell in multiple views
APPLICATIONS & Experiments:
• Building 3D geometric models
• Symmetry extraction, detection, and matching
• Camera calibration
CS 682 J. Kosecka
1
2
CS 682 J. Kosecka
Scene knowledge and symmetry
Scene knowledge and symmetry
Parallelism (vanishing point)
Orthogonality
Translational invariance
Rotation and reflection
Symmetry is ubiquitous in man-made or natural environments
CS 682 J. Kosecka
3
CS 682 J. Kosecka
4
1
SYMMETRY & MUTIPLE-VIEW GEOMETRY
Wrong assumptions
Ames room illusion
Necker’s cube illusion
• Why does an image of a symmetric object give away its structure?
• Why does an image of a symmetric object give away its pose?
• What else can we get from an image of a symmetric object?
5
CS 682 J. Kosecka
Equivalent views from rotational symmetry
CS 682 J. Kosecka
6
Equivalent views from reflectional symmetry
90O
CS 682 J. Kosecka
7
CS 682 J. Kosecka
8
2
Recovery using rectangular structure
Equivalent views from translational symmetry
• Recovery of the camera displacement from a planar rectangular
structure
• Rectangle – reflectional symmetry
• Square – rotational and reflectional symmetric
• Special cases of symmetric objects
CS 682 J. Kosecka
9
CS 682 J. Kosecka
Camera pose recovery
„
10
Homography Estimation
Assume partially calibrated camera
„
Decouple known and unknown structure
• Explicitely parametrize the homography
• Estimate the unknown homography
• Explicitely parametrize the unknown structure
CS 682 J. Kosecka
11
CS 682 J. Kosecka
12
3
Homography Factorization
„
Example
Exploiting orthogonality constraints
100
0
200
20
• Directly estimate the focal length
40
300
60
80
400
100
500
120
50
0
y
z
50
0
−50
600
x
50
100
150
200
250
300
350
400
450
−100
−50
• remaining parameters and final pose
• With shared segment the pose can be reconciled and we
obtain single consistent pose recovery up to scale and
error ~ 3 degrees
CS 682 J. Kosecka
13
Example for multiview matching
14
Example
• Recovery of the camera displacement from a planar structure
• Recovery of the camera displacement from a planar structure
CS 682 J. Kosecka
CS 682 J. Kosecka
15
CS 682 J. Kosecka
16
4
Symmetric multiview geometry (chap 10.)
Symmetry-based reconstruction (experiments)
Rotational
symmetry
Definition. A set of 3-D features S is called a symmetric structure
if there exists a non-trivial subgroup G of E(3) that acts on it such
that for every g in G, the map
is an (isometric) automorphism of S. We say the structure S has a
group symmetry G.
Detailed treatment in chapter 10 (some examples next)
17
CS 682 J. Kosecka
Symmetry-based reconstruction (experiments)
18
CS 682 J. Kosecka
Symmetry-based reconstruction (experiments)
Reflectional
symmetry
Translational
symmetry
Algorithm
Algorithm
1. Recover R’,T’ from homography constraint
1. Recover T’ from homography constraint
(no rotation in this case)
known,
recovered
Projection of the origin of the
Symmetric structure
CS 682 J. Kosecka
19
CS 682 J. Kosecka
20
5
Alignment of multiple symmetric structures
Alignment of multiple symmetric structures
Pick the image of a point
on the intersection line
?
CS 682 J. Kosecka
21
Alignment of multiple symmetric structures
CS 682 J. Kosecka
CS 682 J. Kosecka
22
Alignment of multiple symmetric images
23
CS 682 J. Kosecka
24
6
Alignment of multiple symmetric images
CS 682 J. Kosecka
Alignment of multiple symmetric images
25
Alignment of multiple symmetric images
CS 682 J. Kosecka
26
APPLICATION: Symmetry detection and matching
Extract, detect, match symmetric objects
across images (over an arbitrary motion),
and recover the camera poses.
CS 682 J. Kosecka
27
CS 682 J. Kosecka
28
7
Symmetry verification & pose recovery
Segmentation & polygon fitting
• Symmetry verification (rectangles)
• Color-based segmentation (mean shift)
• Polygon fitting.
CS 682 J. Kosecka
• Single-view pose recovery
29
30
CS 682 J. Kosecka
Pose and 3D recovery
Symmetry-based matching
• Only one set of camera poses is consistent
with all correctly detected symmetry objects
87.6o
Accuracy (aspect ratios)
Reconstruction
CS 682 J. Kosecka
31
CS 682 J. Kosecka
Ground Truth
Whiteboard
1.54
1.51
Table
1.01
1.00
32
8
Multiple-view matching and recovery
CS 682 J. Kosecka
APPLICATION: Building 3D geometric models
33
34
Building 3D geometric models (rendering)
Building 3D geometric models (camera poses)
CS 682 J. Kosecka
CS 682 J. Kosecka
35
CS 682 J. Kosecka
36
9
APPLICATION: Calibration from symmetry
APPLICATION: Calibration from symmetry
Calibrated homography
Calibration with a rig is also simplified: we only need
to know that there is sufficient symmetries, not
necessarily the 3D coordinates of points on the rig.
Uncalibrated homography
(vanishing point)
CS 682 J. Kosecka
37
CS 682 J. Kosecka
38
SUMMARY
Multiple-view 3-D reconstruction in presence of symmetry
• Symmetry based algorithms are accurate, robust, and simple.
• Methods are baseline independent and object centered.
• Alignment and matching can and should take place in 3D space.
• Camera self-calibration and calibration are simplified and linear.
Related applications
• Using symmetry to overcome occlusion.
• Reconstruction and rendering with non-symmetric area.
• Large scale 3D map building of man-made environments.
CS 682 J. Kosecka
39
10