Inferring Temporal Order of
Images from 3D Structure
Grant Schindler
Frank Dellaert
Sing Bing Kang
Gatech
Gatech
MSR, Redmond
Outline
•
•
•
•
Problem Definition
Algorithm Overview
Applications
Things to think about
What can be done with n images?
What can be done with n images?
Feature Extraction
Correspondence
Structure from Motion
What Now?
Temporal Ordering and 4D
Walkthrough
1920
1951
1966
2003
Outline
•
•
•
•
Problem Definition
Algorithm Overview
Applications
Things to think about
F1
F2
C1
I1
F3
I2
C2
SFM tells us:
Camera Matrices
3D points for features
Visibility of 3D points in images
F1
F2
C1
I1
F3
I2
C2
SFM
info
Image 1 (I1)
Image 2 (I2)
F1
Visible
???
F2
???
Visible
F3
Visible
???
F1
F2
C1
I1
F3
I2
C2
SFM
info
Image 1 (I1)
Image 2 (I2)
F1
Visible
???
F2
Occluded
Visible
F3
Visible
Out of View
F1
F2
C1
I1
F3
I2
Notion of
missing at that
time
C2
SFM
info
Image 1 (I1)
Image 2 (I2)
F1
Visible
???
F2
Occluded
Visible
F3
Visible
Out of View
Classification of 3D point for an Image
• Visible – SFM tells us
• Out of View – Camera Matrix tells us
• Missing / Occluded - ???
– for an occluded point, there must be an occluder
Point ‘F’ Missing / Occluded ?
• Find out occluders
– 3D Triangulation of all visible points
– No occluder should occlude a visible point
• Visibility check for F
occluders
F1
occluded
F2
missing
Camera centre
Visibility Matrix
I1
I2
...
In
F1
S11
S12
…
S1n
F2
S21
S22
…
S2n
…
…
…
…
…
Fm
Sm1
Sm2
…
Smn
Sij € {visible, occluded, missing, out of view }
Constraints of Visibility Matrix
Combinatorial Algorithm to find Best
Configuration
• Local search method
• Starts at a random configuration
• Small moves which reduce the number of
constraints violated
Issues leading to Finding
Approximate Solution
• Problems in feature detection
• Mislabeling of points
– Triangulation strategy
– Inaccuracy in SFM
– Features occluded by undetected occluders (fog,
trees etc)
Structural Segmentation from
Temporal Ordering
• Clustering temporally coherent features
• Separate triangulation of each cluster
• Texture by projecting on images
Algorithm Overview
Possible Applications
•
•
•
•
Historic Preservation
Virtual Tourism
Urban Planning
Spatio-Temporal models as a new way to
interact with a vast collection of imagery
Things to Think about
• Feature extraction (done manually here)
• Better methods for finding occluders – problems with
triangulation method
– Very coarse structure
– Can have triangles for no occluders
• Using Goesele’s work (ICCV 2007) for structural
segmentation
– High number of images required (this paper used 20-30 images)
• Validation
– Correspondence between the best ground truth solution and
best approximate solution of ordering
• Increasing the scale technically and physically
An Interesting Insight….
• Assume no building can be demolished once
it’s built
• Assume every image is a node of graph
• Edge from A to B if A precedes B
– (B has visible features missing in A )
• Directed Graph (acyclic in ideal case)
B1
B2
B3
A
B2
B2
B
C
Input Images
B
A
C
Directed Graph (Acyclic in ideal case)
B3
B1
B2
B3
A
B2
B2
B
C
B3
Input Images
B
A
B
C
Directed Graph (Acyclic in ideal case)
C
Topological Sort
Solution !
A
More insights about Graph Model
• Every edge has a confidence value based on
quality of features and SFM procedure
• In general, there can be back edges in this
graph
• Problem to find the best topological sort
maximizing the confidence measure
Graph Complexity
• Increases with more constraints
• Modeling constraints involving more than 2
images at a time - how??