3D City Modeling using Cognitive Loops
Nico Cornelis1
Bastian Leibe2
Kurt Cornelis1
1 KU
Luc Van Gool1,2
2 ETH
Leuven
Leuven, Belgium
Zurich
Zurich, Switzerland
{firstname.lastname}@esat.kuleuven.be
{leibe,vangool}@vision.ee.ethz.ch
D ESCRIPTION
In this video we show the combined results from two recent
publications [1], [2]. In [1], we introduce a real-time 3D
City Modeling algorithm which is able to build compact 3D
representations of cities using the assumption that building
facades and roads can be modeled by simple ruled surfaces. A
typical result of this method can be seen in the left of Figure 1.
The main advantage of this algorithm is its exceptional speed.
It can process the full SfM and reconstruction pipeline at
25-29fps — thus, the reconstructed model can directly be
created online, while the survey vehicle is driving through
the streets. However, due to the simple geometry assumptions,
this original algorithm is unable to model cars which are everpresent in cities and obviously visually degrade our resulting
3D city model.
In [2], we therefore propose to combine the 3D reconstruction with an object detection algorithm based on Implicit Shape
Models [3]. The two components are integrated in a cognitive
feedback loop. The 3D reconstruction modules inform object
detection about the scene geometry, which greatly helps to
improve detection precision. Using the knowledge of camera
parameters and scene geometry from [1], the 2D car detections
are temporally integrated in a world coordinate frame, which
allows to obtain precise 3D location and orientation estimates
(Figure 2). Those can then be used to instantiate the virtual
3D car models which improve the visual realism of our final
3D city model (Figure 1).
Our final system is able to create an automatic 3D city
model from the input video streams of a survey vehicle,
identify the locations of cars in the recorded real-world scene,
and replace them by virtual 3D models in the reconstruction.
Besides improving the visual realism of the final 3D model
(see Figure 1), this has as the additional benefit that it also
solves privacy issues by removing personalized information
from the resulting final city model. Therefore, object recognition can aid 3D reconstruction in achieving more realistic
results. On the other hand, the object recognition algorithm
itself can benefit from the higher-level scene knowledge which
is available through 3D reconstruction. It is exactly this bidirectional nature of interactions between both the reconstruction
and recognition algorithm which earns it the name of cognitive
loop.
ACKNOWLEDGMENTS
This work is supported by the European IST Programme
DIRAC Project FP6-0027787. We also wish to acknowledge
Fig. 1
( LEFT ) O RIGINAL 3D C ITY M ODEL ( RIGHT ) F INAL 3D C ITY M ODEL
WITH
3D
VIRTUAL CARS WHOSE POSITIONS HAVE BEEN DETERMINED BY
THE OBJECT RECOGNITION MODULE .
Fig. 2
( LEFT ) C AR DETECTIONS USING AN ISM- BASED OBJECT DETECTOR AND
SCENE GEOMETRY CONSTRAINTS . ( RIGHT )
3D CAR LOCATION ESTIMATES
OBTAINED BY TEMPORAL INTEGRATION . T HEY SERVE TO INSTANTIATE
THE VIRTUAL CARS IN THE FINAL
3D CITY MODEL .
the support of the K.U.Leuven Research Fund.s GOA project
MARVEL, Wim Moreau for the construction of the stereo rig,
and TeleAtlas for providing additional survey videos to test on.
R EFERENCES
[1] N. Cornelis and K. Cornelis and L. Van Gool. “Fast compact city modeling for navigation pre-visualization,” in IEEE International Conference on
Computer Vision and Pattern Recognition (CVPR’06), New York, 2006.
[2] N. Cornelis and B. Leibe and K. Cornelis and L. Van Gool. “3D city
modeling using cognitive loops,” in 3rd Intern. Symp. on Data Processing,
Visualization, and Transmission (3DPVT’06), Chapel Hill, NC, June
2006.
[3] B. Leibe and E. Seemann and B. Schiele. “Pedestrian Detection in
Crowded Scenes,” in IEEE International Conference on Computer Vision
and Pattern Recognition (CVPR’05), San Diego, 2005.