True Single View Point Cone Mirror Omni-Directional Catadioptric System1
Shih-Schön Lin, Ruzena Bajcsy
GRASP Laboratory, Computer and Information Science Department
University of Pennsylvania,
[email protected] , [email protected] (also, [email protected])
Abstract
Pinhole camera model is a simplified subset of geometric
optics. In special cases like the image formation of the
cone (a degenerate conic section) mirror in an omnidirectional view catadioptric system, there are more
complex optical phenomena involved that the simple
pinhole model can not explain. We show that using the
full geometric optics model a true single viewpoint cone
mirror omni-directional system can be built. We show
how such system is built first, and then show in detail
how each optical phenomenon works together to make
the system true single viewpoint. The new system
requires only simple off-the-shelf components and still
outperforms other single viewpoint omni-systems for
many applications.
1. Introduction
With the ideal pinhole camera model in mind, it has
been shown that conic section convex mirrors can be
used to create single viewpoint catadioptric omnidirectional view systems that can be unwarped
perspectively without systematic distortions [1]. It has
also been shown that cone mirror is a degenerate shape of
the conic section family that still possesses a single
viewpoint [1]. The problem was that, with a true pinhole
camera the single viewpoint of a cone (located at its tip)
seems not useable because all the visible rays seem to be
blocked by the cone itself [1], while if the virtual pinhole
was placed at some distance away from the tip of the
cone, the whole system no longer possesses a single
viewpoint [10;1;8].
With this problem in mind, cone mirrors have not
been used to generate precisely unwarped perspective
images. However, cone mirrors have been used to aid
navigation, collision avoidance, and pipe inspections
without single viewpoint [9;10;2;6;8] In those
1
applications, cone mirror images are used as is, and no
attempt was made to perform a perspectively correct
image unwarping. A cone mirror is also used to construct
mosaics [5], but that requires a special line scan camera,
time consuming scanning, and mechanical moving parts.
In real world applications we use finite aperture lens
systems to simulate the projective properties of a
theoretical pinhole camera. The real cameras have more
useful properties that a pure pinhole lacks. We show in
this work that under the geometric optics image
formation model, the single viewpoint of the cone mirror
is actually usable, i.e. we can physically place the
effective pinhole position at the single viewpoint and still
get the exact true single viewpoint image we need.
The geometric optics image formation model is more
powerful than pinhole camera model, i.e. it is a more
accurate description of what really happens in the
physical world. For example, when an object is placed
inside the front focal point of a real convex lens, it forms
an enlarged virtual image. This is completely
unexplainable if you treat the lens system as a perspective
pinhole. A perspective pinhole camera always gets real
images regardless of the object distance and image plane
distance, and it can never be out of focus. Baker and
Nayar [1] used a simple optical model to analyze some
omni-mirrors but they did not use any optical modeling
on cone mirrors. This turns out to be the key deficiency
in modeling the cone mirror/perspective camera
combination as we will elaborate shortly.
The cone system proposed here uses only off-the-shelf
ordinary lens and CCD camera. The cone is among the
simplest shape to produce, and it has much higher
resolution for scenes around the horizon [8]. It adds the
least optical distortion to the resulting image because it is
the only omni-view mirror with a longitudinally flat
surface. All other omni-mirror types are curved in both
longitudinal and lateral directions. The ability to pair
with readily available perspective camera instead of
expensive and complex orthographic cameras is also a
very important advantage over existing omni-view
This work has been supported in part by NSF-IIS-0083209, ARO/MURI-DAAH04-96-1-0007, NSF-CDS-97-03220, DARPA-ITO-DABT63-99-1-0017,
and Advanced Network and Services. Special thanks to members of MOOSE Project: V. J. Kumar, Kostas Daniilidis, Elli Angelopoulou, Oleg Naroditsky,
and Geoffrey Egnal for their invaluable comments and support. Also grateful thanks to members of GRASP Lab for their wonderful feedback and help.
systems. For example, when you want to build an omnisystem outside the visible range, it is often the case that
only perspective cameras are available. That is an added
reason why we go through all the trouble to prove and
build a cone mirror omni-directional camera system.
2. True single viewpoint cone mirror omniview system
As proved by Baker and Nayar [1], when the
viewpoint of a perspective camera coincides with the tip
of the cone, you have a single viewpoint omni-view
system. The only question then is whether or not you can
see an image in such configuration. The COPIS by Yagi
[10] did not put the lens camera viewpoint at the tip of
the cone because of this concern and thus did not form
single viewpoint omni-system. It turns out that we can
indeed see an image in this single viewpoint
configuration. We put the viewpoint of our lens camera
right at the tip of the cone and have a working true single
viewpoint cone mirror omni-view system. See Figure
1(geometry), Figure 2, Figure 3(full models) and Figure
8(real experimental setup).
image plane
effective viewpoint
of lens camera
p = (0,0)
effective viewpoint
v = (0,0)
A
a
B
M
mirror
N
b
Figure 1 True Single Viewpoint Cone Mirror
Geometry
In geometric optics image formation model [3], plane
mirror is an optical system by itself. The cross section of
a cone mirror through its axis of rotation consists of two
plane mirrors joined at the tip of the cone. The image
formation property of a plane mirror is that it always
forms virtual images behind its mirror surface. Notice
that the right side mirror does not actually extends to
point N, but that does not prevent the virtual image b
from forming. Within geometric optics, the position of
the virtual image point b does not change, either. As long
as there is some portion of the plane (ideally infinitely
large) is occupied with piece(s) of real mirror, the virtual
images corresponding to any world point on the front of
the mirror plane will be formed on the back of the mirror
plane. The positions of the virtual images are not effected
by the size of the actual mirror, but its “visibility” will
depend on the size and position of the actual mirror.
Since the virtual images are formed by the real plane
mirrors, the second converging optical system must ‘see’
the virtual image point through the real plane mirror.
One can regard real plane mirror as a “window” to look
at the “virtual world” on the other side of the mirror. The
surface of the cone mirror, after it accomplishes the task
of forming a virtual world behind it, acts as a window
opening for the 2nd converging optical system, i.e. the
perspective lens camera, to look into the virtual world. So
the cone is not blocking our view, it IS the view itself.
Whether the lens camera has its effective pinhole at the
cone tip or not, the lens camera “sees” the virtual image
points just like any other real world point. These virtual
points get imaged perspectively the same way as if there
is a real point there. In Figure 1, the virtual images a and
b formed by the catoptric system are imaged by the
dioptric system to form real images α and β respectively.
As proved by Baker and Nayar [1], if we can put the
effective pinhole of the lens camera at the tip of the cone
and still see image perspectively we have a single
viewpoint omni-view catadioptric system. We have just
shown that we can do this. We put the effective pinhole
of the lens camera right at tip of the cone and construct a
true single viewpoint cone mirror omni-directional view
catadioptric system. See Figure 8.
Full geometric optic ray tracing for the cone
combined with Gaussian optics ray tracing for the lens
camera will yield exactly the same results we just
showed. Notice here in geometric optics the term “ray
tracing” has very different meaning than the same term
used in computer graphics literatures. Computer graphics
“ray tracing” traces only one ray for one world point
while geometric optics “ray tracing” traces no less than 2
rays (ideally infinitely many) for one world point. The
computer graphic “ray tracing” is a simplified way of
finding image position valid only for focused real images.
Analyses based on tracing single rays alone can not
model how the single viewpoint of cone mirror omniview system works.
Since we need to trace many rays for each world
point, we trace only point B in Figure 2 and Figure 3 to
avoid confusion (too many rays in one small graph if we
trace both points). Interested reader can trace point A and
find it imaged exactly at the image position α.
In Figure 2 the tip of the cone is cut out a little so the
cone tip is not physically present. This does not alter the
single viewpoint geometry. Only the brightness of the
single viewpoint image is reduced a little. The two
configurations (Figure 2 and Figure 3) yield exactly the
same image for scenes points that are in focus vertically
(geometry-wise, the physical focus settings are different).
We shall explain details in the following section.
3. System Characteristics
3.1 Perfect vertical imaging
Curved surfaces, do not form exact image points for
all scene points. For example, parabolic mirror forms
perfect image only for points at infinity, while hyperbolic
mirror forms perfect virtual image only for one point, its
outside focal point. Baker and Nayar [1] has good
analysis of vertical defocus blur caused by vertically
curved surfaces. Because cone is the only vertically flat
omni-view mirror, the cone mirror based omni-view
device is the only device free of vertical defocus blur.
back focal point
O is projected to I via ray 2. In image space the two
image positions are always the same because the lengths
CH and VI are always the same. The lengths are the
same because under Gaussian optics conditions ray 1
goes parallel to VW after refracted by the lens and will
hit point I when the object is in-focus. Here the
conceptual camera at F does not produce any physical
image but the real image seen by camera at C will be
exactly the same since they are 1:1 orthographic
projection of each other.
Gauss developed the theory for paraxial rays only and
founded Gaussian optics [3]. Within Gaussian optics all
the rays that come from the same object point in the
world and enter the focused optical system converge at
the same point. This is an approximation but in practice
is very close to perfect in many situations. Given that,
trace any two such rays from a world point and we can
find its image point.
effective viewpoint
of lens camera #2
front focal point
p = (0,0)
real image plane
(CCD position)
lens
Virtual image plane
effective viewpoint
v = (0,0)
1.000
f
image plane
effective viewpoint
of lens camera #1
p = (0,0 )
effective viewpoint
v = (0,0 )
B
lens iris
B
4.000
So
mirror
N
mirror
b
N
b
Figure 3 Optical ray tracing using front focal point
of the lens as its virtual pin-hole
O
Object
Figure 2 Optical ray tracing using lens center as
virtual pin-hole
3.2 Dual effective pinholes for lens camera
This applies to most ordinary perspective lens
cameras that are in focus. In Figure 4, with a conceptual
perspective camera with viewpoint at F and image plane
at C(perpendicular to the axis WV), an arbitrary object
point O is projected to H via ray 1. While with a second
conceptual perspective camera with viewpoint at C and
image plane at V(perpendicular to WV), the same object
Lens(Aperture)
Back Focus Screen
(CCD Chip)
W
V
1
2
3
H
F
Front Focus C
(Pin Hole#2) Lens Center
(Pin Hole #1
I
Image of Object
Figure 4 Thin lens geometry implies dual effective
viewpoints
For convenience, we usually pick the two most easily
traced rays.
1. The ray that passes through the front focal point
will be refracted by the lens and then advances parallel to
the optical axis until it hits the image plane.
2. The ray that passes through the center of the lens
will pass straight through unaltered.
The Gaussian Thin Lens formula:[3] can be derived
directly from the same similar triangles we just used to
derive the dual effective pinholes.
BB
effective viewpoint
of lens camera #1
p = (0,0)
back focal point
image plane
effective viewpoint
v = (0,0)
lens iris
L
B
R
mirror
Equation 1 Gaussian Thin Lens Formula
1
so
+
1
si
=
N
1
f
where so object distance, si is image distance, and f is
focal length. The term “focal length” may cause some
confusion as many computer vision literature uses “focal
length” to mean the distance from the viewpoint to the
image plane. The term “focal length” in Gaussian optics
is a lens parameter that remains fixed while we change
the position of image plane. The two “conceptual
cameras” has different “object distance”/”image plane
distance” parameters but their resulting images are the
same.
This is why we can use the setup of either Figure 2 or
Figure 3 and get the same results. The choice of effective
pinhole effects only the value of so, si , and f, but not the
pixel position of any image point (compare Figure 2,
Figure 3, and Figure 4). Krishna and Ahuja [4] also use
the front focal point as the viewpoint for their panoramic
imaging device.
b
Figure 5 World points on the opposite side form
extremely weak images which are not visible in most
practical situations
Cone Omnisystem Vertical FOV
Real Perspective Camera
33.4°
FOV with cone mirror
cone tip
16./2
7°
Mirror Image of Half of the Camera
cone edge
53.3°
/2
106.6°
Cone Mirror
/2
53.3°
3.3 Image brightness and visibility
Since the profile of a cone consists of two plane
mirrors, each point inside the cone should be the virtual
image point of two world points simultaneously, one on
each side, see Figure 5.
In practice, we only “see” images on the same side of
the mirror. Figure 5 shows clearly that the point b is also
the position of the virtual image of point BB on the other
side. But in the same figure we can also see that very few
rays from point BB can actually reach the image plane
because of occlusion. Also, due to the symmetry of the
geometry, for a point b that is to the right of the central
axis, there is always more area of the mirror on the right
side (from tip of the cone p to point R) than the area of
mirror on the opposite side (from p to L) that is useful in
terms of imaging. So we always see B instead of BB in
practice because the image intensity of BB is far weaker
than that of B and thus the image of B always dominates
in practice.
/2
16.7°
FOV without cone mirror
Figure 6 Cone Shape and Vertical FOV
4. Unwarping algorithm
Unlike most existing omni-view system, the vertical
field of view for cone omni-system is not continuous
across the zenith. As shown in Figure 6, if the tip of the
cone subtends an angle α, and the normal angular
FOV(Field Of View) of the perspective camera is φ, then
the upper limit of viewable elevation is α-π/2. The
elevation here is defined to be 0 at level, 90 degree
straight up and -90 degree straight down. The same as
the system used for a gun turret. The lower bound of
viewable elevation is α-(π/2)-(φ/2).
When “unwarping”, assume the camera is perfectly
positioned and lined up, we establish a 2D image polar
coordinate system. For a given azimuth and elevation
angle in the “unwarped view”, the azimuth match the
polar angle directly. The polar radius variable r is related
to the elevation as
Equation 2
1.0044 pix^2 in the horizontal direction and 0.4425
pix^2 in the vertical direction. The square root of mean
squared distance differences is 1.2029 pix. This is
equivalent to 0.5 degree in view angle.
r = f p × tan(α − π2 − θ )
where fp is the effective focal length(image plane to
viewpoint distance) of real camera in real camera
pixels(i.e. the pixel unit in the original omni-view image,
not the pixels in the unwarped image) and θ is the
elevation angle of a point in the ‘unwarped’ view we
want to create.
Figure 10 Cone and Parabolic mirror placed side by
side; Right: Test pattern
5. Experiments
Figure 11 Omni-views, Left: Cone/perspective, Right:
parabolic/orthographic
Figure 7 Cone mirror prototype. Right: Overview of the
prototype setup.
Figure 12 Unwarped images. Left: Cone, more stripes
Figure 8 The tip of the cone actually coincides with the
effective pinhole. Right: cone omni-view seen by the
camera. The left side is a test pattern with dots 4” apart.
Figure 9 Actual perspective image taken by real
perspective camera (left) The unwarped perspective
image from the above looking at the same test pattern is
shown at the right.
To demonstrate the ability to correctly unwarp image
into perspective view, a test pattern of rectangular grids
is imaged in a omni-view picture. The right picture in
Figure 8 shows the omni-view containing the grids.
Figure 9 shows the unwarped result compared to an
actual picture taken by a normal perspective camera.
There is no visible radial distortion in Figure 9, which
means the radial distortion within our FOV is very small.
Tsai’s algorithm is suitable for compensating small
radial lens distortions. [7]
For the pattern of 45 points grids, the best match
between the two pictures has mean squared error of
preserved. Right: Parabolic, stripes tend to merge and not
separable due to lower vertical resolution.
To demonstrate the high vertical angular resolution
we put a custom made integrated parabolic/orthographic
system along side the cone prototype and let them look at
the same color foam pattern placed about 1m away.
In traditional non-omni cameras, there is a preferred
direction one can move the camera closer to the scene to
improve resolution, but in our omni camera case there is
no special direction to “move closer”. If you move the
camera “closer” toward one direction, the scene in the
opposite direction becomes “farther” at the same time. In
other words, cone mirror gets high resolution in all
direction easily in normal environment while other omni
cam system gets high resolution only in very confined
space where everything is close by.
The lens used is COSMICAR 6 mm 1:1.2 TV lens
stopped down to f/11 or f/16. A SONY XC-77 camera is
connected through a video box model PS-12SU made by
CHORI AMERICA. If your lens cannot get proper focus
within normal adjustment range, using extender rings
and/or use smaller aperture settings. Small aperture
setting in a dark room may need to digitally enhance the
brightness of the image. Some lens set has effective
viewpoints surrounded by light shield which blocks most
FOV. Avoid using such lens or remove the light shield
before use.
6. Discussions
The cone mirror is the only system that can get
perfect image vertically when you use a perfect lens
system. All other longitudinally curved mirrors introduce
defocus blurs before the light enter any lens system [1].
So even with a perfect lens system you can not get a
perfect image in those mirrors. In the rotational
symmetric direction, all omni-mirrors are curved with
the same horizontal cross-section shape: circles.
As a degenerate case of conic section, the cone mirror
does not have a full hemispherical view. It sees all 360
degrees in the horizontal direction, but the vertical field
of view (FOV) is not continuous over the zenith, see
Figure 6. Other omni-mirror system may in theory have
complete hemispherical view but in practice the lens
camera or secondary mirrors are always blocking the
view around the zenith, so in practice no similar system
can see the whole hemispherical view anyway. Smaller
vertical angular FOV is actually good for many
applications because we get higher angular resolution
with the same number of pixels concentrating on the
most interesting scene. The region not visible is occupied
by the lens camera, the sky or the floor, all of which are
of secondary interests in many applications like driving a
ground vehicle.
7. Conclusion
The Cone mirror has been identified as having a
single viewpoint before but that single viewpoint has
been regarded as useless because previous analysis was
done with the idealized pinhole camera geometric model.
This work proves that with more accurate modeling of
the optical properties of real cameras used in the real
world, the single viewpoint inherent in the cone mirror
geometry is usable. Prototype system based on our theory
is constructed and experiments confirm our theoretical
predictions. The total field of view is smaller but more
flexible. The new system is simple to build and maintain,
easier to analyze, costs a lot less, and yet yields much
higher performance in the peripheral region where most
interesting scenes reside.
References
[1] Baker, S. and Nayar, S. K., "A Theory of Single-Viewpoint
Catadioptric Image Formation," International Journal of
Computer Vision, vol. 35, no. 2, pp. 175-196, Nov.1999.
[2] Bogner, S. Introduction to panoramic imaging. 3100-3106.
1995. Proceedings of the IEEE SMC Conference.
[3] Hecht, E., Optics, 3 ed. Reading, MA, USA: Addison Wesley
Longman, Inc., 1998, pp. 1-694.
[4]
Krishna, A. and Ahuja, N. Panoramic image acquisition. 379384. 6-18-1996. San Francisco, CA. International Conference
on Computer Vision and Pattern Recognition.
[5] Nayar, S. K. and Karmarkar, A. 360 x 360 Mosaics. 2000.
Hilton Head Island, South Carolina. Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition.
[6] Southwell, D., Vandegriend, B., and Basu, A. A Conical Mirror
Pipeline Inspection System. 4, 3253-3258. 4-22-1996.
Minneapolis, Minnesota, USA. Proceedings of the 1996 IEEE
International Conference on Robotics and Automation.
[7] Tsai, R. Y., "A Versatile Camera Calibration Technique for
High-Accuracy 3D Machine Vision Metrology Using Off-theShelf TV Cameras and Lenses," IEEE Journal of Robotics and
Automation, vol. RA-3, no. 4, pp. 323-344, Aug.1987.
[8] Yagi, Y., "Omnidirectional sensing and its applications," IEICE
TRANSACTIONS ON INFORMATION AND SYSTEMS, vol.
E82D, no. 3, pp. 568-579, Mar.1999.
[9] Yagi, Y. and Kawato, S. Panoramic scene analysis with conic
projection. 1990. Proceedings of the International Conference on
Robots and Systems.
[10] Yagi, Y., Kawato, S., and Tsuji, S., "Real-Time
Omnidirectional Image Sensor (COPIS) for Vision-Guided
Navigation," IEEE TRANSACTIONS ON ROBOTICS AND
AUTOMATION, vol. 10, no. 1, pp. 11-22, Feb.1994.