Stereovision
Four weeks project in
Introduction to multimedia technologies
Abstract:
This project contains theory about finding the corresponding points by the area -based search
method for stereo images. There are implementation of a single and
multiple windows
methods for finding corresponding points for a pair of parallel stereo images.
It contains also the theory and an implementation of finding the epipolar line by the 8 -points
algorithm for a pair of convergent stereo images.
Saif Alghanemi:
Alaa ElHussuna:
Aws Abdul-Muhsin:
Nicolai Johannesen :
Stereovision
Table of contents
INTRODUCTION ................................ ................................ ................................ ............... 4
GENERAL STEREO VISION................................ ................................ ............................ 6
CORRESPONDENCE ................................ ................................ ................................ .............. 6
AREA-BASED METHOD ................................ ................................ ................................ ........ 8
Disparity ................................ ................................ ................................ ......................... 8
Depth Map ................................ ................................ ................................ ...................... 9
The Concept of Search Window ................................ ................................ ....................... 9
Single Window Method................................ ................................ ................................ .. 10
Multiple Window Method ................................ ................................ .............................. 11
Adaptive search window (ASW)................................ ................................ ..................... 11
PARALLEL STEREO IMAGES ................................ ................................ ...................... 14
IMPLEMENTATION OF SINGLE WINDOW ................................ ................................ ............... 14
IMPLEMENTATION OF MULTIPLE WINDOWS ................................ ................................ ........ 15
RESULTS ................................ ................................ ................................ ........................... 16
Disparity/Depth Map................................ ................................ ................................ ..... 17
Single Window method ................................ ................................ ................................ .. 17
Multiple Window method................................ ................................ ............................... 18
NON-PARALLEL STEREO IMAGES ................................ ................................ ............ 19
EPIPOLAR GEOMETRY ................................ ................................ ................................ ........ 19
THE ESSENTIAL MATRIX................................ ................................ ................................ .... 21
THE FUNDAMENTAL MATRIX ................................ ................................ ............................. 22
RECOVERING EPIPOLAR GEOMETRY AND EIGHT POINT ALGORITHM ................................ ... 23
THE IMPLEMENTATION OF THE EPIPOLAR LINE ................................ ................................ ... 26
RESULTS ................................ ................................ ................................ ........................... 27
CONCLUSION ................................ ................................ ................................ .................. 29
REFERENCES ................................ ................................ ................................ .................. 30
APPENDIX FOR PSEUDO CODE................................ ................................ ................... 31
Pseudo code for the main function SingleWindow................................ ..........................
Pseudo code for the function SWCorrespondence ................................ ..........................
Pseudo code for the function SMW ................................ ................................ ................
Pseudo code for the function SMWCorrespondence................................ .......................
31
31
32
32
APPENDIX FOR CODE ................................ ................................ ................................ ... 33
Code for the function SingleWindow................................ ................................ .............. 33
Code for the function SWCorrespondence ................................ ................................ ..... 34
Code for the function CorrespondenceTest ................................ ................................ .... 35
Code for the function SMW................................ ................................ ............................ 36
Code for the function SMWCorrespondence ................................ ................................ .. 37
Code for the function SMWCorrespondenceTest................................ ............................ 38
Code for the function scale2uint8 ................................ ................................ .................. 39
Date 11-05-2002
Introduction to multimedia technologies
Page 2 of 2
Stereovision
Code For The Main Function DrawEpipolarline ................................ ...........................
Code For The Function eight................................ ................................ .........................
Code For The Function findA ................................ ................................ ........................
Code For The Function findTheF ................................ ................................ ..................
40
41
42
42
APPENDIX: THE PAPER “EFFICIENT STEREO WI TH MULTIPLE INDOWING ”
................................ ................................ ................................ ................................ ............ 43
APPENDIX: THE PAPER “A STEREO MATCHING ALG ORITHM WITH AN
ADAPTIVE WINDOW: THEORY AND EXPERIMENT” ................................ ............ 44
APPENDIX: THE PAPER “IN DEFENCE OF THE 8-POINT ALGORITHM” .......... 45
Date 11-05-2002
Introduction to multimedia technologies
Page 3 of 3
Stereovision
Introduction
Stereo vision is the process of constructi ng a 3D model of a scene through processing of two
2-D images of this scene.
Viewing a scene from two (or more) different positions simultaneously allows us to make
inferences about 3 -D structure, provided that we can match up corresponding points in the
images. The visual systems of humans and some other animals make use of this, and it is very
important in attempts to develop practical computer vision systems.
Stereo vision uses two cameras to observe the environment, finds the same point in each
image, and measures depth to that point by triangulation; that is, by intersecting the lines of
sight from each camera to the object. Finding the same object in each image is called
correspondence (or matching) and is the fundamental computational task underlyin g stereo
vision.
One of the most useful aspects of vision is that of depth perception. Depth perception is the
process of extracting the depth of each pixel in the image, i.e., finding the third dimension in
the image, denoted by (z). A 3-D image has many advantages over its 2D counterpart. The 2D
images give only limited information about the physical shape and size of an object in a
scene.
The 3D images express the geometry in terms of three -dimensional coordinates, hence the
size and shape of an obje ct can be computed from its three -dimensional coordinates.
In order for machines to guide themselves correctly, they need to know how far away things
are.
Some examples and applications that use the 3D -depth information are: bin-picking, in which
a robot is required to identify the position and orientation of a 3D object and then pick it up.
Another application is vehicle guidance of automobile. One more application is
industrial
inspection for 3 -D objects. Plenty of applications are coming to the field o f computer vision
and 3 -D reconstruction. This reflects the important demand for finding a good and accurate
computer vision systems.
This project is about stereo vision and the depth of the image. The depth of the image is
simply found by estimating corr
espondence of each pixel in the image. From the
correspondence one can calculate the disparity of each pixel. This will form the disparity map.
For the care of calibrated camera, when we know the focal length, and the distance between
the two centers of the cameras, called the base line, we can calculate the depth map.
The depth map includes the depth of each pixel in the image.
Cameras can be oriented in different way, the parallel and nonparallel orientation.
The parallel cameras are situated in away th at the two image plans included within the same
view plan.
When the cameras are not situated in parallel, then they are in the nonparallel situation. The
nonparallel camera, are more difficult to analyze, but still the general case for most practical
cases.
Date 11-05-2002
Introduction to multimedia technologies
Page 4 of 4
Stereovision
To solve the problem of finding the correspondence of the world object, we have to, some
extent, to calibrate the two cameras. There are two types of parameter relate the camera with
the geometry of the vision system. These parameters are known as the e xternal (extrinsic) and
the internal (extrinsic) parameters. These are defined as follow:
•
•
The external parameters: are the parameters that define the location an orientation of the
camera reference frame with respect to a known world reference frame.
The internal parameters: are the parameters necessary to link the pixel co -ordinates of an
image point with the corresponding co-ordinates in the camera reference frame [2]
The parallel orientation will be studied in this project, and solutions for finding th
correspondence points in the two images will be implemented.
For the nonparallel orientation of we will study a way of minimizing the search of the
correspondence point from the entire image to just a single line called the epipolar line.
e
The 3 -D recon struction of the image is beyond the scope of the project, and will not be
covered here.
Date 11-05-2002
Introduction to multimedia technologies
Page 5 of 5
Stereovision
General Stereo Vision
The problems of stereo vision systems can be divided into two parts.
•
•
The correspondence: which is the process of finding the corresponding poi
nts of each
pixel in two images.
The reconstruction: which is the process of building the 3 -D model of the image.
Finding the correspondence of a point in two images, is not an easy task. The different
methods and problems of finding the correspondence ar e explained in the following sections.
Once a correspondence has been established, we can calculate the disparity, which is a
measure of the depth of the point. If we know the internal parameters of the two cameras, the
focal length and the distance betwe en the cameras, we can form the depth map, which shows
the depth of each pixel in the image. From the depth map, the 3 -D reconstruction of the model
can be calculated.
The work in this project will covers the study and implementation of finding the
correspondence and constructing the depth map, while the 3 -D reconstruction is beyond the
scope of this project.
Correspondence
The process of establishing the matches between points in a pair of images is called
correspondence.
Thus assuming that we have tw o images taken form two different view points, the process of
finding a correspondence is just to determine which point in the left image corresponds to the
point in the right image.
However, one of the difficulties finding the correspondence is that not a ll the points in the left
seen are visible by the right one, because of the different orientation of the cameras. So a
stereo system must be able to determine the image parts that should not be matched.
Through out this project, we will assume that most o f the scene points are visible from both
viewpoints, and that corresponding image regions are similar.
This assumption will reduce the problem of finding the correspondence to just a search
problem, i.e. given an element in the left image we search for the corresponding element in
the right image. This is what is called a conjugated pair.
The classical way of finding the conjugate pair of a point in two images, is by choosing a
point in one of the images and determining a search line on the other image, ca
lled the
Epipolar line. So the search region can always be reduced to a 1 -D problem, independent of
the relative positions of the cameras. If the two images are taken from two parallel cameras,
the search line is the same row in the two images. If the imag
es are taken from two
nonparallel cameras, then the search line should be the line that passes through the
correspondence point and another point called the epipole.
Date 11-05-2002
Introduction to multimedia technologies
Page 6 of 6
Stereovision
When the search line has been found, we can compute similarity scores for each point on t he
line by comparing an area around the point (called the window) on the first image and an area
around each point on the search line. This method of finding the correspondence is called the
Area-Based Method.
But there are several factors making the cor respondence problem difficult:
1. Occlusions: this is where points on one image don’t correspond to any points in the
other image i.e. an object close to the cameras will obscure part of the first image you
can see in the second image and vice versa or an ob ject is obstructing a area for one
camera and not the other (see figure 1)
2. Noise: when we have a noise in one image, which is not correspondent in the other
image
3. Photometric distortions: make the projected intensity of a world point different in the
two images i.e. windows will reflect different intensity for different angles.
4. Projective distortion : make the projected shapes different in the two images. This
effect can be seen in two ways: "(1) as a surface recedes from the viewer its markings
become s maller. (2) As a surface is inclined, its markings appear compressed in the
direction of inclination" (see figure 2 ) [3]. Projective distortion gives the human brain
a general idea of the shapes of the objects in the visual field. It can use this
information to make educated guesses about the shape of the objects that it sees. So
that it can fill in the blanks left by binocular disparity.
5. Non-Texture Areas : In regions where there is no intensity variations, then we will
have the same amount of similarity
scores for the whole region.
Multiple
corresponding points can accrue when there is no texture in a region larger than the
window size. This makes it impossible for the similarity to distinguish between the
windows before and after.
Object
Object
Left image
(ImL)
Left image (ImL)
Right image
(ImR)
z=0
Focal length
(f)
Focal length
(f)
Date 11-05-2002
dr
Right image
(ImR)
z=0
Baseline (T)
Baseline (T)
Figure 1. This is an example of Occl
usion.
Where an object is obstructing the line of one
camera and not the other.
dl
Figure 2. This is an example of
projective distortion.
Introduction to multimedia technologies
Page 7 of 7
Stereovision
Area-Based Method
The area-based method is used to find the correspondence between two stereo images. In the
Area Based method the element to match are windows of the images taken around the point. It
is the amount of the similarity between the windows in the two i mages, which will decide the
correspondence of a point.
Finding a successful correspondence point will enable us to measure the disparity between the
two images, and hence provide us will what we need to construct the disparity map. Finding
the disparity of two corresponding matches is explained in the next section.
There are two ways to find the similarity scores of the windows in the two images. The
comparison of the areas is either done by cross
-correlation or SSD (sum of squared
difference).
Cross-correlation:
c ( x, y , d ) =
Im L ( x + ξ , y + η ) ⋅ Im R ( x + ξ − d , y + η )
(1)
(ξ ,η )
SSD (sum of squared difference)
c ( x, y , d ) =
− [Im L ( x + ξ , y + η ) − Im R ( x + ξ − d , y + η )]
2
(2)
(ξ ,η )
Where ξ = [− n, n] is the height of the windows, η = [− m, m] is the length of the window, x
and y is the co-ordinate for the point in the left image (ImL) and, d is the displacement.
The area-based method will be used through out the implementation of finding the disparity
map of a parallel stereo image.
Disparity
The disparity is the difference in position (the shift) between t wo corresponding points. The
disparity is found by finding the index of the maximum value of the correlation (SSD) of all
the displacement points:
disparity(x, y) = arg max (c(x, y, d))
d
(3)
SSD is usually preferable over cross -correlation, because it is not biased by t he presence of
regions with very small or very large intensity value [2 p. 147].
The disparity of all the points in the image forms the so -called the disparity map , which can
be displaced as image. If the geometry of the stereo system is known the depth map can be
calculated. The process of calculating the depth map is explained below.
Date 11-05-2002
Introduction to multimedia technologies
Page 8 of 8
Stereovision
Depth Map
When the disparity has been found for all the points in the left image we can calculate the
depth map . The depth formula is derived from to similar triangles (se e figure 3). Take the
base of triangle 1, and divide it by the height of the same of triangle. This is equal to the base
triangle 2, divided by the height triangle 2. The depth of the point P is the same as the height
of triangle 1.
P
z
xl
Left image (ImL)
Focal length (f)
cl
xr
pr
pl
Right image (ImR)
cr
Triangel 1
Triangel 2
z=0
Baseline (T)
Figure 3. Finding the depth from the disparity.
Let xl and xr are the co -ordinates of pl and pr with respect to the principal points cl and cr .
So in the example of figure 3 xr is negative and xl is positive.
T T +d
where d = x R − xL
=
z z− f
zT − fT = (T + d ) z
z (T − T + d ) = fT
T
z= f
d
(4)
So the depth of each point in the seen, is the focal length f times the baseline T divided by the
disparity of that point d.
The Concept of Search Window
The idea behind using a window at all is to use the neighboring pixels to help the match. This
assumes that the disparity function varies little on the scale of the window. By and large this
is true, disparity is often piecewise smooth, which large discontinuities present between
Date 11-05-2002
Introduction to multimedia technologies
Page 9 of 9
Stereovision
objects and a smooth change of disparity across the surfaces of objects, called surface
discontinuities.
For this purpose a search window is constructed, the selection of an appropriate window
shape and size is crucial to achieve a smooth and detailed disp arity map.
Several methods have been suggested to make the process of finding the correspondence
points and constructing the depth map more accurate like:
1- Single window methods.
2- Multiple window method.
3- Adaptive search window method.
These different met hods were suggested to solve some of the problems like occlusion,
projective distortion and multiple corresponding points associated with finding the
correspondence between two images.
Single Window Method
This method is the simplest one. It’s assumes that the match window to be of fixed size and
shape during the correspondence search process.
The choice of the size and shape of the window is decided by the user from the beginning.
Once fixed, the size of the window will not be changed.
To optimize the speed of the method we choose to determine the length on the search line in
between maximum and minimum values. The chose of these maximum and minimum values
will depend on the nature of the image itself (see figure 4).
The Region to search
in on the line
Left window
The chosen point
Left image ImL
The window is
moving along
the line.
Right image ImR
Figure 4. Finding the search line for a parallel stereo image.
The advantage of the Single Wind ow method, is it very simple and faster than the other two
methods.
The disadvantages of the Single Window method is it’s does not solve the problem of the
multiple corresponding points, intensity variations, projective distortion and occlusions. So it
is not accurate in estimating the depth map.
Date 11-05-2002
Introduction to multimedia technologies
Page 10 of 10
Stereovision
A solution to the problem of multiple corresponding points using the Single Window method
could chose to add one to the point where the test of finding a single correspondence fails.
This will improve the result but will not, but will not solve the overall problem.
More trivial solution could use the Multiple Window method, or the Adaptive Search
Window presented bellow.
This method was implemented and tested using MATLAB. (see the implementation & result
of the Single Window method)
Multiple Window Method
In the Multiple Window Method, nine different windows, shown in the figure 5 bellow, are
taken to find the correlation for each pixel and compute the disparity with the smallest SSD
error so that the disparity will lead to the choice of the appropriate window [1].
Figure 5 . The nine asymmetric correlation windows. The pixel
for which disparity is computed is highlighted.
The Single Window search algorithms have the drawback of inefficiency in poorly textured
areas.
In Multiple -Windows method the advantage of having multiple windows to estimate the
disparity will h elp to compute the uncertainty as “the estimated variance of the disparity
measures obtained with the various windows, the occluded points are assigned infinite
variance” [1], consequently, this will improve the results in poorly textured areas.
The adva ntages of the Multiple -Windows method that it’s improves the disparity estimate,
and gives more accurate depth map. It’s still faster than the Adaptive Search Window method.
The Multiple-Windows method also solves the problem of occlusion [1].
The disadvantages are that it’s still fails to handle big non -textured areas, and still incapable
of completely solving the projective distortion.
This method was implemented and tested using MATLAB. (see the implementation & result
of the Multiple Window method)
Adaptive search window (ASW)
To solve the problem of finding the corresponding points, the Adaptive Search Window was
suggested [4].
The algorithm assumes the use of different shapes and size for the search window, and hence
called the adaptive search window.
Date 11-05-2002
Introduction to multimedia technologies
Page 11 of 11
Stereovision
The shape of the window is of great importance because the image properties like intensity,
disparity and texture are piecewise smooth; they vary smoothly at the most points, but change
dramatically at the object boundaries. Furthermore, because it acts as a filter on the image,
introduces systemic bias into the matching process.
In this method the window size and shape are iteratively changed based on the local variation
of the intensity and current depth estimate. Adaptive Search Window gives far better results
but still computationally expensive.
The idea is that the size of the window must be large enough to contain adequate intensity
variation for accurate and acceptable matching however, it should be small enough to
minimize the effects of projective distortion.
Very small window may not include enough intensity variation so it gives a poor disparity
estimates because the signal (the intensity variation in this case) to the noise ratio would be
low.
On the other hand large window may also giv e poor results this time because the position of
maximum correlation or minimum SSD will not match due to the effect of projective
distortions in the left and right images.
For all the above reasons the size of the window must be selected adaptively acco rding to the
intensity variation (see figure 6-8).
y-plus
x-plus
x-minus
y-minus
Figure 6. Window expansion.
The method suggested aims to “select an appropriate window by evaluating the local variation
of the intensity and the disparity”.
To implement this a statistical model for the disparity distribution with in the window should
be built. This model will help to construct a window, which searches for the disparity
estimates with the minimum uncertainty for each pixel in the image.
The size as well as the shape (rectangular) of the window is adaptively changeab le as stated
above.
One of the difficulties associated with the Adaptive Search Window is the estimation and then
use of the disparity, because disparity variation is the end result we are looking for in stereo vision at the same time it is one of the cr iteria we intend to use to estimate disparity.
Date 11-05-2002
Introduction to multimedia technologies
Page 12 of 12
Stereovision
P6
P5
P0
P2
P0
P1 P4
Figure 7. Positions for which size and
shapes of the selected windows are
examined.
P6
P3
P0
P5
P2
P1
P4
Figure 8. Selected windows for each position are shown (P refers to the pixel).
Date 11-05-2002
Introduction to multimedia technologies
Page 13 of 13
Stereovision
Parallel Stereo Images
There are two ways by which the two cameras can be oriented. They can be oriented either in
parallel, or nonparallel.
In the parallel camera , we assume that their optical axe s are lined up parallel to one another
and that they are side by side - or more exactly, that the line joining their optical centers is
parallel to their x-axes. This means that the image of a point will have the same y coordinate
for the two cameras. The line connecting the camera lens centers is called the baseline.
In the case of two parallel cameras, finding the depth map is much simpler than when they are
non-parallel. This is because any conjugate pairs must lie on the same row in both images, i.e.
their y- value will be the same. Finding the correspondence will not need to rectify the image,
neither changing its orientation. The process will just need to chose for each point in the left
image, search for the corresponding point on the same row. The c
riteria for finding the
correspondence could be measured either using cross-correlation or sum of squared difference
(SSD). Either one of the methods proposed to find the correspondence, i.e. Single Window,
Multiple Windows or Adaptive Search Window could be used during the process. Surely each
one will give different results.
In the section bellow we will show the implementation of the Single Window and Multiple
Windows to calculate the depth map for a pair of test images, taken from parallel cameras.
Implementation of Single window
The correspondence problem for parallel cameras was implemented using single window and
multiple window method. All the depth map calculations are implemented in MATLAB
functions. The first is single window implementation, wh ere the window is centered on point
we are finding the disparity for.
The method for finding the correspondence using the Single Window method could be
implemented to the whole image, or it could be implemented to single pixel. We have
implemented a tool to visualize the output of a single pixel of this method (see the code
appendix for SWCorrespondenceTest).
The Single Window implementation is made by two functions, the main SingleWindow and
SWCorrespondence (see the Pseudo code and the code in the appen dix).
SingleWindow is keeping track of which point we are calculating the corresponding point for,
deciding which point to take if there is multiple corresponding points, calculating the depth
(z) and to re -scale the depth map to give a graphical represe ntation of it. Sometimes the
disparity is 0 this makes a divide by zero error when the depth is calculated, because of this
these points is set to –50 this makes it appear white in the visual representations.
SWCorrespondence is called by SingleWindow. S WCorrespondence is calculating all the
similarity scores between a point in the left window and a region of the search line in the right
Date 11-05-2002
Introduction to multimedia technologies
Page 14 of 14
Stereovision
image. The region is chosen so that we know that the correct corresponding point is within
this region. By making the s earch region large enough the maximum correlation will be
within this region. The similarity scores can either calculated by SSD, normalized SSD or
cross-correlation
Because of the way the point is centered in the window it is not
possible to calculate t he correspondence in an outer rime of the image
see figure 9.
We also expect some problems finding corresponding point in a rim
furthest left of the left image. This is because this part is not
represented in the right image at all.
Figure 9. Boundarys
og the calculations
To visualize the resu lts a MATLAB function is used called imshow.
This function only accepts values in uint8 so the images of results are scaled to by a function
called scale2uint8 (see the code in the appendix).
Implementation of Multiple Windows
The second is a multiple w indow implementation, where the algorithm chooses between nine
different windows, which give better disparity estimations.
We also implemented a tool to visualize the output of a single pixel of this method (see the
code appendix for SMWCorrespondenceTest).
The implementation of multiple windows is like the single window implementation. Now it is
finding the most likely corresponding points from a matrix instead of a vector. There by
making it more likely too find better -correlated windows. The size of th e matrix is the region
times the nine windows.
The multiple window implementation is made of two functions, the main function SMW and
SMWCorrespondence (see the Pseudo code and the code in the appendix).
Date 11-05-2002
Introduction to multimedia technologies
Page 15 of 15
Stereovision
Results
To test the implementations there wa s found a pair of parallel stereo images on the Internet.
The images are found on http://www-dbv.cs.uni-bonn.de/stereo_data/ and they are artifice
images of a corridor (se figure 10).
Left image
Right image
Figure 10. Shows a pair of parallel stereo images.
When we use t he little graphical tool we can see that SSD is a lot better to find the
Corresponding points than the Cross-correlation’s
Figure 11. The figure shows the correspondence of a single point between the left the right image.
We can also se that maximal legal disparity is around -10 with is found in the right of the
images and the disparity in the end of the corridor is –1 (see figure 11).
Date 11-05-2002
Introduction to multimedia technologies
Page 16 of 16
Stereovision
Disparity/Depth Map
The disparity map and depth map has the same characters, as the depth is only a scaling of the
disparity. The dark area represents the areas away from the cameras the light represents the
areas close to the cameras for all the disparity maps and depth maps.
Single Window method
We can see on calculation of the disparity map that the corridor is sections in layers as
expected because the disparity is in integer values form –1 to –10 (see figure 12). The method
is not fin grain enough to properly represent the cone. The ball better represent probably
because the ball is a larger object. The two strait lines in the left of the windows are errors
with come from the way the Multiple Corresponding Points is handled.
Figure 12. Left the disparity map and right the disparity m ap calculated with SSD by the single
widow implementation
If we do not take the disparity from the point before when we have multiple corresponding
points but take the first of the points in correspondence vector the results is as in figure 13.
Figure 13. Left the disparity map and right the disparity map calculated with SSD by the single
widow implementation, but if there are multiple corresponding points the first point is taken.
Date 11-05-2002
Introduction to multimedia technologies
Page 17 of 17
Stereovision
Now it is possible to see where there are multiple corresponding points in the image this is the
area where the image scale from gray to white or the disparity is getting larger and larger.
The single window method in this implementation it takes approximately 5 minutes for the
test image (that is 256 by 256).
Multiple Window method
The result is a lot better for Multiple Window method the cone and ball have shaper edges
(see figure 14 and 15). But there is also some new errors introduce the disparity is shifting on
the top of the corridor end. This properly because it shifting between the different windows.
It is a wary slow method in this implementation it takes approximately 30 minutes for the test
image (256 by 256 pixels).
Figure 14. Left the disparity map and right the disparity map calculated with SSD by the Multiple
widow implementation, but if there are multiple corresponding points the first point is taken.
Figure 15. Left the disparity map and right the disparity map calculated with SSD by the single
widow implementation
Date 11-05-2002
Introduction to multimedia technologies
Page 18 of 18
Stereovision
Non-Parallel Stereo Images
The case of the two cameras aligned in parallel is not the general case, in the stereovision, the
parallel cameras limits the region of space by which objects are visible in both images. It is
more general to put the cameras so that their axes are angled inwards. This situation is called
the converging cameras , or the nonparallel cameras . In this case the corresponding points
will not generally lie at the same y co-ordinate.
The geometry of stereo with two images taken from two non
epipolar geometry [9].
-parallel cameras is called the
We will introduce in this project the geometric property known in computer vision as the
epipolar constraint .
There are two ways for extracting three-dimensional structure from a pair of images:
In the first, and classical method, known as the calibrated ro ute. We firstly need to calibrate
either cameras (or viewpoints) with respect to some world coordinate system, calculate the
epipolar geometry by extracting the essential matrix of the system, and from this compute the
three-dimensional Euclidean structure of the image scene.
The second or the un -calibrated system, a quantity known as the
fundamental matrix is
calculated from image correspondence and this is then used to determine the projective three dimensional structure of the imaged scene.
Epipolar Geometry
Given a stereo pair of cameras any point in 3D space, P, defines a plane, π, passing through P
and the centers of projection of both cameras. The plane π is called the epipolar plane and the
straight lines where π intersects the image planes are cal led conjugated epipolar lines . It is
the image in one camera of a ray through the optical center and image point in the other
camera. The image in one camera of the other camera’s center of projection is known as the
epipole. In other word, The epipole is the point of intersection of the line joining the optical
centers, that is the baseline, with the image plane (see figure 16)
P
Figure 16. The epipolar geometry and the epipolar lines
Date 11-05-2002
Introduction to multimedia technologies
Page 19 of 19
Stereovision
In figure 15, the epipolar plane is the plane defined by a 3D -point P and the optical center C
and C`.
With the exception of t he epipole, only one epipolar line passes through any image point. All
epipolar lines of one camera pass through the camera’s epipole.
The epipolar constraint states that corresponding points must lie on conjugated epipolar lines
[2].
The epipolar geomet ry is the intrinsic projective geometry between two views. It is
independent of scene structure, and only depends on the cameras internal parameters and
relative position.
Clearly the determination of the scene position of an object point depends upon mat ching the
image location of the object point in one image to the location of the same object point in the
other image.
At first it might seem that correspondence requires a search through the whole image, but the
epipolar constraint reduces this search to a single line.
Thus a point pl in one image generates a line in the other on which its corresponding point pr’
must lie. We see that the search for correspondence is thus reduced from a region to a line,
hence it is a1-D problem (see figure 17)
Figure 17. Shows that corresponding point must lie on the epipolar line
Date 11-05-2002
Introduction to multimedia technologies
Page 20 of 20
Stereovision
The Essential Matrix
The reference frames of the left and right cameras are related via the external parameters.
These define a rigid transformation in 3 -D space by a translation vector T = ( Or-Ol), and a
rotation matrix R. Given a point P in space the relation between Pl and Pr is therefore [2]:
Pr = R (Pl –T)
(5)
The relation between a point in 3 -D space and its projections is described by the equation of
the perspective projection equation:
fi
Pl
Zl
fr
p r = Pr
Zr
pl =
(6)
(7)
The equation of the epipolar plane through P can be written as the cross product of the vectors
Pl , T and (Pl –T) as bellow:
using (5) we obtain
(Pl –T)T × Pl = 0
(8)
× Pl = 0
(RT Pr)T T×
(9)
we can write the cross product as a rank deficient matr ix, than we have:
T × Pl = SPl
(10)
Where:
0
− Tz Ty
S = Tz
0
− Tx
− Ty Tx
0
(11)
then (5) becomes:
PrT E Pl = 0
(12)
E = RS
(13)
Where
The matrix E is called the Essential matrix .
The Essential matrix directly links the epipolar constrain with th e external parameters of the
camera system.
The essential matrix maps points in one camera to epipolar lines in the second camera.
The essential matrix maps points in the camera frame via the epipolar constrains:
prT E pl = 0
Date 11-05-2002
(14)
Introduction to multimedia technologies
Page 21 of 21
Stereovision
We can think of Epl as the projective line in the right plane
epipole er .See the figure 18:
lr that goes through pr and the
lr = Epl
(15)
lr
lr = Epl
Figure 18. Shows the epipolar constrain.
The Fundamental Matrix
Notice that the whole discussion used coordinates in the camera reference Frame but what we
actually measure from images are pixel coordinates Figure (19).
Figure 19. Shows the relationship of the camera frame with
respect to the reference (world reference) frame.
Therefore in order to be able to make profitable use of the essential matrix, we need to know
the transformation from camera coordinates to pixel coordinates, that is, the internal
parameter. This limitation is removed by using the fundamental matrix.
The fundamental matrix is often use in the reconstruction of objects taken with two
uncalibrated cameras. It is a powerful mapping between stereo images. It includes both the
Date 11-05-2002
Introduction to multimedia technologies
Page 22 of 22
Stereovision
internal and the external pa rameters of the system and enables us to describe the epipolar
geometry from matching points only without any prior information about the stereo system.
Assuming Ml be the matrix of internal parameters of the left camera. A point in image
coordinates p^l is written in the camera frame as [2]:
pl = M l−1 pl^
(16)
Similarly, for points in the right camera:
(17)
pr = M r−1 pr^
by substituting these equation into equation (10) above, we get:
prT F pl = 0
(18)
Where
F = M r−T EM l−1
F here is the fundamental matrix.
As with equation (15), for the fundamental matrix we have,
(19)
lr = Epl
(20)
Equation (20) can be thought of as the equation of the projective epipolar line that
corresponds to the point pl.
The di fference between the essential matrix and the fundamental matrix is that the
fundamental matrix is defined in terms of pixel coordinate and that the essential matrix is
defined in terms of camera coordinates. Hence if we can estimate the fundamental matrix
from a number of points matches in pixel coordinates, we can reconstruct the epipolar
geometry with no prior information at all about the internal or external parameters.
Recovering Epipolar Geometry and Eight Point Algorithm
Many techniques are found for epipolar recovery. We now discuss how the epipolar geometry
can be derived from the image match points. We will introduce the straightforward and often
used 8-points algorithm .
The 8-point algorithm is often used to compute the fundamental matrix from a set of eight or
more matching points.
The advantage of the 8
-point algorithm is that it is linear, hence fast and easily
implemented. If 8 point matches are known, then the solution of a set of linear equations is
used.
The disadvantage of this 8 -point algorithm was that it was very susceptible to noise. But
translation and scaling (normalization) before formulating the linear equations improve the
result. Indeed these transformations don't increase really the complexity of the algorithm [10].
Date 11-05-2002
Introduction to multimedia technologies
Page 23 of 23
Stereovision
Assume that n point correspondence have been established between left and right images.
Each correspondence provides a homogeneous linear equation of the form:
prT F pl = 0
Given at least eight correspondences ( n≥8), a homogeneous linear system is formed.
If pl = (x, y,1) and pr = (x’,y’,1) are the two corresponding points in the left and right images,
with their homogenous co-ordinate.
From the epipolar constrain, equation (18), for each point i we have:
[x
]
'
i
f1
yi' 1 f 4
f7
f2
f3
xi
f5
f8
f6
f9
yi =0
1
(21)
Where f1 through f9 are the element of the fundamental matrix.
Expanding (21) for each one of the points we will have eight equation of the form:
xx'f1 + xy'f2 + xf3 + yx'f4 + yy'f5 + yf6 + x'f7 + y'f8 + f9 = 0
(22)
This can be rearranged as AF = 0, where A is a ( n×9) measurement (equation) matrix, and F
is the fundamental matrix represented as a 9 -vector
f1
f2
1 f3
: f4
: f5 = 0
: f6
1 f7
f8
f9
x1' x1
:
:
:
'
xn xn
x1' y1
:
:
:
'
xn yn
x1'
:
:
:
xn'
y1' x1
:
:
:
'
y n xn
y1' y1
:
:
:
'
yn yn
y1'
:
:
:
yn'
x1
:
:
:
xn
y1
:
:
:
yn
(23)
For the eight point algorithm, we have (n=8).
This is a standard linear algebra problem Because if the system is homogeneous, unique
solution exists up to an unknown scale factor.
Date 11-05-2002
Introduction to multimedia technologies
Page 24 of 24
Stereovision
The solution is the eigenvector with minimum eigenvalue of ATA. If we expand A using the
Singular Value Decomposition , the SVD, method then we can write A as [12]:
A = UDV’
(24)
The solution is of SVD s ystem is the column of V associated with the least singular value of
A.
Date 11-05-2002
Introduction to multimedia technologies
Page 25 of 25
Stereovision
The Implementation of the Epipolar Line
In our project the implementation of the epipolar geometry estimate will be build on the 8
point’s algorithm.
We wrote a program using the M ATLAB. The steps to implement the 8 - point’s algorithm is
as follows:
1. Read the left and right images.
2. Chose 8-corresponding points from the left and the right images. First we have to chosen
the first point in the left and chose its corresponding one in t he right image, and so on.
3. Construct the system matrix A from equation (22).
4. Analyze the matrix AAT (8 ×9), using the Singular Value Decomposition, SVD. This is a
build in function in MATLAB. This will return the value
AAT = UDVT
5. The columns of U (8 ×8) are the eigenvectors of AAT and the columns of D (9 ×9) are the
eigenvectors of AAT. While D is (8 ×9) diagonal matrix, contain r singular values on the
diagonal, representing the square root of the nonzero eigenvalues of both
AAT and ATA
[12].
6. Solve the syste m (AAT)F = 0 (equation (19)). The solution is just the components of the
column of V corresponding to the least singular value (the eigenvalues) of AAT.
7. Having the fundamental matrix F, for a matter of stability, we compute it singular value
decomposition,
F = UDVT
8. We set the smallest singular value in the diagonal of D equals to 0; if D’ is the corrected
matrix.
9. The correct estimate of F, is Fp, is given by:
Fp = UD’VT
10. Now we have to use the corrected fundamental matrix
epipolar line.
Fp to calculate and draw the
11. We read a point pl from the left image pl = (x ,y ,1). Then we multiply it by the
fundamental matrix Fp this will give us the parameter for the epipolar line on the right
image:
(a,b,c )= Fp pl
Date 11-05-2002
Introduction to multimedia technologies
Page 26 of 26
Stereovision
12. The equation for the epipolar line is given b y:
ax+by+c = 0
ax + c
y=−
b
13. Draw the epipolar line on the right image, using the above equation. This is done just by
direct substitution.
Results
The algorithm for finding the fundamental matrix and the epipolar line has been i mplemented
using different images; some of the images that used are the following:
The Epipole
Epipolar
lines
Selection
Points
Selection points
Figure 19. This figure shows the epipolar line of four different points. It
also shows where the epipole is located.
Selection
Points
Epipole
Epipolar
lines
Selection
Points
Figure 20. This figure shows the epipolar line of four different points. It
also shows where the epipole is located.
Date 11-05-2002
Introduction to multimedia technologies
Page 27 of 27
Stereovision
The above result was found, using the 8 -point algorithm. After taking the eight corresponding
points from the left and right image, the program calculate the fundamenta l matrix Fp. Then it
finds the parameter of the epipolar line, from the co -ordinates of the point in the left image
and the fundamental matrix. Its draw the line on the right image as shown.
Since we are really choosing the 8 corresponding points by hand,
then the calculated
fundamental matrix, is just an estimation, and hence its subjected to errors. But compared to
its simplicity and its speed, the 8 -point algorithm is still one of the best -used algorithms to
estimate the epipolar geometry.
Date 11-05-2002
Introduction to multimedia technologies
Page 28 of 28
Stereovision
Conclusion
The work on this project was mainly concentrating on the correspondence problem. This was
done by investigating the Area Based method for finding the matches between two images
taken from parallel cameras.
Many algorithms and methods were suggested to f ind the best corresponding points, these
include the Single Window, Multiple Windows and Adaptive Search Windows.
Solutions to the problems of Multiple Corresponding Points, and Projective Distortion were
discussed and implemented
From the implementatio n of the Single Window method, the depth map was calculated. The
results was suffering from inaccuracy in the areas where there is a big change is disparity
(disparity shift).
The result was improved using the Multiple Windows method. It shows better disp
arity
estimation in the areas where the disparity shift appears. And hence a more accurate depth
map was obtained using this method.
The Single Window method was faster than the Multiple Windows method. Using MATLAB
the Single Window method is about 6 ti mes faster. It takes about 5 minute to calculate the
depth map of a single image (256 ×256) using the Single Window method, and about 30
minuets using the Multiple Windows method. The speed of the program could be improved
using C++ code. This will make th e Multiple Window method faster and hence could be used
practically.
The case of finding the epipolar line in one image that correspond to a point in the other
images, was studied and implemented for the case of a pair of nonparallel stereo images. This
proves the epipolar constrain, that simplify the process of finding the correspondence in a pair
of nonparallel images.
The method was implemented practically using the 8
-point algorithm for finding the
fundamental matrix. The 8 -point algorithm suffers fr om noise, due to the inaccuracies in the
measurement or specification of the 8 -corresponding points, since they were done manually
by the user. The result was much improved by enforcing the singularity constrain of the
fundamental matrix by normalizing the fundamental matrix, this was done using the Singular
Value Decomposition (SVD) method. The fundamental matrix was more stable using this
method.
The 8-point algorithm is very simple and fast compared to other abstract iterative algorithms
and still easy to code and implement.
Further work could try to improve the method of estimating the fundamental matrix, and
finding more accurate epipolar lines.
Also one can implement the rectification for the image, and then apply any one of the
methods used to find the correspondence in the parallel camera case. This can lead to
estimating the disparity map and calculating the depth map, and hence building the 3-D model
of the image.
Date 11-05-2002
Introduction to multimedia technologies
Page 29 of 29
Stereovision
References
1. A. Fusiello, V. Roberto and E. Trucco, “Efficient Stereo with Multiple Windowing”, Proc.
IEEE Intern. Conf. on Computer Vision and Pattern Recognition CVPR97, Puerto Rico,
1997. (see appendix)
2. E. Trucco and A. Verri, “Introductory techniques for 3.D computer vision”, Pentice -Hall
1998, ISBN 0-13-261108-2
3. Jonathan Potter, “ Computer Algorithms for the Reconstruction of Spatial Geometry from
Stereoscopic Images”, http://cpcug.org/user/scifair/potter.html
4. T.Kanade, , M.Okutomi, “A stereo matching algorithm with an adapti ve window: Theory
and experiment”, IEEE Transactions on pattern analysis and Machine intelligence, vol 16
No.9 September 1994. (see appendix)
5. R.Szeliski, “Stereo algorithm and representations for image
Brithish machine vision conference (BMVC’99).
-based rendering”, 10
6. H.Ishikawa, D.Geiger, “Occlusion, discontinuities, and epipolar lines in stereo”, 5
European conference on computer vision (ECCV’98).
th
th
7. Y.Boykov, O.Veksler, R.Zabih, “A Variable window Approach to early vision”, IEEE
Transactions on Patte rn Analysis and Machine Intelligence, Vol. 20, No. 12, December
1998
8. C.L.Zitnick and T.Kanade, “A co -operative algorithm for stereo matching and occlusion
detection”, IEEE Transaction on pattern analysis and Machine intelligence, vol.22, No.7
July 2000.
9. Richard Hartley, Andrew Zisserman, “Multiple View Geometry in Computer Vision”,
Cambridge Univ Pr (Short); ISBN: 0521623049, pp 219-227.
10. Richard I. Hartley, “ In Defence Of The 8
-Point Algorithm”, Proc. 5 th International
Conference Of Computer Vision, Cam bridge, pp.1064-1070 (1995). (see appendix)
11. Milan Sonka, Vaclav Hlavac, Roger Boyle, “Image Processing, Analysis and Machine
Vision ”, Chapman & Hall Computing ISBN: 0 -412-45570-6.
12. Gilbert Strang, “linear algebra and its applications”, Harcout Brace & Co
addition, ISBN:0-15-551005-3, appendix A, pp443.
Date 11-05-2002
Introduction to multimedia technologies
mpany, 3 rd
Page 30 of 30
Stereovision
Appendix for Pseudo Code
Pseudo code for the main function SingleWindow
Load left and right image
Find the area of the image to operate in
For y = 1 to the row end of operation area
For x = 1 to the column end of operation area
CorrespondenceVector = SWCorrespondence
If the CorrespondenceVector is a single value
Correspondence(y,x) = CorrespondenceVector
Else
If x != 1 then
Correspondence(y,x) = Correspondence(y,x-1)+1
Else
Correspondence(y,x) = Correspondence(y-1,x)
End
End
End
Disparity(y,x) = Correspondence(y,x) - x
End
End
Save Disparity map
Init the Focal length and the Baseline
Calculate Depth map
Depth map Scale To uint8
Show scaled Depth map7
Pseudo code for the function SWCorrespondence
Make the window for image 1
Make the region boundaries
If SSD in chosen
For displacement (d) = min of region to the max of region
Calculate SSD for window 1 and slid the window of image 2 by d
End
Return a vector of the index of all the minimal SSD values
Else
If Normalise SSD in chosen
For displacement (d) = min of region to the max of region
Calculate the Normalise SSD for window 1 and slid the window of image 2 by d
End
Return a vector of the index of all the minimal Normalise SSD values
Else
For displacement (d) = min of region to the max of region
Calculate the cross-correlation for window 1 and slid the window of image 2 by the d
End
Return a vector of the index of all the maximal cross-correlation values
End
End
Date 11-05-2002
Introduction to multimedia technologies
Page 31 of 31
Stereovision
Pseudo code for the function SMW
Load left and right image
Find the area of the image to operate in
For y = 1 to the row end of operation area
For x = 1 to the column end of operation area
For w = 1 to number of windows (9)
CorrespondenceMatrix = SMWCorrespondence
WindowCorrespondence(w) = The index of the most probable of CorrespondenceMatrix
End
Disparity(y,x)= The index of the most probable of WindowCorrespondence(w)
End
End
End
Save Disparity map
Init the Focal length and the Baseline
Calculate Depth map
Depth map Scale To uint8
Show scaled Depth map7
Pseudo code for the function SMWCorrespondence
Make the window for image 1
Make the region boundaries
If Normalise SSD in chosen
If Normalise SSD in chosen
For displacement (d) = min of region to the max of region
Calculate the Normalise SSD for window 1 and slid the window of image 2 by d
End
Else
For displacement (d) = min of region to the max of region
Calculate the cross-correlation for window 1 and slid the window of image 2 by the d
End
Return a vector of the index of all the maximal cross-correlation values
End
Return a Calculatevector
Date 11-05-2002
Introduction to multimedia technologies
Page 32 of 32
Stereovision
Appendix for Code
Code for the function SingleWindow
function SingleWindow
% //////////////////////////// ///////////////////////////////////////////////////////////////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001
//
% ///////////////////////// //////////////////////////////////////////////////////////////////////////////////
tic
warning off
ImL = imread('corridor_l_original.tiff');
ImR = imread('corridor_r_original.tiff');
[xL yL] = size(ImL);
[xR yR] = size(ImR);
WindowsSize = 7;
half = fix(WindowsSize/2);
ymax = yL-(WindowsSize*2);
xmax = xL-(WindowsSize*2);
Region = 20;
Disp = zeros(ymax,xmax);
CorrespondingPoints = zeros(ymax,xmax);
ClusterNo = 0;
NCP = 0;
MCP = 0;
for y = 1 : ymax;
for x = 1 : xmax;
CorrespondenceVector = SWDisparity(x+WindowsSize,y+WindowsSize,WindowsSize,ImL,ImR,Region);
CorrespondingPoints(y,x)= CorrespondenceVector(1);
Disp(y,x) = CorrespondingPoints(y,x)-(x+WindowsSize);
end
end
save('DisparityHeleRegionNoMPC.mat','Disp','-mat')
Disp(find(Disp == 0)) = -50;
figure(1),imshow(scale2uint8(abs(Disp)));
%Position left eye: (250, -150, 170) looking at (100, 1600, 100)
%Position right eye: (259.963,-149.146,170) looking at (109.963,1600.85,100)
f = sqrt(sum(([100, 1600, 100] -[250, -150, 170]).^2))
baseline = 109.963-100;
z = double((baseline*f)./abs(Disp));
figure(2),imshow(scale2uint8(z));
t = toc
Date 11-05-2002
Introduction to multimedia technologies
Page 33 of 33
Stereovision
Code for the function SWCorrespondence
function CorrespondenceVector=SWCorrespondence(x,y,WindowsSize,Im1,Im2,Region,str)
% /////////////////////////////// ////////////////////////////////////////////////////////////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001
//
% //////////////////////////// ///////////////////////////////////////////////////////////////////////////////
[xIm2 yIm2] = size(Im2);
half = fix(WindowsSize/2);
xcounter = x-half:x+half;
ycounter = y-half:y+half;
WindowIm1 = Im1(ycounter,xcounter);
if (x-Region < 0)
% if the min of the regiont is less than the image
dmin = 0;
dmax = x+Region;
else
if (x+Region >xIm2-WindowsSize-1)
% if the max of the regiont is bigger than the image
dmin = x-Region;
dmax = xIm2-WindowsSize-1;
else
% if the regiont is inside than the image
dmin = x-Region;
dmax = x+Region;
end
end
c = zeros(1,dmax-dmin);
if (strcmp(str,'SSD') == 1 )
for d = 1: dmax-dmin;
c(d) = sum(sum((double(WindowIm1)-double(Im2(ycounter,d+dmin:d+WindowsSize-1+dmin))).^2));
end
CorrespondenceVector = find(c == min(c))+half+dmin;
else
if (strcmp(str,'SSDN') == 1 )
for d = 1: dmax-dmin;
WindowIm2 = Im2(ycounter,d+dmin:d+WindowsSize-1+dmin);
c(d) = (sum(sum((double(WindowIm1)-double(WindowIm2)).^2))./...
sqrt(sum(sum(double(WindowIm1)))*sum(sum(double(WindowIm2).^2))));
end
CorrespondenceVector = find(c == min(c))+half+dmin;
else
for d = 1: dmax-dmin;
c(d) = sum(sum(double(WindowIm1).*double(Im2(ycounter,d+dmin:d+WindowsSize-1+dmin))));
end
CorrespondenceVector = find(c == max(c))+half+dmin;
end
end
Date 11-05-2002
Introduction to multimedia technologies
Page 34 of 34
Stereovision
Code for the function CorrespondenceTest
function CorrespondenceTest
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001
//
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
ImL = imread('corridor_l_original.tiff');
ImR = imread('corridor_r_original.tiff');
[xL yL] = size(ImL);
figure(1),subplot(1,2,1), imshow(ImL),subplot(1,2,2), imshow(ImR);
WindowsSize = 9;
half = fix(WindowsSize/2);
while(0==0)
[x,y,button] = ginput(1);
x = round(x);
y = round(y);
if( (x < xL-WindowsSize) & (x > WindowsSize))
CorrespondenceVectorSSD = SWCorrespondence(x,y,WindowsSize,ImL,ImR,20,'SSD');
CorrespondenceVectorCorr = SWCorrespondence(x,y,WindowsSize,ImL,ImR,20,'Corr');
figure(1),subplot(1,2,1), imshow(ImL);
hold on
plot(x,1:yL,'r')
plot(1:xL,y,'r')
plot(x,y,'rx')
drawnow;
hold off
xlabel(sprintf('x:%d y:%d value:%d',x,y, double(ImL(y,x))))
subplot(1,2,2), imshow(ImR)
title(sprintf('Yello SSD disp:%d Blue Cross-Correlations disp:%d',...
CorrespondenceVectorSSD(1)-x,CorrespondenceVectorCorr(1)-x));
hold on
plot(1:xL,y,'r')
plot(CorrespondenceVectorSSD,y,'yx')
plot(CorrespondenceVectorCorr,y,'bx')
drawnow;
hold off
xlabel(sprintf('SSD x:%d y:%d value:%d',CorrespondenceVectorSSD(1),...
y,double(ImR(y,CorrespondenceVectorSSD(1)))))
end
end
Date 11-05-2002
Introduction to multimedia technologies
Page 35 of 35
Stereovision
Code for the function SMW
function SMW
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001
//
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
tic
warning off
ImL = imread('corridor_l_original.tiff');
ImR = imread('corridor_r_original.tiff');
[xL yL] = size(ImL);
[xR yR] = size(ImR);
WindowsSize = 7;
half = fix(WindowsSize/2);
ymax = yL-(WindowsSize*2);
xmax = xL-(WindowsSize*2);
Region = 20;
NoWindows = 9;
dLw = zeros(ymax,xmax,NoWindows);
dL = zeros(ymax,xmax);
ValueL = zeros(1,NoWindows);
CorrespondL = zeros(1,NoWindows);
for y = 1 : ymax;
for x = 1 : xmax;
for w = 1 : 9;
[CorrespondenceMatrixL, dminL] = SMWCorrespondenceSSD5(x+WindowsSize,...
y+WindowsSize,ImL,ImR,w,WindowsSize,Region);
[ValueL(w),CorrespondL(w)] = min(CorrespondenceMatrixL);
dLw(y,x,w) = (CorrespondL(w)+WindowsSize + dminL) -(x+WindowsSize);
if (length(find(CorrespondenceMatrixL == min(min(CorrespondenceMatrixL)))) > 1)
if(x==1)
dLw(y,x,w) = dLw(y-1,x,w);
else
dLw(y,x,w) = dLw(y,x-1,w);
end
else
dLw(y,x,w) = (CorrespondL(w)+WindowsSize + dminL) -(x+WindowsSize);
end
end
[Value,Window] = min(ValueL);
dL(y,x) = dLw(y,x,Window);
end
end
save('DisparitySMWRegion.mat','Disp','-mat')
dL(find(dL == 0)) = -50;
figure(1),imshow(scale2uint8(abs(dL)));
%Position left eye: (250, -150, 170) looking at (100, 1600, 100)
%Position right eye: (259.963,-149.146,170) looking at (109.963,1600.85,100)
f = sqrt(sum(([100, 1600, 100]-[250, -150, 170]).^2))
baseline = 109.963 -100;
z = double((baseline*f)./abs(dL));
figure(2),imshow(scale2uint8(z));
t = toc
Date 11-05-2002
Introduction to multimedia technologies
Page 36 of 36
Stereovision
Code for the function SMWCorrespondence
function [CorrespondenceVector,dmin]=SMWCorrespondence(x,y,Im1,Im2,w,WindowsSize,Region,str)
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001
//
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
[xIm2 yIm2] = size(Im2);
half = fix(WindowsSize/2);
counterhalf = [-half half];
counterright = [0 WindowsSize-1];
counterleft = [-WindowsSize+1 0];
counterdown = counterright;
counterup = counterleft;
Windows = [[counterdown counterhalf]; [counterup counterhalf]; [counterhalf counterright]; [counterhalf counterleft];... [counterhalf
counterhalf]; [counterdown counterright]; [counterdown counterleft]; [counterup counterright];...
[counterup counterleft]];
WindowIm1 = zeros(WindowsSize);
WindowIm2 = zeros(WindowsSize);
WindowIm1 = Im1(y + (Windows(w,1):Windows(w,2)) ,x+(Windows(w,3):Windows(w,4)));
if (x-Region < 0)
% if regiont is less than the image
dmin = 0;
dmax = (x+Region)-dmin;
else
dmin = x-Region;
if (x+Region >xIm2-(WindowsSize+dmin))
% if regiont is bigger than the image
dmax = (xIm2-(dmin+(WindowsSize*2)));
else
% if regiont is inside than the image
dmax = Region*2;
end
end
c = zeros(1,dmax);
if (strcmp(str,'SSDN') == 1 )
for d = 1: dmax;
WindowIm2 =Im2(y + (Windows(w,1):Windows(w,2)),d+WindowsSize+dmin+(Windows(w,3):Windows(w,4)));
c(d) = sum(sum((double(WindowIm1)-double(WindowIm2)).^2));...
sqrt(sum(sum(double(WindowIm1)))*sum(sum(double(WindowIm2).^2)));
end
else
for d = 1: dmax;
WindowIm2 =Im2(y + (Windows(w,1):Windows(w,2)),d+WindowsSize+dmin+(Windows(w,3):Windows(w,4)));
c(d) = sum(sum(double(WindowIm1).*double(WindowIm2)));
end
end
CorrespondenceVector = c;
Date 11-05-2002
Introduction to multimedia technologies
Page 37 of 37
Stereovision
Code for the function SMWCorrespondenceTest
function SMWCorrespondenceTest
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001
//
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
ImL = imread('corridor_l_original.tiff');
ImR = imread('corridor_r_original.tiff');
[xL yL] = size(ImL);
WindowsSize = 9;
half = fix(WindowsSize/2);
counterhalf = [-half half];
counterright = [0 WindowsSize-1];
counterleft = [-WindowsSize+1 0];
counterdown = counterright;
counterup = counterleft;
Windows = [[counterdown counterhalf]; [counterup counterhalf]; [counterhalf counterright]; [counterhalf counterleft];... [counterhalf
counterhalf]; [counterdown counterright]; [counterdown counterleft]; [counterup counterright]; ...
[counterup counterleft]];
figure(1),subplot(1,2,1), imshow(ImL),subplot(1,2,2), imshow(ImR);
while(0==0)
[x,y,button] = ginput(1);
x = round(x);
y = round(y);
ValueL2 = zeros(1,WindowsSize);
CorrespondL2 = 0;
ValueL = zeros(1,WindowsSize);
CorrespondL = zeros(1,WindowsSize);
for w = 1 : 9;
[CorrespondenceMatrixL, dminL] = SMWCorrespondence(x,y,ImL,ImR, w,WindowsSize,20,'Corr');
[ValueL(w),CorrespondL(w)] = max(CorrespondenceMatrixL);
CLw(w) = CorrespondL(w)+WindowsSize + dminL;
end
Window = 0;
[Value,Window] = min(ValueL);
CorrespondenceVector = CLw(Window);
figure(1),subplot(1,2,1), imshow(ImL);
hold on
plot(x,1:yL,'r')
plot(1:xL,y,'r')
plot(x,y,'rx')
plot(x+Windows(Window,3),y + (Windows(Window,1):Windows(Window,2)),'y')
plot(x+Windows(Window,4),y + (Windows(Window,1):Windows(Window,2)),'y')
plot(x + (Windows(Window,3):Windows(Window,4)),y + Windows(Window,1),'y')
plot(x + (Windows(Window,3):Windows(Window,4)),y + Windows(Window,2),'y')
drawnow;
hold off
xlabel(sprintf('x:%d y:%d value:%d',x,y, double(ImL(y,x))))
title(sprintf('Window:%d disp:%d',Window,CorrespondenceVector(1)-x))
subplot(1,2,2), imshow(ImR)
hold on
plot(1:xL,y,'r')
plot(CorrespondenceVector,y,'yx')
drawnow;
hold off
xlabel(sprintf('x:%d y:%d value:%d',CorrespondenceVector(1),y, double(ImR(y,CorrespondenceVector(1)))))
end
Date 11-05-2002
Introduction to multimedia technologies
Page 38 of 38
Stereovision
Code for the function scale2uint8
function ScaledIm = scale2uint8(Im)
% ////////////////////////////////////////////////////////////////////////////////////// /////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001
//
% /////////////////////////////////////////////////////////////////////////////////// ////////////////////////
minIm = min(min(Im));
Im = (Im-minIm);
maxIm = max(max(Im));
Im = (Im./maxIm).*255;
Im=-(Im-255);
ScaledIm = uint8(Im);
Date 11-05-2002
Introduction to multimedia technologies
Page 39 of 39
Stereovision
Code For The Main Function DrawEpipolarline
function drawEpipolarLine (ImL,ImR)
% //////////////////// ///////////////////////////////////////////////////////////////////////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001
//
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
[PL,PR] = eight(ImL,ImR);
A = findA(PL,PR);
[F,Fp] = findTheF(A)
[i j] =size(ImR);
% show the two images
figure(1),subplot(1,2,1), imshow(ImL),subplot(1,2,2), imshow(ImR);
while(0==0)
[xL,yL] = ginput(1);
pl = [xL; yL; 1];
pr = Fp*pl;
a = pr(1);
b = pr(2);
c = pr(3);
x = 1:j;
y = (-a.*x-c)/b;
hold on
subplot(1,2,1), plot(xL,yL,'rx')
drawnow;
hold off
subplot(1,2,2)%, imshow(ImR)
hold on
plot(x,y(x),'r')
drawnow;
hold off
end
Date 11-05-2002
Introduction to multimedia technologies
Page 40 of 40
Stereovision
Code For The Function eight
function [PL,PR] = eight(ImL,ImR)
%//////////////////////////////////////////////////////////////////////////////////////////////////////
% This function is written for the 4- week project
//
% of the Stereo Vision of the Introduction to Multimedia / MAY 2001 //
%//////////////////////////////////////////////////////////////////////////////////////////////////////
%
% This is the function which reads the eight points and
% put them in a matrix form to pass them to matrix A
onesRow = ones(1,8);
% show the two images
figure(1),subplot(1,2,1), imshow(ImL),subplot(1,2,2), imshow(ImR);
% read the 8 - points from the two images: the left one and then the right one...
for i = 1:8
i
[xL,yL] = ginput(1);
XL(i) = round(xL);
YL(i) = round(yL);
[xR,yR] = ginput(1);
XR(i) = round(xR);
YR(i) = round(yR);
end
% construct the points matrix as bellow:
%
[x1L x2L x3L ... x8L]
%PL = [y1L y2L y3L ... y8L]
%
[1 1 1 ...
1]
%
% Enter the 8 - corresponding points on the right image as a matrix as bellow:
%
%
[x1R x2R x3R ... x8R]
%PR = [y1R y2R y3R ... y8R]
%
[1 1 1 ...
1]
PL = [XL; YL; onesRow];
PR = [XR; YR; onesRow];
Date 11-05-2002
Introduction to multimedia technologies
Page 41 of 41
Stereovision
Code For The Function findA
function A = findA(PL,PR)
% ///////////////////////////////////////////////////////////////////////////////////////////////////////////
% This function is writen for the 4- week project
//
% of the Stereo Vision of the Intoduction to Multimedia / MAY 2001
//
% ///////////////////////////////////////////////////////////////////////////////////////////////////////// /
% the input is the Matrices points, PL and PR
% the output is the Coefficient Matrix A that can be used to solve the equation AF = 0 using the SVD
% where F is the fundamental Matrix
[m,n] = size(PL);
x = zeros(n,n+1);
for i = 1:n
x(i,:) = [PL(1,i)*PR(1,i) PL(1,i)*PR(2,i) PL(1,i) ...
PL(2,i)*PR(1,i) PL(2,i)*PR(2,i) PL(2,i) PR(1,i) PR(2,i) 1];
end
A = x;
Code For The Function findTheF
function [F,Fp] = findTheF(A)
% /////////////////////////////////////////////////////////////// ////////////////////////////////////////////
% This function is writen for the 4- week project
//
% of the Stereo Vision of the Intoduction to Multimedia / MAY 2001
//
% ///////////////////////////////////////// /////////////////////////////////////////////////////////////////
% This function is used to find the fundamental matrix.
% A - equation matrix
% F - fundamental matrix
% Fp - the fundamental matrix after correction
[U,S,V] = svd(double(A'*A));
% Find the minimum eigenvalue in S
minS=S(1,1);
columnS=1;
for i=1:9
if S(i,i) < 0.0001
minS=S(i,i);
columnS=i;
break;
break;
elseif S(i,i) < minS
minS=S(i,i);
columnS=i;
end;
end;
% Cosnstruct The Fundamental Matrix F
F=[V(1,columnS), V(4,columnS), V(7,columnS);...
V(2,columnS), V(5,columnS), V(8,columnS);...
V(3,columnS), V(6,columnS), V(9,columnS);];
% correct F by making D(3,3) = 0 and recalculate F
[Q1, D, Q2]=svd(double(F));
D(3,3)=0;
Fp=Q1*D*Q2';
Fp = Fp/Fp(3,3);
save 'fundemental.mat' F Fp;
Date 11-05-2002
Introduction to multimedia technologies
Page 42 of 42
Stereovision
Appendix: The paper “Efficient Stereo with Multiple
Windowing”
Date 11-05-2002
Introduction to multimedia technologies
Page 43 of 43
Stereovision
Appendix: the paper “A stereo matching algorithm with an
adaptive window: Theory and experiment”
Date 11-05-2002
Introduction to multimedia technologies
Page 44 of 44
Stereovision
Appendix: the paper “In Defence Of The 8-Point Algorithm”
Date 11-05-2002
Introduction to multimedia technologies
Page 45 of 45