IPOLJournal·ImageProcessingOnLine
HOME·ABOUT·ARTICLES·PREPRINTS·WORKSHOPS·NEWS·SEARCH
Quasi-EuclideanEpipolarRectification
PascalMonasse
article demo archive
published
reference
→ BibTeX
2011-09-13
PascalMonasse,Quasi-EuclideanEpipolarRectification,ImageProcessingOnLine,
1(2011).http://dx.doi.org/10.5201/ipol.2011.m_qer
CommunicatedbyLuisAlvarez
DemoeditedbyAgustinSalgado
[email protected]
Overview
Thebinocularstereopipelineperformsthefollowingtask:fromapair
ofimagesofascenecapturedbytwocameras(orthesameone)at
differentpositions,computethedistancemaptoonecamera.Under
particular conditions (called therectifiedcase) corresponding points
areatsameordinateinbothimagesandthedistanceistheinverse
of an affine transform (whose coefficients depend on camera
parameters) of the abscissa difference, called thedisparity. In
general,theoutputissimplythedisparitymap.
Content
Overview
References
TheFusiello-Irsaramethod
OnlineDemo
Algorithm
Implementation
Examples
Sourcecode
Acknowledgments
If the cameras satisfy the pinhole model assumption, an adequately chosen combination of rotations
and adjustment of cameras' internal parameters yield the rectified case. This amounts to applying
homographiestotheimages.Findingandapplyingthesehomographiesiscalledepipolar rectification.
Theinputisadiscretesetofcorrespondingpointsinimages.Sincetherearemoredegreesoffreedom
than constraints, several methods exist, each trying to minimize the change applied to images
according to its own measure. One method to achieve this and more discussion can be found inthis
book.
The method ofreference2, expanded inreference3,assumesbothcamerasarethesame(thusthey
must have the same size) but only partial knowledge of camera's internal parameters (uncalibrated
case): square pixels, unknown focal length and principal point at image center. It then simulates
appropriatepurerotationsofeachview,whichcanbedonewhenthecorrectfocallengthisestimated.
References
1. RichardHartleyandAndrewZisserman(2003).MultipleViewGeometryinComputerVision.
CambridgeUniversityPress.ISBN978-0-521-54051-3
2. AndreaFusielloandLucaIrsara,Quasi-Euclideanuncalibratedepipolarrectification,19th
InternationalConferenceonPatternRecognition(ICPR),Tampa,Florida,2008.DOI:
10.1109/ICPR.2008.4761561.
3. AndreaFusielloandLucaIrsara,Quasi-Euclideanepipolarrectificationofuncalibratedimages,
MachineVisionandApplications,SpringerBerlin/Heidelberg,2010.DOI:10.1007/s00138-0100270-3.
TheFusiello-Irsaramethod
Background
Theepipolarconstraintequationiswritten
w i t hXl andXr 3-vectors of homogeneous coordinates of corresponding points andF the 3x3
fundamentalmatrix.Intherectifiedcase,Fhasthespecialform(uptoascalefactor)
Inthatcase,epipolesareatinfinityinhorizontaldirectionsince Fe1=F Te1=0,andcorrespondingpoints
havesameordinatesince
AnydecompositionofafundamentalmatrixFinto
withHl andHr invertible 3x3 matrices, yields left and right homographies rectifying the images since
then
Rotations
TherotationofmatrixRofacameraarounditsfocusproducesahomographyoftheimageofmatrix
withKtheupper-triangularmatrixbuiltfromcameraparameters,whichisreasonablysupposedwritten
as:
with(xC,yC)theprincipalpoint,assumedtobetheimagecenter,andftheunknownfocallength.
TheFusiello-Irsaramethodlooksforrotationmatrices RlandRrandfocallengthfsuchthat
whichcanbesimplifiedbyremovinginnermosttwoinstancesof Ksince
This equality also shows that in the rectified case, we can always change the internal parameters of
bothcamerastoKn,amountingtohomography
andremainrectified.
AlsoifRxisanyrotationaround x-axis,wehave
so we can ignore thex-axis rotation inRl for example. Finally, the unknown matrices can be
parameterizedby
sothatexpectedvalueofgisintheinterval[-1,1],thesameorderofmagnitudeasangles.
TheSampsonerrorassociatedtoalgebraicerror Eis
withJthe1x4matrixofpartialderivativesof Ewrtthe4variables:
leadingto
Themethodlooksforthesetofparameters
minimizingthesumofSampsonerrorsovermatchingpairsbyLevenberg-Marquardtmethod.
Jacobiancomputation
TheiterativeerrorminimizationrequirescomputationoftheJacobianmatrix.Notinganyvariable p,we
have
with
andtheoverlineoperatordefinedas
Thepartialderivativeswrtanyofthe5anglesareeasytocompute,aseachoneisinvolvedinonlyone
rotationmatrix:
with
with
beingitsderivative,andsimilarlyforRyandRz.
Now,thederivativewrtgis
with
Fixingfreeparameters
We take advantage of the invariance properties of the rectified configuration, wrt to rotation around
baseline (x-axis) and change of internal parameters, to fix the center point of each image through
applicationofrectifyinghomography.
First, we apply anx-rotation so as to keep ordinate of central pixel fixed, meaning replacing the
homographiesby:
withtheangle
Finally,weadjustindependentlytheabscissaofleftandrightcameratokeepthecenterpointatcenter.
Noting
andknowingtheresultingabscissax'ofcenterpointthroughH'l,basedonmatrixK=Kn(w/2),weneed
totranslatebyw/2-x',yieldingthefinalhomographies:
OnlineDemo
The online demo of the algorithm has additional steps to make it more widely testable, as the
demonstratedalgorithmitselftakesasinputonlycorrespondingpointsbetweenimagesandthesizeof
imagesandoutputshomographies:
1. ExtractkeypointswithSIFTalgorithmineachimage;
2. Matchkeypointsbasedondescriptors;
2. Matchkeypointsbasedondescriptors;
3. RemoveoutliercorrespondencesbyacontrarioRANSACalgorithm(ORSA)lookingfora
consensusonfundamentalmatrix;
4. ApplyFusiello-Irsaraepipolarrectificationalgorithmdescribedabove;
5. Applyhomographiestobothimages.
Rememberthatinputimagesmusthavethesamesize.Acommonfailurecaseiswhenoneepipoleisin
the image. Another problem is when geometrical distortion is noticeable in images (this happens
especiallywithwide-anglelenses).Inthatcase,itispreferabletocorrectthisdistortion(useforexample
thisIPOLalgorithm).
Theimagerectanglecanbemappedtoalargequadrilateraliftheepipoleisoutsidebutclosetoimage
boundary. To avoid potentially very long transfer time over the network, we do not show the rectified
imagecontainingthefullquadrilateral.Instead,weshowonlytheportionofsamesizeasoriginalimage
and with same center point, since this point has been fixed by adjustment of parameters of
homography.
Hereisanexampleofalgorithmoutput:
01sift::1stimage:473keypoints
02sift::2ndimage:496keypoints
03sift::matches:146
04seed:1286188619
05Remove19/146duplicatematches
06Optimizedstochasticmode(ORSA).
07nfa=-130.795size=111(niter=1)
08nfa=-147.135size=102(niter=6)
09nfa=-156.207size=106(niter=10)
10nfa=-190.564size=103(niter=16)
11nfa=-207.249size=107(niter=32)
12nfa=-207.436size=105(niter=424)
13nfa=-211.225size=106(niter=475)
14bestmatchingfound:106pointslog(nfa)=-211.225(500iterations)
15F=[-3.42803e-094.35331e-07-0.000178794;-5.00755e-07-4.76413e-080.00
123017;0.000225377-0.00117995-0.0373931]
16Geometricerrorthreshold:0.869643
17LMiterations:8f=760.855
18Finalrectificationerror:0.206727pix
19Disparity:-90-26
Lines01and02showthenumberofSIFTpointsextractedineachimage.Line03showsthenumberof
SIFT correspondences. Lines 04 to 16 are output from the ORSA algorithm. Line 04 shows the seed
used for random number generation and is useful only for debugging purpose. Line 05 shows the
numberofremainingcorrespondenceswhenduplicatesareremoved:someSIFTpointsmaybefound
to have several main directions in their descriptor, which creates duplicates and can fool the ORSA
match independence assumption. Lines 07 to 13 show each time a more meaningful group of
correspondencesisfound.Line14sumsupthebestconsensus.Line15showsthefundamentalmatrix.
Line16displaysthegeometricthresholdthatwasselectedbyORSAtodiscriminateinliersandoutliers.
Here,everySIFTpointmusthaveitscorrespondingSIFTpointatlessthan0.87pixelfromtheepipolar
line. Lines 17 and 19 show the result of the rectification itself. The number of Levenberg-Marquardt
iterationsishere8andthefoundfocallengthisdisplayed.Line18showstheresultingmeanSampson
error.Finally,line19givestheminimumandmaximumdisparityofSIFTpointsintherectifiedimages.
Algorithm
ThealgorithmisaLevenberg-MarquardtiterativeminimizationoftheSampsonerror,startingwithall6
unknownsat0.Itstopswhenoneofthefollowingconditionsissatisfied:
1. theRMSEisbelow0.1pixel.
2. therelativeerrorchangefromoneiterationtothenextisbelow1E-3.
3. 300iterationshavebeenperformed.
Thefirstcaseisconsideredasuccessandthelastoneafailure.Thesecondcasemaybeeither:the
global minimum is reached but the data inaccuracy prevents the error from dropping below 0.1
(success)oralocalminimumisreached(failure).
(success)oralocalminimumisreached(failure).
Implementation
Eliminatingnullequations
DuringLevenberg-Marquardtminimization(seereference1),onehastosolveasystemof6equations
withJ the Jacobian matrix of the error term, of size Nx6 withN the number of correspondences.The
solutionXistheupdatetoapplytothecurrentsetofparameters.Oneofthese6equationsmayhappen
tobethetrivialone
ThisisthecasewhenonecolumnofJisnull.Thishappensinparticularatthefirstiterationforthe gderivative,whenall5anglesare0,since
Notice that the diagonal elements ofJTJ are the square norms of column vectors ofJ. So, to avoid
division by zero or by a negligible quantity, we find the maximum diagonal entryM ofH and put in
m_nullColstheindicesofdiagonalelementslowerthankM, withk=1E-9.Weremovetheselinesand
columnsfromthelinearsystemtosolveinthemethod
voidMinLM::compress(matrix<flnum>&JtJ,vector<flnum>&B).
Thesolutionofthesmallersystemisthencompletedwith0atremovedindicesinthemethod
voidMinLM::uncompress(vector<flnum>&B),
sincetheseelementscorrespondtovariableswithnullderivative,whichmustnotbechanged.
Homographyapplicationwithanti-aliasingfilter
Applying directly the computed homographies to images can result in aliasing artifacts.Although less
severe than in the case of a zoom out for example, because we simulate rotations in our case, it is
nonetheless important to attenuate the aliasing. This happens when there is a compression in one
direction:ateachpointoftheoriginalimage,thehomographycanbeapproximatedbyits2x2Jacobian
matrix. When one of the singular values is below 1, compression occurs and anti-aliasing filter is
recommended. For this, we have to know the maximum unidirectional compression factor over the
image.Undernormalconditions(imageoflineatinfinityoforiginalimagedoesnotmeettheframeof
theresultimage),themaximumcompression(lowestsingularvalueofJacobian)isreachedatoneof
thefourcornersoftheoriginalimage.Theprocedureisthefollowing:
1. Ateachcorneroftheoriginalimageframe,compute2x2Jacobianmatrixofhomography Hand
itsminimumsingularvalue.Noteztheminimumoftheseoverall4corners.
2. Ifz<1,composetotheleft Hwithazoominoffactor s=1/z;applytheresultinghomographyto
theimage,yieldinganimageofsizesw_sh_;convolvewithaGaussianfilterofstandarddeviation
0.8(s*s-1)^(1/2);downsamplethesmoothedimagebyafactor s.
3. Ifz>=1,regulartransformisappliedwithoutanyanti-aliasingfilter.
ThediscreteGaussiankerneliscutat4timesitsstandarddeviation.Interpolationisdonewithsplines
oforder5.
Differenceswithauthors'implementation
Fusiello and Irsara's Matlab implementation of their algorithm, availablehere does not correspond
exactlytothetextintheirpaper.weunderlineherethedifferences,bothwiththepaperversionandtheir
implementation.
1. TheywritetheepipolarconstraintXrFXl=0,whereaswewriteit XlFXr=0.Inotherwords,the
1. TheywritetheepipolarconstraintXrFXl=0,whereaswewriteit XlFXr=0.Inotherwords,the
fundamentalmatricesaretransposeofeachother.Wefindourconventionmorelogicalas
subscriptl(left)correspondstothetermontheleftofthefundamentalmatrix.
2. TheMatlabcodecontainsfullydevelopedformulasfortheJacobianmatrix,probablygottenfrom
somesymboliccomputationprogram.Weusematrixcalculustocomputeit.Moreover,theMatlab
codehasthewrongJacobian:itcorrespondstotheJacobianmatrixofthesquareerrors.Thisis
harmlessintheirimplementationastheyspecifytousefinitedifferencesforthecomputation,not
theclosedformulas.
3. TheydecomposearotationmatrixwithEuleranglesinthisorder: RxRzRy,wepreferthemore
naturalRzRyRx.
4. Theycancelthex-rotationintherighttermofF,wedoitintheleftterm.
5. ThepaperclaimsusingLevenberg-Marquardt(LM)minimizationwith lsqnonlinMatlabfunction.
Thewaythisfunctioniscalledintheircodeusesthedefault"trustregionreflective"algorithm.On
someexamples,wefoundthisminimizationtobelesssusceptibletogetcaughtinalocal
minimimumoftheenergy.However,weimplementeditwiththebetterknownLMalgorithm,as
claimedinthepaper.ModifyingtheirMatlabcodetouseLMinlsqnonlin,wegetresultsvery
similartoours.
6. TheMatlabcodetriestheminimizationupto20timeswithrandominitializationofthefocallength
whiletheresultsarenotsatisfactory.Wetryonlyoncewithg=0.
7. TheMatlabcodehastheoptionofkeepingthecenterimagepointfixed.Thisisdoneby
translatingtheprincipalpoint.Wepreferusingonlyanx-translation(tofixabscissa)ofthe
principalpointandanx-rotation(tofixordinate).
Examples
Weshowsomeexamplesofsuccessfulrectificationbytheabovealgorithm.Inmostcases,10orfewer
iterations of the Levenberg-Marquardt algorithm are necessary. To check visually the result, you can
makeyourbrowserwindowlargeenoughtogettherectifiedimagessidebyside,thenchangeitsheight
soastocuttheimagesatsomelevel;checkthatdiscerniblefeatures(blobs,corners)movealongthat
line.
Ineachcase,thefirsttwoimagesaretheoriginalonesandthelasttwotherectifiedones.
Notice that different runs of the full pipeline with same data do not necessarily yield the exact same
resultsasthereisarandomcomponentinORSA.
Beijinglion
107correspondencesarefoundandthefinalRMSEis0.25pixelsafter6iterations.
Cachanstatue
964correspondencesarefoundandthefinalRMSEis0.13pixelsafter5iterations.
Clunychurch
110correspondencesarefoundandthefinalRMSEis0.27pixelsafter35iterations.
Carcassonne
503correspondencesarefoundandthefinalRMSEis0.19pixelsafter55iterations.
Birthdaycake
146correspondencesarefoundandthefinalRMSEis0.69pixelsafter11iterations.Thefinalerroris
higherthaninpreviousexamples,duetosomegeometricdistortionofthecamera.
Sourcecode
C++sourcecodeofthepipeline(containingSIFTpointdetectionandmatching,ORSA,rectification)is
C++sourcecodeofthepipeline(containingSIFTpointdetectionandmatching,ORSA,rectification)is
providedinacompressedtararchive: MissStereo.tar.gz.TestedunderlinuxUbuntu10.04withGNU
g++4.4.3.
Acknowledgments
ThisworkwassupportedbyCentreNationald'EtudesSpatiales(CNES)andbyAgenceNationaledela
RechercheANR-09-CORD-003(Callistoproject).Differentpartsofthesupportcodeusedinthedemo
werewrittenbyToniBuades,NicolasLimare,LionelMoisanandZhongweiTang.
feeds&twitter•sitemap•contact•privacypolicy•ISSN:2105-1232•DOI:10.5201/ipol
supportedbyCMLA,ENSCachan•DMI,UniversitatdelesIllesBalears•Fing,UniversidaddelaRepública•Gandi.net
©2009-2014,IPOLImageProcessingOnLine&theauthors