Using
Matching
Feature
Graph-Based
Parameters
andRelational
Descriptive
Olaf Hellwich and Wolfgang Faig
Abstract
A graph-basedmatching algorithm for edgefeatures was de'
veloped.It conductsa breadth-first search in two nert-neigh'
bor graphs starting at hypothetically matched feature pairs.
A match is establishedby comparing feature properties
using o rnotch /unction. Main advantageof the matching aIgorithm is its applicability to various domqins, such os
stereovision or matching images and cts data. Digitized
stereoimage pairs for the investigation of automobile acci'
dentsare used to illustrate the method. For obiectswith distinctive edges,Ba b 90 percentof the resultantfeature paits
werefound to be correct.
Introduction
The fundamentalidea of the proposedmatching method for
edgefeaturesis that any property of the edgefeaturescan be
comparedwith the heip of a match function. In this way, the
propertiescontribute to a measureof similarity of the candidate features.As no constraintshave to be fulfilled, one or
severalpropertiesthat differ for candidatefeaturepairs do
not prevent the algorithm from identifying a correct match,
as Iong as the correspondencecan be proven with a high
similarity value resulting from the match function,
This simplicity is seenas the main advantageof the approach,becauseit results in a high flexibility and a wide
range of possibleapplications.It also provides a robust
matching when noisy imagery or unusual object arrangements are encountered.
The searchstrategyfor the matching of featurepairs is
straightforward.Initially, particularly significant edgesare
graphs.The two sets
extractedfrom two edge-neighborhood
of significant edgesare comparedin ail combinations.Candidate featurepairs which acquire a high similarity value are
acceptedas initial hypothetic matches,They are taken as
start nodes for a further searchfor correspondingnodes in
the graphs,i.e,, for each hypothetical match, the generation
of a connectedcomponentof matched featurepairs is tried.
During the breadth-firstsearchin the graph, a best first
choice selectionof matching pairs is performed.
This meansthat both the relational information of the
graph as well as the similarity identified by the match function support the establishmentof correctly matched features.
Previous
Work
Mclntosh and Mutch (1988)used a match function in order
to comparestraight lines. The match function computesa
Departmentof SurveyingEngineering,University of New
Brunswick, Fredericton,N.B. E3B 5A3, Canada.
O. Hellwich is presentlyat the TechnischeUniversitdt
Miinchen, Germany.
PE&R9
weighted mean of edgeparametersimilarities' Edgeparameters comparedare, for instance,Iength, direction, and contrast.Mclntosh and Mutch's (1988)approachwas extended
towards a comparisonof relational edgeparameters.
Ayacheand Faverjon(1987)proposeda methodof prediction and propagationof hypothesesapplied to a graphbaseddescriptionof images.The imagesare first reduced to
are
neighborhoodgraphs of edge segments.Linear seg-ments
edgeelementsin
generatedby a-polygonalapproximation-of
ihe images.-Theneighborhoodrelations betweenthe edgesegmentsare establishedwith the help of a bucketing techni[ue. An imageis subdivided into rectangul"r grid meshes,
Al[ edgeswhich belong partially or completelyto the same
grid mish are consideredneighbors.The constraintsused by
the stereomatching algorithm are epipolar constraint,geometric similarity constraint,continuity constraint,and
uniouenessconstraint.
A graph-basedmatching approachmore elaboratethan
Ayache and Faverjon'stechniquewas Presentedby H^oraud
and Skordas(1989).An image is describedin form of a relational graph, In the matching processnot only the features
and theirittributes are evaluated,but also the relationships
betweennearby featuresare considered.In order to find a
mapping function betweenthe imageswhich considersrelatioris beiween featuresin the Ieft and in the right image,a
correspondencegraph is built. The stereocorrespondence
oroblem is thereforeformuiated in form of a searchfor maximum cliques in a graph. Horaud and Skordasperform an exhaustive as well as a simplified heuristic searchin the
correspondencegraph.
Features
ofEdge
Comparison
For the formulation of a matching algorithm for edgefeatures, some definitions are made. An edgeis defined as a
one-dimensionaldiscontinuityJike irregularity in the brightness function that has a comparativelyhomogeneousbrighiness and steepness.When averagebrightness,contrast,and
steepnessof such an irregularity changesconsiderably,this
confinuation of the irregularity is regardedto be another
edge.A chain of pixels along an edgeis called an edge
streak.
Corner points, i.e., discontinuitiesin the direction of an
edgestreaksubdivide edgestreaksinto edgesegments,The
taJk to be solved here is the generationof matchesof edge
segmentsfrom two correspondingedgeneighborhoodgraphs
PhotogrammetricEngineering& RemoteSensing,
Vol. 60, No.4, April 1994,pp. 443-450.
oO99_7t72I 94/6004_443g03.00/0
@1994American Societyfor Photogrammetry
and RemoteSensing
that describeneighborhoodrelations betweenthe edgesegments.
Matchesare establishedby comparingvarious candidate
edgesegmentpairs. A correct match of two edgesegmentsis
assumedwhen the similarity of the edgesfor both segments
is (1) higher than for other candidatesegmentcombinations
(i.e.,the edgepair is a mutually best match) and (2) higher
than a threshold.The similarity of a candidatepair is derived from parametersdescribingthe edgesegmentsand the
relati.onsbetweenneighboringedgesegments.The set of parameterssupplied to the match function is easily adjustable
to specific matching tasks.
where ts, to are the directions of the left and right segment in
radhoo.
The value 6 is interpretedas the maximum allowable
differencebetweenthe directions compared.If the magnitude
of the differencelf, - fnl is larger than 6, then the similirity
value v, will be negative.6 is computed accordingto
6 = 6-*'
50
min(50, IL + lR)
t4)
where 6-"* is set to B0 rad/100,and ,[, /p are the lengths of
the left and the right segments.This meansthat 6 is generally B0 rad/100;a value that has been determinedempirically. 6 is increasedwhen the sum of the segmentlengths is
Iessthan 50 pixels, as directions of short segmentsagiee
considerablyless than directions of longer segments.
For the propagationstep, i.e., the generationof connected componentsusing the graph description,for both the
left and the right edgesegmentsI and ^R,'parent" nodes in
the edge-neighborhoodgraphs are known. this provides the
possibility to compa-rerelational parameters.The similarity
valge of two relations rr, rn betweenparent and neighbor
nodesis computed accordingto
\
lt,L - t,nl * * umin(d',,d.n)
,-' : ! (6
* ( d ' . d , J + u* +.P +- s /
(5)
5\
6
EdgeParameters
The parametersused in the implementedstereomatching
method can be subdivided into descriptiveand relational parameters.
Descriptiveparametersdescribethe propertiesof an edge
segmentand, hence,make them distinctive from other segments. The descriptiveparametersare derived by an analysis
of edge-supportregionswhich include the edgesegments
and their vicinity. The parametersused in the implemented
version of the matching algorithm are maximum and minimum brightness,contrast,width, steepness,averagebrightnessin the edge-supportregion, length, direction, and
curvature,
Relational parametersgive information about the neighborhood relationsbetweenan edgeand adjacentedges.They
have either topologic or geometriccontent. The relational parametersused in the implementedversion of the algorithm
are closestdistancebetweenthe edgesegments,direction to
the neighboringedge,perpendicularity,parallelity, collinearity, and the side on which the neighboringedgeis situated.
where f." : direction between related edge segmentsin the
Ieft graph,
f"n : direction betweenrelated edgesegmentsin the
right graph,
d,r = distancebetweenrelated edgesegmentsin the
Ieft graph,
d"n : distancebetweenrelated edgesegmentsin the
right graph,
6 : computed value accordingto Equation 4,
TheMatchFunction
The match function for the comparisonof candidatesfor
matching evaluatesthe edgeparimeters (Mclntosh and
Mutch,19BB):
2 @,',)
D -
\l,,,
(1)
PI
100,if rr, ro e P
)
100,if rr, r, C N
I
100, if b", rn 4P A rlro Slrl)
l,
0, if ((4 € Pn rn € M V (r" € NA r, e P)) |
50 otherwise
)
t
where v, is the similarity value for a parameter pair, and w,
is the weight of parametert'. The result of the match function
is a similarity value s of the featurescompared.The weights
w, are defined according to the image or object type.
The similarity values v, of the parametersare computed
accordingto the nature of the specific parameters.Generally,
the computationis conductedas follows (afterMclntosh and
Mutch, 19BB):
.. _ min(lp,/,lponl)
"': -*(ffiffi
444
S
[ 1 0 0 ,i f r r , r n € I ]
: J 1 0 0 ,i f r " , r o E R l ,
(0 otherwise
)
Here P : {rlr is a relation of parallel edgesegments},
,y : {rlr is a relation of perpendicularedgesegments},
.L : {rlr is a relation betweena parent node and child
node on its left side), and
A : {rlr is a relation between a parent node and child
node on its right side).
(For parent and child nodes in the propagationtree, see
section on the PropagationAlgorithm,)
Thus, the similaritv of two relations dependson the direction and the distanc-ebetweenthe related segments,edgestreakmembership,and topology.
The parameterscomparedby the match function are determined by the application domain stereovision. For other
domains,such as matching of digitized maps, the parameters
(2)
where prz and pe,,are the instantiations of the compared parameters,e.9.,the lengths of segment.T
in the left imageL and
segmentk in the right imageR.
For some parameters,Equation 2 is not suitable,
The similarity value for segmentdirections f", f, is computed as (Mclntoshand Mutch, 19BB):
6-lfr-tRl
u, = ---;-
and
(3)
PE&RS
Figure1. Generalshapeof the weightfunctionof a
Darameler,
p and, if necessary,the parametersimilarity functions vo(p)
have to be adaptedto the specific problems,Imagegeometry
can also be taken into account as a parameterof the match
function. It would, however,be impbssibleto rigorously observegeometricalconstraintssuch as epipolar geometry,as
this would contradict the basic principle not to enforceany
constraints.The geometryparametercould only be evaluated
using a moderatelyhigh weight ry that doesnot prevent
other important parametersfrom influencing the match function value.
TheMatching
Procedure
The matching processconsistsof three steps:(1) the prediction of hypotheses,(2) the propagationof the predicted hypothesesbasedon the edgeneighborhoodgraph, and (3) the
solution of conflicts betweenconcumentconnectedcomponents. Ayache and Faverjon(19S7)used a similar algorithm.
ThePredictionol Hypotheses
In the prediction step a set of particularly obvious matchesis
attempted.First, sets of comparativelydistinctive edgesegments are extractedfrom the edgeneighborhoodgraphs
using a selectionfunction that weights the segmentparameters. Then, the match function (Equation1) is applied to the
setsof distinctive edgesexhaustively.
The selectionfunction derivesa measureof distinctiveness-dfor each edgesegment.It computesthe averagesof
weight function values for specifiedparameters:
)_
2 *,@)
(6)
where p, : argumentof parameteri, w,0 : weight function
for parameterspr, and n = number of parametersevaluated.
A weight function w,,is given in digitized form as a set
of .x-valuesdefining a standardfunction curve. The general
shapeof the graph of a weight function is shown in Figure 1..
The x-valuesof the points 1 to 6 are defined for each parameter. Figure 2 gives an examplefor the derivation of a weight
function value for the specific parameterlength (in pixels).
The graph indicatesthat lines of a length /< 10 pixels have a
weight of 0. Lines of a length 100<1<200have a weight of 1,
This implicitly means that theselines are regardedas most
significant for matching purposes.Lines with a length /> 300
have a weight of 0. This is taking into account that very long
lines probably are split up in the other image and, therefore,
cannotbe matchedreliably.For lines with 10</<100 and
2Oo <l< 300, a linear interpolationis conducted;e.g.,a line
with,l : 55 has a weight of 0.5.
When the measuresof distinctivenessd have been derived for all edgesegments,a histogramfor d-valuesis computed, With a known percentagep of edgesegmentsto be
selected,a threshold value d, is derived from the histogram.
All edgesegmentsi with d > d, are elementsof the set of
selectedsignificant segmentsS. This evaluationis conducted
for both edgeneighborhoodgraphs.
Then all possiblecombinationsof significant edgesegments (sr",s,r) from the left and right edgeneighborhood
graphs are comparedusing the match function (Equation1);
i.e., the crossproduct S, x S^ is computed.The set of mutually best matchesM, i.e.,
M: {(r,,,,*,1fr[i;,:,:]
: }"#[i;:il,# ?)
where m0 : match function, and n", nn = n[mber of edge
segmentsin the left and right edgeneighbourhoodgraphs,is
consideredfor further processing,As can be seenin Equation
7, "mutually" best means that both the assignmentof the
best matching right segmentto the left segmentas weii as the
assignmentof the best matching left segmentto the right segment result in the samepair of edgesegments.When the
similarity measurem(sr.,s,p)is greaterthan a threshold, the
match is acceptedas an initial match of a hypothesis,i.e., of
a connectedcomponentto be generated,
In the prediction step, the parameterscomparedby the
match function are all the descriptiveedgeparametersthat
have been assigneda weight factor w,,
ThePropagation
Algorithm
In the propagationstep, connectedcomponentsor cliques are
generatedby searchingthe edgeneighborhoodgraphs for further potential matches,starting with the initial hypothetic
matches.The neighboringedgesof the initial match (p, pr)
are comparedin all combinations.This meansthat they are
assignmentsamong the crossproduct of neighboringedges,
i.e.,elementsof
{m(sr, sp)ls,e N"n sn € NaA (s",sr) € ru, x iV,r}
where
Figwe2. Weightfunctionof the parameter
length.
1r':{sls is neighbor of p}
Two classesfor the acceptanceof a detectedmatch are
defined as follows:
(! Matched edgesegmentpairs which
(A) are elementsof the set of mutually best matches
It4'(accordingto Equation 7) or matcheswhere one
segments, is matched to a segments, which is
'I
4
-
,'rl'
I
t
.I
]:t
right image
left image
graphs.The most distinctiveedgesare 3 and E. An
Figure3. Two edge-neighborhood
initialmatch of a hypothesiscould be the match (3, E).
s1belongingto the edgestreak
matchedto a segment
of the set
to which s; alsobelongs,i,e.,elements
fu : {(t,", s,') suchthat
tfi[:;,
i;l: xillill::;,;;l#l
t;[3;:
i;]: mH8;'j;,ldl
si1ii1 e edge streak kl V
(B)
sin,ein € edgestreaki])
and which
(B) have a match function value greaterthan a
threshold t,:
m(sr, s") > t,
Thesematchesare regardedas valid matchesof a
connectedcomPonent.
(II) matcheswhich fufill condition A as before and
(B) have a match function value greaterthan a
threshold fr, and less than the threshold t':
t. > m(s1,sn) > t,
Thesematchesare not consideredbeing membersof
the set of connectedcomponents,but they are used
for further propagationof matchesin order to find
more matchesof classI.
The searchalgorithm for matchesbetweenthe edge
graphs is now explained.
neighborhood
-The
propagationstrategyis basedon a breadth-first
searchin-a treJ. The searchstartsby comparingneighborsof
the segmentsof an initial hypothetical matglr' Thus, the initial mitch is the root node of a tree. Its child nodes are the
matchesof classesI and II among the neighborsof the initial
segments.As a breadth-firstsearchis conductedin the
griph, the first generationchild nodes are establishedin diieci seque.tce.fhen, the secondgenerationchild nodes are
searchedby comparingall neighborsof a firstgeneration
match. This means that the breadth-firsttree degeneratesto a
Iinear list with generationindices,
In every generation,all matchesof classesI and II become further tree nodes of the next generation,if they are
446
not alreadypart of the tree as a node of an earlier or of the
samegeneiation.The propagationis ended as soon as no further trie nodes can be geneiated,or if there are more than
three generationsof classII nodes in sequence.
Ffuure 3 shows two neighborhood graphs to be matched,
Figure 4 gives an examplefor the generationof the propagatio-ntree, and Table 1 shows the classI and classII matches
establishedduring the propagation.
During the build-up of the propagationtree, the potential
matchesofclass I are cbnsideredto belong to a connected
component.Ambiguous assignmentsare solved at the end of
the processingof i hypothesis.If multiple matchesoccur between different edge streaks,the match with the greater similarity value is preferred.
When a rnatch which is found during the propagationof
a hypothesisis the initial match of another hypothesis,the
other hypothesisis discarded.This savescomputation time
as the iecond hypothesis would result in a connected com-ponent similar or even equal to the one generated.
According to the details and featuresdescribedbefore,
shown in
the propagationalgorithm was implemented-a,s
FiguleJs ind 6. The procedure Match-Ileighbors-of does
O
clurrlnoo*
ffi
ctasslnoaes
tree. Eventhoughthe
Figure4. The propagation
edgesegments2 and B do not seemto match
verywell (seedifferencein direction),match
(7,K)wasfound.
PE&RS
l
TreLE 1,
parent
node
(3,E)
(2,8)
MArcHEs
rnov HyporxesrsPRopAGAloN.
Mercr.r(6, G) Wns
Ovenwnmeru
By MArcH(6, l).
classI matches
classII matches
(6,C)(1,A) (5,F)(4,D)
{7,K)
(6,I)
(2,8)
discarded
matches
(6,G)
the comparisonof neighboringnodes of an established
match. The procedureMake-Tree--IrJode
adds a node to the
propagationtree, The procedureMake-Match adds a match
to the connectedcomponent.During the tree generation,several entries per edge segment can occur. Afterwards, the
main procedurediscardsthe ambiguousmatchesin the connected component.
Solutionof GonflictsBetweenDifferentConnectedComponents
After all hypothesesare processed,the resulting connected
componentshave to be combined.Generally,this contributes
to coveringthe areaof image overlapwith matched segments. However, conflicts between concuning hypotheses
arise as well. The decision as to which of the conflicting hypothesesis preferredis basedon the strengthof prediction of
a hypothesis.The connectedcomponentwith the largest
number of matched segmentpairs is consideredto be the
correct one.
When two matches are conflicting, the match belonging
to the strongerhypothesisis always preferred.Then the
strengthof the weaker hypothesisis reducedby one. During
the processof solving contradictions,only the original
strengthsof prediction are compared.This avoids that the order in which the hypothesesare evaluatedfor ambiguous
matchesinfluences the result of the process.
After the solution of conflicts beiween concurring connected components,the remaining strengthsof prediction are
comparedwith a threshold. The threshold should be set according to the image type and the number as well as reliability of the matchesneeded.Hypothesesweaker than the
thresholdingstrengthare discardedcompletely,as those hypothesesoften are not basedon correct matches(Ayacheand
Faver;'on,
1987).
for all hypotheses{
treat initial march( pL, pR) as root node;
while tree not completely processed{
get tree node (tZ, ,R);
Match_Nejghbours_of (il" rx)
I
ior both edge neighbourhoodgraphs {
for all segmens {
- discard ambiguousmatches;
\
I
)Figure5. Propagation
algorithm.
ProcedureMatch_Neighboun_of (pz, pn) {
compute matc-hfuriction va-iues'ioi'Mr i ,rrn;
for all left neighbours{
if mutuallymatched{
if ( m(sr, sI) > r.r) {
if (st, si)€tree, Make_Tree_Node;
Make Match:
checkTor not yet processedhypothesis;
) else if (m(nr,, nR)> q {
if (st, sn)fitree, Make_Tree_Node;
\
) el6e if sr is matchedto a right segment sn which is
matchedto anothersegment lI of the edgestreakof sf {
if ( rn(s,, sn) > tl) i
if (st, sR)fftrec, Make_Tree_Node;
Make Match:
yet processedhypothesis;
check-fornot
f
o
) elSeif sn is matchedto a left segment sl which is
matchedto anothersegment qR of the edgestreakof sn{
if ( m(o, sR)> u) T
if (i{,, sn)etree, Make_Tree_Node;
Make Match:
not yet processedhypothesis;
"n*o*t
,
,
))
Figure6, Algorithmfor the procedureMatch-Neighborsof.
Experimental
Results
The developedfeature-basedmatching method was implemented on Sun workstationsas part of a stereovision software packageusing the X-Window user interface (Hellwich,
1992).The program systemwas applied to imagery taken for
car accidentinvestigation.The final goal of the digital stereo
measurementswas the determinationof the depths of dents
in the car bodies.
Imagesof cars are characterizedby well-determined
curved edgesof various shapesas well as by specularreflections in someparts. AII imageswere taken outdoorsunder
common conditions for close-rangephotogrammetry.No special measureswere observed.The imageswere taken with a
24- by 36-mm cameraon color slide film. They were digitized with an Apple Mclntosh slide scannerusing the DeskScan softwarefor the FrameGrabbercard. Then a reduction
from color to greyscalewas conducted,The imageshave a
resolution of about 600 bv 400 pixels.
The first picture shows the^imagepair of a grey 1987
Skoda (Figure 7). The damageddoor of the car is used as the
PE&RS
object of demonstration.The matchesgeneratedare displayed in Figure B. The match numbers enablethe alert eye
to identify matched edgesegments.The unlabeiededgesegments are unmatched.Another imagepair is presentedin
Figure 9 while Figure 10 shows the resulting matcheson the
object of interest,
Intensivetests of the matching method resulted in about
B0 to 90 percent correct matchesamong the matchesfound
on the objectsof interest.
Figure 11, an image pair of an oil tank, demonstratesthe
limitations of the match function in the proposedform. The
welding seamsare well-defined edges.In spite of that, only a
few correct matchesare found, as the edgesare not sufficiently distinctive. An evaluationof disparity gradientsin
the match function would solve the problem. Disparity gradients can be obtained from the disparitiesof parent and
child nodes in the propagationtree.
Figure7. lmagepair of a grey1987 Skoda.
Figure8. Matchedsegmentssuperimposed
to the Gaussianconvolvedimage.
figure 9. lmage pair of a white MercuryGrandMarquis.
448
PE&RS
Figure10. Matchedsegmentsof the white MercuryGrandMarquisimagepair.
Figure11. Matchedsegmentsof an imagepair of an oil tank.
Conclusions
A matching algorithm was developed using a graph structure
to control the search for matches and to exploit relational
properties for the identification of matches.
The main advantages of the matching algorithm are
e its applicability to various domains,such as stereovision or
the matching of images and cIs data;
o its independencefrom epipolar geometrywhich makesthe
algorithm applicable to images taken with different imaging
devices:and
o the availability of reliability information in form of the simi
larity values of establishedmatchesand the numbers of
matches(strength)of a connectedcomponent.
Experimental results indicated difficulties of the matching
method for images of objects with well-defined but indistincPE&RS
tive edges.The problem is expectedto be solved by
uation of disparity gradientsin the match function.
References
AyacheN., and B. Faverjon,1987.Efficient Registrationof Stereo
Imagesby Matching Graph Descriptionsof EdgeSegments,International Jounal of ComputerVision,pp. 107-737.
Hellwich, O., 1992. Graph-BasedEdgeMatching in Stereo Vision,
M.Sc.E.Thesis,Dept. of SurveyingEngineering,Univ. of New
Brunswick, Fredericton,113 p.
Through
Horaud, R., and T. Skordas,1989. StereoCorrespondence
Feature Grouping and Maximal Cliques, IEEE Transactionson
PatternAnalysisand Machine Intelligence,Vol. 11, No. 11, pp.
116 8 - 11 8 0 .
Mclntosh, J. H., and K. M. Mutch, 1988.Matching StraightLines,
Computer Vision Gmphics and Image Processing,Vol. 43, No. 3,
pp. 386-408.
(Received7 August 1992; accepted15 December1992;revised26
/anuary 1993)
Meeting Announcement
ISPRS CommissionII
Systemsfor Data Processing,Analysis, and Presentation
held in conjunction with
The 6th Canadian Conferenceon GIS
June 6-10, 1994
0ttawa, Ontario
ISPRSCommissionII, 'Systemsfor DataProcessing,Analysis,andPresentation"
will offer a largeexhibitarea,
aswell as a strongtechnicalprogramof paperandpostersessions,The Technicalprogramwill providea vastopportunity
to showcasethe leadingtechnologiesand the latestresearchprojectsfrom around the globe. This Symposium,since
accompanied
by the 6th CanadianConferenceon GIS, will not only providean informativeprogramon Photogrammetry
and RemoteSensing,but alsoon GIS, management,
education,and research,as well as the latestin systemstechnology.
Delegatesto eitherthe ISPRSCommissionII Symposiumor the 6th CanadianConferenceon GIS will be free to
exploreand accesssessionsof the other.
As a specialguestspeaker,Her RoyalHighnessPrincessMahaChakriSirindhornof theRoyalFamily of Thailand
will be openingthe Symposium.The Princessholds a greatinterestin GIS and remotesensingand their environmental
applications.
An AccompanyingPersonsProgramas been designedto explore the National Capital Area, its world class
museuns,and monumentsof nationalsignificance.
For more information, please contact:
Dr. Mosaad Allam
Chairman
ISPRS CommissionSymposium
615 Booth Street, Room 700
Ottawa, Ontario
CanadaKIA 089
450