TheoryandMethodsfor Accuracy
MaPsUsing
of Thematic
Assessment
FuzzySets
Sucharita Gopal and Gurtis Woodcock
Abstract
The use of fuzzy setsin map occuracy ossessmenfexpands
the amount of information that can be prouided regarding
the nature,frequency, magnitude, and soutce af errorsin -a
thematic map. The need for using fuzzy sets arisesfrom the
observation fhat all map locations do not fit unambiguously
in a single map category.Fuzzy setsallow for varying levels
of set memberchipfor multiple map categories.A linguisticmeasurementscale allows the kinds of comments commonly
made during map evaluationsto be used to quantify map
accuracy. Four tables resultfrom the use offuzzy functions,
and when taken togetherthey provide more infotmation
than traditional confusion matrices. The use of a hypothetical dataset helps illustrate the benefitsof the new methods'
It is hoped that the enhanced ability to evaluate maps_rcsulting-from the use of fuzzy sets will imprcve our understandingof uncertainty in maps and facilitate improved
error modeling.
TheAssessment
of MapAccuracy
Introduction:
Thematic (categorical)maps are made for a specific purpose
and portray information using some system of classification
for the landscape (soil taxonomy, vegetation classes,land
use, etc.).They are increasinglybeing used by a variety of
people involved in the managementof land and resources.
The increase in their use is largely due to remote sensing
and geographic information systems (cls). Remote sensing is
now used widety to produce thematic maps. cls technology
both employs thematic maps as input data sources and creates new thematic maps utilizing the analysis of existing
maps
- and data.
The purpose of this paper is to present new methods for
assessingthe accuracyof thematic maps basedon fuzzy sets.
The intent of the new methods is to allow explicitly for the
possibility of ambiguity regarding the appropriate map label
It any loiation. In addition, expanded possibilities for error
analysisresult from the use of fuzzy sets.The focus of this
paper is on the theory behind the approanhand on a thorough desuiption of the methods,In another paper,the-se
methods are applied to assessthe accuracyof a map of forest
vegetation, which helps illustrate the utility of the proposed
approach(Woodcockand Gopal, tssz).
Background
The difficulties associatedwith assessingthe accuracyof
thematic maps are the result of the nature of thematic maps.
In thematic maps, each location on the ground has to be assignedto a category(or class).In essence,the continuum of
variation found in the landscape has to be divided into a fi-
Departmentof Geography,Boston University, Boston,
02275.
PE&RS
nite set of categories.Typically, the-categoriesare easily differentiablein their pure states,and becomeless readily
separablenear the dividing lines betweenthe categories.For
exlmple, considerthe differencebetweenthe vegetationcategoriesconifer forest and hatdwood forest. At their extremes
t[ere is no question regarding the appropria-tecategory.However, all degieesof mifing of coniferousand hardwood trees
mav be found. When coniferoustreesdominate,the appropriite label may still be coniferousprest, but what happens
is the mix appioaches50 percent of each?At that point the
decision becbmesarbitrary and neither categoryis either entirely right or entirely wrong. One solution is to add another
category to the map that is mrxed /orest. This new category
solvesihe problem in one case(the 50-50 mix), but now
there is a problem defining the break between mixed forest
and both hardwood forest ar'd conifer forest'
One can argueihat theseproblemsof ambiguity concerning the appropriate map label for a given location can be
solvedby i piecise setbf specificationsfor the definitions of
each of the map categories,and for these categoriesto cover
all possiblesituationi found on the ground, and for the categoriesto be mutually exclusive.In fact, thesekinds of definifions for map categorieswould minimize the ambiguity..
Most mapping proiects,however,are not conductedwith an
extensive ivtie* of breakpoints defining the differences between categories,Accurati and precisemeasurementsof the
required pirameters on the ground are often either difficult
or-imposiible.Take, for example,the differencebetweena
meaiow category and a grass/ond category,which may be primarilv a function of annual water balanceand may not be
ineasurable at a single visit to a site, or even from measurements from a singleyear. Beyond the problem of category
definitions and measurementdifficulties, map categoriesare
frequentlv constructsthat do not Iend themselvesto physical
Examplesmight be maps of land or residential
meisu.erire.,.t,
cost or habitat suitability. There are often descriptionsfor
thesecategories,but rarely definitions that can be used to
make unairbiguous decisionsfor individual sites.The problem that makJs accuracyassessmentdifficult is that there is
ambiguity regardingthe appropriatemap Iabel for some locations, thi siiuation of one categorybeing exactly right and
all other categoriesbeing equally and exactly wrong often
doesnot exist.
The representationof map categoriesa^lsodependson
scale,as siale generallyimplies a degreeof spatial generali-
PhotogrammetricEngineering& RemoteSensing,
V o l , 6 0 , N o . 2 , F e b r u a r y7 9 9 4 ,p p . 1 8 1 - 1 8 8 .
B1$03.00/0
0099-1112/94/6002-1
@1994American Societyfor Photogrammetry
and RemoteSensing
181
zation and a minimum mapping unit (Woodcockand Harward, 1992).The minimum mapping unit is defined as the
size of the smallestobjectto be representedon the map. If an
object is below the minimum mapping unit, then it is
mergedwith its surroundings.However,if the size of the
minimu-mmapping unit changes,the object may be representedon the mao.
There are two primary motivations for assessingthe acc-uracyof a map. The first one concerns trying to understand
the errors in the map. Both producersandusers of thematic
maps are interested in this kind of information. producers
can improve methods of making the maps and of presenting
the information on accuracv and errors to the end user. In-formation about the errors in the map, in turn, can help map
use_rsto interpret and use the map more effectively. Tlre second motivation for assessingthe accuracyof mapi is to provide an overall assessmentthat can be used as an indication
of the generalreliability of a map. For example,one use of
such an overall assessmentwould be to comparetwo maps
in order to determinewhich is better.
The four kinds of information that are desired from an
accuracyassessmentabout the errors in a map are their noture,frequency, magnitude,and source.Thenature of the errors concernswhich categoriesare confusedin the map,
meaningbetweenwhich categoriesare there mismatchbsbetween tlre category assignedin the map and the reality at the
ground location. In the simple casediscussedprevioully, are
there situationswhere conifer treeshave beenissigned map
Iabels of hardwood foresf, and vice versa? FrcquenCyrefers
to how often thesemismatchedsituationsbetween-themao
and the ground occur. A more subtle concept concernsthe
magnitude of errors. At one level, certain kinds of mismatchescan be consideredgreaterin error than others.For
example, assigning a map label of. water to an area of coniferous trees-might be considereda larger emor than assigningit
a map label of hardwood foresf. At a second level, assigning
an-areaof 100 percent coniferoustreesa hardwood forestIabel might be a more seriouserror than assigningthe samelabel to an areathat is 60 percent coniferoustreei and 40
percent hardwood trees. The last kind of information desired
is the sourceof errors,which is less well defined and prim-arilyof concern to the people who make maps. Essentially,
what is desired is any information about the cbnditions in
which errors tend to be made that can help reduce the likeli
hood of the samekinds of errorsbeing made in the future.
Traditional
Methodsof Assessing
Accuracy
Traditionally, the accuracyof a thematic map is determined
empirically by comparing the map with corresponding reference or ground data. The results are tabulatedin the form of
a squarematrix whose columns usually representthe ground
data (i.e.,assumedcorrect)and rows indicite the mapped
data. Each elementin the matrix gives the number of areas
on the map assignedto a particular categoryrelative to the
ground data. The elementsof the principal diagonalof the
matrix representthe correctmatchesand the remaining elements, mismatches, This matrix is popularly known as an e.rror or confusion matrix (Card, 1982) and forms the basis for
a seriesof descriptiveand analSrticalstatisticaltechniques
(Congalton,1991; Hoffer and Fleming, 7978;Rosenfieldand
Fitzpatrick-Lins,1986).
The ideal situation in map accuracyis representedby a
diagonalmatrix where only principal diagonalelementshave
non-zerovalues;all areason the map have been conectly
classified(van Genderenand Lock, iszz;Mead and Szajgin,
ta2
1982;Congallonet al,1983). This situationis rarelv the case.
A distinction is often made betweenproducer'sand user's
accuracy.Produc-er's
accuracyindicaies the probability of a
test or referencelocation known to belong to category-Cis
accuratelylabeledas categoryC. It is a miasure oI omjssjon
error. This ac-curacymeasureis obtained by dividing the total number of correct entries in a categoryii.e., diag6naleleTelt) !I the total number of entries of that categoryas
derived from the referencedata, i.e., the column-total.On the
other hand, user's accuracyindicatesthe probability that a
test or referencelocation labeledas categoryC belongsto
categoryC (Story and Congalton,1986).This is a measureof
commissionerror. It is estimated by dividing the total number of correct entries in a category by the tolal number of entries that were classified in that category,i.e., the row total.
- One advantageof thesemethods is that they yield a single overall map accuracyindex, usually presentLdas a percent correct.This measurecan simplv be the sum of the
major diagonaldivided by the total-nirmberof samples,or
the accuraciesof each categoryin the map can be weighted
by its areain the map and thesevalues summed (Card]
1982).Similarly, a statistic, kappa, can be calculatedthat
provides a measureof differencebetweenthe observedagreement betweentwo maps and agreementthat is contributed
by chance (Congaltonind Mea-d,1983;Rosenfieldand Fitzpatrick-Lin, 1986; Rosenfield,1981).Conditional kappa
coefficientscan also be calculatedfor individual map-iategories.For those interestedreaders,Congaltonhas recently
provided a thorough review of methods for assessingaccuracy of thematic maps and the relevant current reseirch
(Congalton,1991).
The traditional methods of assessingaccuracy,outlined
above,suffer from a number of limitations:
(1) It is assumed
that eachareain the mapcanbe unambieuously-assigned
to a singlemapcategory.
In assessing
gr:ound
truth for mapaccuracy,
the experthasto resolvethislssue
by selecting
a singlecategory
for eachgroundlocationand
matchingthis againstthe mapvalue.
(2) Informationon the magnitudeof errorsis limited to noting
thepatternof mismatches
betweencategories
in themap.
Dataconcerning
the magnitude
or seriousness
of thesemismatchesas indicatedby the conditionsof the groundsite
cannotbe used.
(3) Third,the userneedsto be providedwith morecomplete
and interpretableinformationaboutthe mapthan is currentlypracticed(Aronoff,1982a;19S2b).
Detailedinformation on errorswill help the usersto checkif the map can be
used for a particular purpose.
The methodspresentedin this paper provide information about thematic maps and the eirois in those maps pertinent to the three issuesoutline above,The intention is io
improve the utility of thematic maps through a better understanding of their enors.
Fuzzy
Sets
This sectiol provides someessentialbackgroundon fuzzy
sets and a descriptionof the use of fuzzy sets in the contlxt
ot map accuracyassessment.
The concept of a fuzzv set was
introduc_edby Zadeh (7s6s, tg73) to deicribe imprecision
that is characteristicof much of human reasoning,particuIarly in domains such as pattern recognition,communication
of information, and abstraction.Zadeh (f soS; 1973) and others (Duboisand Prade,1980;Kaufmann,1975;Goguen,1969J
have outlined quantitative techniques for dealing ivith
vaguenessin complex systems.
PE&RS
Formally, afizzy set can be defined as follows:
Let X be a universe of points (or objects)with a generic
element of X being denoted by x. A fuzzy setof X, labeled A,
is characterizedby a membershipor charactetisticfunction,
p,^,which associateswith each point in X a real number in
the closedinterval (0,1).The value of pa@ at x represents
the grade of membership of x in A. This can be designatedas
6:{(x,p.6@\l"eE.
Note that the nearerthe value of p^(x) to 1.0, the higher
the grade of membershiP of x in A'
underlyingfuzzy set theory is that the
ih"
from membershipto nonmembershipis seldom a
transition"ttntttption
step function, Rather,there is a gradualbut specifiable
chinee from membershipto nonmembership.In (classical)
set tlieory, a membershipfunction, p^(x) has only two values
0, 1. In particular,
,^,,: {l ii:Etr
Fuzzv setshave been used in remote sensingfor image
interpretition and image classification.While interpretingan
image,expertse*ptesslhei. assessmentusing.qualitative lineuisiic.raiues.Fulzv setsin this context can be used to
irathematize these linguistic values and to obtain a consensus if the experts pton-idedifferent (linguistic) values. Thus,
thev provide a consistentway to measureand model qualitative vilues that is useful for subsequentdecision-making
(Hadiprionoef o1.,1991).Current m-ethodsof classification
emplovedin remote sensinggenerallyassumethat each pixel
belbnjs to one classonly. A ilassification algorithm basedon
fuzzv"representation(such as Fuzzy C-Meansclassifier) has
been sh6wn to be useful in extracting more information from
remotely sensedimages than conventional -algorithms(Trivedi and Bezdek,1986; Wang, 1990) as well as in generating
"hard classification"(Kent and Mardia, 19BB)'The amount
of land cover within a pixel is reflected in the fuzzy membership value derived from the fuzzy classifier(Fisherand
Pathirana,1990).
Fuzzy set theory can also be used in cts applications'.
More spelifically, ii can be used to representuncertainty in
spatial database!and manipulate uncertainty as data are
transformed by various GISfunctions (Robinson and Strahler,
1984; Leung, igag; Robinson 1988).For example,arnap
overlay funltion results in the creation of a composite map
that contains uncertainty and errors' Using fuzzy sets,it is
possibleto compute the uncertainty in the compositemap if
the membershipvalues of sites in each cover classwere
known for each map layer (Veregin,1989)' Fuzzy sets-can
also be used to estimate similarity and other relationships'
Fuzzy relational databases,like FRsIS(Kollias and Voliotis,
1991j, can handle imprecision in data representationand
manioulation and allow for individualization of data' There
other examplesfrom land survey (Wang et al., 1990),
"r.
"iro
(Burrough,19BB),and-representationof
analysis
soil
viewshedsderived from a DEM(Fisher' 1991; 1992)'
Assessment
FuzzySetsin the Contextof MapAccuracy
We define some terms and notations in order to describe
how fuzzy sets can be used to analyze the-a-ccuracyof th9-.
matic maps. Let X be the finite universeof discourse,which
in the present context is a set of sites (or polygons) represented in the map. Let C denote the (finite) set of c/asses
PE&RS
of
assigned
'=m.
-fo. to sites in X let m be the number
{or cotegories)
be
to
we
define
X,
x
e
{(x)
site
each
cateeori;slcl
the dlassalsigned to x in the map' The set
- a , 1 : { @ , y ( x )l)x e X }
the maP data'
defines
--fo. the eviluation of map accuracy'usually a subsetS C
X of n samplesites is used' A fuzzy set
Ac:{$1tclx))lxeS}
is associatedwith each classC € 4 where p6(x) is the charicieristic ot membershrpfunction of C. Membership.funcJerived from ihe linguistic values provided by the .
lirJ;;;
in
exoerts the processof evaliating map accuracy(described
beiow). Thus, ihe data set used for the fuzzy analysisis.an n
[u - itiultx of membershipfunctions, denotedas a4' The
i,Jri rGt are the rows of.o4 and the classesare the columns
of o4. Abusing the notation slightly, we will also use '46 to
denotethe co]umn of I correspondingto the class C' See
Table 1 for a hypothetical exampleusing I 40 by 4 matrix'
This table also-includestwo additional columns representing
r""rot" site numbers,x : 1,,..,,40, or S and the label assienld to the polygon in question,or x(x)' The different
.-turr"t A, B, C, ani D are-in the set 4, 1(x) denotesthe map
Iabel; and p.r(x)denotesthe expert evaluationfor categoryA
at site (x), ps(x) for a cateSoryB, and so on'
of a LinguisticScale
Construction
ifrr -ttnoaology-presented here was developedbasedon
the observatiorilttit expertsmost often use linguistic constructs to describemap accuracy.It has been the observation
of the authors over 10 years of evaluatingthe accuracyof
*"pr ttr"t qualitative,linguistic terms.are generallyused t.o
the quality of vaiious map labelsJor individual sites
;;i;"tt
when visited in tne fieta. Verbal reports of a group of experts
were collectedand anaduring map accuracyassessment
the protbcol analysistechniquesdescribedin Ertyzea"usin^g
iisson and Simon (1984).It revealedthat expertsare not
to distinguish;'absolutelyright" from "absolutely
""fr "Ui"
values but Le aho able to identify intermediatevalues
*J";';
betwJen these two extremes.At least three different intermediare clearly distinguished by all of the experts'^Based
""iuttanalysis,a five-point membershipscaleranging from .
"i"
o., tttit
;"Ur"t"ttty ilg}ti" to "a6solutelywrong" values was developed'
The linguistiiva/ues and the descriptionsused by the experts
to evaluatea maP class at a site are:
Wong: This answeris absolutelyunacceptable'
(7) Absolutely
VerYWrong.
is
(2)
'-' Unierstaniablebut Wronq:Not a-goodanswer'There
understandanswer
the
makes
lhat
site
the
about
somethi"g
is clearlya betteranswer'This answerwould
ablebut t"here
posea problernfor usersof the map'Not Right'
)r'Iaybenot the bestposAns.w.er:
or Acceptable
(S)
' heasonob/e
this answerdoesnot pose
sibleanswerbut it is acceptable;
a problemto the userif it is seenon the map'Right'.
(q) GoodAnswer:Wouldbe happyto find this answergivenon
the maP.VerYRight.
(s) Absoluielynight:No doubtaboutthe match'Perfect'
The following procedureis used to obtain the linguistic
membershipscalei.the expert'sjob is to evaluateeach landus" classaf each site and ihen choosethe most suitablelinof
e"LU" ;"f". to describehis/her perceptionof the nature
il"f"ft t"t*"en each map categoiyand the ground truth' The
r*pu.t aott not know what the map class is prior to or duri.rf the tu^luation procedure'Linguistic values obtained
r83
T e e L E 1 .A H Y P o r H E r t c A t - B AnM: 4
P0I E
;=
,m
Site
No.
7
2
3
4
5
6
6
I
10
11
1'
13
74
IJ
16
77
18
19
20
Map
Label
4 , A N D C : U , B , C , D ) , T n e C o r - u r r , t r . r s p( pe^\ )( ,dp, c ( x ) ,p o ( & )r)o n v r H E 4 0x 4 M n r n x A , T n e L E s 2
ro 5 Ans DeRveornou ne Detl PnEserureo
tNrxrs TaeLr.
Site
No.
Expert Evaluation (A)
x8)
Pa8)
ttaS)
Pc8)
ttnF)
D
1
1
A
c
4
7
1
4
L
4
1
4
1
4
1
c
B
D
A
D
A
B
c
A
B
7
1
7
5
1
5
3
1
5
c
c
D
D
I
7
1
7
a
7
J
I
2
7
1
3
2
1
7
3
3
4
1
4
L
3
2
3
4
1
7
1
5
3
2
4
3
7
4
3
1
J
c
D
B
A
B
J
4
L
4
1
4
1
3
1
1
1
2
Accuracy
Assessment
UsingFuzzy
Operators
A seriesof measureshave been developedbasedon fuzzv
functions that provide indications about the quality of a'map
(Zadeh,1965;1973;1975;Goguen,1969;
and its categories
Dubois and Prade, 1980; Kaufman, 1975). The results of
these functions are four tables which, when taken together,
give more overall information than is possible from a traditional confusion matrix. The use of these functions and the
tables they produce are presented in the context of an example using the Iinguistic scale previously described. The data
used for the example is hypothetical, but is designed to include some of the properties we have found in our initial
testsusing this approachfor map accuracyassessment
(Woodcockand Gopal, 1992).Each table is evaluatedin
terms of its contribution to understanding the nature, frequency, magnitude, and source of errors in a thematic map.
144
Expert Evaluation (.A)
x8)
ttaFJ
2r
B
22
23
24
25
26
-t1,
1
5
7
1
,a
28
29
30
3\
32
33
34
35
36
1
from the expert for each class at every site form the input
data to Table 1.
For the sake of convenience,the linguistic values are
converted into a numerical scale ranging from 1 to 5. Thus,
instead of a continuously measured membership function
ranging between the usual 0 - 1 used in fuzzy sets, the
membership scale used in the present researchis discrete
and is based on the five linguistic levels. Each membership
function A. in Table 1 is in the numerical format ranging between 1 and 5.
One of the differences between the traditional and proposedprocedureis worth noting. Becauseeach map categoryis
evaluatedat each test site, the expertscan rccognizethe heterogeneity of ground cover and ambigurty of map classesand use
it to describethe degreeof match between each map class and
ground data. For example,they might use linguistic values
such as "absolutely righf' or "reasonably right'' to describe the
nature of the match. The expert is not limited to a single match
for a site or bound to the edstence of a perfect match for each
site. On the other hand, the traditional method allows onlv for
a "yes" (match) or "no" (mismatch)condition.
Map
Label
38
39
40
D
C
D
ttaE)
ttc&)
1
3
3
4
D
n
7
J
C
D
A
1
1
J
1
4
I
7
1
1
o
R
c
A
D
B
J
1
c
B
D
A
C
J
1
1
1
c
1
2
4
I
2
1
L
1
1
ttoF)
4
1
I
7
1
3
Frequency
of MatchesandMismatches
In this subsection a procedure is described that measuresthe
accuracy of the map in terms of the frequency of matches
and mismatches between the sample map data and the expert data, i.e,, between a14 and -4. The procedure divides the
traditional question of "how accurate is the map?" into the
following two more precise questions:
. How frequentlyis the categoryassignedin the mapthe best
choicefor the site?
. How frequentlyis the categoryassignedin the map acceptable?
Define a Boolean function o that returns a result of zero or
one basedon whether x belongsto the class C with respectto
the mahix .1. This evaluation of sample site x from the set S /x
€ 5) and map category4 dependson the criteria used to define
a. That is, o [x,d) = 1 if the x "belongs" to C, and o{rC) = 0
if x doesnot "belong" to C. The definition of "belong" depends on o. We will consider two different definitions of o
(seebelow for details),If o(xX(x)) = 1, then we say that there
is a mafcft between the map data and the expert data at site &
and if a (x, X(x)) = 0, then there is a mr'smotch.
For each map category C e C, we compute two quantities
ar6 : l{x e S I x(x) : C and o(x,C) : 1yl
: C a n d o ( x , C ): 0 } 1 .
That is, we count the number of sites x with Xk) : C in
terms of matches (ar") and mismatches (66,).
We now define two functions which can be used as o.
One of them, called uAx, is defined as follows:
6c=l{x€Sl/x)
MAx(x,c) :
(t
.l;
if p6[x) > pc,[x) for all C' e C,
otherwise.
That is, vAX (x,C) is 1 if the value of the membership function for x in category C (p" (x)), is maximum among iU map
categories(p.{x)).
The other function that we define is called nrcnr. It is
PE&RS
or EnnonsUsINGMAxANDRIGHT
Nerunenno Dtsrnteuflof't
faau2.
Exnert Evaluation
nIcHr(R)
uax(M)
Map
Label
Sites
Mismatch(6c)
Match(toc)
ClassA
ClassB
ClassC
Class D
10
10
10
10
10 (100.00%)
6 ( 60.00%)
4 ( 40.00%)
6 ( 60.00%)
Total
40
26 ( 65.00%)
0 ( 0.00%)
4 (40.00%)
6 (60.oo%)
4 (4o.0o%)
14 (35.00%)
defined with respect to some prespecified threshold value c
where r is one vilue in the linguiitic scale or a specified
value in the membership function. A site x "belongs" to-a
category C if its membeiship function, pck) > t Formally,
rt p6@)'>r
nrcrrlxcr
'", :- {1 otherwise.
tO
The threshold value r in the present research equals any
degreeof right that is a value > g in the linguistic scale described before.
Table 2 is derived from the hypothetical data set shown
in Table 1, using the vax and the plcrn functions. The first
column shows the map cateSoryor label for-purposes of comparison and the second iolumn shows the total number
and misof siies under each map category.The matches (1116')
matches @c,)using the two functions are given both in
counts uttal.t perientages in columns 3 to 6 of the table. The
last column of the table shows the improvement in accuracy
associatedwith using the RIGHTfunction instead of the vax
function.
Tvpicallv the MAX function is more conservative than
function, but that is not always the case.It is posnrcirr
the
sible for a site not to have any map categoriesthat fit the
RIGHTfunction, but still have a highest score' Using our linguistic scale, this situation would occur for a site whose
scorewas 2'
highest
Traditional confusion matrices usually follow the approach of either the trlax or the RIGHTfunction' The tuax
iunction would apply when the confusion matrix comes
from blind samplei (i.e., when prior knowledge-of the map
label is not used at the time of the evaluation of the site),
and the RIGHTfunction when the map label is known during
its evaluation, Often, information about the level of knowleds,eof the map label at the time of its evaluation is not provid'ed with a confusion matrix. By explicitly separatingthe
wxtctt Sxows BorH THE
OPERAToR,
TISLE3. Resulrs oF THEDIFFERENCE
or EnnoRsAcnoss Cereeontes
Mnenruog ANDFREQUENoY
Map
Label
ClassA
ClassB
ClassC
ClassD
Total
PE&RS
Matches
Mismatches
Sites-4
10
10
10
10
40
-3
-2
-1
01
2 3 4
01018
2 7 02
02770
4 0 02
0000
7!02
0024
0013
9
6
416I
7
0
Arithmetic
Mean
3.60
o.20
- 0.10
0.10
0.95
Mismatch(6")
Match(to6)
10 (100.00%)
6 ( 60.00%)
8 ( 80.00%)
I ( 80.00%)
o ( o.oo%)
32 ( 80.00%)
I (2o.oo%)
Improvement
(R-M)
4 (4o,0o%)
2 (20.00%)
2 (20.ooo/o)
o ( o.0o%)
0 ( 0.oo%)
4 {40.0oo/o)
2 (20.00%\
6 (15.00%)
results for the MAX and RIGHTfunction, no confusion in this
is possible.
-reeard
-"
D"t" f.o- the vAx and RtGHt functions taken together
are more useful than either taken separately.In this regard,
the results in Table 2 for Class C are interesting. If just the
MAx function were used, it would indicate ClassC to be a
major problem for the map' H-owever,the results for the
mcirr tunction indicate t6at the impact on the user might
not be as severeas the uax function indicates' ClassC is
clearly a poor class,where the best answer is usually not^
p.i"ii.a, U"t an acceptableanswer is given 8-0percent o.fthe
iime. Similarly, the strengthof the-result-forClassA indiUy the tiiCHt funclion is enhanced by the knowledge
""t"a
of its results for the UAX function.
of Enors
Magnitude
Oni of the main benefitsof the methodspresentedhere
based on fuzzy sets is the ability to evaluate the magnitude
or seriousnesiof errors.In ordei to measurethe magnitude
of error, a function A S --+Z is introduced.For a given site x,
A(x) measuresthe differencebetweenthe scoreof the map-all
x(x) of x and the highest score given to x among
"ui"goty
othe"rcitegoiies of c. Formally, A(x) is defined as
A(x) :
pxt*t(x) -
max ttc@).
(1)
Cl X(xl
In the presentstudy the linguistic values rangebetween
1 and 5, to -a < A(x)'< 4. For the ideal case,where the
mapped categoryis perfectly right (score5) and all other cat(score1), the abovedefined
wroni (sdore
utely wrong
egorresare a6soiuteiv
"nori"t
irinction, called the bmnnrNce function, yields 4. All sites
that are matches using the vAx function have DIFFERENCE
valuesgreaterthan oiequal to 0 and all.mismatchesare-negative, imismatch with i orrrnnNcn value of 1 would cora scoreonly
received
label
map
respond to a casewhere the
o"6 t.tt than the highest scoregiven. Clearly,this kind of
error is not as troubfesomeas those where a 4 is found'
ihe results of the use of this function are dependenton the
manner in which values are assignedfor the membership
functions. In our example,we have used a simple linear
iunction to convert linguistic answersto valu-esin a membership function, but ther-eare many other possibilities'This
approachlends to the situation where using this function, a
difierenceof -2, produced when the mapped categoryre'
ceived a scoreof d and the maximum scoreis 5 at a site, is
equivalent to a situation in which the mapped cateSoryreceived a value of 2 and the maximum scoreis 4'
For each cateSoryC e C arrd integer -4 < i < 4, we
compute the quan-tity D; the number of sites x such that 1(x)
TeeLE4,
Ser MEMeeRsHrps
AcRossCnreoonrEsUsrNGMAX(seere<r)
MEMBERSHIP
(x)
Map
Label
ClassA
ClassB
ClassC
ClassD
Total
Sites
10
10
10
10
M
000
000
000
000
M
N
990
633
110
32r
t9
15
M
7L0
tJl
936
743
27
t7
10
: C and A(x) : i. Table 3 illustrates the results of the nr'FERENCE
function for the hypothetical test dataset of Table 1.
The computed values Dj are shown under matches and mismatches (using the vAx function) columns in Table 3. Several trends are interesting to note. First, ClassesB and D,
which have similar frequencies of errors as shown bv the
MAx function, have quite different magnitudes for their errors. Class B generally has enors of higher magnitude than
those in Class D. But Class D has a larger number of zero differences (sites where there is no difference between the map
categoryand other categories).Also, as one might expect,
Class A tends to have high positive DIFFERENCE
values, while
Class C rarely has large positive DIFFERENCE
values. As the
results from Table 3 imply, Class C has frequent negative differencevalues.
The last column shows the arithmetic mean of all scores
for each class.It can be used as an indicator of the quality of
a class.For example,the most accurateclass,ClassA, shows
the higher positive arithmetic mean. Class B and D have
lower arithmetic means. Class C has a negative arithmetic
mean, indicating the larger number of errors in this class.
The use of compositevalues for map categoriesbasedon
these DIFFERnNCE
values mav provide useful indices, It mav
even be appropriateto use a weighting schemeof someklnd
to accentuatethe effects of the negative values. One possibility might be to averageonly the negative values to produce
somethinglike an error magnitudeindex for each category.
The use of the DIFFERENCE
function is dependent on the link
betweenthe linguistic scaleused in evaluatingmap sites and
the set membershipfunction. In this paper, a simple discrete
membershipfunction is used, with unit differencesbetween
each level in the linguistic scale.Clearly,other possibilities
exist for relating linguistic scalesto set membership functions (Zadehet aI., 7975iLakofl 1973)and future researchin
this area is warranted.
Sourceol Errors
Categoricalerrors in thematic maps frequently arise due to
the heterogeneous
nature of ground composition.This aspect
of errors can be examined using a MEMBERSHIP
function ,tr: N
+ Z which measuresthe representation of multiple cover
classesat each site as evaluated by the expert. The function
selectsthose categorieswhose p" (x) is > r. Formally, it is
defined as follows:
, \ ( x ) : l { C l C e C a n d p . " ( x ) >r } 1 ,
(z)
where r is the threshold value defined in the section on Frequency of Matches and Mismatches.In the example describedin Table 1, lcl : 4, so ,\(x) rangesbetween0 and 4.
(Note that ,\(x) can equal 0 if r > p".(xlfor all C.)
The MEMBERSHr
function provides data on the frequency
186
of set membershipsat each site. One of the sbengths of.fuzzy
setsis the ability to recognizemultiple set membershipsand
gradesof set membership.The five linguistic levels given represent degreesof set membership.The marnnsrUp function
calculatesthe frequency of set membershipfor individual sites,
In this case,some degreeof right 1i.e.,values greaterthan or
equal to 3) constitutesset membership.
For each categoryC € 4and integer 1 < i < 4, we compute the number of sites such that X(x) : C and ,t19 : i.
Theseare estimatedfor the hypothetical data of Table 1 and
shown in Table 4. A one-membersite has a membershio
value greaterthan r for only one categorywhile a multiplemember site has membership values for two or more categories.The total number of sites (T) in each group is further
distinguishedinto matched [M) and mismatchedsites (N)
using MAX.Nine out of a total of ten sites in ClassA are single-membersites.All nine single-membersites are correctly
matched as shown in the seconddata column marked "M"
in the table, as is the two-membersite. On the other hand,
the least correct category,ClassC, shows the following pattern. It has the least single-membersites.Its only singlemember site is conectly matched [seecolumn "M" under set
membership1). ClassC has nine two-membersites.Six of
thesesites are mismatched,meaning that some other class
has been given a higher scorethan C in thesesites.The remaining cover classes,ClassB and D, fall betweenthe two
extremesA and C (seeTable 4). Neither zero membership
nor three or four membershipsites occur in this dataset.
The intent of the MevBERSHTp
function is to explore the
possiblesourcesof error in maps, which can be indicated in
the frequencyof matchesand mismatchesfor the different
numbers of set memberships.In the examplepresented,the
proportion of matchesis higher for the single membership
sitesthan the multiple membershipsites.This pattern is the
oppositeof what would be expecteddue to random effects.
While the datasetused is hypothetical,the results mirror
those found in our initial tests of this approach(Woodcock
and Gopal, 1992).The use of the table is in giving some indication of the environmental conditions in which errors are
being made.If there are significant numbers of mismatches
(errors)in the single membershipsites,then the mapping
processis breaking down in the unambiguouslocations.If
the mismatches are concentrated in the multiple membership
'
sites,efforts to improve the mapping methodi should focus
on the areasof heterogeneouscover and relative ambiguity
betweencategories.Also, notice that the frequencyof multiple membershipsis not the samebetweencategories.The
high frequencyof occurrenceof multiple set membershipsor
the lack thereof can be an indication of the characterof a
map category to the users of a map.
If all sites had single set memberships,there would not
be any need for the use of fizzy sets and the methods presentedjn this paper.For those sites there is one unambiguous right answer.Thus, the frequencyof multiple
membershipsites gives some indication of the need for the
use of fuzzy sets.
Natureof Errors
One important kind of information is the categorical nature
of the enors, or betweenwhich categoriesis their confusion,
In a traditional analysis,this information is containedin the
off-diagonal elements of an error or confusion matrix, Similar
tables can be constructed using the CONFUSTON
and AMBtcUITY functions and fuzzy sets. The coNzusroN function (: N
--+2a identifies categorieswith a rating greater than the
PE&R5
mapped category.In particular, for a site x, (x) computes the
set of categoiieswhose membership values are greater than
the membershipvalue assignedto the map category1(x).
((x) = {C I C e C and ps(x) > &xr*r(x)}
It is identical to a traditional confusion matrix except for
the fact that more than one category can have a ruting {p.6)
higher than the mapped cateSory(prut\)) at a single site (x).
For each (distinct) pair of categoriesC, C' e C we compute
the quantity (.", the number of sites x such that 119 : 5
and p""(x) > pc!). The values of this coNFUSIoNfunction
are the first values given in each cell of Table 5.
function q : N=+ 2c identifies categories
The AMBIGUITY
with the same rating as the mapped category.Formally:
q(x) : {C I C € 4and p6(x) : p11*y(x)}
For each pair of (distinct) categoriesC, C' e C we compute
the quantity 46.6,,the number of sites x such that f(x) : 6
function
and p",(9 : pc@. The values for the AMBIGLIITY
are given in the secondcolumn of each cateSoryin Table-5.
Thelesults are presented only where the membership values
are some degreeofright (valuesgreateror equal to 3)' The
information on the equally rated or ambiguous categories
will be particularly interesting for users of the maps if the
matrix ii not symmetric. The asymmetriesindicate which
categoriesmore frequently include sites whose membershipis
ambiguous,From our example,ClassD includes many_sites
ratedlqually with ClassB, but ClassB doesnot include any
sitesraGd equally with ClassD. The conclusion for a user of
the map would be to expect sites in the ambiguousareasbetween ClassesB and D to have been mapped as ClassD.
Discussion
To summarize,the proposedmethodshave severaladvantagesover the traditional approachin assessingthe accuracy
of-thematic maps. Some of the improvements are in the form
of additional information about errors of the kind provided
from traditional analyses.For frequency of errors, the combined use of the vax and rucgt functions constitutes an improvement. Similarly, using the CONzuSIoNand Al'tstculrv
iunctions provides additional information on the nature of
errors. There are also benefits obtained in the form of new
kinds of information about errors. The olFfnnntcE measure
yields information on the magnitude of errors, of a kind
unobtainable from traditional approaches'The benefits assofunction may prove
ciated with the use of the DIFFERENCE
valuable in understanding the distribution of errors' The
function also provides information outside the
MEMBERSHIP
domain possible using classical set theory. The MEMBERSHIP
function provides indications of the kinds of environments
in which-errors are concentrated,which may help isolate the
sourcesof errors in maps.
There are many isiues and questions that remain to be
addressedregarding the use of.fuzzy sets in map eccuracy
assessment.Our hope is that this paper will stimulate interest in the use of fuzzy sets and that others will expand on
our ideas. One area that needs attention concerns the ability
to use thesemethods for the purposesof map comparison'
Most of the information in the tables presented in this paper
relates to the problem of understanding the errors in a single
map. For comparingmaps, additional researchis neededon
a couple of topics, First, a method of distilling the results to
a single measureor index of map accuracy may be required
as well as a way to produce a kappa-likestatistic.A related
PE&RS
FuNclloNs (sEETDc FoR DErAlLs)
coNFUsloNANDAMBIGUITY
TreLe 5.
Expert Evaluation
Map
Label
ClassA
ClassB
ClassC
ClassD
ClassA
(cc'
tlcc'
ClassB
(cc'
rlcc'
ClassD
ClassC
tcc' tlcc' tcc' lcc'
No. of
mismatches
(cc'
rlcc'
xx00000000
00xx420042
0030xx5080
00243OXX54
Total
1a
concern is a way to standardize results from different experts. Someexpertswill be more lenient in the number of
multiple set membershipsgiven, and this factor will need to
be talien into account.Another dimension of neededresearch
concerns the ability to derive global estimatesfor map categiven-b-yHay (1988)
goriesalong the lines of the ap-proaches
used by simply rebe
could
Their
methods
Ind Iupp fises).
coori.u"tit g a traditional error matrix using the fuzzy^data,
but a metho=dthat took advantageof the additional information would be preferable.
-of
The issue errors and inaccuracy in a spatial database
is an important researchtopic in GtS'Decisionsmade using
the end products of crs analysis are dependent on how errors
propagatethrough the system.This-researchis concerned
i,rriti o*netype of error - categorical or tlrematic error. The
methods ouilined in this paper describe how magnitude, nature. and sourceof theseirrors can be identified and measured using fuzzy sets. It fits into Level I and II of the
"hierarchy-ofneeds" model of cIS-proposedby Veregin
(19Sg),Uirtil recently, error modeling has not receivedmuch
attention (Goodchildet a1.,7992iLanter andYeregin, 1992)'
Understanding the nature and distribution of errors in a map
is a step in this direction. The paper has describedvarious
measuies that can be used to analyze errors in greater detail.
This researchhas implications in the areaof error modeling
and developinga model of error propagationin the GIScontext, as its iesults can be easily incorporatedto produce better and more predictive error models.
Snatial dita is often impreciseand cannot be captured
bv bo6leanloeic. The propoled methodologycan cope with
impreciseor imbiguoris data' It permits subjectiveevaluation
of ixperts to be iniorporated into the design. The results of
this research can be incorporated into error modeling and
conveyed to the map user.
Conclusion
Categoricalmaps are commonly Ploduced to represent complex-seosraphical patterns' As such, it is essential that
greatir
b" made to deal explicitly with the measureerrors in categorical maps. This paper describesa
irent of"Ifoitt
new approachto assessthe accuracyof a thematic map
basedin fuzzy sets.The suitability of fuzzy setsin the map
accuracy contLxt is demonstratedby deriving a set of measures to analyze the nature, frequency, so-urce'and magrri
tude of er.ori. The approachis lllustrated using a simple
example, and the results obtained from it are compared with
those'of the traditional approach.The feasibility of the approach is further discussbdusing empirical data (Woodcock
and Gopal,1992).
The results of the analysisprovide useful information to
both the producer and user of thematic maps. Producerscan
ta7
improve methods of making the maps and presenting the information on accuracy and enors to the end user. UJe of thematic maps will be enhanced by an improved understanding
of their reliability.
Acknowledgments
The authors thank Vida Jakabhazy,Scott Macomber, Soren
Ryherd, Ken Casey,Ralph Warbington, and |ack Levitan for
their valuable assistancein collecting data. This researchwas
su-pportedin part by a grant from the Timber Management
Office of Region 5 of the U.S. Forest Service.The authors
also thank the anon5[nous reviewers, as the paper has been
significantly improved as a result of their comments.
References
Aronoff, S., 1982a.Classificationaccuracy:A user approach.photogrammetricEngineeilng& RemoteSensing,48(8):1299-1307.
1982b. The map accuracy report: A user's view. photogrammebic EngineeringI RemoteSensing,48(8):1309-1312.
Burrough, P. A., 1989. Fuzzy mathematical methods for soil survev
and land evaluatiou. lournal of Soil Science,40i477.
Card, A., 1982. Using known map'category marginal frequencies to
improve estimatesof thematic map accuracy. Photogrammetric
Engineering& RemoteSensing,48(3):431-439.
Congalton,R., 1991.A review of assessingthe accuracyof classifications of remotely senseddata. Remote Sensingof the Environrnent,37:35-46.
Congalton, R., R. Oderwald, and R. Mead, 198A.Assessinglandsat
classification accuracy using discrete multivariate anilysis statistical techniqu es.Photogammetric EngineeringI Remote
Sensrhg,49(72):167t-767 B.
Dubois, D., and H. Prade, t9BO.Fuzzy Sefsond Systems:Theory and
Applications. Academic Press,New York.
Ericcson,K. A., and H. Simon, !984. ProtocolAnalvsis.MIT press,
Cambridge,Massachusetts
Fisher,P., 1991.First experimentsin viewsheduncertainty:The accuracy of the viewshed area.PhotogrammetricEngineering&
RemoteSensrng,57(t0):I32t-I327 .
-^ , 1992.First experimentsin viewshed uncertainty:Simulating
fuzzy viewsheds. Photogrammetric Engineeilng& Eemote Sens-tng 5B(3):345-352.
Fisher, P., and S, Pathirana, 1990. An evaluation of.fuzzv membership of landcover classesin the suburban zone.Reiote Sensing
of the Envircnment, 34i727-732,
Goguen,J., 1969.The logic of inexact concepts.Synthese,lg:325J/3.
Goodchild,M., S. Guoqing,and Y. Shiren, 1992.Developmentand
test of an error model for categorical data. Infernafrbnal Journal
of GeographicalInformation, 6(2):87-IO4.
Hadipriono_,_F.
q., I. G.Lyon, and L. Thomas,1991.Expert opinion
in satellite data interpretation. PhotogrammetricEngineeiing &
RemoteSensing,57(t):7S-7B.
Hay, A.M., 1988.The derivation of global estimatesfrom a confusion
_-matrix. International Journal of RemoteSensrn& 9:1392-139g.
Hoffer, R., and M. Fleming, t978. MappingVegetative Coverby
ComputerAided Analysis of SatelliteDofo. Purdue University,
I,ARS Technical Report 1IIIZB.
]upp, D.L.B., 1989.The stability of global estimatesfrom confusion
matrices. International Journal of RemoteSensing,10(9):15631569.
Kaufmann, A., 1975. Inboduction to the Theory of Fuzzy Subsets
(translatedfrom French by D. L. Swanson).Academicpress,
New York.
Kent, f.T., and K.V. Mardia, 1988. Spatial classificationusing fuzzy
membership models. IEEE Transactionson pattern Analvsis and
Machine Intelligence,10(5):659-621.
Kollias, V.J.,and A. Voliotis, 1991.Fuzzy reasoningin the developm9."! oJ geographical information systems.FRSIS: a prototype
soil information system with fuzzy retrieval capabiliiies. Intir- national Journal of Geographical Information, S1Z\:ZOS-ZLS.
Lakofi G., 1973. Hedges:A study in meaning criteria and the Iogic
_ of fuzzy concepts./ournal of PhilosophiCalLogic, 2:4SB-SOBI
Lanter, D.P., and H. Veregin, 1992. A research paridigm for propagating error in layer-basedGlS. Photogramhetric-Engineeriig A
RemoteSensing,58(6):825-S35.
Leung, Y. C., 1988. Spotral Analysis and Planning lJnder Imprecisrbn. North Holland, Amsterdam.
Mead, R., and f. Szajgin,1982.Landsatclassificationaccuracyassessmentprocedures. PhotogrammetricEngineering& Remote
Sensing,a8(1):139-141.
Robinson,V. B., 1988. Someimplications of fuzzy set theory applied
to geographic databases.Computers,Environment, and IJrban
Systems,72t89-97.
Robinson, V. B., and A. Strahler, 1984. Issuesin designing geographic information systemsunder conditions of inexictness.
Machine Processingof Remotely SensedData Symposium,pp,
798-204.
Rosenfield, G., 1981. Analysis of variance of thematic mapping exq99Tgn! qata. Photogrammetric Engineering & Remo{e-Seising,
47(12):7685-7692.
Rosenfield, G., and K. Fitzpatrick-Lins, 1986. A coefficient of agreement as a measure of thematic classification accuracv. pioTo_
_ grammetic EngineeyingI EemoteSensrng,S2(2)tZ2S-227.
Story, M., and R. Congalton,1986.Accuracvissessment:a user's
perspective.PhotogrammebicEngi neering& RemoteSensing,
52(3):3
77-399.
Trivedi, M.M., and f.C. Bezdek,1986.Low-levelsesmentationof aerial images with fuzzy clustering. IEEE Transaclionson Systems,
Man and Cybernetics,16(4):589-598.
van Genderen,f., and B. Lock, t972. Testingland-usemap accuracy.
PhotogrammetricEngineering& Remote-sensrng43(9i:11387737.
Veregin,H., 1989.Error modeling for the map overlayoperation,Itre
Accwacy of Spatial Database.s(M. Goodihild anil S. Gopal, edi
tors), Taylor Francis,London, pp. 3-18.
Wang, F., 1990.Improving remotesensingimageanalysisthrough
fuzzy information repiesentation. photogrimmetric Engineeiing
I RemoteSensing,56(S):1163-1169.
Wang, F., G. B. Hall, Subaryono,1990.Fuzzy information representation and processingin convontionalGIS software:databasedesig-nand application, International loumal of Geographical
Information Systems,4:261-28J.
Woodcock,C., and S. Gopal, 1992.AccuracyAssessmentof the Stanislaus Fore1 Veget-ationMap using Fuziy Sets, proceedingof
the Fourth Biennial Remote Sensing Applications Conferenlce,'
Orlando, Florida, April, 1z p.
Woodcock,C., and V. Hirward,^t992. Nestedhierarchicalscene
model" and image segmentation, International lournal of Remote Sensing,13(16)1316Z-3189.
Zadeh,L., 1965.Fuzzy sets.In/ormotron and Control,8:338-353.
1968. Probability measuresof fuzzy events./ournal of Mathematical Analysis and Applications, 101427427.
1973. Outline of a new approachto the analysisof complex
or imprecise concepts.IEEE Transactions:Systems,Man and
Cybernetics,SMC-3:28-44.
Zadeh,L., K. Fu, K. Tanaka,and M. Shimura (editors),t975. Fuzzy
Setsond Their Applications to Cognitive and Decision
Processes.
AcademicPress,New York.
(Received12 March 1992;revisedand accepted15 December1992:
revised26 January1993)
Rememberto mail in our electionballot by April 6, 19941
PE&BS