2009‐11‐20
THENATUREOFEVALUATION
MichaelScriven
ClaremontGraduateUniversity
DEFINITIONS
1.Evaluationistheprocessofdeterminingmerit,worth,orsignificance(abbreviatedhere
asm/w/s).Inmanybutnotallcontextsthesethreepropertiesareroughlythesameas,
respectively:quality,value,andimportance.
2.Anevaluationistheproductofthatprocess.
3.Anevaluandiswhateverisevaluated.(Thetermevalueeisoftenusedforhumanevalu‐
ands.)
REFINEMENTS
Theprocessofevaluationmaybejudgmentalorinferential;eithercanoftenbedoneatthe
everyday/commonsenselevelorataprofessionallevel.Examples:JudgesofOlympicdiv‐
ingorskating,andrefereesincompetitivesports,areexamplesofprofessionalevaluators
largelyusingjudgmentratherthaninference;undergraduatescheckinganallegednew
proofofFermat’sLastTheoremareamateursusinginference.Englishteachersinsecond‐
aryschools,gradingessays,includemanythatuseoneortheotherapproachexclusively;
oftenthejudgmentalgradersaresuperior.(Reference:…)
Professionallevelevaluationisdistinguishedbytheuseofvalidatedskillsortechniques
thatrequireextensivetraining,andbyrespectforstandardsofqualityandethicsinthe
work;anditofteninvolvesanextensivecommitmentoftimeandotherresourcesthatis
usuallyonlypossibleforthosepaidtodothiswork.
Thetransdisciplinaryviewofevaluation
Definition4.Adisciplineis:(i)abodyofknowledgeorskillswithacommonsubjectmatter
orfieldofactivity,thatis(ii)distinguishablefromthatofotherdisciplines,and(iii)often
hassomedistinctivekindorkindsofinvestigativemethodsthatareexplicitlystudiedand
debated(althoughtheymaybeimplicitratherthanexplicitinthecommonpracticeand
teachingofthephysicaldisciplines),(iv)thatalwaysincluderespectforlogicalrequire‐
mentsonevidenceandinference(e.g.,consistency,transitivity,defensibility);andadisci‐
plineis(v)normally—i.e.,exceptinitsbirthingprocess—regardedasaseriousfieldof
workforasignificantnumberofhighlytrainedpersonnel,whosepursuitis(vi)normally
expectedtohaveproducedatleastsomeresultsofnotablesocial,intellectual,orpersonal
value,asjudgedbythoseinotherfields.
Note4.1:thissetofconsiderationsisofferedassufficient;somemaynotbeneces‐
saryconditions.
Note4.2:besidestheacademicdisciplines,ofwhichevaluationisherearguedtobe
one,therearealsomanyphysicaldisciplinese.g.,boxing,dance,tai‐chi,marathon
training,regimentalmarching,andtheartisticdisciplinessuchasplayingthepiano
andportraiture.Evaluationappliesequallytothem(detailslater),althoughitisof
coursenotoneofthem;cf.‘intradisciplinaryevaluation,’below.
DEFINITION5.Atransdisciplineisaspecialclassofdisciplinethatmeetstwocondi‐
tions:(i)ithasthestatusofanautonomousdiscipline,meaningthattherearepeo‐
plestudyingitanditsmethodsandpracticesfortheirownsake;butalso(ii)its
methodsandresultsprovideimportanttoolsusedinasignificantnumberofother
disciplines.Statisticsisagoodexampleofatransdiscipline;mathematicsisamore
generalexample,sinceitincludesstatistics;logicisasupremetransdisciplinesince
itsstandardsarepartofthedefinitionofdisciplineitself;andevaluation,soitwill
bearguedhere,istheonlyotheruniversallyessentialtransdiscipline.
Thetransdisciplinaryviewofevaluationisdefinedbyasetofpropositionsdealing
withvariousaspectsoftheview,asfollows(forthemostparttheserefertoprofess‐
ionalevaluation):
GroupA:Propositionsreferringtothegeographyofknowledge
PropositionA1.Evaluationcannowberegardedasadisciplinesinceithas:(i)adefined
andfairlydistinctiveterritory—thestudyoftheprocessofdeterminingm/w/s;(ii)some
distinctivemethods,describedbelow;(iii)adherencetotheusualresearchstandardsof
logicandevidenceacceptedbythegeneralbodyofdisciplines;(iv)aworkforcenumbering
inthesixfiguresworldwide,alongwithdedicatedbooks,encyclopedias,journals,websites,
professionalassociationsandconventions;(v)contributionsofsomeintellectuallyandso‐
ciallyvaluableresults,asjudgedbythosenotcommittedtoit,severalofwhicharesetout
inthiswork.
PropositionA2:Evaluationisamegadiscipline.Evaluationhasavastrangeofestablished
sub‐specialties(a.k.a.sub‐disciplines)andanunlimitedrangeoffieldsinwhichoneor
moreofthesecanbeappliedprofessionally.Evaluationactivitycanberepresentedasa
three‐dimensionalspace,withaxesfor:(i)thesub‐fieldsofevaluation,(ii)theapplication
areasforevaluation,and(iii)methodsofinvestigation.(Thereissomeoverlapbetween
these.)
PropositionA3:Evaluationisatransdiscipline.Itisnotonlyadiscipline,asarguedabove,
butitisappliedineverydisciplineasthepartofapplyinglogicandreasonthatrelatesto
andindeedprovidestheentirecoverageofmeetingthestandardsofevidenceandinfer‐
encethatjustifyclaimsaboutthequalityofallresearchorknowledgeinthediscipline,and
henceitsclaimtobeadisciplineatall.Thereasonastrologyisnotadisciplinebutmerelya
subjectofstudyissimplyafailureofthequalityofitsdataandhypotheses,i.e.,afailureto
meetthequalitystandardsofadiscipline—andqualitystandardsarethebusinessofeval‐
uationtodefineandsupervise,atthegenerallevel.
[Fig1abouthere.]
2
Thedefinitionoftheterm‘transdiscipline’requiresthatevaluationisnotonlyadis‐
ciplineinitsownright,withitsmanysub‐divisionsandappliedfields,including
theoreticalaswellaspracticalresearch,butitservesanessentialroleasatoolfor
muchresearchinotherdisciplines.Infact,theclaimofeverydisciplinetothatvery
titledependsontheextenttowhichitmaintainsgoodstandardsintheassessment
ofevidencequality,hypothesisrating,andcontinuedproductionofsignificantra‐
therthantrivialresults—allofthesebeingexamplesof‘intradisciplinaryevalua‐
tion’.
Intradisciplinaryevaluationisbestclassifiedasprofessionaljudgmentalevaluation,in
mostcases,therebeingonlyafewheuristicsandnoalgorithmstomakeitamatterofde‐
monstrablyvalidinference.Learninghowtodointradisciplinaryevaluationisanessential
partofbecomingcompetentasaresearcher,oradvancedpractitioner,inanydiscipline.
Thequalitycontrolsysteminalmosteverydisciplinedependsultimatelyonpeerreview.Of
course,‘peers’areexpertsinthatfield;unfortunately,theyareusuallynotexpertsinevalu‐
ation,withtheresultthatpeerreview,ascurrentlypracticed,turnsouttobeanextremely
defectivesystem.Itcanbegreatlyimproved,andsuggestionsfordoingthisarediscussed
laterinthiswork.
NoteA3.1:whenevaluatorsidentifytheirspecialty,theytypicallymentionbotha
specialtyareaofevaluationandanapplicationarea;e.g.,‘(evaluationof)leadership
traininginthemilitary.’
NoteA3.2:mostofthenamesofthesub‐areasinevaluationmakesomereferenceto
an‘applicationarea’,inthesenseofafocusononetypeofevaluand;butthetermin
quotesishereusedtorefertotypesofcontext(knowledgearea,jobenvironment)
inwhichtheevaluandtypeislocated.Thisdistinctionbetweentwosensesof‘appli‐
cationarea’isnotarbitrary;itreflectsthefactthatthetoolkitofspecializedtech‐
niquesforgoodevaluationismoresubstantiallydifferentbetweenevaluandtypes
thanbetweenareasofknowledgewheretheevaluandofinteresthappenstobelo‐
cated.Buttherearestillsignificantdifferencesbetweenareas,e.g.,leadershiptrain‐
ingforthemilitarycallsforconsiderableknowledgeofthemilitary’sneeds,capaci‐
ties,andconstraintsthatdiffersfromthosefactorsintrainingbusinessorcollege
leaders.
[UnpackingFigure1,above:]
(i)Evaluationspecialties
Manybutnotallofthesearealsoproperlycalledsub‐fields(a.k.a.divisionsorsu‐
evaluations)ofevaluationbytypeofevaluand.Therearethreegroups:
(a)7recognizedautonomoussub‐divisions(a.k.a.sub‐disciplinesor‘specialties’)in
evaluation,e.g.,personnelandproductevaluation.Eachhasthousandsortensof
thousandsofprofessionallytrainedpractitioners,publications,conventions‐‐the
paraphernalia.Thefulllistis:programevaluation(whichincludestheevaluationof
processes,projects,systems,organizations,services),products(includingproduce),
performances,personnel,proposalsandplans,policies,portfolios;
3
(b)3newsub‐fieldsidentifiedasabyproductofthedevelopmentofadisciplineof
evaluation—intradisciplinaryevaluation,meta‐evaluation,andpublicevaluation
(we’lldefineanddiscusstheselater);
(c)6classicaldisciplinesthathavealwaysbeenlargelydevotedtothepracticeof
evaluation(ethics,logic),andcertainlyinvolvedinittoanessentialdegree(medi‐
cine,engineering,politicalscience,architecture).Thesearenotusuallyorproperly
calledsub‐divisionsofevaluationbecausetheyhavelongsincebeenrecognizedas
independentdisciplines,andincludemuchthatgoeswellbeyondevaluatione.g.,
hugeareasofpurelyexplanatoryknowledge.Butpartsofthemare,logicallyspeak‐
ing,exactlythat.
(ii)Applicationareas(a.k.a.environments)forevaluation.Therangeofthings,andaspects
ofthings,thatcanbeevaluatedisunbounded.Everythingthatexistsorcanbeconceived
canbeevaluated:evenGodisjudgedgood,andevenquarkshavecharm.Thisisonereason
evaluationisnotonlyamegadisciplinebutpossiblythelargestoneintermsofscope.But
inpracticetheturfonwhichevaluationsaredone—theenvironmentoftheevaluation
wheretheclientsandaudienceslive,demand,andconsidertheevaluationreports,canbe
usefullydividedintodifferenttypes.Intherealworldbusinessofprofessionalevaluation,
thetwomostcommonareasofapplicationare:
a. Thefieldsofmostcommonhumanendeavor,e.g.,manufacturing,services,educa‐
tion,health,military,mining,travel,agriculture,management,art,andathletics.
Anypracticalapplicationofthesefieldsinvolvesandrequiresevaluation.
b. Theacademicdisciplines,eachofwhichisbuiltonevaluationofeverytooltheyuse
andeveryresulttheyproduce;andwhosepracticalapplicationsarealsosubject
toandinvolveevaluation.
(iii)Methodsofinvestigationorpresentation.Surveyingallthespecialtiesinevaluation,it
becomesclearthatalmosteveryknown‘software’researchmethod(bycontrastwithre‐
searchhardwaresuchastelescopesandtheLargeHadronCollider)isusefulsomewherein
evaluation.Manyofthesearequalitativeandmanyarequantitative;usingbothinone
study,alsoacommonandoftenoptimalpractice,iscalled‘mixedmethod’evaluation.There
arefrequentlymanyalternativemethodsthatwillyieldagoodevaluation.Modelsetc.
[Furtherdiscussionofsomeaspectsofthetransdisciplinarymodel.]
4
Intradisciplinaryevaluation.Thedefinitionoftheterm‘transdiscipline’requiresthateval‐
uationisnotonlyadisciplineinitsownright,withitsmanysub‐divisionsandapplied
fields,includingtheoreticalaswellaspracticalresearch,butitservesanessentialroleasa
toolformuchresearchinotherdisciplines.Infact,theclaimofeverydisciplinetothatvery
titledependsontheextenttowhichitmaintainsgoodstandardsintheassessmentofevi‐
dencequality,hypothesisrating,andcontinuedproductionofsignificantratherthantrivial
results—allofthesebeingexamplesof‘intradisciplinaryevaluation’.Itisbestclassifiedas
professionaljudgmentalevaluation,inmostcases,therebeingafewheuristicsbutnoalgo‐
rithmstomakeitamatterofdemonstrablyvalidinference.
Learninghowtodointradisciplinaryevaluationisanessentialpartofbecomingcompetent
asaresearcher,oradvancedpractitioner,inanydiscipline.Thequalitycontrolsystemforit
inalmosteverydisciplinedependsultimatelyonpeerreview.Ofcourse,‘peers’areexperts
inthatfield;unfortunately,theyareusuallynotexpertsinevaluation,withtheresultthat
peerreview,ascurrentlypracticed,turnsouttobeanextremelydefectivesystem.Itcanbe
greatlyimproved,andsuggestionsfordoingthisarediscussedlaterinthiswork.
ONESTANDARDLISTOFTHEDISCIPLINES
·2Humanities

2.1History

2.2Languagesandlinguistics

2.3Literature

2.4Performingarts

2.5Philosophy

2.6Religion

2.7Visualarts
·3Socialsciences

3.1Anthropology

3.2Archaeology

3.3Areastudies

3.4Culturalstudiesandethnicstudies

3.5Economics

3.6Genderandsexualitystudies

3.7Geography

3.8Politicalscience

3.9Psychology
5

3.10Sociology
·4Naturalsciences

4.1Lifesciences

4.2Chemistry

4.3Earthsciences

4.4Physics

4.5Spacesciences
·5Formalsciences

5.1Computersciences

5.2Logic

5.3Mathematics

5.4Statistics

5.5Systemsscience
·6ProfessionsandAppliedsciences

6.1Agriculture

6.2Architectureanddesign

6.3Business

6.4Divinity

6.5Education

6.6Engineering

6.7EnvironmentalstudiesandForestry

6.8Familyandconsumerscience

6.9Healthsciences

6.10Humanphysicalperformanceandrecreation*

6.11Journalism,massmediaandcommunication

6.12Law

6.13Libraryandmuseumstudies

6.14Militarysciences

6.15Publicaffairs

6.16Socialwork

6.17Transportation
6