DECI-2: Research Study
LEARNINGWITHANDUSINGDECIMATERIALS:
LESSONSLEARNED
Sonal Zaveri and Wendy Quarry
May 2016
CONTENTS
ExecutiveSummary......................................................................................................................3
Introduction.....................................................................................................................................4
Findings..............................................................................................................................................4
1)MaterialFindings:...............................................................................................................4
2)ThesurpriseofIntegration:............................................................................................6
3.LookingAhead–Popularizingtheapproach?.........................................................7
a)Labeling–firstweneedtogetitright:..................................................................7
b)Makingitsimple–itisaframework(s)!..............................................................7
c)Roleofmentor–thementorrelationshiptomaterials..................................8
d)Materialsandwebsite...................................................................................................9
f)Popularizingtheapproach–areweready?......................................................10
AttachmentOne:TermsofReference...........................................................................12
AttachmentTwo:QuestionnaireforMentors............................................................14
2
ExecutiveSummary
Thisreportsummarizesareviewofthematerialsthatmentorsusedwithprojectsatthe
networkandsub-projectlevel.TheseincludedbothDECI1and2printmaterialsaswell
asonlinetoolssuchaswebinars,presentationsandmodules,availablethrougha
websiteplatform(www.evaluationandcommunicationinpractice.net)aswellasmore
recenthybridmaterialsthatsynergizedevaluationandcommunicationapproaches.The
studyrelatedtoa)understandinghoweffectiveandusefulthematerialswereandwhy
b)feedbackontherelativelynewermaterialsthatdescribedthehybridapproachandc)
suggestionsforpopularizinganddisseminatingtheapproachestestedintheDECI2
project.
Evidencewasgatheredfromaquestionnairetoallmentors(DECI1and2)andone
mentee,followedupbyinterviewsforclarification.Inall,14personsprovidedevidence
forthispaperincluding11mentors,onementee/funderandtwofacilitatorsof
communitiesofpractice.
Keylearningswereasfollows:
1. Materialswereuseful,diverseandadaptabletothecontext.Whatwasused
dependedonthementor’srole(evaluationmentorsusedUFEand
communicationpersonsusedtheResCommaterials);specialmentionwasmade
ofthePrimer,followedbythemodules,checklists,presentations,webinars
(less)andcasestudies(less)withthemorerecentmaterialssuchasVeriCom
anddraftarticlelessunderstoodandused.Materialshave‘title’confusion
possiblybecausesomearesubsumedinothers.Forexample,indescribing
templates,posters,checklists-theposterispartofthetemplatebutis
sometimesmentionedseparatelyandthereportingtemplatewasactuallythe
checklisttosome!NavigatingtheDropBoxtoaccessbothvarietyandversionsof
materialswastimeconsuming.
2. Vericom(theacronymappearsartificial)materialsthatemergedinDECI2,
suggestingasynergybetweenUFEandResCom(especiallytheinitialsteps)are
toonewandrequirefieldverification.Infact,UFEandResComcouldalsobenefit
fromsimplerlabelsoracronyms.
3. Brandingoftheapproach(es)UFE,ResCom,VeriComisimportantbutneeds
tobelessmystifying,selfexplanatorywithacleariterationofitsvalueto
learningorganizationstosupportevaluativethinkingandevidencebased
decisionmaking.
4. Mentor-materialsrelationshipindicatedimportanceofmentor’srole,
readinessandcapacityfortimelyprovisionofselectedmaterials,relevantto
organization’slearningstyle,contextandcapacity.Thisisinsharpcontrastto
uniformmaterialsprovidedatcapacitybuildingworkshops.
5. Materialsweredevelopedforthepurposeofmentoringorganizationsin
evaluationandcommunication.To‘evangelize’theapproach,DECI2willhave
tostrategizeforwhom,whyandwhatmaterialswillneedtobedeveloped.
Branding,advocatingtheapproachandbuildingacommunityofpracticewill
likelyneedareviewofcurrentmaterial,co-creationandwidersharingof
materials.
Aconcludingthoughtisthatalthoughtherearetriedandtestedwaystodevelop
andusematerialstodisseminatetheapproach(es)andlearnings(andOutcome
MappingandRAPIDhaslessonstolearnfrom),furtherexperimentationand
experiencemaybenecessarytotestdrivethe‘integrated’approach(es)alongwith
variationsofthementoringprocesswithothersectorsanddonors.
3
Introduction
UFE(DECI1andDECI2)1andResCom(onlyDECI2)developedanumberofmaterials
tomentorselectedprojectsatthenetworkandsub-projectlevel.Theseincludedprint
materialsaswellasonlinetoolssuchaswebinars,presentationsandmodules,available
throughawebsiteplatform2.DuringthecourseofDECI2,somehybridmaterialswere
alsodeveloped.Thematerialsgrewoutoftheactionresearchthatcalledforanattempt
tosynergizeevaluationandcommunicationtools.Forthisresearchstudy,twomentors3
wereasked(SeeAttachmentOneforTORs)toengagewithothermentorsto
understand:
1) Howeffectivewerethematerialstosupportaction-researchandasreference
tools?Whathasworked;whathasnot;andwhatarethepossibleexplanations?
2) WithreferencetotheintegrationofahybridapproachundertheVeriComtitle
throughadrafte-Primer(forpractitioners)andadraftjournalarticle(for
researchers):Whatarethemeritsandlimitationsofthedrafts?Whatactionsare
neededtomakethemmorerelevanttoeachaudiencegroup?
3) Whatvariationsandopportunitiesareavailablewiththewebsiteduringthe
remainderoftheDECI-2projecttopopularizetheapproach(VeriCom,UFEand
ResCom)?
Forthepurposeofthestudy,aquestionnaire(seeAttachmentTwo)wassenttoall
mentors(DECI1and2).Thiswasfollowedupbyinterviewsforclarification.Inall,14
personsprovidedevidenceforthispaperincluding11mentors(alsoDal,Ricardo,Sonal
andWendy),twothoughtleaders(whohadfacilitatedcommunitiesofpractice)andone
mentee/funder(Sylvia).
Discussionwiththementorsindicatedalotof‘learnings’relatedtoquestiona)with
fewercommentsforb)andc).
Findings
1)MaterialFindings:
a)Thematerialshavebeenusefulandcanbeadaptedtothecontextandneedsofthe
mentees.
TheDECI2mentorswereaskedtheusefulnessofthematerials:
MaterialsUsage
DECI1
DECI2
High
UFEchecklist
Same
High
Modules
High
Casestudies
High
Primer
High
Conferencepresentations
High
UFETemplate(Step12–
TellingtheStory)
1DevelopingEvaluationandCommunicationCapacityinInformationSociety
Research
2www.evaluationandcommunicationinpractice.org
3WendyQuarryandSonalZaveri
4
Mostlyhigh
ResComChecklistand
modules
ResComTemplate(poster)
UFETemplate
VeriComTemplate
Webinar
Casestudies
DraftEPrimer
DraftArticle
Mediumtohigh
Mediumtohigh
LowtoMedium
Medium
Medium
Low
Low
Mentorsusedthematerialsmostrelevanttothemi.e.evaluatorsusedUFEmaterialsand
communicationspecialistsusedtheResCommaterials.Allmentorsfeltthatthe
materialswereflexibleandlentthemselvestoadaptation.Forexample,thechecklists
suggestastep-by-stepmethodologythatdoesnotnecessarilyadheretoreallife
situations–butdohavevalueastheyprovideaframeworkagainstwhichtocheck
progress.Similarly,thosedoingResComfounditusefultoaskparticipantstodescribe
whattheywerealreadydoingincommunicationandthenpulloutthetemplate(poster)
tohelpillustratequestionsaroundpurpose,audience/stakeholdersandmethods.
ThevarietyoftoolsofferedinDECI2wasseenasastrengthasitenabled
accommodationtodifferentlearningstylesandcanbeusedindifferentcombinations.
Forexample:
“Iusedifferentmaterials–itismoreamatterofcombiningthem–forexample,the
checklistalonewouldmakeitseemverytaskorientedbutthepresentationhelpsthemto
seethebigpicturefirstasdoestheResComtemplate”.
“Whatworksisthetimely,focusedlinktoaresourcethatrespondstoaperson’simmediate
needs”
“OnceIfinishthepresentation,peopledobrowsethroughthePrimerusuallyinbetween
themeetings–somereadit,anotherbrowsedandreadwhateverwasinteresting.The
ResComTemplate……..peoplegothroughitfillinginthesections.”
Ultimately,itwasHOWthementorsusedthematerialsthatdeterminedhoweffective
theywere.Asonementorstated,“Theeffectivenessofthetoolsdependsonasetof
factorsratherthanthetoolitself”.Itisimportanttounderstandthementee,his/her
constraintsandfromthereoffertherightcombinationofmaterials.Inotherwords,
differentmaterialswereusedwithdifferentmenteesdependingoncontextand
byandlargewerefounduseful.
ThePrimerwasfoundtobeuseful4sinceitoffersasuccinctpictureofhowUFEcouldbe
activelyapplied–itis“easytodigest”plusitcanbedownloaded.Itservesasan
excellentreferenceforpeopletoeitherbrowseordigoutspecificexamplestoguidethe
readertothewayahead.
Themodules,checklists,presentations,primer,webinars(tosomeextent),UFEand
ResComtemplateswereusedwidelybymentors(menteesalsousedthemwith
assistancefrommentors)buttheVeriComtemplate,eprimerandarticlewereusedless.
4MichaelPatton’sbookwastoobigtoread
5
However,itturnedoutthatthecasestudiesoverallwereofgreatervaluetothosewho
haddevelopedthemratherthantonewparticipantsastheywereintended.5
ComparingtheuseofUFEandResCommaterials,amentorsaid:
“HavedevelopedResCommaterialsbutnotuseditenough–toreflectwhatyouaredoing
incommunicationwhicheveryoneisdoinganywaybutnotdoingitstrategically..also
buriedintheDropboxfiles”
b)Thereappearstobe‘title’confusion–thesametoolsandtemplatesarelabeled
differently.
TitleconfusionisprimarilywithDECI2mentors.Templates,posters,checklists
referredtodifferenttools!Forexample,theposterispartofthetemplatebutis
sometimesmentionedseparately;thereportingtemplatewasactuallythechecklistthat
wasused,thetemplateforcasestudy(DECI1)nowaddsonResCom(forDECI2)andit
isunclearwhichreportingtemplatewasbeingreferredto.TheUFEDECI2reporting
templateisfromStep3to8butincludestheposter!So,istheposteramatrixora
snapshot?Duringthedatacollection,wehadtoaskmentorstoexplainwhatmaterial
theyweretalkingabout!6
c)MaterialsareavailablebutaccessibilitythroughtheDropboxisdifficulttonavigate.
MostmentorsfoundthearrangementofthematerialsintheDropboxdifficultto
understandandmostlyaskRicardotosendmaterialsviaemail,whichhedoespromptly
sothereisnoneedtoexploretheDropboxfurther!Barringthat,asearchfeaturewould
beusefultofindmaterials.DECI1mentorsdonothaveaccesstotheDropBox.
2)ThesurpriseofIntegration:
a)Althoughthevalueofcombiningcommunicationwithevaluationisbecomingevident,
theVeriComtemplateneedsfurtherdiscussion–commentsonthetemplatevariedfrom
“artificial”to“haspotentialforapplication”.
TherealityisthatthereareveryfewexampleswhereUFEandCommunicationhave
beenmentoredintandem.Ricardo,DalandWendyhadabrushwiththisthroughan
experiencewiththeWorldBank.RicardohastrieditoutwithtwoCanadabasedsmall
NGOswhileitisjustgettingstartedwithLondonbasedJusticeForumandaprojectin
BuenosAires,AssociationdeDerechosCivilesorCiudadanos(ADC)
However,throughthisrelativelysmallexperiencewearequicklyseeingthatitisthe
firstfourstepsforbothUFEandResComthatformthebasisforthejointapproach.The
samequestionstobeaskedaroundUFEaresimilartosomeforResCom.Bothneedto
lookatcontext(situationalanalysis);bothdecideonaudienceoruserswhilebothneed
toconsiderthesequestionsattheinitiationandearlystartupofprojectinitiatives.
5Thisissimilartoearlierfindingsthatitisthewriters/creatorsofmanualsandworkbookswho
tendtolearnthemost
6EvenattheendofseveralinterviewsSonalandWendyhadtogoovertheterminologyagain
andagaintomakesuretheywereusingthesamewords
6
b)Vericommaterialsaretoonewtobeunderstoodwellandhavenotbeenusedmuch,if
atall.
MosthavenotusedtheVeriCommaterialsanddonotfullyunderstandthem.UFEand
ResComareclearandalthoughthereisaconnectionbetweenthetwo,thetermorthe
waybothareasofworkhavebeencombinedseemsdifficulttoworkwith.Also,theterm
VeriComwassuddenlyintroducedsoatfirstthemeaningwasunclearuntilthe
materialswerestudied.Thislabelisnotyeta‘brand’andsoneedsfurtherexplanation.
3.LookingAhead–Popularizingtheapproach?
a)Labeling–firstweneedtogetitright:
Mentorssuggestedthatthelabelsortheacronymshouldbeself-explanatory.Inthecase
ofUFEorUtilizationfocusedevaluation,itisquiteobviousandclearthatitisaboutuse
ofevaluation.Therewasonesuggestiontorenameit“Usefocusedevaluation”forseveral
reasons:itiseasierintranslation,useofevaluationsisacurrentconcernandthe
acronymcanremainthesame.ResComontheotherhandevokedamixedresponse.For
somementors,itrequiredtoomuchexplanationespeciallytheResearchword.Similarly,
mentorsunderstandthevalueofResComandUFEbutbecauseperhapsVeriComwas
‘suddenly’introduced,thetitledidnotmakesenseuntiltheyhadreadthroughthe
explanation.Infact,onementorquestionedwhat,ifUFEandResComhaveclarity,isthe
advantageofintroducinganewtermlikeVeriCom?
OverallVeriComdidnotappeartobeapopularlabelforthehybridapproach.One
personmentionedthatitsoundedlikeaninternetcompany,thatitrequiredtoomuch
explanationastheworditselfdidnotmeananythingbyitself,thatthebrand(ifitwas
one)wasunrecognizableatthismoment.
b)Makingitsimple–itisaframework(s)!
UFEandResComareframeworkstohelpapersonthinkthroughhowtodoan
evaluationthatisusefulandhowtodevelopacommunicationstrategythatreaches
targetedpersonsfordefinedpurposes.Theseframeworks(UFEandResCom)canbe
easilyunderstoodbecausetheyareessentiallyawayofthinkingandplanning.However,
wemayhavemadeitappearunnecessarilycomplicatedaswhenUFEisexplained
throughadiagramthatshowsintertwiningstepswhichactuallyintimidatesmorethan
explains.Theargumentisthatwithsomuchinformationoverload,itisbesttokeepit
simpleandstraightforward.
Asonementorputit,itisparticularlyimportantnotto‘sell’UFEandResComasasilver
bulletforallcontexts.Therearelessonslearnedfromotherinitiativesaswell:
“SASwasabouttoolsbutnottheproblemstobesolved.SodoNOTfollowtheSASpathto
popularize.………SASisaboutmechanics,OMishowyouthink.UFEisnotlike1,2,3steps
anditismoreaboutconceptualthinkingofhowtoviewevaluation.UFEisanapproach
notamethod,accordingtoPatton.Myquestion–areweconveyingtheconceptandthe
approachratherthanthesteps.Theevaluatorknowsthesteps–butyoudonotneedto
statethemasasteptothementeeorprimaryusersdonotneedtoknowaboutthat.Sojust
gothroughtheprocess.IwoulddothesamewithResCom.”
7
Aswasalreadynoted(paraabove),itisreallythefirstfourstepsofbothUFEand
ResComthatrequireourimmediateattentionandformthebasisofthetwoentities
workingintandem.Thesestepsarerelativelysimpleandstraightforward:whatarethe
uses(UFE);whatarethepurposes(ResCom);Whoaretheusers(UFE);Whoarethe
audiences(ResCom);Whatarethekeyevaluationquestions(UFE);whatarethemedia
andmethods(ResCom)
Showntogether,itistheposters/templatesthatillustratetherelatednatureofthese
stepsoratleasthaveprovensointhefewtimesofactualapplication.Infact,this
combinedapproachisakintoa‘silobuster’andavaluableframeworkfordecision
making!
“….It(UFEandResCom)doesthreethings–pushesprojectstomakeclearwhatthey
aredoing–whatISthisthattheyaredoing?(TOC);howdoIvalueitandmeasure
it/documentit(evaluation)andwhoisinterestedinthis…howdoIuseit
(communication)….”7
Ifitistrue(aswesuspect)thatitisthepersonwhoactuallyproducesthematerialor
writesthecasestudiesthatlearnsthemostabouttheprocessthroughthisexercise,
couldwerevisittheroleofthementorandfindawayforthementortolearntoproduce
his/herownmaterialsoradapttheexistingmaterialstohis/herownuse?
c)Roleofmentor–thementorrelationshiptomaterials
Lessonslearnedabouttheroleofthementorandthementoringprocessturnedoutto
beembeddedintheresponsetothematerials.DECIrevolvesaroundthebeliefthat
workshopstopromoteframeworksormethodologiesarenotusefulwithout‘justin
time’mentoringtohelpindividualsadaptnewprocedurestotheiruniquework
environment.Whilethisapproachtomentoringhasproventobeausefulone,atthe
sametimehandholdingshouldnotbeviewedasabottleneck.Itisimportanttoconvey
thatwhilepeoplecandothisworkontheirownitcanbebetterwithsupport.Perhaps
thereisawaytoprovidementoringasapackage–responsivetotheorganization’s
decisionastohowmuchandwhen.ThepotentialtouseUFEandResComishuge,and
severalreasonssupportitsapplication:itis‘internal’totheorganization;both
emphasizelearning;haveinherentflexibility(thementorsguidetheprocessof
learning);canbeusedanytimeduringtheprojectcycleandisoverall,non-threatening
asthereareno‘outsiders’,e.g.anexternalevaluator.
Thementor’sreadiness(orcapacity,asonementorpreferred)isthemostcomplex
becauseitnotonlydependsonthereadinessoftheorganizationandmenteebutalso
howadaptablethementorisabletobe.Itisbothaskillaswellasapersonalityissue
sincethementorneedsto‘connect’withthementee.Timeandpatienceareother
qualitiesofthementorthatwerefoundtobeuseful.Likethementee,thementorhasto
beopentolearning–understanddifferentcontextsandperspectives.Learningthrough
mentoringisinvaluableyeta‘luxury’becauseonlyonedonor(IDRC)hasbeenwillingto
workwiththismodel.
“Mentorneedstobeveryflexibleandneedtoallowpeopletostruggleabitandthose
mentorswhoaremoreprocessorientedareabetterfitratherthanbeingtaskoriented
andtogetthingsdone.”
7DiscussionwithRicardoRamírez.
8
d)Materialsandwebsite
TheResearchgateaccountsindicatethattherewereabout250downloadsofthePrimer.It
must,howeverbestatedthatexceptforsomePelicanexchangesorrequestsformaterial,
thePrimerandothermaterialswerenotactivelypromoted.Indeed,traffictothewebsiteis
low,perhapsindicatingthatitismoreofaprojectstorefrontandrepository8.
LinkstorelatedmaterialssuchasRAPID,useofmoretoolssuchas“storyofchange”,
blogsandecontributionswherementeescouldarticulatetheirownlearningwould
enhancethevarietyandsophisticationoftoolsavailable.Alsotheavailabletoolsare
bestforthosewhoarealreadyconvincedoftheapproach.Forexample,theprimeris
usefulbutsomebodyunfamiliarwiththeapproachneedssomesimplematerialsthat
combineboththeoryandpractice.Perhapsashortprimerandshorterillustrative
materialscouldbeanoption.
Toget‘converts’,thereisaneedforadvocacytoolsandsimplerwaystoexplainwhat
UFEis.OneoftheDECI1mentorsmentionedthatvisitingthenewwebsitedidnotgive
aneasyinsighttoResComorVeriComandhencethereisaneedforsimpler
explanations.Thewebsitewasdescribedas‘flat’andneededarevamp.
Acceptancewithintheacademiccommunityisalsoimportantandjournalarticlesonthe
UFEandResComframeworkswouldbringcredibility.Decisionswouldhavetobemade
astowhatextentastepbystepapproachshouldbepromotedand/oraprinciple(read
‘framework’)approachand/orexemplars.
e) Abrieflookatotherinitiatives
ThosefamiliarwithOMmentionedthatIDRCplayedamajorroleinsupportingthe
understandingoftheapproach,buildingacommunityofpractice,supportingan
interactivewebsiteandencouragingawidersupportforit.
TheChairoftheOutcomeMappingcommunity(HeidiSchaeffer)explainedthatyes,itwas
IDRCbackingthatprovidedtheoriginalhelpandsupporttoOutcomeMappingbutthe
waytheydiditiswhatreallyhelped.IDRCneverassumedorpretendedthatOutcome
Mappingwasanewtoolthattheyhadinventedbutinsteadstressedthatitwasa
frameworkforthinking,aworkinprogressthathadnotbeendevelopedbutcontinuedto
bedevelopingwithallthepeoplewhoparticipatedintheexperience.Heidiwentontosay
thatmorethananythingitwastheideaofOMbeingaframeworkorawayofthinkingwas
themostimportantaspectofitscontinuingsuccess.
ODI’sJohnYoungfeltthatitwasthesustainedcommitmentonthepartofthemany
peoplecomingfromtheoutcomemappingcommunitywhoalsogotembeddedinother
organizationsthatmadethedifference.HesummeditupbysayingthattheOutcome
Mappingsuccessisdueto:
• Theclarityofitsprinciplesandhighqualitymaterialsthatpeoplecanpickupand
runwithattheirownpace
• SustainedcommitmentfromIDRC
• Apreparednesstoadaptandsuitotherorganizations’needsandberesponsive
• Anactivelivingandlearningprocessgoingonthroughalong-termactivity
• Beingroutedinhumanbehaviorbutalsoremainingflexible
8InterviewwithRicardoRamírez.
9
Youngadmitsthathisownorganization(ODI)hasspentagreatdealoftimetryingto
figureouthowtobestpopularizeawayofdoingwork.Everyfouryearsorsotheystep
backandfigureouthowtomaximizelikelyuptake–whatcontractstolookforand
whichoneswouldbeimportanttotake.Theythenwriteconceptnotesandfloatthem
arounddonorstotrytogethired.
TheycallthisapproachStrategicallyOpportunistic,orashesays,fantastically
opportunistic:Thismeansthatfirsttheytrytouseconferencesandpublishmaterialsto
attractthosewhomightshowinterest.Oftenthisresultsinpeopleapproachingthemto
askforhelp.Theyarequiteselectivewiththeseprospectivecontractsandworkwith
peoplewhotheyfeelmightbe‘ready’foruptakeandhelpproducematerialsthatmight
laterbeaddedtothemix.9
TheRoma,hesaysisaninterestingstory.ODIhadalreadydoneanassortmentoftool
kitsandtrainingcourseswithResearchInstitutesforuptakebutthendecidedthatall
thesedisparatebitswerenotgoodenough.SotheyproducedRomathatisessentiallya
compilationofjustabouteverythingtheyhavelearned.
Romaresultedinaplethoraofrequestsforhelpfromcompletelydifferentclientssuch
asResearchCouncilsandUNagencies.TherewasahugeuptakefromwritingRoma.
ODIhadhopedtocomeoutwithaloadofpracticalexperiencetobuildtheirresource
baseandlessonsastohowtheirtoolshadbeenusedbutintheendtheirtoolsdidnot
getusedunderthesecontracts.Whatcameoutoftheworkwiththenewclientswasa
setofprinciplesidentifiedthroughtheRomadocument.ThroughthisprocesstheUN,
forexamplerealizedthattheywouldfirsthavetoturnthemselvesintoalearning
organizationbeforebeingabletoactuallyusetheresearchuptaketools.
f)Popularizingtheapproach–areweready?
DECI2mentorsremindedusthatuptonowDECIhasbeenfocusedontest-drivingthe
approachandnotpopularizingit.Somefeltitmightbeerroneoustoexpectustostart
“evangelizing”10theapproachsincethiswouldrequireaverydifferentsetofactivities.
HoweverothersfeltthatatthemomentthereisawindowofopportunityforDECIto
makeitsworkknownandhopefullyadopted.Thereisacurrentbuzzaround‘useof
evaluations.’Wecould/shouldcapitalizeonthisinterestbecauseneitherevaluatorsnor
decisionmakersreallyknowhowtomakeithappenwhileUFEprovidesaneasy
frameworktouse.
DECI1mentorsmentionedthatUFEtakestimeandisgearedtowardsorganizations
thatareseriouslylookingatchangeandreformratherthan‘paylipservicetolearning,
orwhohaveavestedinterestforastampofapproval.’Theapproachrequiresa
‘readiness’intheorganizationtolearn,spendtimeandbewillingtobepartofajourney
orprocess.11Manydonors/organizationsthatcommissionorexpecttoorganize
evaluationsareinterestedinusingtheOECDcriteria,haveformedtheKEQand
identifiedtheusers(bythedonorsusually)–suchorganizationswillnotbeinterested
9RecentlytheyworkedwithsomeUNorganizationswhotheyfoundwerenot‘ready’
butfirstneededtobuildlearningmechanismsintotheirownorganizationsbeforebeing
abletouptakethe
10InterviewwithaDECI2Mentor
11ThisechoesJohnYoung’sexplanationofODIexperiencewiththeUN
10
intheUFEprocess.Besidesmostevaluationmanagersdonothavethecapacitytodothe
UFEbecauseoftimeandskillconstraints.
Onementormentionedthattherearesomanyoptionsavailable,evaluatorswillnot
wanttoacquireacertainsetofskillsasrequiredbyUFEifthereisnodemandforit
fromthedecisionmakers.TheUFEisalsodependent(likepolicy)onawindowof
opportunity–manyoftheprojectsthatsuccessfullycompletedUFE(anduseditwell)
wereforcedtoseizetheopportunityastheywerefacingacrisissuchasdonor
withdrawal.Hence‘readiness’oftheorganizationisimportantforUFE.
Itmight,howeverbepossibletobuildacommunityofpractice–beginningwithasmall
cohortoffundedprojects(DECI2)andthensystematicallymovingtoothersthatwould
beintroducedtotheapproach.Asonementorstated,
“Oneeffectivewayisforthementorstokeepapplyingtheapproachintheir(extraDECI)assignments,talkingabouttheinnovativeapproachesANDdemonstrating
howdifferentand,ifpossible,moreeffective(ifthatturnsouttobe)theyareto
traditionalprocesses.Inaddition,ifthementors(andothersadoptingUFE,
ResCommandVericom)cantryandidentifyandpointouthowthoseapproaches
havebeenappropriatetotheregion’sculture,institutionalsystemsand
programs…(itwouldhelptopopularize)”
Blogs,socialmedia,mailinglistsandnetworksforcommunicationandevaluationare
toolsthatcouldbebroughtintosupportthiswork.Or,asODItriestodo–itmightbe
possibletotaketheideastoconferencesandintroduceUFE(andResCom)aspartofthe
teachingcurriculumatvariousuniversitiesorapproachtheVoluntaryOrganizationsfor
ProfessionalEvaluatorsorEvaluationnetworks.
Finallywhynotapproachdonors(includingIDRC)topromotetheframeworkacross
differentsectors(besidesICT4D)suchasclimatechange,forestryandsoonandlookto
researchinstitutesandinternationalresearchprojectssuchasthosefundedbyAusAid,
DFID,ACIAR(inAustralia),CIFOR(hasabigpresenceinIndonesia)usingeitherthe
communication‘entrypoint’ortheevaluationone.
4.Conclusion
ThediscussionoverthetwodaysclarifiedthattheDECIteamstillfeltitselftobeinthe
testingstagebutwasreadytocautiouslymoveforwardtoconsiderpopularization.The
suggestionstodothiswerevaried.InthebackgroundwasthehopethatIDRCmightbe
preparedtoassistinfurtherpromotingtheDECIapproach(asinoutcomemapping)
whileitwouldbetheDECIteamresponsibilitytofindtheirownmethods.Thesecould
includesimplifyingthematerialssothateachmentorcouldadaptaccordingtoneed;
testingtheuseofsocialmedia,blogs,update(andenliven)thewebsiteandconsiderthe
possibilityofdevelopingacommunityofinterest(COI)withpeoplecommittedtothe
UFEandResComapproach.Furtherthoughtsincludedapproachingtheacademic
community,writingarticlesforpublicationandsearchingotherdecision
makers/donorsinterestedinadaptingtheirevaluationuseandcommunication
practices.Intotaltherewasagreementthattheideaofcombiningevaluation
approacheswithcommunicationstepsisprovingtohavemeritandworthpopularizing
beyondthefewDECIpartners.
11
AttachmentOne:TermsofReference
ResearchStudy-DECI-2Learning&disseminationreview
InpreparationforourMay2016teammeeting,wewillreviewhowwecanintegrate
learninganddissemination.ThiseffortrespondstoObjectives1and5ofthisproject
(seeTextBox).
DECI-2objectivesrelatedtolearninganddissemination
1.Meta-levelaction-research:Todevelopandtest-driveacombinedapproachto
UFEandResCommentoring.
5. Sharing lessons: To communicate the DECI-2 project findings and training
approachtopractitioners,researchersandpolicymakers.
Forobjective#1,wehavedevelopedlearningsupportmaterials(modules,checklists,
reportingtemplates,webinars).Aswehavestartedtointegrateevaluationand
communicationintoahybridapproach,wehavebegunproducingadditionalmaterials
thataddressobjective#5:theVeriComdocumentation(2015),adrafte-Primer,anda
workingdraftforajournalarticle.
Thisreviewfocusesonrevisitingwhatwehaveproduced,reviewingwhetherthe
materialscanbecomeeffectiveleaningtools,reviewingourdissemination
productsandsuggestingastrategyfortheremainderoftheproject.
Possibleguidingquestions:
c) Wehavebeentest-drivingUFEandResCombymentoringresearchnetworks
andselectsub-projects.Towhatextenthaveourmaterialsbeeneffectivetoolsto
supportaction-research?Whathasworked;whathasnot;andwhatarethe
possibleexplanations?
d) Wehavesoughttoproduceon-linereferencetools,especiallywebinarsand
modules.Underwhatconditionshavetheybeenuseful?Whathavebeenthe
limitationsandbarrierstoeffectiveuse?
e) WehavebegunintegratingahybridapproachundertheVericomtitlethrougha
drafte-Primer(forpractitioners)andadraftjournalarticle(forresearchers).
Whatarethemeritsandlimitationsofthedrafts?Whatactionsareneededto
makethemmorerelevanttoeachaudiencegroup?
f) Wehaveastaticwebsitewithnosocialmediaorblogcomponents:what
variationsandopportunitiesareworthintegratingduringtheremainderofthe
DECI-2project?
NextSteps
1. InviteSonalandWendytoteamuptocoverthispreparatoryactivity,includinga
reviewoftheguidingquestions.
2. Co-PI’stoprovidesupporttotheprocessandtocommentondrafts
3. PrepareasummarypresentationbeforetheCapeTownmeeting
4. PresentthefindingsandsuggestionsattheCapeTownmeetingforgroup
discussion
5. Considertakinganactiveroleinrevisingthee-Primerandco-writingthearticle
(afterMay)
12
Approximatelevelofeffort:
a.Finalizethequestionsanddistributingthework:2dayeach
b.Read,review,analyzejointly:3dayeach
d.Prepareashortpresentation:1dayseach
Totalof12days,6forSonaland6forWendy
Approximatetiming:FebruaryandMarch:produceafulldraftbyearlyApril.
13
AttachmentTwo:QuestionnaireforMentors
ForDECIMentors:
Why:
a)assesstheDECIproductswithourmentorsandmenteestoseewhatworksandwhy
andwhatisnotsogoodandwhy-plusafewideasforimprovement
b)whatdoweneedtodotohelpgainmoreconvertstotheDECIVericomapproach
What:
Pleasereviewthequestionsbelow,jotdownafewpointsandemailtousbeforeApril7
sothatwecanfollowupwithaninterviewtogetamoredetailedordescriptiveversion
ofyourexperience(short30to45mins).
HereisalistofmaterialsfromDECI-1andDECI-2andwheretofindthem
DECI1
Website
Dropbox-UFE Dropbox–Casestudy
Dropbox
directory
Casestudies
Reporting
Casestudies
Conference
template
presentations
Checklistsand
modules
Primer
DECI2
Website
Dropbox
Dropbox
Dropbox–
Dropbox–
UFE
Rescom
VeriCom
Modulesfor
Reporting
Reporting
ModifiedVericom
Drafte-primer
ResCom
template
template
template
Checklists
Draftarticle
webinars
Thewebsite:evaluationandcommunicationinpractice.net
1. Whichmaterialshaveyouused–pleaselist.
2. Whatworked,whatdidnot?
3. Couldyouonascaleof1-5(with5beingthehighest)rankthematerialsyou
used?
4. Haveothersyoumayknowaboutusedthesematerialsandwhatwastheir
experience?
5. Tohelpusmoveforwardandtosharethismorewidelywewanttomakesure
wehavetherightcombinationofmaterialsandyourcommentsarewelcome.
14
• Whattoinclude
• Whatnottoinclude
• Whattoadd
6. Arewelabeling(UFE,Vericom,ResCome.g.)thematerialsproperly?
7. Topopularizeourapproach,anylessonsfromotherinitiativessuchasOM,SAS,
RAPID?
8. Howdowepopularizeintheregionandwhatcouldbetheroleoftheregional
mentors?
ActionPoints:
1 WelookforwardtoyourresponsebeforeApril7
2 Pleaseindicatewhenitwouldbeagoodtimetointerviewyou–ideallyafterwe
haveyourresponse!
3 Pleasesuggestmenteenamesandcontactdetailssothatwemaysendthema
shorterversionofthisq’airre.
THANKS!
SonalandWendy
15