Diversity of
Methods in Impact
Evaluations
IPDET 2014
Rob D. van den Berg
Director
June 13, 2014
Increasing demand for impact information
Political:
 Demand in parliament, in public debates, media
Audit/accountability:
 Need to demonstrate that public funding has
“impact”
Learning:
 What works (and what does not)
 Why does it work (or why does it not work)
 How to achieve more impact
Role of independent evaluation
 Multilateral organizations tend to have central internal
evaluation units
 They tend to be independent of management and to
report directly to the governing body, to ensure
impartiality and objectivity
 Independence in itself does not guarantee good
evaluations – for that we need credible approaches and
methods
 Credible approaches and methods may lead to excellent
evaluations from an accountability perspective, but does
not guarantee learning
 Utility of evaluations is the third and last pillar of
independent evaluation to enable learning
Impact of the GEF (1)
 Impact in general (based on OECD/DAC*): Positive
and negative, primary and secondary long-term
effects produced by an intervention, directly or
indirectly, intended or unintended
This definition is used by the DAC Evaluation Network,
the Evaluation Cooperation Group of the International
Financial Institutions and the UN Evaluation Group
It includes 8 perspectives on impact
 The goal of the GEF is to achieve environmental
impact, which can be reached through stress
reduction and change in environmental status
 Over time stress reduction leads to improved
environmental status
Impact of the GEF (2)
 The time dimension of the GEF impact is identified
in:
 Direct, immediate impact
 Long term impact
 Impact can be measured at different scales:







At single sites
At multiple sites
In landscapes or seascapes
Market-wide
In local or national administrations
In regions
Global (worldwide)
System-level
impact
Impact of the GEF (3)
 Impacts may have a local and a global significance
 Saving a unique local species has global impact; it has local
impact as well, as it may be a source of eco-tourism income
 Globally significant impacts have local impact as well, but not
all local impacts have global significance
 Social and economic impacts are studied to find out
whether behavior changes reduce or enhance threats and
whether they lead to sustainable development
 Andy Rowe: environmental interventions are about
interactions between humans and the environment and
thus have two evaluands
 Anybody kept count?
Impact of the GEF (3)
14 impact perspectives
that can be measured at
7 different scales
Generic Theory of Change
Generic Theory of Change
Project level and performance
evaluations
South China Sea Impact Evaluation
Social Network Analysis
Social Network Analysis
Methods & Tools (1)
• Social network analysis is a tool used to describe and
examine the interactions among actors in a defined
population, as well as to explore the patterns that
may emerge from these interactions
• It consists of:
–
–
–
–
–
Identifying actors
Scoring actors for relationships
Survey / interviews to verify
Analysis
Presentation
Methods & Tools (2)
• Relationships are scored in an Excel spreadsheet
• Visualization software UCINet 6.289 / NetDraw
2.097 were used to present the graphs
Impact Evaluation of GEF Support to Climate
Change Mitigation
• Four emerging economies: China, India, Mexico,
Russia
• 18 completed projects
Key questions:
• What have been the GEF contributions to GHG
emission reduction and avoidance?
• What has been the progress made to market
change leading towards market transformation
for climate change mitigation?
• What were GEF contributions in the process?
Contributions to Emissions Reductions
Most GEF projects examined reported
direct and indirect emission reductions.
Direct impacts:
GHG emission reductios
> 1 Mt CO2 äq p.a.
>100,000 t CO2 äq p.a.
> 50,000 t CO2 äq p.a.
> 30,000 t CO2 äq p.a.
> 10,000 t CO2 äq p.a.
> 1,000 t CO2 äq p.a.
< 1,000 t CO2 äq p.a.
not measureable
# projects
1
3
3
3
1
4
1
2
Countries
China
China, India (2)
Mexico (2), China
India, Mexico (2)
India
India (2), Russia (2)
China
Mexico, China
project short names
TVE
CRESP, AE, Biomethane
Ilumex, Landfill gas, CREDP
Energy Efficiency, Agriculture, BRT
CBM
PVMTI, Hillly Hydel, Buildings (2)
FCB II
Wind, FCB I
Indirect Emission Reductions
Country
short name direct GHG,
indirect impact (e.g. triggered or follow-up investment in energy
p.a., in 1000 t efficiency, MW installed)
CO2 äq/yr
China
TVE II
2.049,00
China
CRESP
971,30
India
Alternate Energy
660,00
India
China
Biomethane
CREDP
236,38
62,95
technologies were entered into national technology catalogue and some
made mandatory. Estimated savings: 10 Mt p.a. in coking, cement and
brick making industries respectively, 1 Mt in foundries
60 GW of wind power capacity have been installed but not fully
attributable to CRESP
31.000,00
18.3 GW wind / 1151 MW of solar today in India; project also
influenced the small hydro sector and led to significant installations in
four southern states
201 GW of waste to energy projects
market change, other rural PV programs picking up; ICR claims 5,6 GW
of wind and 80 MW of solar PV as indirect impact of this program (i.e.
status of RE deployment in 2006).
5.220,00
Mexico BRT
Mexico Ilumex
Mexico Agriculture
35,75
84,90
30,00
follow-up projects in other countries (are not counted)
CFLs have been phased out in Mexico in 2010
subsidy program was continued; 808 systems already by end of the
project
India
Energy Efficiency
34,00
unable to assess due to lack of data
India
PVMTI
India
CBM
India
Hilly Hydel
Russia
Russia
6,30
indirect impact
after project
completion; in
1000 t p.a.
32.000,00
550,92
11.000,00
35,00
3.400,00
15,00
helped develop private sector; effects twice as large as Alternate Energy
(GEF
ID 76)
one replication
project with 200.000 m³/d of methane
1.200,00
1,90
basis for calculation is total SHP installations in India
4.020,00
Vladimir
3,00
not quantifiable
Education
2,20
14 follow-up investments
14,48
500,00
28,27
Generic Theory of Change
GEF Theory of Change Framework :
Mechanisms to Broader Adoption
 Sustaining: continue using outputs beyond the actual
project
 Mainstreaming: Information, lessons, or aspects of a GEF
initiative are incorporated into a broader initiative such as
policies, institutional reforms, and behavioral
transformation
 Replication: reproducing at a comparable scale, often in
different geographical areas
 Scaling-up: expanding to address concerns at larger
geographical, ecological or administrative scales
 Market change: GEF activity catalyzes changes in the
market, through stimulation of significant demand or
reduction of the more polluting technology or practice in a
significant share of the market
Progress to Market Transformation
and Mechanisms at Play
GEF ID
Short name
Impact rating
Sustain
Mexico
575
Ilumex
High
x
x
x
Mexico
1155
BRT
High
China
446
CREDP
High
China
622
TVE II
High
China
943
CRESP
High
325
CBM
Significant
India
370
Biomethanation
Significant
India
386
Hilly Hydel
Significant
Mexico
784
Landfill gas
Significant
India
76
Alternate Energy
Significant
India
112
PVMTI
Moderate
Mexico
643
Agriculture
Moderate
India
404
Energy Efficiency
Moderate
Russia
1646
Education
Moderate
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
India
x
x
x
x
x
x
x
x
x
x
x
China
941
FCB I
Unable to assess
China
2257
FCB II
Unable to assess
Russia
292
Boilers
Low to negligible
Mexico
1284
Wind
Low to negligible
x
Replicate
x
Scaling-up
x
x
x
x
Market Change
 Four different types of market changes:
 Improved product quality
 More and better suppliers
 More demand for the sustainable energy technology
/ practice
 Lowered incremental costs
 14 out of 18 projects led to change in one or
more of these dimensions
 The analysis demonstrates that there are many
forms by which projects can improve markets
Types of Market Change
Country
GEF ID
short name
product /
technology
quality
Alternate Energy
more and
better
suppliers
India
76
India
112
PVMTI
Russia
292
Vladimir
India
325
CBM
India
370
Biomethane
India
386
Hilly Hydel
India
404
Energy Efficiency
China
446
CREDP
x
x
Mexico
575
Ilumex
x
x
China
622
TVE II
x
Mexico
643
Agriculture
x
x
Mexico
784
Landfill gas
China
941
FCB I
China
943
CRESP
x
x
Mexico
1155
BRT
x
x
Russia
1646
Education
Mexico
2222
Wind
China
2933
FCB II
count:
more
demand
cost
reduction
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
8
x
12
7
7
Progress towards impact with causal links to GEF
projects
 Broader adoption of technologies, approaches
and strategies with causal links to GEF projects
was observed in 17 cases
 Projects demonstrating high progress towards
impact are those which have adopted
comprehensive approaches to address market
barriers and specifically targeted supportive
policy frameworks
Counterfactual Analysis
 Establishing causal links to GEF support is not
enough to claim a contribution to the process
 Need to discount rival hypotheses (other causal
links could have taken over)
What would have happened in the absence of GEF
Support?
 Qualitative Comparative Analysis, based on and
verified by:
 Contextual analysis
 Opinions of diverse key informants
How likely is it that the activity would have
taken place without the GEF?
Very
unlikely
2
Mexico: BRT (GEF #1155), Agriculture (GEF
#643)
6
India: Energy Efficiency (GEF #404), CBM
(GEF #325), Biomethanation (GEF #370),
PVMTI (GEF #112)
Mexico: Ilumex (GEF #575), Wind (GEF
#1284)
5
China: FCB I+II (GEF #941 and GEF #2257),
RESP (GEF #943)
Mexico: Landfill gas (GEF #784)
Russia: Education (GEF #1646)
Very likely
4
India: Alternate Energy (GEF #76)
China: TVE II (GEF #622), REDP (GEF #446)
Russia: Boilers (GEF #292)
N.N.
1
India: Hilly Hydel (GEF #386)
Not likely
Likely
..but slower
4
China: FCB I+II (GEF #941 and
GEF #2257), RESP (GEF #943)
Mexico: Landfill gas (GEF #784)
3
India: Alternate Energy (GEF #76)
China: TVE II (GEF #622), REDP
(GEF #446)
…but not at
international
standards
2
China: RESP (GEF #943)
Russia: Education (GEF
#1646)
What can be attributed to GEF?
Role
Number of projects
Catalytic: without GEF activities would not have started
8
GEF has speeded up existing progress toward impact
7
GEF has enhanced quality of the progress toward impact
2
Note: “Very unlikely” and “Not Likely” have been combined as catalytic role of the GEF
Attribution / Contribution
• GEF is a partner in environment/development, so by
definition “contributes”
• The many pathways to impact point to many
possible cases:
– GEF supported intervention is a necessary but not
sufficient condition for progress toward impact
– GEF supported intervention is a sufficient but not a
necessary condition for progress toward impact
– GEF supported intervention is neither necessary nor
sufficient, but was the “spark that lights the fire”
• QCA is able to provide counterfactual analysis on
contribution!
Thank you
[email protected]
www.gefeo.org