The Comparative Method for
Assessing Impacts and Cost
Effectiveness of Development
Assistance Programs
Paul Clements
Western Michigan University
It is possible to substantially enhance the
impacts of development assistance with
evaluations employing the comparative method.
This depends on two, related arguments:
1. An analytic argument
2. An organizational or structural argument
2
The analytic argument - overview
Linear analysis, the method for rigorous impact evaluations (e.g.
with randomized controlled trials) is grounded in situational
analysis.
Situational analysis can be rendered more rigorous through the
comparative method with synthesis anchored in cost
effectiveness.
Applying lessons from evaluations always requires (an act of)
synthesis.
Weak synthesis in development practice is due about equally to
analytic shortcomings and to weak incentives (and the melding
of these in organizational cultures).
Stronger synthesis (hence more cost effective development
interventions) can be achieved through a particular organization
of evaluation based on the comparative method.
3
The organizational/structural
argument - overview
(See “Reducing World Poverty by Improving Evaluation of
Development Aid,” by Paul Clements, Thomaz Chianca and Ryoh
Sasaki, American Journal of Evaluation, June 2008, pp.195-214.)
Aid evaluation today, e.g. under the DAC criteria or results-based
approaches, has uneven quality and inconsistent units and
evaluative judgments even for similar projects.
It is controlled by donor and implementing agencies with incentives for
positive bias, so evaluative conclusions tend to be too narrow (not
addressing cost effectiveness) or positively biased, and both of
these amount to positive bias.
Evaluations are usually framed in individual, project specific terms.
Together these factors undermine learning and accountability in the
development assistance community.
4
The proposal - overview
It is possible to dramatically improve the quality of
evaluations, the discourse of the development community,
and the impacts of aid interventions, by professionalizing
aid evaluation.
A professional association (Evaluation International) with
guidelines and standards for evaluation based on impacts
and cost effectiveness.
The association’s evaluations also:
- explain impacts in terms of project design and
implementation,
- address strategic issues in the sector, and
- employ the comparative method.
Units and evaluative judgments are consistent between
evaluations.
5
The Comparative Method
Introduction
Cost effectiveness is the correct analytic anchor for the
discourse of development management (we want to
maximize impacts given our resources).
Linear Analysis
Rigorous impact estimates based on the well known formula:
(Ben t1 – Ben t0) – (Control t1 – Control t0)
Sets of statistical procedures and analytic requirements for
valid and precise results.
Linear analysis depends on situational or constructive analysis
for setting up the design and for applying the results.
The comparative method builds on and adds rigor to
situational or constructive analysis
6
The Comparative Method:
situational/constructive analysis
Integrates sense of mechanism, of causal
relations, vis a vis an objective
Involves our sense of the relative importance of
several contributing factors
Involves triangulation
7
The Comparative Method:
Forms of Reasoning (a)
Given similar known complex
phenomena a, b, c and a new one, x
a > b > c ; where does x fit in this range?
8
The Comparative Method:
Forms of Reasoning (b)
Consider two microfinance programs in environments with:
Program A
Program B
High and variable inflation
Low and steady inflation
Scattered, sparse population
Dense population
Largely subsistence-based
society
Fully monetized society
Low GDP growth
High GDP growth
9
The Comparative Method:
Forms of Reasoning (c)
Two microfinance programs:
Program A
Program B
Weak training
program
Strong training
program
Weak sense of
mission
Strong sense of
mission
10
The Comparative Method:
Forms of Reasoning (d)
Estimating future impacts
Evaluation of condition C at time t1
Ct1 – Ct0 = x
what is Ct2 – Ct1?
11
The Comparative Method:
Forms of Reasoning (e)
Routine, every-day forms of reasoning
Employed in complex integrated judgments (hence
constructive)
When grounded in a single value – cost effectiveness –
exceed lay person’s normal analytic capacity
Can be systematized
12
The Comparative Method
Integrated
To evaluate a primary health care project:
Assume 20 evaluations of completed primary health care projects,
more and less rigorous, all estimate impacts and cost effectiveness
All explain impacts and c. e. in terms of the project’s design and
implementation and characteristics of the target population and of
the context or environment.
- Evaluator locates the project on the multi-dimensional spectrum
established by these earlier evaluations
- Evaluator investigates hypotheses based on this placement
- Analysis of baseline and counterfactual
- Sample survey to estimate changes in conditions
- Estimates future impacts and costs
- Inferences informed by technical literature and prior evaluations as
well as survey results, project records, etc.
13
The Comparative Method
How to Develop and Apply it
This can be achieved, rendered rigorous, and
generalized through the establishment of a
professional association along the lines of
associations of accountants and auditors.
Proceeding on a sector basis:
Consistent units, consistent evaluative
judgments given similar evidence.
14
The Comparative Method
Supports Learning and Accountability
Given consistent estimates of cost effectiveness
- Better portfolio management by donor and
implementing agencies
- Stronger within- and across-sector learning across
the development community
- New basis for personnel management
- More cost effective resource allocation
15