Quality of Economic
Analysis in World Bank
Education Projects
Harry Anthony Patrinos
March 6, 2008
Why Economic Analysis?

Improve quality of portfolio


1.
Research indicates that:
Strong relation between quality of cost benefit/cost-effectiveness
analysis & project outcomes

2.
3.
Reduce unsatisfactory projects under supervision
indicative of importance of good quality analysis at design
Quality of fiscal impact analysis, beneficiary assessment, poverty
analysis & overall economic analysis related to performance
Poorly prepared projects more likely to perform worse

Bank-wide, projects rated poor 4X more likely to be unsatisfactory than
those rated good – consistent across regions
Vawda, A., P. Moock, J.P. Gittinger, H.A. Anthony Patrinos. 2003. “Economic analysis of World Bank education projects and project
outcomes.” International Journal of Educational Development 23: 645–660.
Evolution of the Application of
Economics in Education Projects
“Rate of
Return”
Not
Applicable
“Sector” Rates
of Return
Pre-1990s ECON
Reports
1992
“Sector” RoR
and “ProjectSpecific” C/B
Priorities and
Strategies,
First Economics
1995
of Education
Course
Project
Specific
RoR
Ranking
Education
Projects
2000
2008
Overall Ranking of Projects in
the Education Sector Improved
100
CY93
80
FY98
60
40
20
0
Good and
Acceptable
Marginal and
Poor
Recent Performance
Good &
acceptable
100
80
Good &
acceptable
Good &
acceptable
60
40
Marginal &
poor
Marginal &
poor
Marginal &
poor
20
0
2005
2006
2007
Recent Performance
2005 and 2006 no better than early 1990s
 Fortunately 2007 performance as good as late
1990s
 However, too many are just acceptable

Too Many Projects are Merely
Acceptable
100
80
2007
2005
60
2006
40
20
2006
2007
2005 2006
2005
2007
0
Good
Acceptable
Marginal
Characteristics of FY07
Education Projects

28 loans approved

$2.0 billion



$507 million IBRD; $1.5 billion IDA
(1 project for additional funds, no PAD, so not included)
Assessment




7 (26%) good
18 (67%) acceptable
2 (7%) marginal
None poor
10 Dimensions of Economic
Analysis
1. Linkage with ESW
2. Quantitative analysis of alternatives
3. Financial & fiscal impact assessment
4. Cost-benefit and cost-effectiveness analysis
5. Sensitivity analysis
6. Institutional analysis
7. Poverty impact analysis
8. Environmental assessment
9. Outline of economic performance indicators/criteria
10. Overall project justification
Table 1 . Ranking of Characteristics in FY07 Education PADs and Project Documents
Projects (#)
Characteristic/Ranking
Overall quality
1
2
3
% in Category
4
1
2
3
4
7
18
2
0
26%
67%
7%
0%
Project coherently set in ESW and CAS
25
2
0
0
93%
7%
0%
0%
Rationale for public sector involvement
7
12
8
0
26%
44%
30%
0%
Evidence of consideration of alternatives
11
10
6
0
41%
37%
22%
0%
Fiscal impact analysis
15
7
5
0
56%
26%
19%
0%
Quality and clarity of economic analysis
9
6
12
0
33%
22%
44%
0%
Clarity of price and quantity assumptions
0
13
14
0
0%
48%
52%
0%
Institutional risk analysis
17
10
0
0
63%
37%
0%
0%
Beneficiary assessment and poverty analysis
15
11
1
0
56%
41%
4%
0%
Testing for learning performance*
15
5
5
0
60%
20%
20%
0%
1 = Good. 2 = Acceptable. 3 = Marginal. 4 = Poor.
*In two projects it was judged that testing for learning performance was not applicable.
Good Practice: Overall Quality
Ecuador Inclusion and Quality Education
 Guatemala Education Quality and Secondary
Education
 Vietnam Second Higher Education

Good Practice: Consideration of
Alternatives
Argentina Lifelong Learning and Training
 India Vocational Training Improvement
 Nigeria Science and Technology Education PostBasic

Good Practice: Fiscal Impact
Analysis
Burundi Education Sector Reconstruction
 Haiti Education for All
 Kenya Education Sector Support
 Nepal Second Higher Education

Good Practice: Quality and
Clarity of Economic Analysis
Argentina Lifelong Learning and Training
 Ecuador Inclusion and Quality Education
 Guatemala Education Quality and Secondary
Education
 Nepal Second Higher Education
 Vietnam Second Higher Education

Testing for Learning
Performance
3/5 of projects include provision for testing &
judged good
 Another 1/5 judged to acceptable

Testing for Learning
Performance
3/5 of projects include provision for testing &
judged good
 Another 1/5 judged to acceptable
 An example of a good approach is:


Bangladesh Third Programmatic Education Sector
Development Program – support annual student
assessments (based on internationally recognized
assessments such as TIMSS) piloted in Grades 8 and
12
Testing for Learning
Performance



3/5 of projects include provision for testing & judged
good
Another 1/5 judged to acceptable
An example of a good approach is:


Bangladesh Third Programmatic Education Sector
Development Program – support annual student assessments
(based on internationally recognized assessments such as
TIMSS) piloted in Grades 8 and 12
Several projects support development of national testing
programs

Georgia Education System Realignment and Strengthening
Project – help to establish a system of student assessment so
as to identify difficulties in student learning and develop remedial
programs accordingly
Type of Assessment by Fiscal Year
100.0%
100.0%
88.9%
90.0%
80.0%
75.0%
72.2%
71.4% 71.4%
70.0%
60.0%
55.6%
50.0%
50.0%
50.0%
50.0%
44.4% 44.4%
42.9%
41.7%
40.0%
30.0%
22.2%
20.0%
16.7%
8.3%
10.0%
0.0%
0.0%
0.0%
0.0%
2002
2003
2004
Fiscal Year
2005
2006
School-managed assessment
National assessment
Regional assessment
International assessment
Impact Evaluation


FY06: Only 4 had any discussion of evidence or plan for
evaluation
FY07: 2/5 (10 of 27) projects make reference to
proposed impact evaluation







In several it was clear that a rigorous impact evaluation
envisioned
LAC: Argentina, Guatemala, Haiti
SAS: Bangladesh
ECA: Bulgaria
AFR: Kenya
EAP: Vietnam
In others, however, unclear just how rigorous an evaluation was
proposed
Towards Better Economic
Analysis




CBA is still a potentially important tool in the economists’
arsenal
Too many projects lack adequate economic analysis
More work needs to be done on analyzing alternatives
for project
Important research agenda for future is to measure
effects of interventions on learning outcomes
Today’s Agenda
Appropriate public sector roles
 Project versus sector justification
 Impact evaluation
 Introduction to incorporating learning
outcomes

Today’s Agenda

Appropriate Public-Private Roles: Emmanuel Jimenez

Project versus Sector Justification: Price Gittinger

Keynote Speaker: George Psacharopoulos

Impact Evaluation: Felipe Barrera

Impact Evaluations in Difficult Circumstances: Tazeen Fasih

Incorporating Learning Outcomes in Project Objectives
Panel: Eduardo Velez, Elizabeth King, Marguerite Clarke,
Jee-Peng Tan, Robin Horn