Private involvement in education:
Measuring Impacts
Felipe Barrera-Osorio
HDN Education
Public-Private Partnerships in Education,
Washington, DC, March 31, 2010
The quest
1.
Ideally, we want to know what is the effect of private
participation in the provision of education
A.
Does donation into an school produce better learning
outcomes of students?
B.
Does a donation that goes to teacher training,
increases the pedagogic capacity of teachers and
better education outcomes of kids?
C.
Does a private funded computer program induce in the
long run higher labor opportunities for individuals?
D.
Does a Public-Private Partnership induce higher
learning outcomes?
The quest (continuation)
1.
Also, we want to know how we can improve design of
programs that will lead to improvement of donations
A. If the program donate to schools, what is the best
strategy to allocate resources: towards
management, or towards educational management
information systems, or towards teacher training?
B. Should we donate computers with specific
software for education or generic software?
C. Can we improve efficiency of private donation by
inducing involvement of the community?
Why is this important?
1.
2.
3.
4.
5.
Impact evaluation can provide reliable estimates of
the causal effects of programs
Impact evaluation can potentially help improve the
efficacy of programs by influencing design and
implementation
Impact evaluation can broaden political support for
programs
Impact evaluation can help in the sustainability of
successful programs and the termination of programs
that are a failure
Impact evaluation can help expand our understanding
of how social programs produce the effects that they
do
These questions are not easy to
answer
1.
2.
Impact evaluation is a new name for an old quest: What is the effect of
programs?
A good evaluation will have the following characteristics:
A.
A clear definition of the intervention: the concrete objective, what is
being modified, the new set of incentives, and to whom the
modifications apply.
B.
A description of how the intervention is expected to achieve the
final desired outputs: how the intervention will lead to the desired
result.
C.
A definition of the identification strategy that allows to attribute
causal effects between an intervention and a set of outcome
variables.
Definition of the program: objectives
1.
2.
Sometimes, the objectives are difficult to measure
A. For example, a program can have an objective
such as “social cohesion”
Solution: formulate a concrete, measurable
objective
A. For example, an objective can be to increase
learning outcomes of students
B. A measurable, concrete outcome can increase
accountability of a program
Definition of a program: an special
challenge for the private sector
1.
2.
Usually, the evaluation is not about the
private donation, but about the specific
type of intervention
For example, a private individual who
donate to schools for teacher training
A.
The evaluation is not about the dynamics of
the private sector into the schools, but
about the effect of teacher training
How the intervention will lead to the
desired result.
1.
2.
It is critical to understand the pathways in
which the program operate
An example: a computer donation to schools
A.
B.
C.
Is the program affecting the way teachers
teach?
Are the students using the computers?
What are the students using the computer
for?
Channels of transmission
1.
PPP contracts allow more flexibility in the
provision of education than in the public
sector

2.
Monitor critical areas of flexibility: payroll of
teachers, length of shifts, firing and hiring of
personnel
Usually private providers in PPP contracts
are chosen by an open bid process based
on quality criteria

Quality of providers: previous experience
Channels of transmission (cont.)
3.
A PPP contract can achieve an optimal level of
risk-sharing between the government and the
private sector

4.
The private sector may have higher standards in
the delivery of education services

5.
Very difficult to quantify
Analysis of educational outcomes like test scores,
internal efficiency, etc
PPPs can induce competition in the market for
education

Important to track entering and exiting students
Attributing causal effects
1.
Why is so difficult to find the effects of a program?
A. Suppose a computer program: donation of private
sector, schools receive them, teachers use them in
classrooms, students have direct access to computer
and software
B. Suppose that we follow two strategies:
a. First, we compare the students that receive the
program before and after the program
b. Second, we compare students that receive the
program versus students that did not receive the
program
The basic intuition: only data before
and after the program
Y
Impact
of the
program?
NO!
We need a
contrafactual
Time
t=1
after program
t=0
before program
Intervention
We need a comparison group….
Y
Impact
of the
program
Control
Time
t=1
after program
t=0
before program
Intervention
The basic intuition 2: only data after
the program….
Y
Impact
of the
Program?
Control
Time
t=1
after program
t=0
before program
Intervention
We need the right comparison group!
Y
Control
Impact
of the
Program?
NO!
At t=0, two
groups
were very
different…
Time
t=1
after program
t=0
before program
Intervention
Some possibilities to find or construct
the right control group
1.
2.
Prospective evaluation
A.
Randomization of benefits: a lottery to get benefits
B.
Randomization of entry: a lottery to determine order of entry
Retrospective evaluation
A.
Regression discontinuity analysis: groups are found using an
index (e.g., if an individual score above certain number, she
receives the benefits)
B.
Differences in differences: information at baseline and follow
up for a group that received the program and for a group that
didn’t
C.
Propensity and matching estimators: very detail data at some
point before the program
When do we apply each method?
1.
Ideally, randomization is first best
A.
B.
2.
RD
A.
3.
Individual / geographic randomization: program is
a pilot and not universal
Phase-in randomization: program is universal and
implementation is done in steps
Program is targeted using an instrument
DD and Matching:
A.
If there is good amount of information and the
program is not universal
Who should be pay for an
evaluation?
1.
2.
If the evaluation produces a public good the
evaluation should be finance by governments
/ multilateral institutions.
If the evaluation produces a private good
(e.g., the results are appropriated by the
private agent), the evaluation should be
finance by the private agent
A. If so, what are the incentives to provide a
real, strong evaluation?