MScEPS EC8014: Economic Evaluation:
Theory, Techniques and Applications
Infrastructure and
Environment
Dr Edgar Morgenroth
Associate Research Professor
Economic and Social Research Institute
Adjunct Professor
Trinity College Dublin
[email protected]
“In the US no one can market a prescriptive
medicine for male pattern baldness without
evidence that it is “safe and effective”.... Few
public actions, even those of tremendous
importance, are ever evaluated to a standard
required of even the most trivial medicine.”
Pritchett, 2002
 “the value of infrastructure is not equal to its
cost” Pritchett (1996)
2
Outline





Why evaluate?
Evaluation methodologies
Key parameters
Examples
‘Accompanying Measures’ – institutions,
implementation, optimism bias.
3
Why Evaluate?






Resources are limited;
One can think of an almost unlimited number of desirable (at
least on the face of it) projects;
Governments do not have unlimited budgets;
Governments need to prioritise;
The economic approach to prioritisation is to do so on the
basis of the most efficient allocation of resource;
Evaluation of public projects is different to evaluation of
private projects – correct prices, market failure, taxation.
4
Not all infrastructure is
created equal







Some types of infrastructure investment have a higher return
than others.
This is related to the adequacy of the existing stock –
eliminating bottlenecks or other capacity constraints has the
highest return.
The return to infrastructure investment is also related to the
relevance of the infrastructure for the economy – presidential
palaces tend to have no impact.
Core infrastructure has been found to have the highest return
– energy, telecoms, transport and water.
Lower returns on education, health and public buildings.
With limited resources, prioritisation is even more important.
The lawnmower approach to cutting capital (and current)
expenditure is an expedient but flawed solution!
5
Environment





Just as not all infrastructure is created equal
there are differences across environmental
investments.
The cost of reducing carbon emissions differs
across different technologies.
What is the net benefit of avoiding different
types of pollution?
Habitat protection
The environment should also be considered in
6
infrastructure projects
Infrastructure Investment in
Ireland




In common with all countries, there has been public
investment in Ireland since the foundation of the State.
This has varied in scale and in scope, reflecting economic
cycles and policy focus.
As a less developed EU member Ireland received EU
Structural Funds.
In recent years capital expenditure has fallen but is still
significant.
7
% of GDP
General Government GFCF Average
1970-2013
4.0%
3.5%
3.0%
2.5%
2.0%
1.5%
1.0%
0.5%
0.0%
Source: Own calculations using data from EU DG-ECFIN AMECO database.
8
Evaluation



In order to prioritise we must evaluate.
Typically evaluation is cheap relative to the overall
costs of any policy instrument e.g. the cost of the exante evaluation of the last NDP 2007-2013 was
0.0002% of the total planned expenditure!!
So why is there so little hard evaluation?
9
Who Plans and Evaluates
Infrastructure Investment in Ireland?




Current situation involves promoting agencies in
charge of evaluation – some oversight by the relevant
government departments and the department of
finance/public expenditure and reform.
It has been (is?) government policy not to publish
evaluations or to withhold key aspects of the
evaluation.
The public is usually the only party that does not have
full knowledge of a project/programme.
In the past public investment in Ireland was
significantly co-financed by the EU Structural Funds –
the EU insisted on evaluation and planning.
10
Planning & Evaluation

1.
2.
3.
4.
5.
Why did the EU insist on planning and evaluation?
Not all investments have the same return – evaluation helps
identify those with the highest return;
Investments should relate to economic needs – where are
the constraints?
Different investments may be complements or substitutes;
Investments typically run over several years – multi-annual
plans give funding certainty.
Agreed plans are easier to evaluate ex-post, than ad-hoc
investments.
11
Institutions and Public Investment –
the impact of corruption



“Any project evaluation can be disrupted if
corruption and bad regulation distort the
decision making process” (Florio and Myles,
Fisc.Stud. 2011)
Low-quality governance leads to higher level of
public investment (Kiefer and Knack,
Rev.Econ.Stat. 2007) .
An increase in public investment can raise the
level of corruption (Hanousek and Kocenda,
Fisc.Stud. 2011)
12
The political economy of evaluation


Pritchett (2002) constructs a simple political
economy model where projects/programmes are
support by “advocates” who are more committed
to pursuing their project that the general public
and where the latter is split into three groups
according to their attitudes towards the project.
He finds that except in the case where advocates
know that the project will have the desired
outcome, they will prefer not to evaluate the
project i.e. they will prefer ignorance!
13
Infrastructure Institutions in Ireland



Current situation involves promoting agencies (a
large and very heterogeneous group) in charge of
evaluation – some oversight by the relevant
government departments and the department of
finance/public expenditure and reform.
It has been (is?) government policy not to publish
evaluations or to withhold key aspects of the
evaluation (where is the CBA on water meters?).
The public is usually the only party that does not
have full knowledge of a project/programme.
14
Evaluation Strategies: General
Considerations










Evaluation can take place at different points in the project life cycle and at
different scales:
Individual project
Programme of investment
Ex-post (after the investment has been put into place)
Ex-ante (before the investment is put in place)
Mid-term (during a programme of many individual investments)
Short-run impact
Long-run impact
Indirect and induced effects.
There are many evaluation methods. All have strengths an weaknesses
related to what, when and which impact(s) is to be evaluated i.e. The
purpose of the evaluation should determine the method that is to be
used.
15
Evaluation Methodologies





A wide set of evaluation methodologies could be used:
Cost-benefit analysis (CBA), cost effectiveness analysis, multicriterion decision analysis (MCDA), case studies, simple
econometric modelling, input-output models (I-O),
computable general equilibrium models (CGE), small macroeconometric models, large macro-econometric models, time
series modelling (e.g. ARIMA or VARs).
Some methods are focused on individual projects (micro)
while others are focused on all investment (macro)
Each method has advantages and disadvantages - one
method on its own may net be enough!
The methodology should be able to measure the impact in
relation to the objectives.
16
Evaluation Strategies






Macro-evaluation – evaluate the impact of all the
investments together.
Wider coverage including spillovers.
Data is often more readily available.
Fully specified models (general equilibrium or econometric)
have the advantage that they capture direct and indirect
benefits.
This has the disadvantage that it will not be able to identify
the poorly performing projects/measures.
This requires the use of a model.
17
Evaluation Strategies






Micro-evaluation
Detailed analysis of specific projects or programmes – often
identifies why certain projects should not go ahead – not
popular but very necessary!
There is also evidence that this type of analysis is sometimes
badly done so that a project looks good.
It is difficult to get an overview of total impact (hence the
need for macro analysis).
Micro analysis is a significant task for large programmes of
investment.
Background research on particular areas is important.
18
Input-Output Models
Sets out the interrelationship between different sectors of
the economy – supply and use of output.
Advantages: straightforward (I-O tables are published for most
countries), usually very detailed.
Disadvantages: static – short-run effects only, no structural
change, lacks behavioural aspects, constant marginal product,
double counting.
Examples: Beutel, (2002), very popular with consultants!

19
I-O tables
Mining &
Quarrying
Mining &
Non-Metallic Construction .... Total
Quarrying Minerals
111
286
240 ...
3,907
Non-Metallic
Minerals
15
234
1,225
...
2,022
Construction
11
11
1,708
...
13,048
....
...
...
...
...
...
3,907
2,022
13,048
1.332
1.685
1.703
Total
Output
Multiplier
20
Computable General Equilibrium
Models
Related to I-O Models but involve a detailed set of
equations describing the economy (behavioural).
Advantages: Often very detailed theoretically based
structure, don’t need time series data, some ready
made models can be used.
Disadvantages: Usually static => best suited to evaluate
short-run effects, takes some effort to set up,
parameters are usually drawn from other studies or
assumed and calibrated to one years data.

21
Single Equation Econometric
Analysis
Advantages: Straightforward to implement,
estimated using data.
Disadvantages: Partial equilibrium, often
simplistic, dynamics often not well specified
(no distinction between long-run and shortrun impacts as channels are not modelled),
usually focused on ex-post analysis.
These are better suited to provide supporting
analysis.
22
Example

These inputs are used to produce output, Y.
Production function is given as:
Yt  f K t , Lt , KGt
(1)


where the t subscripts denote time.

to estimate (1) we need to choose a functional form. The most common
choice in the literature has been the familiar Cobb-Douglas:
(2)
 

Yt  At K t Lt KGt
where a has been added to represent the state of technology.
differentiating the production function with respect to infrastructure yields:
  MPKG ,t
KGt
Yt
(3)
where MP denotes the marginal product of infrastructure
23
Elasticities with respect to
Infrastructure
0.80
0.60
0.73
0.59
0.40
0.20
0.33
0.25
0.10
0.00
-0.07
-0.09
-0.19
-0.20
-0.40
-0.48
-0.60
Output
Cost
TFP
Interpretation: A 1% increase in infrastructure increases
output by 0.25% on average. Outer bounds correspond to two
standard deviations from mean.
Based on parameters taken from studies on the economic impact of infrastructure
in Greece, Ireland, Italy, Spain and Portugal.
24
Time-Series Models - VARs
Advantages: Reasonably straightforward to
estimate, no need to specify a theoretical
model, very good on dynamics.
Disadvantages: Lacks theoretical foundations,
not terribly good at identifying long-run
structural impacts, simplistic aggregate
relationships only (parsimony), data hungry
(ideally quarterly data).
25
Small Macroeconometric Models
HERMIN (27 EU Countries plus some others e.g.
Turkey)
Advantages: Theory consistent, parameters usually
estimated using data, more comprehensive than
single equation approaches, long-run and short run
impacts.
Disadvantages: Need to simplify (aggregate), can be
misleading if not properly understood (maintained
assumption of infrastructure deficit implies that
investment always has a positive impact).

26
HERMIN
Consumption function
Wage bargaining
Factor demands
Expenditure (GDE)
Income (GDI)
[6] Behavioural and
identity equations
GDP
Output (GDP)
[1] National
accounting
Framework
Behavioural
equations
influenced
by theory
Manufacturing (T)
Identities adding up,
definitional & closure
productivity
Building (BC)
PSBR
Output = Expenditure used
to determine net trade balance
(NTS = GDP - GDA)
where GDA = C + G + I + DS
HERMIN
[2] Output
Utilites
Market services (N)
Other market services
[5] Model as
integrated system
Agriculture
Agriculture (A)
Output = Income
used to determine
corporate profits
(IYC = GDP - YWI)
Fishing
General Government
Non-market services (G)
Public sector
Private consumption
Borrowing requirement
Public consumption
Debt accumulation
[4] Income
[3] Expenditure
Investment
Stock changes
Corporate sector
Household sector
Health
Education
Revenue
Expenditure
Forestry
Private sector
Exports
Imports
27
Fully Specified Macro-Models
HERMES (Ireland)
 Quest (EU Model)
Advantages: Very detailed and comprehensive,
theory consistent, parameters estimated from
data, long-run and short-run impact.
Disadvantages: Costly to built and maintain
(HERMES has over 900 equations), more
difficult to identify causality, results can be
difficult to interpret.

28
Ex-Ante Macro-Model
Evaluation





To carry out an evaluation we do the following:
We carry out a model simulation starting in the a year
previous to the start of the investment programme and
continue the simulation out past the end of the programme
We then “extract” the NDP “policy shocks”, i.e., we set the
NDP expenditures at zero and re-simulate the model.
The difference between the two simulations gives us the
impact of the NDP.
This is quite an artificial way to identify the impact as in a
normal setting investment is unlikely to be zero i.e. The
counterfactual is quite extreme. An alternative would be to
compare the with new NDP scenario with a ‘business as usual’
scenario.
29
Mid-Term Evaluation of the NDP
2000- 2006 (GNP) – impact of
investment for the period 20002003
8
7
6
4
3
2
1
Demand Only
20
10
20
08
20
06
20
04
20
02
0
20
00
%
5
Total
Crowding Out





Building and Construction bids resources away from other
sectors
 Higher wages because labour demand
 Higher house prices affects wages
The tradable sector has to sell abroad
 Badly affected by competitiveness
More construction jobs – less jobs in manufacturing &
business services
Should introduce measures to take the heat out of the
housing market!
More modest NDP recommended.
31
Multi Criterion Decision Analysis
(MCDA)
Implemented through a scoring model, that defines criteria
for different types of investments and weights them in
accordance with priorities.
Advantages: Very flexible, allows for multiple conflicting
objectives, allows for comparison of very heterogeneous
investments, can be done where little data exists, can be used
for loosely defined projects/programmes.
Disadvantages: Can be accused of being subjective, requires
thorough knowledge of the investments and background
analysis.

32
Multi-Criterion Decision Analysis
(MCDA)



Widely applied in management science.
Multi Criterion Analysis, was first applied by Honohan (1997)
and subsequently in ESRI investment priorities and mid-term
evaluations, is becoming more popular (e.g. Cundric, Kern and
Rajkovic, 2008, KPMG, World Bank 2010 etc.)
This is a simple method that entails the classification of
measures and a judgement of the measures against a set of
criteria.
33
MCDA






For each type of intervention a set of criteria is defined importance, contribution, dead-weight, least cost, targeting,
environmental impact……..
The criteria should reflect government objectives and then
economic rationale for such interventions.
It is possible to have a large set of criteria covering all
objectives in detail. Criteria can have different weights.
Each possible investment is judged (scored) against the
criteria.
A composite score is calculated, which allows investments to
be ranked.
The results are remarkably robust once the criteria are well
34
chosen – the cream always rises to the top!!!
Multi-criterion Decision Analysis
(MCDA)
Operationalisation
Goal
1
Selection of
preferred
alternatives
Scoring
Criterion
2
Scoring
Implementation
Ranking
3
4
Decision
5
Monitoring and evaluation
of the results
35
MCDA Results - Examples
Priority
Measure
Sub-Measure
Area
Specific
Local
Non-National
Improvement
Infrastructure
Roads
Grant Scheme Transport
Environmental Management and Rehabilitation of
Infrastructure
infrastructure
Environment
Special Interests
Local Enterprise Tourism
Pursuits
Tourism
Harvesting
Local Enterprise Forestry
Equipment
Agriculture
Composite Score
0.8
0.8
-0.1
-0.1
36
Cost Effectiveness Analysis
Considers the most effective option once a
specific objective is chose – popular to assess
treatment options in health care or other
areas where it may be difficult to monetise all
benefits.
Advantages: Can be applied in situations where
it is difficult to monetise benefits, clear
ranking
Disadvantages: Not practicable for a large set of
heterogenous projects/ programmes.
37

Social Cost Benefit Analysis
(CBA)
Widely used. Involves comparing total (all) expected
benefits with total (all) expected costs. In order to take
account of the timing of benefits and costs a discount
rate is used to convert these into a present value;
Advantages: Very detailed analysis using a common metric,
comparable results across different types of investments.
Disadvantages: Dependent on parameters and
counterfactual, onerous to implement for a large
programme of projects, some projects are difficult to
evaluate (especially ill defined projects/programmes).
Standard methodology – required in Ireland for all public
projects above a threshold size (€20 million)
38
Steps Required in doing CBA
1. What policy or project is being evaluated? What alternatives
are there? - often the only alternative considered is to do
nothing!
2. Whose costs and benefits are to count? – often not all
costs/benefits are considered!
3. Over what time horizon are costs and benefits are counted? –
there could be terminal values at the end of the time horizon
that may be important!
4. Costs/benefits today are not the same as costs/benefits in the
future – need to decide on discounting. – discount rates can
have a significant impact on results.
5. What are the risks to the costs and benefits?
39
6. What are the distributional impacts of costs/benefits?
Social Cost Benefit Analysis
(CBA)






Involves comparing total expected benefits with
total expected costs.
The usefulness of the analysis is critically
dependent on key parameters;
In order to take account of the timing of benefits
and costs a discount rate is used to convert these
into a present value;
Risks;
Counterfactual.
This looks like a simple task??
40
Decision Rule





Having done the measuring one can assess whether
the project should go ahead.
Pareto improvement – do the project if some people
gain and nobody loses.
Few projects would ever meet this criterion.
Hicks-Kaldor criterion – a project should be carried
out if the gainers could compensate the losers and
still be better off.
Of course the losers are often not compensated in
practice!! => implications for the planning system –
objectors!!
41
CBA




What is a benefit? – anything that increases national welfare.
Benefits can be monetary (cost reduction), non-monetary (e.g.
reduction in complaints), non-monetary qualitative (improved
quality of life).
In many cases it is possible to find ways to monetise non-monetary
benefits, but this takes some effort e.g. Environmental quality
effects can be assessed with hedonic models of property prices (the
benefits/disamenities should be capitalised in property prices).
What are costs? – Costs relate to actual resource use in the
economy and reflect the best alternative uses that the resources
could be put to (i.e. they are opportunity costs). It is important to
explore what alternative opportunities may exist.
42
Market Failures




In an ideal world the market system leads to an efficient
outcome. However, we do not live in an ideal world (perfect
competition and perfect information are interesting academic
concepts but are far removed from the ‘real world’)
Market failures/externalities are not taken into account in
normal market interaction including price setting => market
prices are often not appropriate for social cost benefit
analysis
Taking such distortions into account distinguishes social cost
benefit analysis from standard investment appraisal.
Price distortions also make cost-benefit analysis more difficult
to conduct.
43
Market Failures




Monopoly – price is above marginal cost – should we use the
(higher) market price or the marginal cost?
If the input used is diverted from other activity we should use
the marginal cost (because other users of the input loose)
If the input used is additional to that produced then the
market price should be used.
Indirect taxes such as VAT imply that the price faced by
consumers are not equal to ‘free’ market prices – the same
goes for subsidies and tariffs. Taxes that address
distortions/externalities imply that the tax inclusive price is
more efficient – that is the price to be used.
44
Market Failures


Unemployment – if a project employs unemployed people
then the cost of employing them is their lost leisure rather
than the wage they receive, which is typically less than the
wage (involuntary unemployment). Employing unemployed
people also has a consumption effect. If labour is fully
employed an additional project could also have significant
inflationary consequences.
Foreign exchange/trade – inputs into a project could be
imported which implies that the benefits accrue abroad (a
significant proportion of the cost to build the bridge of peace
in Derry went on steel from Wales – local benefits were
limited).
45
Market Failures


Property rights – some externalities are difficult to capture in
property rights (Coase Theorem - if trade in an externality is
possible and there are sufficiently low transaction costs,
bargaining will lead to an efficient outcome regardless of the
initial allocation of property - fails). Current controversies
around wind turbines and pylons are examples – land owners
are compensated if a pylon is placed on their land but
neighbours who get a visual dis-amenity are not
compensated!
Pricing of non-market items – anything that is not traded on a
market does not have an observable market value.
Nevertheless these items may account for a large proportion
of benefits – travel time savings/value of time, lives saved.
46
Shadow Prices




Shadow price = Social opportunity cost;
Market prices can give a misleading signal;
We have already noted the issue regarding unemployment
that the cost of employing unemployed people is the value of
their lost leisure time because that is their opportunity cost.
With full employment the opportunity cost is the wage
earned in another job (i.e. The project would displace
employment elsewhere);
Assuming labour is allocated efficiently the market wage
equals the opportunity cost so that the shadow wage is equal
the market rate – this simplifies a CBA – but is often not valid.
47
Shadow Wage



In practice if the shadow wage is lower than the market wage
then the benefit is calculated by pre-multiplying the wage bill
associated with a project by (1-v) where v is the shadow wage
expressed as a fraction of the market wage.
If v=1 then the shadow wage equals the market wage and
there is no benefit.
But how do we determine the shadow wage?
48
Discount Rate





Costs and benefits do not accrue immediately.
A euro to be received in 10 years time is worth less than a
euro received today.
Converting future values to a present value is accomplished
using a discount rate.
Setting the appropriate discount rate has been a topic of
countless research papers.
Current test discount rate is 5% (4% until recently).
49
Example: €1000 received in the future
Discount Rate
Years
0%
4%
8%
12%
1
€1000
€962
€926
€893
10
€1000
€676
€463
€322
20
€1000
€456
€215
€104
50
€1000
€141
€21
€3
100
€1000
€20
€0
€0
50
Setting the Discount Rates


1.
2.
3.

The choice of discount rate clearly matters.
What is the right rate? Alternative approaches:
Use the cost of public borrowing to proxy the discount
rate – This is taken as a riskless rate, but projects are
inherently risky.
Use the opportunity cost of funds not used for other
projects (private return).
Use a rate that reflects the rate of time preference of
individual with respect to consumption decisions (Ramsey
Equation).
The different approaches yield different results.
51
Ramsey Equation







Set up a simple growth model where investor must decide how much to
save (invest) and how much to consume.
Utility is a function of current and expected future consumption.
The marginal utility loss of consuming a little less today and buying a little
more of the asset should equal the marginal utility gain of consuming a
little more of the asset’s payoff in the future.
Solving the model gives rise to the so-called Ramsey equation:
rtf=φ+ η g
Where rtf is the risk free social discount rate, φ is the pure rate of time
preference used to discount utility, η is the elasticity of marginal utility of
consumption and g is the growth rate of consumption.
Given values for each of the components it is simple to derive an estimate
of the discount rate.
52
Discount Rate Estimates
Source: Bond Rate from OECD, Real net return calculated using CSO Institutional Sector Accounts and Capital Assets, time
preference calculated using consumption from CSO National Accounts (see paper for detailed description) – see Morgenroth
.
(2013)
53
Example – Roads Scheme


-
-
Benefits:
Changes in Travel time
Changes in Operating cost
Changes in Accident costs
Costs
Capital Costs
Changes in maintenance costs
54
Examples – Roads Scheme
55
Examples – Bus Rapid Transit

-
-

-
-
Benefits/Disbenefits
Change in travel time for drivers, transit users
Change in vehicle operating costs for drivers, fares for transit
users
Change in emissions of criteria pollutants and greenhouse
gases
Change in crash costs
Costs:
Capital costs of materials, equipment, infrastructure
construction, new buses
Operations and maintenance costs
56
Risks: Cost & Benefit Estimates




Projects are subject to a range of risks: construction risk,
operating risk, demand risk, financial risks and political risk.
But projects are typically promoted on an “everything goes
according to plan” basis.
Given the uncertainty it would not be surprising to find that
project costs/benefits are not accurately projected – but
one would expect the average error to be zero.
The international literature finds a systematic optimism
bias (Flyvbjerg, 2003, 2004, 2005, Bain, 2009, Pickrel,
1990).
57
Cost Overruns


Flyvbjerg et al (2003) analysed 258 projects from 20 countries
covering rail, bridge, tunnel and road projects. They found
that 90% of projects were subject to cost overruns. The
average cost overrun for rail projects was 45%, bridges and
tunnels were subject to an average 34% cost overrun and
roads cost on average 20% more than initially estimated.
ICT projects in the transport area were found to suffer
particularly large cost overruns averaging 200%!
58
Lower Benefit



Pickrel (1990) considered rail projects in the USA. He found
that for nine projects the actual passenger numbers were
50% lower than expected while for one project the passenger
numbers exceeded the predicted level by 50%.
Flyvbjerg et al (2004) analysed 210 projects from 14 countries
90% of rail projects overestimated demand (average bias
51%). For Roads the estimates on average understated traffic
by 9.5%.
Parthasarathi and Levinson (2010) considered 391 road
projects in Minnesota constructed in the 1960’s and
compared the traffic forecasts with the traffic counts taken in
1978. They found that on average traffic on roads was
underestimated by 19.5% and that the deviations of actual
59
from projected traffic ranged from -60% to +57%.
A Simple Example



The implications of the findings of this literature are best
illustrated by applying them to an example of a rail project
with benefit to cost ration of 2:1.
For rail projects the findings are that benefit (demand) is
overestimated by 50% and that costs are underestimated up
by 40% at the time of project proposal.
Adjusting the benefits and costs accordingly reduces the
Benefit to Cost Ratio (BCR) to less than 0.75:1.
60
Optimism Bias Implications for a
Hypothetical Rail Project (BCR 2:1)
61
How can we Eliminate Bias?





One method proposed by Flyvbjerg is called reference class
forecasting (see Flyvbjerg, 2007, Flyvbjerg and COWI, 2004).
What does this involve?
The conventional approach is to take an “inside view” which
focuses at the project to be assessed.
The alternative is to take an “outside view” considering the
outcomes achieved by other similar projects.
This approach has its origins in behavioural economics and
was initially proposed to adjust for systematic cognitive
biases.
62
Reference Class Forecasting

1.
2.
3.
Reference class forecasting requires three steps:
Identifying the relevant class or type of projects with which
the proposed project is to be compared;
Establish the probability distribution of outcomes for the
relevant reference class;
Establish the probability of a specific outcome being
achieved for the proposed project.
63
Reference Classes



What types of projects – reference classes?
They need to be reasonably homogenous.
There need to be sufficient number of comparable projects
for which appropriate data is available.
64
Reference Classes
Category
Type of Project
Roads
Motorways
Trunk roads
Local roads
Bicycle facilities
Pedestrian facilities
Park and ride
Bus lane schemes
Bus rapid transit
Rail
Metro
Light rail
Mainline rail
High speed rail
Fixed Links
Bridges
Tunnels
Other building
Stations
Airports
IT Projects
IT systems
65
Probability Distribution of
Outcomes



Once the reference class is decided it is necessary to get data
on the relevant information e.g. construction costs,
construction time, demand etc.
This is then used to estimate the probability distribution of
the target variables.
This is straightforward – cumulative distribution.
66
Cost Overruns for Roads
67
Cost Overruns for Rail
68
Probability of Outcome



Having established the probability distribution it is possible to
consider the chance of a project having a particular deviation
from the originally projected cost/duration/benefit.
60% of projects have a cost overrun of up to 20% => there is a
60% chance that cost overruns will not exceed 20%.
Adding 20% to the projected costs would reduce the
probability of any cost overrun to 40%.
69
Limitations



The method requires access to the details of outcomes for a
large number of projects (especially if one wants to be able to
apply the method to different classes of projects).
The probability distributions can be biased due to outliers.
The UK Department of Transport recommends testing
projects by applying the cost uplift as discussed – if this is the
general rule then it is easy to ‘game’ the evaluation by
reducing the projected cost to an even lower level than the
possibly already biased cost estimates.
70
Readings
Bain, R. (2009). “Error and Optimism Bias in Toll Road Traffic Forecasts”, Transportation, Vol. 36, pp. 469-482.
Beutel, J. (2002). The Economic Impact of Objective 1 Interventions for the period 2000 – 2006, Final Report to DG-REGIO, May.
http://ec.europa.eu/regional_policy/sources/docgener/studies/pdf/objective1/final_report.pdf
Bradley J., T. Mitze, E. Morgenroth and G. Untiedt (2006). How can we know if EU cohesion policy is successful? Integrating micro
and macro approaches to the evaluation of Structural Funds. Münster: GEFRA.
http://www.dcu.ie/education_studies/ien/iendata/Bradley-Cohesion%20Policy%20Analysis%20Micro%20and%20Macro%20Approaches.pdf
Brent, R., (2006) Applied Cost-Benefit Analysis. Cheltenham: Edward Elgar
Burgess, D. and R. Zerbe (2011). “Appropriate Discounting for Benefit Cost Analysis”, Journal of Cost Benefit Analysis, Vol. 2, No. 2,
pp. 1-18.
Cundric, A.,T. Kern and V. Rajkovic (2008). “A qualitative model for road investment appraisal”, Transport Policy, Vol. 15, pp. 225231.
De Brucker, K., C. Macharis and A. Verbeke (2011). “Multi-criterion Analysis in Transport Project Evaluation: An Institutional
Approach”, European Transport, Vol. 47, pp.3-34.
Department of Finance (2005). Guidelines for the Appraisal and Management of Capital Expenditure Proposals in the Public
Sector.
http://www.finance.gov.ie/documents/publications/other/capappguide05.pdf
Department of Public Expenditure and Reform (2011). Project Discount and Inflation Rates
http://per.gov.ie/project-discount-inflation-rates/
Evans, D. (2005). “The Elasticity of Marginal Utility of Consumption: Estimates for 20 OECD Countries”, Fiscal Studies, Vol. 26, No.
2, pp. 197-224.
FitzGerald, J., C. McCarthy, E. Morgenroth and P.J. O’Connell (2003). The Mid-Term Evaluation of the National Development Plan
(NDP) and Community Support Framework (CSF) for Ireland, 2000-2006. Policy Research Series No. 50, Dublin: Economic
and Social Research Institute.
71
Readings
Flyvbjerg, B., in association with COWI (2004). Procedures for Dealing with Optimism Bias in Transport Planning – Guidance
Document. London: Department of Transport.
Flyvbjerg, B., M. Holm, K.Skamris and S.L.Buhl, (2003). ‘How Common and How Large Are Cost Overruns in Transport
Infrastructure Projects?’, Transport Reviews, Vol. 23, No. 1, pp. 71-88.
Flyvbjerg, B., M. Holm, K.Skamris and S.L.Buhl, (2004). ‘What Causes Cost Overrun in Transport Infrastructure Projects?’,
Transport Reviews, Vol. 24, No. 1, pp. 3-18.
Flyvbjerg, B., M. Holm, K.Skamris and S.L.Buhl, (2005). “How (In)accurate Are Demand Forecasts in Public Works Projects? The
Case of Transportation”, Journal of the American Planning Association, Vol. 71, No. 2, pp. 131–46.
Flyvbjerg, B., (2007) Eliminating Bias In Early Project Development through Reference Class Forecasting and Good Governance.
Concept Report No.17.
http://www.concept.ntnu.no/Publikasjoner/Rapportserie/Rapport%2017%20kapittelvis/Concept%20176%20Reference%20Class%20Forecasting%20and%20Good%20Governance.pdf
Flyvbjerg, B., (2008) “Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in
Practice”. European Planning Studies, Vol. 16(1), pp.
http://commonsenseatheism.com/wp-content/uploads/2011/09/Flyvbjerg-Curbing-optimism-bias-and-strategicmisrepresentation-in-planning-reference-class-forecasting-in-practice.pdf
Gillespie, G., P. McGregor, K. Swales and Y. P. Yin (2001). “The Displacement and Multiplier Effects of Regional Assistance: A
Computable General Equilibrium Analysis”, Regional Studies, Vol. 35, No. 2, pp. 125-139.
Hahn, R. and P. Dudley (2007). “How Well Does the U.S. Government Do Benefit-Cost Analysis?”, Review of Environmental
Economics and Policy, Vol. 1, No. 2, pp. 192-211.
Harrison, M. (2010). Valuing the Future: The Social Discount Rate in Cost-Benefit Analysis. Visiting Researcher Paper,
Australian Government Productivity Commission.
http://pc.gov.au/__data/assets/pdf_file/0012/96699/cost-benefit-discount.pdf
HM Treasury (2011). The Green Book: Appraisal and Evaluation in Central Government. London: TSO.
72
Readings
Honohan, P., (1998) Key Issues of Cost-Benefit Methodology for Irish Industrial Policy. ESRI General Research Series Paper.
Dublin: ESRI.
Honohan, P. (ed.) (1997). EU Structural Funds in Ireland: A Mid-Term Evaluation of the CSF 1994-1999, Policy Research Series, No.
31, Dublin: The Economic and Social Research Institute.
Juri N., R. and K. Kockelman (2006). “Evaluation of the Trans-Texas Corridor Proposal: Application and Enhancement of the
Random-Utility-Based Multiregional Input-Output Model”, Journal of Transport Engineering, Vol. 132, No. 7, pp. 531-539.
Layard R. and S. Glaister (1994) Cost-Benefit Analysis. Cambridge: Cambridge University Press.
Loewenstein, G. and R. Thaler (1989). “Anomalies: Intertemporal Choice”, Journal of Economic Perspectives, Vol. 3, No. 4, pp. 181193.
Mackett, R.L. and Edwards, M. (1998). ‘The Impact of New Urban Public Transport Systems: Will the Expectations Be Met?’,
Transportation Research A, Vol. 32, No. 4, pp. 231-45.
Mishan, E., and E. Quah (2007) Cost Benefit Analysis. London and New York: Routledge
Morgenroth, E. (2013) “How Can We Improve Evaluation Methods for Public Infrastructure?” in Lunn, P., and F. Ruane (eds.)
Using Evidence to Inform Policy. Dublin: Gill and Macmillan
Mulreany, M., (2002) Cost Benefit Analysis Readings. Dublin: Institute of Public Administration.
Murphy, A., Walsh, B. And F. Barry (2003) The economic appraisal system for projects seeking support from the industrial
development agencies. Forfas
Oosterhavn, J., and T. Knaap (2003) “Spatial Impacts of Transport Infrastructure Investments”, in Pearman, A., Mackie, P. And J.
Nellthorp (eds) Transport Projects, Programmes and Policies: Evaluation Needs and Capabilities. Aldershot: Ashgate.
Parthasarathi, P. and D. Levinson (2010). “Post-construction evaluation of traffic forecast accuracy," Transport Policy, Vol. 17, No.
6, pp. 428-443.
Pearce, D., Atkinson, G., and S. Mourato (2006) Cost-Benefit Analysis and the Environment: Recent Developments. Paris:
OECD.
Pickrell, D.H. (1990). “Urban Rail Transit Projects: Forecast versus Actual Ridership and Costs”, US ‘Department of Transportation.
73
Readings
Pritchett, L. (1996) “Mind Your P’s and Q’s: The Value of Infrastructure is not Equal to its Cost”, World Bank Working Paper No.
1660.
Pritchett, L. (2002) “It Pays to be Ignorant: A Simple Political Economy of Rigorous Program Evaluation”, Policy reform, Vol. 5(4),
pp. 251-269.
Roeger, W. (1996). Macroeconomic Evaluation of the Effects of Community Structural Funds with QUEST II. Mimeo, European
Commission, DG-ECFIN.
http://ec.europa.eu/regional_policy/sources/docconf/eva/download/roeger.pdf
http://www.rug.nl/staff/j.oosterhaven/transtalk03_raem_zzl.pdf
Ramsey, F. (1928). “A Mathematical Theory of Saving”, Economic Journal, Vol. 38, pp. 543-559.
Sah, R. K. and J. Stiglitz (1985) "The social cost of labor and project evaluation: A general approach," Journal of Public Economics,
Vol. 28(2), pp. 135-163
Stiglitz, J. (1994). “Discount Rates: The Rate of Discount for Cost-Benefit Analysis and the Theory of the Second Best” in R. Layard
and S. Glaister (eds.), Cost-Benefit Analysis, Cambridge: Cambridge University Press.
Thaler, R. (1981). “Some Empirical Evidence on Dynamic Inconsistency”, Economics Letters, Vol. 8, pp. 201-207.
Törmä, H. (2008). “Do Small Town Development Projects Matter, and can CGE Help?”, Spatial Economic Analysis, Vol. 3, No. 2, pp.
247-268.
Van Ewijk, C. and P. Tang (2003). “How to Price Risk In Public Investment”, De Economist, Vol.151, No. 3, pp.317-328.
Van Wee, B. (2012). “How Suitable is CBE for the Ex-Ante Evaluation of Transport Projects: A Discussion from the Perspective of
Ethics” Transport Policy, Vol. 19, pp. 1-7.
Venables, A., and Gasiorek (1997) Evaluating Regional Infrastructure: A Computable Equilibrium Approach
Vassallo, J. (2010). “The Role of the Discount Rate in Tendering Highway Concessions under the LPVR Approach”, Transportation
Research Part A, Vol. 44, pp. 806-814.
Viscusi, K., J. Huber and J. Bell (2008).“Estimating Discount Rates for Environmental Quality from Utility-based Choice
Experiments”, Journal of Risk and Uncertainty, Vol. 37, pp. 199-220.
74
Readings
Weitzman, M. (1998). “Why the Far-Distant Future Should be Discounted at its Lowest Possible Rate”, Journal of Environmental
Economics and Management, Vol. 36, No. 3, pp. 201-208.
Weitzman, M. (2001). “Gamma Discounting”, American Economic Review, Vol. 91, No. 1, pp. 260-271.
Weitzman, M. (2010). “Risk-adjusted Gamma Discounting”, Journal of Environmental Economics and Management, Vol. 60, No. 1,
pp. 1-13.
75