SCP Evidence Base Research Programme
Summary review statement
Research Project SCP008
Report reviewed: Cost Benefit Analysis of Sustainable Public Procurement
Completed by: SQW Limited
Date of report:: 31st March 2006
Reviewers: Dr. Gordon Murray (co-ordinating role); Prof. R. Kerry Turner; Dr. Helen Walker
Introduction
The role of external review is to provide an independent challenge to the science
commissioned by Defra, to ensure that policy is informed by a high-quality, robust evidence
base and to raise the perceived standard of Defra-funded science among stakeholders. The
Advisory Group on SCP evidence has a role in quality assurance of research commissioned
under the SCP evidence base research programme. This has been undertaken through
expert sub-groups of the Advisory Group, including experts from beyond the Group’s
membership where appropriate.
Reviewers have been asked to rate the report in 10 categories and provide comments to
justify the scores given. These reports have been used as the basis for this summary report,
which has been compiled by Defra and verified by the coordinating peer reviewer, Gordon
Murray.
Objective of the research project
To undertake a cost benefit analysis of a sample of public procurement initiatives to evaluate
their efficacy as a tool. The case studies aimed to cover the three pillars of sustainability:
Social, environmental, and economic issues, and to embrace efficiency, equity, up-stream,
and down-stream impacts arising from the selected procurement initiatives.
Coordinating reviewer’s comments
The Reviewers felt:
1. The approach is sound, robust and represents sound science;
2. Assumptions are identifiable;
3. The analysis is sound and mostly draws on appropriate, recent and relevant studies;
4. Evidence is generally given appropriate consideration.
The research satisfies the specified objectives although findings cannot be generalised
beyond the specific case studies considered. The approach is robust given the restricted
range of cases considered; a broader range of cases, representing categories of
procurement more typical of public sector procurement would have helped increase the
potential transferability of the learning to application. The research could also have used a
broader approach to evaluation of procurement costs and benefits through considering the
whole procurement cycle, for example, from identifying the need through business case
development, supplier appraisal, tender evaluation, contract management and collecting
81923469
Page 1 of 7
SCP Evidence Base Research Programme
lessons learnt. Consideration could have been given to how changes in procurement
approaches allowed for other efficiency gains, for example, how the investment in the court
case equipment could be used to deliver efficiency gains beyond merely court appearances.
Assumptions are generally clearly stated but heavily dependent on third party valuations with
some unquantifiables/intangibles in need of greater explanation and comparison with
alternative approaches. The Contractor acknowledged this point, however, it was felt that the
time scale and resources available for the project provided a limitation to collect additional
data to that already collected.
It would have been helpful if the case studies included economic and social costs/benefits.
A consistent approach to sustainable procurement CBA, supplemented with a recommended
approach, which would assist in the evaluation of sustainable procurement CBA beyond this
study, would have been preferred.
Presentation would be helped through a link from the methodology to the case studies, and a
summary table at the end of each case study. The Contractor subsequently revised the initial
chapters, and included an annex with tables of data to allow greater transparency of the
underlying calculations.
The Reviewers were divided on the aspect of conclusions and recommendations on which to
base policy decisions; specific concern lay in the absence of a consistent approach to CBA
of sustainable procurement and the absence of clear messages about the CBSA of
sustainable public procurement. The Contractor felt that the underlying approach for CBA
was consistent. The conclusions have later been revised to address some issues relating to
CBA of sustainable public procurement, yet it was felt that the six case studies provided
insufficient evidence to draw clear messages from them.
Dr Gordon Murray, 8th May 2006
81923469
Page 2 of 7
SCP Evidence Base Research Programme
Summary of reviewers’ responses
Reviewers were asked to rate the report on the following headings:
REPORTING AND METHODS
1) Scope and Objectives. Does the report address all aspects of the objectives of the study
stated in the agreed specification?
2) Quality of Approach. Do the approach and methodology adequately address the objectives?
Are there any weaknesses that could cast doubt on the conclusions?
3) Assumptions. Are any assumptions made in the report sound and clearly identifiable?
DATA AND ANALYSIS
4) Evidence Base. Does the evidence on which the analysis is based draw on appropriate,
recent and relevant studies in this field? Is the evidence considered representative of the
evidence that exists?
5) Analysis. Is the analysis sound, clear and appropriate for the report?
6) Presentation of Evidence. Are the figures and tables clear, adequate, not actually or
potentially misleading, and do they support the inferences drawn from them?
CONCLUSIONS
7) Use of Evidence. Is effective use made of relevant subject matter and evidence - is any
evidence ignored or under-represented? (Include evidence from within or outside the report –
please give details).
8) Conclusions and Recommendations. Are the conclusions, policy implications, and
recommendations clearly set out, based on the evidence gathered and logically argued? Are
there any gaps or omissions?
9) Reasoning. Are conclusions based on judgement rather than evidence clearly recognisable?
OVERALL
10) Rigour and Robustness. Does the work represent sound and robust science and are the
conclusions supported by the evidence and analysis presented?
Reviewers rated each aspect on a scale of a-e, where ‘e’ is ‘not applicable’ and the ‘d’ and ‘e’ grades
are sometimes not available. The table below shows the distribution of reviewers’ grades for each
aspect. The detail of the criteria set for each question and grade and a summary of reviewers’
additional comments is provided on the following pages.
1. Scope and Objectives
2. Quality of Approach.
3. Assumptions
4. Evidence Base.
5. Analysis
6. Presentation of Evidence
7. Use of Evidence
8. Conclusions and
Recommendations
9. Reasoning
10. Rigour and Robustness
81923469
a
b
c




















d
e (not
Average score
applicable) (MODE)
B
A
A
B
A
A
A
B
A
A
Page 3 of 7
SCP Evidence Base Research Programme
Full text of questions and summary of reviewers’ comments
Question
Reviewer comments
REPORTING AND METHODS
1) Scope and Objectives. Does the report address all
aspects of the objectives of the study stated in the agreed
specification?
a) all of the stated objectives addressed satisfactorily

b) most of the stated objectives addressed satisfactorily

c) few of the stated objectives addressed satisfactorily
2) Quality of Approach. Do the approach and
methodology adequately address the objectives? Are there
any weaknesses that could cast doubt on the conclusions?
a) Quality of the approach is sound and robust. It is optimal
for the scope / nature of the project.
b) Quality of approach generally sound. Some parts weaker
than others.
c) Weaknesses in approach could draw doubt on some of
the conclusions.
d) Approach is such that conclusions could be flawed.
e) Not applicable.


3) Assumptions. Are any assumptions made in the report
sound and clearly identifiable?

a) Assumptions are clearly identified and sound.
b) Assumptions are identifiable and broadly in line with
current thinking and/or are justifiable in the circumstances.

c) Assumptions are hard to identify and/or could lead to
conclusions being incorrect.
d) Assumptions are not identified and/or are not based on
sound judgement.
e) Not applicable.
81923469
Page 4 of 7
The Reviewers agreed that the key stated
objectives have been met. The limitations
of case studies were referred to by all
reviewers. Also, one Reviewer noted that
the study has not provided a clear
methodology for procurement officers to
adopt if they need to conduct a CBA, and
that the report would benefit from greater
reflection by the authors around the
conclusions and implications.
The Reviewers were in agreement that
the methodology deployed was systematic
and sound. One Reviewer identified a
weakness relating to the absence of a
value based on reputation gains and the
limited portfolio of cases, for example,
there are other types of sustainable
procurement initiatives, not considered,
which could potentially appeal to more risk
averse public sector bodies.
The Contractor agreed that reputation
gains should in principle be valued,
however, such data was not available.
The Reviewers generally found the
assumptions appropriate, but note some
specific issues of concern in the report,
generally related to the reliance on third
party valuations and concern over the
unquantifiable benefits, for example,
reduction in asthma rates due to less
pollution. The Contractor has
subsequently included considerations
related to health benefits. One Reviewer
commented that the NHS furniture study
could have been expanded further with
some extra illustrative assumptions. The
Contractor found that there was
insufficient data to make meaningful
assumptions. Commenting later under
‘Reasoning’ one Reviewer also pointed
out that intangibles are difficult to measure
and that assumptions underlying the
measures adopted need more unpacking
and to be complimented by reflection on
alternative approaches
SCP Evidence Base Research Programme
DATA AND ANALYSIS
4) Evidence Base. Does the evidence on which the
analysis is based draw on appropriate, recent and relevant
studies in this field? Is the evidence considered
representative of the evidence that exists?
a) Evidence for the analysis is drawn from appropriate,
recent and relevant studies in the field.
b) Evidence for the analysis mostly draws on appropriate,
recent and relevant studies in the field.

c) Evidence for the analysis is frequently drawn from
inappropriate, dated and/or irrelevant sources.
d) Evidence for the analysis is not representative of the
evidence that exists.
e) Not applicable.
5) Analysis. Is the analysis sound, clear and appropriate
for the report?
a) Analysis is logical and robust. The most appropriate
techniques / analyses have been used throughout.
b) Analysis is generally sound although more up to date /
appropriate techniques could have been used.


c) Analysis is frequently inappropriate.
d) Analysis is incomplete or flawed. It may have led to
incorrect conclusions being made.
e) Not applicable.
6) Presentation of Evidence. Are the figures and tables
clear, adequate, not actually or potentially misleading, and
do they support the inferences drawn from them?
a) Figures and tables add value to the report and aid
interpretation of the results.
b) Figures and tables are broadly sound and assist the
reader. They could be improved to add clarity.
c) Figures and tables do not add value and in some cases
may mislead the reader.
d) Figures and tables are misleading and do not support
the inferences made from them.


e) Not applicable.
81923469
Page 5 of 7
The Reviewers agreed that evidence is
mostly drawn from appropriate literature,
but point out different issues that could be
improved. One Reviewer noted that the
main onus has been on environmental
costs, and suggested adding sections on
economic and social costs for each case
study. One Reviewer, commented that
whilst selection of criteria of cases was set
out and applied, a weakness lies, in the
case peculiarities, in that focus has been
on some of the more adventurous
initiatives as opposed to areas likely to be
deployed by the average public sector
body. One Reviewer, suggested that it
could be helpful to place a value on the
reputation gains for the public sector body
taking positive steps. Lastly, one
Reviewer suggested a discussion of the
social cost of carbon.
The majority of Reviewers found the
analysis logical. However, one Reviewer
commented on the variation of
approaches used to calculate CBSA and
felt that it would be advantageous to have
clear step-by-step analysis that others
could replicate with flow/decision chart.
The Contractor agreed with this
assessment, and subsequently revised
the initial chapters and provided tables of
data in an appendix.
A majority of the Reviewers found the
report as well presented with good use of
figures and tables. One Reviewer further
commented that each point has been
systematically set out and argued, and the
authors make use of referencing to a
good academic standard. However, one
Reviewer commented that some figures
and tables could be improved, and
suggested adding a summary table at the
end of each case study to help summarise
the findings, and to link more clearly each
table with the narrative.
SCP Evidence Base Research Programme
CONCLUSIONS
7) Use of Evidence. Is effective use made of relevant
subject matter and evidence - is any evidence ignored or
under-represented? (Include evidence from within or
outside the report – please give details).
a) All relevant evidence is considered and given due weight


b) Generally most evidence is given appropriate
consideration with one or two minor exceptions
c) There are some gaps in the evidence given and some
evidence is given inappropriate weight
d) The report ignores or significantly under-represents
pertinent subject matter or evidence
e) Not applicable
8) Conclusions and Recommendations. Are the
conclusions, policy implications, and recommendations
clearly set out, based on the evidence gathered and
logically argued? Are there any gaps or omissions?
a) Conclusions, policy implications and recommendations
are well presented, evidence based, logically argued and
comprehensive
b) Conclusions, policy implications and recommendations
are generally well presented, logically argued, evidence
based and comprehensive with some minor exceptions
c) Conclusions, policy implications and recommendations
are frequently not well presented, logically argued,
evidence based or comprehensive
d) Conclusions, policy implications and recommendations
are very poorly presented, and are not logically argued,
evidence based or comprehensive
e) Not applicable
81923469



Page 6 of 7
The Reviewers found that generally most
evidence has been considered and given
due weight, with one Reviewer
commenting that the authors make good
systematic use of the approach, and
clearly argue selection of cases. One
Reviewer, however, commented that the
furniture study and environmental impacts
estimates could have been expanded
further and the social cost of carbon could
have been discussed. One Reviewer
observed that in the court video
conferencing case study the additional
benefits gained through the use of the
equipment, for example, internal video
conference meetings, could have been
considered if processes were redesigned
and a wider perspective taken.
The Reviewers were divided on the
aspect of conclusions and
recommendations. One Reviewer found
that given the case study limitations, a
sound and logical analysis was presented.
One Reviewer commented that the
authors could do more to set out the
policy implications of their conclusions.
Although the conclusions regarding
transferability has been presented, the
conclusions do not present evidence on
what conditions need to be present for the
benefits to outweigh costs, and it was
suggested that a model utilising the
selection method used by some public
sector bodies of sustainability risk/profile
risk could help. The Reviewer further
noted that as the report considers a very
limited set of case studies, extreme
caution would be required prior to
generalising on those cases. Finally, one
Reviewer felt the authors did not make
recommendations, and suggests the
authors make more clear messages about
the feasibility of sustainable public
procurement and how that fits with various
Government objectives. The Contractor
has added a conclusions section with
comments on CBA and public policy.
SCP Evidence Base Research Programme
9) Reasoning. Are conclusions based on judgement rather
than evidence clearly recognisable?
a) Yes there is a clear distinction between the two
b) Broadly it is possible to distinguish between judgement
and evidence based conclusions
c) It is not clear whether conclusions are based on
judgement or evidence


A majority of the Reviewers found a clear
distinction between judgement and
evidence. One Reviewer commented that
when extra assumptions were introduced
it was made clear and defended. One
Reviewer noted that conclusions
appeared objectively based on evidence,
albeit evidence has been drawn from a
restricted range of case studies. One
Reviewer commented that the report was
very systematic and the quantitative
nature was reassuring in an evidencebased age, however, the Reviewer also
pointed out that intangibles are difficult to
measure and that assumptions underlying
the measures adopted need more
unpacking and to be complimented by
reflection on alternative approaches.
OVERALL
10) Rigour and Robustness. Does the work represent
sound and robust science and are the conclusions
supported by the evidence and analysis presented?
a) Sound and robust science. Conclusions are wholly
supported by the evidence and analysis presented
b) Sound science. Conclusions largely supported by the
evidence. Some improvements in the approach, analysis
and interpretation would improve confidence in the
conclusions
c) The evidence provided does not fully support the
conclusions
d) The evidence is of poor quality. It is not a sufficient base
from which to draw the conclusions made
81923469


Page 7 of 7
The majority of the Reviewers found the
report to be sound and robust science.
Yet, one Reviewer would like more on
conclusions and implications for
sustainable procurement policy-makers
and those seeking to calculate CBAs.
Another Reviewer found that the report
took the case studies as far as was
feasible, with the possible exception of the
NHS Furniture study. Finally one
Reviewer suggested some minor
adjustments to the report structure and
wording, and that the authors add a
general introductory section prior to
moving from the methodology to the case
studies. The Contractor has subsequently
addressed the format and conclusions
issues in the final report.