Peer Review of Defra Funded Science
Final Report Appraisal
Project Title : New Approaches to Evaluating and Quantifying
the Benefits of Chemicals Regulation
This form is to be used for the peer review of
completed final reports of Defra-funded
science. When you have completed this
form (preferably typed), please sign it and
send it via post to Defra at the address given
at the end of the form. If you have any
questions relating to the completion of this
form please contact either:
Victoria Cox - [email protected]
Tel 0207 082 8594
or
Kate Perry – [email protected]
Tel 0207 082 8085
Instructions
If, for whatever reason, you feel that you are
not able to confidently assess the report(s)
you have been sent, please advise Defra as
soon as possible.
Consider each question on the form and tick
the box that most represents the answer to
the question for the report being reviewed.
Given the importance of high quality
refereeing, it is essential that you can speak
with confidence with respect to the project
report(s) you have received, and that you
can justify your comments. It may be that
you feel you are only able to comment on
some aspects of the report. Therefore only
answer questions on which you feel you
are able to give a qualified judgement or
are applicable.
A full justification for each answer must be
included in the ‘Comments’ box. The space
allocated in the box is for guidance only. If
you are using this form electronically the
boxes are expandable, alternatively if you
are using a hard copy, feel free to use
continuation
sheets.
When
using
continuation sheets, please number and
mark them clearly with the question number.
This form and any continuation sheets may
be passed to the researcher responsible for
the project and to the funding policy division
within Defra. Would you prefer your name
NOT to be attributed to this form?
REPORTING AND METHODS
1. Scope and Objectives. Does the report address all aspects of the objectives of the study
stated in the agreed specification?
YES
□ NO X
If YES, please go to question 2.
If NO, are:
Tick ONE box only
□
(b) objectives broadly stated but not all clearly addressed?.................................................X
(c) objectives not clearly stated and/or the report does not address the objectives?........... □
(a) all objectives stated and most addressed satisfactorily?................................................
1
Please comment:
The specification for the work stated 3 clear objectives
1. Identify suitable example substances of concern based on historical evidence to be used as case-studies and collect
the evidence establishing a link between the substance and effects reported in the environment and on human health
through environmental exposure.
2. Quantify the link between the substances’ hazardous properties and the effect(s) reported in the environment and
human health.
3. Develop a framework of hazardous properties vs. cost incurred to be used as a generic model to estimate potential
costs of not implementing future chemical management legislation.
This requires the analysis of historical evidence from substances that have caused harm, the
costs involved in removing the harm, and the monetised benefits that accrued, and using this to
develop a generic model. This as clearly not been done and the project has been built on the use
of predicted effects. This presumably was agreed with Defra during the project progression. A
rationale for this change is provided, and is reasonable up to a point, but the report could put a
stronger case for the approach adopted and why the original approach could not be progressed.
For example, current PBT policy is based strongly on the evidence of existing PBT/POPs and the
harm, both potential and real, caused by their widespread distribution, and also the ‘effects’ of the
measures taken to control these, ie declining environmental concentrations. Monetising the
results of declining environmental concentrations is, of course, the difficult bit, but that is what the
project was for. So a stronger and clearer rationale as to why this was not possible should be
made.
Nevertheless, the approach adopted does offer a way of addressing objective 3, which is clearly
the key output required from the project. The authors if the report, however, feel that the model
developed can only assist on the prioritisation of action, ie which a number of options offers the
best monetised environmental benefit, or as the report says, prioritisation of which chemicals offer
the greatest benefits from regulation. There are a range of tools for the latter process of
prioritisation.
2. Quality of Approach (i.e. development of the model). Do the approach and methodology
adequately address the objectives? Are there any weaknesses that could cast doubt on the
conclusions?
(a) Quality of approach is sound and robust. It is optimal for the research conducted throughout
□
(b) Quality of approach is generally sound. Some parts weaker than others ……………... □
the project……………………………………………………………………………………………
(c) Weaknesses in approach could draw doubt on some of the conclusions………………..X
(d) Approach is such that the conclusions could be flawed…………………………………….
□
Please comment:
The broad approach of defining an environmental level, and the effects it is causing, by
prediction, both before and after regulatory action, as a way of assessing and quantifying the
benefits seems sound. Equally, the principle that as the environmental concentration increases
above a certain no effect level, the severity of the impact or the probability of a severe impact will
increase would also appear sound.
2
The problem is in the detail. The approach essentially uses predictions of actual environmental
and health effects via predictions of exposures and predictions of no effect levels. This is justified
up to a point but is very uncertain given the inherent uncertainties in both PEC and PNEC
derivation. This is recognised by the report’s authors who do not claim to have solved this
problem, but the principal issue is whether these uncertainties are too large for this approach to
offer a meaning estimate of benefit. The following points need to be considered
1. The estimate of PEC via EUSES offers a value for each point in a chemicals lifecycle, but
offers no estimate of the probability that this will actually occur. It is thus an estimate of a
concentration that could arise (and in all probability will arise), but how many locations etc
is ignored until later in the risk benefit analysis. Given the nature of the risk assessment
process, the PEC is likely to be ‘reasonable worst case’ and thus the probability of
occurrence would be low. How is this factored in?
2. The accuracy of this estimation of PEC is highly dependent on the level of information
available. Experience suggests that the more information eg on emissions, that are
available, the lower the PEC (and hence the PEC/PNEC ratio). Thus a high PEC/PNEC
is as likely to be a reflection of high uncertainty as of a high level of effects.
3. Equally, the PNEC generally rises as better and more relevant data are available.
The net effect would be that the highest benefits of regulation may accrue to poorly studied
substances where the appropriate action may be to gather better information. Logically, there
should be a level playing field applied to the information input to the model, but this is impractical
with respect to exposure.
The second problem with using the PEC/PNEC ratio as a measure of the severity of effects is the
flawed logic that species effects/survival is based solely on the toxicant and its concentration.
This is illustrated in table 2.3 where the assumption is that as the PEC exceeds the fish data
derived PNEC by more and more, so fewer fish species survive. There is a logic to this but it does
rather over simplify, and ignores inter-species interactions that makes up a functioning
ecosystem. In simple terms, fish eat food, so if the food dies so do the fish (or at least they swim
of somewhere else where there is food). Thus, you can’t separate fish from an overall ecosystem
effect, although for the purposes of monetisation, I can see the need.
(I wonder here whether the answer lies in seeing this a more an issue of probability. For example,
I would see a ‘no fish’ scenario as occurring at levels well below the PEC/PNEC 1000 mark
simply because they would have stopped reproducing, or their food would have died or stopped
reproducing. Monetisation needs a scalable measure of impact and this might come from
consideration of the probability of the impact occurring, ie the higher the PEC/PNEC (as worst
case), the greater the probability of a no fish environment across many local environments??)
The PEC/PNEC approach does not address PBT/POPs
3. Assumptions. Are any assumptions made during the work sound and clearly identifiable?
(a) Assumptions are clearly identified and sound………………………………………………
□
(b) Assumptions are identifiable and are broadly in line with current thinking and/or are justifiable
in the particular circumstances…………………………………………………………………….X
□
(d) Assumptions are not identified and/or are based on unsound judgement………………. □
(c) Assumptions are hard to identify and/or could lead to the conclusions being incorrect…
3
Please comment:
Many assumptions are made, most of which are described and justified in the interests of model
development. Generic modelling requires this and while some of the more simplistic assumptions
are questionable in purely scientific terms, most in the report are justified (but see comments
above)
Some assumptions used for scaling are wrong, eg the assumed rationale for the PNEC x 1000 on
p12, but while the explanation is not justified, the need for a scalable impact level is understood
and should be justified on that need. Almost any attempt at a scientific justification is likely to
draw criticism. In practice, the scaling adopted is probably OK, although the assumptions
underlaying it do not seem valid
DATA AND ANALYSIS
4. Evidence Base. Does the evidence on which the analysis is based draw on appropriate,
recent and relevant studies in this field? Is the evidence considered representative of the
evidence that exists? Is any evidence ignored or under-represented?
□
(b) Evidence is mostly drawn from appropriate, recent and relevant studies……………….. □
(c) Evidence is not frequently drawn from inappropriate, dated and/or irrelevant sources…□
(d) Evidence is not representative of the evidence that exists………………………………... □
(a) Evidence is drawn from appropriate, recent and relevant studies………………………..
Please comment:
TGD used as the basis for the risk analysis which is sound. Two example ESR assessments
chosen to populate the model, and illustrate the problems inherent in the approach
1. Trichloroethylene showed no environmental risks in the agreed RAR, apart from those
limited ones to plans via atmospheric exposure. Risks to health are not quantified, but
arise as a result of a hazard, ie Carc. Cat 2. In this report, marine risks are included but
these risks are present only because higher AFs are used due to lack of marine ecotox
data and thus higher uncertainty. The regulatory action was implemented not to curb an
aquatic risk, and yet 79% of the benefit appears to be for the aquatic (marine)
environment, for which there is probably no risk.
2. Pentabromodiphenyl ether is a PBT/POP and it would have been useful to see an
attempt at monetising regulation on this basis
4
5. Uncertainties. Are uncertainties in the data identified and recognisable?
(a) Uncertainties in the data are clearly identified and explained. ……………………………
□
(b) Uncertainties in the data are broadly identified although greater clarification
would be beneficial……………………………………………………………………………...X
□
(d) It is not possible to identify where there may be uncertainties in the data .............……..□
(c) Greater detail regarding uncertainty in the data would be beneficial……………………...
Please comment:
See above
6. Analysis. Is the analysis sound, clear and appropriate for the work undertaken?
(a) Analysis is logical and robust. The most appropriate techniques / analyses have been used
throughout…………………………………………………………………………………………..
□
(b) Analysis is generally sound although more up-to-date/appropriate techniques could have been
□
(c) Analysis is frequently inappropriate…………………………………………………………..□
used………………………………………………………………………………………………….
(d) Analysis is incomplete and/or flawed. It may have led to incorrect conclusions being
made…………………………………………………………………………………………………
□
Please comment:
5
7. Presentation of Evidence. Are the figures and tables clear, adequate, not actually or
potentially misleading, and do they support the inferences drawn from them?
(a) Figures and tables add value to the report and aid interpretation of the results…………
□
(b) Figures and tables are broadly sound and assist the reader. They could be improved to add
clarity………………………………………………………………………………………………….X
□
(d) Figures and tables are misleading and do not support the inferences drawn from them..□
(c) Figures and tables do not add value and in some cases may mislead the reader……... .
Please comment:
Tables are generally helpful, although table 1.1 and discussion can be deleted and 2.3 and 2.5
are the same
CONCLUSIONS
8. Conclusions and Recommendations. Are the conclusions, policy implications and
recommendations clearly set out, based on the evidence gathered and logically argued? Are
there any gaps or omissions?
(a) Concluding policy implications and recommendations are well presented, evidence based,
logically argued and comprehensive……………..……………………………………………
□
(b) Concluding policy implications and recommendations are generally well presented, evidence
based, logically argued and comprehensive, with some minor exceptions………………..
□
(c) Concluding policy implications and recommendations are frequently not well presented,
evidence based, logically argued or comprehensive…………………………………………
□
(d) Concluding policy implications and recommendations are very poorly presented, and are not
evidence based, logically argued or comprehensive…………………………………………
□
6
Please comment:
:
9. Reasoning. Are conclusions based on judgement rather than evidence clearly recognisable?
(a) Yes, there is a clear distinction between the two……………………………………………x
(b) Broadly it is possible to distinguish between judgement and evidence-based
□
(c) It is not clear whether conclusions are based on judgement or evidence……………….. □
conclusions…………………………………………………………………………………………..
Please comment:
As far as I can tell, the model is theoretical and based on prediction and extrapolation. There is
little evidence basis and this is recognised in the report and conlusions, and a reasonable
justification provided
10.Rigour and Robustness. Does the work represent sound and robust science, and are the
conclusions supported by the evidence and analysis presented?
(a) Sound and robust science. Conclusions are entirely supported by the evidence and analysis
presented……………………………………………………………………………………………
□
(b) Sound science. Conclusions largely supported by the evidence. Some improvements in the
design/analysis/interpretation would have improved confidence in the conclusions……….
(c) The evidence provided does not fully support the conclusions…………………………
X
□
7
(d) The evidence is of poor quality. It is not a sufficient base from which to draw the conclusions
made…………………………………………………………………………………………………
□
Please comment:
PRACTICABILITY
11. Policy Support and Development. Do you see the outputs of this project being useful in
policy support and development?
(a) The outputs can be immediately used in policy support and development………………
□
(b) With some minor improvement the outputs can be used in policy support
X
(c) The outputs could not be used without major development………………………………. □
and development………………………………………………………………………………..
Please comment:
12. Further Work. How do you see the outputs being developed further for a useful policy tool?
Please comment:
8
13. Remaining Issues. Are there any other issues that you consider are of note and should be
brought to our attention?
Please comment:
Please return via post to:
Victoria Cox
Zone 4/F8
Ashdown House
123 Victoria Street
London
SW1E 6DE
Data Protection Act 1998. In line with Defra's policy on openness and transparency, your
name will normally appear on the appraisal as a matter of course. If you tick the box at the
top of the form your name will not appear. However, you should be aware that we may still
be required to disclose your name in response to a request, including under the Environmental
Information Regulations, the Code of Practice on Access to Government Information and the
Freedom of Information Act 2000. We would, as far as practicable, seek to consult with you if
such a request were received.
9