General enquiries on this form should be made to:
Defra, Science Directorate, Management Support and Finance Team,
Telephone No. 020 7238 1612
E-mail:
[email protected]
SID 5



Research Project Final Report
Note
In line with the Freedom of Information
Act 2000, Defra aims to place the results
of its completed research projects in the
public domain wherever possible. The
SID 5 (Research Project Final Report) is
designed to capture the information on
the results and outputs of Defra-funded
research in a format that is easily
publishable through the Defra website. A
SID 5 must be completed for all projects.
1.
Defra Project code
2.
Project title
This form is in Word format and the
boxes may be expanded or reduced, as
appropriate.
3.
ACCESS TO INFORMATION
The information collected on this form will
be stored electronically and may be sent
to any part of Defra, or to individual
researchers or organisations outside
Defra for the purposes of reviewing the
project. Defra may also disclose the
information to any outside organisation
acting as an agent authorised by Defra to
process final research reports on its
behalf. Defra intends to publish this form
on its website, unless there are strong
reasons not to, which fully comply with
exemptions under the Environmental
Information Regulations or the Freedom
of Information Act 2000.
Defra may be required to release
information, including personal data and
commercial information, on request under
the Environmental Information
Regulations or the Freedom of
Information Act 2000. However, Defra will
not permit any unwarranted breach of
confidentiality or act in contravention of
its obligations under the Data Protection
Act 1998. Defra or its appointed agents
may use the name, address or other
details on your form to contact you in
connection with occasional customer
research aimed at improving the
processes through which Defra works
with its contractors.
SID 5 (Rev. 3/06)
Project identification
PS2301
Web-integrated framework for addressing uncertainty
and variability in pesticide risk assessment (WEBFRAM1)
Contractor
organisation(s)
Central Science Laboratory
Sand Hutton
York
YO41 1LZ
UK
54. Total Defra project costs
(agreed fixed price)
5. Project:
Page 1 of 21
£
278,035
start date ................
01 January 2003
end date .................
31 December 2006
6. It is Defra’s intention to publish this form.
Please confirm your agreement to do so. ................................................................................... YES
NO
(a) When preparing SID 5s contractors should bear in mind that Defra intends that they be made public. They
should be written in a clear and concise manner and represent a full account of the research project
which someone not closely associated with the project can follow.
Defra recognises that in a small minority of cases there may be information, such as intellectual property
or commercially confidential data, used in or generated by the research project, which should not be
disclosed. In these cases, such information should be detailed in a separate annex (not to be published)
so that the SID 5 can be placed in the public domain. Where it is impossible to complete the Final Report
without including references to any sensitive or confidential data, the information should be included and
section (b) completed. NB: only in exceptional circumstances will Defra expect contractors to give a "No"
answer.
In all cases, reasons for withholding information must be fully in line with exemptions under the
Environmental Information Regulations or the Freedom of Information Act 2000.
(b) If you have answered NO, please explain why the Final report should not be released into public domain
Executive Summary
7.
The executive summary must not exceed 2 sides in total of A4 and should be understandable to the
intelligent non-scientist. It should cover the main objectives, methods and findings of the research, together
with any other significant events and options for new work.
This project created a set of computer models for assessing pesticide risks to birds, mammals and aquatic
organisms, using improved approaches that account for variability and uncertainty in factors that influence
risk. The models produced by the project are freely available for use on the internet at www.webfram.com.
Before a pesticide is approved for use, the risks to non-target organisms are assessed. These risks
depend on two main factors: the toxicity of the pesticide to the organisms, and the amounts of pesticide
they are exposed to. Most current methods for assessing pesticide risks are deterministic – they use a
single value for toxicity, a single value for exposure, and produce a single estimate of risk. In the real
world, toxicity and exposure are not fixed; instead they vary both between species and between
individuals. Furthermore, many aspects of risk assessment involve uncertainty – for example, when
extrapolating toxicity from species tested in the laboratory to those exposed in the wild. Consequently, the
effects of pesticides are both variable and uncertain. Deterministic methods cannot represent this
variability and uncertainty. Instead, they provide a snapshot of one possible outcome. This may be
sufficient for simple screening assessments, but for refined assessments a more complete picture of the
possible outcomes and their uncertainty would be desirable.
Probabilistic methods take account of variability and uncertainty using probability distributions. Probability
distributions show the range and relative likelihood of possible values for each input and output, and thus
provide a fuller picture of the possible outcomes. This is a major advantage compared to deterministic
methods, which use only single values for inputs and outputs. However, uptake of probabilistic methods
has been hindered by a number of factors including lack of reliable data for input distributions, lack of
standardised methods for probabilistic calculations, the complexity of the methods, and concerns about
the validity of assumptions.
This project (Webfram1) is one of a set of inter-linked projects that sought to address these obstacles by
developing a suite of refined risk assessment models for addressing uncertainty and variability in higher
tier assessments and making them available to users via the internet.
The specific models that were implemented in this project address risks to aquatic organisms (models
from project PS2302, Webfram2), birds and mammals (models from project PS2303, Webfram3). The
sister projects designed the models, collated, evaluated and processed the relevant data, evaluated the
models by applying them to worked examples (case studies), and provided the algorithms and data for the
finished models to this project for implementation on the internet.
SID 5 (Rev. 3/06)
Page 2 of 21
This project, Webfram1, had an integrating role. First, it reviewed the available approaches for addressing
uncertainty and variability in risk assessment, and developed a general framework for applying these
approaches across the Webfram suite of projects. This work was closely integrated with preceding and
concurrent initiatives on probabilistic risk assessment in Europe and the USA, especially the EUFRAM
project (www.eufram.com). Second, this project implemented the models produced by Webfram2 and
Webfram3 as web-enabled software freely available for use on the internet at www.webfram.com. Third,
this project produced additional, generic models for quantifying variability and uncertainty in toxicity,
exposure and risk, to extend the potential applicability of the Webfram software to other taxonomic groups
in addition to aquatic organisms, birds and mammals. Finally, this project provided overall coordination for
the whole suite of projects, including the organisation of annual meetings with potential end-users and
stakeholders.
This report provides an overview of the results of this project, including a review of alternative methods for
addressing uncertainty and variability, a general framework of principles for using them in risk
assessment, and a summary of the models and how they are organised on the project website. The
models together with extensive supporting information and user instructions are available at
www.webfram.com.
Project Report to Defra
8.
As a guide this report should be no longer than 20 sides of A4. This report is to provide Defra with
details of the outputs of the research project for internal purposes; to meet the terms of the contract; and
to allow Defra to publish details of the outputs to meet Environmental Information Regulation or
Freedom of Information obligations. This short report to Defra does not preclude contractors from also
seeking to publish a full, formal scientific report/paper in an appropriate scientific or other
journal/publication. Indeed, Defra actively encourages such publications as part of the contract terms.
The report to Defra should include:
 the scientific objectives as set out in the contract;
 the extent to which the objectives set out in the contract have been met;
 details of methods used and the results obtained, including statistical analysis (if appropriate);
 a discussion of the results and their reliability;
 the main implications of the findings;
 possible future work; and
 any action resulting from the research (e.g. IP, Knowledge Transfer).
SID 5 (Rev. 3/06)
Page 3 of 21
WEB-INTEGRATED FRAMEWORK FOR ADDRESSING UNCERTAINTY &
VARIABILITY IN PESTICIDE RISK ASSESSMENT (WEBFRAM)
Andy Hart and Willem Roelofs
Central Science Laboratory, York, YO41 1LZ, UK
Introduction
EU and UK regulations require that, before a pesticide is approved for use, the risk to non-target organisms
including birds and mammals has to be assessed. EU Directive 91/414/EEC specifies criteria to be used in an
initial “first-tier assessment”, and states that pesticides which fail to meet these criteria may not be authorised for
use unless an “appropriate risk assessment” shows that it will cause no unacceptable impact. Various options for
these refined, higher tier risk assessments are identified in existing EU Guidance Documents, including both
deterministic and probabilistic approaches.
Most current methods for assessing pesticide risks are deterministic – they use fixed values for toxicity and
exposure, and produce a single measure of risk (e.g. a toxicity-exposure ratio). In the real world, toxicity and
exposure are not fixed, but variable. Furthermore, many aspects of risk assessment involve uncertainty – for
example, when extrapolating toxicity from test species to humans or wildlife. Consequently, the effects of
pesticides are both variable and uncertain. Deterministic methods cannot incorporate variability and uncertainty
directly. Instead, uncertain or variable factors are fixed to worst-case values, or dealt with subjectively using
expert judgement, or simply ignored.
Probabilistic methods take account of variability and uncertainty by using probability distributions to represent
them in risk assessment. Although this is a major advantage, uptake of probabilistic methods has been hindered
by a number of factors including lack of reliable data for input distributions, lack of standardised methods for
probabilistic calculations, the complexity of the methods, and concerns about the validity of assumptions.
This project (Webfram1) is one of a set of inter-linked projects that sought to address these obstacles by
developing a suite of refined risk assessment models for addressing uncertainty and variability in higher tier
assessments and making them available to users via the internet.
The specific models that were implemented in this project address risks to aquatic organisms (models from
project PS2302, Webfram2), birds and mammals (models from project PS2303, Webfram3). The sister projects
designed the models, collated, evaluated and processed the relevant data, evaluated the models by applying
them to worked examples (case studies), and provided the algorithms and data for the finished models to this
project for implementation on the internet.
This project, Webfram1, had an integrating role. First, it reviewed the available approaches for addressing
uncertainty and variability in risk assessment, and developed a general framework for applying these approaches
across the Webfram suite of projects. This work was closely integrated with preceding and concurrent initiatives
on probabilistic risk assessment in Europe and the USA, especially the EUFRAM project (www.eufram.com).
Second, this project implemented the models produced by Webfram2 and Webfram3 as web-enabled software
freely available for use on the internet at www.webfram.com. Third, this project produced additional, generic
models for quantifying variability and uncertainty in toxicity, exposure and risk, to extend the potential applicability
of the Webfram software to other taxonomic groups in addition to aquatic organisms, birds and mammals. Finally,
this project provided overall coordination for the whole suite of projects, including the organisation of annual
meetings with potential end-users and stakeholders.
This report provides an overview of the approaches and results of this project. The models together with
extensive supporting information and user instructions are available at www.webfram.com.
Scientific objectives
The scientific objectives agreed for the project are listed below. All have been fully achieved.
1. Review, evaluate and recommend the most appropriate existing mathematical/statistical approaches to
addressing uncertainty and variability.
2. Develop and evaluate a generic model framework that incorporates uncertainty and variability into the
assessment of pesticide risks to nontarget species.
3. Develop fully web-integrated software, by web-enabling the generic framework and a suite of specific models
(algorithms and data for the latter being provided by sister projects), freely accessible via a browser interface
with options to provide varying levels of access to different stakeholders.
SID 5 (Rev. 3/06)
Page 4 of 21
4. Establish a Coordinating Committee and effective means of electronic communication to ensure efficient
interaction and exchange of results between this project and the sister projects which will provide the specific
models and data for web-enabling in objective 3.
Objective 1. Review of approaches for addressing uncertainty and variability
Variability is defined as real variation in factors that influence risk. For example, toxicity varies between species,
and exposure varies in time and space. Variability matters because risk assessment usually needs to address a
range of relevant species and exposures, not just one particular species and one exposure.
Uncertainty is defined as limitations in knowledge about the factors that influence risk. For example, there is
uncertainty when we extrapolate toxicity from a small number of tested species to other, untested species, and
uncertainty when we extrapolate from mathematical models of exposure to the real world. Uncertainty matters
because decision-makers and stakeholders need to know the range of possible outcomes and their relative
likelihoods.
An important practical difference between uncertainty and variability is that obtaining further data can often
reduce uncertainty, whereas variability can be better quantified but not reduced by further data.
Uncertainties can be classified in various ways (e.g. Morgan & Henrion 1990, Cullen & Frey 1999).
Important types of uncertainty include:
 Uncertainty about distribution shape – often, several different distributions may be plausible for the same
model input, and may show similar goodness of fit to sample data.
 Sampling uncertainty – when a sample is used to estimate distribution parameters or to derive an empirical
distribution, there is uncertainty about their relationship to the true parameters or distribution for the larger
population from which the sample was drawn.
 Measurement uncertainty – various factors may cause random errors or bias in measurements or
experimental data.
 Extrapolation uncertainty – when it is necessary to extrapolate beyond the range of a dataset, or from one
type of data to another (surrogacy), there is uncertainty about how closely the extrapolated values match the
true values that are being estimated.
 Model uncertainty – there is often uncertainty about which of several alternative model structures best
represents real mechanisms and processes.
 Uncertain dependencies – there may be uncertainty about the nature, direction and magnitude of
dependencies between the model inputs.
 Ignorance – the risk may be influenced by factors of which we are unaware.
Quantitative methods for dealing with variability and uncertainty can be divided into two primary categories:
 Deterministic methods use point estimates (i.e. single values) to represent factors influencing risk.
 Probabilistic methods use probability distributions to represent variability and/or uncertainty in factors
influencing risk.
It is important to note that individual assessments often use deterministic methods to represent some factors and
probabilistic methods to represent others. There is thus a continuum between assessments that are entirely
deterministic and those that are entirely probabilistic.
Deterministic methods
Deterministic methods use point estimates to represent one or more factors in a risk assessment and treat them
as if they were fixed and precisely known. Point estimates represent a measured or estimated quantity by a single
number (e.g. the minimum, mean or maximum value), rather than a distribution. When all of the inputs to a risk
assessment are deterministic, the output will also be deterministic.
The key advantage of deterministic methods is simplicity. Their key limitation is that point estimates do not reflect
the variability and uncertainty that, in reality, affect almost all inputs to every assessment. Consequently, the
output is one possible value for the risk or impact: it does not reflect the range of possible values, nor give any
indication of their relative likelihood. This is potentially a very serious limitation, as it provides the decision-maker
with no information on the relative likelihood of different outcomes.
The common way of countering this limitation is to make the inputs conservative: i.e. to choose values that will
tend to exaggerate the risk. Decision-makers can then act on the result and assume that the actual outcome is
unlikely to be worse.
Choice of input values for deterministic assessments
Choosing suitably conservative input values to use for deterministic calculations is often difficult.
SID 5 (Rev. 3/06)
Page 5 of 21



Sometimes there are absolute lower and upper bounds for the possible values of an assessment input.
However, these are often much wider than the range of values which is really plausible, leading to extremely
unrealistic risk estimates.
If measurements are available, it must be remembered that the minimum and maximum recorded values will
generally underestimate the true range of possible values because the true minimum and maximum are very
unlikely to be observed in a limited sample.
Often, data will be lacking, limited, poor in quality, or only indirectly relevant (e.g. measured for a different
population or in different conditions). In these cases it may still be possible to identify a plausible range of
values, based on the available data, expert judgement and common sense.
In all cases, the rationale and justification for the values chosen should be reported in sufficient detail so that
others can review and evaluate the judgements that have been made.
Combining multiple conservative assumptions can quickly lead to a scenario that is extremely conservative or
even beyond the bounds of possibility. It is therefore important to consider which combinations are reasonably
plausible. The most objective approach would be to use probabilistic methods (see later) to estimate the
probabilities of different combinations. Otherwise, the plausibility of different combinations of input values can be
assessed subjectively, taking account of any relevant data or evidence, expert judgement and common sense.
The highly subjective nature of this assessment makes it essential to report the rationale and justification for the
selection of combinations, so that others can review and evaluate the judgements that have been made.
Choosing values for deterministic inputs, and deciding which combinations of values are appropriate to consider
in the assessment, involves two distinct types of judgement. First, it involves evaluating how conservative
(probable) the values are, which is a scientific judgement. Second, it involves deciding how conservative
(precautionary) to be, which is a policy judgement. Therefore, some interaction between scientists and policymakers is needed to decide what values are appropriate to use in a deterministic assessment. In many areas of
regulation this is achieved, in effect, by the establishment of regulations and/or guidance documents containing
standard sets of assumptions for use in deterministic assessments (e.g. EC 2002a,b). Consultation between
scientists and policy-makers is then only required for those cases that deviate from the standard procedure.
Sensitivity analysis for deterministic assessments
Given the difficulty of deciding on appropriate values for deterministic inputs it may be helpful to repeat the
assessment for a number of different scenarios, ranging from more typical or probable values to more
conservative or improbable ones. A set of alternative scenarios should provide the decision-maker with more
insight into the effect of variability and uncertainty in the assessment, and into the range of possible outcomes,
compared to what they would learn from a single result. This is a simple form of sensitivity analysis; a broad term
used to describe the diverse range of methodologies available for analysing the sensitivity of calculation outputs
to their inputs.
It may be helpful to summarise the results of sensitivity analysis graphically, showing the relationship between the
value of a particular input and the resulting assessment outcome. If ranges of values are being considered for
several different assessment inputs, each can be varied in turn while keeping the others constant. More complex
methods for sensitivity analysis provide various ways of systematically exploring multiple combinations of inputs.
Detailed descriptions, examples and evaluations of these methods may be found in the literature (e.g. Saltelli et
al. 2000, Mokhtari & Frey 2005).
Another important benefit of sensitivity analysis is that it helps to identify which inputs contribute most to
uncertainty and/or variability in the output. Those inputs that contribute most uncertainty may be good targets for
research, if it is desired to refine the assessment, whereas those factors that contribute most to variability may be
good targets for risk management, if it is desired to reduce the risk. Sensitivity analysis is also useful in
conjunction with probabilistic methods (see later).
Interval analysis
Interval mathematics can be used in cases when it is possible to specify not just a selection of different possible
values but the absolute minimum and maximum, forming for each input an interval representing the full range of
possible values. Correspondingly, the model output is also an interval: the range of possible outcomes but not
their probabilities. Special arithmetic procedures for calculation of functions of intervals used in this method are
available in computational tools that are capable of performing interval arithmetic on functions of variables that
are specified by intervals (e.g. Ferson 2002).
The primary advantage of interval mathematics is that it can address problems for which the probability
distributions of the inputs cannot be specified. In addition, interval analysis is capable of handling any kind of
uncertainty, no matter what its nature or source. However, this advantage is counterbalanced by the fact that the
method does not characterise the nature of the output interval, as both variability and uncertainty are forced into
one interval, nor does it estimate the probabilities of different values within that interval. Furthermore, as
SID 5 (Rev. 3/06)
Page 6 of 21
discussed earlier in relation to deterministic methods, it is frequently difficult to identify truly absolute upper and
lower bounds for assessment inputs.
Probabilistic methods
Probabilistic methods use distributions for one or more inputs to a risk assessment, to represent variability and/or
uncertainty in those inputs. If at least one of the inputs to a risk assessment is probabilistic, the output will also be
probabilistic.
The key advantage of probabilistic methods is that they provide an objective way to account for variability and
uncertainty in the factors influencing risk, using formal statistical methods. In principle this provides the means to
characterise not only the range of possible outcomes or impacts, but also their relative frequency or likelihood.
A variety of approaches exist for probabilistic analysis; those most commonly considered for risk assessment are
outlined in the following sections. The approaches differ in their underlying theory, in the way distributions are
specified, and in the way distributions are combined or propagated in the assessment, and in the type of output
they produce. Some approaches (e.g. 2-dimensional or 2D Monte Carlo) account for variability and uncertainty
separately, whereas others do not.
Fuzzy arithmetic
Fuzzy arithmetic is a generalization and refinement of interval analysis in which the bounds vary according to the
level of confidence or belief that one has that the value is within an interval. These confidence bounds are
described by so-called membership functions, which can be seen as stacked intervals, where the bounds become
wider when moving down from possibility level 1 to 0. At the highest possibility level, everyone would agree that
the values within that interval could contain the true, but unknown value. Close to the lowest possibility level, the
interval is much wider, indicating that only a few people would expect that the true, unknown value could be as
high or as low as the interval bounds. In other words, the possibility level at a certain value is an estimate of our
degree of belief of whether the parameter could have that value. Just as with confidence intervals, arithmetic
operations could be applied to fuzzy numbers. The output of fuzzy calculations, however, will be a distribution
rather than a single range. Another advantage of fuzzy arithmetic is that it does not require any information about
correlations between parameters. The main disadvantage of fuzzy arithmetic is that it still mixes variability and
uncertainty. In addition, many assumptions are necessary to define the fuzzy numbers, particularly when
information suggests that there are no outer bounds or when the information on possible bounds is scarce.
Probability bounds
Probability bounds is a method of combining uncertain distributions that does not require any assumptions about
either distribution shape or dependencies among variables, although such information can be used to some
extent if available. It first produces absolute (outer) bounds for the cumulative distribution of each input variable,
and uses these to derive absolute (outer) bounds for the output distribution (illustrated in Figure 1). However, an
important limitation is that this method does not provide any information on where the true distribution might lie
between the bounds. The method can work with many types of information, and each piece of information can
itself be uncertain. Probability bounds copes well with most types of uncertainty but has difficulties with sampling
uncertainty, which is important for the small sample sizes that are common in risk assessment. Probability bounds
may be very wide, but this is appropriate if there is genuinely little information about the inputs. Software for
probability bounds is provided by Ferson (2002).
Frequency of effect
(how often)
1
a
Uncertainty
(how sure)
0 .5
0
-2 0
-1 0
0
10
20
Magnitude of effect (how bad)
Figure 1. Example of output from probability bounds analysis. This method provides absolute bounds, within
which the cumulative distribution of the assessment endpoint must lie.
SID 5 (Rev. 3/06)
Page 7 of 21
1D Monte Carlo
Monte Carlo simulation combines distributions by taking large numbers of samples from each distribution at
random. A good introduction is provided by Vose (2000). In 1D (one-dimensional or first order) Monte Carlo, all
the input distributions are sampled together, and produce a single distribution for the output.
The following points are rather complex, but important for the construction and interpretation of 1D Monte Carlo
assessments.
 When all the input distributions exclude uncertainty and represent only variability, the output distribution
represents only variability in the assessment endpoint. It can be used to estimate specific percentiles of the
output distribution (e.g. the 5th percentile), but provides no confidence intervals and may give a false
impression of certainty.
 When input distributions representing variability and uncertainty are combined by 1D Monte Carlo, the output
distribution represents the combined effect of variability and uncertainty on the assessment endpoint. The
output distribution can be interpreted as representing our uncertainty about the assessment endpoint for a
single member of the statistical population or ensemble, selected at random.
The difference between 1D Monte Carlo including and excluding uncertainty is illustrated in Figure 2. The
distribution excluding uncertainty (labelled MLE in Figure 2) represents a central estimate of the CDF, whereas
the distribution combining both variability and uncertainty (labelled 1D) spans a wider range of possible values for
the assessment endpoint.
Monte Carlo methods can take account of dependencies between input variables, if information is available to
characterise them, but require strong assumptions about distribution shape. A number of software packages exist
for 1D Monte Carlo and are fairly easy to use (e.g. @Risk© and CrystalBall©).
100%
Cumulative
frequency
% of species
80%
60%
95% Conf. Interval
Median
EC50
2D
2D
40%
20%
0%
100
1D 1D
MLE
MLE
104
1000
105
Magnitude
Figure 2. Example of the difference between the output of different forms of Monte Carlo simulation for the same
input data. 2D Monte Carlo can produce a median estimate and confidence intervals for each percentile of the
distribution. 1D Monte Carlo combining variability and uncertainty produces a single distribution representing
uncertainty about a single member of the statistical population selected at random. Monte Carlo based on the
maximum likelihood (MLE) estimates of the mean and variance excludes uncertainty and produces a distribution
close, but not identical, to the median curve from 2D Monte Carlo.
2D Monte Carlo
Two dimensional (2D) Monte Carlo is very similar to 1D Monte Carlo except that it separates uncertainty and
variability. Again, a good introduction is provided by Vose (2000). In 2D Monte Carlo, distributions for uncertainty
and variability are sampled separately, so that the combined effect of the uncertainties can be shown as a
confidence interval around the output distribution (illustrated in Figure 2). The ability to show the effect of
uncertainties separately as confidence intervals makes 2D Monte Carlo especially useful. Statistical estimates
can be used for some types of uncertainty (e.g. measurement and sampling uncertainty, and extrapolations
calibrated by regression analysis). Other types of uncertainty must be estimated subjectively, although this has
been criticised as lacking mathematical rigor (Ferson, 2002). Facilities for 2D simulation are included in some
Monte Carlo software (e.g. CrystalBall©), although the number of samples that can be drawn is often limited. Like
1D Monte Carlo, 2D methods can take account of known or assumed dependencies between input variables but
require strong assumptions about distribution shape.
SID 5 (Rev. 3/06)
Page 8 of 21
Bayesian methods
Bayesian methods use a subjective concept of probability that is designed to make use of subjective information
(e.g. expert judgements, which are often used in risk assessment), as well as objective data. Bayes’ theorem is
used to combine existing (“prior”) knowledge with new data, a process sometimes called “updating”. This
produces “posterior” distributions for the assessment endpoint. For variables where no prior information is
available, “uninformative priors” are used. Confidence intervals can be generated for different percentiles of the
assessment endpoint, if required, so Bayesian methods can generate the same types of output as 1D and 2D
Monte Carlo (Figure 2). The concept of updating fits neatly with the process of tiered risk assessment, and can
also be used to combine different types of information (e.g., lab and field). Several different schools of Bayesian
approaches exist, e.g. robust Bayes uses families of uninformative prior distributions to counter concerns that
choosing between them may have undue influence on the results. Some of the theory and computations involved
in Bayesian methods are complex and require specialist expertise (for a simple introduction see Vose 2000).
However, this need not be an obstacle to their use if they can be provided in a pre-packaged form that is known
to be appropriate to the problem in hand (e.g. customised software or lookup tables).
Sensitivity analysis for probabilistic assessments
Sensitivity analysis can be applied to probabilistic assessments, although the practicalities depend on the type of
probabilistic method used. For 1D Monte Carlo, relationships between inputs and outputs can be examined
graphically or by computing correlation coefficients, both of which are provided in the commonly-used Monte
Carlo software packages @Risk© and CrystalBall©. The same can potentially be done for 2D Monte Carlo, with
the addition that sources of variability and uncertainty can be analysed separately, although this is more
cumbersome computationally and is not included in the 2D Monte Carlo function in CrystalBall©. The
contributions of different sources of variability and uncertainty in probability bounds can be analysed by repeating
the calculations and omitting each source in turn. Simple forms of sensitivity analysis can also be used to
examine model assumptions, e.g. by conducting multiple analyses with different assumptions for distribution
shapes and dependencies. This helps to increase the robustness of the assessment as a whole but quickly
becomes cumbersome if there are many such assumptions to consider. More detailed accounts of sensitivity
analysis are given by Saltelli et al. (2000) and Mokhtari & Frey (2005).
Choosing between probabilistic methods
Each probabilistic approach has strengths and weaknesses, and there is no general consensus on which
approaches should be preferred. In a report on quantitative microbial risk assessment, the European
Commission’s former Scientific Steering Committee (SSC) stressed the need for further evolution of the
methodology of quantitative risk assessment, and of criteria for the appropriate use of different probabilistic
approaches. The SSC concluded that “a quick harmonisation at the present state-of-the-art should be avoided”
(European Commission 2003). The lack of a generally preferred approach implies that prospective users need to
consider for each assessment which of the alternative approaches is appropriate.
Issues that users might consider in choosing between alternative approaches include:
 Robustness. Users may reasonably prefer methods that are robust in the sense of not requiring
questionable assumptions. This may lead them to favour interval analysis or probability bounds, since these
do not require information about distribution shape and dependencies between variables, which is often
limited. However, see next issue.
 Bounds versus probabilities. Probability bounds (and also interval analysis) provide upper and lower
bounds on the outcome, but cannot estimate the probability of different outcomes. Bounds are useful if the
nature of the risk is such that it is sufficient to know whether a particular outcome is inside or outside the
bounds of possibility. This may apply to severe outcomes (e.g. extinction of a species) where decisionmakers wish to find management options that ensure a probability of (very close to) zero. But for most risk
problems, the occurrence of adverse outcomes cannot be ruled out and decision-makers want to know their
probabilities, which cannot be estimated from bounding methods. This need may outweigh the preceding
issue of robustness: approximate probabilities may be more useful than no probabilities.
 Separation of uncertainty and variability is useful because it distinguishes the main sources of uncertainty
(which may be good targets for research) from the main sources of variability (which are not reducible by
research but may be options for risk management, if they are controllable). Furthermore, methods that
separate uncertainty and variability are needed if decisions hinge on particular percentiles of a variable, e.g.
the probability that an impact occurs in 1% of fields, rather than the probability that an impact occurs in a
randomly-selected field. 2D Monte Carlo, Bayesian methods and probability bounds separate uncertainty and
variability, whereas 1D Monte Carlo, interval analysis and fuzzy arithmetic do not.
 Use of subjective information. If objective data is lacking for key inputs, then it may be desirable to use
subjective estimates (e.g. expert opinion). Bayesian approaches provide a formal theoretical framework for
this, although subjective information can also be used less formally in 2D Monte Carlo and probability
bounds.
 Ease of use is an important practical consideration. 1D Monte Carlo is relatively easy to use and available in
many software packages; 2D Monte Carlo is of intermediate difficulty and available in fewer software
packages. Probability bounds and Bayesian methods are (or are perceived as) more difficult to use, but may
SID 5 (Rev. 3/06)
Page 9 of 21
provide better solutions depending on the problem. Any of these methods can be made easy to use if readyto-use models are provided within user-friendly software, although the more complex methods will pose
greater challenges to user understanding and acceptance.
Choice of approaches for addressing uncertainty and variability in Webfram
2D Monte Carlo was selected as the primary approach for use in the Webfram framework. This choice was based
primarily on two considerations:
 The desire to quantify both variability and uncertainty with distributions, so as to characterise not only the
range of alternative outcomes but also their relative frequency (variability) and likelihood (uncertainty). This
argues in favour of Monte Carlo or Bayesian approaches rather than deterministic approaches, interval
analysis or probability bounds.
 The desire to separate variability and uncertainty, so that distributions (and hence percentiles) for variable
outcomes can be provided together with confidence or probability intervals to represent their uncertainty. This
was considered more relevant and interpretable for decision-making than a distribution for randomly-chosen
outcomes (e.g. for a random field or random individual). This argues for a 2D rather than 1D Monte Carlo
approach.
It was decided to adopt a 2D Monte Carlo approach rather than fully Bayesian equivalent because the former was
simpler to develop, simpler to implement as online software, and easier to communicate. However, this does not
preclude the use of Bayesian methods to model input distributions for use in (propagation by) 2D Monte Carlo,
where appropriate. For example, Bayesian methods were used for normally- and log-normally distributed
parameters and for a linear regression relationship in the Webfram3 (bird and mammal) models.
Monte Carlo approaches require more assumptions than probability bounds, especially concerning distribution
shapes and dependencies. One possibility is to use Monte Carlo and probability bounds in parallel, and compare
the results (e.g. Regan et al. 2002): the Monte Carlo results provide an estimate of the median distribution of
outcomes and its probability density under specific assumptions regarding distribution shapes and dependencies,
while the probability bounds (which should be wider) provide a measure of the additional uncertainty associated
with those assumptions. That was not done in Webfram partly because the extra work would have reduced the
number of models that could be developed, and partly because of difficulties encountered in implementing
probability bounds (for representing sampling uncertainty, and for modelling variables that must sum to 1 such as
dietary composition in the Webfram3 models).
It was recognised that, whichever approach is adopted, assumptions will be required, only some of the relevant
uncertainties will be quantified, and some parameters that influence risk may be omitted altogether (e.g. the
Webfram3 models consider only exposure via the dietary route). Such limitations are unavoidable in risk
assessment and apply equally (or more so) to deterministic methods. They result in additional uncertainty about
the relation between the assessment outcome and the “true” risk. It is therefore essential to evaluate the potential
impact of these additional uncertainties as far as is possible, so that they can be taken into account in decisionmaking. A qualitative approach for doing this was developed in Webfram3, as a way of responding to questions
raised in the EUFRAM project about the reliability of model outputs. The approach involves systematically
identifying unquantified uncertainties and tabulating them together with an evaluation of their impact on the
assessment outcome. The approach is potentially applicable to all types of assessment, and is therefore included
as a standard part of the Webfram framework (see below). It has also been adopted by the European Food
Safety Authority for dealing with uncertainty in human dietary exposure assessments (EFSA 2006).
Objective 2. General framework for incorporating uncertainty and variability in the
assessment of pesticide risks to non-target organisms
DEFRA’s specification for the Webfram projects stated that the selected mathematical/statistical approaches to
addressing uncertainty and variability should be developed into a generic framework, and that modules for
different groups (aquatic, terrestrial vertebrates, terrestrial invertebrates) should be made compatible with the
framework. It was recognised that the specifics of the models (their structure as well as data) would need to differ
between groups. The aim of the framework is to identify those things that can be done in a consistent way
between groups. This is desirable to ensure best practice and avoid unnecessary divergence of approaches. It
should also lead to savings in effort required to develop model software, as some components will be transferable
between modules.
As anticipated in the project proposal, development of the general framework within Webfram proceeded in close
consultation with parallel work in the EUFRAM project (www.eufram.com). The resulting framework is presented
in the form of a concise description of the main steps that should be taken in developing a risk assessment,
focussing on where and how uncertainty and variability are taken into account, and identifying key issues that
should be considered. In addition, a shorter checklist of the key points is provided, which is intended as an aide
memoire for those developing and reporting risk assessments, and for those evaluating them.
SID 5 (Rev. 3/06)
Page 10 of 21
General framework for incorporating uncertainty and variability in risk assessment
The following sections outline a general framework for conducting quantitative risk assessments. The main
elements are the same for both deterministic and probabilistic approaches. Aspects that are specific to one
approach or the other are identified in the text. The framework can be used when assessing the risk for a
particular pesticide, but in Webfram they are applied to developing scenario-based models that can be applied to
a number of pesticides with similar uses (crop, date of application).
Problem definition
Developing a risk assessment model, whether for a specific pesticide or for repeated use in a given scenario,
should start with a process of problem definition (sometimes called problem formulation) to ensure the
assessment will meet the regulatory need. Problem definition should first define the scope and objectives of the
assessment and then specify what form of output will meet those objectives: these steps are discussed in the
following two sections.
Problem definition is a vital task that requires effective interaction between risk assessors and decision-makers
(sometimes called risk managers) to ensure the risk assessment is fit for purpose, i.e. addresses the relevant
question and communicates the results effectively. Effort will be saved if a similar assessment has been done
previously, or if a standardised problem definition has been established, but a check should always be made to
confirm that the problem definition is truly appropriate for the issue under consideration.
Assessment scope and objectives
The first step is to define the objectives and scope of the assessment, starting with the basic elements that should
also be defined for qualitative assessments:
 Define the scenario to be considered (crop, time of year, non-target species, etc.)
 Define the types of effect to be considered (e.g. direct/indirect, mortality, reproduction, etc.).
 Define the timescale of the assessment (often implied by the type of effect, e.g. acute/chronic).
 Define the spatial boundaries of the assessment (e.g. regional, national, EU).
These choices should be made to reflect as closely as possible the protection goal – what the decision-maker
wants to protect, and what types of effect they wish to limit or prevent. If it is too difficult to assess the types of
effect that ultimately concern the decision-maker, then the assessment can be directed at a simpler measure of
effect (e.g. effects on a species rather than an ecosystem). However, if this is done then it will be necessary to
consider the relation between these two measures of effect when interpreting the results.
If the decision-maker is concerned about particular sources of variability and uncertainty in the assessment
inputs, then it is helpful to identify these as part of the problem definition. However, this should not preclude
additional sources being examined later, if it becomes apparent during the assessment that they are important.
Finally, any limitations on the scope of the assessment should be specified, e.g. if the assessment is restricted to
considering particular routes of exposure or types of impact.
Assessment endpoint (measure of risk)
The next step is to define a measure of risk that will address the assessment objectives and provide a suitable
basis for decision-making: we refer to this key output as the assessment endpoint. Assessment endpoints should
provide a meaningful and readily interpretable measure of impact on something the decision-maker wishes to
protect or prevent, i.e. one of the types of effect listed in the preceding step. Each distinct type of effect will
require a separate assessment endpoint.
The assessment endpoint measure may be a point value, e.g. percentage mortality for individuals exposed to a
pesticide. Here the measure is a single value (though it will be uncertain, see later). On the other hand, it may be
important to the decision-maker to know how the measure of impact is distributed over one or more dimensions,
e.g. how the level of mortality varies between species, or the proportion of species in which it exceeds some
specified value.
Defining the dimensions over which variability is to be assessed is very important in probabilistic assessments, for
two reasons: to ensure that decision-maker and assessor are both clear about what output is to be produced; and
because it affects the structure of the model and the types of input distributions that are needed. For example, if
the assessment endpoint is the proportion of species that exceed some specified level of mortality, the
assessment is likely to require input distributions for variation between species of any factors that influence
mortality (toxicity, exposure, etc.). Two elements are needed to define each dimension over which the endpoint
varies: the units of the distribution, and the population or ensemble of units to which the distribution refers. For
example, the units are species for a distribution representing variation between species, and the population or
ensemble might be all species found in the exposed habitats, or a subset of these (e.g. all fish species).
SID 5 (Rev. 3/06)
Page 11 of 21
Defining the dimensions over which the assessment endpoint varies is also important even if all parts of the
assessment will be treated deterministically. This is because consideration needs to be given to how to choose
appropriate point values to represent each input to the assessment. Otherwise, (a) it is difficult to judge where the
assessment outcome lies in the true distribution (i.e. how conservative the assessment is), and (b)
inconsistencies may arise in the construction of the model, e.g. if an average in space for one input is combined
with an average over time for another.
It is also important to discuss in advance with the decision-maker how uncertainty affecting the assessment is to
be characterised. For example, if uncertainty is to be expressed as a distribution or confidence interval for the
assessment endpoint, this would require a probabilistic method separating variability and uncertainty. If
confidence intervals are required, it is helpful to agree in advance what level they should be (e.g., 90 or 95%
confidence intervals).
Finally, we emphasise that it is important to define the output first, and then use that to infer what the inputs
should be. It is tempting to do the opposite – i.e. allow the available input datasets to determine the nature of the
output. This can cause under- or over-estimation of variability or produce output distributions that have no realworld interpretation1.
Developing and implementing the assessment
Having defined the output that is required, the next step is to develop and implement the an appropriate
assessment model to generate the desired output from the available data. Developing a model may require
substantial effort if the assessment scenario is new. Effort will be saved if a similar assessment has been done
previously, or if a standardised assessment has been established, but a check should always be made to confirm
that the assessment is truly appropriate for the issue under consideration.
The key steps in developing and implementing the assessment are outlined below.
1.
Identify the factors and mechanisms that influence the assessment endpoint, and how they interact.
It may help to draw a diagram representing the various factors and mechanisms and showing their
interactions, e.g. as a flow chart (this is sometimes referred to as a conceptual model). An example from
Webfram3 is shown in Figure 3.
BW: body weight
DEE: daily energy
expenditure
GE: gross energy
content of food
FIR: Food intake
per day (wet wt)
M: moisture
content of food
PT: Fraction of
diet obtained in
treated areas
AE: assimilation
efficiency
AppRate:
application rate
RUD: residue per
unit dose
C: Initial
concentration
PD: Fraction of
food type in diet
Exposure
(mg/kg bw/day)
C: Concentration
on food type
LD50
(mg/kg bw)
AV: Avoidance omitted
Slope of
dose- response
%Mortality
Figure 3. Example of a conceptual model diagram, for acute risks to birds (from project PS2303, Webfram3).
2.
Consider each part of the model in turn, and decide which factors and mechanisms might contribute
significantly to variability in the assessment endpoint. These will need to be represented by suitable
distributions or point estimates in the model.
3.
For each variable in the assessment, identify the appropriate population or ensemble. This depends
on the ensemble of the assessment endpoint (see earlier) and the model structure. For example, if the
assessment endpoint is effects in water bodies adjacent to treated fields, then water bodies adjacent to
treated fields will be the relevant ensemble for spatial variation of exposure, and the set(s) of species found
in such water bodies will be the relevant ensemble for between-species variation in toxicity.
1
For example, combining within-species variation for one parameter with between-species variation for another would create
an output distribution of entities that are all one species with respect to the first parameter but represent multiple species with
respect to the second parameter.
SID 5 (Rev. 3/06)
Page 12 of 21
4.
Identify what data are available that can help in quantifying each factor and mechanism.
5.
Decide whether any extrapolations or adjustments are needed to model the key factors and
mechanisms from the available data. Extrapolation or adjustment may be required to deal with various
issues including:

Lab to field extrapolation.

Non-random sampling.

Surrogacy – where one type of data is used as a surrogate for another type.

Incompatible data populations or ensembles, e.g. a model may require data on variation over time, but
the only available data may be on variation between sites.

Inappropriate averaging/incomplete information. Scientific publications frequently report just averages, or
averages and standard deviations without the raw data or any information on distribution shape, which
are necessary to properly estimate variation and uncertainty.
If issues of this type do require the use of adjustment or extrapolation factors, then they become, in effect,
part of the assessment model and should be identified as such in the model plan. Also, consideration should
be given to whether these factors should be represented by point values or by distributions (if they are
subject to substantial variability or uncertainty).
6.
Consider each part of the model in turn to identify possible sources of uncertainty. Decide which
uncertainties might contribute significantly to uncertainty in the assessment endpoint, and which of these will
be quantified using distributions (if any). Decide what to do about important uncertainties that cannot or will
not be quantified (e.g. use best estimates or conservative values). Different types of uncertainty, and
methods for dealing with them, are discussed under objective 1 (above).
7.
Consider carefully for each input distribution whether it contributes uncertainty or variability to the
output. This again depends partly on the ensembles. For example, a species sensitivity distribution
contributes variability if the output population comprises multiple species, but uncertainty if the output
population comprises individuals of a single species. Distributions estimated from data generally contain both
variability and measurement uncertainty, which can be separated if the distribution for measurement
uncertainty can be specified (e.g. Zheng & Frey 2005). If a distribution includes both variability and
uncertainty and they cannot be separated, it may be better to treat the distribution as uncertainty as this
reflects the state of knowledge more accurately than if it were treated as variability.
8.
Identify potential dependencies affecting the assessment. Dependencies occur where the value of one
variable depends upon the value of another variable (e.g. food intake may be positively related to body
weight). It is important to consider dependencies carefully, as they can have a major impact on the
assessment outcome.
9.
Decide on which type of deterministic or probabilistic method to use. The choice of method will depend
on various factors including whether bounds or probabilities are required for the output, whether it is desired
to separate variability and uncertainty, ease of use, and what expertise is available. It may also be influenced
by the preceding steps e.g. the availability of data for estimating distributions and dependencies. For reasons
discussed earlier in this report, 2D Monte Carlo was adopted as the primary approach for propagating
variability and uncertainty in Webfram.
10. Define the distributions or point estimates to be used for variable and uncertain inputs and
dependencies. This is a critical step, as inappropriate choices will give misleading or invalid results for both
deterministic and probabilistic outputs.
The methods used for representing distributions and dependencies depend on the type of probabilistic or
deterministic approach that has been chosen. For example, probability bounds uses p-boxes to represent
distributions, and assumes either independence or perfect correlation between each pair of variables
(Ferson, 2002). There is a wide range of options for specifying distributions for Monte Carlo and Bayesian
approaches, including parametric and non-parametric methods, and subjective approaches based on expert
opinion. The most familiar form of dependency is linear correlation, but there are many others. Whatever
methods are used, special attention should be paid to ensuring the tails of distributions are reasonable, as
they are frequently critical to the outcome of risk assessment but poorly represented by data except in very
large samples. An extensive literature exists on these topics, including substantial chapters in publications on
probabilistic risk assessment (e.g. Vose 2000, Cullen & Frey 1999, US EPA 1997, 2001).
Deterministic approaches use point estimates to represent variability, uncertainty and dependencies. Ideally,
these should be chosen based on a careful statistical analysis similar to that required for a probabilistic
treatment, so that the relation of the point estimate to the distribution is known. In practice, however,
deterministic point estimates are often chosen in a rather arbitrary way – e.g. by using the average or
maximum value from a small set of measurements, without any statistical analysis. The disadvantage of this
is that it provides very poor control over the conservatism of the assessment.
SID 5 (Rev. 3/06)
Page 13 of 21
It should be emphasised that specifying appropriate distributions and dependencies requires a high level of
statistical expertise, as well as a clear understanding of the assessment model, and expert knowledge of the
parameters in question (e.g. ecology, ecotoxicology, environmental chemistry, etc.).
11. Express the entire assessment model as a set of mathematical equations, so that they can be used for
calculations.
12. Consider conducting sensitivity analyses. These may help in various ways, e.g. exploring alternative
assumptions or scenarios, and analysing the relative importance of different sources of variation and
uncertainty.
13. Select appropriate software to carry out the computations and generate outputs. The selection of software
will depend in part on the selected method for conducting the quantitative analysis, as some methods may
only be possible with specific software.
14. Program and run the model including any sensitivity analysis, and generate the outputs.
15. Check the outputs of the assessment. Results should be evaluated carefully for any indication of model
mis-specification or programming errors, or assumptions that have led to unrealistic results.
16. Qualitatively evaluate sources of variability, uncertainty and dependency that have not been
quantified. Ultimately it is never possible to quantify all uncertainties affecting an assessment. It is therefore
essential to state clearly in the assessment, (a) which uncertainties have been quantified, and (b) what
uncertainties remain unquantified and their potential implications for the assessment. The following steps
may assist in evaluating unquantified uncertainties:

Systematically list or tabulate all unquantified sources of uncertainty.

Qualitatively evaluate the direction and relative magnitude of each unquantified uncertainty in terms of its
impact on the assessment outcome

Consider re-running the assessment with different assumptions to assess the influence of potentially
important uncertainties or dependencies (a form of sensitivity analysis)

Consider using probability bounds to explore the impact of potentially important uncertainties about
distribution shape and dependencies (in effect, putting conservative outer bounds around the assessment
result, e.g. Regan et al. 2002)

Qualitatively evaluate the possible combined effect of all the quantified and unquantified uncertainties and
dependencies on the predicted outcome (how different could the “true” outcome be?).
An example of qualitative assessment of uncertainties using a tabular format is included in the report on the
sister project, Webfram3 (birds and mammals, project number PS2303).
17. Consider other lines of evidence. Assessors should seek to provide an overall characterisation of risk,
combining the outcome of the quantitative assessment with any other relevant lines of evidence, e.g. direct
measurement of the assessment endpoint in field studies, data from monitoring of commercial use, or
evidence from different but related pesticides. However, it should be remembered that such types of
information are also subject to uncertainty and bias. Therefore, if other lines of evidence are to influence the
characterisation of risk, then they should be subject to the same standards of assessment as the quantitative
assessment. Sometimes, different lines of evidence can be compared quantitatively, e.g. using Bayesian
updating to combine model predictions with field measurements. No line of evidence should be discounted
unless there is convincing evidence that it is invalid.
18. Consider the wider ecological consequences. Full characterisation of risk may require consideration of
wider ecological consequences of effects indicated by the assessment. This is often necessary because the
quantitative assessment is constrained (for reasons of practicality) to simple measures of risk that are only
indirect or partial measures of the assessment objectives or protection goal. For example, quantitative
estimates may relate to effects on a local population of one species, but decision-makers may be more
interested in impacts at the level of the community or landscape. Any conclusions about such extrapolations
should be subject to the same standards of evidence as the quantitative assessment itself. For example,
arguments about the ability of the ecological systems to absorb or recover from impacts should be supported
by data.
19. Communicate the assessment and results to decision-makers and others as appropriate. This will include
presenting results for the assessment endpoints that were agreed at the outset, together with explanation
and interpretation to assist understanding. In addition, it is essential to document the assessment in detail,
including the justification for each choice made in planning the assessment model, and each component of
the resulting equations. It may aid communication if the primary output of the assessment is a concise
summary focussing on the key results, but fuller details should be accessible to support the credibility of the
assessment and facilitate peer review.
SID 5 (Rev. 3/06)
Page 14 of 21
Summary checklist
The following list is intended as a concise aide memoire when developing risk assessments, when drafting
assessment reports, or when evaluating reports of assessments produced by others, to ensure that all the major
issues are covered.

Introduction. Is the purpose and context of the assessment clearly explained? Does the assessment as a
whole follow an established approach accepted by the relevant authorities, or is it novel?

Problem definition. Are the protection goals identified? Are the assessment scenario and endpoints
appropriate and consistent with the protection goals?

Assessment model. Is the assessment model appropriate? Does it include all relevant routes of exposure?
Does it include all relevant factors and combine them in appropriate ways?

Probabilistic methods and software. Are any probabilistic methods (e.g. 1D or 2D Monte Carlo, etc.) and
software appropriate, and are they correctly used? Have they been accepted by relevant authorities or
independent peer review? For computational methods, is it demonstrated that the number of iterations is
sufficient to produce stable outputs?

Input data. Consider every model input in turn. Are the data used for each input appropriate? Are they
consistent with the statistical populations needed to generate the assessment endpoint? If extrapolation is
required, has this been justified and are the associated uncertainties considered?

Distributions. Are appropriate distributions used to represent variability and uncertainty? Are they
compatible with the nature of the parameter concerned? Is goodness of fit evaluated graphically and
statistically? Are the tails of the distribution appropriate? Is any truncation justified and correctly applied?

Uncertainties. Are potential uncertainties of all types sufficiently considered? Are potentially significant
uncertainties quantified, or addressed through conservative assumptions, or is their influence evaluated
qualitatively?

Deterministic methods. Where point estimates are used to represent variable or uncertain inputs, is the
choice of values justified? Is their effect on the conservatism of the assessment output evaluated?

Dependencies. Is the possibility of dependencies between variables been sufficiently considered? Are
potentially significant dependencies quantified, or is their influence evaluated qualitatively?

Risk characterisation. Are exposure and effects combined in an appropriate way? Does it quantify the
frequency, magnitude and duration of effects? What is the probability of exceeding any relevant decisionmaking criteria? What types of organisms are at risk? Is sensitivity analysis used and, if so, which sources of
variability and uncertainty are most influential? Are the strengths and weaknesses of the assessment, and
their overall effect on the reliability of the assessment, evaluated? Is it useful to compare probabilistic and
deterministic results? Are other relevant sources of evidence critically evaluated? Are the wider ecological
implications considered? What is the applicability of the results to different scenarios, regions etc.?

Conclusion. Is the overall conclusion fully supported by the assessment? Are options for further refinement
identified, if relevant?

Are sufficient details reported to repeat the assessment?
Objective 3. Develop web-enabled software for a suite of models provided by the sister
projects
The specific models that were implemented in this project address risks to aquatic organisms (models from
project PS2302, Webfram2), birds and mammals (models from project PS2303, Webfram3). The sister projects
designed the models, collated, evaluated and processed the relevant data, evaluated the models by applying
them to worked examples (case studies), and provided the algorithms and data for the finished models to this
project for implementation on the internet.
This project, Webfram1, implemented the models produced by Webfram2 and Webfram3 as web-enabled
software freely available for use on the internet at www.webfram.com. In addition, this project produced a set of
generic models for quantifying variability and uncertainty in toxicity, exposure and risk, to extend the potential
applicability of the Webfram software to other taxonomic groups in addition to aquatic organisms, birds and
mammals.
The organisation of the models on the website is illustrated in Figure 4.
SID 5 (Rev. 3/06)
Page 15 of 21
Figure 4. Diagram of Webfram internet site including main models and user functions.
Welcome & User login
New model
Bird & mammal
Saved models
Aquatic
Instructions & info
Generic
Bird/mammal
Aquatic
Tox data entry
Scenarios
(choice of 5)
Risk modelling
(acute or chronic)
Exposure
Herbest model
(recovery potential)
LIfe history
database
Tox data entry
& goodness
of fit
Exposure
data entry
Effects
Output options
Exposure model
Default data
User RUDs
User DT50
Spray drift
7 scenarios:
- 4 arable
- 3 orchard
SSD
Drainflow
3 soils
3 climates
10 crops
Single value
(eg lowest
or mesocosm)
SSD
Outputs
SSD
Dose response
Exposure
User data
(eg monitoring)
Overlay
graph
Food intake
Exposure
Overlay plot
TERs
% Mortality
Risk graphs
(JPC & FA)
Single value
(eg FOCUS)
Details of the models for birds, mammals and aquatic organisms are described in the final reports of projects
PS2302 and PS2303 (Webfram2 and Webfram3), including data, assumptions, modelling methodologies and
outputs. Full details are provided on the website, via the “instructions and info” page and as help pages attached
to specific steps within each model.
The generic model allows the user to fit lognormal distributions to toxicity and exposure distributions and produce
a number of outputs including species sensitivity distributions (SSDs), exposure distributions, overlay plots (SSD
and exposure distribution plotted together) and 3 types of joint probability curves (JPCs). Examples of these
outputs are illustrated in Figures 5 – 8 below.
The use of SSDs, exposure distributions, overlay plots and joint probability curves is well established in
ecotoxicology literature (e.g. Posthuma et al. 2002 and chapters therein), although only sparingly used in
regulatory assessment in the past. New features introduced in the Webfram generic model are (a) estimation of
confidence intervals for the whole of each distribution (rather than just selected percentiles, e.g. HC5), and (b) a
new format for the JPC, plotted as an exceedance distribution to show the proportion of exposures that exceed
any given effect level (Figure 8).
SID 5 (Rev. 3/06)
Page 16 of 21
100%
95% Conf. Interval
Median
EC50
90%
80%
% of species
70%
60%
50%
40%
30%
20%
10%
HC5
0%
0.1
1
10
100
EC50 [g / L]
1000
104
Figure 5. Example of using a species sensitivity distribution produced by the Webfram generic model, using
hypothetical data. Points represent the EC50 for 9 tested species (species names can be shown as labels). The
arrows show the HC5 (hazardous concentration for 5% of species), together with confidence intervals. In this
hypothetical case, the HC5 is 3.9 g/L (95% confidence interval 0.1 – 18.0 g/L).
100%
90th %-ile
90%
% of exposures
80%
70%
60%
95% Conf. Interval
Median
Data
50%
40%
30%
20%
10%
0%
0.1
1
10
100
Concentration [g / L]
1000
104
Figure 6. Example of a cumulative distribution for exposure produced by the Webfram generic model, using
hypothetical data on concentrations in surface water. The arrows show the 90 th percentile concentration. In this
hypothetical case, the 90th percentile concentration is 22 g/L (95% confidence interval 13 – 50 g/L).
SID 5 (Rev. 3/06)
Page 17 of 21
% of exposures
100%
80%
80%
Median - Exposure
95% Conf. Interval
Median - Toxicity
95% Conf. Interval
Exposure Data
EC50
60%
40%
60%
40%
20%
20%
0% -6
10
10-4
0.01
1
100
104
106
% of species exceeding toxicity endpoint
100%
0%
Concentration [g / L]
Figure 7. Example of risk characterisation using an overlay graph, produced by the Webfram generic model. The
distribution on the left is the same data as in Figure 6, but plotted as a complementary cumulative distribution or
exceedance function. The distribution on the right is the same SSD as in Figure 5. The degree of overlap between
the two curves gives a visual impression of the level of risk.
100%
90%
% of exposures
80%
70%
95% Conf. Interval
Median
60%
50%
40%
30%
20%
10%
0%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%
% of species exceeding toxicity endpoint
Figure 8. Example of an exceedance risk distribution or exceedance profile plot. This is one of three types of joint
probability curve (JPC) that can be produced with the Webfram generic model. The arrows show how to estimate
the proportion of exposures causing more than a given level of effect; in this case,10% or more species will
exceed their toxicity endpoint in 33% of exposures (95% confidence interval 5% - 96%).
Clear communication of assessment outputs is one of the key requirements for successful introduction and
acceptance of probabilistic methods. To assist in meeting this requirement, a series of investigations was
conducted in the EUFRAM project (www.eufram.com) to explore the effectiveness of a range of tabular and
graphical formats. Lessons from these studies were used to design and refine the output formats used in the
Webfram project, including the graphical formats illustrated in Figures 5-8. To maximise the effectiveness of the
graphical displays, the software includes extensive options for user control of the graphical displays, including the
following:
 Plotting distributions in cumulative or complementary cumulative (exceedance) form
 Option to include or exclude confidence intervals, and to vary confidence level (e.g. 90%, 95% etc)
 Option to alter minimum or maximum value on each axis, and to plot in natural or logarithmic scale
 Option to show or hide data points, and labels for points (e.g. species names on SSDs)
 Control on colours used for points and curves
 Default settings on all options for quick use
 Option to download finished graphic in emf, png or fig formats
SID 5 (Rev. 3/06)
Page 18 of 21
The website contains nearly all of the features identified as potentially desirable in the original project proposal, as
shown below (with comments in italics):
User interface
 Home page
 Information pages, help pages, links to relevant websites (material provided by module projects)
 Scenario selection screens
 Data entry screens
 Output/report option selection screens
 Option to do model calculations online or in batch
 Results screens including a range of graphical and tabular outputs, suitable for use in reports/documents,
with option to print/download/save online
 Option to save data/scenarios for later re-use
 Consistent look and feel for all modules
 Default options and data for every module (so user can just run or modify a standard example)
 Information screens on the standard examples (generated by case studies in module projects)
 List-server to announce changes/additions to registered users (could be added easily if Webfram is endorsed
for regulatory use)
 Option to limit functionality available to different types of users (decided in consultation with PSD to make
same functions available to all users)
Model inputs
 Option to vary analysis options (e.g. number of iterations) – but decide during project how much to limit these
choices
 Default distributions for all parameters (from module project case studies)
 Option to specify own distribution (type and parameters) for each parameter (decided in consultation with
PSD to allow changes only for selected parameters, where users are likely to have more relevant data)
 Option to use default or user’s raw data for bootstrapping?
Uncertainty analysis
 Type(s) of uncertainty propagation to be decided during project and likely to be limited to one or two, but may
include: first-order error analysis, 1D and 2D Monte Carlo, Bayesian updating, P-bounds, interval analysis.
(2D Monte Carlo selected, see objective 1 in this report).
 Need to be able to do uncertainty propagation within the software generated by the project, or by running
existing packages using macros.
 Sensitivity analysis (methods to be decided during project) (done for specific models with example datasets in
sister projects PS2302 and PS2303)
 Need to consider what can be done online vs. in batch mode
Objective 4. Coordination with sister projects
A Committee was established at the start of the work, to coordinate interactions between this project and the
other Webfram projects. The Committee comprised:
 Project manager: Helen Thompson (CSL) (chair)
 Consortium members: Colin Brown (Cranfield University, subsequently York University), Theo Traas (RIVM),
Andy Hart (CSL), Jim Siegmund (Cadmus Group, subsequently freelance consultant)
 Technical adviser on web software aspects: Matthew Atkinson (CSL)
 Representatives invited from all the key module projects (Brown and Hart, above, plus Geoff Frampton of
Southampton University), and the project on acceptability of effects (Mark Crane, Crane Consultants,
subsequently Watts and Crane Associates).
 Additional members nominated by DEFRA and PSD.
The Committee met at the start of the project and subsequently approximately annually, in conjunction with the
annual meetings with stakeholders.
The Coordinating Committee held four annual stakeholder meetings at CSL (near York). Invitations were sent to a
range of invitees including members of the Environmental Panel of the Advisory Committee on Pesticides, and
the pesticides industry. Each meeting included presentations on the progress of all the Webfram projects and
extensive discussion sessions, and the final meeting also included hands-on practical sessions and break-out
groups to provide feedback on the prototype software. The meetings generated a lot of valuable feedback and
comment which was very useful in refining the outputs of the projects.
SID 5 (Rev. 3/06)
Page 19 of 21
Action resulting from this research
During the course of the project, the ongoing work and prototype models were presented and demonstrated at a
series of events including scientific conferences (SETAC, BCPC, Fresenius, Agchem Forum), workshops
(EUFRAM), and annual meetings with invited stakeholders. The models and supporting documentation were also
subjected to peer review by independent experts from academia and industry. The feedback from all these
activities was used to refine and improve the final versions of both models and software.
The final models web-enabled by the project are freely available for use online at www.webfram.com. They are
being evaluated by the UK authorities to decide on whether and how they should be used in regulatory risk
assessments.
Possible future research
Although this project has made substantial advances, there are many areas in which further work could be
considered. These include:
 Transfer of the software to a new host server
 Enhancements and extensions to the existing models
 Additional models for different scenarios and/or different taxonomic groups
 Refinements to the user interface
Acknowledgements
The authors are very grateful to the UK Pesticides Safety Directorate for funding; to Jim Siegmund (formerly of
The Cadmus Group Inc., now a freelance consultant at [email protected]) for design and
implementation of the Webfram website and programming of some of the models; to Helen Thompson, Colin
Brown and other members of the project steering group; to Mark Clook of PSD for advice and feedback; and to
the many other individuals who have provided valuable feedback and peer review on the models and software
developed by the project.
References
Cullen AC and Frey HC, 1999. Probabilistic techniques in exposure assessment. A handbook for dealing with
variability and uncertainty in models and inputs. Plenum Press, NY.
EFSA 2006. Guidance of the Scientific Committee on a request from EFSA related to Uncertainties in Dietary
Exposure Assessment. The EFSA Journal, 438, 1-54.
European Commission, 2002a. Guidance document on aquatic ecotoxicology in the context of Directive
91/414/EEC. Sanco/3268/2001 rev.4 (final), 17 October 2002, Brussels.
European Commission, 2002b. Guidance document on risk assessment for birds and mammals under Council
Directive 91/414/EEC. Sanco/4145/2000, 25 September 2002, Brussels.
European Commission, 2003. Risk assessment of food borne bacterial pathogens: Quantitative methodology
relevant for human exposure assessment. Appendix 3 in: The future of risk assessment in the European Union.
The second report on the harmonisation of risk assessment procedures. Scientific Steering Committee. European
Commission, Brussels, Belgium.
Ferson, S. 2002. RAMAS Risk Calc 4.0 Software: Risk Assessment with Uncertain Numbers. Lewis Publishers,
Boca Raton, Florida.
Mokhtari A, Frey HC. 2005. Recommended practice regarding selection of sensitivity analysis methods applied to
microbial food safety process risk models. Human and Ecological Risk Assessment 11: 591-605.
Morgan MG and Henrion M, 1990. Uncertainty. Cambridge University Press.
Posthuma L, Suter GW III, Traas TP. 2002. Species sensitivity distributions in ecotoxicology. Lewis Publishers,
Boca Raton, FL.
Regan HM, Hope BK, Ferson S. 2002. Analysis and portrayal of uncertainty in a food-web exposure model.
Human and Ecological Risk Assessment, 8: 1757-1777.
Saltelli A, Chan K, EM Scott. 2000. Sensitivity analysis. John Wiley & Sons Ltd, Chichester.
SID 5 (Rev. 3/06)
Page 20 of 21
US EPA. 1997. Guiding principles for Monte Carlo analysis. US Environmental Protection Agency, Risk
Assessment Forum, Washington DC. EPA Document No. EPA/630/R-97/001, March 1997. Available at
http://epa.gov/osa/spc/htm/probpol.htm.
US EPA. 2001. Risk assessment guidance for Superfund. Volume III – Part A, Process for conducting
probabilistic risk assessment. US Environmental Protection Agency, Office of Emergency and Remedial
Response, Washington DC. Available at www.epa.gov.
Vose, D. 2000. Risk analysis: a quantitative guide. 2 nd edition. John Wiley & Sons Ltd.
Zheng JY, Frey HC, 2005. Quantitative analysis of variability and uncertainty with known measurement error:
Methodology and case study. Risk Analysis 25: 663-675.
References to published material
9.
This section should be used to record links (hypertext links where possible) or references to other
published material generated by, or relating to this project.
The primary output of this project is the modelling software and associated explanatory and supporting
information which are freely available for use online at www.webfram.com. The software includes detailed
instructions and extensive help screens including more detailed information on the methods, data and
assumptions used in the models.
SID 5 (Rev. 3/06)
Page 21 of 21