Atmospheric Environment 43 (2009) 4822–4832
Contents lists available at ScienceDirect
Atmospheric Environment
journal homepage: www.elsevier.com/locate/atmosenv
Skill and uncertainty of a regional air quality model ensemble
R. Vautard a, *, M. Schaap b, R. Bergström c, B. Bessagnet d, J. Brandt e, P.J.H. Builtjes b, J.H. Christensen e,
C. Cuvelier f, V. Foltescu c, A. Graff g, A. Kerschbaumer h, M. Krol i, P. Roberts j, L. Rouı̈l d, R. Stern h,
L. Tarrason k, P. Thunis f, E. Vignati f, P. Wind k
a
LSCE/IPSL Laboratoire CEA/CNRS/UVSQ, 91198 Gif sur Yvette, Cedex, France
TNO Built Environment and Geosciences, Utrecht, The Netherlands
c
SMHI, SE-601 76 Norrköping, Sweden
d
INERIS, Parc Technologique Halatte, Verneuil en Halatte, France
e
NERI, University of Aarhus, P.O. Box 358, DK-4000 Roskilde, Denmark
f
European commission, DG JRC, Institute for Environment and Sustainability, TP 483, I-21020 Ispra, Italy
g
UBA, Wörlitzer Platz 1, 06844 Dessau-Roßlau, Germany
h
Freie Universität, Berlin, Germany
i
IMAU, Institute of marine and atmospheric research, Utrecht, The Netherlands
j
Shell Global, HSE Department, P.O. Box 1, Chester CH1 3SH, United Kingdom
k
EMEP/MSC-W, P.O. Box 43, Blindern, 0313 Oslo, Norway
b
a r t i c l e i n f o
a b s t r a c t
Article history:
Received 30 January 2008
Received in revised form
24 September 2008
Accepted 24 September 2008
Recently several regional air quality projects were carried out to support the negotiation under the Clean
Air For Europe (CAFE) programme by predicting the impact of emission control policies with an
ensemble of models. Within these projects, CITYDELTA and EURODELTA, the fate of air quality at the scale
of European cities or that of the European continent was studied using several models. In this article we
focus on the results of EURODELTA. The predictive skill of the ensemble of models is described for ozone,
nitrogen dioxide and secondary inorganic compounds, and the uncertainty in air quality modelling is
examined through the model ensemble spread of concentrations.
For ozone daily maxima the ensemble spread origin differs from one region to another. In the neighbourhood of cities or in mountainous areas the spread of predicted values does not span the range of
observed data, due to poorly resolved emissions or complex-terrain meteorology. By contrast in Atlantic
and North Sea coastal areas the spread of predicted values is found to be larger than the observations.
This is attributed to large differences in the boundary conditions used in the different models. For NO2
daily averages the ensemble spread is generally too small compared with observations. This is because
models miss highest values occurring in stagnant meteorology in stable boundary layers near cities. For
secondary particulate matter compounds the simulated concentration spread is more balanced, observations falling nearly equiprobably within the ensemble, and the spread originates both from meteorology and aerosol chemistry and thermodynamics.
Ó 2008 Published by Elsevier Ltd.
Keywords:
Ensemble modelling
Air quality model
Model evaluation
Uncertainty
1. Introduction
To effectively reduce the adverse effects of air pollutants on
human health, thorough process knowledge, resulting in reliable
predictions for the future air quality, is mandatory. The evolution of
air quality in the next decades depends on many factors, such as the
growth of pollutant emissions due to the worldwide economic
development and its concentration in areas of high activity like
* Corresponding author.
E-mail address: [email protected] (R. Vautard).
1352-2310/$ – see front matter Ó 2008 Published by Elsevier Ltd.
doi:10.1016/j.atmosenv.2008.09.083
megacities, or climate change (Stevenson et al., 2005; Langner et al.,
2005; Forkel and Knoche, 2006; Hedegaard et al., 2008; Meleux
et al., 2007; Giorgi and Meleux, 2007 and references therein) and
its consequences like more frequent heat waves (Meehl and
Tebaldi, 2004) inducing poor air quality episodes (Vautard et al.,
2005a; Ordonez et al., 2005; Hodzic et al., 2006; Solberg et al.,
2008; Struzewska and Kaminski, 2008). Besides climate change
effects, the evolution of air quality in Europe will be affected by
a combination of the change in european emissions, and the
hemispherical background concentrations (Szopa et al., 2006).
While the latter factor is impossible to control at the European
scale, the former mainly results from the combination of increased
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
economic activity and the concerted efforts put into regional
environmental policies. The evaluation of how efficient such policies might be in controlling air pollution can only be carried out
with the use of numerical models.
Supporting the definition of the European Strategy for Air
Pollution through the Clean Air For Europe (CAFE) Programme and
in the framework of the Convention on Long Range Transport of
Atmospheric Pollution, (CLRTAP United nations – Economic
Commission for Europe), several projects have been set up in order
to evaluate the regional responses to emission reduction scenarios.
These projects rely strongly on long-term air quality simulations
using several chemistry-transport models. CITYDELTA, the first
project (Cuvelier et al., 2007), was devoted to the evaluation of
emission scenarios for 2010 at the scale of European cities. EURODELTA is its regional counterpart (Van Loon et al., 2007; Schaap
et al., submitted for publication), and deals with air quality changes
at the European scale. In both projects several models are used,
revealing the spread of possible modelling responses to emission
scenarios.
Using an ensemble of models rather than a single model to
predict air quality for emission scenarios actually gives two new
informations:
(i) The average (or the median) over this ensemble of responses is
a new response by itself, which is expected to have a smaller
RMS because individual model error cancel each other to
a certain extent.
(ii) The spread of the ensemble can be a measure of the uncertainty in model predictions.
For obvious reasons, it is not possible to directly verify responses
of several emission scenarios and their associated uncertainties.
However, a first evaluation whether these new possibilities, offered
by model ensembles, are realistic can be achieved by simulating
a period in the past. For point (i) a direct comparison between
individual models skills and ensemble mean model skills can be
carried out using routine air quality observations. The better
performance of the ensemble average or median has been shown in
several recent studies for air quality (Delle Monache and Stull,
2003; Pagowski et al., 2005; McKeen et al., 2007; Van Loon et al.,
2007; Schaap et al., submitted for publication) as well as for
transport of passive tracers (Galmarini et al., 2004; Riccio et al.,
2008). The evaluation of the relation between the spread of values
obtained in the model simulations and the actual uncertainty in air
quality simulation has received focus in a few studies (Delle Monache et al., 2006; Vautard et al., 2006). It can be tackled using
methods developed in the framework of ensemble weather forecasting (Molteni et al., 1996; Talagrand et al., 1998; Jollife and Stephenson, 2003). The variability in the EURODELTA model ensemble
spread has been shown to give a fair representation of the uncertainty for ozone and secondary inorganic aerosols (Vautard et al.,
2006; Schaap et al., submitted for publication). However, this
finding was based on global, average information without distinguishing regions within Europe or individual sites, neither for
skill nor for spread analyses. This article is designed to provide
a more general and spatially detailed analysis of the features of air
quality ensemble modelling for Europe. For this purpose we use the
EURODELTA modelling results for several pollutants: ozone,
nitrogen dioxide and secondary organic aerosols, as already used in
two previous studies (Van Loon et al., 2007; Schaap et al., submitted
for publication).
Section 2 contains a brief description of the models and observations used. Section 3 describes the skill characteristics of the
ensemble. The ensemble spread properties and its origins are
described in Section 4. Section 5 contains a conclusion and a short
discussion.
4823
2. Models and observations
2.1. Models
Within the EURODELTA project, seven air quality models have
been run over an extended period of time, the year 2001. The
models are CHIMERE (Schmidt et al., 2001; Bessagnet et al., 2004),
DEHM (Christensen, 1997; Frohn et al., 2002), EMEP (Simpson et al.,
2003; Fagerli et al., 2004), LOTOS-EUROS (Schaap et al., 2008 and
references therein), MATCH (Andersson et al., 2007 and references
therein), REM-CALGRID (RCG; Stern et al., 2003; Beekmann et al.,
2007) and TM5 (Krol et al., 2005). All models simulate ozone,
precursors, and chemically speciated particulate matter (PM) with
at least 2 size categories: fine (size less than 2.5 mm) and coarse
particles (size between 2.5 and 10 mm). All models, except the
global TM5 research model, have been used in their regional-scale,
limited-area versions designed for long-term simulations, which
implies some compromise between vertical/horizontal resolutions
and the extent of processes description. The horizontal resolution
of the models ranges from 25 to 100 km, while the vertical resolution varies from one model to another. The lowest model layer
lies in the range 20–100 m, and only a few model layers generally
resolve the boundary layer. The models are off-line chemistrytransport models forced either by global or regional meteorological
analyses or nudged meteorological simulations. For more details
about model formulations and comparisons the reader is referred
to Van Loon et al. (2007) and Schaap et al. (submitted for publication) and references therein.
Models were run over the full 2001 year and concentrations
were saved every hour. Surface concentrations are interpolated to
the sites where observations are available. All the results presented
along this article are calculated from statistics of these models
output and the corresponding observed concentrations.
2.2. Observations
For ozone and nitrogen dioxide, most observations used in this
study are gathered from the EMEP database (http://www.emep.int)
and some are taken from the AIRBASE database (http://dataservice.
eea.europa.eu/dataservice). To provide comparisons with spatial
scales consistent with models resolution, urban or industrial sites
were removed from the databases. Data for one station in Switzerland was obtained from the Swiss Environmental Agency. Due to
the poor representation of mountainous areas in coarse-resolution
models, only stations below an altitude of 1000 m were considered.
However this did not excluded valley sites. In order to homogenise
data coverage, a reduced set of stations was selected in densely
covered areas. The selection was based on the availability of
observations of additional compounds, such as NO2. Finally, only
stations lying within all model domains are considered. These
choices, together with the constraint of data availability, led to a set
of 96 sites for ozone and 69 for NO2. Additional diagnostics were
carried out using the ‘‘Ox’’ mixture (Ox ¼ O3 þ NO2). These diagnostics were possible only over sites where the two species are
measured simultaneously (67 sites).
The difficulty for models to simulate the mass of particulate
matter (PM10 or PM2.5) over Europe has been recognized (Van
Loon et al., 2004; Schaap et al., 2008). The underestimation of total
particulate mass is, among others, a result from the lack of emissions of fugitive dust, resuspended matter (Vautard et al., 2005b),
a plausible underestimation of primary carbonaceous particles
(Schaap et al., 2004; Tsyro, 2005), the inaccuracy of secondary
organic formation (Simpson et al., 2007), the difficulty of representing primary PM emission from wood burning and other sources
(Tsyro et al., 2007) and a more general lack of process knowledge
(Stern et al., 2008). By contrast, simulated secondary inorganic
4824
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
compounds have been shown to compare better with observations
(Van Loon et al., 2004; Schaap et al., submitted for publication).
Therefore, we chose to study only the characteristics of model
ensemble spread for the last group of compounds. These are also
the most important components from air quality policy considerations as they are mainly anthropogenic in origin.
Observations of these secondary inorganic compounds (NO3,
NH4 and SO4) were gathered from the EMEP database (http://www.
emep.int), together with the additional German Melpitz site. As for
gas compounds, mountain and other high altitude stations above
1000 m were not considered.
Following Schaap et al. (submitted for publication), we distinguish data for nitrate and ammonium measured by different
techniques, which may lead to data misinterpretation (see e.g.
Schaap et al., 2004). Nitrate data obtained from cellulose filters
were interpreted as total nitrate and compared with the sum of
particulate nitrate and gaseous nitric acid simulated by models.
Aerosol nitrate and ammonium observations obtained from inert
filters were used and compared with the simulated aerosol phase
only, although we are aware that they are prone to losses at
temperatures above about 20 C. Total nitrate and ammonium data
which were not obtained in a single (denuder) filter pack set-up
were disregarded.
For total nitrate measurements, there are 20 stations (resp. 20
also for total ammonium measurements) while 6 (resp. 9) stations
have particulate nitrate (resp. ammonium) observations. Sulphate
is taken at 37 sites.
For ozone and Ox we only study the behaviour of daily maxima
during the spring and summer seasons (April to September), in
order to focus on the highest values found in regional episodes. For
nitrogen dioxide and secondary inorganic compounds only daily
averages data were available.
O3 Daily Max Models
NO3 Daily Average Models
Ox Daily Max Models
NH4 Daily Average Models
NO2 Daily Average Models
SO4 Daily Average Models
O3 Daily Max Ensemble
NO3 Daily Average Ensemble
Ox Daily Max Ensemble
NH4 Daily Average Ensemble
NO2 Daily Average Ensemble
SO4 Daily Average Ensemble
1.8
r=0.3
1.6
r=0.4
r=0.5
r=0.6
1.4
r=0.7
1.2
r=0.8
1
0.8
r=0.9
0.6
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Fig. 1. Taylor plots of Ozone, Ox (daily maxima), NO2, NO3, NH4 and SO4 (daily averages), averaged over each set of stations. The radial coordinate is the standard deviation of the simulated concentration normalized by the standard deviation of the
observations, and the azimuthal coordinate is the arccosine of the correlation. Each
colour represent a species (see legend in the graph). Solid circles stand for individual
models while triangles stand for the skill of the ensemble mean.
3. Skill of the model ensemble
To represent the global skill of the models and of their ensemble
average, Fig. 1 shows, in a single diagram, the normalized ‘‘Taylor
plots’’ (Taylor, 2001) for all models and various compounds.
Correlations lie above 0.5 in most cases, with higher values for
ozone species than for PM or NO2 species. Not shown in Fig. 1 are the
total nitrate and ammonium skill which exhibit poorer skill than
the nitrate and ammonium skill. However, a general feature is the
superior skill (correlation coefficients around 0.7 or more) of the
ensemble mean relative to any individual model. For air quality
compounds, this was also found by previous studies (Delle Monache
and Stull, 2003; Van Loon et al., 2007) and can be explained, in
mathematical terms (Van Loon et al., 2007), by the mutual cancellation of various types of errors.
3.1. Gas phase species
Ozone and Ox daily maxima are the best simulated compounds,
with average correlations lying between 0.5 and 0.85. The simulation of Ox has a slightly superior skill than that of ozone because
local, non-representative, titration effects by fresh nitrogen oxide
emissions do not affect Ox while it does for ozone. The variance of
simulated oxidant concentrations is generally underestimated,
a possible consequence of the relatively low model grid resolutions,
which does not allow sharp and concentrated plumes and other
subgrid scale variability to be represented correctly.
Fig. 2 shows the spatial distribution of the biases of the
ensemble mean for each compound. For ozone daily maxima,
the ensemble mean bias is negative over central and southern
Europe, especially in the Alps and around the Mediterranean
Sea, while it is positive along Atlantic coastal areas. A careful
investigation and comparison of the simulated and observed
concentrations behaviour in North-western Europe shows that
only 2–3 models exhibit strong overestimation along the Northwestern coastal areas, and this affects the ensemble mean
model predictions in this area. One possible explanation is the
overestimation in boundary conditions for ozone for these
models. The Ox concentrations are also overestimated in coastal
areas, but to a smaller extent, which is consistent with this
hypothesis. However, most models use observation-based
(hence unbiased) ozone boundary conditions: EMEP, RCG,
LOTOS are based on Logan (1998) climatology, MATCH also uses
observations. DEHM, TM5 and CHIMERE use ozone boundary
concentrations from large-scale simulations. Another origin of
negative bias in coastal areas is insufficient titration with
nitrogen monoxide, as the bias is reduced in this area when
considering Ox instead of ozone. The negative bias for daily
maximum Ox in continental areas is larger than for ozone. This
may be partly due to a significant systematic overestimation of
daytime NO2 concentrations by NO2 monitors using molybdenum converters (Steinbacher et al., 2007). Nevertheless, the
analysis reveals a general difficulty for models to reproduce the
absolute value of the high ozone daily maxima. This is especially
true in most polluted areas where NO2 is also present in
significant amounts, or in complex-terrain areas where ozone
plumes are guided and concentrated within valleys, and where
plumes are smaller than the grid resolution.
The spatial distribution of time correlation of the model
ensemble mean is shown in Fig. 3. For ozone daily maxima highest
values are obtained in central/northern Europe, and lowest values
in complex-terrain or coastal areas. As expected the resolution of
models do not generally allow meso-scale breeze processes to be
described. Possibly, for coastal stations, the great difference in
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
O3 Bias
4825
Ox Bias
45
45
35
35
25
25
15
15
5
5
-5
-5
-15
-15
-25
-25
-35
-35
-45
-45
-55
-55
NO2 Bias
SO4 Bias
2.6
2.2
1.8
1.4
1.0
0.6
0.2
-0.2
-0.6
-1.0
-1.4
-1.8
-2.2
-2.6
-3.0
26
22
18
14
10
6
2
-2
-6
-10
-14
-18
-22
-26
-30
NO3 Bias
NH4 Bias
5.00
2.00
1.00
0.50
0.35
0.25
0.15
0.05
-0.05
-0.15
-0.25
-0.35
-0.45
-0.50
-1.00
-2.00
-5.00
5.00
2.00
1.00
0.50
0.35
0.25
0.15
0.05
-0.05
-0.15
-0.25
-0.35
-0.45
-0.50
-1.00
-2.00
-5.00
Fig. 2. Spatial distribution of concentration biases (time average simulated ensemble mean minus observed) for all studied compounds and stations, in mg m3. The time average
runs over spring and summer 2001 for daily maximum of ozone and Ox species, and over the whole 2001 year for all other species. For nitrate and ammonium solid squares stand
for total (gas þ particle, TNO3 and TNH4) concentrations and circles for particulate-phase only (NO3 and NH4).
deposition velocities to sea and land surfaces also influences the
model skill, because this process is sensitive to land-fractions in
their respective grid points.
It is found that Ox correlations are slightly higher than ozone
correlations almost everywhere. The most likely reason is the
weaker sensitivity of Ox than O3 to nitrogen monoxide levels, which
may have a small-scale structure due to non-representative local
emissions or complex-terrain boundary layer flows (see also Van
Loon et al., 2007). The skill of simulated NO2 at regional-scale was
not documented in previous studies. For this species, correlation
coefficients on a daily basis are more homogeneous among models
and range between 0.5 and 0.7 (see Fig. 1). Because it is rapidly
transformed from emitted NO, this species can be almost considered to be a primary species, with a variability mostly driven by
dispersion and emission. Hence, the skill of a model strongly relies
on its ability to simulate low-dispersion conditions (stable
boundary layers, low winds). As shown in Fig. 1, the range of
simulated variability for NO2 is larger than that for ozone, reflecting
this higher sensitivity to meteorology and model resolution. Again
the skill of the ensemble average (r ¼ 0.7) is superior to that of any
single model. Its standard deviation, however, is twice as low as the
observed one, indicating that the ensemble mean does not capture
the extremes.
This is reflected by the distribution of the bias (Fig. 2), which is
negative almost everywhere. The distribution of correlation is fairly
homogeneous in the Northern part of Europe. The smallest correlations are obtained in Spain and also in mountainous areas of
central Europe. Correlations are highest over the Benelux region
where the NO2 concentrations are generally high due to the high
emission density of nitrogen oxides.
4826
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
O3 Correlation
Ox Correlation
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
NO2 Correlation
SO4 Correlation
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
NO3 Correlation
NH4 Correlation
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
-0.1
-0.2
Fig. 3. As in Fig. 2 for the distribution of correlation coefficients.
3.2. Secondary inorganic aerosol species
The particulate matter species studied in this article are
moderately well simulated. As noted by Schaap et al. (submitted for
publication), the ensemble mean sulphate is generally underestimated while that for (total) nitrate is overestimated (Fig. 2). The
single models either over or underestimate these species systematically. Total as well as particulate ammonium exhibits a more
balanced simulation. The correlation coefficients for individual
models range from 0.4 to 0.7 (Fig. 1) for sulphate, ammonium and
nitrate, but total nitrate and ammonium have a much poorer
correlation (not shown). The ensemble mean has a significantly
higher correlation than individual models for nitrate and sulphate,
exceeding 0.6. The same holds for ammonium.
The correlations for the secondary inorganic aerosols (SIA)
compounds are generally highest in North-Western and Central
Europe and lowest in Spain, as for NO2, showing that the
concentrations are also sensitive to meteorology and dispersion. In
addition, Schaap et al. (submitted for publication) identified
uncertainties in formation routes as key issues related to the
modelling of SIA. The relatively low number of stations unfortunately does not allow one to draw strong conclusions. This point
both with the challenge represented by the measurement of some
of these compounds (nitrate especially) make it difficult to express
the real uncertainty of the model results.
4. Spread of the ensemble and uncertainty
4.1. Theoretical concepts
We now examine whether the daily concentration distributions
issued from the ensemble provide a useful representation of
uncertainty. The ensemble is generally assessed by two essential
properties: reliability and resolution. An ensemble is said to be
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
reliable if, on average over a large number of cases, the probability
density of the observation is equal to the probability density of the
ensemble values. If the ensemble spread is too small, e.g. due to
a missing source or formation route, we expect the observation to
fall more often than expected off the ensemble range. Such would
be the case if all models used the same driving meteorology with
identical errors. On the contrary, the ensemble can also use an
exaggerated range of (input) parameters, in which case one expects
the predicted concentration distribution to have a wider range than
that of the observations. In both cases the ensemble has a poor
reliability and the uncertainty is either underestimated or overestimated. Reliability can be measured by means of a rank histogram (Talagrand et al., 1998), showing statistics of the rank of the
observation within the predicted ensemble of values. A reliable
ensemble has an equiprobable distribution of rank, which should
be reflected by a flat histogram.
The resolution of the ensemble is linked to the sharpness of the
predicted distributions and to the added value of the information
provided by the ensemble. If, for instance, the predicted distribution is every day the ‘‘climatological’’ distribution, the ensemble
prediction system has no added value (no resolution). Provided that
the predicted distributions are reliable in the above sense, the
added value of the ensemble increases as the ensemble spread
decreases. The resolution is therefore linked to (and can be
measured by) the average ensemble spread. These concepts can be
formally derived from the Brier skill score for ensemble prediction
(Talagrand et al., 1998; Jollife and Stephenson, 2003).
4827
low ranks (resp. for TNH4 the high ranks) are overrepresented. The
rank histogram for NO3 indicates a too large ensemble spread for
this compound. This will be analysed further below. The NO2
histogram exhibits an insufficient spread, probably due to the too
coarse horizontal and vertical resolutions in combination with
a poor description of vertical exchange in a stable boundary layer.
The highest NO2 values are not reached by any of the models.
When discussing the balance of rank histograms the question is
often raised as to whether it is not only due to chance, and is only
valid for this specific set of models. Such is not the case, as
demonstrated in Fig. 5, where the Ox histograms are shown when
using all models, and when excluding successively each model
contribution to the data set. For each experiment, the rank histogram (which contains 7 cases instead of 8) remains relatively
balanced. The more accentuated U shape obtained when excluding
Model #1 or Model #5 shows that these models significantly
contribute to the ensemble spread. For other species than Ox, the
general features of the rank histograms are also fairly insensitive to
the exclusion of any model.
As shown in Vautard et al. (2006), for ozone only, the histograms
have a more symmetrical shape when the model ensemble bias is
removed (histograms not shown here). However, the ensemble bias
removal does not modify the ensemble spread so that ensemble
reliability remains affected by too large or too small spreads. Hence,
we find here that NO2 histogram still exhibits a strong distortion
indicating an underestimation of uncertainty while NO3 has the
reverse property. For the other species the global rank histograms
are relatively flat (not shown).
4.2. Characteristics of the EURODELTA ensemble
4.3. Spatial distributions of the ensemble properties
Fig. 4 shows the rank histograms of the 6 compounds studied in
this article, with data from all sites put together in a single histogram. Histograms have shapes that depend on species. For Ox the
histogram indicates a reliable behaviour of the ensemble. For
ozone, sulphate and ammonium the histogram has an imbalance
due to a bias of the ensemble: Ozone is generally overestimated
while sulphate and ammonium are underestimated. For TNO3 the
We now examine the spatial distribution of the ensemble
characteristics. In this section only unbiased ensembles are
considered in order to focus on spread properties. We display, in
Fig. 6, the normalized spread, calculated as the ratio of the timeaveraged standard deviation of daily ensembles to the standard
deviation of the observed concentration, for each site, for ozone and
Fig. 4. Rank histograms of the EURODELTA ensemble for all compounds under study, data from all stations being taken together. The histograms show the number of counts of the
rank of the daily averaged observation (ozone and Ox daily maximum) within the ensemble of 7 models. The large difference between the counts range of values for different species
is due to the large difference in the number of sites.
4828
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
Fig. 5. Rank histograms obtained for Ox when using all models (a) or when successively excluding each model’s contribution from the ensemble.
NO2. We also display a measure of the reliability, the rank histogram distortion (U shape or reversed U shape) of the unbiased
ensemble, as the difference between the 4 extreme rank counts
(ranks 0,1, 6 and 7) and the central rank counts (ranks 2, 3, 4 and 5),
divided by the total number of counts. When this reliability index is
positive, the rank histogram has a U shape and the ensemble
underestimates the uncertainty, while the reverse occurs when the
reliability index is negative.
For ozone, the normalized spread is relatively large over the
coastal Atlantic regions. The mean difference between models is
almost as large as the standard deviation of the observations,
meaning that the ensemble has a low resolution and the added
value, relative to the ‘‘climatological distribution’’ of the ensemble
prediction is weak. Over this area and more generally over NorthWestern Europe, the reliability index is negative, meaning that the
rank histograms have a reversed U shape at each station. The
uncertainty of the prediction is overestimated, the observation
falling more often in the middle of the ensemble range than near its
boundaries.
Fig. 7 shows the time series of the unbiased ensemble and the
observation, over the spring and summer 2001, for the site of
Keldsnor (a coastal site at the Baltic Sea, situated in the southern
part of Denmark), where the reliability index is low and negative
(0.42), and the normalized spread is large (0.995). The variability
of ozone daily maximum at this site is weak with only a few
moderate values near 140 mg m3, and a relatively constant value
during spring, while the model range is large. Such behaviour is
found in many North-western Europe sites, indicating that the large
ensemble spread is due to the spread in the models upwind
boundary conditions, but could also result from differences in
model processing of coastal areas (land/sea mask, dynamics, .).
The situation is somewhat different at the site of Zegveld (The
Netherlands, Fig. 7b), where the normalized spread is much lower
(0.60), but the reliability index remains negative (0.38). In this
case, due to the proximity of large regional precursor emissions, the
site undergoes relatively frequent summertime ozone episodes.
The observed variability increases and now dominates the
ensemble spread due to boundary conditions, so that the ensemble
prediction has an added value. However the uncertainty remains
overestimated by the ensemble and the observation often falls in
the centre of the range.
Over Central and Southern Europe the normalized spread is low,
thus the range of the ensemble predictions is small compared to the
observed variability, and thus the ensemble has a high resolution.
However at many sites the reliability index is positive, meaning that
the uncertainty is underestimated and the range of predicted
values is too small. Such is typically the case at the site of Ternay
(Fig. 7c), which actually is located in the suburb of the city of Lyon.
In this case the sharp and concentrated ozone city plumes are not
captured by the regional scale models, and the ensemble fails to
represent the highest peak values. This situation also occurs in
mountainous areas where ozone plumes can be concentrated in
valleys. A more ideal situation is found in other inland sites such as
the Donon site (Fig. 7d), for which the reliability index is close to
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
4829
Fig. 6. Spatial distribution of the reliability index and normalized spread (see text), for ozone and NO2.
Fig. 7. Time series of the unbiased ensemble prediction (thin black curves) and the
observation (thick red curve) at selected sites showing characteristic behaviour for
ozone. (For interpretation of the references to colour in this figure legend, the reader is
referred to the web version of this article.)
Fig. 8. Same as Fig. 7 for unbiased NO2 ensemble time series and observations at other
selected sites.
4830
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
NO3 Spread
NO3 Reliability
5.0
4.0
3.0
2.0
1.5
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
NH4 Spread
5.00
2.00
1.00
0.50
0.35
0.25
0.15
0.05
-0.05
-0.15
-0.25
-0.35
-0.45
-0.50
-1.00
-2.00
-5.00
NH4 Reliability
5.00
2.00
1.00
0.50
0.35
0.25
0.15
0.05
-0.05
-0.15
-0.25
-0.35
-0.45
-0.50
-1.00
-2.00
-5.00
5.0
4.0
3.0
2.0
1.5
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
SO4 Spread
SO4 Reliability
5.00
2.00
1.00
0.50
0.35
0.25
0.15
0.05
-0.05
-0.15
-0.25
-0.35
-0.45
-0.50
-1.00
-2.00
-5.00
5.0
4.0
3.0
2.0
1.5
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Fig. 9. Same as Fig. 6 for the secondary organic particulate matter compounds.
zero (0.03), while the normalized spread is low (0.39). In this case,
the added value of the ensemble is high and the uncertainty is fairly
well estimated. Several sites, mainly located from Eastern France to
Eastern Germany, have the same properties.
For NO2, the ensemble underpredicts the uncertainty in most
stations (large positive reliability index). This may have several
causes. The proximity of local centres of emissions to the
measurement stations, especially in valleys, is a source of variability
that cannot be captured by large-scale regional models with relatively low resolution. Such is for instance the case of the site of
Gravanches (central France, Fig. 8c), where reliability index is close
to 1 and normalized spread is close to zero. A reverse situation is
found at a few stations along the coast of the North Sea or British
Channel. In this case, the ensemble spread is large, probably due to
the coarse model resolution and the treatment to process emissions
and boundary layer in coastal areas. At the Lullington Heath site
(South-East England, Fig. 8a), the summertime variability of
observed concentrations is weak (range 0–20 ug m3), while the
models give a larger range of predictions (0–50 ug m3). There are
however several scattered sites where spread is moderate with
a high reliability, such as in Svratouch (spread 0.39, reliability 0.01;
Fig. 8b) or Kolumerwaard (spread 0.44, reliability 0.04; Fig. 8d).
Fig. 9 shows the spatial distribution of the unbiased ensemble
properties for the secondary inorganic aerosol species. It is more
difficult to distinguish clear patterns than for ozone and nitrogen
dioxide. Regional trends are not present, such as the North-West/
South-East seasaw for ozone. Coastal or complex-terrain sites do
not seem to exhibit a specific behaviour. For nitrates normalized
spread values can be very large, with values reaching or exceeding
1, showing the excessive model-to-model variability, especially in
Southern regions where models exhibit very dispersive results. The
reliability index is often negative, and the model concentration
range overestimates the actual uncertainty. Except for a few
stations, overestimation is homogeneous for nitrate. For ammonium and sulphate, the reliability index can be positive or negative,
and the normalized spread is much smaller than for nitrate.
In order to further identify the source of the large model spread,
we display, in Fig. 10 the time series of the unbiased ensemble
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
Fig. 10. Time series of the unbiased ensemble for NO2, NO3, NH4 and SO4 obtained at
the site of Vredepeel in the inland part of The Netherlands. Individual models are
displayed with thin black curves and observations with a thick red curve.(For interpretation of the references to colour in this figure legend, the reader is referred to the
web version of this article.)
obtained for the aerosol species over one site (Vredepeel, The
Netherlands) which exhibits a characteristic behaviour. At this site,
the normalized spread is relatively large for all compounds (1.00 for
NO3; 0.76 for NH4; 0.63 for SO4 and 0.67 for NO2). However, for NO2
the reliability index is close to zero (0.04) while for aerosol
compounds it is significantly negative (0.31 for NO3; 0.22 for
NH4; 0.27 for SO4). This indicates that dispersion and meteorology
are not the major source of uncertainty. The additional source is
most probably related to the aerosol formation routes and the
sensitivity to the description of the precursor concentrations for
ammonium nitrate, as ammonium nitrate is semi-volatile. The
individual evolutions of the species at this site (Fig. 10) show indeed
large model-to-model variations (especially for nitrate and
ammonium), some models giving small concentrations and some
other models large peaks.
It is however difficult to generalize these results to all stations.
The most striking conclusion for the aerosol species is that despite
the complexity and the number of processes involved in the
formation and transport of secondary inorganic aerosols the
ensemble gives, at least for sulphate and ammonium, a fairly reliable picture of the uncertainty with added value relative to the
climatology. The normalized spread, for NH4 or SO4, is in the same
range as for ozone.
5. Conclusion
Both for air quality forecasting and for the assessment of the air
quality response to emission control policies for the future it is
necessary to predict the regional distribution of pollutant
4831
concentrations and their uncertainty. Ensembles of regional models
have been used in several studies of policy assessment within the
Clean Air For Europe programme, such as the EURODELTA project
(Van Loon et al., 2007; Schaap et al., submitted for publication). A
first global evaluation of the ensemble ability to simulate uncertainty was presented for ozone in a previous study (Vautard et al.,
2006). In the present article we presented a more exhaustive
evaluation of the EURODELTA ensemble, focusing on the spatial
distribution, over Europe, of the ensemble skill in simulating
various pollutants and estimating their uncertainty.
Our results lead us to several conclusions. First, we verified that
the ensemble mean of the seven participating models has almost
always a global skill superior to that of any individual model. The
spatial distribution of the skill of the ensemble mean has been
studied for ozone, nitrogen dioxide and secondary inorganic
aerosols. For spring/summer ozone daily maxima a positive bias is
found over Atlantic coastal areas, presumably due to overestimation of boundary conditions for some models or to underestimated titration by NO. In central and southern Europe the bias
is negative. Correlation is highest in Central/Northern Europe, and
lowest in southern coastal or mountainous areas. The ensemble
mean fails to predict high NO2 daily mean concentrations, probably
resulting from the combination of local emissions and stable
meterological conditions difficult to capture with the model’s
resolutions. This produces an overall negative bias. However
correlation for NO2 remains significant in most areas, with values
culminating over the Benelux (r ¼ 0.7–0.9) area, meaning that at
least models capture the right timing of variations. For secondary
inorganic aerosols, moderate biases are found, and significant
correlation values are obtained, especially over Central/Northern
Europe. Lower skill is found over the few scattered southern Europe
sites.
The ability of the ensemble distribution to simulate uncertainty
has been evaluated using rank histograms and an analysis of the
ensemble spread. For ozone we found that the ensemble spread is
excessive over coastal Atlantic areas, probably due to large differences between boundary conditions used or to processing of
coastal areas by the different models. In these areas the ensemble
overestimates the actual uncertainty, because the observation is
more often found near the centre of the ensemble range than near
its boundaries. Over mountainous areas or near large polluted areas
the ensemble spread is too small and provides an underestimated
representation of the uncertainty. For NO2 the ensemble generally
underestimates the uncertainty also, because the predicted range
fails to extend to highest values, probably due to too coarse model
resolution. For inorganic aerosol species no specific spatial pattern
was found for spread properties, but the ensemble often overestimates the uncertainty for nitrates by producing an excessively
large range of predicted daily means.
An important overall conclusion of this study is that despite the
complexity of processes involved in the formation and transport of
secondary pollutants such as ozone and most inorganic aerosols,
the ensemble mean of the models provides a skilful prediction of
these species. For all species but NO2 and nitrates the ensemble
spread provides a generally fair representation of their prediction
uncertainty.
For nitrogen dioxide the ensemble gives poor results, most
probably because NO2 is almost a primary pollutant and therefore
is very sensitive to model resolution; pollutants that have maximal
concentrations after several hours of transport and dilution, such as
e.g. ozone and secondary inorganic aerosols, are much less sensitive to model resolution.
Our result presents optimistic perspectives for the future use of
ensembles for emission policy assessment or air quality forecasting.
However a major limitation is that we have not considered here
uncertainty in emissions themselves, since all models used the
4832
R. Vautard et al. / Atmospheric Environment 43 (2009) 4822–4832
same emission set. Whether introducing this additional source of
uncertainty would dramatically change our results remains to be
investigated. However this limitation is not of major importance for
the assessment of emission policies, where emissions are precisely
the control variables.
References
Andersson, C., Langner, J., Bergström, R., 2007. Inter-annual variation and trends in
air pollution over Europe due to climate variability during 1958–2001 simulated with a regional CTM coupled to the ERA40 reanalysis. Tellus B. 59, 77–98.
doi:10.1111/j.1600-0889.2006.00196.x.
Beekmann, M., Kerschbaumer, A., Reimer, E., Stern, R., Möller, D., 2007. PM
measurement campaign HOVERT in the greater Berlin area: model evaluation
with chemically specified observations for a one year period. Atmos. Chem.
Phys. 7, 55–68.
Bessagnet, B., Hodzic, A., Vautard, R., Beekmann, M., Cheinet, S., Honoré, C.,
Liousse, C., Rouil, L., 2004. Aerosol modeling with CHIMERE: preliminary
evaluation at the continental scale. Atmos. Environ. 38, 2803–2817.
Christensen, J., 1997. The Danish eulerian hemispheric model – a three dimensional
air pollution model used for the Arctic. Atmos. Environ. 31, 4169–4191.
Cuvelier, C., Thunis, P., Vautard, R., Amann, M., Bessagnet, B., Bedogni, M.,
Berkowicz, R., Brandt, J., Brocheton, F., Builtjes, P., Coppalle, A., Denby, B.,
Douros, G., Graf, A., Hellmuth, O., Honoré, C., Hodzic, A., Jonson, J.,
Kerschbaumer, A., de Leeuw, F., Minguzzi, E., Moussiopoulos, N., Pertot, C.,
Pirovano, G., Rouil, L., Schaap, M., Stern, R., Tarrason, L., Vignati, E., Volta, M.,
White, L., Wind, P., Zuber, A., 2007. CityDelta: a model intercomparison to
explore the impact of emission reductions in 2010 in European cities in 2010.
Atmos. Environ. 41, 189–207.
Delle Monache, L., Stull, R., 2003. An ensemble air-quality forecast over western
Europe during an ozone episode. Atmos. Environ. 37, 3469–3474.
Delle Monache, L., Hacker, J.P., Zhou, Y.M., Deng, X.X., Stull, R.B., 2006. Probabilistic
aspects of meteorological and ozone regional ensemble forecasts. J. Geophys.
Res. 111, D24307. doi:10.1029/2005JD006917.
Fagerli, H., Simpson, D., Tsyro, S., 2004. Transboundary Acidification, Eutrophication
and Ground Level Ozone in Europe. EMEP Status Report 1/2004, Unified EMEP
Model: Updates. pp.11–18. The Norwegian Meteorological Institute, Oslo, Norway.
Forkel, R., Knoche, R., 2006. Regional climate change and its impact on photooxidant
concentrations in southern Germany: simulations with a coupled regional
climate-chemistry model. J. Geophys. Res. 111. doi:10.1029/2005JD006748.
Frohn, L.M.J.H., Christensen, J., Brandt, 2002. Development of a high resolution nested
air pollution model – the numerical approach’’. J. Comput. Phys. 179, 68–94.
Galmarini, S., Bianconi, R., Addis, R., Andronopoulos, S., Astrup, P., Bartzis, J.C., Bellasio, R.,
Buckley, R., Champion, H., Chino, M., D’Amours, R., Davakis, E., Eleveld, H., Glaab, H.,
Manning, A., Mikkelsen, T., Pechinger, U., Polreich, E., Prodanova, M., Slaper, H.,
Syrakov, D., Terada, H., Van der Auwera, L., 2004. Ensemble dispersion forecasting –
part II: application and evaluation. Atmos. Environ. 38, 4619–4632.
Giorgi, F., Meleux, F., 2007. Modeling the regional effects of climate change on air
quality. C.R. Geoscience 339, 721–733.
Hedegaard, G.B., Brandt, J., Christensen, J.H., Frohn, L.M., Geels, C., Hansen, K.M.,
Stendel, M., 2008. Impacts of climate change on air pollution levels in the
Northern Hemisphere with special focus on Europe and the Arctic. Atmos.
Chem. Phys. 8, 3337–3367.
Hodzic, A., Vautard, R., Chepfer, H., Goloub, P., Menut, L., Chazette, P., Deuzé, J.-L.,
Apituley, A., Couvert, P., 2006. Evolution of aerosol optical thickness over
Europe during the August 2003 heat wave as seen from CHIMERE model
simulations and POLDER data. Atmos. Chem. Phys 6, 1853–1864.
Jollife, I.T., Stephenson, D.B., 2003. Forecast Verification: A Practitioner’s Guide in
Atmospheric Science. Wiley and Sons, West Sussex, United Kingdom, 240 p.
Krol, M., Houweling, S., Bregman, B., van den Broek, M., Segers, A., van Velthoven, P.,
Peters, W., Dentener, F., Bergamaschi, P., 2005. The two-way nested global
chemistry-transport zoom model TM5: algorithm and applications. Atmos.
Chem. Phys. 5, 417–432.
Langner, J., Bergström, R., Foltescu, V., 2005. Impact of climate change on surface
ozone and deposition of sulphur and nitrogen in Europe. Atmos. Environ. 39,
1129–1141.
Logan, J.A., 1998. An analysis of ozone sondes data for the troposphere: recommendations for testing 3-d models and development of a gridded climatology
for tropospheric ozone. J. Geophys. Res. 10 (D13), 16115–16149.
McKeen, S., Chung, S.H., Wilczac, J., Grell, G., Djalalova, I., Peckham, S., Gong, W.,
Bouchet, V., Moffet, R., Tang, Y., Carmichael, G.R., Mathur, R., Yu, S., 2007.
Evaluation of several PM2.5 forecast models using data collected during the
ICARTT/NEAQS 2004 field study. J. Geophys. Res. 112, D10S20. doi:10.1029/
2006JD007608.
Meehl, G.A., Tebaldi, C., 2004. More intense, more frequent and longer lasting heat
waves in the 21st century. Science 305, 994–997.
Meleux, F., Solmon, F., Giorgi, 2007. Increase in summer European ozone amounts
due to climate. Atmos. Environ. 41, 7577–7587.
Molteni, F., Buizza, R., Palmer, T.N., Petroliagis, T., 1996. The ECMWF ensemble prediction system: Methodology and validation, Quart. J. Roy. Met. Soc. 122, 73–119.
Ordonez, C., Mathis, H., Furger, M., Henne, S., Hüglin, C., Staehelin, J., Prévôt, A.S.H.,
2005. Changes of daily surface ozone maxima in Switzerland in all seasons from
1992 to 2002 and discussion of summer 2003. Atmos. Chem. Phys. 5, 1187–1203.
Pagowski, M., Grell, G.A., McKeen, S.A., Dévényi, D., Wilczac, J.M., Bouchet, V.,
Gong, W., McHenry, J., Peckham, S., McQueen, J., Moffet, R., Tang, Y., 2005. A
simple method to improve ensemble-based ozone forecasts. Geophys. Res. Lett.
32. doi:10.1029/2004GL022305.
Riccio, A., Giunta, G., Galmarini, S., 2008. Seeking for the rational basis for the
median model: the optimal combination from the multi-model ensemble
results. Atmos. Chem. Phys. 7, 6085–6098.
Schmidt, H., Derognat, C., Vautard, R., Beekmann, M., 2001. A comparison of
simulated and observed ozone mixing ratios for the summer of 1998 in
Western Europe. Atmos. Environ. 36, 6277–6297.
Schaap, M., Spindler, G., Schulz, M., Acker, K., Maenhaut, W., Berner, A.,
Wieprecht, W., Streit, N., Mueller, K., Brüggemann, E., J-Putaud, P., Puxbaum, H.,
Baltensperger, U., ten Brink, H.M., 2004. Artefacts in the sampling of nitrate
studied in the ‘‘INTERCOMP’’ campaigns of EUROTRAC-AEROSOL. Atmos.
Environ. 48, 6487–6496.
Schaap, M., Vautard, R., Bergström, R., van Loon, M., Bessagnet, B., Brandt, J.,
Christensen, H., Cuvelier, K., Foltescu, V., Graff, A., Jonson J.E., Kerschbaumer, A.,
Krol, M., Langner, J., Roberts, P., Rouil, L., Stern, R., Tarrason, L., Thunis, P., Vignati,
E., White, L., Wind, P., Builtjes, P.H.J. Evaluation of long-term aerosol simulations
from seven air quality models and their ensemble in the EURODELTA study.
Atmos. Environ., submitted for publication.
Schaap, M., Timmermans, R.M.A., Sauter, F.J., Roemer, M., Velders, G.J.M.,
Boersen, G.A.C., Beck, J.P., Builtjes, P.J.H., 2008. The LOTOS-EUROS model:
description, validation and latest developments. Int. J. Environ. Pollut. 32, 270–290.
Simpson, D., Fagerli, H., Jonson, J.E., Tsyro, S., Wind, P., Tuovinen, J.-P., 2003.
Transboundary Acidification and Eutrophication and Ground Level Ozone in
Europe: Unified EMEP Model Description. EMEP Status Report 1/2003 Part I,
EMEP/MSC-W Report. The Norwegian Meteorological Institute, Oslo, Norway.
Simpson, D.W., Yttri, K., Klimont, Z., Caseiro, A., Gelencser, A., Pio, C., Puxbaum, H.,
Legrand, M.R., 2007. Modelling carbonaceous aerosol over Europe: analysis of
the CARBOSOL and EMEP EC/OC campaigns. J. Geophys. Res. 112, D23S19.
doi:10.1029/2006JD008158.
Solberg, S., Hov, Ø., Søvde, A., Isaksen, I.S.A., Coddeville, P., De Backer, H., Forster, C.,
Orsolini, Y., Uhse, K., 2008. European surface ozone in the extreme summer
2003. J. Geophys. Res., 113, D7. doi:10.1029/2007JD009098.
Steinbacher, M., zellweger, C., Schwarzenbach, B., Bugmann, S., Buchmann, B.,
Ordonez, C., Prevot, S.H., Hueglin, C., 2007. Nitrogen oxides at rural sites in
Switzerland: bias of conventional techniques. J. Geophys. Res. 112, D11307.
doi:10.1029/2006JD007971.
Stern, R., Yamartino, R., Graff, A. 2003. Dispersion modelling within the European
community’s air quality directives: long term modelling of O3, PM10 and NO2.
In: 26th ITM on Air Pollution Modelling and its Application. May 26–30, 2003,
Istanbul, Turkey.
Stern, R., Builtjes, P., Schaap, M., Timmermans, R., Vautard, R., Hodzic, A.,
Memmesheimer, M., Feldmann, H., Renner, E., Wolk, R., Kerschbaumer, A., 2008.
A model inter-comparison study focusing on episodes with elevated PM10
concentrations. Atmos. Environ. 42, 4567–4588.
Stevenson, D.S., Doherty, R.M., Sanderson, M.G., Johnson, C.E., Collins, W.J.,
Derwent, R.G., 2005. Impacts of climate change and variability on tropospheric
ozone and its precursors. Faraday Discuss. 130, 41–57.
Struzewska, J., Kaminski, J.W., 2008. Formation and transport of photooxidants over
Europe during the July 2006 heat wave – observations and GEM-AQ model
simulations. Atmos. Chem. Phys. 8, 721–736.
Szopa, S., Hauglustaine, D. A., Vautard, R., Menut, L., 2006: Future global tropospheric ozone changes and impact on European air quality. Geophys. Res. Lett.,
33, L14805. doi:10.1029/2006GL025860.
Talagrand, O., Vautard, R., Strauss, B., 1998. Evaluation of probabilistic prediction
systems. In: Proc. Seminar on Predictability. ECMWF, Reading, United Kingdom,
pp. 1–26.
Taylor, K.E., 2001. Summarizing multiple aspects of model performance in a single
diagram. J. Geophys. Res. 106 (D7), 7183–7192.
Tsyro, S.G., 2005. To what extend can aerosol water explain the discrepancy
between model calculated and gravimetric PM10 and PM2.5. Atmos. Chem.
Phys 5, 515–532.
Tsyro, S., Simpson, D., Tarrasón, L., Klimont, Z., et al., 2007. Modeling of elemental
carbon over Europe. J. Geophys. Res. 112, D23S14.
Van Loon, M., Roemer, M.G.M., Builtjes, P.J.H. 2004. Model Inter-Comparison in the
Framework of the Review of the Unified EMEP Model. TNO-Report R2004/282.
Apeldoorn, The Netherlands.
Van Loon, M., Vautard, R., Schaap, M., Bergström, R., Bessagnet, B., Brandt, J.,
Builtjes, P.J.H., Christensen, J.H., Cuvelier, K., Graf, A., Jonson, J.E., Krol, M., Langner, J.,
Roberts, P., Rouil, L., Stern, R., Tarrasón, L., Thunis, P., Vignati, E., White, L., Wind, P.,
2007. Evaluation of long-term ozone simulations from seven regional air quality
models and their ensemble average. Atmos. Environ. 41, 2083–2097.
Vautard, R., Honoré, C., Beekmann, M., Rouil, L., 2005a. Simulation of ozone during
the August 2003 heat wave and emission control scenarios. Atmos. Environ. 39,
2957–2967.
Vautard, R., Bessagnet, B., Chin, M., Menut, L., 2005b. On the contribution of natural
aeolian sources to particulate matter concentrations in Europe: testing
hypotheses with a modelling approach. Atmos. Environ. 39, 3291–3303.
Vautard, R., Van Loon, M., Schaap, M., Bergström, R., Bessagnet, B., Brandt, J.,
Builtjes, P.J.H., Christensen, J.H., Cuvelier, K., Graf, A., Jonson, J.E., Krol, M.,
Langner, J., Roberts, P., Rouil, L., Stern, R., Tarrasón, L., Thunis, P., Vignati, E.,
White, L., Wind, P., 2006. Is regional air quality model diversity representative of
uncertainty for ozone simulation? Geophys. Res. Lett. 33, L24818. doi:10.1029/
2006GL027610.