Oecologia (2011) 167:587–597
DOI 10.1007/s00442-011-2106-x
CONCEPTS, REVIEWS AND SYNTHESES
The model–data fusion pitfall: assuming certainty
in an uncertain world
Trevor F. Keenan • Mariah S. Carbone •
Markus Reichstein • Andrew D. Richardson
Received: 11 December 2010 / Accepted: 5 August 2011 / Published online: 8 September 2011
Ó Springer-Verlag 2011
Abstract Model–data fusion is a powerful framework by
which to combine models with various data streams
(including observations at different spatial or temporal
scales), and account for associated uncertainties. The
approach can be used to constrain estimates of model
states, rate constants, and driver sensitivities. The number
of applications of model–data fusion in environmental
biology and ecology has been rising steadily, offering
insights into both model and data strengths and limitations.
For reliable model–data fusion-based results, however, the
approach taken must fully account for both model and data
uncertainties in a statistically rigorous and transparent
manner. Here we review and outline the cornerstones of a
rigorous model–data fusion approach, highlighting the
importance of properly accounting for uncertainty. We
conclude by suggesting a code of best practices, which
should serve to guide future efforts.
Communicated by Russell Monson.
T. F. Keenan (&) A. D. Richardson
Department of Organismic and Evolutionary Biology,
Harvard University, 26 Oxford Street, Cambridge,
MA 02138, USA
e-mail: [email protected]
M. S. Carbone
National Center for Ecological Analysis and Synthesis,
Santa Barbara, CA 93101, USA
M. Reichstein
Biogeochemical Model–Data Integration Group,
Max-Planck-Institute for Biogeochemistry,
07745 Jena, Germany
Keywords Model–data fusion Data assimilation Parameter estimation Inverse analysis Carbon cycle
model
Introduction
‘‘Solum ut inter ista certum sit nihil esse certi’’
‘‘In these matters the only certainty is that nothing is
certain.’’
—Pliny the Elder (23 AD–79 AD)
Many fields within environmental biology and ecology
are rapidly becoming ‘‘data rich’’ (Luo et al. 2008). Novel
technologies (e.g., new instruments and automated measurement devices, wireless networking and communication
protocols, as well as data processing software and storage
hardware) are making the collection of vast amounts of data
possible. For example, continuous measurements of surface–atmosphere CO2 and H2O fluxes via eddy covariance
(Baldocchi 2008) or soil–atmosphere CO2 fluxes via soil
respiration chambers (Carbone and Vargas 2008; Vargas
et al. 2011) provide continuous, high-frequency data on
ecosystem carbon cycling at time scales from minutes to
years. Continental-scale monitoring through ‘‘ecological
observatories’’, such as NEON (http://www.neoninc.org),
ICOS (http://www.icos-infrastructure.eu), FLUXNET (http://
www.fluxnet.ornl.gov; http://www.fluxdata.org), and LTER
(http://www.lternet.edu), offer exciting new possibilities for
research, but will require many scientists to update the
statistical and analytical tools they use in order to meet new
challenges for analysis and synthesis.
One possible approach to making sense of the mountain
of data now available to ecologists is to embrace model–
data fusion (MDF) techniques. With MDF, also referred to
123
588
Fig. 1 The model–data fusion process: a conceptual diagram showing the main steps involved, and the iterative and interactive nature of
the approach. The figure is adapted from Williams et al. (2009)
as ‘‘data-model synthesis’’ or ‘‘data assimilation’’ (in a
broad sense), the process model (chosen according to the
research questions being pursued) is used to provide a
rigorous, analytical framework for data interpretation,
synthesis, and generalization or extrapolation (see Zobitz
et al. 2011 for a detailed primer). MDF results are conditional on the information content of observational data and
on what is known or assumed about uncertainties (Raupach
et al. 2005). Here, we refer specifically to three sources of
uncertainty, namely, those associated with (1) observational uncertainties (incomplete and noisy observational
data, systematic biases, etc.), (2) model structure uncertainties (e.g., in the specification of model processes and
internal relations), and (3) uncertainties in model parameters and states.
Briefly, in MDF, data of diverse types (including, in a
Bayesian context, ‘‘prior knowledge’’) can be used to
constrain model parameters or states, evaluate competing
model structures or process representations, and scale
upwards in space or time, while simultaneously accounting
for—and propagating—different sources of uncertainty.
MDF can be viewed as an iterative comparison of a model
with observations (Fig. 1). At each iteration model
parameters are changed, with the objective being to find the
parameter combinations that give equivalently best fits to
the available data. Such an approach can contribute to
model development and improvement, along with the
synthesis of various data forms (Williams et al. 2009).
MDF analyses can yield mechanistic insight into the processes underlying observed biological responses to environmental forcing and can be used to obtain best estimates
and the associated uncertainties of model parameters (e.g.,
physiological rate constants) and states––what Bayesians
refer to as the ‘‘posterior distributions’’. Recent studies,
(e.g., Fox et al. 2009; Medvigy et al. 2009; Wu et al. 2009;
Alton and Bodin 2010; Carvalhais et al. 2010; Desai 2010;
Richardson et al. 2010; Weng and Luo 2011; Zhang et al.
2010; Zhou et al. 2010), represent the MDF state-of-the-art
123
Oecologia (2011) 167:587–597
in the field of terrestrial ecosystem carbon cycle modeling
[for recent reviews see Wang et al. (2009) and Williams
et al. (2009)].
Uncertainty pervades all aspects of scientific research
(Taylor 1997), both for models and data. The strength of
MDF lies in, and is dependent on, the incorporation of both
model and data uncertainty within the analytical framework (Raupach et al. 2005; Williams et al. 2009; Cressie
et al. 2009). In fact, as suggested by Raupach et al. (2005),
results reached without a full and accurate consideration of
uncertainty might as well not have been reached at all!
Nevertheless, explicit assessments of uncertainty meet with
several conceptual and practical difficulties, ranging from
the accurate quantification of model and data error to
commonly held views that reporting uncertainty devalues
results (Pappenberger and Beven 2006).
Few MDF studies fully characterize model and data
uncertainty. In this paper we review aspects of uncertainty
and implications for MDF results. Using a forest carbon
cycle model in a synthetic experiment, we show that MDF
results are highly conditional on how the associated
uncertainties are treated. Based on this, we make the case
for an open and honest assessment of uncertainty (Brown
2010) in MDF applications and suggest a ‘‘code of best
practices’’ (Table 1). Although we focus on applications to
the terrestrial carbon cycle, the problems raised are relevant to a wide range of MDF applications in environmental
biology and ecology.
Synthetic experiment
A synthetic experiment is an experiment that uses artificially generated data. In such an experiment the ‘‘true’’
values of observations and parameters are known, giving a
controlled environment in which to test the impact of different assumptions (Code of best practices #4, Table 1).
Throughout the text, we make use of results from five
different synthetic MDF experiments using Markov chain
Monte-Carlo techniques (Hastings 1970) with a simple
forest carbon cycle model. The experiments (described
below) are designed to test the ability of the MDF framework to retrieve the parameters that generated the synthetic
data and to assess the importance of different assumptions
regarding data uncertainty.
We use the data assimilation-linked ecosystem carbon
model (DALEC, Williams et al. 2005) at the Howland
forest. The Howland Forest AmeriFlux site is located in
central Maine, USA, at the southern ecotone of the North
American boreal spruce–fir zone and consists largely of red
spruce (Picea rubens Sarg.) and eastern hemlock [Tsuga
canadensis (L.) Carr.]. The model integrates the key
components of forest productivity and carbon allocation
Oecologia (2011) 167:587–597
589
Table 1 Suggested components for a model–data fusion ‘‘code of best practices’’
‘‘Code of best practices’’ for model–data fusion analyses
1. Data uncertainties should be evaluated in an open and realistic manner. Data error should be fully characterized and fed forward into
the model–data fusion (MDF) framework. Enough detail should be provided so that the assumptions can be evaluated and the method
duplicated
2. Alternative model structures should be assessed, particularly for model components associated with unobserved variables (e.g., Zobitz
et al. 2008)
3. Multiple data constraints should be implemented whenever possible, to minimize over-fitting to any single data stream and to constrain
processes operating at different time scales (e.g., Xu et al. 2006; Barrett et al. 2005; Richardson et al. 2010)
4. The MDF framework should be tested against synthetic data with error properties similar to those of the real data in order to assess the
capacity for retrieving accurate estimates of model parameters and states
5. Attempts should be made to validate the optimized model against independent data (e.g., Moore et al. 2008; Zobitz et al. 2008;
Richardson et al. 2010)
(a) It should not be claimed that just because the optimized model can reproduce the data used as constraints that the model is good or
correct (inference only with extreme caution)
(b) It should be acknowledged that, without validation, predictions of an optimized model only represent untested hypotheses
6. Confidence intervals (posterior distributions) on model parameters, states, and predictions should be estimated in a transparent
manner. Alternative approaches to estimating these uncertainties should be considered
Table 2 Synthetic data
experiments
Experiment
Synthetic data description
Parameters
optimized
MDF assumptions
1.
Observed data gaps added and no noise
12 parameters
No error in data
2.
Observed data gaps and 10% noise
12 parameters
Fixed 10% error in data
3.
Observed data gaps and observed noise
12 parameters
Observed error in data
4.
Observed data gaps and observed noise
6 pools
Observed error in data
5.
Observed data gaps and observed noise
12 parameters
Understate data error by 25%
and respiration. Model states and parameters were optimized as described in Richardson et al. (2010; see their
Fig. 2, Table 1) using 4 years of observational data
(1997–2000). The model was optimized against five different data streams in parallel [half-daily net ecosystem
exchange of CO2 (NEE) from tower eddy-covariance
measurements, leaf area index, wood carbon, leaf litterfall,
and soil respiration] (see Richardson et al. 2010 for
details). The optimized model output for 1997 was then
used as the basis for our synthetic data.
Three synthetic data sets were created for five experiments as outlined in Table 2; the same synthetic data were
used in experiments 3 through 5. Gaps added to the model
output for each experiment were distributed according to
the observed gap frequency and inherent sampling frequency of the original data constraints. Experiment 1 used
output from the constrained model with added gaps but no
added noise to test the ability of the MDF framework to
retrieve the parameters that had generated the data. No data
error was assumed, and the cost function (the estimated
model–data mismatch, described below) was constructed
as a non-weighted least squares estimator. For the
remaining experiments, stochastic measurement error was
simulated as random noise added to each data point, drawn
from a normal distribution with a mean of zero. For
Experiment 2, the added data noise was heteroscedastic
with a standard deviation fixed at 10%. For experiments 3,
4, and 5, noise added to the synthetic data approximated the
observed error distribution (as described by Richardson
et al. 2010), including heteroscedasticity. In Experiment 5
we underestimated data uncertainty in the cost function to
illustrate the effect of assuming the data error is less than it
really is (Table 2).
For each experiment, the cost function (a measure of
data-model mismatch which guides the optimization process and parameter exploration stage) was constructed as
the mean (Eq. 2) of individual cost functions (Eq. 1) for
each data stream. Individual data stream cost functions
were constructed as the total uncertainty-weighted squared
data–model mismatch, divided by the number of observations for each data stream (NI):
!
NI X
yI ðtÞ pI ðtÞ 2
ð1Þ
jI ¼
=NI
dI ðtÞ
t¼1
where yI(t) is a data constraint at time t for data stream I,
pI(t) is the corresponding model predicted value, and dI(t)
is the measurement specific uncertainty as specified in
123
590
Oecologia (2011) 167:587–597
Richardson et al. (2010). For the aggregate multi-objective
cost function we use the mean of the individual cost
functions, which can be written as:
!
M
X
J¼
jI =M
ð2Þ
i¼1
where M is the number of data streams used. The cost
function approach applied here gives equal weight to each
data stream, regardless of the number of observations in each.
Three parameter optimizations were performed for each
experiment (and compared to test for convergence). In
total, 200,000 Monte Carlo iterations, with an ensemble of
10,000 sets of parameters accepted on the basis of a v2 test,
were used to characterize parameter distributions and 90%
confidence intervals. The v2 test was performed on each jI
individually, after normalizing each jI by the minimum
error obtained, jopt (see variance normalization: Franks
et al. 1999; Richardson et al. 2010). The retrieved parameter values and their associated confidence intervals were
then compared with the values that were used to generate
the synthetic data.
Data uncertainty
All measurements, whether from a meticulous lab experiment or a remote field campaign and whether recorded by a
human observer or a sophisticated instrument, are subject
to uncertainties (Taylor 1997). In fact, in many cases
(particularly within the environmental sciences), the
uncertainties can be on the same order of magnitude as the
data itself. As an example, tower and chamber measurements of CO2 fluxes have substantial uncertainties associated with them, and these frequently violate commonly
held assumptions of normality and constant variance
(Richardson et al. 2008; Savage et al. 2008). Random
uncertainties on individual, 30-min measurements of the
net ecosystem exchange of CO2 are commonly approximately ±50% (at 95% confidence interval) of the measured
flux, or larger (Richardson et al. 2008). Coupled with
inherent systematic biases, uncertainties on measured
fluxes are also substantial at longer time scales, such as at
the monthly or annual time step.
An additional source of data uncertainty often comes
from the difference between what the measurements represent and the real quantities of interest (Williams et al.
2009; Richardson et al. 2010). For example, flux tower
instrumentation measures the net flux of CO2, which is not
in itself a process that is modeled directly. Various
decomposition techniques are available to estimate the
associated component fluxes of gross photosynthesis (GPP)
and ecosystem respiration (RE) (Desai et al. 2008), but the
measured CO2 exchange does not directly correspond to
123
the processes that we are usually attempting to model.
While it is conceivable that partitioned fluxes could be used
as model constraints (e.g., Williams et al. 2005; Zhou et al.
2010), the partitioning itself is an additional source of
uncertainty.
Here we use our synthetic experiment to show that both
the amount of error present in data and how it is considered
in the MDF process affect the confidence intervals of
extracted parameters and states. Assigning a 10% error
(Experiment 2) increased the confidence intervals on estimated fluxes and decreased the number of constrainable
parameters (Fig. 2b) compared to considering data with no
error (Experiment 1), where data gaps contributed nominal
uncertainty (Fig. 2a). For synthetic data with the observed
scale-dependent error (Experiment 3), the confidence
intervals were much larger, and the potential for equifinality
(i.e., that different combinations of parameters and unobserved variables can give equally acceptable model behavior) (Beven 2006) in modeled processes increased through a
decrease in parameter constraints (Fig. 2c). This finding is
supported by other MDF studies (Braswell et al. 2005;
Zhang et al. 2010), although some other studies have also
suggested that magnitudes of measurement errors were not
found to considerably influence estimated fluxes (Williams
et al. 2005) or parameter identifiability (Luo et al. 2009).
The MDF framework offers a unique opportunity to
account for data uncertainties (Raupach et al. 2005). That
said, misspecification of the error model can yield welldefined but incorrect estimates of parameter distributions
(Beven et al. 2008). Data uncertainty is not considered in
many MDF studies (e.g., Carvalhais et al. 2010); in particular, data uncertainty is often not acknowledged in
studies using partitioned eddy–covariance flux data (e.g.,
Zhou et al. 2010). Many studies simply treat data error as a
data-constant nominal amount (e.g., Zhang et al. 2005,
2010; Wang et al. 2006; Xie et al. 2009; Zhou et al. 2010),
although the error structure of many data sets is well
characterized as heteroscedastic (e.g., flux data—Richardson et al. 2006, 2008). Another common approach is to
estimate data error using the MDF framework itself (e.g.,
Braswell et al. 2005; Sacks et al. 2007). It is often the case
that estimates of data error are not available. A conservative approach would be to take the largest measurement
error estimate (e.g., Desai et al. 2008; Wang et al. 2009;
Richardson et al. 2010). In such cases, one approach would
be to assume a range of error values and then test the
sensitivity of the results. Regrettably, the majority of
studies simply do not specify how data errors were treated
(Code of best practices #1, Table 1).
While a full treatment of data errors increases the
challenge of conducting MDF analyses, the failure to fully
propagate uncertainties may give a false sense of security
and over-confidence in the model output. This can hamper
Oecologia (2011) 167:587–597
591
Fig. 2 Accepted parameter ranges (left column) and subsequent
estimated annual sums (right column, gC m-2 year-1) for net
ecosystem productivity (iNEP), gross primary production (iGPP),
autotrophic respiration (iRa), and heterotrophic respiration (iRh),
when considering no noise (a Experiment 1), 10% fixed rate noise
(b Experiment 2), and observed data noise (c Experiment 3). Whiskers
90% confidence interval, based on a v2 test against measured data
after variance normalization, filled circles original parameter values
used to create the data set. Results presented represent optimization
against 1 year of synthetic data at Howland forest. Synthetic data
were thinned to represent data gap frequency and distribution at
Howland. Parameters are normalized to their initial ranges (see
Richardson et al. 2010 for detailed description). Carbon pools are not
optimized here (fixed to original values). Parameters included in the
optimization are: a Log10 decomposition rate, b fraction of GPP
respired, c fraction of net primary production (NPP) allocated to
foliage, d log10 turnover rate of foliage, e log10 turnover rate of
wood, f log10 turnover rate of fine roots, g log10 mineralization rate
of litter, h log10 mineralization rate of soil organic matter (SOM),
I temperature dependence of respiratory and decomposition processes, j photosynthetic nitrogen use efficiency, k foliage carbon pool,
l fraction of autotrophic respiration partitioned to belowground organs
progress towards improving our understanding of the
underlying processes and mechanisms or towards identifying the sources of uncertainty that could be reduced by
new measurements or experiments (Brown 2010).
between processes and drivers). Using a different model
approach can often give a very different result, particularly
when the model is used for extrapolation in time or space
(Rastetter 2003). Here we highlight model uncertainty from
two sources: (1) that arising from an incorrect or inadequate description of a process (e.g., not accounting for soil
moisture effects in arid ecosystems) and (2) that arising
from fixed internal relations (e.g., prescribed carbon pool
sizes or allocation rates). A best practice approach to MDF
can allow the testing of such uncertainty (Code of best
practices #2, Table 1).
Model uncertainty
Any given process can usually be modeled in a variety of
ways (e.g., the mathematical structure—in terms of how
processes and states are connected to, or feed back upon,
each other—and the functional form of the relationship
123
592
Fig. 3 Time series of modeled net ecosystem productivity (NEP)
from the optimized model (black line) along with 90% confidence
intervals (gray area) for a model–data fusion run for 1 year of
synthetic data (1997) with observed gap frequency and data noise
(Experiment 4). Optimization was performed on initial pool sizes
only, with all parameters fixed to those used to generate the synthetic
data. The 90% confidence intervals are calculated based on a v2 test
for acceptance/rejection. Inset Accepted range of initial pool sizes
(i.e., the possible combination of pool sizes which could generate the
plotted iNEP) for each pool (Cf carbon in foliage, Cr carbon in roots,
Cw carbon in wood, Clit carbon in litter, Csom carbon in soil organic
matter) normalized against the respective prior range. See Richardson
et al. (2010) for initial range values used. Cw here ranged between the
initial Cw value ± 50%
Although almost a trivial example of model error, the
estimated annual ecosystem respiration, extrapolated from
nocturnal eddy covariance measurements, varied by 50%
or more depending on the model used in the study by
Richardson et al. (2006). At a larger scale, the C4MIP
project showed that different model approaches led to very
different answers in terms of the relative sensitivity of
terrestrial productivity versus respiration to future climate
(Friedlingstein et al. 2006). Optimized model parameters
and states are thus conditional on model structure (Franks
et al. 1997). MDF has been successfully used to assess the
error introduced by different process representations (Sacks
et al. 2007; Zobitz et al. 2008). However, systematic model
differences that result from model structure are probably
the most difficult to identify and quantify (Butts et al. 2004;
Abramowitz et al. 2007; Zobitz et al. 2008).
Most models are subject to the setting of fixed internal
relations, such as prescribed pool sizes. We tested the
affect of fixed internal relations in synthetic Experiment 4.
Here, we varied initial pool sizes, which are often set as
fixed values in MDF studies, but kept all other parameters
constant. This allowed us to test if different structural
relations, in the form of pool sizes, could give an equivalent fit to the data. Widely varying combinations of pool
123
Oecologia (2011) 167:587–597
Fig. 4 The 90% confidence intervals for NEP) for a model–data
fusion run using 1 year of synthetic data with observed gap frequency
(Experiment 5). Gray area Confidence interval obtained when
assuming observed data noise, black area confidence interval when
underestimating data error by 25%. Model parameters are optimized,
and initial pools are prescribed. The 90% confidence intervals are
calculated based on a v2 test for acceptance/rejection
sizes gave an equally good model performance (Fig. 3). In
particular, the lack of data on some pool sizes (e.g., labile
carbon, root carbon) allowed the model to use different
initial structural combinations to arrive at a similar final
aggregate flux. On the other hand, pools for which data
constraints were available (foliar and wood carbon) were
well constrained.
Fixed internal parameters and relations are often
assigned based on ‘‘expert opinion’’ or literature values
specific to other sites or species. For example, models often
assign a constant proportion of assimilate to belowground
allocation (e.g., Zhou et al. 2010), whilst there is evidence
that belowground allocation of new assimilate can vary by
up to 500% within the summer season alone (Högberg
2010). Carvalhais et al. (2008) found that the common
assumption that carbon pools are in an equilibrium state
can lead to large differences in estimated parameters, with
results often showing high sensitivity to initial pool sizes
(Santaren et al. 2007). Such fixed internal relations are
good examples of model structural error that can be
addressed in a MDF framework. By including fixed internal relations as variables in the optimization process, their
impact on model performance can be assessed
Confidence intervals on results
While MDF can be used to identify the model parameters
that yield a statistically ‘‘optimal’’ match between data and
model (and thus, in principle, eliminate mis-parameterization as a source of model error), this does not guarantee
Oecologia (2011) 167:587–597
Fig. 5 Modeled daily ecosystem respiration (Reco) its components
[daily autotrophic (Ra) and heterotrophic (Rh) respiration] from the
optimized model (filled circles) along with 90% confidence intervals
(gray area) for a model–data fusion run using 1 year of synthetic data
with observed gap frequency and data noise (Experiment 3). Model
parameters are optimized conditional on the data, and initial pools are
prescribed. The 90% confidence intervals are calculated based on a v2
test for acceptance/rejection
593
that the model output is correct or true. In this context,
statistically valid confidence intervals for predicted model
states and fluxes are needed to strengthen inferences based
on the model analysis.
Since many model–data fusion techniques are rooted in
the Bayesian perspective, it is reasonable (and expected)
that a confidence interval is conditional on both data and
the model, so that the 90% confidence is based on prior
information, data, and information provided by the model.
That said, large variabilities (up to tenfold) in uncertainty
estimates have been reported in previous experiments, even
when algorithms and methods that were, in principle, very
similar, were applied to the same synthetic data with the
same model (Trudinger et al. 2007; Fox et al. 2009). This
large discrepancy leads us to ask the question of what do
the uncertainty estimates really represent? Further work is
clearly warranted to identify the reasons why different
MDF algorithms yield different confidence intervals.
Retrieved parameters are also dependent on an accurate
quantification of the error structure in the observed data
(e.g., Sacks et al. 2007). In our synthetic experiment 5, we
underestimated the data error by 25%. This led to an
overestimation of the apparent confidence in model
parameters (as we show here for daily net ecosystem productivity, Fig. 4). Data uncertainty feeds forward (or is
‘‘mapped’’) to parameter uncertainty, and thus to uncertainty in model predictions. This lends weight to the
importance of adequately characterizing data uncertainty in
order to accurately quantify parameter uncertainty.
A variety of methods are available to estimate confidence ranges, such as bootstrapping approaches [Efron and
Tibshirani (1993); see Richardson and Hollinger (2005)
and Reichstein et al. (2003) for applications]. The simple
v2 statistic, calculated between model and data (Press et al.
2007), can also be used to evaluate whether (for a given
parameter set) data and model are consistent at a given
level of statistical confidence, thereby offering another
basis for generating confidence intervals on model output.
The benefit of such tests is that they take into account both
uncertainties and the degrees of freedom defined by the
number of parameters and observations. Accurately quantifying the degrees of freedom is complicated by difficulties in quantifying the true information content of data. The
current assumption is that all measurements are made at a
frequency that represents the frequency of information
variability in the variable being measured. It may be
advisable to test different approaches to quantifying
parameter and state uncertainty (Code of best practices #6,
Table 1). A conservative approach would then pick the
largest confidence band, keeping in mind that a ‘‘90%
confidence interval’’ is approach- and model-dependent.
123
594
Fig. 6 Acceptable values of modeled annual autotrophic and heterotrophic respiration (iRa, iRh) from the optimized model for a model–
data fusion run using 1 year of synthetic data with observed gap
frequency and data noise (Experiment 3). Each data point represents
an accepted model run (90% confidence intervals) based on a v2 test
for acceptance/rejection. Model parameters are optimized conditional
on the data, and initial pools are prescribed. The figure inset shows the
smoothed temporal correlation of accepted values of Ra and Rh over
the course of the year. Data correspond to those shown in Fig. 3
Inferences, model constraints, and validation
An unobserved variable is a variable in the MDF framework that is not directly observed and thus must be inferred, conditional on the model structure and model
parameters. A good model fit to observations does not
imply a good estimation of the unobserved processes
responsible for the observations (Fig. 5, see discussion
above), as compensating errors have the potential to generate the right overall answer for the wrong reason
(equifinality: Beven 2006; Luo et al. 2009). Notwithstanding, direct inferences about the behavior of unobserved variables are sometimes made on the basis of
optimized model output (e.g., Zhou et al. 2010). This is in
part often unavoidable. For example, the labile carbon pool
of forest ecosystems is an important aspect of carbon
cycling, but data on the size of this pool, at the ecosystem
scale, are scarce due to practical measurement difficulties.
Autotrophic and heterotrophic respirations are good
examples of components often treated in MDF frameworks
as unobserved processes (processes that are not quantified
by direct observations). This is due to the relative lack of
data constraints on belowground processes, which cannot
be constrained by the use of NEE data alone (Sacks et al.
2007; Zobitz et al. 2008). We use synthetic experiment 3 to
highlight the potential for uncertainty in both the annual
contribution to total respiratory fluxes and the temporal
change of that contribution. In our experiment, although
123
Oecologia (2011) 167:587–597
estimates of GPP were well constrained (Fig. 2), both
autotrophic and heterotrophic ecosystem respiration were
subject to large compensating uncertainties (Figs. 5, 6).
This shows that even soil respiration data are not capable
of constraining belowground partitioning between heterotrophic and autotrophic respiration. Furthermore, the
uncertainty associated with the partitioning of ecosystem
respiration into its auto- and heterotrophic components
varied non-linearly throughout the year (Figs. 5, 6). This
reflects the heteroscedastic nature of data errors and nonlinear scaling of respiration with temperature (Fig. 5). The
noisier the data, the lower the model confidence in the
contribution of autotrophic respiration to total soil respiration (Fig. 7). Unobserved variables incorporate all the
uncertainties previously discussed, thus any inference
should therefore be made with extreme caution.
While there are many examples of unobserved variables
that would, in practice, be impossible to measure, others
are simply most often treated as unobserved variables
because they are difficult or expensive to measure. As an
example, total soil respiration is commonly measured (e.g.,
Savage et al. 2008; Vargas et al. 2011), but in MDF
applications, the separate contributions of autotrophic and
heterotrophic respiration to total respiration are more often
treated as unobserved variables. However, methods such as
root exclusion and isotope partitioning (Hanson et al. 2000;
Kuzyakov 2006) can be used to distinguish the contribution
of autotrophic and heterotrophic components. Thus, these
are not necessarily unobserved variables. Similarly, many
Fig. 7 The inferred percentage contribution of autotrophic respiration to total soil respiration, when considering: no noise (a), 10%
fixed rate noise (b), and observed noise (c) (Experiments 1, 2, 3,
respectively). Whiskers 90% confidence interval, based on a v2 test
against measured data. Results represent 1 year of synthetic data at
Howland forest, optimizing model parameters, with initial carbon
pools fixed to original values
Oecologia (2011) 167:587–597
other unobserved variables can be constrained by using
more data streams. For example, as the information content
of tower eddy–covariance data can only constrain a limited
number of parameters (e.g., Knorr and Kattge 2005; Alton
and Bodin 2010), additional data streams are needed to
constrain more parameters even in very simple carbon flux
models (Richardson et al. 2010). Using different data
streams effectively constrains different but non-independent parts of the model, thus reducing equifinality. Additional data, which are not currently widely available, such
as information on the allocation of photosynthetic assimilates or transfer rates among different carbon pools, would
greatly aid in future MDF studies.
Of course, direct validation of results against independent observational data (e.g., Moore et al. 2008; Zobitz
et al. 2008; Richardson et al. 2010) is the acid test to
confirm confidence in model predictions (Code of best
practices #5, Table 1). In the worst-case scenario (no
observed data), the sensitivity of model results to
assumptions regarding parameter estimates and model
structure can be assessed. An operational best practice
would be to use multiple measured data streams (Code of
best practices #3, Table 1) both to constrain the optimization process and provide validation (using independent
training and testing data pools). With respect to validation,
previous studies have demonstrated the benefit of testing
the MDF framework itself by using synthetic data
(Braswell et al. 2005; Zhang et al. 2005; Trudinger et al.
2007; Lasslop et al. 2008; Fox et al. 2009). Even in the
presence of good constraint availability and the potential
for model validation, either through cross-holdout schemes
or outside the optimization domain, conducting a synthetic
experiment as performed in this study is essential to validate the MDF framework itself.
Conclusions
Despite the recent increase in the application of MDF to
questions in environmental sciences and ecology, there
remains no common agreement on best practices with
respect to the characterization and propagation of uncertainties and the validation and testing of model output.
Adequately addressing the uncertainty associated with
MDF results remains a critical challenge. The lack of a
proper consideration of model and/or data uncertainty,
however, invariably gives overconfident and potentially
misleading results. Uncertainty has historically been
regarded as the curse of environmental research (Pappenberger and Beven 2006). However, we would argue the
contrary that acknowledgment of the true inherent uncertainty only serves to improve the quality of both current
and future scientific endeavors. MDF offers a unique
595
framework by which to do this, and we would urge future
studies to apply an open treatment of uncertainty (Brown
2010).
Here, we have discussed the ways in which uncertainties
can be included in MDF analyses, based on the belief that
obtaining accurate uncertainty estimates is every bit as
important as obtaining the ‘‘right’’ result. In Table 1 we put
forward a ‘‘best practices’’ template for model–data fusion.
We encourage further discussion and debate of this
proposal.
Acknowledgments TFK and ADR acknowledge support from the
Northeastern States Research Cooperative, and from the Office of
Science (BER), U.S. Department of Energy, through the Terrestrial
Carbon Program under Interagency Agreement number DE-AI0207ER64355, and through the Northeastern Regional Center of the
National Institute for Climatic Change Research. MR acknowledges
support from the CARBO-Extreme project of the European Commission (FP7-ENV-2008-1-226701). MSC acknowledges support
from the Kearney Foundation of Soil Science.
References
Abramowitz G, Pitman A, Gupta H, Kowalczyk E, Wang P (2007)
Systematic bias in land surface models. J Hydrometeorol
8:989–1001
Alton P, Bodin P (2010) A comparative study of a multilayer and a
productivity (light-use) efficiency land-surface model over
different temporal scales. Agric For Meterorol 150:182–195
Baldocchi D (2008) Breathing of the terrestrial biosphere: lessons
learned from a global network of carbon dioxide flux measurement systems. Aust J Bot 56:1–26
Barrett DJ, Hill MJ, Hutley LB, Beringer J, Xu JH, Cook GD, Carter
JO, Williams RJ (2005) Prospects for improving savanna
biophysical models by using multiple-constraints model-data
assimilation methods. Aust J Bot 53:689–714
Beven K (2006) A manifesto for the equifinality thesis. J Hydrol
320:18–36
Beven KJ, Smith PJ, Freer JE (2008) So just why would a modeller
choose to be incoherent? J Hydrol 354:15–32
Braswell BH, Sacks WJ, Linder E, Schimel DS (2005) Estimating
diurnal to annual ecosystem parameters by synthesis of a carbon
flux model with eddy covariance net ecosystem exchange
observations. Glob Change Biol 11:335–355
Brown JD (2010) Prospects for the open treatment of uncertainty in
environmental research. Prog Phys Geogr 34:75–100
Butts MB, Payne JT, Kristensen M, Madsen H (2004) An evaluation
of the impact of model structure on hydrological modelling
uncertainty for streamflow simulation. J Hydrol 298:242–266
Carbone MS, Vargas R (2008) Automated soil respiration measurements: new information, opportunities and challenges. New
Phytol 177:295–297
Carvalhais N, Reichstein M,Seixas J,Collatz GJ,Santos Pereira J et al
(2008) Implications of the carbon cycle steady state assumptionfor biogeochemical modeling performanceand inverse
parameter retrieval. Glob Biochem Cycles 22:GB2007. doi:
10.1029/2007GB003033
Carvalhais N, Reichstein M, Ciasis P, Collatz GJ, Mohecha MD et al
(2010) Identification of vegetation and soil carbon pools out of
equilibrium in a process model via eddy covariance and
biometric constraints. Glob Change Biol 16:2813–2829
123
596
Cressie N, Calder CA, Clark JS, Hoef JMV, Wikle CK (2009)
Accounting for uncertainty in ecological analysis: the strengths
and limitations of hierarchical statistical modeling. Ecol Appl
19:553–570
Desai AR (2010) Climatic and phenological controls on coherent
regional interannual variability of carbon dioxide flux in a
heterogeneous landscape. J Geophys Res 115:G00J02. doi:
10.1029/2010JG001423
Desai AR, Richardson AD, Moffat AM, Kattge J, Hollinger DY et al
(2008) Cross-site evaluation of eddy covariance GPP and RE
decomposition techniques. Agric For Meteorol 148:821–838
Efron B, Tibshirani RJ (1993) An introduction to the bootstrap.
Chapman and Hall, New York
Fox A, Williams M, Richardson AD, Cameron D, Gove JH et al
(2009) The REFLEX project: comparing different algorithms
and implementations for the inversion of a terrestrial ecosystem
model against eddy covariance data. Agric For Meteorol
149:1597–1615
Franks SW, Beven KJ, Quinn PF, Wright IR (1997) On the sensitivity
of soil-vegetation-atmosphere transfer (SVAT) schemes: Equifinality and the problem of robust calibration. Agric For Meteorol
86:63–75
Franks SW, Beven KJ, Gash JHC (1999) Multi-objective conditioning
of a simple SVAT model. Hydrol Earth Sys Sci 3:488–489
Friedlingstein P et al (2006) Climate-carbon cycle feedback analysis:
Results from the C4MIP model intercomparison. J Climatol
19:3337–3353
Hanson PJ, Edwards NT, Garten CT, Andrews JA (2000) Separating
root and soil microbial contributions to soil respiration: a review
of methods and observations. Biogeochemistry 48:115–146
Hastings WK (1970) Monte-Carlo sampling methods using markov
chains and their applications. Biometrika 57:97
Högberg P (2010) Is tree root respiration more sensitive than
heterotrophic respiration to changes in soil temperature? New
Phytol 188:9–10
Knorr W, Kattge J (2005) Inversion of terrestrial ecosystem model
parameter values against eddy covariance measurements by
Monte Carlo sampling. Glob Change Biol 11:1333–1351
Kuzyakov Y (2006) Sources of CO2 efflux from soil and review of
partitioning methods. Soil Biol Biochem 38:425–448
Lasslop G, Reichstein M, Kattge J, Papale D (2008) Influences of
observation errors in eddy flux data on inverse model parameter
estimation. Biogeosciences 5:1311–1324
Luo Y, Clark JS, Hobbs T, Lakshmivarahan S, Latimer AM, Ogle K,
Schimel D, Zhou X (2008) Symposium 23. Toward ecological
forecasting. Bull Ecol Soc Am 89:467–474
Luo Y, Weng E, Wu X, Gao C, Zhou X, Zhang L (2009) Parameter
identifiability, constraint, and equifinality in data assimilation
with ecosystem models. Ecol Appl 19:571–574
Medvigy D, Wofsy SC, Munger JW, Hollinger DY, Moorcroft PR
(2009) Mechanistic scaling of ecosystem function and dynamics
in space and time: ecosystem demography model version 2.
J Geophys Res 114:G01002. doi:10.1029/2008JG000812
Moore D, Hu J, Sacks WJ, Schimel DS, Monson RK (2008)
Estimating transpiration and the sensitivity of carbon uptake to
water availability in a subalpine forest using a simple ecosystem
process model informed by measured net CO2 and H2O fluxes.
Agric For Meteorol 148:1467–1477
Pappenberger F, Beven KJ (2006) Ignorance is bliss: or seven reasons
not to use uncertainty analysis. Water Resour Res 42: W05302.
doi:10.1029/2005WR004820
Press WH, Teukolsky SA, Vetterling WT, Flannery BP (2007)
Numerical recipes: The art of scientific computing. Cambridge
University Press, Cambridge
Rastetter EB (2003) The collision of hypotheses: what can be learned
from comparisons of ecosystem models? In: Canham CD, Cole
123
Oecologia (2011) 167:587–597
JJ, Lauenroth WK (eds) Models in ecosystem science. Princeton
University Press, Princeton, pp 221–224
Raupach MR et al (2005) model-data synthesis in terrestrial carbon
observation: methods, data requirements and data uncertainty
specifications. Glob Change Biol 11:378–397
Reichstein M et al (2003) Inverse modeling of seasonal drought
effects on canopy CO2/H2O exchange in three Mediterranean
ecosystems. J Geophys Res 108(D23):4726. doi:10.1029/2003
JD003430
Richardson AD, Hollinger DY (2005) Statistical modeling of
ecosystem respiration using eddy covariance data: maximum
likelihood parameter estimation, and Monte Carlo simulation of
model and parameter uncertainty, applied to three simple
models. Agric For Meteorol 131:191–208
Richardson AD, Braswell BH, Hollinger DY, Burman P, Davidson
EA et al (2006) Comparing simple respiration models for eddy
flux and dynamic chamber data. Agric For Meteorol 141:219–
234
Richardson AD, Mahecha MD, Falge E, Kattge J, Moffat AM et al
(2008) Statistical properties of random CO2 flux measurement
uncertainty inferred from model residuals. Agric For Meteorol
148:38–50
Richardson A, Williams M, Hollinger DY, Moore DJP, Dail DB et al
(2010) Estimating parameters of a forest ecosystem C model
with measurements of stocks and fluxes as joint constraints.
Oecologia 164:25–40
Sacks WJ, Schimel DS, Monson RK (2007) Coupling between carbon
cycling and climate in a high-elevation, subalpine forest: a
model-data fusion analysis. Oecologia 151:54–68
Santaren D, Peylin P, Viovy N, Ciais P (2007) Optimizing a processbased ecosystem model with eddy-covariance flux measurements: a pine forest in southern France. Glob Biogeochem Cyc
21:GB2013. doi:10.1029/2006GB002834
Savage K, Davidson EA, Richardson AD (2008) A conceptual and
practical approach to data quality and analysis procedures for
high-frequency soil respiration measurements. Funct Ecol
22:1000–1007
Taylor JR (1997) An introduction to error analysis: the study of
uncertainties in physical measurements. University Science
Books, Sausalito
Trudinger CM, Raupach MR,Rayner PJ,Kattge J,Liu Q et al (2007)
OptIC project: an intercomparison of optimization techniques for
parameter estimation in terrestrial biogeochemical models.
J Geophys Res 112:G02027. doi:10.1029/2006JG000367
Vargas R, Carbone M, Reichstein M, Baldocchi D (2011) Frontiers
and challenges in soil respiration research: from measurements
to model-data integration. Biogeochemistry 102:1–13
Wang YP, Baldocchi D, Leuning R, Falge E, Vesala T (2006)
Estimating parameters in a land-surface model by applying
nonlinear inversion to eddy covariance flux measurements from
eight FLUXNET sites. Glob Change Biol 12:1–19
Wang YP, Trudinger CM, Enting IG (2009) A review of applications
of model–data fusion to studies of terrestrial carbon fluxes at
different scales. Agric For Meteorol 149:1829–1842
Weng E, Luo Y (2011) Relative information contributions of model
vs. data to constraints of short- and long-term forecasts of forest
carbon dynamics. Ecol Appl 21:1490–1505
Williams M, Schwarz PA, Law BE, Irvine J, Kurpius MR (2005) An
improved analysis of forest carbon dynamics using data assimilation. Glob Change Biol 11:89–105
Williams M, Richardson AD, Reichstein M, Stoy PC, Peylin P et al
(2009) Improving land surface models with FLUXNET data.
Biogeosciences 6:1341–1359
Wu XW, Luo YQ, Weng ES, White L, Ma Y, Zhou XH (2009)
Conditional inversion to estimate parameters from eddy-flux
observations. J Plant Ecol 2:55–68
Oecologia (2011) 167:587–597
Xie H, Eheart JW, Chen Y, Bailey BA (2009) An approach for
improving the sampling efficiency in the Bayesian calibration of
computationally expensive simulation models. Water Resour
Res 45:W06419. doi:10.1029/2007WR006773
Xu T, White L, Hul D, Luo Y (2006) Probabilistic inversion of a
terrestrial ecosystem model: analysis of uncertainty in parameter
estimation and model prediction. Glob Biogeochem Cycles
20:GB2007. doi:10.1029/2005GB002468
Zhang Q, Xiao X, Braswell B, Linder E, Baret F, Moore B (2005)
Estimating light absorption by chlorophyll, leaf and canopt in a
deciduous broadleaf forest using MODIS data and a radiative
transfer model. Remote Sens Environ 99:359–371
Zhang L, Luo Y, Yu G, Zhang L (2010) Estimated carbon residence
times in three forest ecosystems of eastern China: applications of
597
probabilistic inversion. J Geophys Res 115:G01010. doi:
10.1029/2009JG001004
Zhou X et al (2010) Concurrent and lagged impacts of an
anomalously warm year on autotrophic and heterotrophic
components of soil respiration: a deconvolution analysis. New
Phytol 187:184–198
Zobitz JM, Moore D, Sacks WJ, Monson RK, Bowling DR, Schimel
DS (2008) Integration of process-based soil respiration models
with whole ecosystem CO2 measurements. Ecosystems 11:250–
269
Zobitz JM, Desai, AR, Moore DJP, Chadwick MA (2011) A primer
for data assimilation with ecological models using markov chain
Monte Carlo (MCMC). Oecologia. doi:10.1007/s00442-0112107-9
123