MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT
CROP MODEL ENSEMBLES UNDER HEAT STRESS
Andrea Maiorano, Pierre Martre, Senthold Asseng, Frank Ewert,
and
Christoph Müller, Reimund P. Rötter, Alex C. Ruane, Mikhail A. Semenov, Daniel Wallach, Enli Wang, Phillip D Alderman,
Belay T. Kassie, Christian Biernath, Bruno Basso, Davide Camarrano, Andrew J. Challinor, Jordi Doltra, Benjamin Dumont,
Ehsan Eyshi Rezaei, Sebastian Gayler, Kurt Christian Kersebaum, Bruce A. Kimball, Ann-Kristin Koehler, Bing Liu, Garry J.
O’Leary, Jørgen E. Olesen, Michael J. Ottman, Eckart Priesack, Matthew P. Reynolds, Pierre Stratonovitch, Thilo Streck,
Peter J. Thorburn, Katharina Waha, Gary W. Wall, Jeffrey W. White, Zhigan Zhao, Yan Zhu
Introduction
Use of crop models in climate impact studies has increased during the last years
Crop models have been used to answer questions for which they were not originally designed
Furthermore most climate change impact assessments for agriculture have not addressed
crop model uncertainties
Model improvements were suggested to improve accuracy of simulations
The use of crop multi-model ensembles (MME) à increase reliability of impact estimates
and to give estimates of uncertainty
Model improvements might also reduce the number of models required for an acceptable level
of simulation uncertainty.
.02
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Introduction
15 models from the AgMIP-Wheat MME were improved for the simulation
of the impact of temperature extremes using a calibration dataset
Comparison of model performances before and after improvements were
performed using and independent dataset
Effects of model improvement on single models and MME performances
and uncertainties were evaluated
.03
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
_01
Materials & Methods
The AgMIP-Wheat model improvement exercise and MME
uncertainty study
.04
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Calibration dataset - HSC
Hot Serial Cereal Experiment, Maricopa, AZ, USA
“Cereal” because it’s on wheat
“Serial” because the wheat was planted serially every 6 wks for 2 y
“Hot” because IR heaters were deployed on some of the planting dates.
(cv Yecora Rojo sown every six week x 2 years + infrared heaters)
15 IR/year/sowing date
combinations
No stresses related to
water, fertilizer, insect
pest, diseases, weeds
Phenological and
growth data available
Grant et al., 2011; Kimball et al., 2015; Ottman et al., 2012; Wall et al., 2011
.05
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
‘Blind’ evaluation dataset - IHSGE
CIMMYT IHSGE
International Heat Stress Genotype Experiment
2 cv (Bacanora and Nasser) grown in 2 years
1990-1992 in Mexico, Egyp, India, Sudan,
Bangladesh, Brazil x 2 sowing dates
Phenology and yield for 1 treatment
provided to modellers to calibrate cultivars
14 site/year/sowing date
combinations
Reynolds et al., 1992, 1994, 1997, 1998; Reynolds,
1994
Measured variables:
- Anthesis/Maturation day
- LAI (2 measurements)
- Grain yield (GY - 2 measurements)
- Above ground biomass (ABGM - 3 measurements)
- Grain number (Gnumber)
- Single grain dry mass (GDM)
.06
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
AgMIP-Wheat Model improvement
15 of AgMIP-Wheat models (>30 crop models) participated to the study
AgMIP-Wheat models minimum requirement: model described in publication and currently in use
APSIM-E
APSIM-Wheat
APSIM-Nwheat
FASSET
GLAM
HERMES
LPJmL
Expert-N-SPASS
Expert-N-SUCROS
OLEARY
SIRIUS2010
SALUS
SIMPLACE
SIRIUSQUALITY
WHEATGROW
Objective: improving the models for the simulations of the impact of high temperatures and heat
stresses on crop development and growth
No indication was given to the teams about how improving the models.
.07
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
_02
Results
The AgMIP-Wheat model improvement exercise and MME
uncertainty study
.08
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Which model improvements?
Nb of models that included or modified key processes related to heat stress during the
model improvement exercise
Heatstressonphenology
Otherprocessesnotdirectly
relatedtoheatstress
Heatstressongrain
numberand/orsize
Canopytemperature
Heatstressongrowth
Heatstressonleafsenescence
Heatstressonleafdevelopment
.09
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Model improvements: model accuracy
Single model accuracy: Root Mean Squared Relative Error
Accuracy of single models were evaluated using the RMSRE
(Martre et al 2015)
1 N ⎛ Yi - Yˆm,i ⎞
RMSRE m = 100 ×
⎟
∑⎜
N i =1 ⎜⎝ Yi ⎟⎠
𝚫𝐥𝐨𝐠 𝟐 𝐑𝐌𝐒𝐑𝐄
Sow − Ant
Ant − Mat
LAI
2
Most of the models improved for all the variables
Worsening in HI for 3 models:
GY and ABGM not improved proportionally
HI
Gnumber
GDM
Worsening of LAI for 2-3 models (for NP and SP very slight).
Biomass
Yield
𝐑𝐌𝐒𝐑𝐄(%)𝐨𝐟𝐔𝐧𝐢𝐦𝐩𝐫𝐨𝐯𝐞𝐝
.010
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Model improvements: effects on MME
Effect of model improvement on the simulation of the impact of the mean
growing season temperatures
HI
Evaluation
Grain DM
Grain number
Calibration
Biomass
LAI
Variation range of GY for T >24°C
decreased by 39% (calibration) and
26% (evaluation)
Evaluation
Yield
Variation range was reduced above all
at high mean seasonal temperatures
Ant − Mat
Sow − Ant
Calibration
Meangrowingseasonaltemperature
.011
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Effects of model improvements on MME
accuracy and prediction skills
MSE MME
1
=
M
M
∑(
Yi − Yˆm ,i
m =1
2
)
= varM (Yˆm ,i ) + bias M (Yˆm ,i , Yi )
(
Bias
ß depends on simulations + observations
Variance ß depends on simulations
)
2
Comparison with known field experiments (hindcast) à accuracy of simulations (MSE)
Not observed conditions (prediction - forecast) à no Bias! MSE? à estimate uncertainty of simulations (MSEP)
2
MSEP = biashindcast
+ varprediction
Bias
ß hindcast (reference)
Variance ß actual simulations
(Wallach et al. framework - presented in session 1.2 Crop Model Improvement)
.012
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Model improvements: MME accuracy and
prediction skills (yield)
Mean squared error (MSE) decomposition of grain yield for the
calibration and evaluation data sets, and the prediction data set
(“unknown” data set)
Variance
Variance was almost halved
while
Bias very similar
- Maybe intrinsic property of this MME and/or
these simulations
- Maybe other uncertainty factors still not
considered explicitely
Bias 2
MSEP
MSE
Further tested with other dataset but important
when estimating uncertainty of prediction
This suggest that future predictions related to
impact of heat stress can be considered reliable
and consistent in relation to observed error
Evaluation dataset
«not known»
2
MSEP = biashindcast
+ varprediction
.013
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Model improvements: MME e-median skills
Bootstrap sampling of ensembles of size 1 to 15
Coefficient of variation of e-median GY calculated on 20,000 bootstrap samples.
In general CV of improved is
41% lower than unimproved
RMSRE of e-median was also
decreased
Benchmark
13.5 %CV
(meta-analysis of 300 agronomic
trials, Taylor et al 1999)
8 models
With improved models the mean coefficient of variation of 13.5% is
reached with 8 improved models in MME.
With unimproved this target was not achieved even with 15 models.
.014
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
_03
Summary
The AgMIP-Wheat model improvement exercise and MME
uncertainty study
.015
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Summary 01
High temperature effects were captured by incorporating/improving a range of different processes
As a consequence,
- accuracy of the models, of MME, and of e-median was increased
- variance among models in the population was reduced
Improvement with good data can improve the applicability of models across diverse management,
environments and climates, including climate extremes
The number of models for MMEs for robust impact assessments can be reduced with improved
models, halved in this case
.016
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
Summary 03
Single model improvements
Still no agreement on the cardinal temperatures
Further modeling and experimental work are needed to reach agreement among models
Improved model versions further tested through sensitivity analysis in order to better
understand the impact of adding and revising processes and additional parameters in model
structures
MME
Still to investigate weighting methods for models in the ensemble:
- are some models better than others (in specific conditions? always? For which
variables/processes)
- Can assign them a weighting coefficient?
In the climate community à increased performances (Tebaldi and Knutti, 2007).
Not considered uncertainty related to weather, soil, and management inputs.
.017
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
.018
Maiorano et al. / MODEL IMPROVEMENTS REDUCE THE UNCERTAINTY OF WHEAT CROP MME UNDER HEAT STRESS
28 / 06 / 2016
TheAgMIP-WheatTeam
https://www.agmip.org/wheat/
Senthold Asseng,FrankEwert,PierreMartre
P.K.Aggarwal,P.D.Alderman,J.Anothai,B.Basso,A.Berger,M.Bindi,C.Biernath,D.Cammarano ,A.J.Challinor, G.De
Sanctis, J.Doltra,B.Dumont,E.Fereres,R.Ferrise, M.Garcia-Vila,S.Gayler,G.Hoogenboom, L.A.Hunt,R.C.Izaurralde,
M.Jabloun, C.Jones, C.Kersebaum,B.A.Kimball ,A.-K.Koehler, D.B.Lobell,A.Maiorano,S.Minoli, C.Müller, C.
Nendel, G.J.O’Leary, J.E.Olesen, M.J.Ottman ,T.Palosuo, J.R.Porter,P.V.V.Prasad,E.Priesack, E.Eyshi Rezaei,M.P.
Reynolds, P.R.Rötter ,A.C.Ruane,M.A.Semenov, I.Shcherbak, N.K.Soora,C.Stöckle,P.Stratonovitch,T.Streck,I.Supit,
F.Tao,P.Thorburn, K.Waha,G.W.Wall,D.Wallach,E.Wang,H.Webber, J.W.White,J.Wolf, Z.Zhao,Y.Zhu