INTERNATIONAL JOURNAL OF CLIMATOLOGY
Int. J. Climatol. (2014)
Published online in Wiley Online Library
(wileyonlinelibrary.com) DOI: 10.1002/joc.4216
Skill of CMIP5 climate models in reproducing 20th century
basic climate features in Central America
Hugo G. Hidalgoa,b* and Eric J. Alfaroa,b,c
a
School of Physics, University of Costa Rica, San Pedro, Costa Rica
Center for Geophysical Research, University of Costa Rica, San Pedro, Costa Rica
c Center for Research in Marine Sciences and Limnology, University of Costa Rica, San Pedro, Costa Rica
b
ABSTRACT: A total of 107 climate runs from 48 Coupled Model Inter-comparison Project 5 (CMIP5) general circulation
models (GCMs) were evaluated for their ability to skillfully reproduce basic characteristics of late 20th century climate over
Central America. The models were ranked according to metrics that take into consideration the mean and standard deviation
of precipitation (pr) and surface temperature (tas), as well as the El Niño-Southern Oscillation (ENSO)-pr teleconnection.
Verification was performed by comparing model runs to observations and a reanalysis dataset. Based on the rankings, the best
13 models were further evaluated. Not surprisingly, the models showed better skill at reproducing mean tas patterns throughout
the year. The skill is generally low for mean pr patterns, except for some models during March, April, and May. With a
few exceptions, the skill was low for reproducing the observed monthly standard deviation patterns for both pr and tas. The
ENSO-pr teleconnection was better simulated in the best 13 model runs compared to the sea-surface temperature global pattern
characteristic of ENSO which showed low skill. The Inter-tropical Convergence Zone (ITCZ) appeared better modeled in July
than in January. In January, there were instances of a double ITCZ pattern. Some models skillfully reproduced the seasonal
distribution of the Caribbean Low-Level Jet index (CLLJ). More detailed research evaluating the specific performance of the
models on a variety of time-scales and using parameters relevant to these and other climatic features of Central America is
needed. This study facilitates a pre-selection of models that may be useful for this task.
KEY WORDS
climate model; ITCZ; Caribbean Low-Level Jet; GCM; Central America; skill
Received 12 March 2014; Revised 2 October 2014; Accepted 24 October 2014
1. Introduction
General circulation models (GCMs) are the main tool
used by the Intergovernmental Panel on Climate Change
(IPCC) for projecting climate impacts at the end of the
century (IPCC, 2013). GCMs are based on mathematical
representations of the physical laws governing the earth’s
climate. Due to limitations in the modeling of small-scale
processes and other deficiencies, GCMs cannot accurately
reproduce certain features of regional and global climate.
It is therefore often necessary to rank them according to
their ability to reproduce observed climatic characteristics
of a region, especially when evaluating large sets (around
100) of model runs, which can be prohibitively expensive
if they are used for dynamical downscaling (Hidalgo and
Alfaro, 2012b).
The task of selecting models is not trivial as the types
and numbers of metrics used for ranking greatly influences the results (Brekke et al., 2008; Hidalgo and Alfaro,
2012b). For example, a model could be very good at reproducing the general pattern of the El Niño-Southern Oscillation (ENSO), but deficient at reproducing the monthly
* Correspondence to: H. G. Hidalgo, School of Physics, University of
Costa Rica, San Pedro, Costa Rica. E-mail: [email protected]
© 2014 Royal Meteorological Society
means or standard deviations of precipitation (pr) or surface temperature (tas) over the area of interest, and vice
versa. Although some biases, e.g. in the monthly means
and standard deviations of pr and tas patterns can be
bias-corrected using statistical methods (Hidalgo et al.,
2013), other factors that affect the model’s skill in representing important climatic phenomena (e.g. ENSO) cannot
be corrected. Model selection, therefore, always implicitly
requires compromise.
Pre-selection of the most skillful GCM runs based on
basic climatic characteristics is useful for determining
an initial set of ‘best’ model runs for further analysis
(see for Sheffield et al., 2013; Neelin et al., 2013). In
this study, 107 available GCM runs from 48 different
models are ranked using basic climatic characteristics, as
a precursor for future studies evaluating the use of the
same models over Central America. That is, in this study,
emphasis is placed on evaluating the ability of the GCM
runs to reproduce basic but desirable historical climatic
characteristics of Central America, namely (1) the monthly
mean patterns of pr and tas, (2) the standard deviations of
pr and tas, and (3) the ENSO-pr teleconnection patterns.
These three metrics are used because they are amongst the
most important climatic sources of variability in Central
America.
H. G. HIDALGO AND E. J. ALFARO
It is recognized that (1) metrics representing other
climatic features important to the region, e.g. the
Inter-Tropical Convergence Zone (ITCZ), the Caribbean
Low-Level Jet (CLLJ) (Amador, 1998; 2008), and the
Mid-summer Drought (MSD) (Magaña et al., 1999)
could have been included as a part of the evaluation of
model skill, and (2) depending on the type and number
of metrics used different rankings could result (Brekke
et al., 2008). The objective of this study, however, is not
to present a full evaluation of the models’ abilities to
reproduce most (or all) climatic features relevant to the
region at various time-scales. Rather, this study seeks to
provide an initial ranking of a large quantity of model
runs based on a limited and manageable number of basic
climatic characteristics, which can then support (by aiding
pre-selection) future and more detailed modeling studies
in the region. As an example of what is possible, however,
the general shape of the ITCZ, the climatology of the
CLLJ and the ENSO-PDO global sea-surface temperature
(SSTs) patterns are also evaluated qualitatively for the
most skillful 13 models determined from the ranking [note
that the Pacific Decadal Oscillation (PDO) is considered
here as part of the ENSO pattern, given that both are
closely related (Polade et al., 2013)].
In this study, the analysis is performed on runs from
48 Coupled Model Inter-Comparison Project, Phase 5
(CMIP5; Taylor et al., 2012b) fully coupled GCMs. Previous studies have determined the skill of CMIP3 (Meehl
et al., 2007) GCMs in reproducing relevant climatic features of the region. For example, Pierce et al. (2008, 2009)
show that the CMIP3 models reproduce with limitations,
ENSO and the PDO (Mantua et al., 1997). Other evaluations can be found in the studies by Rauscher et al.
(2011), Hirota et al. (2011) Delworth et al. (2012) and
Liu et al. (2012) for the ITCZ; Rauscher et al. (2011)
for the MSD; Martin and Schumacher (2011) for the
CLLJ; and in the study by Jiang et al.’s (2012) examination of intra-seasonal variability. Hidalgo and Alfaro
(2012b) also showed that over the Eastern Tropical Pacific,
most of 30 CMIP3 GCM runs evaluated had significant
biases in the monthly mean and standard deviation of
monthly pr and tas patterns with respect to the National
Center for Environmental Prediction/National Center for
Atmospheric Research (NCEP/NCAR) Reanalysis (hereafter the Reanalysis; Kalnay et al., 1996). This suggests
that statistical bias-correction of raw CMIP3 GCM pr and
tas data is a necessary step when using these data in
climate change studies involving statistical downscaling
techniques (Hidalgo and Alfaro, 2012b).
Knutti and Sedláček (2013) evaluated the robustness
and uncertainties in the projections of CMIP5 models.
They found similarities between CMIP3 and CMIP5 models in the projected pr change and in the ‘robustness’ of
the projections, suggesting little improvement between the
two generations of models. There is in general a lack
of improved model convergence in CMIP5 compared to
CMIP3. Knutti and Sedláček (2013), however, also found
that potential improvements in pr projections are larger in
the Tropics, including Central America. Notwithstanding
© 2014 Royal Meteorological Society
deficiencies, Knutti and Sedláček (2013) suggest that the
incorporation of improved understandings of certain processes (i.e. incorporating bold assumptions or previously
ignored factors) inspires confidence that the CMIP5 models capture more of the relevant processes.
Other relevant CMIP5 evaluations include:
• Kim et al.’s (2012) assessment of the CMIP5 decadal
hindcast/forecast simulations of seven state-of-the art
ocean–atmosphere coupled models. All the models
showed high prediction skill for tas over the Indian,
North Atlantic and western Pacific Oceans where the
externally forced component and low-frequency climate
variability is dominant (Kim et al., 2012). The Atlantic
Multi-Decadal Oscillation (AMO; Enfield et al., 2001)
is also predicted in the models with significant skill,
while the PDO shows relatively low predictive skill
(Kim et al., 2012).
• Sheffield et al.’s (2013) evaluation of 17 CMIP5 models
in terms of their skill in reproducing selected regional
climate processes relevant to North American climate,
including cool season western Atlantic cyclones, the
North American monsoon, the United States Great
Plains low-level jet and Arctic Sea Ice. No particular
model stood out as performing better for all analyses,
although some models performed much better for sets
of metrics (Sheffield et al., 2013).
• Neelin et al.’s (2013) report on the skill of CMIP5 models in representing California precipitation. The paper
appears in volume 26 of the Journal of Climate which
was devoted to studies of CMIP5 models’ representation
of North American climate. Although some of the studies (e.g. Sheffield et al., 2013) included Central America, the predominant focus is on climatic processes of
the mid-latitudes.
• Polade et al.’s (2013) study of the teleconnections
between SSTs in the Pacific (north of 30∘ S) and North
American pr using Singular-Value Decomposition
(SVD) analysis. In general, an improvement in the skill
of the teleconnection was found in the CMIP5 models,
compared to the CMIP3. Increased spatial resolution
in the new generation of models and better physics
contributed to the improvement (Polade et al., 2013).
• Langenbrunner and Neelin’s (2013) study of the skill of
CMIP5 AMIP (Atmospheric Model Inter-Comparison
Project; Gates et al., 1999) models in reproducing
the patterns of global ENSO teleconnections for
1979–2005. In AMIP5 models, SSTs are prescribed
and only the atmospheric component of climate is
simulated. In regions of strong signal (equatorial South
America, the western equatorial Pacific, and a southern
section of South America) there was little improvement
in reproducing the amplitude and spatial correlation
metrics of the pr teleconnection for CMIP5 versus
CMIP3 models. However, other aspects, such as the
amplitude of the pr response (root mean-square deviation over each region) were reasonably captured by the
mean of the amplitudes of the individual models, in contrast with the multi-model ensemble mean. Although
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
the simulation of ENSO teleconnections is fairly challenging, the skill of individual models in reproducing
the teleconnection signal amplitude (assessed using
the mean of individual model amplitudes as opposed
to the multi-model ensemble or MME) suggests that
similar measures are trustworthy in the case of climate
change studies. Sign agreement plots also proved to
be a useful tool for such studies (Langenbrunner and
Neelin, 2013).
• Flato et al.’s (2013) summary of CMIP5 model evaluations.
None of the above studies specifically focused on Central
America or the Intra-Americas Sea. Some CMIP5 projection results for Central America and the Caribbean can be
found in the study by Bindoff et al. (2013).
Central America is a region of complex topography
influenced by many climatic mechanisms, such as ENSO,
the migration of the ITCZ (Sachs et al., 2009), AMO,
PDO, the CLLJ, the MSD and others. The climate variability at seasonal and inter-annual scales, in particular those
aspects associated with ENSO, have significant socioeconomic impacts on the countries of Central America
(Waylen et al., 1996; IPCC, 1997; Hidalgo and Alfaro,
2012a, 2012b; Hidalgo et al., 2013). Central America has
also been identified as a climate change ‘hot-spot’ (Giorgi,
2006). In this study, the focus is on the skill of the CMIP5
GCM model runs in reproducing basic climatic characteristics of Central America.
2. Characteristics of Central American climate
High mountains divide Central America into two main climate regions – the Pacific and Caribbean slopes, which
are lee and windward, respectively, of the North Atlantic
trade winds which are the dominant wind regime (Maldonado et al., 2013). The rain over the Pacific slope has a
bimodal annual cycle, with a maximum in May–June and
another in August–September–October, separated by the
MSD in July (Magaña et al., 1999). A first maximum of
deep convective activity (and hence precipitation) occurs
when surrounding SSTs exceed 29 ∘ C (around May). During July and August, the SSTs cool by ∼1 ∘ C due to diminished downwelling solar radiation and stronger easterly
winds associated with the CLLJ, leading to a decrease
in deep convective activity (Amador et al., 2006). The
decreased convection facilitates increased downwelling
solar radiation and a slight increase in SSTs (∼28.5 ∘ C),
so that by the end of August through early September
convection is again enhanced and a second maximum of
precipitation occurs. Alfaro (2002) notes that a different climate regime dominates at stations located on the
Caribbean Coast of Honduras, Costa Rica, and Panama.
There are similarly two rainfall maxima but they occur in
July and November–December, with the latter maximum
being wetter than the first. There are also two minima in
March and September–October, with the first being drier
than the second.
© 2014 Royal Meteorological Society
The annual cycle of air surface temperature in Central
America is tropical, predominantly maritime, with small
annual changes, and dependent on cloud cover and altitude (Taylor and Alfaro, 2005). The dominant annual cycle
(excepting the Atlantic coasts of Honduras and northern Nicaragua) is monsoonal, with highest temperatures
occurring just before the summer rains. Temperatures are
at their lowest during January, largely due to the cooling
effect of the strong trade winds. Maximum temperatures
occur during April and are associated with a decrease in
trade wind strength, less cloud cover and higher values
of solar radiation (Amador et al., 2006). There is a temperature minimum in July that coincides with the onset of
the MSD (or in Spanish ‘veranillo’ or ‘canícula’). During
this period, the trade winds briefly increase in intensity, the
subtropical ridge over the Caribbean intensifies, and a second minimum and maximum in cloud cover and radiation
respectively occur (Alfaro, 2000).
The CLLJ, also known as the Intra-Americas Low-Level
Jet, is another dominant climatic feature of the region. Its
annual cycle is characterized by two wind maxima near
925 hPa in July and January–February (Amador, 2008).
During summer, winds in excess of 13 m s−1 dominate the
central Caribbean Sea (15∘ N, 75∘ W) and extend upward
to 700 hPa. In comparison, the winter component of the jet
is compressed below 850 hPa, with values of up 10 m s−1 ,
and with strong vertical wind shear which is unfavorable
for convection. The summer peak generally starts to
develop in early June just after the onset of the rainy
season, reaches a maximum in July, before weakening in
early September. During September to early November,
the trades are relatively weak, vertical wind shear over the
Caribbean is reduced, hurricane activity peaks, and rainfall spreads across most of the Intra-Americas Sea. In late
November and early December, the trades again increase
in strength, cold surges from mid and high latitudes reach
the Tropics, and the second maximum of wind appears
over the Caribbean Sea.
3.
Data
Simulations from 107 runs from 48 different CMIP5
GCMs for the historical period 1979–1999 and for
ensembles from initializations r1i1p1, r2i1p1 and r3i1p1
were obtained from the Centro Agronómico Tropical de
Investigación y Enseñanza (CATIE) located in the city of
Turrialba, Costa Rica, and from the Earth System Grid
Federation (http://pcmdi9.llnl.gov/esgf-web-fe/), a data
server from the Program for Climate Model Diagnostics
and Inter-comparison (PCMDI) at the Lawrence Livermore National Laboratory of the United States. The
data consist of monthly pr, tas and zonal and meridional
925 hPa wind speeds (u, v). Not all the models have wind
data and therefore much of the analysis performed was
based on the pr and tas data only. The wind data were
used in an analysis of the climatology of the CLLJ index,
which is based in u only and at the level of the jet core
(925 hPa). Therefore, the v data and the u data at other
levels were not used.
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
In Table 1, the characteristics of the runs are shown.
There is at least one run available for each model corresponding to run r1i1p1. In some cases, additional runs
corresponding to r2i1p1 and r3i1p1 were also available.
Detailed explanation of the difference between the runs
can be found in the study by Taylor et al. (2012a). In
summary, the runs examined correspond to equally likely
outcomes for a particular simulation (i.e., they typically
differ only by being started from equally realistic initial
conditions). Historical runs initialized from different
times of a control run are identified by ‘r1’, ‘r2’, ‘r3’, etc.
The other letters in the ensemble identifier distinguish
between initializations of models with different methods
(‘i1’, ‘i2’, ‘i3’, etc.) and different perturbed physics (‘p1’,
‘p2’, ‘p3’, etc.). As previously mentioned, only runs with
initializations 1, 2 and 3 were selected, while the methodof-initialization and perturbed physics were fixed to i1p1.
The choice of runs is in part dictated by the unavailability
of historical (and matching climate change) runs for initializations beyond r3 and for other methods and physics.
Monthly pr, tas and u data from the Reanalysis dataset
for 1979–1999 are compared with the data from the
GCMs. In particular, tas Reanalysis patterns are used
as verification for the tas patterns from the GCM runs.
Where necessary, units were converted to match those
from the models, and the GCM data interpolated to the
2.5 × 2.5 degree resolution of the Reanalysis dataset using
the nearest grid-point method. Although it is recognized
that surface air temperature requires correction for elevation over land regions, especially if they are to be used
for impacts analysis, no correction is carried out in this
study. The purpose of this work is to explore and determine all possible sources of error, including those caused
by different grid resolutions between models. The Reanalysis data were provided by the National Oceanographic
and Atmospheric Administration, Office of Oceanic and
Atmospheric Research, Earth System Research Laboratory, Physical Sciences Division (NOAA/OAR/ESRL
PSD) in Boulder, Colorado, USA, and accessed from their
website at http://www.esrl.noaa.gov/psd.
Monthly pr data from 1979 to 1999 were also obtained
from the Global Precipitation Climatology Project
(GPCP; Adler et al., 2003), version 2.2, provided by
NOAA/OAR/ESRL PSD. The GPCP dataset combines
satellite and rain-gauge sources at a spatial resolution of
2.5 × 2.5 degrees. The GCMs’ data were converted to the
resolution of the GPCP for comparison.
Global monthly SST data (Smith and Reynolds, 2004)
version 3b (hereinafter Reynolds), from 1854 to the
present were obtained from NOAA/OAR/ESRL PSD. The
data are available at 2 × 2 degrees resolution and were
used for verification of ENSO SST patterns and indices
simulated by the GCMs.
4.
Methods
The first part of the analysis compares observed and
modeled climatic patterns from 1979 to 1999 (the common
© 2014 Royal Meteorological Society
period of the models and GPCP data) using an index of
skill. The skill index compares observed and modeled patterns of 5 metrics shown in Table 2. Initially, 12 pr and
12 tas mean monthly patterns, and 12 pr and 12 tas standard deviation monthly patterns are calculated over a target
area bounded by coordinates 4∘ N and 20∘ N, and 95∘ W
and 75∘ W. The 1979–1999 period is used for all pr and
tas metrics. Twenty pairs (modeled and observed) of patterns corresponding to the ENSO-pr teleconnection metrics are also calculated. The procedure for obtaining each
ENSO-pr teleconnection pattern was as follows:
• The Niño1.2 time series (see location of the averaging
area in Figure 1) was calculated from the observed
NOAA/OAR/ESRL PSD SST data for DJF for each year
from 1979 to 1999.
• The precipitation time series was calculated from the
observed GPCP data for each grid point in the target area
(4∘ –20∘ N, 95∘ –75∘ W) for DJF for each year from 1979
to 1999.
• Temporal correlation coefficients between the Niño1.2
time series and the precipitation time series for all the
grid points in the target area were calculated. This gave a
two-dimensional distribution of correlation coefficients
over the target area (i.e. the teleconnection pattern).
• The process was repeated for DJF Niño1.2 and DJF
precipitation time series derived for each CMIP5 model
runs.
• The process was repeated for the Niño3, Niño3.4 and
Niño4 regions during DJF.
• The process was repeated for MAM, JJA, SON and
annual averages of the ENSO indexes and precipitation
data.
The metrics in Table 2 partially differ from those used
by Hidalgo and Alfaro (2012b), as some of the metrics in
the latter study could not be calculated due to the shorter
period of record in this study (a limitation imposed by
the availability of GPCP data). The short period of record
also limits the determination of AMO-pr teleconnection
patterns given the multi-decadal nature of the AMO. Additionally, the ENSO-pr teleconnection metric was not used
by Hidalgo and Alfaro (2012b), but is incorporated in this
study because of the importance of ENSO (second only to
the annual cycle) as an influencing climatic mechanism for
Central America.
Following the study by Pierce et al. (2009), the degree of
similarity between any two climate patterns (e.g. between
the same metric calculated from both GPCP data and a
GCM simulation) was calculated using a Skill Score (SS)
defined by:
[
( )]2 [
]2
Sm
m−o
2
−
(1)
SS = rm,o − rm,o −
So
So
In Equation (1), rm,o is the Pearson’s (ordinary) spatial
correlation between modeled (i.e. GCM) and ‘observed’
(e.g. GPCP) patterns, and Sm and So are the sample spatial
standard deviations for the modeled and observed patterns,
respectively. The m and o correspond to the spatial average
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Table 1. CMIP5 runs used in the study.
No.
Model
Modeling center or group
1
access1_0
2
3
access1_3
bcc_csm1_1_m
4
5
bcc_csm1_1
bnu_esm
6
7
8
9
cancm4
canesm2
ccsm4
cesm1_bgc
10
11
12
13
14
15
16
17
cesm1_cam5_1_fv2
cesm1_cam5
cesm1_fastchem
cesm1_waccm
cmcc_cesm
cmcc_cm
cmcc_cms
cnrm_cm5
18
csiro_mk3_6_0
19
20
ec_earth
fgoals_g2
21
fgoals_s2
22
23
24
25
26
27
28
29
30
31
fio_esm
gfdl_cm2p1
gfdl_cm3
gfdl_esm2g
gfdl_esm2m
giss_e2_h_cc
giss_e2_h
giss_e2_r_cc
giss_e2_r
hadcm3
32
33
34
35
36
37
38
39
hadgem2_ao
hadgem2_cc
hadgem2_es
inmcm4
ipsl_cm5a_lr
ipsl_cm5a_mr
ipsl_cm5b_lr
miroc4h
40
41
42
43
44
45
46
47
48
miroc5
miroc_esm_chem
miroc_esm
mpi_esm_lr
mpi_esm_mr
mpi_esm_p
mri_cgcm3
noresm1_m
noresm1_me
Commonwealth Scientific and Industrial Research Organization
and Bureau of Meteorology (CSIRO-BOM)
CSIRO-BOM
Beijing Climate Center, China Meteorological Administration
(BCC)
BCC
College of Global Change and Earth System Science, Beijing
Normal University (BNU)
Canadian Centre for Climate Modelling and Analysis (CCCMA)
CCMA
CCMA
Centro Euro-Mediterraneo per I Cambiamenti Climatici
(CMCC)
CMCC
CMCC
CMCC
CMCC
CMCC
CMCC
CMCC
Centre National de Recherches Meteorologiques / Centre
Europeen de Recherche et Formation Avancees en Calcul
Scientifique (CNRM)
Commonwealth Scientific and Industrial Research Organisation
in collaboration with the Queensland Climate Change Centre of
Excellence (CSIRO-QCCE)
EC-EARTH consortium
LASG, Institute of Atmospheric Physics, Chinese Academy of
Sciences; and CESS, Tsinghua University
LASG, Institute of Atmospheric Physics, Chinese Academy of
Sciences
The First Institute of Oceanography, SOA, China
Geophysical Fluid Dynamics Laboratory (GFDL)
GFDL
GFDL
GFDL
GFDL
GFDL
GFDL
GFDL
Met Office Hadley Centre (HadCM) additional HadGEM2-ES
realizations contributed by Instituto Nacional de Pesquisas
Espaciais
HadCM-HADGM
HadCM-HADGM
HadCM-HADGM
Institute for Numerical Mathematics (INM)
Institut Pierre-Simon Laplace (IPSL)
IPSL
IPSL
Atmosphere and Ocean Research Institute (The University of
Tokyo), National Institute for Environmental Studies, and Japan
Agency for Marine-Earth Science and Technology. (MIROC)
MIROC
MIROC
MIROC
Max Planck Institute for Meteorology (MPI-M)
MPI-M
MPI-M
Meteorological Research Institute (MRI)
Norwegian Climate Centre (NCC)
NCC
© 2014 Royal Meteorological Society
Runs
1
2
3
3
1
1
3
3
1
3
3
3
2
1
1
1
3
3
2
3
3
3
3
3
3
3
1
3
1
3
3
1
1
3
1
3
1
1
3
3
1
3
3
3
2
3
3
1
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
are the Conditional and Unconditional Biases respectively
(see Pierce et al., 2009). Note that SS not only reflects
correlation coherence between the patterns but also incorporates biases in its calculation.
Once the SSs were calculated, for each model and
for each type of metric they were combined using the
Euclidean distance (Δ) from their optimum solution
SS = 1, such that the lower the Δ, the better is the model in
reproducing observed patterns. An example of the formula
used to calculate Δ for metric 1 is:
√
√ (
)2 (
)2
pr
√ 1 − SSpr
+ 1 − SSFeb mean
√
Jan mean
Δ=√
(3)
√
(
)2
pr
+ … + 1 − SSAnnual mean
Table 2. Metrics used in the study.
No.
Name of
the metric
Description
1
Precipitation
monthly means
2
Surface
temperature
monthly means
Precipitation
monthly
standard
deviations
Surface
temperature
monthly
standard
deviations
ENSO
teleconnections
Seasonal
monthly
means of pr
Seasonal
monthly
means of tas
Seasonal
monthly std.
devs. of pr
3
4
5
Symbols
Seasonal
monthly std.
devs. of tas
Correlations
between SST
(or tas as
surrogate) in
ENSO regions
and
precipitation
(over the
region of
interest) for
DJF, MAM,
JJA, SON,
and annual
of the modeled and observed climate patterns, respectively.
SS varies from −∞ (no skill) to 1 (perfect match between
the patterns). Zero SS values correspond to cases in which
the mean of the observations is reproduced correctly by the
model in a certain region, but only as a featureless uniform
pattern (Pierce et al., 2009).
The right-hand side of Equation (1) is composed of three
squared terms, and therefore SS can also be expressed as:
SS = RHO − CBIAS − UBIAS
(2)
RHO is the square of the spatial correlation between the
observed and modeled patterns, and CBIAS and UBIAS
In the equation, Δ is the Euclidean distance between
the SSs of all patterns in this particular metric (see again
definition of the metrics in Table 2) with the optimum
vector of (1, 1, 1, … 1). Therefore, Δ is computed for each
model and metric. In the example represented by Equation
(3), metric 1 (mean pr patterns) has 13 SSs – one for each
of the monthly mean patterns plus one associated with the
annual average patterns. Sorting the Δs from low to high
for all models, and for that particular metric, results in
the partial rankings for metric 1. The procedure is then
repeated for all metrics of Table 2 and the results are Δ
ranks (partial ranks) for each individual metric. The ranks
for all the metrics are added to produce a sum of rankings,
and the Final rank is produced by ranking the sum of
rankings (Table 3).
Finally, the 13 best models are compared qualitatively
and quantitatively in terms of how good they reproduce
some characteristics of the ITCZ, the seasonal cycle of the
CLLJ and the ENSO global loading pattern associated with
the first principal component (PC) of the annual average of
SST limited to 60o S and 60o N. For the models, tas is used
over the oceans as a surrogate for SST.
5. Results
5.1. SS ranking
Although the partial ranks were constructed using all the
SSs of each individual type of metric (as shown in Tables 2
30°N
20°N
10°N
0°
10°S
160°E
180°W
160°W
140°W
120°W
100°W
80°W
60°W
Figure 1. Location of the target area (solid thick line over Central America). Also shown are the ENSO SST regions: Niño1.2 (dashed line), Niño3
(dark grey region), Niño4 (light fray region) and Niño3.4 (solid thick line over the ocean partially covering Niño3 and Niño4).
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Table 3. Partial and final rankings of the GCM model runs using different metrics and variables. Numbers in parenthesis indicate run
number.
Model
cesm1_cam5(1)
cesm1_cam5(3)
cnrm_cm5(3)
cnrm_cm5(1)
cesm1_cam5_1_fv2(3)
cnrm_cm5(2)
cesm1_cam5(2)
mpi_esm_p(1)
cmcc_cms(1)
cesm1_fastchem(3)
cesm1_fastchem(2)
mpi_esm_p(2)
ccsm4(3)
ec_earth(2)
mpi_esm_lr(1)
mri_cgcm3(2)
mpi_esm_mr(3)
ccsm4(1)
cesm1_cam5_1_fv2(2)
ccsm4(2)
mpi_esm_lr(3)
mpi_esm_lr(2)
cesm1_fastchem(1)
mpi_esm_mr(2)
ec_earth(1)
cesm1_bgc(1)
giss_e2_r(3)
miroc4h(1)
miroc5(3)
miroc5(1)
inmcm4(1)
hadgem2_cc(1)
giss_e2_r_cc(1)
giss_e2_r(2)
miroc4h(2)
miroc4h(3)
giss_e2_r(1)
cesm1_cam5_1_fv2(1)
giss_e2_h(1)
mpi_esm_mr(1)
hadcm3(1)
fgoals_g2(1)
gfdl_esm2g(2)
fio_esm(2)
fgoals_g2(3)
giss_e2_h(3)
gfdl_cm3(2)
miroc5(2)
giss_e2_h(2)
bcc_csm1_1(3)
mri_cgcm3(1)
noresm1_me(1)
cmcc_cm(1)
fgoals_g2(2)
hadgem2_es(2)
gfdl_esm2g(1)
gfdl_cm3(1)
access1_3(1)
noresm1_m(3)
fio_esm(1)
hadgem2_es(1)
cesm1_waccm(2)
Metric 1
(pr)
Metric 2
(tas)
Metric 3
(pr)
Metric 4
(tas)
Metric 5
(ENSO)
Sum of
ranking
Final
rankings
9
5
1
2
12
3
4
19
7
24
31
13
42
6
18
29
65
34
14
30
28
25
32
33
8
55
41
78
15
10
56
63
45
39
80
77
36
16
59
62
20
73
27
46
67
54
68
11
57
48
37
83
23
61
71
49
66
26
87
52
74
35
17
13
18
14
20
16
15
4
21
25
24
3
27
75
5
40
10
30
22
26
1
2
28
11
77
31
92
44
35
34
37
7
95
94
45
54
93
23
99
6
55
66
85
49
67
96
61
33
97
65
41
59
29
63
9
90
56
62
52
48
12
36
18
3
15
26
16
32
28
11
1
38
50
49
39
6
37
57
70
47
19
27
34
44
42
64
25
54
8
24
12
2
75
104
23
13
36
33
17
10
21
63
30
9
35
14
5
46
56
43
29
96
51
67
89
4
106
31
52
97
77
22
102
41
4
10
51
36
6
43
19
60
66
17
2
57
11
48
87
29
39
31
58
8
61
68
3
74
64
1
12
5
86
95
33
32
24
37
7
28
21
90
46
88
98
26
59
35
27
34
23
106
56
45
41
13
65
50
63
84
54
78
9
70
40
80
6
33
1
9
42
3
43
17
27
25
24
14
22
7
8
11
2
46
76
99
68
54
89
13
23
58
48
51
61
77
19
16
38
49
64
41
69
98
12
20
39
72
40
103
81
18
45
60
15
5
94
52
70
100
30
28
55
21
59
96
63
101
54
64
86
87
96
97
109
111
122
129
131
136
141
142
155
166
186
188
189
190
192
193
194
195
197
199
201
202
209
218
220
222
225
232
232
233
236
237
237
239
242
246
246
247
247
248
253
253
254
259
264
274
276
278
279
282
283
284
284
288
291
293
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
Table 3. Continued
Model
fio_esm(3)
cesm1_waccm(1)
bcc_csm1_1(2)
noresm1_m(2)
access1_0(1)
hadgem2_ao(1)
gfdl_cm3(3)
hadcm3(3)
noresm1_m(1)
cmcc_cesm(1)
fgoals_s2(1)
mri_cgcm3(3)
gfdl_esm2g(3)
hadgem2_es(3)
hadcm3(2)
access1_3(2)
miroc_esm_chem(1)
bnu_esm(1)
canesm2(1)
cancm4(2)
ipsl_cm5a_lr(2)
cancm4(1)
giss_e2_h_cc(1)
gfdl_cm2p1(3)
ipsl_cm5a_lr(3)
fgoals_s2(3)
bcc_csm1_1_m(3)
miroc_esm(1)
miroc_esm(3)
gfdl_cm2p1(1)
bcc_csm1_1(1)
bcc_csm1_1_m(2)
ipsl_cm5a_lr(1)
gfdl_cm2p1(2)
canesm2(2)
ipsl_cm5a_mr(1)
canesm2(3)
gfdl_esm2m(1)
ipsl_cm5b_lr(1)
miroc_esm(2)
fgoals_s2(2)
csiro_mk3_6_0(3)
csiro_mk3_6_0(1)
bcc_csm1_1_m(1)
csiro_mk3_6_0(2)
Metric 1
(pr)
Metric 2
(tas)
Metric 3
(pr)
Metric 4
(tas)
Metric 5
(ENSO)
Sum of
ranking
Final
rankings
58
38
64
81
99
98
60
22
85
17
102
47
43
75
21
69
76
90
84
100
93
97
44
40
88
104
92
79
72
51
70
91
95
53
96
101
86
50
89
82
103
106
107
94
105
53
39
69
46
19
32
60
70
58
107
100
42
89
8
71
64
104
74
47
57
72
43
102
87
68
98
83
103
106
86
73
81
76
88
51
78
50
91
38
105
101
79
80
82
84
20
40
94
69
107
101
45
83
68
7
65
86
48
105
81
98
62
60
72
80
87
78
58
59
82
74
93
76
61
55
92
88
84
53
85
90
79
73
100
71
66
99
103
91
95
83
81
30
14
49
71
62
101
15
102
16
52
76
47
99
73
85
25
89
69
18
93
77
105
22
20
53
75
97
103
67
55
44
96
91
42
92
94
72
79
38
107
100
82
104
82
102
44
91
32
4
80
31
84
78
34
90
65
86
56
29
10
104
62
57
93
53
83
74
106
71
47
36
37
79
73
67
88
97
66
85
92
95
107
75
105
26
35
87
50
296
300
301
301
306
306
307
307
310
311
317
317
321
321
328
333
337
353
354
363
363
364
364
365
366
367
368
369
373
374
375
382
387
387
389
396
399
403
406
412
413
417
425
436
438
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
and 3), it is important to have a general idea of which
types of metrics showed the highest deviations from the
observed values and during which months of the year.
Histograms of the (1 − SS)2 terms for a few examples of
each different type of metrics are shown in Figures 2–4.
For pr (Figure 2), the mean observed patterns are generally
reproduced with more skill [lower (1 − SS)2 ] for January
than for July. In fact, there were a few runs that performed
very poorly during July compared to the majority of the
simulations. The standard deviations of pr were generally
also better simulated during January compared to July,
and there were more high-value (1 − SS)2 outliers during
July. In terms of tas (Figure 3), there was generally less
contrast in the skill between January and July, but there
© 2014 Royal Meteorological Society
were more high-value (1 − SS)2 outliers during July. In
terms of the ENSO-pr teleconnection, the skill during all
seasons was similar, except for MAM which (in general)
showed considerably less skill. The annual teleconnection
also showed slightly better skill than for the individual
seasons.
In Table 3, the 107 runs are shown in order of increasing
Final ranks. The partial rankings for each metric are
also included. The cesm1_cam5 model runs (1, 2 and 3)
consistently ranked at position 7 or higher in the Final
ranks. This model performed relatively well (top 10 ranks)
at reproducing the mean patterns of precipitation (Metric
1), and its other partial ranks were also relatively good
(equal or better than 28, i.e. in the top third of all model
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Figure 2. Histograms of (1 − SS)2 corresponding to the Skill Score (SS) between the modelled and observed mean and std. dev. precipitation patterns
for January and July. The 107 runs from Table 1 are considered in this figure.
Figure 3. Histograms of (1 − SS)2 corresponding to the Skill Score (SS) between the modelled and observed mean and std. dev. temperature patterns
for January and July. The 107 runs from Table 1 are considered in this figure.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
Figure 4. Histograms of (1 − SS)2 corresponding to the Skill Score (SS) between the modelled and observed mean and std. dev. correlation patterns
of Niño3.4 region SST average and Central American precipitation for different seasons. The 107 runs from Table 1 are considered in this figure.
runs), with the exception of the ranks of runs 3 and 2 for
Metric 3 (ENSO) which are slightly higher (33 and 43,
respectively). The three runs of the cnrm_cm5 model also
performed relatively well (ranked equal or better than 6) in
the Final rank. In general, the cnrm_cm5 model performed
better in terms of the ENSO teleconnection pattern (Metric 5) than the cesm1_cam5 model, but showed higher
ranks for reproducing standard deviations of tas patterns
(Metric 4). There are models represented with just one
run in the top 25 rankings, but it is remarkable to note
models with at least two runs in the top 25 positions, such
as cesm1_fastchem, ccsm4 and mpi_esm_lr. Among the
models with the poorest performance that showed at least
two runs in the bottom 25 positions were csiro_mk_6_0,
miroc_esm, canesm2, fgoals_s2, some of the ipsl_cm5
runs and others.
5.2.
Characteristics of the ‘best’ 13 runs
The best 13 runs using the Final ranking were selected and
their skill in reproducing characteristics of the 1979–1999
© 2014 Royal Meteorological Society
climate of the region examined. In Figure 5, the SSs
distribution of these 13 runs for metrics 1 through 4 (from
Table 2) are shown. Not surprisingly, there was generally better skill in reproducing mean monthly tas patterns
than mean monthly pr patterns, but both variables showed
poor results with respect to standard deviation patterns.
The runs showed median positive SSs when reproducing
mean pr patterns of February, March, April and May, but
low skill for all other months (including negative SS values), suggesting the presence of large biases or low spatial
inter-correlation between observed and modeled patterns
[Figure 5(a)]. The skill in reproducing pr standard deviations was also generally low [Figure 5(c)], although in
February and March the median SS values were above
zero, suggesting some skill during those months. In terms
of tas, the median SSs for metric 2 (Figure 5(b)) were
always above zero, suggesting significant skill in reproducing the observed patterns. However, for the standard
deviation of tas [Figure 5(d)] the results for some months
(especially September and October) suggested low skill.
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Figure 5. Boxplots of the distribution of the SSs for the best 13 models shown in Table 3. The upper two subplots correspond to the SS calculated
between the mean modelled and observed patterns for pr and tas (metrics 1 and 2 of Table 2), while the bottom correspond to the standard deviation
patterns (metrics 3 and 4 of Table 2). The boxes represent the 25th and 75th percentiles, the median is shown inside the boxes, the whiskers extend
1.5 the interquartile range or to the extent of the data, and the outliers are shown with the asterisk symbol.
Figure 6. Boxplots of the distribution of the SSs for the best 13 models shown in Table 3. The upper two subplots correspond to the SS calculated
between the mean modelled and observed patterns for the ENSO teleconnections (metric 5 of Table 2) for different ENSO indexes. The boxes
represent the 25th and 75th percentiles, the median is shown inside the boxes, the whiskers extend 1.5 the interquartile range or to the extent of the
data, and the outliers are shown with the asterisk symbol.
With respect to ENSO-pr teleconnections (Figure 6),
there was generally less skill during MAM for all ENSO
indicators. However, in the case of annual averages, the SS
medians for the best 13 models were always above zero
for the four ENSO indicators evaluated. Positive median
© 2014 Royal Meteorological Society
skill was found in at least one of the four ENSO indicators
during JJA and SON, while the SS medians during DJF
and MAM were always below zero for all indicators. The
strongest ENSO-pr teleconnection is known to be located
off the Pacific coast of Central America (Waylen et al.,
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
1996) which has a rainy season during May to November
and a dry season from December to April (Alfaro, 2002;
Taylor and Alfaro, 2005). It is noteworthy then, that the
highest ENSO skill of the GCM runs occurs mainly during
the wettest seasons of the year in this climate regime.
In Figures 7 and 8, mean pr patterns for the GCPC, the
Reanalysis and the best 13 GCM models are shown for
January and July, respectively (1979–1999). The area of
interest over which the SSs were calculated is shown as a
red square in Figures 7(a) and 8(a). The SS values inside
the subplots indicate that the GCMs generally performed
as well or better than the Reanalysis in reproducing mean
GCPC pr patterns in the area of interest during January and
July. In fact, all of the first 13 models showed top (highest?)
ranks in terms of metric 1 (Table 3).
Although there is no direct comparison between the
methodologies employed in this study and that used in
the study by Sheffield et al. (2013), some similarities and
differences are noted between results obtained for models common to both studies. For example, the runs corresponding to the crnm_cm5 model are highly ranked
in Table 3 for the partial rank corresponding to metric 1 (mean pr patterns). In the study by Sheffield et al.
(2013), the model showed low pr biases compared to the
models they analyzed for December–January–February
(DJF), but significant pr biases for June–July–August
(JJA). Another difference is that in the study by Sheffield
et al.’s (2013) study, the csiro_mk3.6.0 model showed relatively low pr biases in both DJF and JJA, while in this
study the model did not perform well (Table 3). We do not
think that the mentioned differences are solely explained
by the different methodologies used. They may also result
from differences in the definition of the regions over which
the models were evaluated. It should be mentioned, however, that the mean pr SSs found for the best 13 models
during January and July (Figures 7 and 8) were generally low and in many cases negative, possibly due to the
large biases in the means detected in the GCMs (Hidalgo
and Alfaro, 2012b). Fortunately, some of these biases can
be partially corrected through bias-correction procedures
(Maurer and Hidalgo, 2008; Maurer et al., 2010; Hidalgo
et al., 2013). However, it is desirable to use models requiring the smallest amount of correction and hence an evaluation of the models’ skill such as is being done in this study
is useful.
Although the area of interest is smaller than the domain
shown in the maps of Figures 7–16, the larger domain
facilitates a qualitative evaluation of the shape of the ITCZ
for the months shown. The plots show that the ITCZ over
the eastern Pacific Ocean was stronger and more defined
in July than in January and that the GCMs capture the
shape of the pr maximum over the Pacific Ocean with
some skill (interestingly, the Reanalysis appears to be
missing this feature, but it should be remembered that
the Reanalysis precipitation is modeled and not observed).
From Figures 7 and 8, there was no clear evidence of a
double ITCZ, a problem that was common to GCMs in
the previous generation of models (Hirota et al., 2011;
Liu et al., 2012). To investigate this further, the difference
© 2014 Royal Meteorological Society
maps (modeled − observed) of the pr patterns for the best
13 models were plotted (not shown). In these plots, there
was evidence of a double ITCZ in January in the models
ranked 1 to 7 and 11. During July, although there are errors
present in the model, the patterns do not resemble a double
ITCZ (not shown). The problems with the double ITCZ in
these best 13 models appear to occur in the months when
the ITCZ is the weakest, and the pr over Central America
is the lowest.
In terms of the pr standard deviations (Figures 9 and 10),
there was better skill in the GCMs than in the Reanalysis in January over the region of interest. In July, some
models performed better while others did worse than the
Reanalysis. In general, there was great dissimilarity (over
the entire plot domain) between the large-scale GPCP pr
patterns and those from the Reanalysis or GCMs. It is evident that of all the metrics examined, the standard deviations are the most difficult patterns to be replicated by
the models.
For tas means (Figures 11 and 12), the SSs were
very high and positive during January and July. The
large-scale patterns were also well replicated, including
the contrast between land and sea in some regions of
the maps. The tas standard deviations (Figures 13 and
14) were not well reproduced during January for the
area of interest or the large-scale, Interestingly, however,
some of the GCM runs (cesm1_cam5(1), cesm1_cam5(3),
cesm1_cam5_1_f2, cesm1_cam5(2), cesm1_fastchem(3),
cesm1_fastchem(2) and ccsm4(3)) showed significantly
positive skill in reproducing the standard deviations of the
July tas in the region of interest. This was consistent with
Figure 5(d) which showed that the best skill for standard
deviations of tas was found during the month of July, and
the worst during October.
With respect to the ENSO annual teleconnection using
Niño3.4, all GCM runs were better than the Reanalysis at
reproducing the observed ENSO-pr teleconnection pattern
contained in the GPCP/Reynolds data for the area of
interest (Figure 15). Overall, the SSs were all positive and
most of them relatively high. The large-scale correlation
patterns (considering the entire map domain) generally
showed different amplitudes compared to the observations
(Figure 15), though in many model runs the patterns had
similar shapes. The patterns over the region of interest
showed very good coherence between the observed and
modeled patterns in most of the cases.
In order to examine the ability of the models in reproducing global ENSO patterns, the loading patterns of the
first un-rotated PC of annual SST (or tas as surrogate) over
the global oceans inside 60o S and 60o N were computed
from the Reynolds data, the Reanalysis and the models
(Figure 16). The variances explained by these PCs are
shown in Table 4. On average, the first PCs of the GCM
runs explained around 23% of the variance, while in the
observations this number was close to 21%. The GCM runs
performed poorly in reproducing the global loading pattern with respect to the Reanalysis, as this latter database
includes assimilated tas observations (in contrast to the
GCM runs that are entirely modeled). The closest match
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Figure 7. January mean precipitation (mm month−1 ) for GPCP, Reanalysis and the best 13 GCM runs ordered from lowest to highest Final ranking
(see text). Numbers in parenthesis indicate the run number. The square in the GPCP subfigure represents the area of interest where comparisons
were made between observed and modeled patterns for selecting the best models. The Reanalysis was included for reference (i.e. not ranked with
the GCMs). Numbers in the subfigures represent the Skill Score for the seasonal patterns in the area of interest.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
Figure 8. July mean precipitation (mm month−1 ) for GPCP, Reanalysis and the best 13 GCM runs ordered from lowest to highest Final ranking (see
text). Numbers in parenthesis indicate the run number. The square in the GPCP subfigure represents the area of interest where comparisons were
made between observed and modeled patterns for selecting the best models. The Reanalysis was included for reference (i.e. not ranked with the
GCMs). Numbers in the subfigures represent the Skill Score for the seasonal patterns in the area of interest.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(m)
(n)
(o)
Figure 9. January precipitation standard deviation (mm month−1 ) for GPCP, Reanalysis and the best 13 GCM runs ordered from lowest to highest
Final ranking (see text). Numbers in parenthesis indicate the run number. The square in the GPCP subfigure represents the area of interest where
comparisons were made between observed and modeled patterns for selecting the best models. The Reanalysis was included for reference (i.e. not
ranked with the GCMs). Numbers in the subfigures represent the Skill Score for the seasonal patterns in the area of interest.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
Figure 10. July precipitation standard deviation (mm month−1 ) for GPCP, Reanalysis and the best 13 GCM runs ordered from lowest to highest
Final ranking (see text). Numbers in parenthesis indicate the run number. The square in the GPCP subfigure represents the area of interest where
comparisons were made between observed and modeled patterns for selecting the best models. The Reanalysis was included for reference (i.e. not
ranked with the GCMs). Numbers in the subfigures represent the Skill Score for the seasonal patterns in the area of interest.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Figure 11. January mean temperature (∘ C) for Reanalysis and the best 13 GCMs ordered from lowest to highest Final ranking (see text). Numbers
in parenthesis indicate the run number. The square in the REAN subfigure represents the area of interest where comparisons were made between
observed and modeled patterns for selecting the best models. Numbers in the subfigures represent the Skill Score for the seasonal patterns.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
Figure 12. July mean temperature (∘ C) for Reanalysis and the best 13 GCMs ordered from lowest to highest Final ranking (see text). Numbers
in parenthesis indicate the run number. The square in the REAN subfigure represents the area of interest where comparisons were made between
observed and modeled patterns for selecting the best models. Numbers in the subfigures represent the Skill Score for the seasonal patterns.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Figure 13. January temperature standard deviation (∘ C) for Reanalysis and the best 13 GCMs ordered from lowest to highest Final ranking (see text).
Numbers in parenthesis indicate the run number. The square in the REAN subfigure represents the area of interest where comparisons were made
between observed and modeled patterns for selecting the best models. Numbers in the subfigures represent the Skill Score for the seasonal patterns.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
Figure 14. July temperature standard deviation (∘ C) for Reanalysis and the best 13 GCMs ordered from lowest to highest Final ranking (see text).
Numbers in parenthesis indicate the run number. The square in the REAN subfigure represents the area of interest where comparisons were made
between observed and modeled patterns for selecting the best models. Numbers in the subfigures represent the Skill Score for the seasonal patterns.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Figure 15. Annual correlations between precipitation and Niño3.4 for GPCP/Reynolds, Reanalysis and the best 13 GCMs runs ordered from lowest
to highest delta total (see text). Numbers in parenthesis indicate the run number. The square in the GPCP/Reynolds subfigure represents the area
of interest where comparisons were made between observed and modeled patterns for selecting the best models. The Reanalysis was included for
reference (i.e. not ranked with the GCMs). The numbers in the figures are the Skill Scores between observed and modeled patterns.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
Figure 16. Annual correlations between precipitation and Niño3.4 for annual temperature loading maps of the first principal component of SST (or
tas) over the oceans from 60o S to 60o N. GCMs runs ordered from lowest to highest delta total (see text). Numbers in parenthesis indicate the run
number. The square in the Reynolds subfigure represents the area of interest where comparisons were made between observed and modeled patterns
for selecting the best models. The Reanalysis was included for reference (i.e. not ranked with the GCMs). The numbers inside the maps are the Skill
Scores between the observed pattern (using Reynolds data) and each model for the global domain.
© 2014 Royal Meteorological Society
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Table 4. Variance explained by first principal component of SST
or tas over the global ocean below 60o S and 60o N, for 13 models
of Figure 7.
Model
Reynolds
REAN
cesm1_cam5(1)
cesm1_cam5(3)
cnrm_cm5(3)
cnrm_cm5(1)
cesm1_cam5_1_fv2(3)
cnrcm_cm5(2)
cesm1_cam5(2)
mpi_esm_p(1)
cmcc_cms(1)
cesm1_fastchem(3)
cesm1_fastchem(2)
mpi_esm_p(2)
ccsm4(3)
Variance
explained (%)
20.69
20.03
20.32
26.90
18.73
18.44
22.53
16.63
27.38
19.94
28.56
27.29
21.36
24.94
24.28
Figure 17. Seasonal averages of the CLLJ index for the Reanalysis and
selected models with available wind data. See text for definition of the
CLLJ index. Data for model cesm1_cam5_fv(2) were not available. The
darkest line correspond to the Reanalysis results.
with observations from the subset of 13 best models was
cesm1_fastchem(2) which had a SS of −0.81 compared to
0.57 for the Reanalysis).
A few of the best GCMs had wind speed data. In
Figure 17, the seasonal cycles of the CLLJ index (Amador,
1998, 2008) are shown. The CLLJ index was constructed
by averaging the zonal wind speed at 925 hPa from the
Reanalysis and the GCMs in a region bounded by latitudes
7.5∘ N to 12.5∘ N, and longitudes 85∘ W and 75∘ W. Note
that a stronger jet is associated with a more negative CLLJ
index, as the sign convention of the wind data is negative
for easterly winds (the prevailing direction). Although the
sample size is too small to reach robust conclusions, it is
evident that at least some of the models were very skillful at reproducing the seasonality of the CLLJ. The CLLJ
seasonal cycles of the three cnrm_cm5 runs were very
similar to those of the Reanalysis data (Figure 17), while
other models had significant negative biases. These models
suggest themselves as good possibilities for further future
evaluation of other parameters (e.g. atmospheric circulations, humidity patterns and others) to determine if they
too will display such promising results.
© 2014 Royal Meteorological Society
6.
Summary and conclusions
The skill of 107 GCMs runs in reproducing climatic
patterns of Central America as observed in the GCPC,
Reynolds and Reanalysis data was measured using a SS
index. Results indicated that for the metrics evaluated, the
models had greatest difficulty in accurately reproducing
the monthly standard deviation pr and tas patterns over
the region of interest. With some exceptions, the standard
deviations for pr and tas, were reproduced with low skill,
even in the best models. Reproducing precipitation mean
monthly patterns was also challenging for the models.
Only a few of the best 13 runs have high SSs for reproducing mean pr patterns. For tas, the models, however, show
significant skill. It should be remembered that biases play
an important role in producing low SSs for the mean and
standard deviations of pr and tas. Many of these biases can
be reduced significantly using bias-correction procedures,
which could make these runs useful for climate change
studies (Maurer and Hidalgo, 2008; Maurer et al., 2010;
Hidalgo and Alfaro, 2012b; Hidalgo et al., 2013). However, as previously noted, it is desirable to identify and use
the less biased models, and this study helps with this task.
Other features are more difficult to fix when they are
not simulated well, such as the ENSO teleconnection patterns and ENSO characteristic SST patterns, ITCZ patterns
and the seasonal cycles of the CLLJ index. It is also difficult to determine the causes of the differences between
runs/models, as they all have different physics, initializations and/or spatial resolutions.
The Final ranking of the models showed that differences
in the initialization of the models (resulting in different
runs of a particular model) are not as important when
determining its performance, compared to the difference
between models’ structure. That is, generally the multiple
runs of a particular model have similar ranks, grouped in
the top, middle or bottom of the list.
Finally, features such the shapes and amplitudes of the
ITCZ patterns in the GCMs are in some cases reproduced
with more skill than the Reanalysis. However, the Reanalysis is much better in reproducing the ENSO SST patterns than the GCM runs. The assimilated temperature
data in the Reanalysis may be playing an important role
in this regard. The seasonality of the CLLJ index was
reproduced with great skill by some of the models/runs,
however, the sample is too small to reach conclusive statements. The most promising models can/should be studied
in more detail using other variables and time-scales to validate their usefulness for reproducing climatic features of
the region.
Acknowledgements
This work was partially funded by the International
Climate Initiative (ICI) of the German Federal Ministry
for the Environment, Nature Conservation, Building
and Nuclear Safety (BMUB), as part of the CASCADE
project (‘Ecosystem-based Adaptation for Smallholder
Subsistence and Coffee Farming Communities in Central
Int. J. Climatol. (2014)
H. G. HIDALGO AND E. J. ALFARO
America’). The German Federal Ministry for the Environment, Nature Conservation, Building and Nuclear
Safety (BMUB) supports this initiative on the basis of
a decision adopted by the German Bundestag. Also
by projects 805-B3-413, 805-A9-224, 808-A9-180,
805-A9-532, 805-B3-600 and 808-B0-654, from the
Center for Geophysical Research (CIGEFI) and the
Center for Research in Marine Sciences and Limnology (CIMAR) of the University of Costa Rica (UCR).
Thanks to the logistics support provided by the School
of Physics of UCR. Thanks also to Dr. Pablo Imbach
from CATIE for supplying much of the original raw GCM
data. The authors thank Ricardo Herrera and Andrés
Jiménez who formatted the data and collaborated in
the calculation of some of the indexes. Thanks also to
two anonymous reviewers whose comments considerably enhanced the quality and structure of this paper.
Thanks to Michael Taylor and David Enfield for revising
the manuscript.
References
Adler RF, Huffman GJ, Chang A, Ferraro R, Xie P, Janowiak J, Rudolf
B, Schneider U, Curtis S, Bolvin D, Gruber A, Susskind J, Arkin P.
2003. The Version 2 Global Precipitation Climatology Project (GPCP)
monthly precipitation analysis (1979–present). J. Hydrometeorol. 4:
1147–1167.
Alfaro EJ. 2000. Response of air surface temperatures over Central
America to oceanic climate variability indices. Tóp. Meteor. Oceanog.
7(2): 63–72.
Alfaro E. 2002. Some characteristics of the annual precipitation cycle in
Central America and their relationships with its surrounding tropical
oceans. Tóp. Meteor. Oceanog. 9(2): 88–103.
Amador JA. 1998. A climatic feature of the tropical Americas: the trade
wind easterly jet. Tóp. Meteor. Oceanog. 5: 91–102.
Amador JA. 2008. The intra-Americas sea low-level jet, overview and
future research. Ann. N. Y. Acad. Sci. 1146: 153–188.
Amador JA, Alfaro EJ, Lizano OG, Magaña VO. 2006. Atmospheric
forcing of the eastern tropical Pacific: a review. Prog. Oceanogr. 69:
101–142.
Bindoff NL, Stott PA, AchutaRao KM, Allen MR, Gillett N, Gutzler
D, Hansingo K, Hegerl G, Hu Y, Jain S, Mokhov II, Overland J,
Perlwitz J, Sebbari R, Zhang X. 2013. Detection and attribution of
climate change: from global to regional. In Climate Change 2013:
The Physical Science Basis. Contribution of Working Group I to the
Fifth Assessment Report of the Intergovernmental Panel on Climate
Change, Stocker TF, Qin D, Plattner G-K, Tignor M, Allen SK,
Boschung J, Nauels A, Xia Y, Bex V, Midgley PM (eds). Cambridge
University Press: Cambridge, UK and New York, NY.
Brekke LD, Dettinger MD, Maurer EP, Anderson M. 2008. Significance
of model credibility in estimating climate projection distributions
for regional hydroclimatological risk assessments. Clim. Change 89:
371–394.
Delworth TL, Rosati A, Anderson W, Adcroft AJ, Balaji V, Benson R,
Dixon K, Griffies SM, Stephen M, Lee HC, Pacanowski RC, Vecchi
GA, Wittenberg AT, Zeng FR, Zhang R. 2012. Simulated climate and
climate change in the GFDL CM2.5 high-resolution coupled climate
model. J. Clim. 25: 2755–2781.
Enfield DB, Mestas-Nuñez AM, Trimble PJ. 2001. The Atlantic multidecadal oscillation and its relation to rainfall and river flows in the
continental U.S. Geophys. Res. Lett. 28: 2077–2080.
Flato G, Marotzke J, Abiodun B, Braconnot P, Chou SC, Collins W, Cox
P, Driouech F, Emori S, Eyring V, Forest C, Gleckler P, Guilyardi E,
Jakob C, Kattsov V, Reason C, Rummukainen M. 2013. Evaluation of
climate models. In Climate Change 2013: The Physical Science Basis.
Contribution of Working Group I to the Fifth Assessment Report of
the Intergovernmental Panel on Climate Change, Stocker TF, Qin D,
Plattner G-K, Tignor M, Allen SK, Boschung J, Nauels A, Xia Y, Bex
V, Midgley PM (eds). Cambridge University Press: Cambridge, UK
and New York, NY.
© 2014 Royal Meteorological Society
Gates WL, Boyle JS, Covey C, Dease CG, Doutriaux CM, Drach RS,
Fiorino M, Gleckler PJ, Hnilo JJ, Marlais SM, Phillips TJ, Potter GL,
Santer BD, Sperber KR, Taylor KE, Williams DN. 1999. An Overview
of the Results of the Atmospheric Model Intercomparison Project
(AMIP I). Bull. Am. Meteorol. Soc. 80: 29–55.
Giorgi F. 2006. Climate change hot-spots. Geophys. Res. Lett. 33: 8707,
doi: 10.1029/2006GL025734.
Hidalgo HG, Alfaro EJ. 2012a. Some physical and socio-economical
aspects of climate change in Central America. Prog. Phys. Geogr. 36:
379–399.
Hidalgo HG, Alfaro EJ. 2012b. Global model selection for evaluation of
climate change projections in the eastern tropical Pacific seascape. Int.
J. Trop. Biol. Conserv. 60: 67–81.
Hidalgo HG, Amador JA, Alfaro EJ, Quesada B. 2013. Hydrological climate change projections for Central America. J. Hydrol. 495: 94–112.
Hirota N, Takayabu YN, Watanabe M, Kimoto M. 2011. Precipitation
reproducibility over tropical oceans and its relationship to the double
ITCZ problem in CMIP3 and MIROC5 climate models. J. Clim. 24:
4859–4873.
IPCC. 1997. The Regional Impacts of Climate Change: An Assessment
of Vulnerability. Regional Vulnerability to Global Climate Change,
Watson RT, Zinyowera MC, Moss RH (eds). Cambridge University
Press: Cambridge, UK.
IPCC. 2013. Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the
Intergovernmental Panel on Climate Change, Stocker TF, Qin D, Plattner G-K, Tignor M, Allen SK, Boschung J, Nauels A, Xia Y, Bex V,
Midgley PM (eds). Cambridge University Press: Cambridge, UK and
New York, NY.
Jiang X, Waliser DE, Kim D, Zhao M, Sperber KR, Stern WF, Shubert
SD, Zhang GJ, Wang W, Khairoutdinov M, Neale RB, Lee M. 2012.
Simulation of the intraseasonal variability over the Eastern Pacific
ITCZ in climate models. Clim. Dyn. 39: 617–636.
Kalnay E, Kanamitsu M, Kistler R, Collins W, Deaven D, Gandin L,
Iredell M, Saha S, White G, Woollen J, Zhu Y, Leetmaa A, Reynolds
R. 1996. The NCEP/NCAR 40-year reanalysis project. Bull. Am.
Meteorol. Soc. 77: 437–471.
Kim HM, Webster PJ, Curry JA. 2012. Evaluation of short-term climate
change prediction in multi-model CMIP5 decadal hindcasts. Geophys.
Res. Lett. 39: L10701, doi: 10.1029/2012GL051644.
Knutti R, Sedláček J. 2013. Robustness and uncertainties in the new
CMIP5 climate model projections. Nat. Clim. Chang. 3: 368–373.
Langenbrunner B, Neelin JD. 2013. Analyzing ENSO teleconnections in
CMIP models as a measure of model fidelity in simulating precipitation. J. Clim. 26: 4431–4446.
Liu HL, Zhang MH, Lin WY. 2012. An investigation of the initial
development of the double-ITCZ warm SST biases in the CCSM.
J. Clim. 25: 140–155.
Magaña V, Amador J, Medina S. 1999. The midsummer drought over
Mexico and Central America. J. Clim. 12: 1577–1588.
Maldonado T, Alfaro E, Fallas B, Alvarado L. 2013. Seasonal prediction
of extreme precipitation events and frequency of rainy days over Costa
Rica, Central America, using Canonical Correlation Analysis. Adv.
Geosci. 33: 41–52.
Mantua NJ, Hare SR, Zhang Y, Wallace JM, Francis RC. 1997. A Pacific
interdecadal climate oscillation with impacts on salmon production.
Bull. Am. Meteorol. Soc. 78: 1069–1079.
Martin ER, Schumacher C. 2011. The Caribbean low-level jet and its
relationship with precipitation in IPCC AR4 models. J. Clim. 24:
5935–5950.
Maurer EP, Hidalgo HG. 2008. Utility of daily vs. monthly large-scale
climate data: an intercomparison of two statistical downscaling methods. Hydrol. Earth Syst. Sci. 12: 551–563.
Maurer EP, Hidalgo HG, Das T, Dettinger MD, Cayan DR. 2010. The
utility of daily large-scale climate data in the assessment of climate
change impacts on daily streamflow in California. Hydrol. Earth Syst.
Sci. 14: 1125–1138, doi: 10.5194/hess-14-1125-2010.
Meehl GA, Covey C, Delworth T, Latif M, McAvaney B, Mitchel
JFB, Stouffer RJ, Taylor KE. 2007. The WCRP CMIP3 Multimodel
Dataset: A New Era in Climate Change Research. Bull. Am. Meteorol.
Soc. 88: 1383–1394, doi: 10.1175/BAMS-88-9-1383.
Neelin JD, Langenbrunner B, Meyerson JE, Hall A, Berg N. 2013.
California winter precipitation change under global warming in the
coupled model intercomparison project phase 5 ensemble. J. Clim. 26:
6238–6256, doi: 10.1175/JCLID1200514.1.
Pierce DW, Barnett TP, Hidalgo HG, Das T, Bonfils C, Sander B, Bala G,
Dettinger M, Cayan D, Mirin A. 2008. Attribution of declining western
US snowpack to human effects. J. Clim. 21: 6425–6444.
Int. J. Climatol. (2014)
SKILL OF CMIP5 CLIMATE MODELS IN CENTRAL AMERICA
Pierce DW, Barnett TP, Santer BD, Gleckler PJ. 2009. Selecting global
climate models for regional climate change studies. Proc. Natl Acad.
Sci. USA. 106: 8441–8446.
Polade SD, Gershunov A, Cayan DR, Dettinger MD, Pierce DW. 2013.
Natural climate variability and teleconnections to precipitation over
the Pacific-North American region in CMIP3 and CMIP5 models.
Geophys. Res. Lett. 40: 2296–2301, doi: 10.1002/grl.50491.
Rauscher SA, Kucharski F, Enfield DB. 2011. The role of regional
SST warming variations in the drying of Meso-America in future
climate projections. J. Clim. 24: 2003–2016, doi: 10.1175/2010
JCLI3536.1.
Sachs JP, Sachse D, Smittenberg RH, Zjang Z, Battisti DS, Golubic S. 2009. Southward movement of the Pacific intertropical
convergence zone AD 1400–1850. Nat. Geosci. 2: 519–525, doi:
10.1038/NGEO554.
© 2014 Royal Meteorological Society
Sheffield J et al. 2013. North American climate in CMIP5 experiments,
part I: evaluation of historical simulations of continental and regional
climatology. J. Clim. 26: 9209–9245, doi: 10.1175/JCLID1200592.1.
Smith TM, Reynolds RW. 2004. Improved extended reconstruction of
SST (1854–1997). J. Clim. 17: 2466–2477.
Taylor M, Alfaro E. 2005. Climate of Central America and the
Caribbean. In Encyclopedia of World Climatology, Oliver JE (ed).
Springer: Amsterdam, 183–189.
Taylor KE, Balaji V, Hankin S, Juckes M, Lawrence B, Pascoe S.
2012a. CMIP5 and AR5 data reference syntax (DRS) and controlled
vocabularies. http://cmip.llnl.gov/cmip5/ (accessed 6 January 2013).
Taylor KE, Stouffer RJ, Meehl GA. 2012b. A summary of the CMIP5
experiment design. Bull. Am. Meteorol. Soc. 93: 485–498.
Waylen PR, Quesada ME, Caviedes CN. 1996. Interannual variability on
monthly rainfall in Costa Rica. J. Clim. 9: 2606–2613.
Int. J. Climatol. (2014)