C-Coupler1: a Chinese
community coupler for Earth
system modeling
Li Liu, Cheng Zhang, Ruizhe Li, Guangwen Yang,
Bin Wang, Zhiyuan Zhang
Tsinghua University, China
[email protected]
http://c-coupler.org/index.action
1
Outline
• C-Coupler development
• C-Coupler1
• Bitwise identical reproducibility
• Future work
2
Milestones of C-Coupler
development
Milestones
Time
Initiation
2010.01
Main design and software design
2010.10
Prototype system of C-Coupler
2011.07
Common multi-dimensional remapping software CoR
2012.07
Prototype system of C-Coupler platform
2012.12
Early release of C-Coupler1 (beta version) and FGOALS-gc
2013.09
Parallel 3-D coupling
2013.10
Enhancement for bitwise identical reproducibility on the CCoupler platform
2013.12
Release of C-Coupler1
2014.06
3
Target functions of C-Coupler

Science




Technology



Flux computation
3-D coupling
Two-way model nesting and interactive ensemble
Modularization, extendibility
High parallel efficiency
Application



Powerful coupling functions, user friendliness
Reliability, automatic error detection
Reproducibility of simulation results
4
Architecture of models with C-Coupler
ATM
ICE
C-Coupler
Coupler
component
LND
OCN
5
Software structure of C-Coupler
Standardized component models
Component models code
Component models configuration
ATMs
OCNs
LNDs
ICEs
Carbon
Configuration
system
Coupling
flow
Coupled
models
configuration
configuration
Coupling
Coupling
Runtime
generator
configuration
generator
…
Interfaces
Process MGR
Communication MGR
Grid MGR
Remapping MGR
Restart MGR
Decomposition MGR
Time MGR
Data MGR
Runtime software
Interfaces
system
C-Coupler runtime
software system
Remapping
algorithms
Scientific
algorithms
I/O
algorithms
External coupling algorithms
6
Software modules
Standardized
component
models
External
algorithms
Create case
Runtime
configuration files of
experiment models
From a default setting
C-Coupler
From an existing setting
Namelist
Parallel settings
Compiling options
Initial or restart
Output settings
Start and stop time
Configure
Compile
Run case
Output data
Input data
Experimental
setting package
Outline
• C-Coupler development
• C-Coupler1
• Bitwise identical reproducibility
• Future work
8
C-Coupler1
• C-Coupler runtime software system for 3-D coupling
(in C++)
– CoR1.0: A common multi-dimensional remapping
software for remapping, grid and field data management
– Coupling interfaces (Fortran and C++)
– Function managers, e.g., time manager, communication
manager, etc.
– Parallelization
• C-Coupler platform: a runtime environment for model
development, simulation and reproducibility
• Runtime configuration for the CPL6 coupling flow
• Imported: CPL6 flux algorithms
9
Metadata for runtime configuration
• Timer
– <unit of frequency, count of frequency, count of
delay>
• Field instance
– <field name, component name, parallel
decomposition, grid name>
10
Metadata for runtime configuration
• External algorithm
11
Metadata for runtime configuration
• runtime algorithm list
• Runtime procedure
12
Data transfer and interpolation
• Data transfer
– All fields (of different data types, on different grids, on
different parallel decompositions, or with different
dimensions) to be transferred at the current time step can
be packed into one message
• 3-D interpolation
– 2-D+1-D implementation
• Spline is supported for 1-D interpolation
– Offline and online
– Parallel dynamic 3-D interpolation now (for example, for
coupling between AGCM and atmospheric chemistry
model)
13
Example: parallel 2D flux coupling,
global coupling, climate system model
GAMIL2
LICOM2
C-Coupler1
CLM3
FGOALS-gc
CICE4_
LASG
14
Coupler component in FGOALS-gc
15
Example: Sharing platform for GAMIL
development
FGOALS-gc
CLM2
FGOALS-gc-colm
…
GAMIL
GEOS-Chem
CLM3
CoLM
CLM4
C-Coupler platform
16
Example: parallel 3-D coupling,
regional coupling, direct coupling
WRF
Direct coupling
2D coupling
C-Coupler
MASNUM-Wav 3-D coupling:
2D coupling
2D coupling
Four choices for 1D
interpolation for vertical level
POM
Example: model integration
• Integrating a standalone model versions, e.g.,
CESM1.2.1, CESM1.0.5, WRF and MOM4p1,
onto the C-Coupler platform
– Several configuration files
– Less than 10 lines of source code in the main driver
– Enhancement for bitwise-identical reproducibility to
the simulations
18
Outline
• C-Coupler development
• C-Coupler1
• Bitwise identical reproducibility
• Future work
19
Reproducibility
• A fundamental principle of scientific research
• More and more claims for reproducibility of
published results
– Nature family, Science and Geoscientific Model
Development, etc.
20
Bitwise identical reproducibility?
• It may be unnecessary, because climate simulations
results are generally statistical characteristics of
output data on time scales longer than a few months
• It was extremely difficult to achieve bitwise identical
reproducibility
– The whole simulation setting needs to be recorded
and recovered
• Existing works show that climate simulation results
can be sensitive to round-off error
21
Experimental setups
• Two fully coupled models: CESM1 and FGOALS-g2
• CMIP5 historical experiments: 60 years (1850-1909)
Simulation cases
Model
#1
#2
#3
#4
CESM1
11.1_120_C1
11.1_128_C1
11.1_128_C2
12.1.3_128_C1
FGOALS-g2
11.1_104_C1
11.1_108_C1
11.1_108_C2
12.1.3_108_C1
Model
CESM1
FGOALS-g2
Model
CESM1
Number of processes
ATM
OCN
LND
ICE
CPL
GLC
120
120
120
120
120
120
120
128
128
128
128
128
128
128
104
30
18
24
20
12
-
108
30
18
24
20
16
-
Label
Compiling option
C1
-O2 -convert big_endian -assume byterecl -ftz -FR -fp-model precise
C2
-O2 -convert big_endian -assume byterecl -ftz -FR
-c -r8 -i4 -O2 -zero -132 -convert big_endian -assume byterecl -no-vec -mp1 -fpmodel precise -fp-speculation=safe
FGOAL C1
S-g2
Label
C2
-c -r8 -i4 -O2 -zero -132 -convert big_endian -assume byterecl
22
Climatological mean TS by CESM1
23
Climatological mean TS by FGOALS-g2
Bitwise identical reproducibility is
important to Earth system modeling
Current status of bitwise identical
reproducibility of published results?
24
Design of a survey: 17 journals
Climate Dynamics
Geophysical Research Letters
Geoscientific Model Development
Global and Planetary Change
Global Biogeochemical Cycles
Journal of Advances in Modelling Earth
Systems
Journal of Climate
Journal of Geophysical Research:
Atmospheres
Journal of Hydrology
Journal of Physical Oceanography
Journal of the Atmospheric Sciences
Monthly Weather Review
Nature
Nature Climate Change
Nature Geoscience
Proceedings of the National Academy of
Sciences of the United States of America
Quarterly Journal of the Royal
Meteorological Society
25
Statistical characteristics of paper
selection
Year of publishing
Number of citations
Number of selected
2006 2007 2008 2009 2010 2011 2012 2013 2014
≥10
≥9
≥8
≥7
≥5
≥4
≥3
≥1
≥0
35
35
42
41
42
45
48
46
17
92.1
74.5
65.9
52.6
26.0
31.8
20.4
5.1
0.4
papers
Average number of
citations per paper
26
Results of the survey
• No reply: 283 papers (80.6%)
– No corresponding authors: 5 papers (1.4%)
– Automatic email rejection: 66 papers (18.8%)
– No active reply: 212 papers (60.4%)
• Replied without required information: 54 papers (15.4%)
– Replied without required information and confirmation: 7
papers (2%)
– Inconvenient for reproduction: 47 papers (13.4%)
• Unsuccessful re-run: 4 papers (1.1%)
• Successful re-run: 5 papers (1.4%)
• Successful bitwise identical reproduction: 5 papers (1.4%)
27
Brief summary
• Fellow scientists heavily depend on the authors’
help to reproduce the published simulation results
• It is always inconvenient even impossible to recreate
the same simulation setting as the whole simulation
setting is rarely kept for a long time
Bitwise identical reproducibility of Earth
system
modeling
is currently
very
lowtolevel
• The authors
still have
to spend aat
lotaof
efforts
help the fellow scientists who want to reproduce
these results, even when the whole simulation
setting can be recalled
28
Reproducibility for model code
Configure
Reproducibility for input data
GIT server and code version id for
each component model
SVN server and version id for
each input data file
Code patch for each
component model
Check sum of
each input data file
Reproducibility for input parameters
Reproducibility for parallel settings
Script for generating the input
parameter files of each component
model
Configuration file with the parallel
settings of all component models
Reproducibility for computer system
C-Coupler
platform
Reproducibility for Compiler
and compiling options
Name of the computer system
Configuration file of compiling
options of each component model
Log information for configuring
Information of compiler for each
component model
Username, computer name and
configuration time, and error and
warning report for configuration
Experimental setup package for technical reproducibility
Log files for technical reproducibility Output files for technical reproducibility
Compile and run
Log files for compiling
Name of model version
Log files for the execution of model
simulation
Description of model simulation
29
Time of the corresponding configuration
Flowchart for achieving bitwise identical
reproducibility on the C-Coupler platform
30
An example
31
Worldwide standard of
bitwise identical reproducibility
• Any fellow scientists can independently
obtain the whole simulation setting of
published results and then can independently
reproduce exactly the same simulation output
• Requires scientists’ actions, journals’ actions,
model intercomparison projects’ actions, and
technical supports.
32
A framework for achieving worldwide
bitwise identical reproducibility
33
Outline
• C-Coupler development
• C-Coupler1
• Bitwise identical reproducibility
• Future work
34
Future work
• Coupling generator
• Parallel optimization
• Testing bed with benchmarks
• ASCII configuration file format  XML format
• More coupling functions
35
Thank you
36
Example: Computation performance
Time for a data transfer (low resolution)
37
Example: Computation performance
Time for an interpolation (low resolution)
38