Evaluation of FCM in IPSL Earth System model
A.Caubel (IPSL), with the help of D. Matthews (Met Office)
1. Context – IPSL Earth System model
Through IS-ENES2 project, IPSL has planned to evaluate FCM by moving from FCM V1 to FCM V2 and
to move from FCM-build to the more powerful FCM-make and to report back on the move. IPSL ESM
model includes different components of the climate system (the atmosphere, ocean, cryosphere and
biosphere) to represent both physical and biogeochemical processes. LMDZ is the atmospheric
component and NEMO is the oceanic component. IPSL ESM run currently on two supercomputers :
Curie (TGCC) and Ada (IDRIS). IPSL constraints in term of configuration management are related to :
-
Different components, developped by differents groups
Different configurations (one or more components, coupled or forced)
Different computing centers (different hardware, software, ...)
Users with different backgrounds, different computer literacy level, ...
For all of these reasons, IPSL need a flexible, portable, robust and easy to use tool.
Fig 1. IPSL Earth System model
IPSLCM5 was the previous version of IPSL-CM which was used for CMIP5 intercomparaison project in
2010. IPSLCM6 is the current IPSL Earth System model which is under development : IPSLCM6 will be
used for CMIP6 intercomparison project.
2. Integration of FCM-Make in IPSL Earth System Model
a. FCM in IPSLCM5 Earth System model
FCM is used as compilation tool (build step) in IPSL Earth System model since IPSLCM5 model
version. FCM has been first integrated in LMDZ atmospheric model in 2010, then in oceanic
component NEMO, land surface component ORCHIDEE and atmospheric chemistry and aerosols
component INCA in 2013. The XIOS Input/output library, which is used in IPSL components, uses FCM
as well. FCM sources are extracted within component sources or installed on computing machine.
Model
Version
Path
Launching method
LMDZ
1.2
own binary
through ksh script
INCA
1.2
locally installed
through ksh script
NEMO
1.5
own binary
through ksh script
XIOS
1.5
own binary
through ksh script
Fig 2. FCM version in IPSLCM5 components
b. Integration of FCM Make in IPSLCM6 Earth System model
Two steps are needed to integrate FCM make in IPSLCM6 model. First, we switch from FCM V1 used
in IPSLCM5 (cf fig 2) to FCM v2. Then, we use make functionality (extract, preprocess, build) instead
of only build step, as it was done in IPSLCM5. This work was performed for atmospheric component
LMDZ. Thanks to the work done at Metoffice for NEMO component, an IPSL Earth model (atm-oce
coupled model) version uses FCM make.
i.
LMDZ atmospheric component
LMDZ compilation needs the creation of « include file » depending on the model resolution. This step
must be done between extraction step and preprocess/build step.
fcm make -n extract
extract/lmdz/grid/dimension/makdim 96 95 39
fcm make -n build
Fig3. Extraction and compilation script for LMDZ component
The use of “makedim” command was the current way used by IPSL to compile LMDZ model. A work
around to avoid the use of “makdim” command between extract and build steps would be either to
use CPP flags or to use the “makedim” command via the preprocessing step (as suggested by Dave
Matthews from FCM team).
Two main configuration files have been created : fcm-makeextract.cfg and fcm-makebuild.cfg. These
files are configuration files of the chain of steps.
-
fcm-makeextract.cfg
steps = extract
extract.ns = lmdz
extract.location[lmdz] = http://svn.lmd.jussieu.fr/LMDZ/LMDZ5/trunk@2292
extract.path-root[lmdz] = libf
extract.path-excl[lmdz] = dyn3d phydev phymar dyn3dpar phylmd/dyn1d phylmd/rrtm phylmd/sisvat
dynlonlat_phylonlat/phymar dynlonlat_phylonlat/phydev dynlonlat_phylonlat/phylmd/ce0l.F90
dyn3dmem/ce0l.F90
extract.path-incl[lmdz] = dyn3dmem dyn3d_common filtrez phylmd dynlonlat_phylonlat grid misc
Fig4. fcm-makeextract.cfg
-
fcm-makebuild.cfg
include = /ccc/scratch/cont003/dsm/p86caub/LMDZ_FCM_MAKE/fcm-make/curie.cfg
include = /ccc/scratch/cont003/dsm/p86caub/LMDZ_FCM_MAKE/fcm-make/inc/common.cfg
Fig5. fcm-makebuild.cfg
with options related to the machine in curie.cfg (Fig 6) and options related to the model in
common.cfg (Fig 7).
$netcdf_inc_path{?} = ${NETCDF_INC_DIR}
$netcdf_lib_path{?} = ${NETCDF_LIB_DIR}
$netcdf_lib{?} = netcdff netcdf hdf5 hdf5_hl hdf5_fortran z
$xios_path{?} = /ccc/scratch/cont003/dsm/p86caub/XIOS_FCM_V2/xios-1.0
$xios_inc_path{?} = $xios_path/inc
$xios_lib_path{?} = $xios_path/lib
$xios_lib{?} = stdc++ xios
$ioipsl_path{?} = /ccc/work/cont003/dsm/p86caub/IPSLCM6_rc1/modipsl
$ioipsl_inc_path{?} = $ioipsl_path/lib
$ioipsl_lib_path{?} = $ioipsl_path/lib
$ioipsl_lib{?}
= ioipsl
$mkl_path{?} = /usr/local/intel-14.0.3.174/14.0.3.174/mkl
$mkl_inc_path{?} = $MKL_INCDIR
$mkl_lib_path{?} = $MKL_LIBDIR
$mkl_lib{?}
= mkl_intel_lp64 mkl_sequential mkl_core m
preprocess.prop{fpp.flags}
= -P -C -traditional
build.prop{class,fc}
= mpif90 -c -cpp
build.prop{class,fc.flags}
= -i4 -r8 -auto -align all -O2 -fp-model strict -openmp
build.prop{class,fc.flags-ld} = -openmp
build.prop{class,fc.include-paths} = $netcdf_inc_path $xios_inc_path $ioipsl_inc_path $mkl_inc_path
$mkl_inc_path/intel64/lp64
build.prop{class,fc.libs}
= $netcdf_lib $xios_lib $ioipsl_lib $mkl_lib
build.prop{class,fc.lib-paths} = $netcdf_lib_path $xios_lib_path $ioipsl_lib_path $mkl_lib_path
build.prop{class,ld}
= mpif90
Fig6. curie.cfg(LMDZ component)
steps = preprocess build
preprocess.source = extract
preprocess.prop{fpp.defs} = CPP_PHYS CPP_EARTH CPP_PARA CPP_MPI CPP_OMP CPP_IOIPSL
CPP_XIOS CPP_IOIPSL_NO_OUTPUT
preprocess.prop{class,no-dep.include} = netcdf.inc
preprocess.prop{class,fpp.include-paths} = ${NETCDF_INC_DIR}
build.source = extract
build.prop{class,no-dep.f.module} = netcdf xios typesizes ioipsl iaxis iaxis_attr icontext_attr idate
idomain_attr ifield_attr ifile_attr ixml_tree
build.prop{class,no-dep.include} = mpif.h netcdf.inc
build.prop{ns-dep.o}[lmdz/dyn3dmem/gcm.F90] = lmdz
build.target
= gcm.exe
Fig7. common.cfg(LMDZ component)
ii.
NEMO oceanic component
The use of “fcm make” was integrated in NEMO by D. Matthews. The work done here consisted in
using Dave’s work in IPSL Earth System model. The installation on curie and ada supercomputer was
easy, thanks to the work done by D. Matthews to split options related to supercomputers into
include files.
iii.
Installation on two computing machines
IPSL Earth System model is currently used on two computing centres (TGCC Curie and IDRIS Ada).
Once the split was done between machine options and model options (cf Fig 5), the installation of
the system from Curie to Ada supercomputer is mainly to create a configuration file ada.cfg (cf Fig 8),
both for LMDZ and NEMO component.
$netcdf_inc_path{?} = /smplocal/pub/NetCDF/4.1.3/mpi/include
$netcdf_lib_path{?} = /smplocal/pub/NetCDF/4.1.3/mpi/lib /smplocal/pub/HDF5/1.8.9/par/lib
$netcdf_lib{?} = netcdff netcdf hdf5 hdf5_hl hdf5_fortran z
$xios_path{?} = /workgpfs/rech/ces/rces061/IPSLCM6-VLR_rc0/modipsl/modeles/XIOS_force
$xios_inc_path{?} = $xios_path/inc
$xios_lib_path{?} = $xios_path/lib
$xios_lib{?} = stdc++ xios
$ioipsl_path{?} = /workgpfs/rech/ces/rces061/IPSLCM6-VLR_rc0/modipsl
$ioipsl_inc_path{?} = $ioipsl_path/lib
$ioipsl_lib_path{?} = $ioipsl_path/lib
$ioipsl_lib{?}
= ioipsl
$fftw_inc_path{?} = /smplocal/pub/FFTW/3.3.3_dyn/include
$fftw_lib_path{?} = /smplocal/pub/FFTW/3.3.3_dyn/lib
$fftw_lib{?}
= fftw3 mkl_intel_lp64 mkl_sequential mkl_core m
preprocess.prop{fpp.flags}
= -P -C -traditional
build.prop{class,fc}
= mpiifort -c -cpp
build.prop{class,fc.flags}
= -i4 -r8 -auto -align all -O2 -fp-model strict
build.prop{class,fc.include-paths} = $netcdf_inc_path $xios_inc_path $ioipsl_inc_path $fftw_inc_path
build.prop{class,fc.libs}
= $netcdf_lib $xios_lib $ioipsl_lib $fftw_lib
build.prop{class,fc.lib-paths} = $netcdf_lib_path $xios_lib_path $ioipsl_lib_path $fftw_lib_path
build.prop{class,ld}
= mpiifort
Fig8. ada.cfg (LMDZ component)
Model
Version
LMDZ
FCM 2015.09.0
NEMO
FCM 2015.09.0
Fig 9. FCM version in IPSLCM6 components (working version)
iv.
Difficulties encountered
-
Installation of FCM V2 on ada supercomputer(IDRIS): it is needed to ask IDRIS assistance to
install some missing modules. Error message : « Can't locate Time/Piece.pm”, “Can't locate
Time/HiRes.pm”,…On the other hand, there was no problem to switch from fcm v1 to fcm v2
on curie supercomputer.
-
Dependencies with LMDZ Fortran 77 routines : for Fortran code, dependencies on include
files and modules are handled automatically. For modern code this should be sufficient.
However, for legacy F77 codes which just call routines directly with no use of modules or
interface files (that’s the case with LMDZ model) there is no way to reliably detect these
dependencies. These dependencies need to be specified in the config file as follows :
build.prop{ns-dep.o}[lmdz/dyn3dmem/gcm.F90] = lmdz
This trick makes the main program dependent on all the available targets in the source tree
so everything would be compiled before the link is attempted.
-
export vs checkout : At IPSL, we are used to work with “svn checkout” and so, the location of
extracted files and working files is the same. The extract step of FCM performs “svn export”
which means users are not able to modify the files created by the extract step. Dave
Matthews (from FCM team) explained that if users want to do a make using files which they
have modified in a working copy then they should modify the extract.location to refer to
their working copy rather than a subversion URL.
3. Evaluation
a. Functionality
i.
General functionalities
-
“Include” functionality : very useful (especially in IPSL context) to separate specific options
depending on :
o Configurations
o Models
o Machines/computing centres
The summary in log file is nice to check that what we do fits with what we want to do.
[info] config-file=/ccc/scratch/cont003/dsm/p86caub/NEMO_FCM_MAKE/fcm-make.cfg
[info] config-file= - /ccc/scratch/cont003/dsm/p86caub/NEMO_FCM_MAKE/fcm-make/curie.cfg
[info] config-file= - - /ccc/scratch/cont003/dsm/p86caub/NEMO_FCM_MAKE/fcmmake/inc/common.cfg
[info] config-file= - - - /ccc/scratch/cont003/dsm/p86caub/NEMO_FCM_MAKE/fcm-make/inc/keysorca2_lim_pisces.cfg
Fig9. Tree of include files fir NEMO component
-
verbose functionality (-vv) : easy to increase verbose level.
-
information - log files
o “on-success” log file : very convenient to repeat last successful command especially
in test or development mode.
o log : useful to have result of the command : always a problem with compilation with
classic interactive gmake. Time information spent is nice.
o time information spent for each command is very useful.
ii.
Extract functionality
This “extract” functionality is a new functionality for IPSL ESM (only build step was used before).
IPSL component sources to be extracted are located on local svn servers.
-
easy to extract part of whole sources/model (extract.path-excl, extract.path-incl).
extract.path-excl[lmdz] = dyn3d phydev phymar dyn3dpar phylmd/dyn1d phylmd/rrtm phylmd/sisvat
dynlonlat_phylonlat/phymar dynlonlat_phylonlat/phydev dynlonlat_phylonlat/phylmd/ce0l.F90
dyn3dmem/ce0l.F90
extract.path-incl[lmdz] = dyn3dmem dyn3d_common filtrez phylmd dynlonlat_phylonlat grid misc
-
the whole model is extracted in “.fcm-makeextract/cache” and sources are then duplicated :
that could be a problem to have too many files because of inode quotas on supercomputers.
Dave Matthews (from FCM Team) informed me that they introduced a new “—archive”
option to help this. Besides, they have also made the make directories re-locatable. This
allows to run the fcm make (using the --archive option) in a temporary directory (fast storage
suitable for small file I/O) and then move the make directory to a permanent location on
completion.
find .fcm-makeextract -type f -print | wc -l
1863
find extract -type f -print | wc -l
640
-
combining changes from multiple branches of a project : could be interesting. Not tested.
-
--jobs=N : no significative impact on the extract step because only one single source is used
in this case. But this option could help where there are multiple sources (ESM context) :
either because code is extracted from several different repositories or because changes are
combined from multiple branches.
/ccc/cont003/home/dsm/p86caub/FCM_2.0/fcm-2015.05.0/bin/fcm make -vv -j 8 -n extract 2>&1 |
tee output_extract : 53s
/ccc/cont003/home/dsm/p86caub/FCM_2.0/fcm-2015.05.0/bin/fcm make -vv -n extract 2>&1 | tee
output_extract : 51s
iii.
Build functionality
This functionality was already used in previous version of IPSL ESM (IPSLCM5) and so it is not a new
functionality. However, we still appreciate the reliable dependency analysis.
-
--jobs=N : significative impact on the build step
--jobs=8 : [done] make build
# 407.7s
--jobs=1 : [done] make build
# 1249.8s
-
time information (as default information) spent for each command is very useful.
- inheritance : could be interesting, especially in ESM context (with many models,…). Not
tested.
b. Documentation - support
The documentation helps a lot from a “beginner” to “expert” user level : it gives a detailed overview
of both available functionalities and functionalities user’s guide. Some practical problems related to
integration on a model need to ask support to FCM people. That’s what I did to handle fortran 77
dependencies (cf “difficulties encountered”) and the support was fast and precise (special thanks to
D. Matthews).
4. Conclusions – Perspectives - Recommendations
FCM is a powerful tool with many functionalities/options and we haven’t tested all of them. As it is
written in this report, FCM-make is a useful, reliable and fast tool which allows to extract and
compile configurations and models. Support and documentation to FCM users are efficient.
The work described above was useful for IPSL configuration management working group : a working
revision of IPSL Earth System has been created. This version of IPSL ESM is a working version and will
not be used for next CMIP6 simulations. It remains both an integration work in each component
reference version and a synchronization work between every component to make these
developments available in IPSL ESM coupled model.
Extraction and compilation aspects are not a priority at IPSL at the moment. A work on these aspects
for IPSL components has a cost which is not negligible because it is needed to do it for all of IPSL
components (coherence for ESM model) : that would need a person focused on these aspects and we
do not have human resources to do such a work. IPSL is quite happy with the use of previous version
of FCM (build step) and the direct use of Subversion (extract step). Consequently, for IPSL there is no
plan to switch to new FCM functionalities in reference versions right now but we will advance step by
step to increase usability and reliability of our configuration management system: regarding this
evaluation, FCM seems to be the ideal tool to do that. On the other hand, we strongly recommend
the use of FCM to modeling groups who are looking for an easy to use and efficient configuration
management tool.