End-to-End Workflows for Coupled
Climate and Hydrological Modeling
Kathy Saint (SGI) and Sylvia Murphy (NOAA/CIRES)
iEMSs, Ottawa Canada
July 5-8, 2010
Outline
• Project Objective
• Motivation
• System Description
– Components
– Frameworks
– System Driver
•
•
•
•
Logical Workflow
Data Flow
Challenges
Future Directions
Project Objective
The development of an end-to-end workflow that executes, in
a loosely coupled mode, a distributed modeling system
comprised of an atmospheric climate model using ESMF and a
hydrological model using OpenMI
Motivation
• Hydrological impact studies can be improved when forced
with data from climate models [Zeng et al., 2003; Yong et
al., 2009]
• A technology gap exists:
– Many hydrological models run on personal computers
– Most climate models run on high performance supercomputers
• The leveraging of ESMF and OpenMI can mitigate the
communication difficulties between these modeling types
– ESMF contains web services interfaces that can be used to
communicate across a distributed network
– Both ESMF and OpenMI are widely used within their respective
communities
System Description
• SWAT (hydrology model) runs on PC
Personal Computer
• CAM (climate model) runs on HPC
Driver
• Wrappers for both SWAT and CAM
provide OpenMI interface to each model
• Driver (OpenMI Configuration Editor)
uses OpenMI interface to timestep
through models via wrappers
• Access to CAM across the
network provided by ESMF
Web Services
• CAM output data written to
NetCDF files and streamed
to CAM wrapper via ESMF
Web Services
• Resulting output files archived
to science gateway
OpenMI
SWAT
CAM OpenMI
Wrapper
ESMF Web
Services
ESMF CAM
Component
High Performance Computer
Data Files
Components: SWAT
• The hydrological model
chosen for this project is the
Soil Water Assessment Tool
(SWAT)
• It is a river basin scale model
developed to quantify the
impact of land management
practices in large, complex
watersheds
• It was chosen for this project
because it is widely used, is
open source, and runs on a
Windows platform
Components: CAM
• The atmospheric model chosen for this system is the
Community Atmospheric Model (CAM4), part of the
Community Climate System Model (CCSM4)
• It was chosen because:
• Has ESMF Component
Interfaces
• Our group has an ongoing
collaboration with CCSM
• It is Open Source
Frameworks: Earth System
Modeling Framework
• Is a high-performance, flexible software infrastructure that
increases the ease of use, performance portability,
interoperability, and reuse of Earth science applications
• Provides an architecture for composing complex, coupled
modeling systems and includes array-based, multidimensional data structures
• Has utilities for developing individual models including
utilities to make models self-describing
• Web services are being developed that allow any
networked ESMF component to be available as a web
service. (Target merge into ESMF distribution in Sept.
2010.)
Frameworks: OpenMI
• The OpenMI Software Development Kit (SDK) is a software
library that provides a standardized interface that focuses
on time dependent data transfer
• Primarily designed to work with systems that run
simultaneously, but in a single-threaded environment
[Gregerson et al., 2007]
• The primary data structure in OpenMI is the ExchangeItem,
which comes in the form of an InputExchangeItem and an
OutputExchangeItem (single point, single timestep)
The system driver
• Controls the application flow
• Implemented using OpenMI’s Configuration Editor
• Convenient tool for the testing of the OpenMI
implementations and model interactions
• The OpenMI Configuration Editor will eventually be
replaced to handle more complex workflows
Logical Workflow
Driver
SWAT/OpenMI
CAM/OpenMI Wrapper
ESMF Web Services
ESMF Component
Initialize
Initialize
NewClient
Prepare
Prepare
GetValues
GetValues
Initialize
ESMF_GridCompInitialize
Run
ESMF_GridCompRun
Finalize
ESMF_GridCompFinalize
GetDataValues
Finish
Finish
Dispose
Dispose
EndClient
Logical Workflow
Driver
SWAT/OpenMI
CAM/OpenMI Wrapper
ESMF Web Services
ESMF Component
Initialize
Initialize
NewClient
Prepare
Prepare
GetValues
GetValues
Initialize
ESMF_GridCompInitialize
Run
ESMF_GridCompRun
Finalize
ESMF_GridCompFinalize
GetDataValues
Finish
Finish
Dispose
Dispose
EndClient
Logical Workflow
Driver
SWAT/OpenMI
CAM/OpenMI Wrapper
ESMF Web Services
ESMF Component
Initialize
Initialize
NewClient
Prepare
Prepare
GetValues
GetValues
Initialize
ESMF_GridCompInitialize
Run
ESMF_GridCompRun
Finalize
ESMF_GridCompFinalize
GetDataValues
Finish
Finish
Dispose
Dispose
EndClient
Logical Workflow
Driver
SWAT/OpenMI
CAM/OpenMI Wrapper
ESMF Web Services
ESMF Component
Initialize
Initialize
NewClient
Prepare
Prepare
GetValues
GetValues
Initialize
ESMF_GridCompInitialize
Run
ESMF_GridCompRun
Finalize
ESMF_GridCompFinalize
GetDataValues
Finish
Finish
Dispose
Dispose
EndClient
Logical Workflow
Driver
SWAT/OpenMI
CAM/OpenMI Wrapper
ESMF Web Services
ESMF Component
Initialize
Initialize
NewClient
Prepare
Prepare
GetValues
GetValues
Initialize
ESMF_GridCompInitialize
Run
ESMF_GridCompRun
Finalize
ESMF_GridCompFinalize
GetDataValues
Finish
Finish
Dispose
Dispose
EndClient
Current Data Flow
High Performance
Computer
ESMF
Component/CAM
Personal Computer
GetValues
GetDataValues
CAM/OpenMI
Wrapper
SWAT/OpenMI
Input
Exchange Item
Output
Exchange Item
NetCDF File
The data exchange is initiated by the OpenMI GetValues call, and the data,
retrieved from a NetCDF file as multi-dimensional array, is indexed and
streamed to the CAM Wrapper as a single point, single timestep value
Target Data Flow
High Performance
Computer
ESMF
Component/CAM
ESMF State
Personal Computer
GetValues
GetDataValues
CAM/OpenMI
Wrapper
SWAT/OpenMI
Input
Exchange Item
Output
Exchange Item
The goal is to bypass the NetCDF file, and instead, retrieve the data
directly from an ESMF State object
Model Configurations
•
CAM
–
–
–
–
–
–
Global Atmospheric Model
Model run: 1 day
Timestep: 1800 sec
Dynamical Core: finite volume
Horizontal Grid: 10x15
Export data variables:
• surface air temperature
• precipitation
• wind speed
• relative humidity
• solar radiation
•
SWAT
– Hydrology science information
provided by Jon Goodall of
University of S. Carolina
– Lake Fork Watershed (TX)
– Watershed Area: 486.830 km2
– Model run: 2 years, 1977 – 1978
– Timestep = 1 day
– Weather Stations:
• wea62 (33.03 N, 95.92 W)
• wea43 (33.25 N, 95.78 W)
Challenges
• Bridging the Gaps – dealing with the complexities of
interoperating between 2 diverse models
– Single threading vs. Multi-threading (MPI)
– Desktop platforms vs. High Performance Computing (HPC)
platforms
– Windows vs. Linux/Unix
– Multiple programming languages throughout the system
•
•
•
•
•
CAM and SWAT Models in Fortran
OpenMI in C#
ESMF in Fortran and C++
Web Services in Java, C++, and Fortran
NetCDF interface in J#
Remaining Tasks
• Replace the OpenMI Configuration Editor for more complex
workflows
• Modify CAM to stream output instead of writing to a file
• Automatically archive data from both CAM and SWAT to a
science gateway
• Time stepping through CAM instead of pre-running during
initialization
• 2-way coupling (long-term)
Questions
Earth System Curator: http://earthsystemcurator.org
ESMF: http://earthsystemmodeling.org
OpenMI: http://www.openmi.org
CAM: http://www.ccsm.ucar.edu/models/ccsm4.0/cam
SWAT: http://swatmodel.tamu.edu
[email protected]
[email protected]