Copernicus Climate
Change Service
Climate Data Store, Toolbox
Geneva, 6-8th of December 2016
Cedric Bergeron
Angel Lopez Alos
Baudouin Raoult
[email protected]
[email protected]
[email protected]
Copernicus Climate
Change Service
COPERNICUS
Copernicus, previously known as GMES (Global Monitoring for Environment and Security),
USERS
Service
Component
Space
Component
In-situ Component
Source: copernicus.eu, retrieved April 2014
2
2
Full, free and open access to data
Budget of € 4.3 Bn for 2014-2020
is the European Programme for the establishment of a European capacity for Earth
Observation
Copernicus Climate
Change Service
COPERNICUS Space components
Sentinel-1 (A/B) – SAR imaging
2014
All weather, day/night applications, interferometry
Sentinel-2 (A/B) – Multi-spectral imaging
Land applications: urban, forest, agriculture,…
Continuity of Landsat, SPOT
Sentinel-3 (A/B) – Ocean and global land monitoring
Wide-swath ocean color, vegetation, sea/land surface
temperature, altimetry
Sentinel-4 (A/B) – Geostationary atmospheric
Atmospheric composition monitoring, trans-boundary pollution
Sentinel-5 precursor/ Sentinel-5 (A/B) – Low Earth-orbit
Atmospheric composition monitoring
Jason-CS (A/B) – Low inclination Altimetry
Sea-level, wave height and marine wind speed
Source: ESA
3
2020
Copernicus Climate
Change Service
COPERNICUS Service components
4
Copernicus Climate
Change Service
COPERNICUS Service components
Copernicus Services
*
*
*
*
* Operational & Delegation Agreement
5
JRC
Copernicus Services Deployment
Copernicus Climate
Change Service
Copernicus Services implementation Schedule
6
Copernicus Climate
Change Service
C3S
Vision
The European Commission has entrusted
the Copernicus Climate Change Service – C3S
ECMWF with the implementation of
•
To be an authoritative source of climate information for
Europe
•
Brings a unique pan-European dimension
to Climate Services
•
Build upon, complement and add value
to current capabilities in Europe
•
Provide a ‘one-stop-shop’ access to quality assured
climate information, tools and good practices
•
Facilitate uptake and growth of the climate service
market
•
Cross-cutting role and exploit synergies
with other Copernicus services
7
Copernicus Climate
Change Service
C3S
Components
CLIMATE
DATA STORE
Sectoral
Information
System
Evaluation and
Quality Control
• ECVs past, present
• Monitors quality of C3S
products and services
• Ensures C3S delivers
state-of-the-art climate
information to end-users
• Identifies gaps in service
provision
• Bridges Copernicus with
the research agenda in
Europe (e.g. H2020,
national research
projects)
and future
• Observed, reanalysed and
simulated
• Derived climate
indicators
• Tools to support
adaptation and
mitigation at global
and European level
8
Outreach and
Dissemination
• Web content
• Public outreach
• Coordination with
national outreach
• Liaison with public
authorities
• Conferences,
seminars
• Training and
education
Copernicus Climate
Change Service
The Climate Data Store, Toolbox
Concepts
9
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Context

The Climate Data Store will be at the heart of the C3S infrastructure and will
provide information about past, present and future climate in terms of
Essential Climate Variables and derived climate indicators

The CDS will be designed as a distributed system, providing improved
access to existing datasets through a unified web interface

The CDS will contain observations, global and regional climate reanalyses,
global and regional climate projections and seasonal forecasts

The CDS will also provide an authoritative set of software (toolbox) that will
allow the users to develop applications that will make use of the content of
the CDS

This service will accommodate the needs of the highly diverse set of users
that will include policy makers, experts as well as scientists
10
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Content
Scientific basis:
• Essential Climate Variables as defined
by GCOS
• GCOS Status Report (GCOS-195)
• IPCC, CMIP
Reprocessed
CDRs, reference
observations
Support for data
rescue, climate
data collections
Climate
reanalysis
Global
atmosphere,
ocean, land
Regional
reanalysis
Coupled climate
reanalysis for
100 years
Seasonal
forecasts and
climate
projections
Multi-model
seasonal
forecast products
Access to CMIP
data and
products
European multimodel climate
projections
11
Climate Indicators
Observations
Global estimates
of ECVs from
satellite and insitu data
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Content
Example of Climate datasets
expected to be accessible via the
Climate Data Store at an early
stage
12
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Content
Example of Climate datasets
expected to be accessible via the
Climate Data Store at an early
stage
13
Copernicus Climate
Change Service
Sectoral Information Systems
Overview
Proof Of Concept
Development of
sectoral applications
Copernicus
Climate
Change
Service
14
Copernicus Climate
Change Service
Sectoral Information Systems
Overview
Seven proof of concept SIS contracts have been awarded:

SIS water management:



SIS energy:



SWICCA (Service for Water Indicators in Climate Change adaptation) – lead SMHI (Sweden)
EDgE (End-to-End demonstrator for improved decision making in the water sector in Europe) –
Lead CEH (UK)
CLIM4ENERGY (Climate for Energy) – Lead CEA (France)
ECEM (European Climatic Energy Mixes) – Lead UEA (UK)
SIS others:



AgriCLASS (Agriculture Climate Advisory Services) – Lead Telespazio – Vega (UK)
WISC (Windstorm Information Service) – Lead CGI (UK)
URBAN-SIS (touching health, infrastructure, water) – Lead SMHI (Sweden)
15
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Toolbox Challenges
Technical challenges:
 Diversity of users
 Diversity of data sets
text
 Very large data volumes
 Data residing at different locations
 Interoperability, efficiency
 User-defined workflows
 Variety of presentation methods
 Need for interactivity
 Access via API
graphs, maps
 User management
 Performance monitoring
16
controls
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Concept
The Toolbox relies on the CDS Infrastructure
Cloud Hosting
17
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Web Portal

Content Management System (articles, news, events, …)

Catalogue of products
 ISO19115, for interoperability with INSPIRE, GEOSS, WMO Information System, …

Toolbox
 Catalogue of analytics tools that can be invoked on the data and products of the CDS
 Describes algorithms and methods, input and output data, possible parameterization

Users’ settings
 Profiles, preferences, licenses, …

Users’ requests
 Current and past users requests: data retrievals and computations
 Request management: monitor, cancel, download results
18
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Broker/Scheduler & Adaptors

Broker
 Manages all users requests (data retrievals and computations)
 Knows where is what
 Dispatches requests to the right location

Adaptors
 Map broker requests (data retrievals and computations) to data supplier specific
protocols
•
Quality of Services
 Needed to guarantee a good service when CDS will be used by many simultaneous
users
 Uses queues to throttle processing of requests so that available resources (CPUs,
disks, network) are not exhausted
 Implements limits for a fair use of the system (per user, per dataset, per type of
computation, etc) and a for a fine grained control of resource usage
 Implements priorities to schedule important requests (e.g. VIP users, interactive works)
before others (e.g. batch access, large computations)
19
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Data Suppliers

Data suppliers will be:

Distributed
 Data and products will mostly remain at the data provider’s location

Offering Datasets
 Reanalysis (Petabytes, GRIB)
 Climate projections (Petabytes, NetCDF)
 Seasonal forecast (Terabytes, GRIB/NetCDF)
 Gridded observation (Gigabytes to Terabytes …)
 Station observation (Gigabytes …)
20
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Toolbox

The Toolbox will be composed of:




Tools that perform basic operations on data, such as the computation of statistics, subsetting, averaging, value at points, etc.
Workflows that combine tools by chaining them so that the output of some tools is used as
input to others
Applications that make use of workflows and selected data and products of the CDS, to
build interactive web-pages allowing end-users to interact with the CDS
A Toolbox Compute layer:
 When possible tools will be executed next to the data (at the data suppliers)
 Otherwise, computations will be performed in a dedicated compute layer
 Use of cloud technologies will be considered
 Compute layer will also hold intermediate results
21
Copernicus Climate
Change Service
Climate Data Store / Toolbox
SOA and Orchestration

Service Oriented Architecture and Orchestration


Tools are available either at the data providers or in the compute layer
Tools will be invoked remotely by the broker as services (SOA), using adaptors when
necessary

Tools are combined into workflows

An orchestrator will manage the execution of workflows
 Parallelisation of execution whenever possible
 Minimisation of data transfers
22
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Concept
The CDS Toolbox, from data to applications
23
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Application Workflows
24
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Application Framework
25
Copernicus Climate
Change Service
The Climate Data Store, Toolbox
Implementation
26
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Architecture Overview
External Information
Systems
Data Supplier
End User
CDS Administrator
Browse
API
Browse
CDS EQC
Catalogue
Export
Data/Products Catalogue
Discover/View/
Download
Users Management*
Authentication
Tools/Workflow
Data/Product
Metadata Management*
Management*
Application Management*
Applications Use
*Creation Modification Deletion
TOOLBOX INFRASTRUCTURE
BACK END
Data/Products
Catalogue
User
Settings
User
Requests
Web Pages
Content Management
System
Tools/Workflow
Inventory
BROKER / Scheduler
Compute
Queues
Status
Adaptor
Applications
inventory
Monitoring / Metrics
Monitoring / Metrics
Data/Products
Catalogue
Iso19115 Export
CLIMATE DATA STORE
WEB PORTAL
Tools/Workflows
Repository
Adaptor
Data Elaborated
Data Elaborated
Adaptor
Computation
Computation
Tools Repository
Tools Repository
DATA
DATA
DATA
DATA
Site 1
Site 2
Site 3
Site 4
27
Staging
Computation Exec.
Results/Cache
Adaptor
DATA
Site n
DATA
DATA
DATA
SUPPLIERS
Adaptor
Computation Exec.
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Operations


Monitoring
Reporting
 Capacity planning
 Usage statistics



Service level agreement
On-call and support
Help desk


High-availability
Backup
28
Copernicus Climate
Change Service
Climate Data Store Infrastructure
Release Cycle
 Project Duration: 36 months
 Kick-Off: 06 Jul 2016
 One release every 6 months
29
KO
We Are
here
04 Mar 17
04 Oct 16
6 Sep 16
5 Sep 16
Development Timeline to release 1.0
30 Sep 17
Copernicus Climate
Change Service
Climate Data Store Toolbox
4 weeks
6 x 4 weeks
Sprint 0
Sprints 1-6
Initial Product
Backlog
6 x 4 weeks
Sprints 7-13
SC1
End
Roadmap
Release 0.1
Roadmap
Release 1.0
v0.1
Review
v1.0
Review
Development
Environment
Review Meetings
(at ECMWF)
Q1
Review
30
Copernicus Climate
Change Service
Climate Data Store / Toolbox
Development and environment tools
 Climate Data Store (CDS) Toolbox






Programming languages: Python and Javascript
Virtualisation technology: Docker / OpenStack (From infrastructure)
Scientific Python packages: NumPy, SciPy, Pandas
Climate community software tools: Metview, Magics, CDO, NCL, NCO,GDAL, ..
Web Framework: React, D3
Software quality: pycodestyle, pyflakes, mypy
31
Copernicus Climate
Change Service
Thank You
climate.copernicus.eu
32