Sentinel 3 Marine Copernicus Data Access
User Manual
https://coda.eumetsat.int
Sentinel 3 Marine Copernicus Data Acces
Doc.No.
Issue
Date
WBS/DBS
:
:
:
:
EUM/OPS-SEN3/MAN/16/880763
v2 Draft
24 January 2017
EUMETSAT
Eumetsat-Allee 1, D-64295 Darmstadt, Germany
Tel: +49 6151 807-7
Fax: +49 6151 807 555
http://www.eumetsat.int
© EUMETSAT
The copyright of this document is the property of EUMETSAT.
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Table of Contents
1
INTRODUCTION .............................................................................................................................. 3
1.1 File naming convention .......................................................................................................... 3
1.2 Acronyms scheme .................................................................................................................. 5
2
GRAPHICAL USER INTERFACE ................................................................................................... 6
2.1 Search Panel .......................................................................................................................... 6
2.2 Map tool.................................................................................................................................. 7
2.3 Search result .......................................................................................................................... 7
2.4 User Profile Panel ................................................................................................................ 10
2.5 Batch products download from CODA GUI .......................................................................... 11
2.6 APIs and Batch Scripting ..................................................................................................... 13
3
OPEN DATA PROTOCOL (ODATA) ............................................................................................. 14
3.1 URI Components .................................................................................................................. 14
3.2 Responses ........................................................................................................................... 15
3.3 XML Responses ................................................................................................................... 15
3.4 Service Metadata Document ................................................................................................ 15
3.5 Using OData to discover products in the CODA Web Service archive ................................ 16
3.5.1 Querying the content of the products archive: usage of the /Products resource path ... 16
3.5.2 Querying a single Product in the archive ....................................................................... 17
3.5.3 Querying the products (paging) ..................................................................................... 18
3.5.4 Using OData to filter the products in the archive ........................................................... 18
3.5.4.1 Filtering the Products on time-based criteria............................................................ 19
3.5.4.2 Filtering the products by Ingestion Date ................................................................... 19
3.5.4.3 Filtering the Products using the file name ................................................................ 19
3.5.4.4 Filtering the Products by Sensing Time (file name) ................................................. 20
3.5.5 Querying Product Attributes and Metadata Contents .................................................... 20
3.5.6 Get product UUID from product name ........................................................................... 20
3.6 Download ............................................................................................................................. 21
3.6.1 Download full product from its UUID .............................................................................. 21
3.6.2 Discover Product Nodes ................................................................................................ 21
3.6.3 Download manifest file from the UUID ........................................................................... 22
3.6.4 Download of Sentinel-1 quick-look file knowing the product UUID ................................ 22
3.6.5 Verifying Download integrity using the MD5 checksum ................................................. 22
4
OPEN SEARCH ............................................................................................................................. 24
4.1 URI Components .................................................................................................................. 24
4.2 Using Open Search to discover products in the CODA Web Service archive ..................... 24
4.2.1 Discover the list of the products stored in the archive ................................................... 24
4.2.2
Paging results ................................................................................................................... 24
4.2.3
Sorting results .................................................................................................................. 25
4.3 Discover the products over a predefined Area Of Interest (AOI): Geographical Search ..... 25
5
BATCH SCRIPTING ...................................................................................................................... 27
5.1 Download via Curl ................................................................................................................ 27
5.2 Download via Wget .............................................................................................................. 27
5.3 Scripts Examples ................................................................................................................. 28
5.3.1 dhusget script ................................................................................................................. 28
5.3.1.1 USAGE: .................................................................................................................... 28
5.3.2 Odata-demo.sh script ..................................................................................................... 29
5.3.2.1 USAGE: .................................................................................................................... 31
6
KNOWN ISSUES AND USEFUL TIPS FOR CODA WEB SERVICE ........................................... 32
Page 2 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
1
INTRODUCTION
The Sentinel-3 Marine CODA (Copernicus Online Data Access) Web Service provides
complete, free and open access to Sentinel-3 user products provided by EUMETSAT on behalf
of the European Commission.
The Web Service consists of a rolling archive with http access and 30 days of data online. It
provides Level-1 and Level-2 user products for the following instruments:
•
OLCI (Ocean and Land Colour Instrument),
•
SLSTR (Sea and Land Surface Temperature Radiometer),
•
SRAL (SAR Altimeter).
Data are provided in different modes:
- Near-Real-Time (NRT): products shall be available to the users within three hours after
sensing.
- Short Time Critical (STC): products available to the users within 48 hours after sensing.
- Non-Time-Critical (NTC): products available to the users within one month after
sensing.
1.1
File naming convention
The file naming convention for CODA products is identified by the sequence of fields described
here:
SA3_SS_L_TTTTTT_yyyymmddThhmmss_YYYYMMDDTHHMMSS_YYYYMMDDTHHM
MSS__GGG_.SEN3
SS
Data source of the instrument
L
Processing level
TTTTTT
Data Type ID
-
-
-
-
OL = OLCI
SL = SLSTR
SR = SRAL
"1" for Level-1
"2" for Level-2
Level 1 OLCI data
- “EFR___” = TOA radiances at full resolution
- “ERR___” = TOA radiances at reduced
resolution
Level 2 OLCI data
- “WFR___” = Full Res. Ocean Colour, Water
& atmosphere parameters
- “WRR___” = Reduced Res. Ocean Colour,
Water & atmosphere parameters
Level 1 SLSTR data
- “RBT___” = TOA Radiances and Brightness
Temperature
Level 2 SLSTR data
- “WST___” = L2P Sea Surface Temperature
Level 1 SRAL data
- “SRA___” = echo LRM, SAR Ku, SAR C
Level 2 SRAL data
- “WAT___”= Water products
Page 3 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
yyyymmddTh
hmmss
YYYYMMD
DTHHMMSS
YYYYMMD
DTHHMMSS
<instance_id>
GGG
<class_id>
Data Start time (Sensing time year, month, day, T, hour, minutes, seconds
for the Instrument data products)
Data Stop time (Sensing time year, month, day, T, hour, minutes, seconds
for the Instrument data products)
Creation Date
year, month, day, T, hour, minutes, seconds
Instance ID for the instrument - Products disseminated in "stripes"
data products disseminated in Duration,"_", cycle number, "_", relative orbit
"stripes"
number,"_",
4
underscores
"_"
DDDD_CCC_LLL_____
Instance ID for the instrument - Products disseminated in " frames "
data
products
Duration, "_", cycle number, "_", relative orbit
disseminated in "frames"
number, "_", frame along track coordinate
Product Generating Centre
- LN1 = Land OLCI Processing and Archiving
Centre
- LN2 = Land SLSTR and SYN Processing and
Archiving Centre
- LN3 = Land Surface Topography Mission
Processing and Archiving Centre MAR = Marine
Processing and Archiving Centre SVL = Svalbard
Satellite Core Ground Station
identifies the class ID for - P indicates the platform (O for operational, F for
instrument data products with
reference, D for development, R for reprocessing)
conventional sequence
- XX indicates the timeliness of the processing
workflow (NR for NRT, ST for STC, NT for
NTC)
- NNN indicates the baseline collection or data
usage.
Page 4 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
1.2
Acronyms scheme
Please, note that Sentinel-3 products are provided according to the following acronyms scheme:
PRODUCT
TYPE
LINK TO EUMETSAT
PRODUCT NAVIGATOR
OLCI (Ocean and Land Colour Instrument)
- OLCI Level 1B Full
Full resolution top of
Resolution in NTC
OL_1_EFR___
Level 1
atmosphere radiance.
- OLCI Level 1B Full
Resolution in NRT
- OLCI Level 1B Reduced
Reduced resolution top of
Resolution in NRT
OL_1_ERR___
Level 1
atmosphere radiance.
- OLCI Level 1B Reduced
Resolution in NTC
- OLCI Ocean Colour Full
Full resolution water &
Resolution in NRT
OL_2_WFR___
Level 1
atmosphere geophysical
- OLCI Ocean Colour Full
products.
Resolution in NTC
- OLCI Ocean Colour Reduced
Reduced resolution water &
Resolution in NRT
OL_2_WRR___
Level 2
atmosphere geophysical
- OLCI Ocean Colour Reduced
products.
Resolution in NTC
SLSTR (Sea and Land Surface Temperature Radiometer)
- SLSTR Level 1B Radiances
and Brightness Temperatures
Brightness temperature and
in NRT
SL_1_RBT___
Level 1
radiances.
- SLSTR Level 1B Radiances
and Brightness Temperatures
in NTC
- SLSTR Sea Surface
Level 2P Sea Surface
Temperatures (SST) in NRT
SL_2_WST___
Level 2
Temperature (GHRSST like) - SLSTR Sea Surface
Temperatures (SST) in NTC
SRAL (SAR Radar Altimeter)
Echos parameters for LRM,
- SRAL Level 1B in NRT
SR_1_SRA___
Level 1
PLRM and SAR mode
- SRAL Level 1B in NTC
(resolution 20Hz).
- SRAL Level 1B in STC
- SRAL Altimetry Global in
1-Hz and 20-Hz Ku and C
NRT
bands parameters
- SRAL Altimetry Global in
SR_2_WAT___
Level 2
(LRM/SAR/PLRM),
NTC
waveforms. Over Water.
- SRAL Altimetry Global in
STC
LEVEL
DESCRIPTION
Page 5 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
2
GRAPHICAL USER INTERFACE
To access EUMETSAT CODA GUI go to the link https://coda.eumetsat.int.
This is how CODA GUI appears:
User Profile
Full text Search bar
Save Search
Draw Region of
Interest / Navigate
Map
Advanced Search bar
Select base map
Map Tool
About
Figure 1: CODA Graphical User Interface
In order to use CODA GUII functionalities it is necessary to have a CODA user account.
If you have already your CODA account, click on LOGIN button and compile the form with
username and password. If you have not a CODA account, click on the SIGN UP button and
follow the instructions to create a new account.
2.1
Search Panel
Figure 2: CODA Search Panel
The Search Panel provide the user with two different search functionalities:
 Full text search: the user has just to write the keywords in the field.
 Advanced Search: the user can apply different filter parameters to the search and order
the results.
Page 6 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Figure 3: CODA Advanced Search Panel
It is also possible to Sort the elements of the result list using different parameters (Ingestion
Date, Sensing Date, Cloud Cover) and then order the sorted list by ascending or descending.
2.2
Map tool
The map tool provides a geographical search feature. To define a region of interest click on the
icon to set "draw region of interest" modality and then click and drag on the map for drawing a
selection box. By clicking on the "Search" button
the application will search for any product
whose footprint intersects the selected region.
Thanks to this geographic filter, the map tool can be combined with the full text search and the
advanced search to further restrict the results.
2.3
Search result
The Search results list provides all the products matching the submitted search query. Each
result consists of:
Page 7 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual




64x64 thumbnail (when available),
product name,
product URI,
mission name, the instrument name, the sensing date, the file size.
The footprints of the products matching the search query are displayed on the map. Each
footprint gets highlighted when hovering on the corresponding product in the search results list.
Red footprints show Sentinel-3 SLSTR products, blue footprints show Sentinel-3 OLCI products
and purple footprints show the SRAL products.
NOTE: the map tool shows only the footprints of the products in the current search list page. To
widen results you can change the page size (25, 50, 75, 100, 125, and 150).
Figure 4: Search results footprints
In case of searches that are recurrently performed, it is possible to save search’s parameters
clicking on the
icon in order to save the search in the user profile and use them again the next
times.
For every product in the results list, the following options are displayed (
 Select product,
 Zoom to product,
 View product details,
 Add product to Cart,
 Download product.
Page 8 of 33
):
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Product Options
Figure 5: Search results – Product Options
When clicking on "View Product Details" icon, a window is open showing the product preview
with the following elements:
 Product name
 Product URL
 Footprint
 Quicklook
 Attributes(Summary, Product, Platform, Instrument)
 Inspection(Annotation, Measurement,Preview, Support, Manifest.safe)
From this window, it is possible to download the entire zipped file but also the single elements
of the zipped file, from the Inspector section.
Page 9 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Figure 6: Search results – Product Options
2.4
User Profile Panel
It is possible to access the User Profile Panel clicking on the ‘User Profile’ icon
right of the web page.
on the top
Figure 7: User Profile settings
From the User Profile Panel it is possible to check the allowed functions (in the case of the
Figure 5: Search function and Download function), to check the Cart and to check the Saved
Searches.
This is how User Cart appears:
Page 10 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Figure 8: CODA User Cart
It is possible to display the entire list of products that the user added to cart to be downloaded.
For every product, it is possible to check product details, remove it from the list or download the
file.
2.5
Batch products download from CODA GUI
Once the user has chosen the products to download and added them to his profile cart, it is
possible to perform a batch download of all the products at once.
One possible way to perform a batch load is a follows (Please, note that to do this, it is necessary
to use an updated version of Mozilla Firefox at least 49.02 ver.):
1) Install
Firefox
Plugin
DownThemAll!
(https://addons.mozilla.org/enUS/firefox/addon/downthemall/).
2) Go to your User Cart page and click on the bottom right button DOWNLOAD CART.
This will automatically open a DownThemAll! plugin popup that will display the list of
products to download, the folder to save the files in and the mask to use to rename the
files.
Page 11 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Figure 9: How to batch download files with DownThemAll! Firefox plugin
3) Click on ‘Start!’ and the plugin will start the download of the selected products in the
selected folder.
Figure 10: DownThemAll! Firefox plugin download window
Page 12 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
4) Once downloads are complete, it is possible to find the .zip files of the downloaded
products in the folder selected for the download.
2.6
APIs and Batch Scripting
CODA Web Services allows two dedicated Application Program Interfaces (API) for browsing
and accessing the EO data stored in the rolling archive. The APIs are:
•
Open Data Protocol (OData)
•
Open Search (Solr)
The OData interface is a data access protocol built on core protocols like HTTP and commonly
accepted methodologies like REST that can be handled by a large set of client tools as simple as
common web browsers, download-managers or computer programs such as cURL or Wget.
OpenSearch is a set of technologies that allow publishing of search results in a standard and
accessible format. OpenSearch is RESTful technology and complementary to the OData. In fact,
OpenSearch can be used to complementary serve as the query aspect of OData, which provides a
way to access identified or located results and download them.
Page 13 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
3
OPEN DATA PROTOCOL (ODATA)
The Open Data Protocol (OData) enables the creation of REST-based data services, which allow
resources, identified using Uniform Resource Identifiers (URIs) and defined in a data model, to
be published and consumed by Web clients using simple HTTP messages.
The OData protocol provides easy access to the CODA Web Service and can be used for
building URI for performing search queries and product downloads offering to the users the
capability to remotely run scripts in batch mode.
3.1
URI Components
A URI used by an OData service has up to three significant parts: the Service Root URI, the
Resource Path and the Query Options.
 the Service Root URI identifies the root of the OData service
 the Resource Path identifies the resource to be interacted with. The resource path
enables any aspect of the data model (CODA Web Service Products, CODA Web
Service Collections, etc.) exposed by the OData service
 the system Query Options part refines the results
Example of and OData URI exposed by the CODA Web Service Service broken down into its
component parts:
https://coda.eumetsat.int/odata/v1/Products?$skip=10&$top=50&$format=xml
\________________________________/\_______/\___________________________/
|
|
|
odata
service root URI
resource path
query options
OData Service Root URI for the CODA Web Service:
https://coda.eumetsat.int/odata/v1
CODA Web Service Resource Paths:
/Products
/Collections
Query Options admitted by the CODA Web Service:
 $format: Specifies the HTTP response format of the record e.g. XML or JSON;
 $filter: Specifies an expression or function that must evaluate to true for a record to be
returned in the collection;
 $orderby: Determines what values are used to order a collection of records;
 $select: Specifies a subset of properties to return;
 $skip: Sets the number of records to skip before it retrieves records in a collection;
 $top: Determines the maximum number of records to return;
For examples, the result of the search of this OData URIs is a list of the records of the last 100
products published on CODA Web Service:
https://coda.eumetsat.int/odata/v1/Products?$orderby=IngestionDate desc&$top=100
Page 14 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
This search skips the first 100 records of the products published on the CODA Web Service and
then returns the next 100:
https://coda.eumetsat.int/odata/v1/Products?$orderby=IngestionDate
desc&$top=100&$skip=100
3.2
Responses
When users query the CODA Web Service via the URI, the request is sent to the CODA Web
Service OData service. This service provides the responses. The description of the responses is
provided here below focusing on the response body and not covering the standard HTTP
response envelope.
The default response format is Atom [RFC4287], an XML-based document format that describes
Collections of related information known as “feeds”. The responses containing a single Product
entity differ from those containing a collection of Product entities. Generally speaking, the single
Product entity is returned as a bare Atom entry element, while for a collection the same Atom
entries denoting the Product entities are wrapped into an Atom feed element.
The response format can however be controlled from the requests through the $format query
option already introduced above. The formats currently implemented by CODA Web Service
service are Atom (or XML) and JSON which is a non-standard output of OData but is an
experimental behaviour useful for EO context.
The following sub-sections describe the response types for Atom/XML and JSON output
formats. The very last sub-section deals with the specific responses returned in case of error.
3.3
XML Responses
The Atom/XML format is the default response format, though it can be forced by setting the
$format query option to “atom” or “xml” indifferently as in the following examples:
https://coda.eumetsat.int/odata/v1/Products?$format=xml
3.4
Service Metadata Document
In order to help clients discover the OData services, the CODA Web Service OData Service
Metadata Document exposes the Entity Data Model of the service including among others, the
Entities and Properties that can be queried. This document can be queried as with the following
URL:
https://coda.eumetsat.int/odata/v1/$metadata
The following example is extracted from the complete Service Metadata Document exposed by
CODA Web Service OData service. Some sections or XML Namespace declarations have been
removed for brevity:
Page 15 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
<edmx:Edmx Version="1.0">
<edmx:DataServices>
<Namespace="CODA">
<EntityType Name="Node" m:HasStream="true">
<Key>...</Key>
<Property Name="Id" Type="Edm.String" Nullable="false"/>
<Property
Name="Name"
Type="Edm.String"
m:FC_TargetPath="SyndicationTitle"/>
<Property Name="ContentType" Type="Edm.String"/>
<Property Name="ContentLength" Type="Edm.Int64"/>
<Property Name="ChildrenNumber" Type="Edm.Int64"/>
<Property Name="Value" Type="Edm.String"/>
...
</EntityType>
<EntityType Name="Product" m:HasStream="true">
<Key>
<PropertyRef Name="Id"/>
</Key>
<Property Name="Id" Type="Edm.String" Nullable="false"/>
<Property
Name="Name" Type="Edm.String"
m:FC_TargetPath="SyndicationTitle"/>
<Property Name="ContentType" Type="Edm.String"/>
<Property Name="ContentLength" Type="Edm.Int64"/>
<Property Name="ChildrenNumber" Type="Edm.Int64"/>
<Property Name="Value" Type="Edm.String"/>
...
</EntityType>
<EntityType Name="Collection">
<Key>
<PropertyRef Name="Name"/>
</Key>
3.5
Using OData to discover products in the CODA Web Service archive
3.5.1
Querying the content of the products archive: usage of the /Products resource path
The following OData URI addresses the resource "Products" exposed by the CODA Web
Service which includes the records of all the products stored in the CODA Web Service archive.
Each product record includes the Universally Unique Identifier (UUID), the product name, the
link for download, the links for navigating through their nodes and attributes and some
properties.
The syntax is:
<ServiceRootUri>/Products
Example:
https://coda.eumetsat.int/odata/v1/Products
By default it provides a list of 50 records showing the properties of the latest 50 products
ingested by the CODA Web Service.
Page 16 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Examples of properties are:
- IngestionDate: date on which the Product was ingested by the contacted CODA Web
Service instance. The date is given in UTC.
- ContentDate (Start and End): time period associated to the Product payload. This date
regards the period of the subject of the content and not the content itself. For example,
for an EO product, the ContentDate deals with the observation date and not to the
downlink or processing dates.
- ContentGeometry: footprint polygon coordinates.
- Id: The universally unique identifier (UUID) is a standard identifier used in software
construction to uniquely identify information without significant central coordination. A
UUID is a 16-octet (128-bit) number. In its canonical form, a UUID is represented by 32
lowercase hexadecimal digits, displayed in five groups separated by hyphens, in the form
8-4-4-4-12 for a total of 36 characters (32 alphanumeric characters and four hyphens).
3.5.2
Querying a single Product in the archive
The responses containing a single Product entity differ from those containing a collection of
Product entities. Generally speaking, the single Product entity is returned as a bare Atom entry
element, while for a collection the same Atom entries denoting the Product entities are wrapped
into an Atom feed element.
Syntax:
<ServiceRootUri>/Products(‘UUID’)
The following example is an actual response of a CODA Web Service OData service that
returned a single Product entity identified by the UUID: 4c6ec540-08ad-466e-9e5f120fa38a82fa.
Again, this snippet has been reformatted for readability purpose but nothing has been removed
from the actual response:
https://coda.eumetsat.int/odata/v1/Products('4c6ec540-08ad-466e-9e5f-120fa38a82fa')
Response:
<id>
http://coda.eumetsat.int/odata/v1/Products('4c6ec540-08ad-466e-9e5f120fa38a82fa')
</id>
<title type="text">
S3A_OL_1_ERR____20161218T073800_20161218T082207_20161219T141423_2646_012_149_
_____MAR_O_NT_002
</title>
<updated>2016-12-19T14:16:04.373Z</updated>
<category term="CODA.Product" scheme="http://schemas.microsoft.com/ado/2007/0
8/dataservices/scheme"/>
<link href="Products('4c6ec540-08ad-466e-9e5f120fa38a82fa')" rel="edit" title="Product"/>
<link href="Products('4c6ec540-08ad-466e-9e5f120fa38a82fa')/$value" rel="edit-media" type="application/octet-stream"/>
<link href="Products('4c6ec540-08ad-466e-9e5f120fa38a82fa')/Products"rel="http://schemas.microsoft.com/ado/2007/08/dataser
vices/related/Products" title="Products" type="application/atom+xml;type=feed
Page 17 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
"/>
<link href="Products('4c6ec540-08ad-466e-9e5f120fa38a82fa')/Nodes"rel="http://schemas.microsoft.com/ado/2007/08/dataservic
es/related/Nodes" title="Nodes" type="application/atom+xml;type=feed"/>
<link href="Products('4c6ec540-08ad-466e-9e5f120fa38a82fa')/Attributes"rel="http://schemas.microsoft.com/ado/2007/08/datas
ervices/related/Attributes" title="Attributes" type="application/atom+xml;typ
e=feed"/>
<link href="Products('4c6ec540-08ad-466e-9e5f120fa38a82fa')/Class"rel="http://schemas.microsoft.com/ado/2007/08/dataservic
es/related/Class" title="Class" type="application/atom+xml;type=entry"/>
<content type="application/octet-stream" src="Products('4c6ec540-08ad-466e9e5f-120fa38a82fa')/$value"/>
<m:properties>
<d:Id>4c6ec540-08ad-466e-9e5f-120fa38a82fa</d:Id>
<d:Name>
S3A_OL_1_ERR____20161218T073800_20161218T082207_20161219T141423_2646_012_149_
_____MAR_O_NT_002
</d:Name>
<d:ContentType>application/octet-stream</d:ContentType>
<d:ContentLength>742531500</d:ContentLength>
<d:ChildrenNumber>0</d:ChildrenNumber>
<d:Value m:null="true"/>
<d:CreationDate>2016-12-19T14:16:01.655</d:CreationDate>
<d:IngestionDate>2016-12-19T14:16:04.373</d:IngestionDate>
<d:EvictionDate m:null="true"/>
<d:ContentDate m:type="CODA.TimeRange">
<d:Start>2016-12-18T07:52:12.252</d:Start>
<d:End>2016-12-18T08:26:30.069</d:End>
</d:ContentDate>
<d:Checksum m:type="CODA.Checksum">
<d:Algorithm>MD5</d:Algorithm>
<d:Value>28B33FD16D16ED2390982A56D88FBD4A</d:Value>
</d:Checksum>
3.5.3
Querying the products (paging)
The URI to be used for paging the list of products in the archive shall follow the syntax below:
<ServiceRootUri> /Products?$skip='N'&$top='M'
Where:
- ?$skip='N' is the number of records to skip before it retrieves records in a collection,
- $top=’M’ is the maximum number of records to return.
For example this OData URI allows to list 50 products skipping the first 10:
https://coda.eumetsat.int/odata/v1/Products?$skip=10&$top=50
Note: $top accepts as maximum value 100. For higher values the maximum number of records
returned will still be 100.
3.5.4
Using OData to filter the products in the archive
It is possible to use the OData properties to perform search query to filter the products.
Page 18 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
3.5.4.1 Filtering the Products on time-based criteria
For metadata that are expressed in the UTC format such as
Filter
CreationDate
IngestionDate
EvictionDate
ContentsDate/Start and ContentDate/End
Description
For Future implementation
Time of the products archiving in the CODA
Web Service
For Future implementation
TBW
Filters can be performed on the day and/or month and/or year.
Symbols are not allowed in the OData protocol but they can be substituted with the following
syntax:
Symbol
<
≤
>
≥
=
Regular Expression
lt
le
gt
ge
eq
Meaning
Lower than
Lower or equal than
Greated than
Greater or equal than
equal
Here below we provide some examples.
3.5.4.2 Filtering the products by Ingestion Date
To filter the products by Ingestion Date, the following syntax may apply:
<ServiceRootUri> /Products?$filter=year(IngestionDate) years[and
month(IngestionDate)][and day(IngestionDate)]
For example this URI selects the products that have been published in the CODA Web Service
on February 2017:
https://coda.eumetsat.int/odata/v1/Products?$filter=year(IngestionDate) eq 2017 and
month(IngestionDate) eq 02
This URI selects the products that have been published in the CODA Web Service after 22nd Feb
2017:
https://coda.eumetsat.int/odata/v1/Products?$filter=IngestionDate gt datetime'2017-0222T00:00:00'
3.5.4.3 Filtering the Products using the file name
The products file name can be used for filtering the products. It shall be noticed that this query
criteria is not based on Medatata indexed from the products content but the criteria is search
products matching a predefined string on the file name.
Page 19 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
In this case, Entity Data Model (EDM) strings are used for metadata like Name which indicates
the name of the product. Every metadata containing a string can be filtered using the substringof
filter.
For example, this OData URI lists the products having OL_1 (that indicates the OLCI files) in
the file name:
https://coda.eumetsat.int/odata/v1/Products?$select=Id&$filter=substringof(‘_OL_1’,Name)
3.5.4.4 Filtering the Products by Sensing Time (file name)
To filter the products by the Sensing Time (start and stop) the S1 product filename can be used,
using the filter substringof. Next section will describe how to make query based on indexed
metadata.
The syntax is:
<ServiceRootUri> /Products?$select=Id&filter=substringof('<string>', Name)
For example this OData URI lists the products having in the file name the string
20170203T154103. It means it returns the products with a predefined sensing time start (e.g.
2017-02-03, T154103):
https://coda.eumetsat.int/odata/v1/Products?$select=Id&$filter=substringof(%2720170203T154
103%27,Name)
3.5.5
Querying Product Attributes and Metadata Contents
OData can be used also for querying the values of products metadata content.
3.5.6
Get product UUID from product name
To retrieve the UUID of a predefined product, the syntax is:
<ServiceRootUri> /Products?$filter=Name <Regular Expression> '<Filename>'
In this example, this URI retrieves the UUID of the product named
S3A_OL_1_EFR____20170128T133750_20170128T134050_20170128T153137_0179_013_35
2_3240_MAR_O_NR_002:
https://coda.eumetsat.int/odata/v1/Products?$filter=Name eq
'S3A_OL_1_EFR____20170128T133750_20170128T134050_20170128T153137_0179_013_3
52_3240_MAR_O_NR_002'
Page 20 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
3.6
Download
OData URL can be used for download full products as well as partial products by using the
combination of the Nodes and Attributes indexed for the products and adding $value at the end
of the URI.
3.6.1
Download full product from its UUID
To download a full product the syntax is:
<ServiceRootUri> /Products('UUID')/$value
For example, this URI starts the download of the product with Id 52e48b83-c717-484a-a33af4ebc941dd84:
https://coda.eumetsat.int/odata/v1/Products('52e48b83-c717-484a-a33a-f4ebc941dd84')/$value
3.6.2
Discover Product Nodes
Products from all processing levels (Level-1 and Level-2) are disseminated in SENTINELSAFE format. The data delivered is packaged as a file structure containing a manifest file in
XML format listing general product metadata and subfolders for measurement data, annotations,
previews and support files. The SENTINEL-SAFE format wraps a folder containing image data
in a binary data format and product metadata in XML. This flexibility allows the exploration of
parts of the products without download the complete product.
In this section we will provide examples to how explore products nodes.
The following URI will return the XML file including the list of the first level nodes:
<ServiceRootUri> /Products('<UUID>')/Nodes
Example:
https://coda.eumetsat.int/odata/v1/Products('52e48b83-c717-484a-a33a-f4ebc941dd84')/Nodes
Generally the first node is the one whose name is [Product_Name.SAFE] and the URI to
visualize the XML schema of this node is the following:
<ServiceRootUri> /Products('<UUID>')/Nodes('[PRODUCT_NAME.SAFE]')/Nodes
Examples of the nodes of the Level-1 Sentinel Product are:
 Manifest
 Annotation
 Measurement
 Preview
 Support
Page 21 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
3.6.3
Download manifest file from the UUID
To download manifest file from the UUID the syntax is:
<ServiceRootUri>
/Products('UUID')/Nodes('Filename')/Nodes('manifest.safe')/$value
Example:
https://coda.eumetsat.int/odata/v1/Products('52e48b83-c717-484a-a33af4ebc941dd84')/Nodes('S3A_OL_1_EFR____20170128T133750_20170128T134050_20170128
T153137_0179_013_352_3240_MAR_O_NR_002.SEN3')/Nodes('manifest.safe')/$value
3.6.4
Download of Sentinel-1 quick-look file knowing the product UUID
To download the quick-look file from the UUID the syntax is:
<ServiceRootUri>
/Products('UUID')/Nodes('Filename')/Nodes('preview')/Nodes('quicklook.png')/$value
Example:
https://coda.eumetsat.int/odata/v1/Products('52e48b83-c717-484a-a33af4ebc941dd84')/Nodes('S3A_OL_1_EFR____20170128T133750_20170128T134050_20170128
T153137_0179_013_352_3240_MAR_O_NR_002.SEN3')/Nodes('preview')/Nodes('quicklook.png')/$value
3.6.5
Verifying Download integrity using the MD5 checksum
This can help to reveal if the download was incomplete.
Each product published on CODA Web Service provides an MD5 checksum of the
downloadable ZIP file. The Message Digit of the file can be discovered using the following
OData query:
https://coda.eumetsat.int/odata/v1/Products('UUID')/Checksum/Value/$va
lue
where UUID is the Universally Unique Identifier of the product.
Example:
https://coda.eumetsat.int/odata/v1/Products('0ae32ee3-efc8-4080-8e24cdc01f2ee118')/Checksum/Value/$value
provides the following MD5: 187C459878D55D3494304E21FD6AF8C5.
Page 22 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
The integrity of the downloaded product can be checked by comparing the MD5 value of the
product at the source with the MD5 of the product located on the users’s system. If they are
identical the download has been successful.
Different Operating Systems require different methods for verifying the checksum of the
downloaded products. Find below a list of the most common ones:
Linux
Open a terminal and type: md5sum path/filename
OS X
Open a terminal and type: md5 path/filename
Windows
The File Checksum Integrity Verifier (FCIV) utility does not come pre-installed on windows and
needs to be downloaded. Instructions can be found here.
Open a command prompt and type: FCIV -md5 -sha1 path\filename
OpenSSL
This method can be done on any system with OpenSSL installed (i.e. Windows, Linux and
MAC).
Open a command prompt or terminal and type: openssl md5 path/filename
Page 23 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
4
OPEN SEARCH
OpenSearch (Solr) is a set of technologies that allow publishing of search results in a standard
and accessible format. OpenSearch is RESTful technology and complementary to the OData. In
fact, OpenSearch can be used to complementary serve as the query aspect of OData, which
provides a way to access identified or located results and download them. The CODA Web
Service implementation uses the Apache Solr search engine.
4.1
URI Components
<CODAhostname>:<port>/<path>/search?q=<query>[&start=<page_start_value>][&rows=
<page_size>]
where:


<CODA_hostname>:<port>/<path> is the Service Root (in our case CODA_hostname is
https://coda.eumetsat.int/ )
/search?q=<query>[&start=<page_start_value>][&rows=<page_size>] is the Open
Search query
4.2
Using Open Search to discover products in the CODA Web Service archive
4.2.1
Discover the list of the products stored in the archive
The syntax is:
<CODA_hostname>:<port>/search?q=*
Example:
https://coda.eumetsat.int/search?q=*
The above URI returns an XML file including the list of the nodes of every products stored in
the CODA Web Service archive.
4.2.2
Paging results
A query can be completed with the pagination options. The available options are:
 rows=<page_size> , where page_size is the number of results listed per page
 start=<page_start_value>, where page_start_value identifies from which
result position starts the pagination.
If nothing is specified by default page_size and page_start_value are set to rows=10 and start=0 .
So the complete search will be:
<CODA_hostname>:<port>/<path>/search?q=*&rows= <N> &start= <N>
Page 24 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Note: The maximum number of rows to be returned in a single query is set to 100. Requests for
more than the maximum supported number of rows will result in an error (http 500).
To display just the first 10 results of the previous query, the can be completed with
&rows=10&start=0
Results sets over the maximum can be obtained through paging of from different start values.
 Page 1: https://coda.eumetsat.int/search?start=0&rows=100&q=*
 Page 2: https://coda.eumetsat.int/search?start=100&rows=100&q=*
 Page 3: https://coda.eumetsat.int/search?start=200&rows=100&q=*
4.2.3
Sorting results
The list of results can be sorted using the option:
orderby=
possible values are:
 beginposition asc: sorts results by sensing date arranged in ascending order
 beginposition desc: sorts results by sensing date arranged in descending order
 ingestiondate asc: sorts results by ingestion date arranged in ascending order
 ingestiondate desc: sorts results by ingestion date arranged in descending order
Example:
https://coda.eumetsat.int/search?q=*&orderby=beginpositionasc asc
4.3
Discover the products over a predefined Area Of Interest (AOI): Geographical
Search
It is possible to search products on the basis of a geographical area of interest, e.g. get the list of
products over a geographic area delimitated by the polygon having vertices:
<CODA_hostname>:<port>/<path>/search?q=footprint:"Intersects(
type> )"
<geographic
The <geographic type> value can be expressed as a polygon or as a point according to the syntax
described below.
Polygon:
<geographic type> =POLYGON((P1Lon P1Lat, P2Lon P2Lat, …, PnLon PnLat, P1Lon
P1Lat))
where P1Lon and P1Lat are the Longitude and Latitude coordinates of the first point of the
polygon in decimal degrees (DDD) format (e.g. 2.17403, 41.40338) and so on.
The coordinates of the last point of the polygon must coincide with the coordinates of the first
point of the polygon.
Page 25 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
The polygon describing the geographical area can have a maximum of 200 points that must be
within an area described by 10 degrees of latitude and 10 degrees of longitude.
Example:
https://coda.eumetsat.int/search?q=footprint:"Intersects(POLYGON((-4.53 29.85, 26.75 29.85,
26.75 46.80,-4.53 46.80,-4.53 29.85)))"
the AOI in this case is a bounding box around the Mediterranean Sea.
Point:
<geographic type>= Lat, Lon
where the Latitude ( Lat ) and Longitude ( Lon ) values are expressed in decimal degrees (DDD)
format (e.g. 41.40338, 2.17403 ).
Example:
https://coda.eumetsat.int/search?q=footprint:"Intersects(41.9000, 12.5000)"
Open Search Queries examples
The [query] in the open search URI will follow the same syntax used in the full text search. Here below
we provide some examples.
Example
Searches
every
product
with
OL_1_EFR___ product type or
products containing the string
“OL_1_EFR___” in the metadata.
Search every products ingested in the
last day
Search every products ingested in the
last month
Search every products ingested in the
last hour
Search every products having sensing
in the last three months
Open Search
https://coda.eumetsat.int/search?q=OL_1_EFR___
https://coda.eumetsat.int/search?q=ingestionDate:[NOW1DAYS TO NOW]
https://coda.eumetsat.int/search?q=ingestionDate:[NOW30DAYS TO NOW]
https://coda.eumetsat.int/search?q=ingestionDate:[NOW1HOUR TO NOW]
https://coda.eumetsat.int/search?q=beginPosition:[NOW3MONTHS TO NOW] AND endPosition: [NOW3MONTHS TO NOW]
Search every products delimitated by https://coda.eumetsat.int/search?q=footprint:"Intersects(POL
the polygon vertices: -4.53 29.85, YGON((-4.53 29.85, 26.75 29.85, 26.75 46.80,-4.53 46.80,26.75 29.85, 26.75 46.80,-4.53 46.80,- 4.53 29.85)))"
4.53 29.85
Page 26 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
5
BATCH SCRIPTING
The above OData and OpenSearch URIs can be combined to create complex queries to be
executed in non-interactive scripts using programs like cURL and Wget.
5.1
Download via Curl
Using cURL it is possible to create a script to login to the CODA Web Service via the following
command line:
curl -u {USERNAME}:{PASSWORD} <URI_QUERY>
where:
<URI_QUERY>
-u <username>:<password>
5.2
is a valid OData URI or OpenSearch URI.
option to specify user and password to use when fetching
Download via Wget
It is also possible to download the products from the CODA Web Service archive using wget.
The following example shows how to download a single product, identified by its own CODA
Web Service universally unique identifier {UUID}, using an OData URI:
wget --no-check-certificate --continue --user={USERNAME} --password={PASSWORD}
"https://coda.eumetsat.int/odata/v1/Products('{UUID}')/$value"
(Please, note that the “-“ symbol corresponds to the Hyphen-Minus (U+002D). Pay attention to not confuse it with other symbols
or you will get an error).
The option --continue is very useful when downloads do not complete due to network problems.
Wget will automatically try to continue the download from where it left off, and repeat this until
the whole file has been retrieved.
The following example shows how to download the manifest file (identified by the universally
unique identifier {UUID}) of a Sentinel-1 product using and OData URI with wget:
wget
--no-check-certificate
--user={USERNAME}
--password={PASSWORD}
"https://coda.eumetsat.int
/odata/v1/Products('{UUID}')/Nodes('{PRODUCT_FILENAME}')/Nodes('manifest.safe')/$value
"
where {UUID} is the value of the universally unique identifier of the product, and
{PRODUCT_FILENAME} is the filename of the product.
Page 27 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
5.3
Scripts Examples
Please, note that, since because those scripts are provided in a shell (.sh) file, they can be used
under a Unix/Linux environment.
5.3.1
dhusget script
dhusget.sh is a simple demo script illustrating how to use OData and OpenSearch APIs to query
and download the products from any CODA Web Service. It allows:
 Search products over a pre-defined AOI
 Filter the products by ingestion time, sensing time and coordinates
 Save the list of results in CSV and XML files
 Download the products
 Download the manifest files only
 Perform the MD5 integrity check of the downloaded products
You can download the script from here
Tip: once the shell file is downloaded, open it with a text editor, go to the section ‘Load input
parameter’ and change the values of USERNAME and PASSWORD with your username and
password. In this way it will be not necessary to pass them as arguments when running the
script.
5.3.1.1
USAGE:
# dhusget.sh [LOGIN OPTIONS]...
OPTIONS]... [DOWNLOAD OPTIONS]...
[SEARCH
QUERY
OPTIONS]...
[SEARCH
RESULT
LOGIN OPTIONS:
-d
-u
-p
<CODA URL>
<username>
<password>
URL of the CODA Web Service Service to be polled;
CODA Web Service username;
CODA Web Service password provided after registration;
Search Query options
-i <instrument name>
-t <time in hours>
-f
<file>
-c
<coordinates i.e.:
lon1,lat1:lon2,lat2>
-T <product type>
instrument name
search for products ingested in the last <time in hours> (integer) from the time of
execution of the script. (e.g. '-t 24' to search for products ingested in the last 24
Hours)
search for products ingested after the date and time provided through the
input <file>. The file is updated at the end of the script execution with the ingestion
date of the last successful downloaded product.
coordinates of two opposite vertices of the rectangular area of interest
product type of the product to search (for examples: OL_1_EFR___) ;
Search Result options
-l
-P
-q
<results>
<page>
<XMLfile>
-C
<CSVfile>
maximum number of results per page [1,2,3,4,..]; default value = 25
page number [1,2,3,4,..]; default value = 1
write the OpenSearch query results in a specified XML file. Default file is
'./OSquery-result.xml'
write the list of product results in a specified CSV file. Default file is './productslist.csv'
DOWNLOAD OPTIONS:
-o <option>
what to download; possible options are:
 'manifest' to download the manifest of all products returned from the search or
 'product' to download all products returned from the search
Page 28 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
-O <folder>
-N <1...n>
-D
-r <file>
-L <lock folder>
-n <1...n>
 'all' to download both
save the Product ZIP files in a specified folder.
set number of wget download retries. Default value is 5. Fatal errors like
'connection refused' or 'not found' (404), are not retried;
if specified, remove the products that have failed the MD5 integrity check from
disk. By deafult products are not removed;
download the products listed in an input <file> written according to the following
format:
 One product per line.
 <space><one
character><space><UUID><space><one
character><space><filename>.
by default only one instance of dhusget can be executed at a time. This is ensured
by the creation of a temporary lock folder /home/dhus/dhusget_tmp/lock which is
removed a the end of each run.
For running more than one dhusget instance at a time is sufficient to assign
different lock folders using the -L option (e.g. '-L foldername') to each dhusget
instance;
number of concurrent downloads (either products or manifest files). Default value
is 2; this value doesn't override the quota limit set on the server side for the user
HELP
--help
Help
Examples:
 ./dhusget.sh –l 100 –o products to download the latest 100
products

5.3.2
./dhusget.sh -l 100 -c '-4.530,29.850:26.750,46.800' to download
the latest 100 products intersecting the specified coordinates
Odata-demo.sh script
The script odata-demo is a demo script performing the following selective actions:
1. List the collections
2. List <n> products from a specified collection
3. List first 10 products matching part of product name
4. List first 10 products matching a specific ingestion date
5. List first 10 products matching a specific aquisition date
6. List first 10 products since last <n> days, by product type and intersecting an AOI
7. Get product id from product name
8. Get polarisation from a product id
9. Get relative orbit from a product id
10. Download Manifest file from a product id
11. Download quick-look from a product id
12. Download full product from its id
You can download the script here
Page 29 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
NOTE: It
requires the installation of xmlstarlet.
Page 30 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
5.3.2.1 USAGE:
-h,
-j,
-u,
-p,
-v,
-V,
--help
--json
--user=Name
--password=PASSWORD
--verbose
--version
displays a help message
Use json output format for OData (default is xml)
Use Name as username for the DataHub Server;
Use PASSWORD as password for the DataHub Server;
Display curl command lines and results
Display the current script version
Page 31 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
6
KNOWN ISSUES AND USEFUL TIPS FOR CODA WEB SERVICE
In this paragraph we are going to list the already known issues about CODA Web Service, and
some useful tips to improve the usability of the application, meanwhile we are working on the
improvement of the service in order to give our users a better experience.
Please, take note of these limitations, and consider that we are working to fix them in the next
CODA Service updates.
KNOWN ISSUE
At the moment, it is not yet present in CODA
web page a “Forgot Password” functionality
that allows the user to recover his lost password.
In the User profile there are some parameters
which seem to be default (country: Afghanistan,
domain: Atmosphere)
WHAT USER CAN DO FOR NOW
Write an email to EUMETSAT User Service
HelpDesk at [email protected] asking for a new
password.
Those parameters represent the default values when
the user accesses the ‘Edit Profile’ section.
At the moment it is not possible for the user to
display his/her own parameters from user profile.
Wait for a new CODA update.
When the user clicks on Home button ( ), a Login again.
new CODA session is opened in new tab, but Do not use the Home button to refresh the page.
the user is no more logged in.
When a search is performed, the map is not Map does not automatically update the centre and the
centred on the resulting images footprint, but it zoom based on search’s results. If you want to centre
remains the same.
the map on one of the resulting products, click on one
of the results in the list, and then click on the icon
“Zoom to product” (see image below).
The same button has two functionalities:
- Navigate On Map that allows the user to
move on the base layer map
- Draw Region of Interest that allows the
user to draw a rectangle on the map to
filter the search results geographically.
This button seems to have the opposite function
with respect the icon it is displaying.
There are two ways to add a product to Cart:
- Select the product you want to add to
Cart then click on the icon ‘Add Product
to
Cart’
(see
figure
below)
-
Please, note that when the Navigate on Map icon
is displayed it is possible to draw a rectangle on the
map and that when Draw Region of Interest icon
is displayed it is possible to move the map.
The icon represents the functionality that will be
activated when the button is clicked.
If you want to access your cart with the list of
products ready to be downloaded, click on ‘User
Profile’ button
and select ‘Cart’. This will lead
you to the list of products you chose.
Select one or more products you want to
add to Cart then click on the icon ‘Add
selected products to Cart’ displayed at
the bottom of the results list.
Please, note that the
icon will not take you
to your list of products chosen for the download.
Page 32 of 33
EUM/OPS-SEN3/MAN/16/880763
v2 Draft, 24 January 2017
CODA user manual
Page 33 of 33