SDMX IT Tools
SDMX use in practice in NA
National Account use case
Jean-Francois LEBLANC
Christian SEBASTIAN
June 2015
Eurostat Unit B3 – IT and standards for data and metadata exchange
Eurostat
Table of contents
1. Objectives
2. National Accounts (NA)
1.
2.
3.
4.
5.
NA
NA
NA
NA
NA
business needs
Domains
Technical Integration
Data Validation Process
Project Cycle
2
Eurostat
Table of contents
2. SDMX Tools in practice
1.
2.
3.
4.
5.
Data Structure Wizard
SDMX Converter
SDMX Reference Infrastructure
Tools relationship
SDMX Tools – Summary
3. SDMX Tools for Data Providers
1.
2.
3.
4.
Possibilities for Data Providers
IT Architectures for Data Providers
How to use the tools
Conclusion
3
Eurostat
1 Objectives
• To reflect how National Accounts work
• To present the SMDX IT tools
• To expose how to work with the data and SDMX
in real situations
• To give a base understanding of the tools and
their relationship that will be of help for further
explanations
4
Eurostat
2 National Accounts (NA)
National
Accounts
BOP
FDI
Excel questionnaires
GESMES files
GFS
Main
Aggregates
Sector
Accounts
Supply & Use
Input /
Output
5
Eurostat
2 Why SDMX for NA?
Within
Eurostat
Within the
ESS
Internationally
6
Eurostat
2.1 NA business needs
Within
Eurostat
Internationally
• Main trigger:
new methodology
• SNA 93  SNA 2008
• ESA 95  ESA 2010
• Business process reengineering
• Corporate production systems
• Integration of production
chains
Within the
ESS
• Harmonised coding
7
Eurostat
2.1 NA business needs
Cross-domain integration
• Balance of Payments
• Foreign Direct Investment
Within
Eurostat
Internationally
Within the
ESS
ESS Business process
reengineering
• Standardisation
• Improved validation
• Reuse of tools
8
Eurostat
2.1 NA business needs
Data quality
• 1 country  1 year  1 GDP
• Shared validation rules
• Across all organisations
Within
Eurostat
Lowering reporting burden
• 1 data flow from national to
international
International cooperation
Internationally
Within the
ESS
• Inter-Agency Group on
Economic and Financial
Statistics = SDMX Sponsors
• Example: PGI Website
http://www.principalglobalindicators.org
9
Eurostat
2.2 NA Domains
National
Accounts
Shared
Concepts and
Code Lists
Cross Domain
Concepts and
Code Lists
SNA 2008
ESA 210
Other related domains
Balance
of
Payment
FDI
statistics
Statistics
Other domains
10
Eurostat
2.3 NA Technical Integration
Source
Source
National
DBs
Mapping
SDMX
DSDs &
Data Flows
Source
National data
sources
National statistical
organizations
International
organizations
Metadata driven process based on SDMX standards and tools
Full automation of the data exchange possible
11
Eurostat
2.4 NA Data Validation Process
Before/during transmission
“First Level”
After Transmission
“Second Level”
Currently covered by SDMX
• Format Check (SDMX-ML)
• Code Check (SDMX DSD)
• Code used correctly
(SDMX Dataflow & Constraint)
Not covered by SDMX
• Detailed value check
• Validation expressions
• …
12
Eurostat
2.5 NA Project Cycle
13
Eurostat
3. SDMX Tools in practice
Registry
• Find DSDs
• Edit/Create DSDs
DSW
• Data files -> SDMX files
Converter
• Database -> SDMX WS/files
SDMX-RI
14
Eurostat
3.1 Data Structure Wizard
• Java tool (open source and platform independent)
• Supports
• Maintenance of SDMX v2.0 and v2.1 structures
• Import and export of artefacts from/to SDMX-ML v2.0,
v2.1 and GESMES format
• Interaction with Euro SDMX Registry (retrieval,
modification and search of artefacts)
• Sample dataset creation
15
Eurostat
3.2 SDMX Converter
• Java tool (open source and platform independent)
• File based conversion
• Different ways of using it:
•
•
•
•
Graphical user interface
Command Line interface
Web Service interface
Common API interface
• Formats*: SDMX-ML 2.0, SDMX-ML 2.1, SDMX-EDI
(Gesmes), FLR, CSV, Google DSPL, predefined Excel
templates
*Exceptions may apply
16
Eurostat
3.3 SDMX Reference Infrastructure
Data Provider
Data Collector
SDMX Registry
Mapping
Assistant
DSD
Web
Svc
Non-SDMX
local database
Test
Client
SDMX-RI
Eurostat
Web
Client
SDMX data set
17
3.4 Tools relationship
produces
uses
stores
DSD
SDMX
Registry
DSW
MSD
produces
DSD
uses
SDMX
Converter
produces
CSV or other
datafile
produces
SDMX
Dataset
ESS-MH
MSD
uses
SDMX-RI
exposes
uses
Mapping
Assistant
SDMX
MetaDataset
produces
Mapping
Census Hub
National DB
Eurostat
18
3.4.1 Tools relationship – Example
Eurostat produces NA DSDs
DSW
Eurostat stores NA DSDs
SDMX Registry
DSD
DSD
SDMX-RI
Mapping
Assistant
SDMX-ML
Dataset
NSI uses
NA DSDs
NSI maps
NA DSDs
Mapping
National
DB
NSI produces
SDMX
Converter
Eurostat
NSI uses
NA DSDs
NSI exports NA data
(CSV, XLS)
19
3.5 SDMX Tools – Summary
• Eurostat tools  Eurostat SDMX Info Space
• SDMX Converter
• SDMX Reference Infrastructure & Mapping Assistant
• SDMX Registry & Data Structure Wizard
20
Eurostat
4. SDMX Tools for Data Providers
Tools offered
Use
Action
Web Forms (EU)
Excel-like templates
Transmission of low
volumes of data
No costs for EU organisations
Manual work for senders
(type, copy/paste)
SDMX Converter
Converts data files from
different file formats to
SDMX format
Local installation on sender’s
PC
Installation on server
SDMX Reference
Infrastructure
A set of tools that allows Installation on server
to transform directly your Mapping of database to DSDs
data from a database to
SDMX
Common API
Supports the
development of local
SDMX systems
Local development in
organisations
21
Eurostat
4.1 Possibilities for Data Providers
Webforms
EDAMIS
Single Entry
Point
SDMX-ML file
Excel
sheets
SDMX Converter
SDMX-ML file
Database
export
SDMX Converter
SDMX-ML file
Database
SDMX Reference
Infrastructure
Same
DSD

Same
Format
SDMX-ML file
SDMX Web Service
22
Eurostat
4.1.1 Possibilities for Data Providers. NA.
Not applicable
Webforms
in
National
Accounts
Will be phased
Excel
out in
National
sheets
Accounts
Database
export
SDMX Converter
SDMX-ML file
SDMX Converter
SDMX-ML file
Database
SDMX Reference
Infrastructure
EDAMIS
Single Entry
Point
SDMX-ML file
Same
DSD

Same
Format
SDMX-ML file
SDMX Web Service
23
Eurostat
4.2 IT Architectures for Data Providers
Webforms
Web-based
Push
mode
(EDAMIS)
Excel
sheets
SDMX Converter
Local conversion
Push
mode
(EDAMIS)
Database
export
SDMX Converter
Local conversion
Push
mode
(EDAMIS)
Database
SDMX Reference
Infrastructure
Pull
mode
24
Eurostat
4.3 How to use the tools
If you are a Data Collector
Design your DSD using
the DSW
Make the DSD available in
the SDMX Registry
Inform your data providers
for the available DSD
25
Eurostat
4.3 How to use the tools
If you are a Data Provider
Download/pull the DSD from the
SDMX Registry
If you
have a
DDB
DDB
Data
stores as
files
Install
and use
the SDMXRI
Install
and use
the
Converter
Expose the data to be
pulled (WS, HUB)
Push via
EDAMIS
Eurostat
26
4.4 Conclusion
SDMX statistical and technical standards
& SDMX IT tools and IT infrastructure
provide the "toolkit" for
 the harmonisation and integration of data
structures
 the rationalisation of national/international data
flows
 finally improving the data quality of official
statistics.
27
Eurostat
SDMX in practice in NA
28
Eurostat