1. SDMX:
Background and purpose
Edward Cook
Eurostat
Unit B5: “Central data and metadata services”
1
SDMX Basics course, 27-29 October 2015
Eurostat
BACKGROUND:
Exchanging data and metadata:
- is taking place against a backdrop of
developments;
- is faced with a number of issues;
- BUT offers various opportunities.
2
Eurostat
BACKGROUND DEVELOPMENTS
- Increasing demand for data
(ease of use of Internet);
- Faster exchange of electronic data;
- More frequent and important exchanges;
- Growing types of information exchange;
- between businesses
- between businesses and customers
- between individuals
3
Eurostat
BACKGROUND ISSUES:
- Data being collected in multiple ways
(surveys, files, web queries, metadata etc.);
- Data being transmitted in various formats
(paper, excel, flat files etc.)
- Data being transmitted in various media
(email, CD-ROMs, file uploads etc.);
- Data being stored in various places
(USB, hard drive, servers, cloud etc.);
4
Eurostat
BACKGROUND ISSUES:
- Multiple organisations collecting similar or
same data;
- Similar concepts in wording can have a
different content;
- Increasing burden on organisations
(collection, maintenance and managing);
- The intensive and manual nature of data
collection;
- Errors and inconsistencies;
5
Eurostat
' A pessimist sees the difficulty in every opportunity;
an optimist sees the opportunity in every difficulty.'
Winston Churchill
6
Eurostat
BACKGROUND OPPORTUNITIES
- Afforded by new technological
developments;
- For the process to be more efficient;
- To improve trust and reliability;
- To improve web dissemination;
7
Eurostat
ESPECIALLY OPPORTUNITIES FOR:
- Simplification
(streamlining data flows, central
management);
- Standardisation
(software tools, data sharing);
- Harmonisation
(data structures, concepts and code lists);
8
Eurostat
So what is SDMX?
The 'Statistical Data and Metadata eXchange'
is an international initiative aimed at
developing and employing more efficient
processes for the exchange and sharing of
statistical data and metadata among
international organisations and member
countries.
It consists of technical and statistical
standards, guidelines, an IT service
infrastructure and IT tools.
9
Eurostat
What is the business case for SDMX?
- SDMX is a global response:
7 international organisations as sponsors,
in collaboration with countries throughout
the world;
- SDMX is an ISO IS standard (17369):
- a document, established by consensus;
- approved by a recognised body;
- providing rules and guidelines;
- for common and repeated use;
- for optimum degree of order;
- viewed as safe, reliable, good quality.
10
Eurostat
- SDMX improves timeliness:
- faster access to data;
- move towards automation.
- SDMX improves accessibility:
- bilateral, gateway and data-sharing;
- push and pull modes;
- SDMX improves interpretability:
- standardises structural metadata
(the identifiers and descriptors of data);
- standardises reference metadata
(the content and quality of data);
11
Eurostat
- SDMX improves coherence:
- uses cross domain concepts;
- uses shared code lists;
- uses content oriented guidelines;
- reuse across domains and agencies
- aims for single figure dissemination.
- SDMX can reduce data errors:
- some automated validation;
- agreed structures for transmission;
- time saved on conversion, mapping;
- less manual intervention.
12
Eurostat
- SDMX can reduce the reporting burden:
- pre-validated content;
- automated publication;
- possible 'pull' by collecting agencies.
- SDMX can reduce IT development and
maintenance costs:
- open source approach;
- no licensing costs;
- shared toolbox;
- improved interoperability between
systems and applications.
13
Eurostat
SDMX is well suited to supporting a data sharing process:
reporting every number only once.
14
Eurostat
SDMX is about changing from a multiple, diverse and
complex exchange system, to a common, harmonised and
standardised exchange system.
Eurostat
15
What are the downsides?
- SDMX is not investment free: it means
training; it means changes.
- SDMX is not a magic wand: it is suited
to aggregated data but not microdata.
- SDMX is dynamic: software versions are
updated to increase functionalities and
overcome bugs.
16
Eurostat
KEY messages:
- SDMX responds to a business need;
- SDMX improves quality in data and
metadata exchanges;
- SDMX is an international standard
based on shared experiences;
- SDMX offers cost-efficiencies.
17
Eurostat