3. An overview of the SDMX
implementation process
Edward Cook
Eurostat
Unit B5: “Central data and metadata services”
1
SDMX Basics course, 1-2 March 2017
Eurostat
Who is involved in SDMX exchanges?
What are our roles?
- Initiator?
- Facilitator?
- Implementer?
2
Eurostat
What is a process?
It is a series of actions or steps taken to achieve a
particular end.
Why is understanding a process important?
Without explaining the process, stakeholders risk
becoming alienated and resistant to change.
Why is understanding the SDMX implementation
process important?
SDMX is quite complicated. But, understanding the
process makes it more accessible, more collective and
more likely to succeed.
3
Eurostat
THEORY
Any implementation process is about
approach to (among other things):
-
Planning;
Organising;
Managing; and
Optimising resources.
a
methodical
(work and people)
(workflows, meeting, paperwork)
(even in public administrations!)
And all within the constraints of:
-
Scope;
Time;
Budget; and
Quality.
(final & intermediate deadlines)
(with competing demands)
(fitting within a C.O.P.)
4
Eurostat
PRACTICE
The SDMX implementation process can be broken down
into key phases. This is the ESTAT process:
- Phase 1: Preparation
- Phase 2: Compliance
- Phase 3: Implementation
- Phase 4: Production
5
Eurostat
'By failing to prepare, you are preparing to fail.'
Benjamin Franklin
6
Eurostat
THEORY
The 'Preparation Phase' is arguably the most critical phase.
- It determines the objectives, scope, expected benefits
and outputs of the project.
- It requires a thorough discussion and assessment
of needs, of stakeholders (and their roles)
and of the business process.
- It bears in mind policies, mandates, technological
changes and different conceptual languages.
7
Eurostat
PRACTICE
- Bearing in mind the SDMX mandate for the ESS;
(Directors' Meeting April 2009):
As a first step before making SDMX compulsory for all domains in
Eurostat, the use of SDMX would be made compulsory for all new or
considerably changed data sets and reference metadata sets.
- Being conscious of the efficiency gains that could be
harnessed from technological advances; and
- Being mindful of the different technical languages spoken
by IT units and production units…
8
Eurostat
…good communication is essential.
'Communication and trust are the two main ingredients
for a successful relationship.'
Unknown
9
Eurostat
PRACTICE
There is no agreed list of preparatory questions for kick-off
discussions, but these are the kind of topics to be covered:
- A needs assessment:
- Why do you want SDMX? What should it solve?
What constraints? What timetable? What risks?
- An overview of the current production process:
- What production systems are used? What file formats for
transmission? What code lists are used?
10
Eurostat
PRACTICE
- An overview of the data / metadata as well as the basic
source of information for the DSD:
- What is the frequency of the data flows? What will be
used to build the DSDs? Existing questionnaires,
legislation, methodologies …?
- An overview of stakeholders, both external and internal:
- Who is involved? What agreements are in place?
Who are the data providers?
11
Eurostat
THEORY
At the end of the preparatory phase all stakeholders should
be clear about:
- The goals of the project;
- The timeline for implementation;
- A draft of the project plan; and
- Their roles and responsibilities.
12
Eurostat
THEORY
This is the phase of the project during which:
- The groundwork is carried out;
- The system is designed; and
- The workflow is sequenced.
It is the phase during which a high investment of time is
likely to be required, in part because loops back into the
preparation tasks are likely.
13
Eurostat
PRACTICE
The following steps are taken in the compliance phase:
- Analysis of the current data exchange;
- Decision on the appropriate structure for the exchanged
data;
- Application of SDMX statistical guidelines; cross-domain
concepts, re-use of existing code lists
(http://sdmx.org/?page_id=11 );
14
Eurostat
PRACTICE
The following steps are taken in the compliance phase:
- Design, then agreement, then creation of DSDs and
MSDs;
- Development of draft maintenance agreements;
15
Eurostat
'Coming together is a beginning.
Keeping together is progress.
Working together is success.'
Henry Ford
16
Eurostat
THEORY
This is the phase of the project during which:
- Plans are implemented;
- Communication with stakeholders is emphasised;
- Problems revealed / solutions sought / corrective action
taken;
17
Eurostat
PRACTICE
The following steps are taken in the implementation phase:
- SDMX artefacts are uploaded to the registry –
particularly DSDs;
- Appropriate IT infrastructure for SDMX compliant
transmission established;
- Pilot projects conducted:
- DSD structure reviewed, and
- IT infrastructure tested;
- …
18
Eurostat
- …;
- Feedback from testing incorporated;
- Finalise DSDs and maintenance agreement;
- Roll-out schedule agreed;
- Support provided through formal training events,
helpdesk services, capacity building etc.
19
Eurostat
The following steps are taken in the production phase:
- SDMX compliant data exchange used in production;
- Maintenance of the SDMX artefacts at regular intervals;
20
Eurostat
21
Eurostat
WHAT IS GOVERNANCE ?
A common and shared effort by all SDMX stakeholders to define
clear organisational and operational rules and arrangements to
which everyone should abide.
More simply,
• Guidance: recommendations, best practices;
• Organisation: rules, administration, control, law, authority etc.
22
If your experiment needs statistics, you ought to have done a better experiment (E. RUTHERFORD)
Eurostat
Why is governance important ?
• Establish a quality threshold
• Save time by avoiding uncertainty on what to do
• Have an agreed, central, unambiguous
reference on what is allowed and what are the
best practices
• Fit the needs of the SDMX community
• Support standardisation
23
Definition of statistics : The science of producing unreliable facts from reliable figures (E. ESAR)
Eurostat
GOVERNANCE BODIES
SDMX
governance bodies
Project
governance bodies
24
Eurostat
MAINTENANCE AGREEMENTS FOR GLOBAL DSDs
General arrangements (Example from National Accounts DSD)
• Three types of changes are foreseen:
- structural changes,
- annual maintenance changes and
- fast-track maintenance changes
• Agreed timetable established for change requests
25
Eurostat