Methods for
DataIntegration
Ton de Waal
14 March, 2017
Overview of project
• Project “Estimation methods for the integration of administrative
sources”
• Aim of the project: identifying and presenting statistical methods
for the integration of administrative data into a statistical
production system
• April 2016 – (end of) March 2017
• Part of ESS.VIP Admin Project (Work package 2 “Statistical
methods”, Lot 1 “Methodological support”)
• Sogeti is main contractor
• Experts from ISTAT, Statistics Netherlands and University of
Southampton
Overview of project
• Task 1: Specify usages of admin data
• Task 2: Identification and description of statistical tasks where using
estimation methods can be envisaged in order to integrate
administrative sources
• Task 3: Comprehensive identification and enumeration of possible
estimation methods that could be used for cases identified in Task 2
• Task 4: Literature review presenting actual examples for types of
usage and tasks identified in Task 2 or 3
• Task 5: Methods description
• Task 6 & 7: Final presentation and report
Task 1: Usages of admin data: Direct
• Direct Tabulation: Admin data used to produce statistics without
resorting to any statistical data.
‒ Exploiting only one administrative data source
‒ Exploiting multiple administrative data sources
• Substitution and supplementation for direct collection: Admin data
directly used as input but are not sufficient for achieving all
objectives
‒ Split-population approach Population is split into two or more
parts. Admin data used for units where these data are of
sufficient quality, and statistical sources used for the remainder
of the units
‒ Split data approach Administrative data used to provide some of
the variables for all population units
Task 1: Usages of admin data: Indirect
•
•
•
•
Creation and maintenance of registers and survey frames
•
Identification of frame units and their connections to population elements
•
Identification of classification and auxiliary variables
Editing and imputation
•
Construction of edit rules
•
Construction of models to find errors in data
•
Auxiliary data to construct imputation models
Indirect estimation
•
Creation of population benchmarks to be used for calibration
•
Use administrative data in a predictive setting
•
Estimation where administrative and statistical data are used on an equal footing
Data validation/confrontation
•
Validation of survey estimates and/or other administrative data sources
•
Address quality issues
Task 2: Possible statistical tasks
•
We have matched statistical tasks to usages by means of GSBPM
•
Statistical tasks for using integrated administrative data
I. Data editing and imputation
II. Creation of joint statistical micro data
a) Data linkage: Identification of the set of unique units residing in multiple datasets
b) Statistical matching: Inference of joint distribution based on marginal observations
III. Alignment of statistical data
a) Alignment of units: Harmonization of relevant units, creation of target statistical units
b) Alignment of measurements: Harmonization of variables, derivation of target variables
IV. Multisource estimation at aggregated level
a) Population size estimation: multiple lists with imperfect coverage of target population
b) Univalent estimation: numerically consistent estimation of common variables
c) Coherent estimation: aggregates that relate to each other
Task 3: Possible estimation methods
I. Data editing and imputation
• Most methods usually applied for surveys can also be applied for
Admin data
• There are editing methods developed specifically for data obtained
through an integration process (micro-integration)
II. Creation of joint statistical microdata
• Identification of the set of unique units residing in multiple
datasets and probabilistic record linkage
• Inference of joint distribution based on marginal observations
(statistical matching)
Task 3: Possible estimation methods
III. Alignment of statistical data
• a) Alignment of units
• b) Alignment of measurements: recently, latent variable models
have been proposed
IV. Multisource estimation at aggregated level
a) Population size estimation: multiple lists with imperfect coverage of
population
b) Univalent estimation: numerically consistent estimation of common
variables
• Obtaining univalent estimates at cross-sectional level
• Obtaining univalent estimates at longitudinal level
c) Coherent estimation: aggregates that relate to each other in terms
of accounting equations
Task 4: Examples at NSIs
• We have focused on those examples that offer most interesting
information
• We have given actual examples in NSIs for
• Direct tabulation
• Split data approach
• Indirect estimation
• Data validation
Task 4: Examples at NSIs
• Direct tabulation:
Use of probabilistic record linkage for statistics on victims and
injured people of road accidents (Istat)
• Spit data approach:
Creation of a social policy simulation database by means of
statistical matching (Statistics Canada)
• Indirect estimation:
Use of repeated weighting and macro-integration for the
construction of the Dutch Population and Housing Census (Statistics
Netherlands)
• Data validation:
Estimation of classification errors in admin and survey variables on
home ownership (Statistics Netherlands)
Task 5: Methods description
We will describe
• Editing and imputation methods, including micro-integration
• Methods for creation of joint microdata, including probabilistic
linkage and statistical matching
• Methods for alignment of statistical data, including latent variable
models
• Methods for multi source estimation at aggregated level, including
multiple lists with imperfect coverage of population, and methods for
obtaining univalent estimates (at cross-sectional level and at
longitudinal level)
Thank you
• Thank you for your attention
• Any questions or comments?