ESSnet “Preparation of standardisation”
Analysis of methodological handbook:
“Survey Methods and Practices”
ISTAT
Rome, 6-7 June 2011
Essnet STAND-PREP Rome, 6-7 June 2011
Analysis of methodological handbook “Survey Method and Practices”
Attributes of the entity “normative document”
Title of the document (of the handbook)
ISTAT, ONS: Survey methods and practices
Version: indication of the version, if available
(otherwise, year of publishing)
2010
Type: standard, pre-standard, technical
specification, code of practice, regulation
ISTAT: Code of practice
ONS: Technical specification
Level: international (=global), regional (e.g.:
European), national, provincial
ISTAT, ONS: National
Body: organisation(s) responsible for the normative
document
ISTAT, ONS: Statistics Canada
Consensus: if any, indication of the agreement
ISTAT, ONS: Statistics Canada
Aim:
ISTAT: This manual is primarily a practical guide
to survey planning, design and implementation. It
covers many of the issues related to survey taking
and many of the basic methods that can be
usefully incorporated into the design and
implementation of a survey.
ONS: A practical guide to survey planning, design
and implementation
Essnet STAND-PREP Rome, 6-7 June 2011
Analysis of methodological handbook “Survey Method and Practices”
Total number of chapters
13
Chapters analysed by ISTAT
4. Data Collection Methods
5. Questionnaire Design
6. Sample Designs
7. Estimation
8. Sample Size Determination and
Allocation
10. Processing
Chapters analysed by ONS
1. Introduction to Surveys
2. Formulation of the Statement of
Objectives
3. Introduction to Survey Design
4. Data Collection Methods
5. Questionnaire Design
Essnet STAND-PREP Rome, 6-7 June 2011
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 4. Data Collection Methods
Summary
ISTAT
ONS
Total number of
Provision Groups
1
Main GSBPM
2.3. Design data collection 2.3. Design data collection
methodology
methodology
Secondary GSBPM
3.1.Build data collection
instrument
3.1. Build data collection
instrument
Collocation
Chapter 4
Chapter 4
Number of identified
Provisions
5
1 statement
3 instructions
1 recommendation
14
3
9
Methodologies
Essnet STAND-PREP Rome, 6-7 June 2011
1
4 statement
10 descriptive provision
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 4. Data Collection Methods
Provision Group 1
Provision Group 1
ISTAT
ONS
Main GSBPM
2.3. Design data collection 2.3. Design data collection
methodology
methodology
Secondary GSBPM
3.1. Build data collection
instrument
3.1. Build data collection
instrument
Collocation
Chapter 4.
Chapter 4.
Description
Provision related to
Data Collection
Methods
Data Collection
Methods
Number of identified
Provisions
5
1 statement
3 instructions
1 recommendation
14
3
9
Methodologies
Essnet STAND-PREP Rome, 6-7 June 2011
4 statement
10 descriptive provision
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 4. Data Collection Methods – Pag.37
Data collection is the process of gathering the required information for each selected unit in the survey. During data collection,
members of the population – be they individuals or organisations – are located and contacted and their participation in the survey is
sought. A questionnaire is then administered and answers recorded. This process is expensive, time consuming, requires extensive
resources and has a direct impact on data quality. Since it is the general public’s main contact with the statistical agency, it
contributes to the image of the statistical agency and has a broad impact on the agency’s relevance and on the quality of its data.
During the planning phase of a survey, many decisions must be made regarding the method of data collection. Should the
questionnaire be administered by an interviewer? If yes, should the interview be conducted in person or over the telephone? Should a
combination of methods be used – should respondents fill out the questionnaire themselves and nonrespondents be followed-up with
a telephone interview? Should the questionnaire be paper or computer-based? Should administrative data be used to collect some of
the survey data? Should data collection for several surveys be combined?
The method of data collection should be chosen to achieve a high participation rate and collect data that are as complete and
accurate as possible while minimising the burden to the respondent and satisfying the client’s budget and operational constraints.
The method of data collection should be chosen to achieve a high participation rate and collect data that are as complete and
accurate as possible while minimising the burden to the respondent and satisfying the client’s budget and operational constraints.
The purpose of this chapter is to present the various methods of data collection – including selfenumeration, interviewer-assisted,
computer-assisted, administrative data and others – and the criteria for deciding which method is most appropriate. Data collection
operations in general (with an emphasis on interviewer-assisted methods), including how to locate the sampling units, elicit cooperation and capture responses are covered in Chapter 9 - Data Collection Operations.
4.1 Basic Data Collection Methods
The basic methods of data collection are:
i. Self-EnumerationWith self-enumeration, the respondent completes the questionnaire without the assistance of an interviewer. There are a variety of
ways that the questionnaire can be delivered to and returned by the respondent: by post or facsimile, electronically (including the
Internet) or by an enumerator. (If the questionnaire is returned by facsimile or electronically, then a secure line or encryption is
needed to ensure the confidentiality of respondent data). When paper-based, this method is called Paper and Pencil Interviewing
(PAPI), when computer-based it is called Computer-Assisted Self Interviewing (CASI).
ii. Interviewer-assisted (Personal Interviews or Telephone Interviews)
Essnet STAND-PREP Rome, 6-7 June 2011
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 4. Data Collection Methods
ISTAT
ONS
Provision group
1
1
Provision
1
1
Collocation
Description
Page 37
Data collection is the process of
gathering the required
information for each selected
unit in the survey
Page 37
Data Collection
Type
Sequence
Methodology
Description
statement
none
Data collection
Methodology related to the
process of gathering required
information for each selected
unit in the survey
statement
none
Quality dimension
timeliness, accuracy and
efficiency
all but timeliness and accuracy
Essnet STAND-PREP Rome, 6-7 June 2011
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 4. Data Collection Methods
ISTAT
ONS
Provision group
Provision
Collocation
1
2
Page 37
Description
The methods of data collection should be chosen to achieve Self-enumeration
a high participation rate and collect data that are as complete
and accurate as possible, while minimising the burden to the
respondent and satisfying the client's budget an operational
constraint
Type
Sequence
Methodology
Description
instruction
none
Data collection
Methodology related to the process of gathering required
information for each selected unit in the survey
statement
none
Quality
dimensions
Methods:
timeliness, accuracy and efficiency
all but timeliness
and accuracy
-
1
2
Page 37
Paper and pencil interviewing
Computer Aided Self Interviewing
Computer Assisted Telephone Interviewing
Computer Assisted Personal Interviewing
Essnet STAND-PREP Rome, 6-7 June 2011
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 4. Data Collection Methods
ISTAT
ONS
Provision group
1
1
Provision
4
2
Collocation
Description
Page 38
Self-enumeration methods require a
very well-structured, easy to follow
questionnaire with clear instructions for
respondent
instruction
none
Self-interviewing
With self-enumeration, the respondent
completes the questionnaire without the
assistance of an interviewer
timeliness, accuracy and efficiency
Page 37
Self-enumeration
Type
Sequence
Methodology
Description
Quality dimension
Methods:
-
Paper and pencil interviewing
Computer Aided Self Interviewing
Essnet STAND-PREP Rome, 6-7 June 2011
statement
none
all but timeliness and
accuracy
ISTAT Provisions
ISTAT Methodologies
ONS Provisions
ONS Methodologies
Data collection is the process of gathering the
required information for each selected unit in
the survey
Methodology related to the
process of gathering required
information for each selected
unit in the survey
(Methods: PAPI, CASI, CAPI,
CATI)
Data collection
The methods of data collection should be
chosen to achieve a high participation rate
and collect data that are as complete and
accurate as possible, while minimising the
burden to the respondent and satisfying the
client's budget an operational constraint
Data collection
Methodology related to the
process
of gathering the required
information for each selected
unit
(Methods: PAPI, CASI, CAPI,
CATI)
Self-enumeration
Issues that must be considered when
selecting a method of data collection:
- Collection information available on the survey
frame
- Characteristics of the target population
-Nature of the questions being asked
-Available resources
-How easy the questionnaire is to complete
-Privacy considerations
-Data quality requirements
Data collection
Methodology related to the
process
of gathering the required
information for each selected
unit
(Methods: PAPI, CASI, CAPI,
CATI)
Interviewer assisted:
personal interviews
Self enumeration methods require a very
well-structured, easy to follow questionnaire
with clear instructions for respondent
Self interviewing
With self-enumeration, the
respondent completes the
questionnaire without the
assistance
(Methods: PAPI, CASI)
Interviewer assisted:
telephone interviews
Interviewer assisted methods are very useful
for survey populations with low literacy rates
or
when concepts or questionnaire are complex
Interviewer assisted
The questionnaire is
administrated
by an interviewer
(Methods: CAPI, CATI)
Self-enumeration
Self-enumeration
Description of selfenumeration methods,
advantages and
Disadvantages
Interviewer-assisted
Methods
Interviewer-assisted
Provides a description of
interviewer assisted methods
of data collection
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 4. Data Collection Methods
In general:
 ISTAT identified a small number of provisions each of them
associated to methodologies
 ONS identified more provisions, often corresponding to the
methodologies defined by ISTAT.
Essnet STAND-PREP Rome, 6-7 June 2011
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 5. Questionnaire design
Summary
ISTAT
Total number of
Provision groups
ONS
6
Main GSBPM
2.3. Design data
collection methodology
1
2.3. Design data
collection methodology
Secondary GSBPM 3.1. Build data collection
instrument
3.1. Build data collection
instrument
Collocation
Chapter 5.
Chapter 5.
Number of identified
Provisions
41
3 statement
20 instructions
18 recommendation
26
1
19
Methodologies
Essnet STAND-PREP Rome, 6-7 June 2011
8 statement
18 descriptive provision
Analysis of methodological handbook “Survey Method and Practices”
Comparative analysis of Chapter 5. Questionnaire design
In general:
 Istat defined the procedure of designing questionnarie as a
series of steps (groups of provisions) with a lot of associated
provisions of different types (statements, instructions and
recommendations) and only one methodology relating to
questionnaire testing
 ONS defined a single group of provisions with many associated
provisions specified by the related methodologies
Essnet STAND-PREP Rome, 6-7 June 2011
Analysis of methodological handbook “Survey Method and Practices”
Example of analysis of Chapter 6. Sampling Designs
Provision group
Provision
Description of Provision
1. Non-Probability Sampling
Provision 1 Pages 87-90
Non-probability sampling is a method of selecting
units from a population using a subjective (i.e.,
non random) method.
2. Probability Sampling
Provision 1 Pages 91-116
Probability sampling is a method of
sampling that allows inferences to be
made about the population based on
observations from a sample…
Type
Methodology
Description
Statement
Non-probability sampling
Non-probability sampling is a method of selecting
units from a population using a subjective (i.e.,
non random) method
Statement
Probability sampling
Probability sampling is a method of
sampling that allows inferences to be
made about the population based on
observations from a sample
Quality dimension
Methods:
Provision
Description of Provision
Accuracy and efficiency
6 Methods
Provision 2 Pages 88
Non-probability sampling can be applied to
studies that are used as:
- an idea generating tool;
- a preliminary step towards the development of a
probability sample survey;
a follow-up step to help understand the
results of a probability sample survey.
Accuracy and efficiency
10 Methods
Type
Methodology
Quality dimension
Recommendation
Non-probability sampling
Accuracy and efficiency
Essnet STAND-PREP Rome, 6-7 June 2011