Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
A Strategy for Prioritising Non-Response Follow-Up
to Reduce Costs Without Reducing Output Quality
Gareth James
UK Office for National Statistics
Abstract
•
In Office for National Statistics (ONS) business
surveys, a considerable amount of resource is spent
following-up sampled businesses that either delay their
response or do not respond at all. There are pressures
to economise by reducing the amount of follow-up but there are unavoidable quality implications.
•
ONS is investigating methods for targeting nonresponders. Current practice in ONS is to aim for an
overall response rate by survey. We aim to develop a
mechanism that will allow us to prioritise follow-up in
terms of pre-defined quality criteria. Whatever strategy
is employed will require a better understanding of, and
treatment for, non-response; performance measurement
will depend on the quality of imputation for nonresponse. Other issues are: treatment of different types
of non-responders - ie new, large, and chronic nonresponders; and how many small businesses need
following up to avoid bias.
Any inherent differences between the respondents
and non-respondents can lead to non-response bias
being introduced to the estimates. The degree of
this depends on whether values are missing at
random (MAR), missing completely at random
(MCAR) or not missing at random (NMAR) as
described by Little and Rubin (1987).
Failure to address non-response from businesses
can lead to the ingraining of bad habits when the
same businesses are selected on future occasions
for that, or other, surveys. In other words,
allowing non-response in one wave of the survey
may make non-response more likely in later
waves.
ONS conducts many sample surveys of business, and
most are statutory (ie businesses that receive
questionnaires have a legal obligation to comply by
providing requested information). However, nonresponse still exists. Continual improvements are made
to ONS surveys, which should result improved
response, and effort is spent on re-contacting
businesses that haven’t responded to try to gain their
co-operation.
Keywords: Non-response, Bias, Business Survey,
Scoring, Follow-up
1. Introduction
The initiatives which are used to develop the survey
and, in turn, enhance response are often focussed on
questionnaire improvements and the sample design.
Examples of changes to the sample design include
altering rotation patterns so that businesses are retained
in the sample for fewer periods, better control of
overlap between samples of different surveys, and
making changes to the sample allocation itself. The
work conducted on data collection instruments (paper
questionnaires are used for most ONS businesses
surveys) can lead to less burden on businesses and
makes a response more likely. In particular, effort is
made to improve businesses’ understanding of
questions and definitions; improve the layout and
routing of questionnaires; and reduce the number of
questions being asked. Additionally, in a few surveys,
some businesses are offered ways to respond other
than completing a paper questionnaire; this includes
completion of a spreadsheet (for some complex
surveys or businesses) or telephone data entry.
Non-response is an issue which affects many, if not
most, sample surveys. Two different forms are often
defined: unit non-response (when a sampled statistical
unit fails to respond at all to the survey), and item nonresponse (when the unit responds to some but not all of
the questions asked in the survey). Either way, nonresponse means that a full sample is not achieved for
the question under consideration.
There are a number of consequences that can arise
from non-response, many of these particularly relevant
to the periodic, repeating business surveys of the
Office for National Statistics (ONS), which comprise
panel and rotating-sample elements in their stratified
designs.
• The effective (returned) sample size is smaller in
the presence of non-response, leading to less
precise estimates. This can be particularly
pronounced in sampling strata that have small
allocated despatch sample sizes.
On a more routine basis, non-responding businesses
are re-contacted during the course of the survey
1408
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
processing cycle. It is only through action such as this,
that response rates can be improved once the
questionnaires have been despatched. Further details
on the current strategy for prioritising responsechasing are described in Section 2.1.
register turnover (that being the auxiliary variable
‘turnover’ held on the sampling frame (the Interdepartment Business Register, or IDBR)).
This rest of this paper describes recent work (it is still
in progress) at ONS to better understand non-response
in business surveys and the impact that it has on
estimates. The overall aim is to develop a new strategy
for response-chasing that will be more efficient than
the current one: it should deliver better quality results
for the same inputs. To do this, information gathered
about non-response will be included in a new method
of prioritising non-responding businesses for follow-up
through the development of a scoring system. This
work is not yet complete, but the initial results are
presented here.
A previous analysis of responses to businesses surveys
had revealed a number of patterns which were used in
designing a intensive follow-up exercise (Section 3)
and will also be used in the specification of a scoring
system to prioritise response-chasing.
2.2 Analysis of response-rates
The main results from the analysis were as follows.
• Particular industries were identified across a range
of surveys and time periods as suffering from
lower response-rates. These industries included
Catering and Hotels, and the evidence was
supported by experience from staff who work on
the response-chasing teams. Possible reasons for
lower response include establishments that are not
open for custom during the day, which makes
contacting them more difficult; missing or
incorrect contact details (an issue which is more
prevalent amongst smaller businesses); and
possible language barriers or communication
problems.
• Bigger businesses (in terms of employment)
tended to take longer to respond to ONS surveys.
Conversely, smaller businesses tended to return
questionnaires earlier, which may be reflect the
greater ease with which small businesses can
provide data. However, when reviewed later in the
survey processing cycle, it was apparent that
bigger businesses had the higher response rate,
although that is somewhat due to the current
response-chasing policy of targeting bigger
businesses before smaller ones when responsechasing.
• The relative ordering of industries by responserate remained fairly stable throughout the survey
processing cycle. In other words, those industries
which had relatively high-response rates at the
early stages of the processing cycle, tended to also
have the relatively high rates towards the end of
the period.
2. Current Response-chasing Practice in ONS
Business Surveys
2.1 Overview of current practice at ONS
In most ONS business surveys, reminder letters are
sent to all non-responding businesses, some of which
are then followed-up by telephone calls. The number
of standard reminder letters varies between surveys,
and in some cases the number sent has been cut in
recent years to help meet budget constraints.
Additional action is sometimes taken in the case of
particular non-respondents (especially those that have
persistently failed to respond); further reminder letters
are sent to the chief executive of the business, and
legal enforcement action could be taken as a last resort.
Response rate targets are used by most ONS business
surveys. These are essentially the ratio of return-todespatch sampled numbers of businesses, but adjusted
for any out-of-scope units. However, the targets
usually don’t take account of the quality or usability of
the information returned. Targets are often set overall
for the survey, although in practice a balance is struck
by trying to achieve ‘reasonable’ response rates in all
of the major output groups (eg if the overall target for
the survey is 80% response, then a similar target would
be set for each main output within the survey).
3. Intensive Follow-Up Study
A more recent development has been the introduction
of weighted response rate targets, where the weights
are derived from auxiliary information held about each
business. An example is the Monthly Inquiry into the
Distribution and Services Sector (MIDSS), which
formerly had a response rate target of 80% in terms of
counts of businesses, but has recently started a trial of
using joint targets of 75% by count and 80% by
3.1 Introduction to the study
To learn more about reasons for non-response and to
provide data that could be used to assess whether nonresponse bias might be present in survey estimates, a
small-scale Intensive Follow-Up (IFU) study of
MIDSS turnover data was run during winter 2006/7. In
the exercise, selected businesses that had failed to
1409
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
respond were targeted for further action in an attempt
to gain responses that wouldn’t have otherwise been
collected by the survey. Those businesses were
subjected to several more telephone calls to try to elicit
a response. Once additional responses had been
obtained, the survey estimates could be re-worked and
the impact of the extra responses measured, giving an
indication of non-response bias.
3.2 Results of the IFU
3.2.1 Background and summary information
MIDSS is a monthly survey of businesses and collects
only a few variables (often just one, total turnover). At
the time of the IFU, the total monthly sample size was
approximately 30,000. Once each quarter (reference
months March, June, September and December), an
additional question about employment is asked to a
sub-sample of 20,000 businesses. MIDSS is stratified
by industry (according to groups of Standard Industrial
Classification (2003) codes and into employment size
bands, according to register employment). Ratio
estimation is used for monthly total turnover. MIDSS
was selected for the IFU due to its relatively simple
nature and ONS-standard estimation methods.
So as not to interfere with the usual processes of the
survey operation, an additional team of staff was
employed to conduct the IFU telephone calls. To
distinguish more easily those returns that would have
come in anyway, from those that came only as a result
of the IFU, the exercise took place after the closedown of the survey processing cycle for the
compilation of first estimates.
Only part of the MIDSS sample was chosen for the
IFU. Particular industries were targeted, and different
industries were included in the exercises in each of the
three months in which the IFU ran. The choice of
industries was based upon an evaluation of the likely
effect of non-response; all industries in MIDSS were
first stratified into three groups based upon the
variability of returned data and typical response rates.
Table 1 gives summary information about the IFU and
the questionnaires which were returned.
Survey
reference
month
Number of
nonresponding
businesses
selected for
IFU
IFU
businesses
successfully
contacted
(and as %
of selected)
IFU
businesses
which
returned
data (and as
% of
contacted)
Oct 2006
856
590 (69%)
425 (72%)
Nov 2006
783
557 (71%)
416 (75%)
Dec 2006
792
550 (69%)
356 (65%)
All
2431
1697 (70%) 1197 (71%)
Table 1: IFU summary information, relating to MIDSS
Stratum 1: High variability (of data)
Stratum 2: Low variability and low response-rate
Stratum 3: Low variability and high response-rate
Only a small number of industries fell in Stratum 1,
and all of these were selected as any non-response
could have a big impact on results; about half the
industries in Stratum 2 were selected and only a few
from Stratum 3. In Strata 2 and 3 the industries chosen
for the IFU were selected at random. The total number
of industries selected meant that it was possible to
schedule follow-up calls to all the non-responding
businesses within the industry, and a schedule for
calling was planned with reference to information
learned from a similar exercise on a different survey at
ONS a few years ago.
It was not possible to contact all business selected for
the IFU; about 70 per cent of business were
successfully contacted. Though a breakdown of
reasons for failing to achieve 100 per cent contact is
not available, they can be attributed to some businesses
having ‘died’ but still remaining on the register; some
having out-of-date or otherwise incorrect contact
details; or various other reasons.
The response rate from those businesses where a
contact was made varied over the months between 65
and 75 per cent. It was anticipated that more returns
would be achieved, but the relatively low response rate
shows how difficult it can be to obtain responses from
a hard-core element of non-responders. The somewhatlow response rate means that the hoped-for analysis to
determine non-response bias is not possible as that
would require 100 per cent (or close) response;
therefore only an analysis of the relative non-response
bias is possible.
It was planned for businesses to receive up to five
telephone calls as part of the IFU (therefore these calls
were in addition to any they had received as part of the
usual survey process). Successive calls were timed to
be a few days apart. For each successful contact, the
ONS staff member would record the length of call and
the nature of the call (whether positive: the business
contact was courteous, helpful, etc.; neutral; or
negative: aggressive, irritated, unhelpful). In addition,
on the first successful contact, the business would be
asked the reason for its failing to respond originally to
the survey.
1410
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
3.2.2 Reasons given for failing to respond
had been completed – this will be referred to as the
‘final’ data set. In the period between these times,
additional, actual returns had been added to the data
file, replacing values which had been imputed
previously. In turn, the imputations for businesses still
outstanding may also have changed (since the values
imputed for non-respondents are based on the set of
returns of responding businesses). It is also possible for
data already received to be amended if errors are
detected or other information becomes available.
Although all businesses contacted in the IFU were
asked the reason for them failing to respond originally,
not all gave one. Table 2 gives a breakdown of the
reasons where they were given and the corresponding
success rate in gaining a survey response:
Reason for not
responding
Frequency
% of businesses
that then returned
data
85%
Figure 1 shows the percentage difference in industry
estimates between first and final data sets. Industries
are defined here as those at the level of sampling, and
only industries not included in the IFU are in Figure 1;
data from all three months have been combined. For
the non-IFU industries, no additional calls were made
after the usual response-chasing had been completed,
so any additional responses received were not as a
result of IFU response-chasing action, and this
distribution provides a control group for comparison
with industries included in the IFU.
Forgot/ missed
573
date
Too busy to
343
77%
respond
Actively decided
56
39%
not to respond
Table 2: Reasons given for not responding to MIDSS
Of most interest is the relatively low proportion of
questionnaires returned from those businesses which
had consciously decided not to return the
questionnaires originally. However, to have turnedaround over one-third of such businesses could also be
considered something of a success.
F
r
e
q
u
e
n
c
y
3.2.3 Other information recorded
The attitude of the business contact was found to be
generally positive (about 80 per cent of calls); the next
biggest group (about 15 per cent) were neutral and the
other five per cent were negative. These proportions
remained fairly constant with respect to each round of
calls (first call, second call, etc.) whereas one might
have expected businesses to become more irritated
after receiving several response-chasing calls.
100
50
0
- 32
- 24
- 16
-8
0
8
16
pc_change_est
Figure 1: Difference in first and final data set estimates for
the non-IFU industries. Summary measures of the
distribution are: mean = 0.44%, median = 0.48%, standard
deviation = 4.16%.
The duration of calls also seemed not to vary much
with the call-round number. Overall, most calls (about
85 per cent) lasted one to two minutes, about 15 per
cent lasted between three and five minutes (duration
was usually recorded to the nearest whole minute), and
five per cent lasted six minutes or more.
The percentage change in estimates due to later
responses of the 36 industries included in the IFU
study over the three months is shown in Figure 2.
F
r
e
q
u
e
n
c
y
3.2.4 Estimation of non-response bias
Two approaches have been used to assess whether any
significant non-response bias might be present in the
estimates of total turnover from MIDSS.
10
5
0
- 32
- 24
- 16
-8
0
8
16
pc_change_est
The first approach compared the estimates of turnover
at two different points in time: the first data set was
that frozen at the time of the close-down of the survey
for compilation of first estimates (ie after the usual
response-chasing has taken place but before the IFU
commenced), and the second some time after the IFU
Figure 2: Difference in first and final data set estimates for
the IFU industries. One outlier (-198%) is not shown – it was
subsequently found to be caused by a data error - with this
point removed, the summary measures of the distribution are:
mean = 0.90%, median = 0.80%, standard deviation = 5.00%.
1411
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
There is no significant difference between the
distributions of changes in estimates for the non-IFU
industries and the IFU industries. However, there are
some observations (industries) in the tails of the
distributions (for both IFU and non-IFU industries)
that are notable for the absolute size of the revision,
suggesting that getting a higher response rate, or
perhaps getting a response from particular contributors,
is important for estimates in particular industries.
over-burdening the smallest businesses. However, the
process of selecting ‘key’ businesses is not necessarily
a scientific one: it is left to the survey manager to
identify the key businesses using whatever analysis she
or he wishes, together with expert knowledge and
experience of the survey. This means that the process
is not particularly transparent, varies from survey-tosurvey and can lead to an ever-expanding number of
key businesses, as the markers used to denoted key
businesses tend not to be removed as frequently as they
are applied.
The biggest change in the estimate of total turnover, a
198% decrease, came from an industry in the IFU.
Surprisingly, it was a Stratum 3 (low variability, high
response) industry. Upon investigation, however, it
was discovered that almost all of the difference could
be attributed to error in the data being detected and
corrected; the large change was not caused by
additional responses being received. Similar examples
have been found in other observations too.
The overall aim of scoring businesses is two-fold. The
first is to systematise the process of response-chasing,
thus removing the need for the survey manager to
identify ‘key’ businesses. Even if those businesses
chosen by the survey manager really are the most
important, using scoring would give a more defensible,
transparent system. The second aim is to select
businesses in a more scientific and efficient way, so
that only those businesses whose response is important
(and there are many dimensions to this) are targeted.
The second approach taken to assess possible nonresponse bias was an analysis of the auxiliary
information held on the IDBR of responding and nonresponding businesses.
Once scores are derived for each business (whether for
all units at the time of selection, or just the nonrespondents) they can be used for a number of
purposes. The first is to prioritise the non-responding
businesses (possibly into groups or individually) for
follow-up, and a second is to categorise businesses for
different follow-up treatments, eg phone calls, e-mail
reminders, chief executive letters.
Analysis of the differences in auxiliary information of
those industries included in the IFU was conducted
within each industry-by-size-band sampling stratum
(so that the effect of chasing the biggest businesses
first is removed). The analysis revealed some
significant differences in the auxiliary information of
respondents and non-respondents in some strata, but
there was no discernible pattern in terms of the timing
(ie whether using the first or final data sets), the
variable analysed (employment or turnover) or the
employment size band across industries. The main
notable observation was that, in almost every case
where the difference was significant, the average value
of the auxiliary variables of the respondents was larger
than that of the non-respondents.
3.3.2 Non-response bias in ONS business surveys
This section is written with reference to MIDSS, but
the principles also apply to other ONS business
surveys as many processes are common.
In MIDSS, where a response is not received from a
business selected for the sample, a value is imputed in
its place. Ratio imputation is used to create a value: the
average growth from the previous period to the current
period of businesses in the same stratum that did
respond is applied to the previous-period survey value
of each non-responding business. In cases where the
business is new to the sample, a value is constructed
based upon its auxiliary information held on the IDBR.
The approach of imputing a value in each case of nonresponse (rather than re-weighting) is used as it is
thought to give more accurate estimates with the
auxiliary information available. Use of this approach
means that any error resulting from non-response
manifests itself as imputation error (ie the value
imputed being different from the value that would have
been returned if the business had responded).
3.3 Scoring methods to prioritise response-chasing
3.3.1 Introduction to scoring
The current response-chasing strategy for most ONS
business surveys is to first send written reminders to
businesses that have failed to respond to the survey.
After that, some of the non-responders are telephoned.
The priority order for doing so is first to target
businesses identified as ‘key’, and then to chase other
non-responding businesses by descending order of size
of register employment. This aim of this procedure is
to get responses from the most ‘important’ businesses
(which assumes that, after ‘keys’, the biggest
businesses are the most important ones) while not
1412
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
In recent work at ONS, trials have been conducted on
historical MIDSS data sets to test the feasibility of
scoring methods. Although the work is still in
progress, some preliminary results are available to
report here. The survey variable which has been used
in the trials is (monthly) total turnover, and will be
denoted by yi, where i represents the business i, i = 1,
…, n (here n ƒ 30,000, the full, despatched sample
size). An imputed value for business i will be denoted
ŷi.
imputation error cannot be known in advance, and
therefore must be predicted. Using historical data sets
has meant that such scores can be calculated and then
compared against reality to test the effectiveness of the
scoring system.
Once scores have been calculated there are several
ways of proceeding, but the route that ONS is likely to
take is to rank all non-responding business by their
scores and divide them into groups, chasing all
businesses within each group before moving onto the
next group. The reason for this is a practical one; it is
easier to implement it in this way when a team of
several staff members will be responsible for carrying
out the calls.
The estimate of total monthly turnover is given by
tˆy =
∑w y
i
*
i
i∈S
where:
wi represents the survey weight applied to the
response variable of business i, and is
calculated as the product of the design weight
and a calibration weight (since ratio
estimation is used);
An example of this type of method is described by
McKenzie (2000). In his paper he calculated scores
based upon imputation error in previous data returns,
ranked all businesses according to these scores, and
divided them into deciles denoted 0 (smallest scores, ie
lowest imputation error) to 9 (greatest imputation
error). Businesses which were new to the sample, or
were persistent non-responders (ie those that didn’t
have sufficient previous data returns to allow an
imputation score to be calculated) received a score of
10, and became the highest priority businesses for
response-chasing. These businesses were scored as top
priority to encourage good response-behaviour from
them on future occasions.
the response variable, yi*, is given by
⎧ yi if i ∈ S R
yi* = ⎨
⎩ yˆi if i ∈ S NR
where SR and SNR respectively denote the
responding and non-responding parts of the
sample at any given time.
The total potential imputation error is given by
wi ( yi − yˆi )
∑
A slightly modified method was tested on MIDSS at
ONS in 2001-2. However, problems were encountered
with the implementation, and in particular that too
many businesses were being scored ‘10’, and they
often weren’t the businesses that had the biggest
impact on the results. The trial was somewhat shortlived.
i∈S
ie the weighted sum of the differences between the
actual values that would be returned and the values that
would be imputed in the case of non-response. (Note
that in practice this is difficult to calculate or define
precisely as the values of the imputations depend upon
the set of other responses that have been received).
An example of a scoring method that has been
successfully implemented at Statistics Canada is
described by Daoust (2006). Weighted contributions of
businesses form the basis of the scores, from which
businesses are allocated to one of three groups for
follow-up: a group of businesses that must be
followed-up for a response, a group that won’t be
followed-up and a group which may be followed-up
depending on the resource available.
It is possible that the effect of individually large
imputation errors (which should be avoided where
possible) could be cancelled out by differing signs, and
therefore the quantity of interest for the trials was
chosen to be the total absolute potential imputation
error:
wi yi − yˆ i
∑
i∈S
3.3.3 Preliminary results from on-going work at ONS
A good scoring system will be defined as one that
would give the largest scores to the businesses which
are likely to suffer the greatest weighted imputation
error if they fail to respond. Therefore, the greatest
scores would be attributed to businesses with the
largest values of wi yi − yˆi . Of course, the actual
To test the effectiveness of different score functions,
the proposed methods have been applied to historical
MIDSS data sets. For comparison, the ‘actual’ absolute
imputation error has been calculated using true values
that were returned by businesses and by constructing
1413
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
the values that would have been imputed otherwise
(some simplifying assumptions were made to avoid the
iterative nature of imputation). The businesses were
then divided into quintiles based on their actual
imputation error and the total of the absolute
imputation errors within each quintile is shown in
Table 3.
to that score. Table 4 shows, for a range of score
functions, the percentage of the actual imputation error
contained within each quintile of the prediction score
function.
Score
quintile
Score
quintile
4
3
2
1
0
Actual
% of total absolute
imputation
error score
88
8
3
1
<< 1
4
3
2
1
0
Table 3: Distribution of imputation error scores (percentages)
Imp.
error
88
8
3
1
<< 1
Weighted
Prediction
Imp.
error
t-1
73
12
10
3
2
Reg.
turn.
Reg.
emp.
Unweighted
prediction
Reg.
emp.
68
15
8
5
4
42
20
11
9
18
40
15
12
18
15
Table 4: Percentages of actual imputation error contained
within each quintile of various prediction score functions,
including register turnover and employment.
The distribution has a positive skew, one-fifth of the
sample accounts for 88 per cent of the error.
Examination of the individual scores revealed that the
majority of the error comes from just a handful of
businesses, highlighting the need to ensure responses
are gained from those particular few businesses. In
practice, this means it is likely that a threshold will be
used to mark the most important units instead of taking
an arbitrary 20 per cent (or any other fixed proportion)
of businesses.
The best score function of those shown is the weighted
imputation error from the previous month, as its
distribution is most similar to the that of the actual
error. However, there is not much difference between
the effectiveness of that function and that of weighted
register turnover. The use of the latter is appealing,
because it is available directly from the IDBR – it does
not have to be computed and it is available for all
businesses (both aspects unlike previous imputation
error). Further, and more detailed investigation into the
properties of these score functions is planned before a
choice is made.
Different score functions have been tested on their
ability to predict the likely imputation error. Functions
included in the trials have included the previous
period’s imputation error (akin to McKenzie’s
methods), weighted register turnover (similar to
Daoust’s) and register employment (similar to the
current ONS strategy of biggest-first). In general, all
the scores functions take the generic form:
It can be seen that register employment is not so good
a predictor of imputation error for the survey variable
total turnover. When looking at these results, however,
one should bear in mind that particular businesses had
been identified as ‘key’ for MIDSS. These units have
not be treated any differently in the scoring trials
reported here – if they had, then the combined use of
‘keys’ and register employment would lead to more
efficient response-chasing. However, this approach
makes the system more complicated, and it appears
that a simpler and more scientific approach than the
current strategy is available through score functions.
scorei,t = f(wi, imp erri ,(t-1), imp erri ,(t-2), …,
reg empi,t, reg turni,t, …)
where the ‘imp err’ is the imputation error calculated at
a previous period, and ‘reg emp’ and ‘reg turn’ refer to
the auxiliary values of employment and turnover held
on the IDBR (sampling frame). A combination of these
variables – average, maximum, etc. – could also be
used.
3.3.4 Future refinements to scoring
Work will continue to refine the score functions and
decide on which is best to use in terms of the variables
and the number of periods of previous data to include.
Inclusion of further refinements to the score will also
be considered, and ideas for these are discussed.
Finally, once testing has concluded using historical
data sets, live testing is planned. This may well give
further insights to the effectiveness of the system to be
In the trials using MIDSS data, a score has been
computed for each business in the sample according to
the given prediction score function. For functions
which use previous imputation error, new-to-sample
businesses or persistent non-respondents have been
assigned a score equal to the stratum average of
businesses that had responded. Once all businesses
have been assigned a score, they are ranked according
1414
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
proposed, as the responses contained in the historical
data sets have all been collected under the current
response-chasing strategy.
The future direction and strategy for response-chasing
will build on these results, resulting in a scoring
system to identify the most important businesses to
improve the quality of survey estimates and outputs.
The proposed system should be simple to use and
justify, and will be more transparent and efficient than
the methods currently used. In addition, it will be
designed in conjunction with other survey processes,
such as sample design and editing tools.
A number of other issues and ideas will be considered
when developing the new response-chasing strategy
using score functions.
• Thresholds will be established to determine those
businesses that are much more important than
others.
• There is an option on whether to score the survey
as a whole, or separately within any specified subset. The effects will be tested, but scoring within
each of the main output groups seems likely.
• All businesses selected could be scored or only
those that have failed to respond.
• Additional changes to individual businesses’
scores may be included. An example would be to
increase the score of persistent non-respondents,
making them more likely to be subject to
response-chasing.
• The score function will be extended to cover
multivariate surveys; McKenzie (2000) gives
ideas on this.
• Scores will need to be updated during the survey
processing cycle to reflect changing priorities after
other returns have been received. For example, if
the three non-responding businesses in the same
stratum have the same, high-priority score, after
any two of them have returned data, getting a
response from the third might become much less
important. The frequency with which scores are
updated needs to be considered carefully.
• The system used by the response-chasing team
must be easy to use and clearly show which
businesses need to be chased and in what order. It
must also include a stopping mechanism, which
may be target-based. However, the use of targets
will be reviewed.
Acknowledgements
Many people have been involved with this project,
both in helping with the IFU study and in the analysis
of data. Particular thanks go to Sarah Green, Nicholas
Woodhill, Anthony Szary and Gary Brown of ONS,
and Pedro Silva from the University of Southampton
for their contributions to this paper and the
presentation at the conference.
References
Daoust, P., 2006, ‘Prioritizing follow-up of nonrespondents for the Canadian quarterly survey of
financial statistics for enterprises’, Conference of
European Statisticians
Little, R.J.A. and Rubin D.B., 1987, Statistical
Analysis with Missing Data, New York: John
Wiles & Sons
McKenzie, R., 2000, ‘A framework for priority contact
of non respondents’, Proceedings of the second
International Conference of Establishment
Surveys
4. Conclusions
This paper has described some of the work in progress
on measuring non-response bias in ONS business
surveys and has described some results from the initial
investigations carried out. Overall, it seems that
increasing all response-rate targets would not be an
efficient way to improve quality. Rather, carefully
targeted response-chasing would be a better approach.
It appears that getting responses from particular
businesses can impact heavily on industry estimates,
and that some industries are more prone to this than
others.
1415