Public Opinion Quarterly, Vol. 77, No. 3, Fall 2013, pp. 666–695
IMPROVING SURVEY PARTICIPATION
COST EFFECTIVENESS OF CALLBACKS TO REFUSALS
AND INCREASED CALL ATTEMPTS IN A NATIONAL
TELEPHONE SURVEY IN FRANCE
Abstract The general decrease in telephone survey response rates
leads to potential selection and estimation biases. As nonrespondents
can be broken down into noncontacts and refusals, different strategies
can be deployed—increasing the number of call attempts before abandoning a number, and calling back refusals/abandonments to persuade
them to participate. Using a two-stage random-digit-dialing sample of
8,645 individuals aged 15–49 for a survey on sexual and reproductive
health (SRH), we compared the effects of the two strategies: including hard-to-contact respondents (more than twenty call attempts with
no upper limit) and including respondents from two successive waves
of call-back among initial refusals/abandonments. Comparisons were
Stéphane Legleye is the head of the Survey and Sampling Department (SSD) at the National
Institute for Demographic Research (INED), Paris, France; he is also associate researcher at
the National Institute of Health and Medical Research (INSERM), University Paris-Sud and
University Paris Descartes, Paris, France. Géraldine Charrance and Nicolas Razafindratsima
are in the SSD, INED, Paris, France. Aline Bohet is on the Gender, Sexual, and Reproductive
Health (GRSH) team at the Centre for Research in Epidemiology and Population Health (CESP),
INSERM, Paris, France. Nathalie Bajos is research director on the GRSH team at the CESP,
and associate researcher at the INED, Paris, France. Caroline Moreau is assistant professor in
the Department of Population, Family, and Reproductive Health at the Johns Hopkins Bloomberg
School of Public Health, Baltimore, MD, USA, and senior researcher on the GRSH team at the
CESP, Paris, France. The authors would like to thank the respondents to the survey. The FECOND
survey was supported by a grant from the French Ministry of Health, a grant from the French
National Agency of Research [#ANR-08-BLAN-0286-01 to N. B. and C. M.], and funding
from the INSERM and the INED. The authors have no conflict of interest to declare. *Address
correspondence to Stéphane Legleye, INED, 133 Boulevard Davout, Paris, France; e-mail: stephane.
[email protected].
Advance Access publication September 30, 2013
doi:10.1093/poq/nft031
© The Author 2013. Published by Oxford University Press on behalf of the American Association for Public Opinion Research.
All rights reserved. For permissions, please e-mail: [email protected]
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
STÉPHANE LEGLEYE*
GÉRALDINE CHARRANCE
NICOLAS RAZAFINDRATSIMA
ALINE BOHET
NATHALIE BAJOS
CAROLINE MOREAU
Effectiveness of Call-Backs and Call Attempts
667
The telephone is theoretically a very efficient way to perform general population surveys in France: in 2011, more than 74 percent of people aged 12 and
over have both a landline and a mobile phone, 15 percent a landline only, and
10 percent a mobile phone only. Only one percent of the population is unreachable by phone (Bigot and Croutte 2011). The range of available telephone
services has grown more complex in recent decades. Not only has this share
of mobile phones increased rapidly, but also many households (58 percent)
use digital lines linked to Internet services. This evolution has contributed to
greater coverage of the general population but complicates the task of producing representative samples: there is no exhaustive unique directory of subscribers to landline or mobile phone services from which a random selection can
be drawn, and respondents can potentially be reached from multiple sources
(Brick et al. 2011; Gautier et al. 2006). However, the main difficulty confronting survey producers is the general decrease in participation rates, which has
been observed for all data-collection methods and particularly with the telephone (Lan 2009; Schouten, Cobben, and Bethlehem 2009). Nonresponse bias
results from a combination of two factors: the proportion of nonrespondents
(i.e., the rate of nonresponse) and the difference between nonrespondents and
respondents with respect to the survey’s variables of interest. High nonresponse rates are problematic only insofar as the difference between respondents and nonrespondents is large enough to affect a given estimate (Groves
2006). Unfortunately, correct estimation of nonresponse bias requires collecting information from nonrespondents on the survey’s questions of interest and
on their sociodemographic characteristics (which are expected to be linked to
the variables of interest). The first of these requirements is impossible to fulfill
by definition; as for the second, it is often very difficult to achieve in the case
of telephone surveys based on random-number generation, as numbers provide
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
based on sociodemographic bias, differences in SRH behaviors, multivariate logistic modeling of SRH behaviors, post-calibration weighting,
and cost estimation. The sociodemographic profile of hard-to-contact
and call-back respondents differed from that of easy-to-interview
respondents. Including hard-to-contact respondents decreased the sociodemographic bias of the sample, while including call-back respondents
increased it. Several significant differences in SRH behaviors emerged
between easy-to-interview and hard-to-contact respondents, but none
between first-wave and call-back respondents. Nevertheless, the determinants of SRH behaviors in call-back and hard-to-contact respondents
differed with respect to easy-to-interview respondents. The trade-off
between bias and financial costs suggests that the best protocol would
be to mix the two strategies but with only one call-back wave involving
a limited number of call attempts to achieve a sufficient sample size with
optimal quality.
668
Legleye et al.
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
no sociodemographic or geographical information on their possessors (only
half of landline subscribers are registered in a public directory that provides
an address, and no such directory exists for mobile subscribers). The correction of nonresponse bias thus depends on weighting the sample to match it
to a reference population (calibration process). This practice is based on the
hypothesis that respondents with similar characteristics to nonrespondents can
be substituted for them in the sample. Ideally, we should have a sample of nonrespondents against which to compare associations between sociodemographic
characteristics and key survey indicators.
Several studies have shown that there is little connection between nonresponse rates and the size of the nonresponse bias (Curtin, Presser, and Singer
2000; Keeter et al. 2000). But there are relatively few studies of this type, and
the results are potentially sensitive to national and historic contexts as well as
to the survey topic, so survey researchers must exercise caution when drawing on results from other countries. Survey researchers thus seek to include
methodological interventions in the data-collection process to ensure that the
households/individuals who are the hardest to interview are included in the
sample. These are based on the hypothesis that there is a continuum starting
from the respondents who respond most readily, followed by respondents who
require further efforts, and finally by nonrespondents—in other words, that
individuals who are hard to interview are similar to nonrespondents.
As nonrespondents can be broken down into two subpopulations—noncontacts
and refusals—different measures should be considered to reduce nonresponse.
The noncontact rate can be reduced by increasing the number of call attempts
before abandoning a number. The benefits of increasing the maximum number
of contact attempts have already been studied (Beck, Legleye, and Peretti-Watel
2005; Firdion 1993; Razafindratsima 2012), as well as the utility of including
late respondents (after 31 days) with early respondents in monthly surveillance
surveys (Qayad et al. 2010). These studies concluded that the sociodemographic
profiles of persons who are difficult to contact differ substantially from those of
other people. Differences in behaviors (drug use, sexual behaviors, political attitudes, health status and behaviors, etc.) are observed and are not always reducible
to the sociodemographic characteristics alone. But in these studies, a limit was
imposed on the total number of calls. Two main methods are used to lower the
refusal rate: call-backs to refusals and financial incentives (Groves, Singer, and
Corning 2000; Groves 2006). Financial incentives are still uncommon in public
research in France, due to a strong political tradition that values the free and voluntary participation in scientific research and public statistics as a contribution
to the public good, as opposed to the common practice of financial incentives in
commercial surveys conducted by private firms.
Trying to convert the refusals/abandonments has become common practice, including in France (Beck and Guilbert 2007), but no study has assessed
its utility. Using six national surveys conducted in the UK during the 1990s,
a study showed that efforts to convert initial refusals are less effective than
Effectiveness of Call-Backs and Call Attempts
669
Methods
The Survey
The FECOND (Fertility, Contraception, and Sexual Dysfunction) survey is
aimed at analyzing contemporary challenges in sexual and reproductive health
with a view to developing a comprehensive approach that simultaneously
explores a range of sexual and reproductive health (SRH) topics, from both
women’s and men’s viewpoints. The FECOND study has received the approval
of the relevant French government oversight agency (CNIL, the Commission
Nationale de l’Informatique et des Libertés) and CCTIRS, a consultative committee on the data processing regarding health research.
This survey was carried out by phone between June 2010 and January 2011,
with a one-month summer break in August 2010. The sample was obtained
by two-stage stratified sampling (household and interviewee) by phone type,
landline and mobile phones: the “mobile phone” group refers to people who
can be joined only by a mobile phone and not by landline, while the “landline”
group refers to people who can by joined by landline phone, whether they
possess a mobile phone or not. The first stage was the household sampling.
For this purpose, random-digit-dialing (RDD) was used to generate landline
and mobile phone numbers. Mobile phone numbers represented 16 percent of
the sample, which corresponds to the proportion of persons in the population
with a cell phone but no landline. To avoid double-counting between landline
and mobile phone samples, interviewers asked respondents at the beginning of
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
those aimed at hard-to-contact respondents in face-to-face surveys (Lynn and
Clarke 2002). But no similar study has been published for telephone surveys.
Moreover, no multivariate analyses have been performed to compare the two
strategies in a single survey and no systematic comparison of the costs of the
two strategies or their combination has been produced.
Using a telephone survey targeting the general population in France in 2010–
11, this study characterizes the two categories of hard-to-interview respondents
(due to their initial refusal or their unavailability) in terms of sociodemographic
profile and response to key survey indicators. In addition, the sociodemographic
determinants of certain sexual behaviors are compared to test the hypothesis of
substitutability between easy-to-interview and hard-to-interview respondents.
This approach is based on the rationale that, in this survey, no limit was set on
the number of calls and the refusals were called again twice: the questionnaires
obtained by these means would not have been collected in the vast majority of
telephone surveys and may thus be considered questionnaires from people who
would have been nonrespondents otherwise. Finally, we compare the costs of
each possible combination of these two interventions (increasing the number of
calls versus calling back the refusals/abandonments).
670
Legleye et al.
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
the interview whether they were reachable on a landline phone or not. If the
answer was yes, these respondents were eliminated. The two samples were
thus disjointed, which allows putting them together and improving the coverage rate. There was no geographic stratification because no such information
was available in RDD. Generated numbers were then matched against the root
base of the French Telecommunications and Posts Regulator (to eliminate unavailable numbers) and the phone directory (to eliminate professional numbers)
before being called automatically to eliminate non-eligible numbers (out of
service, disconnected). A new base was thus constituted and used to call numbers. This protocol makes it possible to cover 99 percent of the population. For
households whose postal address could be found, a cover letter was sent a few
days before the first call.
The second sampling stage involved selecting one person aged 15–49 per
phone number using the Kish method, with a higher probability for women (to
overrepresent them in the final sample).
During the data-collection protocol, two interventions were implemented to
improve response rates: calling back refusals and allowing a very high number of call attempts. We chose to apply these two interventions in an unlimited manner as an experimental approach. Consequently, the questionnaires
obtained by these means can be considered questionnaires from people who
would have been nonrespondents otherwise. This approach allowed us to
determine practically which intervention produces the greatest improvement
in data quality.
Refusals were called back seven days (minimum) after the refusal. The
rationale was that most of these refusals were “household refusals” that
occurred before the survey was presented and the respondent selected (at
least for landline numbers). A second attempt would have more chance of
finding a less busy interlocutor, and of completing the selection process. The
best interviewers in contact-making were mobilized, and the contact process
was adapted to the potential respondents’ behavior and the reasons given for
refusal during the previous wave (violent refusals were not called back). Two
call-back waves were thus implemented for landline numbers, but only one
for mobile numbers. This was because, contrary to a landline phone, a mobile
phone is often used by only one person, so the same individual is more likely
to answer, and he or she is more likely to refuse again. Moreover, the results
from the first call-back wave were not convincing.
For the call-back strategy, the results were the following: out of the 92,516
valid generated numbers called in the first wave, the initial response rate was
35.3 percent (AAPOR 2011); the noncontact rate was 25.3 percent; and the
refusal/abandonment rate was 34.7 percent (table 1). The remaining 4.85
percent were unable to take part in the survey due to prolonged absence or
language problems. Among the 16,726 recontacted refusals/abandonments
(second wave), 15.1 percent accepted to respond, 68.2 percent refused or
abandoned the survey, 12.8 percent could not be reached, and 3.9 percent
Effectiveness of Call-Backs and Call Attempts
671
Hard-to-Contact and Call-Back Respondents
In the final sample (all waves combined), 9.9 percent of the interviews
(n = 856, 13.6 percent of the respondents in the mobile sample, 9.3 percent in the landline sample) were performed beyond the twentieth call of a
given wave (the corresponding individuals were labeled “hard-to-contact”
respondents), while 19.0 percent (n = 1,645, 13.3 percent of the respondents
in the mobile sample, 20.0 percent in the landline sample) of the interviews
were made after an initial refusal/abandonment (the corresponding individuals were labeled “call-back respondents”). Together, they were considered
“hard to interview,” as opposed to the others, who were considered “easy to
interview” (figure 1).
1. The website of the project is https://cesp.inserm.fr/fr/les-equipes/nathalie-bajos/nb-etudes-encours/4824-fecond-fr-fr.html.
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
proved impossible to interview. A final phase of call-backs to 6,552 refusals/
abandonments (third wave), this time only with landlines, led to a 6.9-percent
response rate and a 73.0-percent refusal rate. The number of violent refusals
(that were not called back) is the difference between the number of refusals
during a wave and the number of phone numbers that were called back during the following wave. For the second wave, we considered landline phone
numbers only because mobile phones were not included in the third wave.
For the first wave, the number of violent refusals was estimated at 2,180 (11.5
percent), and for the second wave, at 1,596 (19.6 percent).
In this survey, the restriction was not on the total calls per number but on the
total of contactless attempts per number. The rule was as follows: a number
was called up to 20 times if no contact was established. Beyond this limit, the
number was classified as unreachable. Once contact was established, a quota
of 20 further calls was set for administering the questionnaire. Note also that
the call counter was reset to zero at the start of each wave: calls made during
the preceding wave were not taken into account in the management of calls in
the current wave. With this methodology, the number of calls was not really
limited. The maximum number of calls for one number was 373 (all outcomes)
and 158 for an interview for landline phones, and 73 and 58, respectively, for
mobile phones, while 14. 9 percent of landline phones and 25.3 percent of
mobile phones were abandoned or interviewed after 30 calls.
The final participation rate was 44.8 percent. Nearly 30 percent of numbers
never answered, while 20.2 percent maintained their refusal to participate in
the survey and 5.1 percent were impossible to interview (language problems,
prolonged absence). Table 1 summarizes this information. The definitions of
the variables and the wording of the questions used in this study are presented
in the appendix (see the supplementary material online).1
6,232
5,263
969
772
610
162
7,004
5,873
1,131
92,516
11,664
7,004
4,660
2,691
654
822
310
183
30,696
14,257
1,831
1,061
13,547
Outcomes of calls
Number of phone numbers called
Eligible (1 + 2)
Interviews (1)
Eligible cases that are not interviewed (2)
Household-level and respondent refusal (2.a)
Break-off (2.b)
Respondents never available (2.c)
Impossible interviewsa (2.d)
Failed interviews (technical problems) (2.e)
Unknown eligibility (3)
Always busy/no answer (3.a)
Call blocking (3.b)
Household-level language problem (3.c)
Immediate refusal (3.d)
16,726
4,215
1,339
2,876
2,028
209
400
154
85
8,799
410
992
188
7,209
1,256
1,097
159
83
68
15
1,339
1,165
174
7,789
6,661
1,128
856
679
177
8,645
7,340
1,305
92,516
12,751
8,645
4,106
1,703
341
1,403
534
125
25,101
14,918
1,605
1,302
7,276
301
301
–
1
1
–
302
302
–
6,552
1,604
302
1,302
1,004
35
181
70
12
4,150
251
619
53
3,227
Continued
Total
Wave 3
(second call-back wave)
refusal/ abandonment
(landlines only)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Number of interviews
Interview rank ≤ 20 (easy-to-reach)
Landline phones
Mobile phones
Interview rank > 20 (hard-to-reach)
Landline phones
Mobile phones
Total
Landline phones
Mobile phones
Wave 1
Wave 2
(first call-back wave)
refusal/ abandonment
Table 1. Number of Interviews Performed at Each Step of the Survey and Total after Each Wave
672
Legleye et al.
15.1%
68.2%
12.8%
3.9%
4.8%
3,712
419
62
959
2,235
37
–
35.3%
34.7%
25.3%
50,156
17,348
495
4,876
10,008
285
17,144
Wave 1
Wave 2
(first call-back wave)
refusal/ abandonment
Total
54,666
17,791
558
5,841
13,003
329
17144
44.8%
20.2%
29.9%
5.1%
Wave 3
(second call-back wave)
refusal/ abandonment
(landlines only)
798
24
1
6
760
7
–
6.9%
73.0%
17.4%
2.7%
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Note.—Recalled numbers during waves 2 and 3 belong to the following categories: “Household-level and respondent refusal,” “Break-off,” “Failed interviews,”
“Call blocking,” and “Immediate refusal.”
Two rates of eligibility are applied to “unknown eligibility” category depending on whether the number is identified as a household:
•rate of households among all numbers: a = (1 + 2 + 4.b + 4.c + 4.d + 4.e)/(1 + 2 + (4 - 4.f))
•rate of eligible households among households: b = (1 + 2)/(1 + 2 + 4.b + 4.c + 4.d + 4.e)
For (3.a), (3.b), and (3.d), both rates are applied successively to the number. For (3.c), only the second rate (b) is applied. Finally, the denominator of each
rate is (1)+(2)+a*b*((3.a)+(3.b)+(3.d))+b*(3.c). For response rate, the numerator is equal to interviews (1). For refusal/abandonment rate, the numerator is
(2.a)+(2.b)+a*b*(3.c). For noncontact rate, it’s (2.c)+(a*b*((3.a)+(3.b)), and for rate of impossible to interview, it is (2.d)+(2.e)+b*(3.c). Reference can be found at
http://www.aapor.org/AM/Template.cfm?Section=Standard_Definitions2&Template=/CM/ContentDisplay.cfm&ContentID=3156, page 46 (formula RR4).
a
An interview is viewed as impossible when there is a language problem (with the respondent or another person of the household), if the respondent is physically
or mentally unable to be interviewed, or if he/she is absent for a long time.
Indicators
Response rate
Refusal/abandonment rate
Noncontact rate
Rate of impossible to interview households/
individuals
Not eligible (4)
Business (4.a)
Not in metropolitan France (4.b)
Secondary residence/nonexclusive mobile (4.c)
No eligible respondent (4.d)
Quota filled (4.e)
Wrong numbers/fax (4.f)
Table 1. Continued
Effectiveness of Call-Backs and Call Attempts
673
674
Legleye et al.
Figure 1. Description of the Subsamples Obtained during Data Collection.
We first compared the distributions of subsamples of respondents (“easy” versus “hard” to interview) for each of the two interventions (call-backs, increased
number of call attempts) relative to a few sociodemographic characteristics
(gender, age, education, etc.), using percentages and Pearson’s Chi2 tests.
We then examined the contribution of including each hard-to-interview category of respondents in reducing distortion of the sample distribution with
respect to the target population, the general population living in France based
on the 2008 Census. Bias was measured by Chi2 distance, which is computed
like a classic Chi2 statistic but with an arbitrarily fixed total sample size each
time, in order to permit comparisons between the distribution of one variable
in two samples of different sizes. Our goal was to observe the change in the
distance between the initial sample of easy-to-interview and the target population on the one hand, and between the sample with both easy- and hard-tointerview and the target population on the other, for each intervention.
We measured and compared the prevalences of several SRH indicators in
the subsamples using Pearson’s Chi2 test. Lacking any external source for
these variables, we could not measure the possible reduction in bias associated with each methodological intervention.
We then sought to estimate possible effects of the two interventions on the
sociodemographic determinants of SRH indicators, using logistic regressions
to test for differences in the associations between sociodemographic characteristics and SRH outcomes by subsample populations (call-back and hardto-contact respondents versus others). The contribution of all interactions
together, as well as of each interaction, to the quality of the models was estimated using Chi2 tests. For the sake of readability, only the models related to
the number of sexual partners over the lifetime and in the past 12 months will
be presented. Finally, we compared the two interventions and their combinations in terms of sample size, bias, and costs.
Among our measurements of nonresponse bias, we also used the ratio
between calibration weights and sampling weights. Annual Census microdata, available from the website of the French National Institute of Statistics
and Economic Studies (INSEE), were used to obtain social and demographic
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Statistical Analysis
Effectiveness of Call-Backs and Call Attempts
675
statistics on the target population. The calibration weights were then computed
with the CALMAR macro from INSEE (Sautory 1993), using the eight sociodemographic variables presented in tables 2, 4, 5, and 6. All analyses were
performed using SAS V9.2.3, with the level of significance set at 0.05.
Methodological Assumptions
Results
Call-Back Respondents
Table 2 (left-hand side) presents a comparison of the respondents’ sociodemographic profiles by interview wave (initial versus call-back respondents).
A greater proportion of call-back respondents belonged to the highest and lowest age groups (15–24 years and 40–49 years); had a low level of education;
were nonworking or unemployed; were living with a partner or their parents;
and were living in households of three or more people.
To determine whether the inclusion of these individuals led to a reduction
in bias, we compared the initial sample (wave 1) and the final sample (waves
1 through 3) to the 2008 census data (table 2, right-hand side). The inclusion
of the call-back respondents increased the distortion of the sample distribution
(measured by an increase in Chi2 distances in the comparison with the target
population) as compared to the initial sample. But results show no significant
difference in SRH behaviors between first-wave respondents and call-back
respondents (table 3, left part).
In a further attempt to explore the effects of the call-back strategy on survey
results, we tested the differences of the effects of the sociodemographic characteristics on SRH indicators in subsamples (initial versus call-back respondents)
in table 4. For the number of sexual partners (lifetime and past 12 months), the
hypothesis of equality of all coefficients in the two subsamples is rejected for
women (at the 5 percent level), suggesting that the sociodemographic determinants of the modeled SRH behaviors differ between subsamples. For lifetime number of sexual partners, it is due to different effects of age and level
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
We examined the effect of interview rank across all waves combined. The
profiles of hard-to-contact respondents in wave 1 and those in waves 2 and
3 were very similar, although differences were observed regarding higher
education, living arrangement, and geographical location. However, these
characteristics were controlled for in the multivariate analyses. Furthermore,
as mobile phones received only one wave of call-backs, we conducted the
analyses across all waves combined. For similar reasons, we combined the
two waves of call-backs (waves 2 and 3) and considered mobile and landline
phones together in the analysis.
45.2
54.9
13.4
20.6
9.7
10.2
13.8
15.4
17.0
15–17
18–24
25–29
30–34
35–39
40–44
45–49
22.4
23.8
23.1
18.1
12.7
None, middle school
Vocational
High school
1 to 3 years U.
> 3 years U.
63.9
36.1
Employed
Nonworking/unemployed
(0.77)
68.3
31.7
(149.55)
16.7
21.9
23.2
22.0
16.3
(11.47)
9.1
20.0
12.2
12.4
15.0
15.4
15.9
(12.75)
45.5
54.5
(1)+(2)
Initial sample
(wave 1)
(2.91)
67.4
32.6
(117.55)
17.8
22.2
23.1
21.2
15.6
(22.75)
9.9
20.1
11.8
11.9
14.8
15.4
16.1
(13.64)
45.3
54.7
(1)+(2)+(3)+(4)
Final sample
(waves 1, 2 & 3)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
68.3
31.7
***
Employment status
16.7
21.9
23.2
22.0
16.3
***
Educational level
9.1
20.0
12.2
12.4
15.0
15.4
15.9
***
Age
45.5
54.5
Men
Women
(3)+(4)
ns
(1)+(2)
Call-back sample
(waves 2 & 3)
Gender
Subsample
Initial sample
(wave 1)
Continued
69.2
30.8
25.9
24.0
20.4
14.7
15.0
8.1
18.4
13.4
14.0
15.3
15.5
15.2
49.5
50.5
Target population
(census)
Table 2. Sociodemographic Profile of Respondents by Interview Wave (percent, Chi2 p-value, and Chi2 value)
676
Legleye et al.
93.3
6.7
59.5
25.2
15.3
With a partner
With parents
Other situations
100
1,641
100
7,004
Total
Respondents
100
7,004
(17.09)
11.7
16.9
52.8
18.6
30.9
36.0
(17.75)
33
(13.01)
57.7
21.4
20.9
(36.69)
92.6
7.4
(1)+(2)
Initial sample
(wave 1)
100
8,645
(30.28)
10.9
16.3
53.5
19.3
30.9
36.0
(15.71)
33.1
(15.96)
60.0
21.7
18.3
(38.99)
92.8
7.2
(1)+(2)+(3)+(4)
Final sample
(waves 1, 2 & 3)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
13.0
19.8
50.3
16.9
28.4
34.3
37.3
56.3
21.9
21.8
88.3
11.7
Target population
(census)
Note.—U.: University, college, or equivalent. In brackets and italics: distance from target population (Chi2 value); see Statistical Analysis for more details.
***, ns: Chi2 p-value < 0.01 and nonsignificant when comparing two subsamples (columns 2 & 3), respectively.
7.6
13.8
56.7
21.9
11.7
16.9
52.8
18.6
Living alone
Two people
Three or four people
Five or more people
30.3
35.6
31.0
36.2
***
34.2
32.8
Ile-de-France, Parisian
Basin
North, East, West
Center and South
Household size
ns
Place of residence
60.1
20.8
19.1
***
Living situation
92.6
7.4
In France
Abroad
(3)+(4)
ns
(1)+(2)
Call-back sample
(waves 2 & 3)
Birthplace
Subsample
Initial sample
(wave 1)
Effectiveness of Call-Backs and Call Attempts
677
94.6
93.6
58.6
35.2
17.0
8.9
46.4
Sexual intercourse
over the past 12 months
Men
Women
Five or more lifetime
opposite-sex sexual partners
Men
Women
Two or more opposite-sex
sexual partners over the
previous 12 months
Men
Women
Use of the pill
Women
47.2
16.2
8.2
56.7
36.2
95.1
94.5
(3)+(4)
Call-back sample
(waves 2 & 3)
ns
ns
ns
ns
ns
ns
ns
P
46.2
16.1
8.6
57.4
35.4
94.5
93.7
(1)+(3)
Interview rank 1–20
(all waves)
P
ns
ns
**
ns
***
*
ns
(2)+(4)
96.3
94.8
66.5
35.3
22.9
11.3
50.6
Interview Rank 21 and
above (all waves)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
a
The weighting takes into account individuals’ probability of being selected and the proportion of individuals with mobiles only by gender in the target population.
***, **, *, ns: Chi2 p-value < 0.01, < 0.05, < 0.1, and nonsignificant when comparing columns 2 & 3 and 4 & 5, respectively.
(1)+(2)
Subsample
Initial sample
(wave 1)
Table 3. Weighteda Means of a Few Variables of Interest for Surveyed Individuals by Interview Wave, Interview Rank, and
Gender (percent and Chi2 test)
678
Legleye et al.
***
***
–0.83***
–0.33
–0.39***
–0.45
Ref.
Ref.
0.08
0.40
–0.05
0.48*
0.01
–0.22
11.24 (0.047)
***
*
–0.09
–0.23
–0.10
–0.64***
Ref.
Ref.
0.32***
–0.24
0.43***
–0.40*
12.16 (0.016)
***
***
–0.95***
–0.79*
–0.34**
0.22
Ref.
Ref.
–0.20
0.37
–0.03
0.65*
0.06
0.74**
5.21 (0.391)
***
ns
–0.23
–0.19
0.04
–0.16
Ref.
Ref.
–0.12
0.04
–0.46***
–0.06
3.57 (0.467)
Educational level
None, middle school
Vocational
High school
1 to 3 years U.
> 3 years U.
TEC: Chi2 (p-value)
Waves
2 & 3
(3)+(4)
Age
18–24
25–29
30–34
35–39
40–44
45–49
TEC: Chi2 (p-value)
Wave 1
(1)+(2)
Subsample
Waves
2 & 3
(3)+(4)
Women
Waves
2 & 3
(3)+(4)
ns
ns
0.04
0.35
0.27
–0.35
Ref.
Ref.
0.04
–0.19
0.09
0.62
5.43 (0.246)
*
ns
0.25
0.15
0.31
0.80
Ref.
Ref.
–0.45*
–0.10
–0.14
0.25
0.08
0.84
2.29 (0.807)
Wave
1 (1)+(2)
Men
Waves
2 & 3
(3)+(4)
Continued
ns
ns
0.25
0.07
–0.01
–1.03**
Ref.
Ref.
0.29*
–0.42
0.21
–1.08**
7.56 (0.109)
***
***
0.73***
2.39***
0.08
–0.06
Ref.
Ref.
–0.04
0.62
–0.62**
0.00
–0.40
–1.28
13.02 (0.022)
Wave 1
(1)+(2)
Women
≥ 2 Opposite-sex partners (past 12 months)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Wave 1
(1)+(2)
Men
≥ 5 Lifetime opposite-sex partners
Table 4. Determinants of SRH Behaviors by Wave (results of logit models)
Effectiveness of Call-Backs and Call Attempts
679
ns
ns
Ref.
Ref.
–0.04
0.30
2.54 (0.111)
ns
ns
Ref.
Ref.
–0.48***
–0.14
1.16 (0.281)
***
***
Ref.
Ref.
0.17
–0.28
0.80***
0.63***
1.57 (0.455)
***
***
Ref.
Ref.
–0.32***
–0.39**
–0.85***
–0.88***
0.12 (0.939)
ns
ns
Ref.
Ref.
–0.02
0.01
0.00 (0.954)
***
**
Ref.
Ref.
0.18
0.96**
0.80***
0.54*
4.79 (0.091)
ns
ns
Ref.
Ref.
–0.03
–0.10
–0.31*
–0.23
0.26 (0.880)
Birthplace
France
Abroad
TEC: Chi2 (p-value)
Living situation
With a partner
With parents
Other situations
TEC: Chi2 (p-value)
Household size
1–2
3–4
5+
TEC: Chi2 (p-value)
Waves
2 & 3
(3)+(4)
ns
ns
Ref.
Ref.
–0.21
–0.06
0.27 (0.602)
Wave 1
(1)+(2)
Employment status
Employed
Nonworking/unemployed
TEC: Chi2 (p-value)
Waves
2 & 3
(3)+(4)
Women
Subsample
Men
≥ 5 Lifetime opposite-sex partners
Waves
2 & 3
(3)+(4)
Waves
2 & 3
(3)+(4)
Continued
***
ns
Ref.
Ref.
–0.20
0.00
–0.95***
0.13
3.08 (0.214)
***
***
Ref.
Ref.
2.02***
0.90*
2.51***
2.74***
5.55 (0.062)
***
***
Ref.
Ref.
2.58***
3.54***
2.71***
3.54***
3.35 (0.187)
*
ns
Ref.
Ref.
–0.22
–0.03
–0.63**
0.63
4.64 (0.099)
ns
ns
Ref.
Ref.
–0.07
0.19
0.16 (0.692)
*
ns
Ref.
Ref.
–0.27*
–0.20
0.03 (0.862)
Wave 1
(1)+(2)
Women
ns
ns
Ref.
Ref.
–0.04
–0.19
0.04 (0.842)
ns
ns
Ref.
Ref.
0.18
0.12
0.02 (0.878)
Wave
1 (1)+(2)
Men
≥ 2 Opposite-sex partners (past 12 months)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Wave 1
(1)+(2)
Table 4. Continued
680
Legleye et al.
37.34 (0.007)
Waves
2 & 3
(3)+(4)
Waves
2 & 3
(3)+(4)
15.64 (0.681)
33.56 (0.016)
33.91 (0.013)
–3.41***
–3.69***
0.11 (0.734)
–2.88***
–3.73***
1.04 (0.307)
15.13 (0.653)
**
*
Ref.
Ref.
0.37**
–1.25*
5.85 (0.016)
Ns
Ns
Ref.
Ref.
–0.10
–0.14
0.10
0.04
0.02 (0.991)
Wave 1
(1)+(2)
Women
ns
**
Ref.
Ref.
0.12
0.86**
2.56 (0.109)
ns
ns
Ref.
Ref.
–0.20
–0.55
–0.10
–0.36
0.76 (0.685)
Wave
1 (1)+(2)
Men
≥ 2 Opposite-sex partners (past 12 months)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Note.—Ref: Reference category; TEC: Test of equality of coefficients; IdF: Île-de-France; U.: University/college. In bold: Chi2 test significant for the difference
between the coefficients in the two subsamples (e.g., when modeling ≥ 5 lifetime opposite-sex partners among women, for “vocational” as educational level, –0.10
compared to –0.64).
***, **, *, ns: Wald Chi2 p-value < 0.01, < 0.05, < 0.1, and nonsignificant when comparing a coefficient to the reference, respectively.
18.63 (0.481)
All coefficients, including
the constant
TEC: Chi2 (p-value)
34.59 (0.011)
–0.48***
–0.16
0.75 (0.386)
0.71***
–0.01
2.71 (0.100)
Constant
TEC: Chi2 (p-value)
17.89 (0.463)
ns
ns
Ref.
Ref.
0.17
–0.11
0.80 (0.3708)
***
ns
Ref.
Ref.
0.34***
0.30
0.01 (0.911)
Telephone
Landline
Mobile
TEC: Chi2 (p-value)
All coefficients except the
constant
TEC: Chi2 (p-value)
***
Ns
Ref.
Ref.
–0.12
0.07
0.23***
0.23
1.13 (0.568)
Waves
2 & 3
(3)+(4)
***
ns
Ref.
Ref.
–0.42***
–0.06
0.13
0.09
2.45 (0.294)
Wave 1
(1)+(2)
Place of residence
IdF, Parisian Basin
North, East, West
Center and South
TEC: Chi2 (p-value)
Waves
2 & 3
(3)+(4)
Women
Subsample
Men
≥ 5 Lifetime opposite-sex partners
Wave 1
(1)+(2)
Table 4. Continued
Effectiveness of Call-Backs and Call Attempts
681
682
Legleye et al.
Hard-to-Contact Respondents
Compared to easy-to-interview respondents, hard-to-contact respondents were
more likely to be 25–39, employed, born abroad, and living in small households, and were less likely to be living with a partner or parent (table 5, lefthand side). The inclusion of hard-to-contact respondents led to a significant
improvement (measured by a drop in Chi2 distances in the comparison with
the target population) in sample structure for all variables apart from level of
education (table 5, right-hand side).
In addition, hard-to-contact respondents were more likely to report having
multiple opposite-sex partners over the past 12 months (table 3, right part).
Men in this group were also more likely to have had more than four lifetime
partners.
Overall, based on the test of the equality of all coefficients, the sociodemographic determinants of the modeled SRH behaviors did not significantly differ between subsamples (table 6). However, variable-by-variable tests revealed
that certain characteristics were differently associated with sexual behaviors
among hard-to-contact respondents and others. Age, level of education (for
women), and country of birth (for men) were differently related to the number
of lifetime partners according to interview rank. Fewer differences were seen
in the effects of study subgroups on the determinants of the number of partners
over the past 12 months, and no differences were noted in connection to contraceptive use (pill use) and sexual activity in the past 12 months.
Comparison of the Two Interventions: Sample Size, Bias, and Costs
In order to determine the optimal allocation of field effort and to facilitate
future survey protocol decisions, we compared results according to different
intervention scenarios. In scenario 1, which corresponds to a scenario with no
intervention to reduce nonresponse rate, only the easy-to-interview respondents were considered; i.e., the initial respondents interviewed in the first 20
calls during wave 1, as presented in figure 1 (sample 1). Scenario 2 adds the
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
of education. For the number of sexual partners in the past 12 months, both
age and type of phone seem to influence behavior in a different way. We also
checked (data available on request) that the factors associated with current use
of the birth-control pill differed in the two subsamples: among initial respondents, higher education was positively associated with pill use, whereas among
call-back respondents, women with higher education were less likely to be
current pill users.
For men, the sociodemographic determinants of these SRH indicators did
not significantly differ between subsamples. Furthermore, in all models, an
attenuation of the significance of the coefficients among call-back respondents
emerged, which may reflect greater homogeneity in behaviors but could also
result from smaller sample size.
47.6
52.4
Men
Women
8.6
19.7
14.2
14.5
15.8
14.9
12.4
15–17
18–24
25–29
30–34
35–39
40–44
45–49
16.4
23.9
22.3
21.3
16.1
None, middle school
Vocational
High school
1 to 3 years U.
> 3 years U.
***
73.0
27.0
66.8
33.2
Employed
Nonworking/
unemployed
33.2
(5.19)
66.8
(116.38)
18.0
22.1
23.2
21.2
15.5
(28.63)
10.1
20.1
11.5
11.6
14.7
15.5
16.5
(15.25)
45.1
54.9
Interview rank
1–20 (1)+(3)
8.1
18.4
13.4
14.0
15.3
15.5
15.2
25.9
24.0
20.4
14.7
15.0
69.2
(22.75)
9.9
20.1
11.7
11.9
14.7
15.4
16.1
(117.55)
17.8
22.2
23.2
21.2
15.6
(2.91)
67.4
Continued
30.8
49.5
50.5
(13.64)
45.3
54.7
32.6
Target population
(census)
Final sample (all
interview ranks)
(1)+(2)+(3)+(4)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Employment status
18.0
22.1
23.2
21.2
15.5
ns
Educational level
10.1
20.1
11.5
11.6
14.7
15.5
16.5
***
Age
45.1
54.9
ns
Interview rank 21
and above (2)+(4)
Gender
Subsample
Interview rank
1–20 (1)+(3)
Table 5. Sociodemographic Profile of Respondents by Interview Rank (percent, Chi2 p-value, and Chi2 value)
Effectiveness of Call-Backs and Call Attempts
683
89.0
11.0
100
856
100
7,789
Total
Respondents
100
7,789
(40.15)
10.6
15.8
53.9
19.7
(18.34)
32.7
31.0
36.3
(20.79)
60.2
22.0
17.8
(46.27)
93.2
6.8
Interview rank
1–20 (1)+(3)
88.3
11.7
56.3
21.9
21.8
37.3
28.4
34.3
13.0
19.8
50.3
16.9
(38.99)
92.8
7.2
(15.96)
60.0
21.6
18.4
(15.71)
33.1
30.9
36.0
(30.28)
10.9
16.3
53.5
19.3
100
8,645
Target population
(census)
Final sample (all
interview ranks)
(1)+(2)+(3)+(4)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Note.—U.: University, college, or equivalent. In brackets and italics: distance from target population (Chi2 value); see Statistical Analysis for more details.
***, *, ns: Chi2 p-value < 0.01, < 0.1, and nonsignificant when comparing two subsamples (columns 2 & 3), respectively.
13.8
21.1
50.2
14.8
10.6
15.8
53.9
19.7
Living alone
Two people
Three or four people
Five or more people
36.5
30.2
33.3
***
32.7
31.0
36.3
*
Household size
Ile-de-France, Parisian Basin
North, East, West
Center and South
Place of residence
57.8
18.0
24.2
With a partner
With parents
Other situations
60.2
22.1
17.8
***
Living situation
93.2
6.8
In France
Abroad
Interview rank 21
and above (2)+(4)
***
Interview rank
1–20 (1)+(3)
Birthplace
Subsample
Table 5. Continued
684
Legleye et al.
ns
Ref.
–0.26
ns
Ref.
0.06
ns
Ref.
–0.10
Employment status
Employed
Nonworking/unemployed
Rank 21+
(2)+(4)
*
Ref.
0.28*
ns
Ref.
–0.46
ns
ns
–0.09
1.62**
0.08
0.88
Ref.
Ref.
–0.12
0.95*
0.07
1.02
6.72 (0.151)
ns
ns
0.09
1.21
0.18
1.44**
Ref.
Ref.
–0.41
–0.55
–0.28
1.03
0.09
0.58
6.14 (0.293)
Rank 1–20
(1)+(3)
Men
Rank 21+
(2)+(4)
ns
Ref.
–0.20
Continued
ns
Ref.
–0.82
ns
ns
0.24
–0.20
–0.14
–0.53
Ref.
Ref.
0.11
0.43
–0.02
0.32
2.24 (0.691)
***
ns
1.09***
0.52
0.24
–0.44
Ref.
Ref.
0.21
–0.77
–0.55**
–0.68
–0.45*
–0.23
4.19 (0.523)
Rank 1–20
(1)+(3)
Women
≥ 2 Opposite-sex partners (past 12 months)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
*
Ref.
–0.73*
***
ns
–0.02
–1.03**
–0.22**
–0.22
Ref.
Ref.
0.27***
–0.66**
0.32***
–0.43
13.78 (0.008)
**
ns
–0.32**
0.45
–0.05
0.09
Ref.
Ref.
–0.16
0.39
–0.39***
–0.41
3.48 (0.481)
Rank 21+
(2)+(4)
Educational level
None, middle school
Vocational
High school
1 to 3 years U.
> 3 years U.
TEC: Chi2 (p-value)
Rank 1–20
(1)+(3)
***
ns
–0.81***
–0.12
–0.37***
–0.79*
Ref.
Ref.
0.16
–0.09
0.09
–0.39
0.00
–0.74*
11.53 (0.041)
Rank 21+
(2)+(4)
***
ns
–0.99***
–0.555
–0.27*
–0.058
Ref.
Ref.
–0.11
0.040
0.08
0.441
0.18
0.731
1.25 (0.940)
Rank 1–20
(1)+(3)
Women
Age
18–24
25–29
30–34
35–39
40–44
45–49
TEC: Chi2 (p-value)
Subsample
Men
≥ 5 Lifetime opposite-sex partners
Table 6. Determinants of SRH Behaviors by Interview Rank (results of logit models)
Effectiveness of Call-Backs and Call Attempts
685
***
ns
Ref.
Ref.
–0.33***
–0.25
–0.83***
–0.81*
0.10 (0.950)
***
Ref.
–0.06
0.24***
ns
ns
Ref.
Ref.
0.00
–0.45
–0.25*
–0.69
1.57 (0.457)
***
Ref.
–0.38***
0.11
Household size
1–2
3–4
5+
TEC: Chi2 (p-value)
Place of residence
IdF, Parisian Basin
North, East, West
Center and South
ns
Ref.
–0.44
0.14
***
***
Ref.
Ref.
0.16
–0.31
0.78***
0.81***
0.59 (0.744)
***
ns
Ref.
Ref.
–0.15
0.22
–0.74*
0.08
3.53 (0.171)
Living situation
With a partner
With parents
Other situations
TEC: Chi2 (p-value)
Rank 21+
(2)+(4)
ns
Ref.
–0.28*
–0.13
ns
Ref.
–0.14
–0.28
ns
ns
Ref.
Ref.
–0.20
–0.44
0.42*
–0.18
0.46 (0.794)
***
***
Ref.
Ref.
2.69***
2.86***
2.76***
3.12***
0.48 (0.788)
ns
ns
Ref.
Ref.
–0.04
–0.25
0.08 (0.779)
1.91 (0.166)
Rank 1–20
(1)+(3)
Men
Rank 21+
(2)+(4)
ns
Ref.
–0.15
0.17
Continued
ns
Ref.
0.13
–0.54
**
ns
Ref.
Ref.
–0.14
–0.26
–0.84
0.75
5.68 (0.0585)
***
***
Ref.
Ref.
1.80***
1.99***
2.52***
2.39***
0.19 (0.908)
ns
ns
Ref.
Ref.
–0.02
–0.71
0.93 (0.335)
1.31 (0.252)
Rank 1–20
(1)+(3)
Women
≥ 2 Opposite-sex partners (past 12 months)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
ns
Ref.
–0.24
0.19
***
**
Ref.
Ref.
–0.36***
–0.87**
1.40 (0.237)
**
Ref.
Ref.
0.11
–1.01**
4.69 (0.030)
Rank 21+
(2)+(4)
Birthplace
France
Abroad
TEC: Chi2 (p-value)
Rank 1–20
(1)+(3)
0.91 (0.341)
Rank 21+
(2)+(4)
Women
2.29 (0.131)
Rank 1–20
(1)+(3)
Men
≥ 5 Lifetime opposite-sex partners
TEC: Chi2 (p-value)
Subsample
Table 6. Continued
686
Legleye et al.
26.76 (0.110)
All coefficients, including
the constant
TEC: Chi2 (p-value)
26.44 (0.118)
26.37 (0.092)
Rank 21+
(2)+(4)
Rank 21+
(2)+(4)
18.96 (0.459)
24.28 (0.186)
20.96 (0.281)
–3.53*** –2.61***
1.53 (0.216)
–2.82***
–3.98***
1.79 (0.180)
16.09 (0.586)
ns
ns
Ref.
Ref.
0.22
–0.15
0.50 (0.477)
4.17 (0.124)
Rank 1–20
(1)+(3)
Women
ns
ns
Ref.
Ref.
0.24
0.03
0.22 (0.636)
0.37 (0.832)
Rank 1–20
(1)+(3)
Men
≥ 2 Opposite-sex partners (past 12 months)
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Note.—Ref: Reference category; TEC: Test of equality of coefficients; IdF: Ile-de-France; U.: University/college. In bold: Chi2 test significant for the difference
between the coefficients in the two subsamples (e.g., when modeling ≥ 5 lifetime opposite-sex partners among women, for “vocational” as educational level, –.10
compared to –.64).
***, **, *, ns: Wald Chi2 p-value < 0.01, < 0.05, < 0.1, and nonsignificant when comparing a coefficient to the reference, respectively.
20.01 (0.332)
–0.48***
0.38
3.62 (0.057)
0.53***
1.20**
1.46 (0.226)
Constant
TEC: Chi2 (p-value)
All coefficients except the
constant
TEC: Chi2 (p-value)
ns
ns
Ref.
Ref.
0.15
–0.18
0.89 (0.346)
***
ns
Ref.
Ref.
0.40***
–0.13
2.39 (0.122)
Rank 21+
(2)+(4)
Telephone
Landline
Mobile
TEC: Chi2 (p-value)
Rank 1–20
(1)+(3)
1.42 (0.492)
Rank 21+
(2)+(4)
Women
0.15 (0.929)
Rank 1–20
(1)+(3)
Men
≥ 5 Lifetime opposite-sex partners
TEC: Chi2 (p-value)
Subsample
Table 6. Continued
Effectiveness of Call-Backs and Call Attempts
687
688
Legleye et al.
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
hard-to-contact respondents of wave 1 to sample 1 to assess the effect of
increasing the number of call attempts alone to reduce noncontact. Scenario 3
combines initial and call-back respondents (all waves) who were easy to contact (i.e., who responded in the first 20 call attempts) to estimate the value of
reducing refusal rates alone. The fourth scenario offers a glimpse of what the
sample would have been if the number of calls had been increased only to capture initial respondents but not during the call-back waves. Finally, scenario 5
combines all respondents and represents the current final sample.
Changes in sample size according to the various scenarios in comparison
to scenario 1 are presented in table 7. Compared to scenario 1, call-backs
with no increase in the number of call attempts (scenario 3) led to a 25.0-percent increase in sample size, while scenario 2 (increased number of contact
attempts without call-backs) led to a 12.4-percent increase in sample size. The
combination of the two interventions in scenario 4 led to an increase of 37.4
percent of the sample size, while the transition from scenario 4 to scenario 5
was associated with an improvement in sample size of around one point.
To study the impact of these measures on sociodemographic bias, we compared the sample structure with that of the target population for each of the
eight variables used in all previous tables (gender, age, educational level,
employment status, birthplace, living situation, place of residence, and household size). While increasing the number of call attempts substantially reduced
selection bias with respect to seven out of eight sociodemographic characteristics, the inclusion of call-back respondents, on the contrary, led to greater
distortion for six out of eight characteristics. Combining the two interventions
led to an intermediate outcome.
Another way to illustrate bias is through ratios of weights (weight after calibration/initial survey weight): ratios far from 1 occur when the structure obtained
with the initial weighting is very different from the target structure. The distribution of weight ratios (coefficient of variation, maximum and minimum) thus can
be used to evaluate distance with respect to the reference population. Compared
to scenario 1, increasing the number of call attempts (scenario 2) improved
the coefficient of variation of the ratio of calibration to initial weights, with a
decrease from 49.8 percent to 48.3 percent. The ratio of maximum to minimum
fell from 36.2 percent to 34.9 percent. Conversely, the inclusion of call-back
respondents (scenario 3) did not improve the variance of the weight ratio.
Finally, we evaluate the financial cost of each scenario. Call-backs (in scenarios 3, 4, and 5) led to a greater increase in costs than increasing the number of call attempts (scenario 2), but they also resulted in the inclusion of
more respondents. In total, the extra cost associated with the two interventions
exceeded 57 percent. The examination of the relative costs reveals opposite
conclusions: interviews of hard-to-contact individuals (samples 2 and 4) were
significantly more costly than others. Call-back respondents who responded
in the first 20 call attempts (sample (3)) led to substantially lower extra costs
per interview (1.19).
6,232
–
Ref.
7 variables/8
(1)+(2) vs. (1)
Extra costs
Ref.
(1)+(2) vs. (1)
24.56%
(1)+(3) vs. (1)
29.67%
0.51
48.70
0.16
6.67
41.69
0.47
2 variables/8
(1)+(3) vs. (1)
Continued
(1)+(2)+(3)+(4) vs. (1)
57.06%
4 variables/8
4 variables/8
(1)+(2)+(3) vs. (1)
54.23%
(1)+(2)+(3)+(4) vs. (1)
(1)+(2)+(3) vs. (1)
0.48
46.00
0.17
6.71
39.47
0.43
8,645
2,413 (38.7%)
8,561
2,329 (37.4%)
0.49
47.29
0.16
6.43
40.19
0.45
(1)+(2)+(3)+(4)
Scenario 5
(final sample)
(1)+(2)+(3)
Scenario 4
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Data collection financial costsc
(1)+(3)
Scenario 3
7,789
1,557 (25.0%)
Statistics on ratios of weights (weight after post-stratification/sampling weight)
Standard deviation
0.51
0.50
Coeff. of variation
49.82
48.27
Min
0.17
0.17
Max
6.15
5.93
Max/min
36.18
34.88
Interquartile range
0.45
0.42
Variables closer to the distribution of
the target populationb
7,004
772 (12.4%)
(1)+(2)
(1)
Measure of sociodemographic bias
Comparison with the structure of the target population
Sample sizes
Sample size
Added respondents to scenario 1: n (%)
Subsamplea
Scenario 2
Scenario 1
Table 7. Comparison of the Methodological Interventions
Effectiveness of Call-Backs and Call Attempts
689
(1)+(2)+(3)+(4)
(2) = 1.98; (3) = 1.19;
(4) = 2.10
(1)+(2)+(3)+(4) = 1.13
(2) = 1.98;
(3) = 1.19
(1)+(2)+(3) = 1.12
Scenario 5
(final sample)
(1)+(2)+(3)
Scenario 4
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Note.—Scenario 1: Only the easy-to-interview respondents were considered; i.e., the initial respondents interviewed in the first 20 calls during wave 1.
Scenario 2: Adds the hard-to-contact respondents of wave 1 to sample 1.
Scenario 3: Initial and call-back respondents (all waves) who were easy-to-contact (who responded in the first 20 call attempts).
Scenario 4: The number of calls had been increased only for the first wave but not during the call-back waves.
Scenario 5: Combines all respondents and represents the current final sample.
Sample 4 does not appear, as it cannot be produced alone in the data-collection process (see figure 1).
Ref.: reference.
a
See figure 1 for the description of the subsamples.
b
Based on the difference between: the Chi2 distance between the distributions of the variables in the current scenario and in the target population on one side; the
2
Chi distance between the distributions of the variables in scenario 1 and in the target population on the other side.
c
The costs are based on average time of a call given the outcome: an interview costs 41 (average time in minutes), a call attempt without answer costs 1; an
immediate refusal costs 2, etc. All calls are considered for each phone number.
(1)+(3) = 1.04
1
Total
(1)+(2) = 1.11
1
Mean relative cost per interview
By subsample
(1)+(3)
(3) = 1.19
(1)+(2)
(1)
Subsamplea
Scenario 3
(2) = 1.98
Scenario 2
Scenario 1
Table 7. Continued
690
Legleye et al.
Effectiveness of Call-Backs and Call Attempts
691
Discussion
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
In order to reduce potential nonresponse bias, interventions seek to reach out
to individuals who are harder to interview (because they are either unreachable
or reluctant to respond). This practice relies on the assumption of a continuum from individuals who are the most easily interviewed to nonrespondents,
with those hard to interview being more similar to nonrespondents. But this
hypothesis cannot be verified. In order to explore its relevance, our protocol distinguished two (nonexclusive) types of hard-to-interview respondents:
hard-to-contact (more than 20 call attempts to complete the questionnaire) and
call-back respondents (who initially refused the questionnaire), who are usually not included in the final sample of respondents and who may thus be considered as close to nonrespondents. We compared the two in order to determine
who contributes more to bias reduction. First, we found that the inclusion of
call-back respondents resulted in greater distortion of the sample distribution
while the inclusion of hard-to-contact respondents tended to reduce selection
bias. Second, we found several differences in SRH behaviors between easyto-interview and hard-to-contact respondents, whereas differences between
first-wave respondents and call-back respondents are not significant. Third,
we found that the sociodemographic determinants of SRH behaviors varied
more with call-back interventions than with interview rank, especially for
women, but that they also varied in opposite ways, especially among women,
for age. In the case of this survey, focusing on SRH in a limited age range
(15–49), we cannot firmly conclude that hard-to-contact respondents are more
different than call-back respondents from easy-to-interview respondents. And
it is not clear whether one category is closer to the nonrespondents than the
other. However, we can conclude that the two hard-to-interview populations
are different, and cannot substitute for each other. Thus, it seems important to
maintain both efforts (increasing the number of call attempts and the call-back
of initial refusals/abandonments), as they have a complementary impact on
sample quality. Consequently, nonresponse correction by calibration is all the
more inappropriate given that hard-to-interview populations are not included
in the sample (because their absence leads to major biases).
As the response rate of the second call-back wave is very low (6.9 percent),
a reasonable choice would be to recommend a first wave with an extended
number of call attempts and one call-back wave with a limited number of call
attempts (20 or less). The relative cost of such a recommendation would not
be higher than the actual one that mixes the two strategies. But the final choice
will depend on whether the goal of the survey is to estimate prevalences or to
model behaviors.
These results are in line with previous studies based on telephone surveys
(Lynn and Clarke 2002; Qayad et al. 2010), as well as research reporting on
face-to-face surveys (Chiu, Riddick, and Hardy 2001; Heerwegh, Koen, and
Loosveldt 2007). In two previous national surveys conducted in France in 2000
692
Legleye et al.
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
(Baromètre santé, or Health Barometer) and 2006 (Contexte de la sexualité en
France, or Sexuality in France), the authors reported substantial sociodemographic differences between hard-to-contact respondents and easy-to-interview populations (Beck, Legleye, and Peretti-Watel 2005; Razafindratsima
2012). Moreover, hard-to-contact individuals were more likely to report a
higher average number of lifetime sexual partners than other respondents
(Razafindratsima 2012), which corroborates the results of our study. However,
these prior surveys did not call back refusals/abandonments, a growing population in national telephone surveys. Such a distinction between hard-to-contact and initially refusing populations was investigated in six surveys carried
out by the British government (Durrant and Steele 2009). The authors showed
that the determinants of nonresponse among refusals and noncontacts were
very different. The difference between easy-to-interview and noncontacts
seemed mostly related to the tendency to be at home, and was therefore associated with household or living situation characteristics. Refusals reflected a
more complex social phenomenon, more likely related to individual characteristics, such as socioeconomic status, education, etc. They also showed that the
combination of the two methodological interventions leads to a balancing of
effects and thus masks the differences in profile of the two populations of nonrespondents. We confirm these observations in the French FECOND study:
call-back respondents and hard-to-contact respondents differed substantially
in terms of age, occupation, and household size, and both need to be included
to substantially improve data quality. Lynn and Clarke (2002) also indicated
that the profile of hard-to-contact individuals was more stable than that of the
refusal population.
Based on these results, our study provides an opportunity to reflect on the
costs and benefits of including hard-to-contact respondents and initial refusals.
In line with the classical deontological argument, which considers the refusals clearly as such, we found few arguments to support the efforts to convert
initial refusals into respondents (namely, the sample size and the existence of
some difference in the sociodemographic determinants of SRH). Moreover,
although call-back interviews were less costly than those among hard-to-contact individuals, the long-term effect of such practice may contribute to lowering confidence in public research. As seen before, the proportion of violent
refusals increased between waves, from 11.5 percent in the first wave to 19.6
percent in the second. This illustrates that the procedure caused more irritation
during the call-backs. But increasing the number of call attempts can also be
criticized on the same basis, as calling a number more than 50 times may be
considered a form of harassment. These interventions may thus be viewed as
a cost for the interviewees and a long-term cost for the survey producers and
the public research community.
Finally, we turn to a discussion of limitations and avenues for future research.
Because preliminary analyses showed little differences by type of telephone
(cell phones versus landlines), we provide results for both types combined.
Effectiveness of Call-Backs and Call Attempts
693
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
The substantial effort to include a large sample of cell phone users (n = 1,305)
in the FECOND study serves as an additional intervention to increase sample
representativeness, as these individuals were considered “impossible to reach”
populations in earlier studies (Beck, Legleye, and Peretti-Watel 2004; Beck,
Legleye, and Peretti-Watel 2005; Gautier et al. 2006; Hu et al. 2011). However,
the refusal rate in the mobile phone sample was higher (33.1 percent versus 30.6
percent in the first wave, 21.7 percent versus 17.5 percent in the final sample).
As all refusals and noncontacts were excluded, the question of the residual
nonresponse bias remains open. Moreover, telephone sampling makes it impossible to use synthetic indicators of representativeness such as the R-indicator
(Schouten, Cobben, and Bethlehem 2009). Similarly, we did not study item
nonresponse and data quality, which could also vary with the respondents’
characteristics.
For our purposes, we considered the two interventions (high number of call
attempts/call-back of refusals/abandonments) separately in our bivariate and
multivariate analyses, and we did not consider their combinations, or subcategories of hard-to-contact or call-back respondents. Defining combinations
of the two categories of “hard-to-interview” respondents would allow further
testing of the hypothesis of a continuum between easy-to-interview respondents and nonrespondents.
In addition, these results must be interpreted with caution because they concern a limited population (15–49) and deal with a sensitive topic, so the conclusions cannot necessarily be generalized to others’ research themes or other
populations.
Despite these limitations, we believe that our results provide new insights
into the costs and benefits of efforts to include hard-to-contact respondents or
initial refusals, not only to improve overall response rates, but more importantly to reduce distortion of the sample distribution and improve estimations.
Such findings need to be coupled with studies of decision-making mechanisms (Adua and Sharp 2010; Groves, Singer, and Corning 2000; Singer 2011;
Wenemark et al. 2011) to better understand the reasons for refusals and to
design tailored strategies to improve participation and reduce selection bias
without compromising the overall acceptance of surveys. Allocating resources
to include a sample of hard-to-interview persons, by face-to-face interview or
by varying contact modes, for example, in order to check whether they differ
from the final respondents or not, would enable us to identify whether there
is a continuum between easy-to-interview individuals and subcategories of
hard-to interview (by telephone) people (i.e., a combination of a high number
of call attempts and/or refusals) and to determine the proportions of each of
these subpopulations. These results, especially the proportion of each kind of
hard-to-interview and the possibility to consider that one is close to the nonrespondent, could then be taken into account to improve the calibration process,
taking inspiration from Hansen and Hurwitz (1946), or perhaps to confirm the
pertinence of the standard process.
694
Legleye et al.
Supplementary Data
Supplementary data are freely available online at http://poq.oxfordjournals.org/.
References
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
AAPOR. 2011. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for
Surveys. Available at http://www.aapor.org/AM/Template.cfm?Section=Standard_Definitions2
&Template=/CM/ContentDisplay.cfm&ContentID=3156.
Adua, Lazarus, and Jeff S. Sharp. 2010. “Examen De La Participation Aux Enquêtes Et De
La Qualité Des Réponses: L’importance De L’intérêt Du Sujet Et Des Primes D’incitation.”
Techniques d’Enquêtes 36(1):103–18.
Beck, François, and Philippe Guilbert. 2007. “Baromètres Santé: Un Éclairage Sur Leur Méthode
Et Leur Évolution.” In Baromètre Santé 2005: Attitudes Et Comportements De Santé,
Baromètre Santé, edited by F. Beck, P. Guilbert, and A. Gautier, 27–43. Saint-Denis: INPES.
Beck, François, Stéphane Legleye, and Patrick Peretti-Watel. 2004. “Using the Telephone in
General Population Surveys on Drugs.” In European Studies on Drugs and Drug Policy:
Proceedings of the 14th International Conference of the European Society for Social Drug
Research (Essd) in Ghent from 2–4 October 2003, edited by T. Decorte and D. J. Korf, 113–
40. Brussels: VUB Press.
———. 2005. “Aux Abonnés Absents: Liste Rouge Et Téléphone Portable Dans Les Enquêtes En
Population Générale Sur Les Drogues.” Bulletin de méthodologie sociologique 86(April):5–29.
Bigot, R., and P Croutte. 2011. La Diffusion Des Technologies De L’information Et De La
Communication Dans La Société Française. Paris: CREDOC.
Brick, J. Michael, Ismael Flores Cervantes, Sunghee Lee, and Greg Norman. 2011. “Erreurs Non
Dues À L’échantillonnage Dans Les Enquêtes Téléphoniques À Base De Sondage Double.”
Techniques d’enquête 37(1):1–16.
Chiu, Pei-Lu, Howard Riddick, and Ann M. Hardy. 2001. “A Comparison of Characteristics
between Late/Difficult and Non-Late/Difficult Interviews in the National Health Interview
Survey.” Paper presented at the Annual Meeting of the American Statistical Association,
Atlanta, GA, USA. Available at http://www.fcsm.gov/committees/ihsng/chiupasa2000.pdf.
Curtin, Richard, Stanley Presser, and Eleanor Singer. 2000. “The Effects of Response Rate Change
on the Index of Consumer Sentiment.” Public Opinion Quarterly 64(4):413–28.
Durrant, Gabrielle B., and Fiona Steele. 2009. “Multilevel Modeling of Refusal and Noncontact
in Household Surveys: Evidence from Six UK Government Surveys.” Journal of the Royal
Statistical Society, Series A 172(2):361–81.
Firdion, Jean-Marie. 1993. “L’effet Du Rang D’appel Et De La Présence Du Conjoint Dans Une
Enquête Par Téléphone.” Population 48(5):1281–314.
Gautier, Arnaud, François Beck, Stéphane Marder, Stéphane Legleye, Benoît Riandey,
Anne Gayet, Nathalie Beltzer, and Philippe Guilbert. 2006. “Téléphones Portables Exclusifs:
Résultats D’une Méthode De Génération Partielle De Numeros.” In Methodes D’enquêtes Et
Sondages—Pratiques Européenne Et Nord-Américaine, Québec, edited by P. Lavallée and L.
Rivest, 60–64. Québec: Dunod.
Groves, Robert M. 2006. “Nonresponse Rates and Nonresponse Bias in Household Surveys.”
Public Opinion Quarterly 70(5):646–75.
Groves, Robert M., Eleanor Singer, and Amy Corning. 2000. “Leverage-Saliency Theory of
Survey Participation.” Public Opinion Quarterly (64):299–308.
Hansen, Morris H., and William N. Hurwitz. 1946. “The Problem of Non-Response in Sample
Surveys.” Journal of the American Statistical Association 41(236):517–29.
Heerwegh, Dirk, Abts Koen, and Geert Loosveldt. 2007. “Minimizing Survey Refusal and
Noncontact Rates: Do Our Efforts Pay Off?” Survey Research Methods 1(1):3–10.
Effectiveness of Call-Backs and Call Attempts
695
Downloaded from http://poq.oxfordjournals.org/ at Warsaw University on March 31, 2014
Hu, Sean S., Lina Balluz, Michael P. Battaglia, and Martin R. Frankel. 2011. “Improving Public
Health Surveillance Using a Dual-Frame Survey of Landline and Cell Phone Numbers.”
American Journal of Epidemiology 173(6):703–11.
Keeter, Scott, Carolyn Miller, Andrews Kohut, Robert M. Groves, and Stanley Presser. 2000.
“Consequences of Reducing Nonresponse in a National Telephone Survey.” Public Opinion
Quarterly 64(2):125–48.
Lan, Romuald Le. 2009. “Enquêtes Ménages: Vers La Fin De La Baisse Des Taux De Réponse?”
Courrier des statistiques 128(September–December):33–41.
Lynn, Peter, and Patricia Clarke. 2002. “Separating Refusal Bias and Non-Contact Bias:
Evidence from UK National Surveys.” Journal of the Royal Statistical Society, Series D (The
Statistician) 51(3):319–33.
Qayad, Mohammed G., Pranesh Chowdhury, Shaohua Hu, and Lina Balluz. 2010. “Différences
Entre Les Répondants Et Durée De La Période De Collecte Des Données Dans Le Behavioral
Risk Factor Surveillance System.” Techniques d’Enquêtes 36(2):241–46.
Razafindratsima, Nicolas. 2012. “Data quality: composition of the sample and conduct of
interviews.” In Sexuality in France: Practices, Gender and Health, edited by N. Bajos &
M. Bozon, 57–71. Oxford: The Bardwell Press.
Sautory, Olivier. 1993. “La Macro Calmar: Redressement D’un Échantillon Par Calage Sur
Marges.” Paris: INSEE.
Schouten, Barry, Fannie Cobben, and Jelke Bethlehem. 2009. “Indicators for the Representativeness
of Survey Response.” Survey Methodology 35(June):101–13.
Singer, Eleanor. 2011. “Toward a Benefit-Cost Theory of Survey Participation: Evidence, Further
Tests, and Implications.” Journal of Official Statistics 27(2):379–92.
Wenemark, Marika, Andreas Persson, Helle Nordlind Brage, Tommy Svensson, and
Margareta Kristenson. 2011. “Applying Motivation Theory to Achieve Increased
Response Rates, Respondent Satisfaction, and Data Quality.” Journal of Official Statistics
27(2):393–414.Imolupta mendae niminus autatus maximi, voluptium et et occuptius.