THE EFFECTS OF M O N E T A R Y P A Y M E N T S ON SURVEY RESPONSES:
EXPERIMENTAL EVIDENCE
FROM A L O N G I T U D I N A L STUDY OF E C O N O M I C A L L Y D I S A D V A N T A G E D Y O U T H S ~ /
Stuart H. K e r a c h s k y
1.
and Charles
D. Mallar,
Mathematica
Policy R e s e a r c h
of m o n e t a r y incentives to r e s p o n d e n t s are hyp o t h e s i z e d to be larger in m a g n i t u d e for r e p e a t e d
waves of interviews and for samples of p o p u l a tions that are d i f f i c u l t to locate and i n t e r v i e w - both of w h i c h apply to this experiment.3_ /
Thus,
a large number of e x p e r i m e n t a l o b s e r v a t i o n s are
available for a n a t i o n a l p o p u l a t i o n for w h o m
m o n e t a r y incentives are e x p e c t e d to be particularly effective.
As expected, the impacts of m o n e t a r y incentives on the e f f i c a c y of the search effort and
i n t e r v i e w c o m p l e t i o n s i n c r e a s e d over the successive waves of interviews.
By the third wave of
interviews, the o b s e r v e d impacts on four measures of s e a r c h - e f f o r t success and i n t e r v i e w
c o m p l e t i o n s were s t a t i s t i c a l l y s i g n i f i c a n t and
i m p o r t a n t in magnitude.
For every i00 sample
m e m b e r s who were to be paid, we r e c e i v e d i0 more
r e s p o n s e s to an advance m a i l i n g than for a similar number of sample members who were not paid,
which suggests some clear effects from r e s p o n d e n t
p a y m e n t s in terms of search e f f i c i e n c y and comp l e t i o n rates w i t h m a i l surveys.
Furthermore,
even though the overall c o m p l e t i o n rates for the
second f o l l o w - u p were very high, we o b t a i n e d app r o x i m a t e l y 5 a d d i t i o n a l c o m p l e t i o n s per i00 sample m e m b e r s in the p a y m e n t group.
The e s t i m a t e d impacts of m o n e t a r y incentives
on item n o n r e s p o n s e were m o s t p r o n o u n c e d at baseline interview, and d e c l i n e d over time.
By the
second f o l l o w - u p survey, they were very small in
m a g n i t u d e and were s t a t i s t i c a l l y insignificant.
This s e e m i n g l y a n o m a l o u s p a t t e r n of effects can
be a t t r i b u t e d at least in part to the q u e s t i o n s
b e c o m i n g easier to answer over time and to the
least c o o p e r a t i v e r e s p o n d e n t s d r o p p i n g out of the
n o n p a y m e n t group over time.
The p a y m e n t e x p e r i m e n t shows some clear benefits to m o n e t a r y incentives.
However, p r e c i s e
inferences cannot be d r a w n about the size of
these effects b e y o n d our specific e x p e r i m e n t
w i t h $5 p a y m e n t s to d i s a d v a n t a g e d youths and using an i n t e n s i v e search process.
The size of effects will u n d o u b t e d l y vary by the a m o u n t of the
payment, by the c h a r a c t e r i s t i c s of the u n d e r l y i n g
population, and by the intensity of search procedures.
The next two sections d e s c r i b e in more detail
the e x p e r i m e n t a l setting and design, r e s p e c t i v e l ~
The fourth section o u t l i n e s the field p r o c e d u r e s
that were followed.
The e x p e r i m e n t a l findings
are r e p o r t e d in Sections 5 and 6 - - f i r s t for
search e f f i c a c y and i n t e r v i e w completions, and
then for the q u a l i t y of the data.
Finally, in
Section 7, we summarize our c o n c l u s i o n s from the
experiment.
I N T R O D U C T I O N AND SUMMARY
This paper presents findings from an experiment d e s i g n e d to test the e f f e c t i v e n e s s of providing m o n e t a r y p a y m e n t s to survey r e s p o n d e n t s in
order to improve their r e s p o n s e s to p e r s o n a l interviews.
A $5 p a y m e n t per i n t e r v i e w was o f f e r e d
to a randomly selected p o r t i o n of a survey panel
of e c o n o m i c a l l y d i s a d v a n t a g e d youths for three
waves of interviews u n d e r t a k e n in c o n j u n c t i o n
with an e v a l u a t i o n of the Job Corps program.
The
effects of these r e s p o n d e n t p a y m e n t s were then
tested for improving the q u a n t i t y of r e s p o n s e s
(search e f f i c a c y and i n t e r v i e w completions) and
the q u a l i t y of responses (interview c o m p l e t i o n s
and item nonresponse).
The b e n e f i c i a l effects of m o n e t a r y p a y m e n t s to
r e s p o n d e n t s in surveys with p e r s o n a l interviews
have been h y p o t h e s i z e d to stem p r i m a r i l y from
the i n c r e a s e d c o o p e r a t i o n of sample m e m b e r s and
of individuals who can be of a s s i s t a n c e in locating them.
Also, r e s p o n d e n t p a y m e n t s are e x p e c t e d
to lead to increased p r o d u c t i v i t y of i n t e r v i e w e r s
who feel they are m a k i n g less of an i m p o s i t i o n on
respondents when r e s p o n d e n t s are paid.
However,
little is known about the m a g n i t u d e of the
effects of r e s p o n d e n t p a y m e n t s in surveys with
p e r s o n a l interviews.
M o n e t a r y incentives have been shown to be effective in increasing the q u a n t i t y of r e s p o n s e s
for mail surveys (see A r m s t r o n g 1975; F r i e d m a n
and A u g u s t i n e 1979; and Hansen 1980) and for surveys whose burdens are s u b s t a n t i a l l y g r e a t e r than
a p e r s o n a l interview, such as those w h i c h ask the
r e s p o n d e n t to submit to a p h y s i c a l e x a m i n a t i o n
(see Miller, Kennedy, and Bryant 1972).
Similarly, m o n e t a r y incentives have been shown to be effective in getting r e s p o n d e n t s to accept g r e a t e r
burden in p e r s o n a l interviews, such as agreeing
to complete more m o d u l e s in return for an additional p a y m e n t (see C h r o m y and H o r v i t z 1978).
However, previous e m p i r i c a l r e s e a r c h has been inc o n c l u s i v e about the effects of p a y m e n t s on
search efficacy and i n t e r v i e w completions for personal interviews, due largely to small sample
sizes and the consequent i m p r e c i s i o n of estimates
(see Ferber and Sudman 1974).
Furthermore, for data quality, the current
state of research leaves even the d i r e c t i o n of
the e f f e c t - - p o s i t i v e or n e g a t i v e - - i n serious
doubt (see Cannell and Henson 1974; Ferber and
Sudman 1974; F r i e d m a n and A u g u s t i n e 1979; and
Hansen 1980).
While it is h y p o t h e s i z e d that
m o n e t a r y payments will help r e s p o n d e n t s p r o v i d e
more t h o u g h t f u l and accurate answers to q u e s t i o n ~
sample m e m b e r s who are m o t i v a t e d to r e s p o n d because of the p a y m e n t may provide responses that
are of p a r t i c u l a r l y low quality.
The e v a l u a t i o n of the economic impact of Job
Corps p r o v i d e d a large n a t i o n a l p r o b a b i l i t y sample for which m o n e t a r y incentives were e x p e c t e d
to be effective.2--/ This sample included a longitudinal panel of over 5,800 e c o n o m i c a l l y disadv a n t a g e d youths, a p p r o x i m a t e l y 70 p e r c e n t of whom
were offered p a y m e n t s on the basis of r a n d o m assignments in an e x p e r i m e n t a l design.
The effects
2.
THE S E T T I N G FOR THE E X P E R I M E N T
The Job Corps p r o g r a m p r o v i d e s job training,
basic education, and r e l a t e d services in a residential setting to e c o n o m i c a l l y d i s a d v a n t a g e d
youths b e t w e e n the ages of 16 and 21.
Our d e s i g n
for the e v a l u a t i o n of the economic impact of Job
Corps included a p e r s o n a l b a s e l i n e i n t e r v i e w that
was a d m i n i s t e r e d in early 1977 to C o r p s m e m b e r s
258
who were then enrolled in centers and a series of
personal follow-up interviews a d m i n i s t e r e d perio d i c a l l y after these C o r p s m e m b e r s left the program.~/
The baseline interviews obtained inform a t i o n on Corpsmembers' previous work experience,
training, education, and related behavior, as
well as on their d e m o g r a p h i c and socioeconomic
backgrounds.
The follow-up interviews extended
these data with information on the p o s t p r o g r a m
experiences of Corpsmembers, so that at each
follow-up we attempted to r e i n t e r v i e w those C o r p ~
members who had been out of Job Corps for a sufficient p e r i o d of time.
In addition to Corpsmembers, we interviewed on the same schedule a comparable group of d i s a d v a n t a g e d youths who had
not a t t e m p t e d to enroll in Job Corps.
O b s e r v a t i o n s were obtained from two area prob a b i l i t y samples--one for C o r p s m e m b e r s and another for the comparison group.
The Job Corps
sample included all C o r p s m e m b e r s who were in the
p r o g r a m during May 1977 and who lived in threedigit zip-code areas (the Primary Sampling Units,
PSUs) that were r a n d o m l y selected.
The comparison group was drawn in 15 geographic sites scattered throughout the country.
These comparison
PSUs were also selected based on zip-code areas
but w i t h a stratified p r o b a b i l i t y procedure designed b o t h to yield a r a n d o m sample of sites
similar to the areas from w h i c h C o r p s m e m b e r s
came and to o v e r s a m p l e locations in which Job
Corps did not recruit heavily in 1977.
Within
comparison sites, youths were randomly selected
from lists of school dropouts and employment service registrants, stratified to yield a probability sample similar on average to C o r p s m e m b e r s in
terms of age, educational level, length of time
out of school, race-ethnicity, poverty status,
and related socioeconomic c h a r a c t e r i s t i c s . ~ /
3.
THE E X P E R I M E N T A L
m e a s u r a b l e effects on survey response for a longitudinal sample of e c o n o m i c a l l y d i s a d v a n t a g e d
youths and a personal interview of moderate
length (approximately one-half hour).
Sample Size 9 n d ALlocation.
The design decisions on the appropriate sample size and allocation also reflect the constraints under which
the payments experiment was conducted.
First,
the overall sample size was d e t e r m i n e d by the
primary objective of the study--i.e., evaluating
the economic impact of Job Corps.
Second, in
allocating this sample to payment and n o n p a y m e n t
statuses, the needs of the payments experiment
had to be traded against the needs of the primary objective of the study.
O n the one hand,
the experiment with payments w o u l d have been
served best by assigning 50 percent of the sample to payment and 50 percent to n o n p a y m e n t
status.
On the other hand, the p r e f e r r e d option
for m e e t i n g the primary objective of the evaluation would have been to assign the entire sample
to payment status.
Our r e s o l u t i o n of this sample a l l o c a t i o n
tradeoff was to assign 70 percent of-the sample
to payment and 30 percent to nonpayment status.
With this sample allocation, adequate statistical power was obtained to detect payment effects
that were large enough in magnitude to be relevant without jeopardizing the primary objective
of the evaluation.
For the follow-up surveys,
if the "true" effect for a p r o p o r t i o n a t e variable were 0.050, for example, we would have a
r e a s o n a b l y good chance of observing a statistically significant finding (approximately 90
percent power for a I0 percent significancelevel test), even though the effect on the overall sample would be a r e l a t i v e l y small 0.015
(i.e., 0.30 times 0.050 equals 0.015).
E x p e r i m e n t a l Assignments.
A random procedure
was used to assign sample to either payment or
nonpayment status based on the geographic areas
which formed the PSUs.
This area clustering of
p a y m e n t s was adopted for three reasons.
First,
some sample members lived close together w i t h i n
geographic areas.
Thus, varying experimental
assignments within sites would likely have
caused biases from information being spread to
some youths in the nonpayment group that others
were receiving payments.
Second, an area saturation of payments was desired %n order to be able
to engender community support when attempting to
locate respondents.
Third, an area p r o b a b i l i t y
a s s i g n m e n t was convenient, because both the Job
Corps and comparison groups had been selected in
that way at baseline and the same PSUs could be
used.
DESIGN
The objective of the payments e x p e r i m e n t was
to test w h e t h e r or not m o n e t a r y incentives could
be an effective survey tool for increasing the
quantity and quality of r e s p o n s e s to personal interviews.
The design of this e x p e r i m e n t was constrained in three ways:
(i) the u n d e r l y i n g population was limited to e c o n o m i c a l l y d i s a d v a n t a g e d
youths, (2) the payments e x p e r i m e n t was a secondary objective of the overall project, and (3) only a small amount of resources could be used for
fielding and analyzing the payments experiment.
It was w i t h i n the context of the above objective
and constraints that the e x p e r i m e n t a l - d e s i g n
issues were decided.
A m o u n t and Type of PaYment.
Because of limitations with the u n d e r l y i n g p o p u l a t i o n and the
available sample, we decided to limit the payments to a fixed amount of money.
V a r y i n g the
amount or type of payment to learn more about
m o n e b a r y incentives w o u l d have been very useful.
However, we first w a n t e d to be able to learn
whether m o n e t a r y payments could be effective, and
the responses from our limited p o p u l a t i o n w o u l d
not allow accurate estimates for more general
populations.
Also, the implicit sample sizes
were too small to support e x p e r i m e n t a t i o n with
several alternative payment amounts and mechanisms.
The amount and type of p a y m e n t per interview were set at $5 checks or money orders,
based on expectations that this would lead to
4.
FIELD PROCEDURES
FOR THE E X P E R I M E N T
The surveys u n d e r l y i n g this experiment consisted of a baseline and two follow-up interviews with the baseline respondents forming the
sampling frame for subsequent follow-ups.
At
baseline we attempted to interview a p r o b a b i l i t y
sample of Corpsmembers who were then in residence at centers and a p r o b a b i l i t y sample of
comparison youths.
In each of the two followup surveys we attempted to r e i n t e r v i e w everyone
in the comparison group who had been interviewed
at baseline and those Corpsmembers interviewed
at baseline who had been out of Job Corps for a
259
p r o x i m a t e l y nine months after b a s e l i n e and included e v e r y o n e in the c o m p a r i s o n group who comp l e t e d the b a s e l i n e i n t e r v i e w and e v e r y o n e in
the C o r p s m e m b e r group who both c o m p l e t e d the
b a s e l i n e i n t e r v i e w and had been out of Job Corps
for at least four months.
All the i n t e r v i e w s
for the first f o l l o w - u p were again a d m i n i s t e r e d
in p e r s o n and p r i m a r i l y in the youths' homes.
However, there w e r e two m a j o r changes that affected the p a y m e n t s e x p e r i m e n t at the first follow-up--(1) prior n o t i f i c a t i o n of the p o t e n t i a l
m o n e t a r y p a y m e n t was given to e v e r y o n e who was
to be paid and (2) the C o r p s m e m b e r group was included in the experiment.
In an advance m a i l i n g to all p o t e n t i a l resp o n d e n t s a l e r t i n g them to the first follow-up
and r e q u e s t i n g address updates, we n o t i f i e d
those in the p a y m e n t group that they w o u l d be
paid for interviews.
In addition, the c o m p a r i s o n
sample m e m b e r s in the p a y m e n t group were told at
b a s e l i n e that they w o u l d again receive $5 paym e n t s if and w h e n they were r e i n t e r v i e w e d .
The second f o l l o w - u p survey was c o n d u c t e d app r o x i m a t e l y 14 m o n t h s after the first and made
use of m o s t of the same procedures.
However,
there were two notable changes--(1) a d d i t i o n a l
C o r p s m e m b e r s from b a s e l i n e had left Job Corps
and were included in the second f o l l o w - u p and
(2) the i n t e r v i e w i n g was changed from i n - p e r s o n
to a m i x e d - m o d e p r o c e d u r e of t e l e p h o n e attempts
f o l l o w e d by in-person a t t e m p t s for a p o r t i o n
(over 60%) of those who could not be i n t e r v i e w e d
by telephone.
Both the i n c l u s i o n of additional C o r p s m e m b e r s
and the m i x e d - m o d e i n t e r v i e w i n g for the second
f o l l o w - u p c o m p l i c a t e the a n a l y s i s of the paym e n t s experiment.
The i n c l u s i o n of a d d i t i o n a l
C o r p s m e m b e r s m e a n s that even by the time of the
second f o l l o w - u p some sample members were b e i n g
paid for interviews for the first time.
The
m i x e d - m o d e i n t e r v i e w i n g m e a n s that the results
for the second f o l l o w - u p cannot be c o m p a r e d
d i r e c t l y to those for the first f o l l o w - u p and
baseline.
Consequently, to s i m p l i f y interpretations, we have limited the second f o l l o w - u p
a n a l y s i s to o b s e r v a t i o n s of sample m e m b e r s who
were i n t e r v i e w e d at both earlier surveys and who
were to be a t t e m p t e d in p e r s o n if the telephone
a t t e m p t s were unsuccessful.-71/
s u f f i c i e n t p e r i o d of time.
The b a s e l i n e q u e s t i o n n a i r e was d e s i g n e d to c o ~
lect d e t a i l e d i n f o r m a t i o n on seven topics:
soc i o e c o n o m i c background, Corpsmembers' r a t i n g s of
Job Corps, d e m o g r a p h i c c h a r a c t e r i s t i c s , employment and income, e d u c a t i o n and training, criminal
behavior, and a d d r e s s e s for survey tracking.
The
first f o l l o w - u p o b t a i n e d u p d a t e d i n f o r m a t i o n on
the last six of these topics and the second follow-up o b t a i n e d u p d a t e d i n f o r m a t i o n on the last
five.
All of the b a s e l i n e i n t e r v i e w s were administered in p e r s o n - - f o r C o r p s m e m b e r s , at their centers; for c o m p a r i s o n youths, at their homes.
No
prior contact or n o t i f i c a t i o n of the b a s e l i n e
survey was given to sample members, so those who
were to receive p a y m e n t s were informed of them
through the search process or w h e n contacted.6--/
However, a more serious l i m i t a t i o n on the payments e x p e r i m e n t at b a s e l i n e was that only comp a r i s o n - g r o u p m e m b e r s could be paid at that time,
b e c a u s e the C o r p s m e m b e r sample was still r e s i d i n g
at centers and r e c e i v i n g pay a l l o w a n c e s from Job
Corps.
It was e x p e c t e d that p a y m e n t s to Corpsm e m b e r s at b a s e l i n e w o u l d be u n n e c e s s a r y and
hence ineffective while they were in r e s i d e n c e at
centers (substantiated by a 97% c o m p l e t i o n rate).
More importantly, it was feared that such paym e n t s w o u l d be d i s r u p t i v e at centers and the results w o u l d be b i a s e d by the p r e s e n c e of b o t h
p a y m e n t and n o n p a y m e n t samples at the same centers.
If, instead, the e x p e r i m e n t a l a s s i g n m e n t s
were made on the basis of centers, the interm i n g l i n g of p a y m e n t and n o n p a y m e n t samples w o u l d
have been s u b s t a n t i a l for the f o l l o w - u p surveys.
The r e l a t i v e l y small sample sizes for the payments e x p e r i m e n t at baseline, as shown in Table
i, r e f l e c t the fact that the p a y m e n t s e x p e r i m e n t
was i n s t i t u t e d only in c o m p a r i s o n sites at that
time.
The sample sizes for search e f f i c a c y and
i n t e r v i e w c o m p l e t i o n s are b a s e d on the n u m b e r of
a t t e m p t e d interviews, and those for data q u a l i t y
are based on full interviews.
Furthermore, only
5 of the 15 c o m p a r i s o n sites were rural, and
search efficacy was e x p e c t e d to (and did) differ
s u b s t a n t i a l l y b e t w e e n rural and urban sites w i t h
g r e a t e r v a r i a t i o n in the rural sites.
All five
rural sites were a s s i g n e d to p a y m e n t status because they w o u l d not contribute m u c h i n f o r m a t i o n
on the e f f e c t i v e n e s s of p a y m e n t s and this ensured
b e t t e r data for the evaluation.
Consequently,
youths in three urban c o m p a r i s o n sites were rand o m l y a s s i g n e d to n o n p a y m e n t status at b a s e l i n e
w i t h g e o g r a p h i c a l l y s t r a t i f i e d a s s i g n m e n t s used
to increase the e f f i c i e n c y of s t a t i s t i c a l comparisons.
The analysis of search e f f i c a c y and i n t e r v i e w
c o m p l e t i o n s at b a s e l i n e c o n t r a s t s the e x p e r i e n c e
in the three n o n p a y m e n t sites with that in five
urban sites where m o n e t a r y incentives were offered.
Two urban p a y m e n t sites were o m i t t e d from
this analysis because of special fielding complications, w h i c h could bias e s t i m a t e s (one site because of p r i o r n o t i f i c a t i o n and the other b e c a u s e
of erroneous addresses).
The samples for analysis of data q u a l i t y include youths in these
latter two urban sites and the five rural sites,
b e c a u s e the r e s p e c t i v e fielding and g e o g r a p h i c
d i f f e r e n c e s should have little effect once an
i n t e r v i e w is begun.
The first f o l l o w - u p survey was c o n d u c t e d ap-
5.
FINDINGS
ON Q U A N T I T Y OF INTERVIEWS
Table 2 summarizes findings on search e f f i c a c y
and i n t e r v i e w c o m p l e t i o n s based on d i f f e r e n c e s in
sample m e a n s b e t w e e n the p a y m e n t and n o n p a y m e n t
groups.
In a d d i t i o n to estimates of d i f f e r e n c e s
in group means, for the f o l l o w - u p surveys the
a v a i l a b i l i t y of b a s e l i n e data e n a b l e d us to compute r e g r e s s i o n estimates that control for other
d i f f e r e n c e s b e t w e e n the two samples and to test
for d i f f e r e n c e s in effects among subgroups.
However, the r e g r e s s i o n e s t i m a t e s of o v e r a l l effects
are n e a r l y identical to the sample m e a n estimates
r e p o r t e d here, as e x p e c t e d w i t h a classic experim e n t a l design.
Also, we did not find any differences in effects across subgroups w i t h the
r e g r e s s i o n estimates.
S e a r c h Efficacy.
Two v a r i a b l e s can be used
to m e a s u r e search efficacy--(1) the p r o p o r t i o n of
the sample that r e t u r n e d p o s t c a r d s from the advance letters and (2) the p r o p o r t i o n of the
260
TABLE 1
SAMPLE SIZES F O R THE PAYMENTS
Sample
A.
For
i.
2.
3.
B.
For
i.
2.
3.
EXPERIMENT
paylnent G r o u p
A n a l y s i s of Search E f f i c a c y
Baseline
First F o l l o w - u p
Second F o l l o w - u p
A n a l y s i s of Data Q u a l i t y
Baseline
First Follow-up
Second F o l l o w - u p
Nonpayment
Group
and I n t e r v i e w C o m p l e t i o n s
sample that was located.
The first m e a s u r e s the
c o s t - e f f e c t i v e n e s s of the search (those who return p o s t c a r d s are easier to locate), and the
second m e a s u r e s the ultimate success of the
search.
The advance letters included a stamped postcard that was to be r e t u r n e d w i t h a d d r e s s and
t e l e p h o n e number updates or v e r i f i c a t i o n s .
At
the first f o l l o w - u p (see Table 2) there was only
a very small and m a r g i n a l l y s i g n i f i c a n t increase
in the p r o p o r t i o n of the p a y m e n t sample that returned these postcards, as compared to the nonp a y m e n t sample.
However, by the second followup, there was a large and s t a t i s t i c a l l y significant increase for the p a y m e n t group.
For the first follow-up, the sample m e a n difference b e t w e e n p a y m e n t and n o n p a y m e n t groups
shows an increase of just over 1 r e t u r n e d postcard for every i00 sample members, from a base
of a p p r o x i m a t e l y 27 r e t u r n e d p o s t c a r d s per i00
sample members in the n o n p a y m e n t group.
For the
second follow-up, the d i f f e r e n c e is 1 r e t u r n e d
p o s t c a r d for every i0 sample members, w i t h a statistical s i g n i f i c a n c e of one percent.
Furthermore, this estimate for the second f o l l o w - u p is
a s s o c i a t e d w i t h a small sample base of only 17
r e t u r n e d p o s t c a r d s per i00 sample members in the
n o n p a y m e n t group.
These findings suggest that,
in a l o n g i t u d i n a l study, m o n e t a r y p a y m e n t s can
reduce the costs of l o c a t i n g r e s p o n d e n t s and increase the r e s p o n s e s to mail surveys.
The second m e a s u r e of search e f f i c a c y is the
p e r c e n t of the sample that was located.
For the
b a s e l i n e survey we were able to locate just over
71 p e r c e n t of the n o n p a y m e n t sample--a low base
b e c a u s e of the i n a c c u r a t e a d d r e s s e s o r i g i n a l l y
o b t a i n e d for the c o m p a r i s o n sample.
We found a
r e l a t i v e l y large (6 p e r c e n t a g e point) and statistically s i g n i f i c a n t increase in the p r o p o r tion of the sample located for the p a y m e n t group
(up from 0.714 to 0.774).
However, the statistical significance is b i a s e d upward b e c a u s e of
the extreme c l u s t e r i n g and the c o n s e q u e n t potential for systematic interviewer error w i t h the
b a s e l i n e findings (the c l u s t e r i n g effect is subs t a n t i a l l y less for the f o l l o w - u p data).
For the first f o l l o w - u p survey, we were able
to relocate a p p r o x i m a t e l y 87 p e r c e n t of the sample and found v i r t u a l l y no d i f f e r e n c e in our
ability to locate youths in either the p a y m e n t
group or the n o n p a y m e n t group.
By the time of
the second follow-up, however, our ability to
locate youths in the n o n p a y m e n t group b e g a n to
decline substantially, as o p p o s e d to m a i n t a i n i n g
nearly an 87 p e r c e n t rate of success in l o c a t i n g
the youths in the p a y m e n t group.
This d i f f e r e n c e
in our ability to locate youths for the second
f o l l o w - u p was a p p r o x i m a t e l y 4 p e r c e n t a g e points
666
3,126
2,270
447
1,262
603
1,185
2,279
1,718
311
934
430
(87 p e r c e n t as o p p o s e d to 83 percent) and statistically significant.
In summary, both of our m e a s u r e s of search
e f f i c a c y show some clear gains to r e s p o n d e n t payments for a l o n g i t u d i n a l survey.
However, the
gains for a single c r o s s - s e c t i o n a l survey are
less.
I n t e r v i e w Completions.
Once a sample m e m b e r
is located, the only r e a s o n for not c o m p l e t i n g
an i n t e r v i e w is a refusal on the part of the
respondent.
In this section we c o n s i d e r empirical e v i d e n c e on the effects of r e s p o n d e n t payments on refusal rates and on the p r o p o r t i o n of
the sample with usable data.
The o v e r a l l refusal rates were very low for
our survey (always less than 3%) b e c a u s e we made
a great effort to convert refusals.
Therefore,
the p a y m e n t effect on refusal rates could not be
very large in magnitude.
For the b a s e l i n e and
first f o l l o w - u p surveys, there was v i r t u a l l y no
d i f f e r e n c e in the refusal rate b e t w e e n the paym e n t and n o n p a y m e n t groups.
For the second follow-up survey, £here was nearly a 2 p e r c e n t a g e
p o i n t lower refusal rate for the p a y m e n t group,
w h i c h was s t a t i s t i c a l l y s i g n i f i c a n t and r e d u c e d
the r e f u s a l s by nearly half.
The refusal rate
i n c r e a s e d over time for the n o n p a y m e n t group but
held r e l a t i v e l y c o n s t a n t for the p a y m e n t group.
The e s t i m a t e d p a y m e n t effect on the p r o p o r t i o n
of sample w i t h usable data r e f l e c t the findings
for the p r o p o r t i o n of sample located and the
refusal rate.
We find a p o s i t i v e effect at baseline, v i r t u a l l y zero effect for the first followup, and a p o s i t i v e and s t a t i s t i c a l l y s i g n i f i c a n t
effect for the second follow-up.
Even though the
overall c o m p l e t i o n rates for the second f o l l o w - u p
were very high, we o b t a i n e d a p p r o x i m a t e l y 5
a d d i t i o n a l c o m p l e t i o n s per i00 sample m e m b e r s in
the p a y m e n t group.
6.
FINDINGS
ON D A T A Q U A L I T Y
We do not have data from s e c o n d a r y sources
w i t h w h i c h to d i r e c t l y v a l i d a t e survey data.
However, we do have two types of proxy m e a s u r e s
for data quality:
(I) i n t e r v i e w c o m p l e t i o n rates
and (2) item n o n r e s p o n s e in c o m p l e t e d interviews.
A higher q u a n t i t y of responses, as found above
for the p a y m e n t group in the second follow-up,
should lead to higher quality data through reduced n o n r e s p o n s e bias.
M e a s u r e s of item nonresponse ("Don't know," refused, and b l a n k answers) are e x p e c t e d to r e f l e c t t h e a m o u n t of
thought and a c c u r a c y r e s p o n d e n t s give to difficult questions.
As shown in Table 3, the e s t i m a t e d impacts of
p a y m e n t s on item n o n r e s p o n s e w e r e most p r o n o u n c e d
at b a s e l i n e and d e c l i n e d over time.
By the
SEARCH
Variable
P r o p o r t i o n o f "sample
that returned postcards
P r o p o r t i o n of sample
located
P r o p o r t i o n located who
r e f u s e d the i n t e r v i e w
P r o p o r t i o n of sample
w i t h usable data
EFFICACY
TABLE 2
AND INTERVIEW
COMPLETIONS
Baseline
NonPayment
Grou p
Difference
F i.rst F o l l o w - U
NonPayPayment
ment
Group
Group
n.a.
n.a.
n.a.
.277
.774
.714
.060***
.867
.029
.025
.004
.751
.696
.055***
Payment
Grou p
.266
p
Second
Difference
Payment
Group
Fo!low-Up
NonPayDifment
ferGroup
ence
. 0 1 1 ". . . . . ..271
.....
.171
.i00"***
.875
-.008
.865
.826
.039***
.017
.023
-.006
.023
.040
.853
.856
-.003
.847
.796
-.017"*
.051"***
TABLE 3
DATA Q U A L I T Y
Variable
"Don't know" answers
per completed i n t e r v i e w
R e f u s e d answers per
completed interview
"Don't know" or r e f u s e d
answers per complete
B l a n k answers per
completed interview
Total Item n o n r e s p o n s e
per complete
Payment
Group
Baseline
NonPayment
Group
3.119
3.873
.331
.820
3.450
4.693
.815
.543
4.265
5.235
Notes to Tables 2 and 3:
*Significantly different
**Significantly different
***Significantly different
****Significantly different
n.a. means not applicable.
from
from
from
from
zero
zero
zero
zero
at
at
at
at
First
Payment
Group
Follow-Up
NonPayDifment
ferGrouP
ence
Payment
Group
-.754****
.608
.957
-.349****
.469
.426
.043
-.489*
.068
.211
-.143
.232
.347
-.115
.676
1.168
-.492****
.701
.772
-.071
1.602
1.967
-.365*
.001
.000
.001
2.279
3.134
-.855****
.703
.772
-.069
Difference
-1.243"***
.272***
-.970*
the
the
the
the
Second
Follow-Up
NonDifPayferment
ence
Group
20% level of s i g n i f i c a n c e (10% for a o n e - t a i l test).
10% level of s i g n i f i c a n c e (5% for a o n e - t a i l test).
5% level of s i g n i f i c a n c e (2.5% for a o n e - t a i l test).
1% level of s i g n i f i c a n c e (0.5% for a o n e - t a i l test).
ference is close to zero and positive.
The e s t i m a t e d p a y m e n t effect on r e f u s a l s to
answer i n d i v i d u a l q u e s t i o n s is similar to those
for "Don' t know" answers.
However, the p a t t e r n s
for r e f u s e d answers are not as pronounced, nor
are any of the i n d i v i d u a l d i f f e r e n c e s h i g h l y
significant.
D i f f e r e n c e s in total item n o n r e s p o n s e at b a s e line were p a r t i a l l y o f f s e t by c o u n t e r - i n t u i t i v e
b u t s t a t i s t i c a l l y s i g n i f i c a n t d i f f e r e n c e s in the
o p p o s i t e d i r e c t i o n for blank answers, w h i c h are
a c o m b i n a t i o n of i n t e r v i e w e r errors and uninterp r e t a b l e answers.
It is d i f f i c u l t to see h o w
p a y m e n t s w o u l d i n c r e a s e the number of b l a n k answers, e s p e c i a l l y b e c a u s e it can be shown that
they were c a u s e d p r i m a r i l y by i n t e r v i e w e r s neg l e c t i n g to indicate in the m a i n q u e s t i o n n a i r e
that they had w r i t t e n answers to c o n f i d e n t i a l
q u e s t i o n s on a separate form.
Total item n o n r e s p o n s e shows a p a y m e n t - r e l a t e d
d i f f e r e n c e at b a s e l i n e of just under one fewer
n o n r e s p o n s e per c o m p l e t e d i n t e r v i e w for the paym e n t group.
The base is s l i g h t l y over five nonr e s p o n s e s per i n t e r v i e w for the n o n p a y m e n t group.
At the first follow-up, the d i f f e r e n c e is only
s l i g h t l y smaller and is s t a t i s t i c a l l y s i g n i f i c a n t
even t h o u g h the base is just over half of w h a t
it was at baseline.
At the second follow-up, the
second follow-up, they were very small in m a g n i tude and s t a t i s t i c a l l y insignificant.
T h i s seemingly anomalous p a t t e r n of effects can be a t t r i b u t e d at least in part to the q u e s t i o n s b e c o m i n g
easier to answer over time (less c o m p l i c a t e d recall, more concrete questions, and i n c r e a s e d
f a m i l i a r i t y on the part of i n t e r v i e w e r s a n d respondents) and to the least c o o p e r a t i v e r e s p o n dents d r o p p i n g out of the n o n p a y m e n t g r o u p over
time.
The results for the number of "Don't know" answers per c o m p l e t e d i n t e r v i e w are the m o s t striking and conclusive.
The e s t i m a t e d e f f e c t of payments at b a s e l i n e is in the e x p e c t e d direction,
large, and s t a t i s t i c a l l y significant.
The magnitude of the effect d i m i n i s h e s for the first
f o l l o w - u p and v i r t u a l l y d i s a p p e a r s by the second.
At baseline, the e f f e c t is to reduce the number
of "Don't know" r e s p o n s e s by three for e v e r y four
c o m p l e t e d interviews.
The a v e r a g e n u m b e r of such
n o n r e s p o n s e s at b a s e l i n e is n e a r l y four per interv i e w in the n o n p a y m e n t group.
By the first follow-up, the d i f f e r e n c e fell to just over one
"Don't know" r e s p o n s e in every three interviews,
but the base had also fallen d r a m a t i c a l l y to
under one in i0 i n t e r v i e w s (as the q u e s t i o n s became easier to answer).
The base falls even
further for the second follow-up, and the dif-
262
1969), interviewers were able to locate only 60%
of the sample, and full interviews were obtained
only from 48% of the sample.
4
Sample members who did not respond at baseline
were not followed.
All those who responded at
baseline were followed in subsequent waves even
if they did not respond to an intervening interview.
5
For more details on the sample designs for the
Job Corps and comparison groups and the statistical methodology used to draw inferences about
p r o g r a m impacts, see Mallar et al. (1980).
6
In the process of searching for comparison
youths in the payments sample, information was
made readily available on the monetary b e n e f i t
to being interviewed.
7
Estimates were also computed for the full sample
at second follow-up and for the telephone attempts.
No substantial deviations were found
compared to the results presented in this paper.
base falls to approximately two item nonresponses
in every three interviews, and the experimental
difference disappears.
W h a t emerges from our findings is some tentative evidence that monetary incentives improve
data quality.
The primary measures on which this
conclusion is based are the proportion of c o m p l e ~
ed interviews and the number of "Don't know"
responses per completed interview.
The number of
completed interviews was significantly higher for
the payment group at the third interview, and
evidence from the first two waves of interviews
indicates that there were beneficial effects of
payments on item nonresponses.
7.
CONCLUSIONS
The findings from our experiment with payments
to survey respondents go far in establishing the
effectiveness of such payments.
The hypotheses
that payments will improve search efficacy and
interview completions for longitudinal surveys
are verified here for our sample of disadvantaged
youths.
After one or two interviews, our ability
to locate potential respondents and to obtain
data from those who were located deteriorated in
the absence of monetary incentives, but not when
$5 payments were offered to respondents.
In
addition, payment effects on the willingness of
sample members to return postcards from advance
letters suggest that m o n e t a r y incentives may be
an effective aid for mail surveys and for reducing the cost of locating respondents for personal
interviews.
The estimated impacts on item nonresponse were most pronounced at baseline and
declined over time.
The payment experiment shows some clear benefits to monetary incentives.
However, precise
inferences cannot be drawn about the size of
effects beyond our specific experiment with $5
payments to disadvantaged youths and using an
intensive search process.
The size of effects
will undoubtedly vary by amount of payment,
characteristics of the population, and intensity
of search procedures.
REFERENCES
Armstrong, J. (1975), "Monetary Incentives in
Mail Surveys," Public O p i n i o n Quarterly, 39,
111-116.
cannell, c. F. and Henson, R. (1974), "Incentive&
Motives, and Response Bias," Annals of Economic
and Social Measurement, 3/2, 307-317.
Chromy, J. R. and Horvitz, D. G. (1978), "The Use
of Monetary Incentives in National Assessment
Household Surveys," Journal of the American
Statistical Association, 73, 473-478.
Ferber, R. and Sudman, S. (1974), "Effects of
Compensation in Consumer Expenditure Studies, "
Annals of Economic and Social Measurement, 3/2,
319-331.
Friedman, H. H. and San Augustine, A. J. (1979),
"The Effects of a Monetary Incentive and the
Ethnicity of the Sponsor's Signature on the
Rate and Quality of Response to a Mail Survey,"
Journal of the Academy of Marketing Science,
7, 95-I01.
Hansen, R. A. (1980), "A Self-Perception Interp r e t a t i o n of the Effect of Monetary and Nonmonetary Incentives on Mail Survey Respondent
Behavior," Journal o f Marketing Research, 17,
77-83.
Harris, L., et al. (1969), A Survey of Ex-Job
Corpsmen, New York:
Louis Harris Associates.
Mallar, C. D., et al. (1980), "Evaluation of the
Economic Impact of the Job Corps Program:
First Follow-Up Report," Mathematica Policy
Research, Inc.
Miller, H. W., Kennedy, J., and Bryant, E. E.
(1972), "A Study of the Effect of Remuneration
Upon Response in a Health and Nutrition Examination Survey," Proceedings of the Social S t a t i ~
tics Section of the American Statistical
Association, 370-375.
FOOTNOTES
1Funding for the research reported in this paper
was provided by the Office of Program Evaluation
of the Employment and Training A d m i n i s t r a t i o n of
the U.S. Department of Labor.
2For more details on the evaluation, see Mallar
et al. (1980).
3
Youths in the age range of Corpsmembers (16 to
21) and with their economically disadvantaged
backgrounds are generally very mobile and difficult to locate.
From Census data, approximately
50% of them can be expected to move in any given
year, with 50% of these moves being outside their
initial metropolitan area.
In a previous, onetime survey of former Corpsmembers (see Harris
263