Encouragement Designs: An approach to
self-selected samples in an experimental
design
Eric T. Bradlow
Dr. Bradlow is Assistant Professor of Marketing and Statistics, Wharton School of Business,
University of Pennsylvania, Philadelphia PA 19104. The author thanks Professor Robert J. Meyer,
Marketing Department, Wharton School of Business for helpful comments.
1
Abstract
When subjects choose their own treatment (are self-selected into treatments
based on their varying compliance with assignment to treatment states), many
of the well-developed techniques of randomization-based experimental design
and analysis are no longer available. With random assignment to treatment and
control conditions, we can reasonable assume that over many replications, the
two groups will be similar in all respects (observed and not observed) but that
of receiving the treatment. With non-random assignment, it may be true that
the groups dier on the variable of interest prior to receiving the treatment, and
that a measured post-treatment dierence (or lack thereof) will be erroneously
attributed to the treatment. Our approach involves making all subjects aware
of the availability of the treatment, but then oering extra encouragement (an
encouragement design) to participate in the treatment to a randomly selected
half of the population. If the encouragement is successful, we show how this
leads to an estimable treatment eect with associated standard error. Two
examples involving measuring the eects of an infomercial on purchase behavior
are presented as well.
2
1 Introduction
A persistent problem of eld experiments is that it is not always possible to randomly
assign respondents to cells. When studying the eects of advertisements on buying
behavior, for example, exposure to an ad is not purely random but rather self-selected.
Standard methods of analysis assuming random assignment (Fisher 1925, Cochran
and Cox 1957) would require assumptions that are typically untenable in order to
yield valid inferences (Rubin 1987).
In this paper we re-examine an approach to overcome self-selection by overlaying the initial treatment assignment with an additional random assignment of an
encouragement condition (e.g. postcards, follow-up letters, phone calls, monetary
rewards). Utilizing (\borrowing randomness") from the encouragement condition under this structure allows for valid inference under a much weaker (believable) set of
assumptions than assuming that the self-selected groups are identical pre-treatment.
The encouragement design was rst implemented by Ball and Bogatz (1970, 1971)
in their evaluation of the TV program Sesame Street, in which a random subsample
of children were encouraged to view the program. Subsequent analyses using these
designs are seen in Cook (1975), Swinton (1975), and Powers and Swinton (1984).
This research formalizes a model for encouragement designs, formally provides a set of
four joint suciently conditions for valid inference under the design, as well as extends
the methodology to include standard error estimates of the eect of the treatment.
A more formal description of the initial non-compliance condition is given in Section 2. The formulation with the encouragement design is given in Section 3. In
Section 4 we give four jointly sucient conditions which yield valid estimates of the
treatment eect under the encouragement design. Two numerical examples are given
3
in Section 5. We nish with a small discussion, Section 6. Details of standard error
computation are given in the Appendix.
2 Inference under self-selection
To illustrate the approach consider a problem where the treatment is the viewing
of a television advertising campaign (e.g. an infomerical), the encouragement is an
awareness campaign regarding the viewing times of the infomercial, and the outcome
is the market share of the advertised product.
1:
Under self-selection of subjects into viewing status we observe results as in Table
Treatment
(2 = 2 + 1; p1)
No Treatment (1 = 1; 1 p1)
Table 1: Mean response and proportion opting for each condition.
A proportion, p1, of the subjects with mean pre-treatment market share 2 voluntarily undergo the treatment (watch the infomercial) and experience an eect of
treatment 1. The remaining proportion, 1 p1, subjects with mean 1 choose not
to undergo the treatment. The problem with such observational studies is that unlike the case of random assignment, where up to random sampling variation, we can
assume 1 = 2, here we have no (or little) idea of the relationship between choosing
to be treated and outcome. For example, we may believe that people who watch
the infomercial are predisposed to buying the product more so than those who don't
4
watch; 2 > 1. On the other hand, it is also plausible that the only persons who
watch the infomercial are those that need to be convinced to buy the product while
those that don't watch will already buy it; 2 < 1. Since there are diculties in
eectively randomizing and forcing subjects to comply with viewing assignments, a
partial randomization must be obtained in an alternative manner. If no alternative
randomization is obtained then the standard estimate of the treatment eect 2 1
will yield 1 (2 1) and not 1 as desired. Some recent work in this area discussing partial randomization (in economics called instrumental variables) is nicely
summarized in Imbens and Rubin (1996) and initially appeared in Zelen (1979) in a
biostatistics context.
3 Encouragement design
If we are justied in making certain strong but reasonable assumptions, we can overlay
a random partial manipulation on the self-selection (no-compliance) process, and
recover an estimate of the treatment eect 1 (the infomercial) uncontaminated by
any dierence in 1 and 2 (the self-selected market shares). The approach involves
making all subjects aware of the availability of a treatment (an infomercial), but then
oering encouragement to participate in the treatment (watch the infomercial) to a
randomly selected half of the population. Operationally, this could be achieved by
sending advertisements or television inserts to half of a randomly selected viewing
audience.
If the encouragement is successful, some additional proportion, p2=2 < 1=2, of the
subjects who would not have participated without encouragement now decide to take
the treatment (watch the infomercial). This process is described in Table 2.
5
Not Encouraged
Encouraged
Treatment
(2 = 2 + 1; p1 =2) (4 = 4 + 2 + 2; (p1 + p2 )=2)
No Treatment (1 = 1; (1 p1)=2) (3 = 3 + 1; (1 p1 p2)=2)
Table 2: Mean response and proportion opting for each condition under an encouragement design.
Half of the subjects receive no encouragement to watch the infomercial, a fraction
of whom (1 p1 )=2 elect not to watch it. These subjects have baseline market
share 1 = 1. The remainder of the non-encouraged sample p1=2 elect to watch the
infomercial and have baseline market share 2 = 2 + 1, where 1 is the eect of the
infomercial in the non-encouraged sample.
Half of the subjects receive encouragement to watch the infomercial, a fraction of
whom (1 p1 p2)=2 elect not to despite encouragement. These subjects have baseline
market share 3 = 3 + 1, where 1 is the eect of encouragement in the sample of
people who don't watch the infomercial. The remaining fraction of subjects (p1 +p2)=2
are encouraged to watch the infomercial and do watch it with baseline market share
4 = 4 + 2 + 2, where 2 is the eect of the infomercial in the encouraged group
and 2 is the eect of encouragement in those who chose to watch the infomercial.
In most studies conducted, the quantities of interest are 1 and 2 the eect of the
infomercial (the treatment), whereas the eects due to encouragement 1 and 2 are
nuisance parameters.
Sample based estimates of the proportions p1 and p2 and market shares 1; 2; 3,
and 4 are easily computed from the data as: p^1 = N2=(N1 + N2), p^1 + p^2 = N4=(N3 +
N4) ^i = Xi=Ni ; i = 1; 2; 3; 4 where Ni; Xi are the sample count and sample sum
6
corresponding to cell i respectively. Under a set of four jointly sucient conditions
described next, estimates of the eect of the infomercial are obtainable from the
estimated p's and 's.
4 Four Jointly Sucient Conditions
We present a set of four conditions under which a consistent estimate of the eect of
the infomercial is obtainable. The plausibility of each of these conditions is discussed
in detail.
(i) The eect of the infomercial is the same for the encouraged and not encouraged
conditions: 1 = 2.
(ii) The eect of encouragement is equal for the subjects who watch the infomercial
and those who don't: 1 = 2.
(iii) Everyone who would watch the infomercial when not encouraged would also
watch the infomercial if encouraged.
(iv) The additional proportion (p2=2) of subjects who watch the infomercial under
encouragement are a representative subgroup of those that do not watch the
infomercial when not encouraged.
We expect the plausibility of condition (i) in a general setting to be strongly associated with the type of encouragement provided. If the encouragement in itself could
have some interaction with the treatment, for example a customer who is \rudely"
encouraged, then this subject will unlikely purchase the product even after watch7
ing the infomercial. This condition puts the burden on the experimenter to nd an
encouragement unlikely to dierentially impact the eect of the treatment.
Condition (ii), similar to condition (i), is related to the interaction of the encouragement utilized and the treatment. In the simplest case, if encouragement has no
eect at all regardless of treatment status (1 = 2 = 0) then condition (ii) holds.
A plausible model however which would violate condition (ii) would state that encouragement has an additive fact for those with strong enough interest to watch the
infomercial 2 > 0, while no eect for those with a lack of interest to watch the infomercial (1 = 0). We expect that condition (ii) will have to be carefully considered
on a case by case basis.
Condition (iii) is maybe the most plausible of the assumptions and could be experimentally veried. If we assume that the form of encouragement is \friendly" or
benecial to the subject then we would imagine that any subject who would watch
the infomercial without encouragement certainly would with encouragement; and may
even be more likely to do so.
Condition (iv), also called \non-skimming", asserts that the additional subjects
who watch the infomercial after encouragement are representative of the subjects who
would not watch the infomercial without encouragement; or stated, encouragement
draws subjects representatively into watching and not watching the infomercial. A
close examination of this condition suggests it is highly suspect. We certainly must
consider the possibility that there exists a segment of the population who will not
watch the infomercial regardless of the encouragement. Other segmented populations
would also likely lead to violations of condition (iv).
Nevertheless, if we assume conditions (i)-(iv), then solving for = the encouragement eect and = the treatment eect provided in Table 2, yields
8
= 3 1;
= (1 + pp1 )(4 3) pp1 (2 1)
2
2
(1)
(2)
Estimates of these quantities are simply obtained by substituting in the sample
based proportions and cell market shares described in Section 3. Methods for computing an asymptotic standard error, condence interval, and test statistic for ^ are
provided in the appendix.
5 Numerical Examples
Consider an example where the observed outcome is the market share for a given consumer product. The corresponding treatment we consider is an infomercial presented
on television. The non-encouraged group receives no information as to when the
infomercial is scheduled to air while the randomly selected encouraged group would
receive a television insert highlighting the infomercial but not promoting the product in any way. This is critical so that the encouragement itself is not confounded
with the impact of the infomercial (conditions (i) and (ii)) , therefore enabling us
to separate out the eect of the television insert from any initial self-selected group
dierences in willingness to watch the infomercial.
We present two numerical examples. In the rst (Table 3) we examine a case in
which those who initially choose to watch the infomercial (the treatment condition)
are more likely to purchase the product with no encouragement (2 = 0:25 > 1 =
0:20), but the infomercial itself has no additional impact on purchasing behavior beyond that due to the encouragement.
9
This case corresponds to no treatment eect as given in (2)
Not Encouraged
Encouraged
Treatment
(2 = 0:25; p1=2 = 0:15)
(4 = 0:23; (p1 + p2 )=2 = 0:25)
No Treatment (1 = 0:20; (1 p1)=2 = 0:35) (3 = 0:20; (1 p1 p2 )=2 = 0:25)
Table 3: No treatment eect.
^ = (1 + pp^^1 )(^4 ^3) pp^^1 (^2 ^1)
2
2
0
:
3
= (1 + 0:2 )(0:23 0:20) 00::32 (0:25 0:20)
= 0:
This is not intuitive as the observed market share is larger in both treatment cells (3
and 4) than in no treatment cells (1 and 2). Clearly a standard analysis which did
not utilize the encouragement model would therefore erroneously attribute a positive
eect due to the treatment!
In a second example, if instead we observed data as in Table 4
then
Not Encouraged
Encouraged
Treatment
(2 = 0:55; p1=2 = 0:15)
(4 = 0:53; (p1 + p2 )=2 = 0:25)
No Treatment (1 = 0:20; (1 p1)=2 = 0:35) (3 = 0:20; (1 p1 p2 )=2 = 0:25)
Table 4: A 30% eect due to watching the infomercial.
^ = (1 + pp^^1 )(^4 ^3) pp^^1 (^2 ^1)
2
2
10
= (1 + 0:30=0:20)(0:55 0:20) 0:30=0:20(0:53 0:20)
= 0:30
a 30% absolute increase in purchase behavior attributable to the eect of the infomercial. The associated asymptotic standard error for ^, assuming a moderate sample of
size 600, is 0.1595 with corresponding z-statistic = 1.883 and p-value 0.03.
6 Summary
This research presents an approach to handle self-selected samples in an experiment.
We do not suggest that this research presents a solution in all cases to this very
pervasive problem; however, we do present a set of experimental conditions under
which an exact solution in some cases is possible. If severe self-selection is occurring
then no method will be able to avoid the direct modeling of the self-selection process
itself. Such models are likely to be case by case specic, dicult to implement, and
experimentally hard to verify. Therefore research (such as presented here) which
may \solve" the problem for a small set of all self-selection cases seen in practice is
extremely useful.
References
Ball, S. and Bogatz, G. A. (1970). The First Year of Sesame Street: An Evaluation,
Princeton, NJ: Educational Testing Service.
{ (1971). The Second Year of Sesame Street: An Evaluation, Princeton, NJ: Educational Testing Service.
11
Cochran, W. G., and Cox, G. (1957), Experimental Designs (ed. 2). New York:
Wiley.
Cook, T. D., Appleton, H., Conner, R. F., Tamkin, G., and Weber, S. J., (1975),
Sesame Street Revisited. New York: Russell Sage Foundation.
Fisher, R. A. (1925). Statistical Methods for Research Workers. Edinburgh: Oliver
and Boyd.
Imbens, G. W. and Rubin, D. B. (1996), \Bayesian Inference for Causal Eects in
Randomized Experiments with Noncompliance", Annals of Statistics, to appear.
Powers, D. E., and Swinton, S. S. (1984), \Eects of Self-Study for Coachable TestItem Types." Journal of Educational Psychology, 76: 266-78.
Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. New York:
Wiley.
Swinton, S. S. (1975), An Encouraging Note, unpublished manuscript.
Zelen, M. (1979), \A New Design for Randomized Clinical Trials", New England
Journal of Medicine, vol. 300, 1242-1245.
Appendix
The following algorithm is used to compute the standard error of ^. It is based on a
linearization approximation (the \delta method") to the likelihood described below.
We dene as in Table 2
Xi = the number of people in cell i = 1; :::4 who had the desired outcome
Ni = the number of people in cell i
i = the market share in cell i; P (Xij = 1); j = 1; : : : ; Ni
where i-th cell corresponds to the encourage/not encourage, treatment/no treatment
cell corresponding to rate i. We then assume that
Xi Binomial(Ni; i)
(N1; N2; N3; N4) Multinomial(N; (1 p1 )=2; p1=2; (1 p1 p2)=2; (p1 + p2)=2):
12
We further dene = (1; 2; 3; 4; p1; p2). Under this setting, the algorithm is given
by
(1) Compute ^i = Xi=Ni the MLE of for cell i = 1; :::; 4.
(2) Compute p^1 = N3=(N2 + N3), and p^2 = N4=(N3 + N4) N2=(N1 + N2), the
MLE's of p1 and p2.
(3) The vector of rst derivatives of with respect to , evaluated at the MLE,
d
p^1
d
p^1
d
p^1
d1 = p^2
d2 = p^2
d3 = (1 + p^2 )
p^1
d
d
d
^4 ^3 ^2 ^1
^ ^3 ^2 ^1) p^p^21
d4 = (1 + p^2 )
dp1 =
p^2
dp2 = (4
2
(4) Compute the estimated asymptotic variance covariance matrix -(d2=d2) 1 with
non-zero elements,
d2
N X
X
d2
N2
N4
d2
d2
N3
N4
d2 = ^2
(^p1 +^p2 )2 dp1 p2 = dp22 = (1 p^1 p^2 )2
(^p1 +^p2 )2
(1 ^ )2 dp21 = p^21
i
i
i
i
i
i
(5) Compute Var(^ ) = dd (
d2
d2
)
1 d 0 .
d
13