Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
Bootstrap Tests of Hypothesis for Survey Data
Jean-François Beaumont1, Cynthia Bocci2
Statistics Canada, R.H.Coats Building, 11th Floor, 150 Tunney’s Pasture Driveway, Ottawa, Ontario, K1A 0T6 Canada1
Statistics Canada, R.H.Coats Building, 11th Floor, 150 Tunney’s Pasture Driveway, Ottawa, Ontario, K1A 0T6 Canada2
hypothesis testing with survey data is that, once a set
of bootstrap weights are provided to the analysts, it is
very easy to implement using standard software
packages that ignore sampling design features.
Therefore, the method can be applied without requiring
specialized software dealing with complex surveys.
Abstract
The bootstrap technique is becoming more and more
popular in sample surveys. So far, the use of the
technique in practice seems to have been mostly
limited to variance estimation problems. In this paper,
we propose a bootstrap methodology for testing
hypotheses about a vector of unknown model
parameters when the sample has been drawn from a
finite population according to a probability sampling
design which may or may not be informative. We
show through a simulation study that it performs well
in comparison with other methods proposed in the
literature when testing hypotheses about a vector of
linear regression model parameters.
We introduce notation and describe the problem in
section 2. In section 3, we describe and justify our
proposed bootstrap methodology for testing
hypotheses. In section 4, we briefly discuss the results
of a simulation study in order to compare the proposed
bootstrap method with other methods of testing
multiple hypotheses which do not rely on the
bootstrap. Finally, we draw some conclusions in the
last section.
Keywords: Complex surveys, Hypothesis testing,
Informative sampling, Pivotal statistic.
2. Preliminaries
We are interested in testing hypotheses about a vector
of unknown model parameters β . We assume that the
finite population U of size N has been generated
according to a model specified by the analyst that
describes the conditional distribution F (yU | XU ; β, θ) ,
1. Introduction
The bootstrap technique is becoming more and more
popular in sample surveys. In most of its applications,
several sets of bootstrap weights accompany the survey
microdata file provided to analysts. So far, the use of
the technique in practice seems to have been limited to
variance estimation problems; mostly for finite
population parameters but at times for analytical
parameters (also referred to as model parameters or
infinite population parameters). Methods for testing
hypotheses on analytical parameters have also been
studied and developed in classical theory (for example,
see Efron and Tibshirani, 1993, Chapter 16; or Hall
and Wilson, 1991). However, there does not seem to
exist in the literature a bootstrap methodology for
testing multiple hypotheses on analytical parameters
when a sample has been taken from a finite population
according to a probability sampling design which may
or may not be informative. In the article, we
investigate this particular problem.
where yU is a vector that contains the population
values of a dependent variable y, XU is a matrix that
contains the population values of a vector of
independent variables x and θ is a potential vector of
additional unknown model parameters that are not of
interest for testing purposes. We also assume that, if
the entire population U could be observed, an
asymptotically pivotal statistic t (yU ; c) , i.e. a statistic
which has an asymptotic distribution that does not
depend on any unknown parameter, would be used to
test the multiple linear hypothesis H 0 : Hβ = c against
the alternative hypothesis H1 : Hβ ≠ c . The Q-row
matrix H is used to define the hypothesis to be tested
and c is a Q-vector of constants specified by the
analyst.
The proposed method uses “classical” statistics
although weighted by survey weights based on the
potentially invalid assumption that the sampling is not
informative. We approximate the distribution under
the null hypothesis of these weighted model-based
statistics by using bootstrap weights. The advantage
of our bootstrap method over existing methods of
Suppose now a random sample s of size n is selected
from the finite population according to a given
probability sampling design p(s ) . Consider the
weighted test statistic tˆ(y s , w s ; c) where the vector
y s represents the sample portion of y U and w s is a
739
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
sample vector containing design-based or modelassisted survey weights.
Note that the statistic
tˆ(y s , w s ; c) is such that tˆ(y s ,1 s ; c) = t (y s ; c) where
bootstrap weights for k ∈ s . Furthermore, we assume
that the bootstrap weights are constructed in a way that
captures the variability induced by the sampling design
and the model. In other words, we assume that the
bootstrap distribution of tˆ(y s , w *sb ; c) is a good
approximation to the distribution of tˆ(y s , w s ; c) under
1 s is a sample vector of one’s. In other words,
tˆ(y s ,1 s ; c) is the classical statistic based on the
hypothesis that the sampling design is not informative.
The statistic tˆ(y s , w s ; c) is simply the weighted
version of tˆ(y s ,1 s ; c) .
the sampling design and the model. Binder and
Roberts (2003) suggest that the variability due to the
model is small with respect to the variability due to the
design when the sampling fraction is small. In this
case, it is sufficient to use a valid bootstrap method
under the design such as that proposed by Rao and Wu
(1988), or also Rao, Wu and Yue (1992).
As an illustrative example, let us assume that y k , for
all population units k ∈ U , are independently and
identically distributed random variables with mean β
and variance θ and that we are interested in testing the
null hypothesis H 0 : β = c . In this case, a natural
sample statistic is given by
( βˆ − c) 2
tˆ(y s , w s ; c) = ws
,
θˆws n
where
We further assume that the distribution of
tˆ(y s , w s ; c) under H 0 which we would like to
approximate by the bootstrap method, is the same
whatever the value of c. This implies that it has the
same distribution as tˆ(y s , w s ; Hβ) under β . Given
the above assumption on the bootstrap weights, the
latter distribution could be approximated by the
bootstrap distribution tˆ(y s , w *sb ; Hβ) if β were
known. As β is not know, it is replaced by a consistent
estimator
forming
the
bootstrap
statistic
*
b
ˆ
tˆ(y s , w s ; Hβ ws ) . Hence, our procedure consists of:
β̂ ws = ∑k∈s wk y k n ,
θˆws = ∑ k ∈s wk ( yk − βˆ ws ) 2 (n − 1) ,
and wk , for k ∈ s , is the standardized survey weight
for unit k (i.e. wk is rescaled so that
∑ k∈s wk = n ).
i.
If the sampling design is a simple random sample or at
least not informative then the asymptotic distribution
of tˆ(y s , w s ; c) under the null hypothesis is usually
fairly simple to obtain.
This, however, is not
necessarily the case in the context of an informative
sampling design. Rao and Scott (1981) proposed a
method whereby the statistic tˆ(y s , w s ; c) is modified
so that it follows asymptotically a known distribution
under the null hypothesis (see also Rao and Thomas,
2003). A Wald test or the Bonferroni method (Korn
and Graubard, 1990) could equally be used.
Alternatively, we propose to approximate the
distribution of tˆ(y s , w s ; c) under the null hypothesis
by using the set of bootstrap weights that accompany
the data files of a survey. This method is described in
the next section.
ii.
iii.
Obtaining the statistic tˆ(y s , w *sb ; Hβˆ ws ) for
b = 1,..., B .
Calculating the observed significance level
# tˆ(y s , w *sb ; Hβˆ ws ) > tˆ(y s , w s ; c)
B
supposing that the null hypothesis is rejected
for large positive values of the statistic
tˆ(y s , w s ; c) .
Rejecting the null hypothesis if the observed
significance level is smaller than a
predetermined
significance
level
(for
example, 5%).
{
}
4. Simulation Study
We performed a simulation study to evaluate the
validity and the power of the proposed bootstrap
method in the cases of an informative and noninformative sampling design under the context of a
one- factor analysis of variance model with five levels.
We compared our method to those of Rao-Scott, Wald,
Bonferroni and a naïve method which consists of using
tˆ(y s , w s ; c) assuming a simple random sample design.
The Rao-Scott and Wald procedures were calculated
using SUDAAN software (Research Triangle Institute,
3. Proposed Bootstrap Method
Suppose that the weight wk has been normalized.
Consider the normalized bootstrap weights, wk*b ,
obtained in a similar fashion for k ∈ s and b = 1,..., B ,
where B is the number of bootstrap replicates so that
∑k∈s wk*b = n . Denote w *sb denote the vector of
740
Papers presented at the ICES-III, June 18-21, 2007, Montreal, Quebec, Canada
2004). More details on the simulation can be found in
Beaumont and Bocci (2007). Here we show rejection
rates for the null hypothesis in the case of informative
sampling.
Binder, D.A., and Roberts, G.R. (2003). Design-based
and model-based methods for estimating model
parameters. Analysis of survey data, edited by R.L.
Chambers and C.J. Skinner. Wiley, New-York.
Efron, B., and Tibshirani, R.J. (1993). An introduction
to the bootstrap. Chapman & Hall, New-York.
Hall, P., and Wilson, S.R. (1991). Two guidelines for
bootstrap hypothesis testing. Biometrics, 47, 757762.
Korn, E.L., and Graubard, B.I. (1990). Simultaneous
testing of regression coefficients with complex
survey data: use of Bonferroni t statistics. The
American Statistician, 44, 270-276.
Rao, J.N.K., and Scott, A.J. (1981). The analysis of
categorical data from complex sample surveys:
chi-squared tests for goodness of fit and
independence in two-way tables. Journal of the
American Statistical Association, 76, 221-230.
Rao, J.N.K., and Wu, C.F.J. (1988). Resampling
inference with complex survey data. Journal of the
American Statistical Association, 83, 231-241.
Rao, J.N.K., Wu, C.F.J., and Yue, K. (1992). Some
recent work on resampling methods for complex
surveys. Survey Methodology, 18, 209-217.
Rao, J.N.K., and Thomas, D.R. (2003). Analysis of
categorical response data from complex surveys:
an appraisal and update. Analysis of survey data,
edited by R.L. Chambers and C.J. Skinner. Wiley,
New-York.
Research Triangle Institute (2004). SUDAAN language
manual, release 9.0. Research Triangle Park, NC:
Research Triangle Institute.
Table 1 shows that the naïve method is too
conservative which leads to a blatant loss of power
while the Wald method is too liberal. The Bonferroni
method improves the situation but not enough. The
Rao-Scott and the proposed bootstrap method give
similar results with a rejection rate close to 5% when
the null hypothesis is true. Moreover, the proposed
method is the one that has a rejection rate closest to 5%
when the null hypothesis is true.
Table 1 : Rejection rates for testing H 0 in the case
of informative sampling.
Method
H 0 true
H 0 false
Naïve
0.5%
5.0%
Wald
16.0%
38.1%
Bonferroni 12.9%
33.3%
Rao-Scott 7.7%
21.5%
Bootstrap
6.9%
20.9%
5. Conclusion
In this article we have proposed a bootstrap
methodology for testing multiple hypotheses when the
sample is taken from a finite population according to a
sampling design. In the context of an informative
sampling design, we have demonstrated empirically
that the proposed method is competitive with other
current methods in the literature. An advantage of this
bootstrap method is that it is easy to implement once
the bootstrap weights are produced since tˆ(y s , w s ; c)
and tˆ(y s , w *sb ; Hβ) can usually be easily obtained
using standard software (such as SAS, for example)
which ignores sample design characteristics. The
proposed method can therefore be applied without
having access to specialized software created for
complex surveys.
Acknowledgements
We sincerely thank J.N.K. Rao from Carleton
University as well as Yves Lafortune from Statistics
Canada for their useful comments and for discussions.
References
Beaumont, J.-F., and Bocci, C. (2007). A practical
bootstrap method for testing hypotheses from
survey data. Article submitted for publication.
741