A Comparison of Variance Estimates
for Schools and Students Using Taylor Series
and Replicate Weighting
Ellen Scheib, Peter H. Siegel, and James R. Chromy
RTI International
Presented at
Third International Conference on Establishment Surveys (ICES-III)
June 21, 2007
3040 Cornwallis Road
Phone 919-541-6000
■
P.O. Box 12194
■
Research Triangle Park, NC 27709
e-mail [email protected]
RTI International is a trade name of
Research Triangle Institute
Acknowledgements
2

The data used in this presentation were produced for
the U.S. Department of Education, National Center for
Education Statistics (NCES), under Project no.
0207818

The views expressed in this presentation do not
necessarily reflect the official policies of NCES or RTI
International; nor does mention of trade names,
commercial practices, or organizations imply
endorsement by the U.S. Government
Introduction – Background and Purpose


3
Choices with replication
●
One or two sampling stages
●
Some or all weight adjustments
●
Overall or replicate-level control totals
●
Finite population correction (fpc)
Replication examples in NCES studies
●
National Postsecondary Student Aid Study (NPSAS)
●
School and Staffing Survey (SASS)
●
Education Longitudinal Study of 2002 (ELS:2002)
Introduction – Variance Estimation Methods
4

Taylor series linearization

Replication
●
Jackknife
●
Bootstrap
●
BRR
Introduction – Overview of ELS:2002

Sample design
●
Base-year
●
First follow-up
●
Transcript
●
Second follow-up
■ Weighting
5
●
Nonresponse adjustment
●
Poststratification/calibration
Replication of School Sampling Stage
6

Formed strata and PSUs for all sample schools

Collapsed strata

200 replicates

FPC not necessary
Replication of Student Sampling Stage
7

Same strata and PSUs as for schools

Used school BRR weight to help compute initial
student BRR weight

Used prior round BRR weight as starting point for
current round BRR weight
Replication of Nonresponse Adjustment
8

1 adjustment for the school weight

2 adjustments for each student weight

Deleted variables from the model, where necessary,
to achieve convergence
Replication of Poststratification/Calibration

Base year schools poststratified to population totals

Base year students not poststratified

Students in follow-up rounds calibrated to previous
round weight sums

Replicate-level control totals
- Computed weight sums for each replicate

9
Deleted variables from the model, where necessary,
to achieve convergence
Comparison of Variance Estimates

10
Variance estimates influenced by:
●
Unequal weighting
●
Stratification
●
Clustering
●
Nonresponse adjustment
●
Poststratification
Comparison of Variance Estimates (cont.)
11

Poststratification to “known” population totals causes
the sampling variance for estimates of the totals to go
to zero

Repeating the poststratification step on each half
sample replicate ensures that the variance estimates
for the control total estimates are zero

Calibration to previous round half sample data causes
the variance estimates for the control total estimates
to not be zero
Comparison of Variance Estimates (cont.)
12

Compared standard errors computed using both the
Taylor series and BRR variance estimation methods

BRR standard errors more conservative

BRR and Taylor series standard errors larger than
simple random sample standard errors
Base Year School Standard Errors
Estimates compared
Estimates with BRR standard error less than
Taylor series standard error
Estimates with simple random sample standard
error less than Taylor series and BRR standard
errors
13
School weight
44
14 (31.8%)
33 (75.0%)
Base Year School Design Effects
25th Percentile
Minimum
Mean
50th Percentile
Maximum
6.0
Design Effect
5.0
4.0
3.0
2.0
1.0
0.0
BRR
Taylor Series
School Weight
14
75th Percentile
Base Year Student Standard Errors
Estimates compared
Estimates with BRR standard error less than
Taylor series standard error
Estimates with simple random sample standard
error less than Taylor series and BRR standard
errors
15
Student weight
204
40 (19.6%)
196 (96.1%)
Base Year Student Design Effects
25th Percentile
Minimum
Mean
50th Percentile
Maximum
9.0
8.0
Design Effect
7.0
6.0
5.0
4.0
3.0
2.0
1.0
0.0
BRR
Taylor Series
Student Weight
16
75th Percentile
First Follow-Up Standard Errors
Estimates compared
Estimates with BRR standard error less than
Taylor series standard error
Estimates with simple random sample standard
error less than Taylor series and BRR standard
errors
17
Cross-sectional
student weight
86
22 (25.6%)
82 (95.3%)
First Follow-Up Design Effects
25th Percentile
Minimum
Mean
50th Percentile
Maximum
3.0
Design Effect
2.5
2.0
1.5
1.0
0.5
0.0
BRR
Taylor Series
Cross-sectional Student Weight
18
75th Percentile
Second Follow-Up Standard Errors
Estimates compared
Estimates with BRR standard error less
than Taylor series standard error
Estimates with simple random sample
standard error less than Taylor series
and BRR standard errors
19
Cross-sectional
student weight
58
14 (24.1%)
57 (98.3%)
Second Follow-Up Design Effects
25th Percentile
Minimum
Mean
50th Percentile
Maximum
3.5
3.0
Design Effect
2.5
2.0
1.5
1.0
0.5
0.0
BRR
Taylor Series
Cross-sectional Student Weight
20
75th Percentile
Conclusions
21

BRR takes into account the variance due to weight
adjustments, so these results are expected

Controlling to replicate-level totals recognizes
variance in base year totals due to sampling
variability, so the results are more conservative

Worthwhile to replicate all stages and all adjustments
if time permits
Questions?
[email protected]
22