NOT LEAVING IT TO CHANCE:
LESSONS FROM THE US CENSUS BUREAU
LISA NEIDERT
UNIVERSITY OF MICHIGAN
IASSIST
VANCOUVER, B.C.
JUNE 2011
Political Interference: US Example (1)
Political Interference: US Example (2)
Political Interference: US Example (3)
US Census Bureau: a tradition of methodological research
United States: Voluntary Option
Mode
Mail
Stratum
Mandatory
Voluntary
Difference
59.5
38.8
-20.7
High Response
64.6
42.4
-22.1
Low Response
43.6
27.7
-16.0
Overall
U.S. Census Bureau: ACS 2003: Voluntary vs Mandatory Test [Report 3]
United States: Voluntary Option
Mode
Mail
Stratum
Mandatory
Voluntary
Difference
Overall
59.5
38.8
-20.7
White
63.3
43.4
-19.9
Black
35.2
22.2
-13.1
All other races
41.7
30.4
-11.3
Hispanic
32.2
20.1
-12.2
U.S. Census Bureau: ACS 2003: Voluntary vs Mandatory Test [Report 3]
Summary Results
Mode
Mandator
y
Voluntary
Difference
Overall
71.8
60.2
-11.6
Mail
59.5
38.8
-20.7
Phone
80.7
66.5
-14.2
In Person
95.6
89.0
-6.7
U.S. Census Bureau: ACS 2003: Voluntary vs Mandatory Test [Report 3]
Cost
Initial
Sample
I Rate
Completed
Cost
VOLUNTARY
3.7 million
60
2.2 m
$214m
MANDATORY
3.0 million
72
2.2 m
$155m
DIFFERENCE
+0.7 million
-12
-
+ $59m
U.S. Census Bureau: ACS 2003: Voluntary vs Mandatory Test [Report 3]
Synopsis of Census Bureau Research
 Dramatic decrease in mail response
 Mail cooperation drops over 20 percentage points
 Reliability of estimates declines
 Costs increase (+$59.2 million)
 Mail/telephone cheaper; in-person expensive
 Need larger sample
 Item non-response similar across both
Continued
 Voluntary methods had a larger impact in high
response areas


50% increase in personal visits for White, Non-Hispanic
<20% increase in face-to-face for Blacks, Hispanics
 Low response areas less impacted, but due to lower
base-line response rates, would not be able to
produce reliable data for those areas
Under Age 30 Population
Characteristic
Landline
Cell
18 to 24
55
70
25 to 29
45
30
Male
48
62
Never Married
64
82
Unaffiliated
26
36
Church Weekly
36
21
http://people-press.org/2008/01/31/the-impact-of-cell-onlys-on-public-opinion-polling/
Literary Digest poll: Landon in a landslide
Praise for the Literary Digest Straw Poll
Any sane person cannot escape the implication of
such a gigantic sampling of popular opinion as is
embraced in The Literary Digest straw vote. I
consider this conclusive evidence as to the desire of
the people of this country for a change in the
National Government. The Literary Digest poll is an
achievement of no little magnitude. It is a Poll fairly
and correctly conducted.
- James A. Farley, Chair DNC
Literary Digest: Sampling Frame
Car & Phone
Car Only
Phone Only
Neither
Roosevelt
55
68
69
79
Landon
45
30
30
19
1
2
0
2
Other
Public Opinion Quarterly, Vol 52: 125-133 (1988)
Literary Digest: Sample
Candidate
Received Poll
Did not receive
Don’t know
Roosevelt
55
71
73
Landon
44
27
25
1
1
3
Other
Public Opinion Quarterly, Vol 52: 125-133 (1988)
Literary Digest: Response bias
Returned Straw
Vote in Election
Do not know
Roosevelt
48
69
56
Landon
51
30
40
1
1
4
Other
Public Opinion Quarterly, Vol 52: 125-133 (1988)
Modern Era: Nonresponse bias
 Low response surveys do not always have nonresponse bias
 Some ways of decreasing nonresponse may lead to higher
nonresponse bias
 Noncontact propensities are likely to be different than refusal
propensities or noninterview propensities due to health,
language, etc.
 Must understand how the influences for/against participation
are related to survey measures
Robert M. Groves. 2006.“Nonresponse rates and Nonresponse Bias in Household Surveys” Public Opinion
Quarterly 70(5): 646-675
Survey Methodology Pop Quiz
July 21, 2010
I want to take this opportunity to comment on a technical
statistical issue which has become the subject of media
discussion. This relates to the question of whether a
voluntary survey can become a substitute for a
mandatory census.
It cannot.
-Munir A. Sheikh
Survey Methodology Pop Quiz
Sending it out to more people doesn’t solve the
problem. The problem is that on a voluntary survey,
people respond who feel like responding.
-Ivan Felligi
Survey Methodology Pop Quiz
We can compensate for the fact that certain
demographic groups are likely to forego completing
the long-form census because statisticians can
ensure validity with a larger sample size.
-Tony Clement
Statistical Summary
 NHS will be an expensive step backward
 NHS sampling frame is adequate
 Cannot compensate for non-response bias with larger sample
 Larger samples improve margin of error, but in this case will
not be able to mitigate non-response error
 Demographic analysis will be impossible, except for
short-form characteristics


What does Canada look like?
How has it changed?
Future??
 Shift towards Privacy & Small government
advocates?
 Will Canada reverse course or is the summer of 2010
the future of Canada?
 Future of federal statistical system
 Canada comes to its senses