Polling
Readings:
Herbert Asher, Polling and the Public
Polling
“Polling”
Polling” and “Surveys” are synonymous.
George Gallup began using the term
“poll”
poll” in his earliest surveys to connect
his questions to the election and add a
degree of importance.
History of Polling
The Harrisburg Pennsylvanian reported a
straw poll in 1824.
In 1904 the New York Herald conducted
street-corner polls.
The Literary Digest began their polling in
1916.
1
Who uses polling?
Campaigns
Media
Government
Marketing
Courts
Academics
Understanding Polls
To evaluate public opinion polls we have to
ask:
Who is questioned?
How are they asked?
What questions are they asked?
Who is
asked?
Drawing a sample
2
Who is questioned?
A “Sample”
Sample”
A subset of the population we use to
make inferences about the population
in general
Why do we use samples?
Types of Samples
 Nonprobability
Samples/Straw Polls
 Probability or Random Samples
Nonprobability Samples
“Straw Polls”
Polls”
People may self-select or be chosen by
some non-rigorous method.
You can not estimate the likelihood
someone will be included
3
Nonprobability polls
“Typical”
Typical” people
Purposive
Volunteer subjects/self selection
Haphazard sampling
Quota
Snowball sample
The Perils of Straw Polls
Shere Hite vs.
vs. Dear Abbey
Hite distributed 100,000 survey and
found that 70% of women married 5 or
more years had affairs.
Abby received 200,000 responses to a
column and found 15% of wives were
unfaithful
 The Literary Digest Poll
The Literary Digest Poll
In 1936 Literary Digest sent
out 10,000,000 ballots
and got back 2,376,523 and
predicted a 57% to 43%
victory for Alf Landon
FDR won with 62.5% of the
vote
4
George Gallup
Gallup predicted more
accurately that FDR
would get 56% of the
public.
Probability Samples
Random samples
Lets us judge how likely what we observe
is not the product of chance.
Judging a fair coin
Let’
Let’s say that you suspect a coin is
“loaded”
loaded” or biased.
How would you know when to be
suspicious?
If you flipped a coin 100 times and it came
up heads 51 times would you be
suspicious?
What if it came up head 95 times?
5
Understanding Samples
What happens if you flip a coin 100 times?
Sampling Error
Probabilities and Standard
Deviations
(c) Probabilities and numbers of standard deviations
Shaded area = 0.683
!
+
68% chance of falling
+
between ! and
Shaded area = 0.954
!2
+2
95% chance of falling
+2
between ! 2 and
Shaded area = 0.997
!3
+3
99.7% chance of falling
+3
between ! 3 and
6
Sampling Error
Arises from representing a population with a
random sample.
Not a mistake on the part of the pollster, it is
caused by the nature of random samples.
Can not be calculated for a non-probability
sample.
Remember:
Remember: A random sample is a calculated
risk.
The Advantage of Random
Samples
Randomness insures that the only source of
error/bias in the sample is sampling error -which we can calculate
Margin of Error or
Confidence Interval
Every poll has a confidence interval.
Accurately reported it will say something like “±
3% with a 95% confidence level.”
level.”
7
Confidence Interval
A poll shows a candidate having the
support of 47% of respondents with ± 3%
This means the candidate’
candidate’ s support is--most
is--most
likely--between
likely--between 50% and 44%.
Confidence Interval
Most surveys use a 95% confidence interval or a
5% alpha level.
This tells us that in 95% of the polls the
percentage found in the sample will fall within
the confidence interval around the actual
percentage that exists in the population
Tradeoffs in the confidence
interval
Level of certainty
Do you want to be 90% or 95% certain?
Precision of results
Is ±3% accurate enough or more precise
than you need
Sample size
The bigger the random sample, the better
8
Sample Size and
Confidence Interval
Gallup Tracking Poll of Likely
Voters
Gallup Tracking Poll of Likely
Voters
9
Simple Random Sample
Respondents are drawn at random from the
individuals in a population
Sub-samples
The margin of error for the sample only
applies to the sample overall.
Stratified Samples
What if you want to know something about
the characteristics of sub-units?
For example, what if you want to compare
GW Bush’
Bush’s rating in Texas to the nation?
10
Stratified Samples
Divides the population into smaller
geographic chunks and randomly surveys
within each chunk.
Cluster or Multistage Sampling
Wrong population sampled
Targeting the wrong people
How do you predict voting?
11
Random samples of what?
Sampling Frame
A listing of the target population from
which a sample is drawn
Usually not possible to assemble
Sampling Lingo
Population (Universe)
What is the difference
between a sample and
All items of interest
a population?
Sample
Portion chosen from population
Parameter
Characteristic of a population
Statistic (Descriptive)
Characteristic of a sample
Parameter and Statistic
An Example
“The mean height of all 10th
graders is 60 inches.
inches.”
Parameter
Population
“The mean height of Mrs. Jones’
Jones’
tenth graders is 60 inches.”
inches.”
Statistic
Sample
12
Declining Response Rates
Polling’
Polling’s “Dirty Little Secret”
Secret”
What happens when people don’
don’t want to
answer pollster’
’
s
questions?
pollster
Are people who do not respond different
from those who do?
Declining Response Rates
Today most polling firms find that they can
not reach about 20% of their sample-even after making 6-7 attempts to reach
them.
Some firms use specially trained
interviewers to contact reluctant
respondents
“Non-coverage”
Non-coverage” errors
Phone surveys
About 30% of phone numbers are unlisted
7% of voters in 2004 were cell phone only users
About 5.5% of homes do not have a phone (25%
of households with income under $15,000 a
year)
Answering machines and caller id are used to
screen calls
5% of phone interviews are not finished because
the respondent hangs up
13
Bias in non-responses
In one experiment, after 1 attempt to reach
respondents, 38% of the sample was working
outside the home
After 4 tries the percentage had risen to 58%
After 7 tries the percentage had risen to 63.5%
Phone-type Bias
CPOs favored Kerry over Bush 54% to 45%
“Weighted”
Weighted” Results
Sometimes in order to correct for
nonresponse errors that lead some groups
to be under-represented in a poll
14
How are they
asked?
By mail, in person or by phone?
Reaching Respondents
By mail or e-mail
May be slow since respondents may respond
at their leisure
Survey is self administered and may not be
filled out by correct person
In person
Presents risks and interviewer bias
Travel time is extensive
Phone Surveys
Most common, but hard to
reach some people and
interviews must be short
15
RDD
Random Digit Dialer
Used to pick numbers/respondents
Proactive dialing has the PC (or central
equipment) dial the number, but an
interviewer is waiting on the line.
Predictive has central dialing equipment dial
and listens. Then interviewers are switched
on the line after someone answers
CATI
Computer Assisted
Telephone
Interviewing helps
speed administering the
survey and analyzing
results.
Caller-id
The use of caller-id (now in about 25% of
homes) allows some to dodge calls from
a “unavailable”
unavailable” number.
Changing to a university’
university’s name from
“unavailable”
unavailable” can raise response rates by
more than 3%.
16
Cell Phones
The Federal Telephone Consumer Protection Act (TCPA)
probits:
probits:
ALL calls made to a cellular phone, without the prior consent of
the person called IF the call is made using an automatic
telephone dialing system (defined as equipment which has the
capacity to store or produce telephone numbers to be called
using a random or sequential number generator and to dial
such numbers) or an artificial or prerecorded voice AND IF
the party is charged for the call
Interviewer Bias
When the response is
altered as a reaction to
the individual asking
the question.
Interviewer Bias- Gender
“The decision to have an abortion is a private matter that
should be left to the woman to decide without
government question.”
question.”
Interviewer
Female
Male
Respondent
Female
Male
84%
77%
64%
70%
17
Interviewer Bias- Race
Respondents will shape their responses
based on the race (or perceived race) of
the person asking the questions.
Perceptions of race
As part of the National Black Politics survey of 1,204 black
respondents researchers asked respondent believed the race of
the interviewer was. Seventy six of the respondents correctly
identified their interviewer as black (only African American
interviewers were used) while 14% thought the interviewer
was white and 10% guessed another race. While 3 of the 25
interviews were guessed to be white or some other race by a
majority of the respondents, none of the interviewer sounded
black to all the respondents.
Perceptions of race
Perceived race of interviewer
“Black
Black””
“White
White””
Black elected officials can best represent
49%
the interests of the black community
64%
American society is fair to everyone
31%
14%
The American legal system is
72%
unfair to blacks
84%
The police are too much like just
42%
another gang to stop gang violence.
52%
18
Perceived race
of interviewers
Black
White
the interests of the black community
64%
49%
American society is fair to everyone
14%
31%
84%
72%
52%
42%
Black elected officials can best represent
The American legal system is
unfair to blacks
The police are too much like just
another gang to stop gang violence.
What is asked?
Questionnaire Construction
Types of Questions
Open-ended
Closed-ended
19
Open Ended Questions
Lets respondents use their own words to
answer questions
“What do you like/dislike about the
Republican Party?”
Party?”
“What do you do you consider the most
important problem facing the nation
today?”
today?”
But, hard to summarize or “code”
code”
Closed ended Questions
Respondents choose from a list of answers
selected by the survey researcher
“Do you approve or disapprove of the
way Bill Clinton is handling his job as
president?
“Respondents are like
cats, if you give them a
box, they will go in it.”
7-point scales
The seven point scale comes from the
psychological conclusion that individuals have
difficulty reliably making distinctions between
more than seven options.
In phone surveys it is generally regarded as
having no more than five response categories.
20
7-point scales
“Recently there has been a lot of talk about women’
women’s rights.
Some people feel that women should have an equal role with
men in running business, industry, and government. Others
feel that women’
women’ s places is in the home”
home”
A woman’s
place is in
the home
Women and men
should have an
equal role
1
2
3
4
5
6
7
Agree Disagree
Respondents are asked to agree or disagree
with a series of statements
Agree Disagree
“Should marijuana be legalized?”
legalized?”
Strongly
disagree
Strongly
agree
+++
++
+
•
-
--
---
21
Feeling thermometer
100° Very warm and very favorable feeling
85° Quite warm or quire favorable feeling.
70° Fairly warm or …
50° No Feeling at all
40° Slightly cold or slightly unfavorable
30° Fairly cold or fairly unfavorable feeling
15° Quite cold or …
0°Very cold or very unfavorable.
feeling
Branching
Rather than having a long question can have a series of
short questions
Branching
One Question
Do you consider yourself a strong
Democrat, a weak Democrat, an
independent leaning Democratic, an
independent, an independent leaning
Republican, a weak Republican or a
strong Republican?
Two Branched Questions
Do you consider yourself a Democrat,
Republican, and Independent or what?
Do you consider yourself a strong or
weak _____?
22
Branching
Do you think the U.S. should or should not make it easier for illegal immigrants to become
citizens of the United States? (Gallup Aug 24-26, 2001)
Should
Should not
No opinion
28%
67%
5%
Which of the following proposals would you prefer for illegal immigrants currently living in the
United States: they should be granted general amnesty, (or) they should be allowed to stay in
the U.S. only if they have worked and paid taxes for a certain length of time]? [Based on
those who say it should be easier for immigrants to become citizens; ±6 PCT. PTS.]
Granted general amnesty
20%
Should stay if they paid taxes 73%
Combining branched questions
Easier, prefer general amnesty
6%
Easier, prefer citizenship for those
who have worked/paid taxes
20%
Easier, no preference
2%
Not easier
67%
No opinion
5%
Batteries of Questions
Sometimes a series of questions can be
used to get at broad issues.
 “Worker satisfaction”
satisfaction”
23
Worker Satisfaction
For each, please tell me how satisfied or dissatisfied you are with your current job in this regard. First, are you
completely satisfied, somewhat satisfied, somewhat dissatisfied, or completely
dissatisfied with…
with…? [Gallup Organization, August 16-19, 2001]
What messes up question?
Loaded Questions
“Push”
Push” Polls
Uncertainty and Ambiguity
Loaded Questions
Questions designed to return a particular
response.
24
Loaded Questions
“A recent study by a psychology professor at a
leading university concluded that the amount
of violence children see on television has an
effect on their likelihood of being aggressive
and committing crimes. From what you have
seen or heard about this subject, do you
agree…”
agree…”
Loaded Questions
“Which segment of the television viewing
audience do you think will suffer more if
federal funding for PBS was cut back
sharply or eliminated--children or
adults?”
adults?”
Loaded Questions
Do you agree or disagree with the statement
“When (Bernhard) Goetz said in his confession that he used dumdum bullets, that he was sorry he didn’
didn’t gouge out the eyes of
the four he shot, and that if he could have reloaded his gun fast
enough, he would have taken out after them, he looked more
like a ‘Death Wish”
Wish” gunman out stalking to kill criminals, not
an innocent victim trying to defend himself.”
himself.”
1985 Harris poll
25
Push Polls
Polls that provide respondents with bits of
information to see what impact they have
on citizens’
citizens’ choices
Abuse of Push Polls
When information is provided that is
intended to influence-rather than
measure-respondents’
measure-respondents’ opinions
Thousands of homes are called.
Push Polls
“As you may know, in 1974, Jerry Springer, who had gotten
married 6 months earlier, was arrested on a morals charge
with 3 women in a hotel room. He also used a bad check to
pay for the women’
women’s services, and subsequently resigned as
mayor of his city. Does this make you much more likely,
somewhat more likely, somewhat less likely, or much less
likely to support Jerry Springer for governor this year?”
year?”
26
Push Polls
“ If you knew that Pat Buchanan had said
women were less equipped psychologically for
the working world,’
world,’ would you be more likely
to vote for him, less likely, or does it make no
difference in your vote?”
vote?”
“Nonattitudes”
Nonattitudes
Sometimes called “Doorstep responses ”
When respondents are unsure about the
facts or the question being asked they
may offer up answers that are not
reliable.
27
Uncertainty
When you ask citizens about something
they know little or nothing about what
does their response mean?
“If you do a 20-minute poll on environmental
issues, you’
you’ve engaged most people in a
longer conversation than they’
they’ve ever had
about the environment in their lives.”
lives.”
Uncertainty
“Do you favor or oppose the Public Affairs Act
of 1975?”
1975?”
33% of respondents expressed a preference
“Do you favor or oppose the Public Affairs Act
of 1975 or don’
don’t you have an opinion?”
opinion?”
Percentage expressing a preference dropped to
2%.
Ambiguity
When the question creates uncertainty
“Does it seem possible or does it seem
impossible to you that the Nazi extermination
of Jews never happened?”
happened?” [1992 Roper Poll]
22% of respondents said that it seemed
possible that the Holocaust never occurred
Another 12% answered “don’
don’t know.”
know.”
28
“Does it seem possible to you that the Nazi
extermination of the Jews never
happened, or do you feel certain that it
happened?”
happened?”
Only 1% expressed doubt
Ambiguity
“How many public officials do you think
are a little bit dishonest-most, some, a
few, or none?”
none?”
Types of Question
Problems
29
Context Bias
Or “Multiple Stimuli”
Stimuli”
Some questions have more than one
question or “stimulus”
stimulus” to which
respondents may react.
“Women in Combat”
Combat”
Context Bias
“President Clinton said that the 1975 Public Affairs
Act should be repealed. Do you agree or disagree?”
disagree?”
36% of Democrats and 16% of Republicans agreed
“The Republican Congress said that the 1975 Public
Affairs Act should be repealed. Do you agree or
disagree?”
disagree?”
36% of Republicans and 19% of Democrats agreed
Context Bias
“The U.S. has 36,000 troops and airmen in South Korea. If North
Korea invaded South Korea, we have a firm commitment to defend
South Korea with our own military forces. If South Korea were
invaded by Communist North Korea, would you favor or oppose
the U.S. using troops, air power and naval power to defend South
Korea?”
Korea?” [43%]
“As things stand now, do you feel the war in Korea has been worth
fighting, or not?”
not?” [27%]
30
Context Bias
Sometimes you hear it said that
there should be a law to ensure
that a certain number of
federal contracts go to
minority contractors. Do you
favor or oppose such a law?
Favor
Oppose
43%
57%
The Congress of the United States-both the House of Representatives
and the Senate--have passed a law to
ensure that a number of federal
contracts go to minority contractors.
Do favor or oppose such a law?
Favor
Oppose
57%
43%
Social Desirability Bias
People don’
don’t want to
admit to unpopular
beliefs or actions.
Just ask them about
Racial attitudes
Sexual behavior
Shere Hite
Consistency Bias
People want to appear consistent
Moderation Bias
n
People want to avoid appearing extreme
31
Agree-ability bias
“Response acquiescence”
acquiescence”
People prefer to agree rather than disagree.
Respondents to consistency agree on
agree/disagree questions while not
seriously reading the questions.
No Middle Point
If people who feel neutral forcing them in
one direction or another draws out
something random or unrelated to the
question
Unbalanced Questions
Having more arguments or responses on
one side than another.
32
Unbalanced Questions
Do you agree or disagree with this statement:
Any able-bodied person can find a job and
make ends meet?
Agree
No opinion
Disagree
65%
10%
25%
Some people feel that any able-bodied
person can find a job and make ends
meet. Others feel that there are times
when it is hard to get along and some
able-bodied people may not be able to
find work. Who do you agree with most?
Can make ends meet
43%
No opinion
18%
Sometimes hard to
get along
39%
Framing Effects
The effect of previous content of interview
may have on specific questions.
Questions about personal finances may
encourage pocketbook voting
Response Choice Order
When questions are read aloud to
respondents they tend to choose among
the last few options
When questions are read by the respondent
they tend to choose among the early
options
33
Other Types of
Surveys
Panel Studies
Polls in which the same voters are
questioned repeatedly over time.
Designed to study change in opinions
Problems with Panel
Studies
People become sensitized by questions and
may end up behaving differently than
other citizens.
34
Exit Polls
Surveys taken as voters leave the polling
place
In recent years conducted by Voter News
Service (VNS) in a cooperative effort
with numerous news outlets
Exit Polls
Short interviews conducted with people as
they leave polling places
Usually large samples
In 2004 150,000 voters from 1,500
precincts were interviewed to insure
samples of most states
Focus Groups
Most often 7 to 10 people (even though
they may range from 4 to 12) who do not
know each other.
Brought together to generate an informal
discussion that will reveal their
perceptions in a friendly, casual
atmosphere
35
Advantages of Focus
Groups
Avoids the “Passive Respondent”
Respondent”
Ability to explore the reasons behind
opinions
Simulates normal interaction
Cost
Focus Groups in Politics
Shape surveys
Probe ideas
Test advertising/themes
Disadvantage of Focus
Groups
Not a representative sample
36
On-line polling
Problems with e-mail polls
Voluntary
No population list
Not anonymous
Computer users younger, whiter, more
male, better educated and wealthier than
population
Web Polls/Panels
Some researchers are using on-line surveys
that contact the same respondents on a
repeated basis to create electronic panels.
37
Internet Polls
People magazine’
magazine’s website poll that named
Howard Stern's sidekick "Hank the
Angry, Drunken Dwarf" as one of the 50
most beautiful people in the world
Electronic ballot stuffing
38