Presidential Approval Ratings
in Perspective
Larry Hugick, Jonathan Best, and Stacy DiAngelo
Princeton Survey Research Associates International
Princeton Survey Research Associates International
911 Commons Way
Princeton, New Jersey 08540
(609) 924-9204
1211 Connecticut Avenue NW
Washington, D.C. 20036
(202) 293-4710
May 2004
Presidential Approval 2
PRESIDENTIAL APPROVAL RATINGS IN PERSPECTIVE.............................................................. 1
ABSTRACT .................................................................................................................................................. 3
INTRODUCTION ........................................................................................................................................ 4
METHODOLOGY ....................................................................................................................................... 8
DATA .......................................................................................................................................................... 8
COMPUTING QUARTERLY AND YEARLY AVERAGES ................................................................................... 9
STATISTICAL ANALYSES............................................................................................................................. 9
ANALYSIS AND RESULTS ..................................................................................................................... 10
DIFFERENCES IN POLLING METHODOLOGY ................................................................................ 10
Interviewing Methods and Practices................................................................................................... 10
Sampling and Weighting ..................................................................................................................... 10
Respondent Selection .......................................................................................................................... 11
Questionnaire Issues........................................................................................................................... 12
Changes in Methodology during the Study Period ............................................................................. 12
TREND LINE FOR ALL POLLS ............................................................................................................ 13
PATTERNS OBSERVED IN YEARLY AVERAGES ........................................................................... 16
STATISTICAL ANALYSIS OF QUARTERLY AVERAGES............................................................... 19
ANOVA Procedures and Findings ...................................................................................................... 19
VISUAL REPRESENTATION OF POLL DIFFERENCES ................................................................... 21
INVESTIGATING THE “DON’T KNOW” FACTOR.......................................................................... 24\\
COMPARISONS WITH GALLUP IN-PERSON APPROVAL RATINGS ............................................ 30
CONCLUSIONS......................................................................................................................................... 34
APPENDIX ................................................................................................................................................. 37
POLLING METHODS QUESTIONNAIRE ........................................................................................... 37
PRESIDENTIAL APPROVAL BY ORGANIZATION: LINE CHARTS............................................... 40
QUARTERLY AVERAGES BY ORGANIZATION.............................................................................. 43
ABC News/Washington Post ............................................................................................................... 43
CBS News............................................................................................................................................ 44
CNN/USA Today/Gallup..................................................................................................................... 45
FOX News/Opinion Dynamics ............................................................................................................ 46
Newsweek............................................................................................................................................ 47
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 3
Abstract
This paper analyzes individual poll results, as well as quarterly and yearly averages for
five major media polls – the ABC News/Washington Post poll, the CBS News poll, the
CNN/USA Today/Gallup poll, the FOX News/Opinion Dynamics poll, and the
Newsweek poll – in order to identify and explain differences in their presidential
approval ratings during the period 1997-2003. Differences in approval are relatively
small. In a given year, the typical difference in percent approve across all polls is only
four points. The level of “don’t know” (DK), however, differs significantly by
organization and analysis shows that reducing the DKs tends to increase approval without
much affecting disapproval. The polls that use a conventional telephone survey
methodology can be divided into two high DK polls (ABC and CNN) with a tendency
toward higher approval scores and two low DK polls with a tendency toward lower
scores (CBS and Newsweek). The FOX poll, which uses a somewhat different
methodology, looks very close to the conventional high DK polls when its approval
ratings are averaged over an entire year, but tends to register lower disapproval ratings,
and its movement from quarter to quarter and poll to poll is less predictable than the
others. Contemporary presidential job performance ratings from telephone polls seem to
be reasonably close to those for past presidents taken from The Gallup Poll’s historic inperson surveys. In the percent approve, however, today’s high DK polls are slightly
closer to the Gallup in-person results than today’s low DK polls.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 4
Introduction
National polls have been asking the public to rate the president’s job performance
since 1935, during Franklin D. Roosevelt’s first term in office. The most widely used
version of this question – “Do you approve or disapprove of the way [president’s name]
is handling his job as president?” – was adopted by The Gallup Poll in 1945.1 While
other question wordings are still employed, such as the one using a four-point scale
(excellent, good, fair, or poor), the standard Gallup wording is now used in most, if not
all, telephone polls sponsored by major national media organizations – network
broadcast and network TV news, as well as the largest national newspapers and news
magazines. While 30 years ago political junkies hungry for the latest results might have
to wait a week or two for a new round of presidential approval data, today hardly a day
goes by without a new set of approval figures being served up by one organization or
another. And that’s not counting the steady stream of poll data provided by organizations
with alternative lower-costs methodologies, like Rasmussen Reports and its automated
polling that generates daily approval numbers. The availability of so many different
measures of the public’s evaluations of the president’s job performance means that one
poll with results that seem aberrant can be checked against others in a timely fashion.
While more data is generally better than less data, it is not without its pitfalls.
Most seasoned poll watchers know that the best practice is to limit trend analysis to polls
of one particular organization – rather than mix and match – but there is a strong
temptation to make cross-comparisons. Even if organizations use the same basic
methodology and question wording, “house effects” caused by factors that are less
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 5
obvious – such as the way a CATI system works and interviewers are instructed –
potentially introduce differences to poll results. Many political reporters, columnists and
other non-survey research specialists who frequently use approval data know less than
they should about differences in methodology between the major national media polls
they monitor, and are misinformed about which differences are likely to have a
significant impact on results and which are not. Variation in party ID distributions from
poll to poll, the presumed political slant of the media organization sponsoring the poll,
and purported day-of-the-week interviewing effects are examples of issues that are
brought up all too frequently by those who know less than they think about poll
methodology. Those of us who conduct these polls for the media often don’t spend
enough time keeping track of what our competitors are doing and our impressions of
other pollsters’ methodology may not always reflect current practices.
Through a careful study of recent presidential approval ratings by five of the most
prominent national media polls, this paper seeks to provide information that will allow
poll watchers to better interpret their results and put them in perspective. It attempts to
answer the following questions: Are the approval ratings of these major media polls
basically interchangeable, or do they differ in ways that can be quantified? If there are
differences by organization, what factors are responsible? Is it reasonable to make a
direct comparison between today’s approval ratings based on telephone surveys and those
for past presidents from Gallup’s in-person surveys? More generally, the paper describes
the current state of methodology of five major media polls and how closely their results
correlate.
1
Edwards III, George C., with Alec M. Gallup. Presidential Approval: A Sourcebook (The Johns Hopkins
University Press: 1990), page 3.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 6
We used two criteria for determining which major media polls to select for this
study: 1) the question posed had to make use of the standard Gallup approval question
wording; and 2) approval figures had to be available from more than 12 individual polls
sponsored by the media organization during the calendar year, excluding one-night polls
– i.e., with a frequency of more than once a month. The five polls meeting these two
criteria are the ABC News/Washington Post poll; the CBS News poll; the CNN/USA
Today/Gallup poll; the FOX News/Opinion Dynamics poll; and the Newsweek poll,
conducted by Princeton Survey Research Associates International.2 Of these five media
polls, the CNN/USA Today/Gallup poll was by far the most active, releasing 41 separate
sets of results over the 12-month period, while the ABC News/Washington Post poll was
least active, with 16 approval measures in 2003. Two other prominent media polls that
regularly ask the standard Gallup approval question – the CNN/Time poll and NBC/Wall
Street Journal poll – fell below the activity frequency threshold. We were aware that the
FOX News/Opinion Dynamics poll differed from the other four polls selected in their
reporting base – registered voters rather than the entire voting-age population. However,
our own recent polling suggested that the registered voter base – often representing over
80% of a general public adult sample – introduces only small differences to poll measures
of political attitudes. What’s more, we believed the study would be enhanced by
including a poll sponsored by top-rated FOX News cable network that has positioned
itself as an alternative to the mainstream media, to see if their poll diverges in any
specific way from the others.
2
In this report organization names are abbreviated as follows: ABC News/Washington Post – ABC; CBS
News – CBS; CNN/USA Today/Gallup – CNN; and FOX News/Opinion Dynamics – FOX.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 7
In defining a time period for this study, we sought to be as contemporary as
possible to reflect current methods and the conditions that might affect our ability to
conduct surveys, but at the same time providing sufficient data to analyze. In part, our
decisions about which polls to study helped us make that decision, since the FOX
News/Opinion Dynamics poll was not launched until 1996. Limiting the study to 19972003 (the period encompassing Bill Clinton’s second term in office and the first three
years of George W. Bush’s presidency) is a reasonable compromise, being reasonably upto-date but still providing a sizable data base of approval ratings for the five
organizations. There is variety in that the two presidents being evaluated are from
different political parties, one during his second term in office and the other during his
first three years in office.
We owe a debt of gratitude to our colleagues at other organizations whose polls
were selected for inclusion in our study, whose cooperation in answering our questions
and disclosing the details about their methods and practices added immeasurably to this
research effort. I would like to thank Gary Langer of ABC News, Kathleen Francovic of
CBS News, David Moore and Jeff Jones of the Gallup Organization, Dana Blanton of
FOX News, and John Gorman of Opinion Dynamics Corporation for their help.
Larry Hugick
May 2004
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 8
Methodology
Data
This research study uses data from national polls conducted by ABC
News/Washington Post; CBS News; CNN/USA Today/Gallup; FOX News/Opinion
Dynamics; and Newsweek.3 Presidential approval ratings from these five organizations
were gathered for the period 1997 to 2003. Any data from polls conducted entirely in
one day were excluded. All polls were conducted by telephone among the 18 and older
general population except for FOX News polls which are limited to registered voters.
The principal sources of approval data for this study are the PollingReport.com web site
and the University of Connecticut’s Roper Center iPoll databank. Historical Gallup inperson approval ratings for 1953-1988 are taken from Presidential Approval: A
Sourcebook, by George C. Edwards III, with Alec M. Gallup.
Spreadsheets were created to analyze data by each organization and include the
following information:
•
•
•
•
•
Sample size and description
Dates of interviewing
Percent approve, percent disapprove, and percent no opinion (i.e. “don’t know” or
“refused” responses)
Quarterly averages calculated from individual polls within each quarter
Yearly averages calculated from quarterly averages within each year
PSRAI requested each of the organizations included in the study to provide some
details about polling methods, including question placement, weighting procedures,
respondent selection, handling of a “don’t know” response, callback procedures, and
3
Some polls included in this study as ABC News/Washington Post were done independently by either
organization and not together with the partner. Some polls included as CNN/USA Today/Gallup polls were
actually conducted independently by Gallup. Some polls identified as CBS News were done jointly with
The New York Times, while others were done independently.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 9
sample type. A detailed summary of responses can be found in the technical appendix
section of this report.
Computing Quarterly and Yearly Averages
Quarterly and yearly averages are used for most of the analysis in this report. Quarterly
averages were computed by averaging approval, disapproval, and no opinion ratings for
individual polls within each quarter.4 Yearly averages were then calculated using the
quarterly averages within each year. Tables in the Analysis and Results section of this
report summarize yearly averages for each president. Line charts using quarterly
averages and scatter plots using individual poll results were also created to help with
analysis. All tables and charts appear in the Analysis and Results section or technical
appendix at the end of this report.
Statistical Analyses
An analysis of variance with post-hoc Bonferroni comparisons was run on quarterly
averaged results in order to test for differences in approval, disapproval and DK ratings
among the five organizations. To test for associations between DK level and
approval/disapproval ratings, linear regressions were run based on individual poll data.
These analyses are discussed in more detail in the Analysis and Results section.
4
Detailed tables of quarterly averages for 1997-2003 can be found in the technical appendix.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 10
Analysis and Results
DIFFERENCES IN POLLING METHODOLOGY
The following is a summary of the differences in methodology of the five polls
studied, based on information provided directly from the researchers who conduct the
polls5 and information the various polling organizations make available through the
Internet and other sources. A review of the methods of five media polls shows FOX, the
new kid on the block, differing most in its procedures.
Interviewing Methods and Practices
FOX and Newsweek have the quickest turnaround time and shortest field periods
– two days. The other three polls tend to have longer field periods although CNN and
CBS, in particular, conduct a fair number of two-day polls. All five polls make callbacks
to non-contacts and ABC, CBS, and Newsweek also make refusal conversion attempts as
a general rule. As a general rule, CNN makes refusal conversions only for polls in the
field more than three days; FOX does not attempt refusal conversions. Only one poll –
CBS – is conducted in-house by the media organization sponsoring the poll, but the other
four polls rely on one field house, or limit their use of a different field house to rare
occasions.
Sampling and Weighting
All five polls use RDD samples for their telephone polls rather than listed sample.
FOX, which reports results based on registered voters instead of the total 18+ population,
also differs from the other four polls in the way it deals with demographic variation in its
sample. Opinion Dynamics weights the FOX poll sample only by race and ethnicity, and
reports using sample controls to achieve demographic balance by gender, age, and region.
5
The technical appendix of this paper includes the responses of those conducting the five media polls to a
short survey about their methods and practices.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 11
Apparently, aside from its race weighting, FOX pollsters take no specific steps to deal
with the tendency of unweighted samples to include too many better educated
respondents and too few respondents with low levels of formal education. The other four
polls take a more conventional approach to demographic weighting. They all weight
their obtained sample by gender, race/ethnicity, age, and education. CBS, CNN and
Newsweek also weight by geographic region. ABC does not specifically weight by
region, but achieves geographic balance through other means. CBS also weights by the
number of adults and the number of phone lines in the household. CNN also weights by
the number of phone lines, but not by the number of adults. Interestingly, none of the
five media polls is routinely weighted by party identification, a technique commonly used
in polls done for political candidates by partisan pollsters.
Typical sample sizes for the five media polls are quite similar. CNN and
Newsweek usually poll about 1,000 adults; although Newsweek’s standard sample size in
polls with approval ratings was 750 adults prior to 2001. ABC’s typical sample size is
1,200 adults, and FOX polls generally have a sample of about 900 registered voters.
CBS’s sample size varies from poll to poll, ranging from about 700 to 1,200 adults.
Respondent Selection
A variety of methods are used by the five media polls to select a respondent
within household. Newsweek asks to speak with the youngest male/oldest female who is
at home, a method CNN also used until late 2000. CNN now uses the “most recent
birthday” method, as does FOX. ABC uses a variation of the most recent birthday
method adjusted so interviewers ask to speak with a male 75% of the time and a female
25% of the time. CBS has the most rigorous approach, first determining the number of
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 12
adults in the household and selecting one at random, whether or not the person is at
home, and making an appointment, if necessary, to interview the selected respondent.
Questionnaire Issues
Four of the five media polls in our study report placing the approval question very
high in the questionnaire. CBS, CNN, and Newsweek report putting the approval ratings
question first, except in election years, when trial heats or other election-related questions
may take precedence. ABC reports asking the approval question first almost always.
FOX is the one poll that does not have a standard policy of placing the approval question
at the front of the interview, reporting that placement varies from poll to poll. Specific
interviewer instructions about whether to probe for a response among those who do not
say they approve or disapprove vary from poll to poll. CBS and CNN indicate that no
probing is done, and such responses are simply recorded as “don’t know.” Interviewers
working on the Newsweek poll are specifically instructed to limit probing to cases where
a respondent say “it depends.” ABC and FOX report using a full probe, re-asking the
approval question of all respondents who do not say they approve or disapprove.
Changes in Methodology during the Study Period
Representatives of the polling organizations were asked if any major
methodological changes were made to their polls during the period of this study, 1997 to
2003. ABC reports no major changes. The four other polls report some changes, none of
which seems truly “major.” CBS reports adding callbacks to “not in-service” numbers in
the past two years. As noted earlier, CNN changed to the “most recent birthday”
respondent selection method in late 2000. FOX reports experimenting with different
CATI systems that vary respondent selection or callback procedures. The Newsweek
poll increased its sample size from 750 to 1,000 in January 2001.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 13
TREND LINE FOR ALL POLLS
Now turning to the polling data collected over this the seven-year period of our
study, Chart 1 displays the key statistic (percent approve) for every individual poll
conducted by the five organizations.6 During Bill Clinton’s second term in office,
approval ratings generally moved within a 20-point range. For the first year of his second
term (1997), Clinton received approval ratings in the mid-50s to low 60s. In late January
1998, Clinton approval jumped to around 70% as the public rallied around him shortly
after the Monica Lewinsky scandal broke. Clinton approval then fell back, but rose again
to the high 60s/early 70s around the time the House voted for impeachment in late
December 1998 and his acquittal in February 1999. For the rest of Clinton’s presidency,
his ratings bounced around the 60% mark.
Approval ratings of George W. Bush during his first three years in office followed
a very different pattern. Prior to September 2001, Bush approval moved within the 50%
to 60% range. After the 9/11 terrorist attacks, Bush approval jumped to over 80% in all
polls, affected by the largest “rally-round-the-flag” effect on record.7 As the rally faded
gradually, Bush approval followed a steady downward path through early 2003, when it
jumped upward in another rally effect from the Iraq war. It then resumed its downward
trajectory until the end of 2003 when Saddam Hussein’s capture produced yet another
jump in Bush approval. With a few exceptions, Bush approval in polls taken from 2001
to 2003 ranged between 50% and 90%.
Chart 1 makes it clear that presidential approval ratings in recent years did not
follow a straight line for very long. Such movement makes it very difficult for us to
6
As noted previously, one-night polls by these organizations are excluded.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 14
compare and contrast the results of polling conducted by the five organizations based on
individual-level data. Since the five organizations tended to poll at different times and
approval ratings were rarely static, timing is such a powerful factor that it tends to
obscure any differences that might result from variation in research methods and
practices. Moreover, the sample size of a typical individual poll is no more than about
1,000; meaning that the difference between two approval ratings must be more than six
percentage points to meet the criteria for statistical significance. Nonetheless, an
examination of the individual poll data in Chart 1 suggests that, although the five polls
follow the same general trend line, there are certain patterns in the results of particular
polls that do not appear to be the result of random variation. In the last two years of
Clinton’s presidency, the FOX approval ratings appear to be consistently higher than
those of other organizations, and the Newsweek results appear to be consistently lower.
During the Bush years, ABC approval ratings seem to be consistently on the high side
and the Newsweek numbers consistently on the low side.
7
The 9/11 rally event and its impact on Bush approval is detailed in a paper by Larry Hugick and Margie
Engle of Princeton Survey Research entitled “Rally Events and Presidential Approval: An Update”
presented at the 2003 AAPOR conference in Nashville, Tennessee.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Chart 1: Presidential Approval Across Organizations
100
--------------------Clinton Years--------------------
--------------------Bush Years-------------------9/11 attacks
84%-92%
90
% Approve
80
SOU Address
Jan 27, 1998
67%-70%
House approves impeachment
Late Dec 98
67%-73%
Iraq War
67%-73%
70
Saddam’s
Capture
54%-63%
60
50
Lewinsky scandal goes
public – late Jan 98
54%-59%
40
Jan 97
Jan 98
Jan 99
Jan 00
Jan 01
Jan 02
Month and Year
CBS
ABC
CNN
FOX
Newsweek
Jan 03
Jan 04
PATTERNS OBSERVED IN YEARLY AVERAGES
In order to reduce sample size limitations and minimize the impact of timing in
comparing poll results, we combined individual polls to calculate quarterly and yearly
averages in the percent approve, percent disapprove, and percent “don’t know” for each
of the five organizations. Table 1, which displays yearly average ratings for George W.
Bush from 2001 to 2003, reduces the data for this period to 15 sets of figures. These
yearly averages confirm the impressions derived from the trend line. Overall, the five
polls’ approval ratings are quite similar, but certain organizations display a tendency
toward higher or lower ratings. The range in yearly averages for Bush approval across all
five polls was only four to five points for these three years, but in two of the three years
(2002 and 2003) ABC’s rating was highest and Newsweek’s the lowest. Oddly, 2001
was an exception with Newsweek’s average approval at the high end of the scale.
Further examination, however, revealed that this departure from the typical pattern is an
artifact of the uneven distribution of Newsweek polls in the third quarter of 2001.
Newsweek did no polls that quarter before the 9/11 terrorist attacks, and therefore its
third quarter average was skewed by the extraordinary rise in Bush approval after 9/11
(85%). When the third quarter data is removed from the calculation of yearly averages
for 2001, the typical pattern re-emerges, with ABC having the highest average approval
(68%) and Newsweek the lowest (65%).
These yearly averages for George W. Bush also show at least as much variation in
the percent disapprove as the percent approve, between 5 and 6 points for the three years.
While ABC ratings tend to be the highest or second highest, FOX, not Newsweek tends
to be lowest. The percent “don’t know” also shows some consistency in the yearly
averages, with ABC tending to be lowest and FOX the highest.
Presidential Approval 17
Table 1
George W. Bush Job Ratings: 2003
ABC
% Approve
61
% Disapprove
35
% No opinion
4
CNN
60
36
4
CBS
59
33
8
FOX
58
31
10
NW
57
34
8
George W. Bush Job Ratings: 2002
ABC
% Approve
74
% Disapprove
24
% No opinion
3
CNN
72
22
5
FOX
72
18
10
CBS
71
21
8
NW
69
22
9
George W. Bush Job Ratings: 2001
NW
% Approve
70
% Disapprove
19
% No opinion
12
ABC
68
25
6
CBS
68
20
13
CNN
66
24
10
FOX
66
19
15
Note: FOX News figures based on registered voters.
Now looking at yearly averages for the four years of Bill Clinton’s second term,
the data has some gaps, and does not show all the same patterns as the Bush data. FOX
did not begin asking the standard approval question until 1998 and therefore no data is
available for 1997. With the exception of one poll, Newsweek did not ask the Clinton
approval question on polls conducted during election year 2000, and is therefore not
represented in the yearly averages. Consistent with the patterns observed in yearly
approval averages during the Bush years, Newsweek has the lowest approval rating in the
three years it was active. However, during the Clinton years it was FOX – not ABC –
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 18
that generally had the highest approval ratings. ABC had the highest average disapproval
during Clinton’s second term. Patterns in “don’t know” are also somewhat different from
the Bush years. ABC’s average ratings tend to be the lowest, but it is Newsweek and
CBS – not FOX – that have the highest average “don’t know.”
Table 2
Bill Clinton Job Ratings: 2000
FOX
% Approve
64
% Disapprove
31
% No opinion
6
CBS
62
32
7
ABC
60
37
4
CNN
60
36
4
NW
n/a
n/a
n/a
Bill Clinton Job Ratings: 1999
FOX
% Approve
62
% Disapprove
33
% No opinion
6
CNN
61
36
4
ABC
60
37
3
CBS
60
33
8
NW
57
34
9
Bill Clinton Job Ratings: 1998
FOX
% Approve
66
% Disapprove
28
% No opinion
6
CBS
64
30
7
CNN
64
32
5
ABC
63
33
4
NW
63
31
6
Bill Clinton Job Ratings: 1997
ABC
% Approve
59
% Disapprove
37
% No opinion
5
CBS
59
31
10
CNN
58
34
9
NW
56
33
11
FOX
n/a
n/a
n/a
Note: FOX News figures based on registered voters.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 19
STATISTICAL ANALYSIS OF QUARTERLY AVERAGES
To put differences among the five media polls to a statistical test, Jonathan Best
used the analysis of variance (ANOVA) technique to conduct a series of analyses with a
subset of the data base created for this study. The data used for these analyses were
limited because some polls were not active in certain years and polling done at certain
times was atypical or distorted by external events. A description of the procedures and
findings follows.
ANOVA Procedures and Findings
In order to compare ratings from the five polling organizations, one-way
ANOVAs were run using quarterly averages for percent approve, percent disapprove, and
percent “don’t know/refused” as the dependent variables. Only quarters where every
organization polled were included. All pre-September 11 results were excluded from the
Bush data. This is because early approval ratings are atypical because of their higher
levels of DK/Refused, and data for the third quarter of 2001 were distorted by the huge
rally effect observed after the terrorist attacks on New York and Washington. Quarters
included in analyses are 1st quarter 1998 through 4th quarter 1999 for Clinton and 4th
quarter 2001 through 4th quarter 2003 for Bush. Results are summarized in Table A.
TABLE A
Approve
Between Groups
Within Groups
Disapprove
Between Groups
Within Groups
DK/Refused
Between Groups
Within Groups
Sum of Squares
df
Mean Square
F
Sig
112.643
6046.746
4
80
28.161
75.509
.373
.827
233.022
5663.415
4
80
58.255
70.793
.823
.514
347.186
184.826
4
80
86.797
2.310
37.569
.000
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 20
Analysis shows no significant difference in approval or disapproval ratings by polling
organization. However, there is a significant difference in levels of “DK/Refused”.8
Running post-hoc Bonferroni comparisons shows that ABC and CNN get
consistently lower “DK/Refused” levels than the three other organizations. On average,
ABC gets over 4 percent lower “DK/Refused” responses than CBS, FOX and Newsweek.
CNN’s “DK/Refused” ratings are almost 3 percentage points lower than CBS and more
than 3 percentage points lower than both FOX and Newsweek. CBS, FOX and
Newsweek average “DK/Refused” levels are all about the same.
TABLE B
Org A
ABC
Org B
CNN
CBS
FOX
NW
Mean Difference
(A-B)
-1.2
-4.1*
-4.8*
-4.9*
Sig.
.219
.000
.000
.000
CNN
ABC
CBS
FOX
NW
1.2
-2.9*
-3.6*
-3.7*
.219
.000
.000
.000
8
When we run the analysis of variance for each president separately we get the same results – no
significant differences in approval or disapproval ratings, but a difference in “DK/Refused”. For this
analysis by president, we dropped quarters where an organization did only one poll (2nd Q, 1998; 3rd Q,
1999; 4th Q, 1999).
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 21
VISUAL REPRESENTATION OF POLL DIFFERENCES
Although not large enough to meet the criteria for statistical significance in the
ANOVA, some consistent differences from quarter to quarter are seen in average
approval ratings for the five media polls. Chart 2 shows graphically how the various
polls tracked with one another over the five years that were used for the ANOVA. As
shown in Chart 2, almost without exception, Newsweek’s approval ratings tend to be the
lowest for both presidents.9 During the Bush years at least, CBS tracks pretty
consistently as second lowest to Newsweek. ABC tends to be at the opposite extreme
during the Bush years, with the highest approval ratings.
Chart 2
Presidential Approval: Selected Quarters
90
-------------------Clinton Years--------------------
-------------------Bush Years--------------------
85
% Approve
80
75
70
65
60
55
50
1998.1 1998.2 1998.3 1998.4 1999.1 1999.2 1999.3 1999.4
2001.4 2002.1 2002.2 2002.3 2002.4 2003.1 2003.2 2003.3 2003.4
Year/Quarter
ABC
CBS
CNN
FOX
NW
9
Newsweek data for second quarter 1998 and third quarter 1999 and ABC News data for third quarter 1999
and fourth quarter 1999 are not shown in charts containing quarterly averages because of low activity (only
one poll in each quarter).
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 22
While Newsweek tends toward a lower percent approve, it tends to be in the
middle as far as percent disapprove. As shown in Chart 3, ABC and FOX are most likely
to be at the extremes as far as disapproval. ABC tends to register the highest disapproval;
FOX the lowest. Again, although these differences are not large enough to be statistically
significant in the ANOVA, their consistency suggests they bear further study.
Chart 3
Presidential Disapproval: Selected Quarters
50
-------------------Clinton Years--------------------
-------------------Bush Years--------------------
45
40
% Disapprove
35
30
25
20
15
10
5
0
1998.1 1998.2 1998.3 1998.4 1999.1 1999.2 1999.3 1999.4
2001.4 2002.1 2002.2 2002.3 2002.4 2003.1 2003.2 2003.3 2003.4
Year/Quarter
ABC
CBS
CNN
FOX
NW
Moving to differences among the five polls in the “DK/Refused” ratings which, in
fact, are statistically significant, Chart 4 shows the clear separation between the “high
DK” and “low DK” polls. Leaving FOX aside for the moment, from quarter to quarter
the DK level for ABC and CNN is always lower than the DK level for CBS and
Newsweek. ABC tends to be lowest; Newsweek the highest. While FOX appears to be a
“high DK” poll consistently different from ABC and CNN, the level of DK in the FOX
poll seems different in the Bush data than it is in the Clinton data. During most of the
Bush years, FOX had the highest DK level, often reaching 10% or higher. In contrast,
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 23
FOX was never higher than both CBS and Newsweek in quarterly averages for DK levels
in the Clinton data.
Chart 4
Presidential No Opinion Ratings: Selected Quarters
15
% Don' t Know
-------------------Clinton Years--------------------
-------------------Bush Years--------------------
10
5
0
1998.1 1998.2 1998.3 1998.4 1999.1 1999.2 1999.3 1999.4
2001.4 2002.1 2002.2 2002.3 2002.4 2003.1 2003.2 2003.3 2003.4
Year/Quarter
ABC
CBS
CNN
FOX
NW
In addition to analyzing the quarterly averages for approval and disapproval of the
five polling organizations, we created an “approval ratio” which removes the DK from
the denominator. Specifically, the approval ratio is defined as follows:
Approval Ratio = Approval ÷ (Approval + Disapproval)
Differences in the approval ratio among the five polls over time are displayed in Chart 5.
At least for the most recent years studied, the four polls with a conventional methodology
– ABC, CBS, CNN, and Newsweek – move even closer together when the approval ratio
replaces basic approval as the statistic used for evaluation. The FOX results, however, do
not move toward the others, and follow a somewhat different path. In its approval ratio,
FOX tends to be higher than the other four polls for most of the selected quarters
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 24
analyzed – Bush post-9/11 and Clinton during the middle two years of his second term,
when all five polls were active.
Chart 5
Approval Ratio: Selected Quarters
90
-------------------Clinton Years--------------------
-------------------Bush Years--------------------
85
% Approve
80
75
70
65
60
55
50
1998.1 1998.2 1998.3 1998.4 1999.1 1999.2 1999.3 1999.4
2001.4 2002.1 2002.2 2002.3 2002.4 2003.1 2003.2 2003.3 2003.4
Year/Quarter
ABC
CBS
CNN
FOX
NW
INVESTIGATING THE “DON’T KNOW” FACTOR
If the levels of “don’t know” is the primary difference in the results of the five
media polls – the only difference the ANOVA found to be statistically significant – what
is responsible for the DK factor? Question order and interviewer handling of the
approval question would seem to be the most likely source of this difference. Leaving
aside FOX to focus on the four polls that use a conventional telephone survey
methodology, question order can be removed as a likely factor. All four polls place the
approval question high in the interview, generally the first substantive question except for
trial heats and other election-related questions. The extent to which the various polling
organizations instruct interviewers to probe DKs to the approval question might explain
the differences between three of the polls – ABC, CBS and Newsweek. ABC, which gets
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 25
the lowest level of DKs, has its interviewers re-ask the approval question of all
respondents who do not answer “approve” or “disapprove” the first time around. The
practice is not shared by CBS and Newsweek, the two conventional polls that typically
get the highest levels of DK. CBS interviewers do not probe at all, while Newsweek
interviewers do only limited probing, when a respondent offers a mixed opinion, such as
“it depends.” The level of difference in DKs between these two organizations and ABC,
about 4 percentage points, is consistent with typical differences observed between polls
that ask trial heats with a “lean” question vs. polls that do not ask a “lean” question. This
does not explain, however, why the other “low DK” poll – CNN – consistently gets a
level of DK almost as low as ABC. Gallup researchers who conduct the CNN poll say
their interviewers do not probe the DKs at all. The interviewer probing explanation also
does not work for the FOX poll, which during the Bush years has registered the highest
levels of DK. In fact, FOX is the only poll beside ABC that reports routinely probing all
DKs to the approval question. What’s more, FOX is the only poll of the five that does
not report always placing the approval question very high in the interview. All other
things being equal, an approval rating asked later in the interview would be expected to
register a lower level of DK, not a higher level.10
In order to see what effect reducing the level of DK has on presidential approval
and disapproval, Jonathan Best conducted the following analyses using a subset of the
study data base. Selecting all polls where the percent DK was 15% or less (n=618),
approval ratings were plotted on a graph by DK (see Chart 6), and a linear regression line
was fit through these data.
10
Crespi, Irving. Pre-election Polling: Sources of Accuracy & Error (Russell Sage Foundation: 1988),
page 104.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 26
Chart 6
Approval by Don't Know: Selected Polls
100
95
90
% Approve
85
80
75
70
65
60
55
50
0
2
4
6
8
10
12
14
% Don't Know
% Approve
Linear (% Approve)
A clear relationship between the level of approval and the level of DK is found.
In fact, this regression line has a significant non-zero slope of -0.823. This means that as
DK increases by one percentage point, approval decreases by 0.823 percent. Conversely,
as DK decreases, as it does with probing, approval increases at the same rate –
approximately eight-tenths of a point for each one-point drop in the DK.
This same relationship is not found between disapproval and DK. The regression
line fit to these data has a slope (-0.176) that is not significantly different than zero. In
other words, as DK changes, disapproval ratings tend to remain about the same. (See
Chart 7).
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 27
Chart 7
Disapproval by Don't Know: Selected Polls
50
45
40
% Disapprove
35
30
25
20
15
10
5
0
0
2
4
6
8
10
12
14
% Don't Know
% Disapprove
Linear (% Disapprove)
To ensure that the elevated Bush approval ratings during the post 9/11 period did not
distort the data, these analyses were also done separately for each president. When the
analysis is re-run only for the Bush data, similar results are found to those for the larger
data set. There is a significant negative relationship between DK and approval, and no
significant relationship between DK and disapproval. When the analysis is re-run for the
Clinton ratings, the same relationship is found between approval and DK, but there is also
a small, but statistically significant, relationship between DK and disapproval. For
Clinton’s ratings, as DK decreases by one percentage point, approval increases by .78
points and disapproval increases by .22 points.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 28
TABLE C: RELATIONSHIP BETWEEN DK AND APPROVAL/ DISAPPROVAL RATINGS
Beta
Sig
Both
Approve
-0.823*
0.000
Disapprove
-0.176
0.123
Bush
Approve
Disapprove
-1.229*
0.228
0.000
0.209
Clinton
Approve
Disapprove
-0.779*
-0.217*
0.000
0.009
These findings give support to the hypothesis that probing and interviewing
effects are the reason for the DK factor. The tendency for lower “don’t knows” to push
the approval level up while leaving disapproval largely unaffected is consistent with a
positivity bias frequently observed in response to survey questions about low salience
issues. When people don’t have especially strong opinions about something, pushing
them for a response tends to elicit positive rather than negative views. But as noted
previously, attributing all these differences to probing does not square with reports about
interviewer practices at CNN and FOX. CNN, whose interviewers are not given specific
instructions to probe, nonetheless gets low DKs; FOX, whose interviewers are instructed
to probe, gets high DKs. It may be that less obvious “house effects” play an important
role. Perhaps in some cases differences created by interviewer characteristics, general
interviewer training, or habits formed from administering other surveys in the same
phone room overpower any specific procedures in place for handling the DKs to the
presidential approval question.
One final piece of analysis in evaluating the DK factor was to compare and
contrast the results of the five polls when approval ratings were well outside the normal
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 29
range in which they travel. Table 3 shows the figures for each of the five polls when they
got their highest approval rating for George W. Bush in the fall of 2001. It is noteworthy
that ABC, with its full probing of the DKs had only one percent DK in their high
watermark poll and the highest approval rating of the five (92%). In these atypical
circumstances, CNN is indistinguishable from the three high DK polls in the percent DK.
FOX again shows itself to deviate from the other polls in an unpredictable manner.
While the four conventional polls hit their high point for Bush approval in late September
to early October, FOX did not reach its peak until mid-November. When other polls
registered approval at or near the 90% mark, the FOX poll had approval in the 80% to
84% range.
Table 3
HIGHEST BUSH APPROVAL RATING FOR FIVE POLLS – FALL 2001
Approve
Disapprove
DK
Dates of Interviewing
ABC
CNN
CBS
FOX
Newsweek
11
%
%
%
92
90
89
88
88
6
6
7
7
8
1
4
5
5
4
10/8-9/01
9/21-22/01
9/20-23/01
11/14-15/01
10/11-12/0111
Ratings in a 10/18-19/01 Newsweek Poll were the same.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 30
COMPARISONS WITH GALLUP IN-PERSON APPROVAL RATINGS
One last area examined for this study is whether current presidential approval
ratings in contemporary telephone polls can be compared directly with approval ratings
from past presidents, as measured by The Gallup Poll’s in-person survey methodology.
Since telephone methodology did not come into its own until the late 1970s/early 1980s,
the in-person data is the only polling data available for presidents before Jimmy Carter.
And since Gallup switched to telephone surveys for its reporting of presidential approval
ratings beginning with President George H.W. Bush, in recent years in-person poll data
has not been available for direct comparisons with telephone poll results. There is one
period, however, when there was sufficient in-person Gallup Poll data and telephone
survey data to allow for good comparisons – the 1980s during Ronald Reagan’s
presidency. The CBS News poll is the obvious choice – and perhaps even the only
choice – for making these comparisons. CBS is the only poll of the five studied that had
the following combination of characteristics: 1) was sufficiently active during the Reagan
years; 2) used the same basic methodology then as it does today12; 3) did not change
phone rooms; and 4) had continuity of leadership.
To make the comparisons between CBS telephone poll approval ratings and those
from Gallup in-person surveys during the Reagan years, the data for each organization
were combined to create yearly averages.13 (The frequency of polling was not as high at
that time and therefore quarterly averages were not an option.) The yearly averages were
then combined to calculate grand average yearly differences for percent approve, percent
12
In an email to Larry Hugick on 4/23/04, Kathleen Francovic indicated that CBS used the same basic
methodology in the 1980s, except for weighting by the number of telephone lines in the household, not
added until the tail end of the Reagan years.
13
The fourth quarter of 1986 was excluded from the calculation for that year because of distorting effect of
the Iran-Contra scandal.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 31
disapprove, and percent DK. As seen in Table D, CBS telephone polls track extremely
close to Gallup Poll in-person results in the grand averages for Reagan approval, not
varying by more than two points on any of the three measures.
TABLE D: CBS TELEPHONE VS. GALLUP IN-PERSON RESULTS: 1981-1988
Approve
Disapprove
DK
%
%
%
1981
CBS
Gallup
Difference
57
57
0
28
31
-3
15
12
3
1982
CBS
Gallup
Difference
45
44
1
43
46
-3
13
10
3
1983
CBS
Gallup
Difference
44
44
0
43
46
-3
13
10
3
1984
CBS
Gallup
Difference
57
56
1
33
35
-2
11
9
2
1985
CBS
Gallup
Difference
61
60
1
28
31
-3
11
9
2
1986
CBS
Gallup
Difference
65
63
2
25
27
-2
10
10
0
1987
CBS
Gallup
Difference
49
48
1
40
42
-2
10
9
1
1988
CBS
Gallup
Difference
52
52
0
36
37
-1
12
10
2
Average
Difference
1
-2
2
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 32
This analysis shows us how CBS compared with the Gallup in-person results in
the past and, inferentially, how it might compare today. But what about the other four
polls? To arrive at rough estimates for how figures from ABC, CNN, FOX, and
Newsweek might compare to Gallup in-person results, two steps were required: 1)
calculating average differences between CBS approval figures and each of the other four
polls; and 2) applying “key” provided by the CBS vs. Gallup differences to extrapolate
how much the other four polls differ from the in-person results. For the comparisons to
CBS, we decided to use the most recent data possible, calculating average differences
between each poll and CBS for each of the three Bush years, and then averaging the
yearly differences. (The 2001 average excluded the problematic third quarter.). Table E
shows step one of this analysis.
TABLE E: ANNUAL DIFFERENCES BETWEEN CBS AND OTHER ORGANIZATIONS
(PERCENTAGE POINTS)
2001
ABC
CNN
FOX
Newsweek
Approve
2
1
1
-1
Disapprove
3
1
-4
1
DK
-4
-3
2
0
2002
ABC
CNN
FOX
Newsweek
3
1
1
-2
3
1
-3
1
-5
-3
2
1
2003
ABC
CNN
FOX
Newsweek
2
1
-1
-2
2
3
-2
1
-4
-4
2
0
Average
ABC
CNN
FOX
Newsweek
2
1
0
-2
3
2
-3
1
-4
-3
2
0
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 33
The differences for each poll from CBS during the George W. Bush years were
then adjusted to take into account CBS differences from Gallup in-person results during
the Reagan years. Table F summarizes the final results of the two-step process,
calculations for how each approval figure for each of the five contemporary telephone
polls compares with those from historic in-person Gallup polls. These findings would
indicate that all five telephone polls come reasonably close to approximating the approval
ratings that had been obtained in the past using an in-person interviewing methodology.
The high DK telephone polls, however, appear to be slightly closer to the in-person
results than the low DK telephone polls on the key measure of percent approve. CBS,
FOX, and Newsweek are all within a point in percent approve. ABC averages three
points higher, and CNN averages two points higher than in-person results. It should be
noted, however, when we go beyond this most widely-reported statistic, the story changes
somewhat. FOX diverges from in-person results more than any other telephone poll in its
average percent disapproval (-5) and average don’t know (+4). ABC estimates of
disapproval are a perfect match to in-person results, while CNN and Newsweek are just a
point lower on this same statistic.
TABLE F: ESTIMATED DIFFERENCE BETWEEN TELEPHONE POLL APPROVAL
RATINGS AND GALLUP IN-PERSON RATINGS (PERCENTAGE POINTS)
ABC
CNN
FOX
CBS
Gallup In-Person
Newsweek
Approve
3
2
1
1
0
-1
Disapprove
0
-1
-5
-2
0
-1
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
DK
-2
-1
4
2
0
2
Presidential Approval 34
Several caveats are appropriate for the preceding analysis. First, it does not account for
real world differences that might change the results of polls today compared with the
1980s. And it also assumes that the patterns seen in differences in poll ratings for one
president (George W. Bush) would hold for a different president.
Conclusions
The five media polls studied track very closely with one another in their
presidential job approval ratings, especially when the focus is narrowed to the mostwidely reported statistic (percent approve) rather than the full set of results that includes
the percent disapprove and “don’t know.” For the period of 1997 to 2003, the typical
difference across all five polls in average annual approval is just four percentage points,
and the five poll trend lines move up and down largely in concert. That said, our analysis
does not suggest that presidential approval ratings for the five polls are truly
interchangeable. There are some differences that follow a consistent pattern over time,
and even one difference – the level of “don’t know” – that is statistically significant.
To put the approval results of these five polls in perspective, our study indicates
they should be divided in two ways. First, it is useful to separate the four polls that
employ a conventional telephone survey methodology with a full demographic weighting
from the one poll that takes a different methodological approach. The ABC
News/Washington Post poll, the CBS News poll, the CNN/USA Today/Gallup poll and
the Newsweek poll all use a conventional methodology. While these four polls differ in
some specific aspects of methodology – such as length of a typical interviewing period
and respondent selection procedures – they have enough in common that in the few ways
their results seem to differ, they do so in a predictable and consistent manner. The FOX
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 35
News/Opinion Dynamics poll, the newest of the five polls, differs from the other four in
basic methodology, using a registered voter base and only limited demographic weighting
of its sample. The FOX poll maintains demographic balance through sample controls
rather than weighting, does not do much to control for respondents’ education levels, and
does not have a general policy of placing the approval question at the front of the
interview. The second important way to divide the five polls is by their typical level of
“don’t know” to the approval question. Two of the polls with a conventional
methodology are “low DK” (ABC and CNN) and two are “high DK” (CBS and
Newsweek). FOX has been a high DK poll, and has shown an average percent approval
remarkably close to CBS and Newsweek in recent years. However, there are enough
differences in FOX and these two polls on other measures to suggest it needs to be kept
separate.
Statistical analysis indicates that ABC and CNN consistently get a lower level of
DK to the approval question than the other two conventional polls, CBS and Newsweek,
and FOX. What’s more, analysis suggests that lower DKs tend to drive approval ratings
higher while leaving disapproval largely unaffected. For each one point reduction in the
level of DK, approval rises by eight-tenths of a point but disapproval is statistically
unchanged. All this suggests that poll watchers should exercise greater caution when
interpreting approval ratings from polls in a different DK category. Newsweek approval
ratings, for example, are more comparable to those of CBS than to those of ABC.
The FOX/Opinion Dynamics poll has tracked remarkably close to the four
conventional polls – particularly those in the high DK category – in percent approve
during recent years. Nonetheless, analysis identifies differences below the surface in the
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 36
FOX poll results that set it apart from the other polls. The FOX poll’s level of DK has
varied more than the others between presidents, often reaching the level of 10 percent or
higher double-digits during the Bush years but tracking well below 10 percent during the
Clinton years. The four conventional polls’ trend lines tend to converge when the DK is
factored out of the equation, but FOX does not join the pack. FOX seems to register
lower disapproval ratings, not only during the Bush years but during the Clinton years as
well. The FOX poll’s high point in Bush approval came a month after other polls
showed his rating at a peak, a lag time that is not easily explained. For all these reasons,
it seems prudent to assume that the FOX poll approval ratings are not interchangeable
with those of the conventional polls, at least until it establishes more of a track record.
Historical comparisons of current presidential approval ratings with those of past
presidents often draw on data from the in-person Gallup Poll. Our analysis suggests that
the differences in results by phone vs. in-person methodology are relatively small, but
that approval ratings from in-person polls are slightly closer to those from high DK
telephone polls like CBS and Newsweek than from low DK polls like ABC and CNN.
FOX is about as close as the other high DK polls to Gallup in-person polling in percent
approve, but tends to register lower disapproval and higher don’t know.
The study failed to clearly identify the factor chiefly responsible for differences in
DK across the various polls. Interviewer instructions to aggressively probe the don’t
knows would seem to be important in distinguishing ABC from CBS and Newsweek, but
does not explain why CNN tends to be a low DK poll and FOX a high DK poll. This is a
fertile area for future research to enhance our understanding of the factor that most
distinguishes approval data from the five polls.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 37
APPENDIX
POLLING METHODS QUESTIONNAIRE
PSRAI is conducting a study on presidential approval ratings for the period of 1997-2003 for
presentation at the 2004 AAPOR conference. This short survey will help us gather important
information for our analysis. The following questions refer to telephone surveys in which
respondents are asked their opinions on presidential approval, i.e. “Do you approve or disapprove
of the way [sitting president] is handling his job as president?” For each question please mark an
“x” in boxes that apply to the methods your company currently uses.
1. Where is the presidential approval question placed on your questionnaire? (Please choose
one)
CBS ABC CNN FOX NW
Always first
•
Second only to horserace questions
•
•
•
Always later in the interview
No specific placement/Varies from poll to poll
•
2. Which of the following variables do you use to weight your sample? (Please choose all that
apply)
CBS ABC CNN FOX NW
Gender
•
•
•
∗
•
Age
•
•
•
∗
•
Race/Hispanic origin
•
•
•
•
•
Education
•
•
•
•
Region
•
•
∗
•
Party identification
Income
Number of adults in household
•
∗
Number of phone lines in household
•
•
* These variables controlled by CATI system.
3. If conduct interviews at an outside organization, does your company…
ABC FOX
•
•
NW
•
Always use the same outside organization
Use multiple outside organizations
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 38
4. Are interviewers instructed to ask to speak with… (Please choose one)
CBS ABC CNN FOX NW
•
•
•
•
•
Any adult who is at home
The youngest male/oldest female over 18
The person over 18 with the most recent birthday
Other (Please specify)
CBS – Interviewer asks for number of adult men and women; random selection made from all
adults in household (whether or not at home); appointments made with selected respondent
ABC – respondent now at home; last birthday; sex
5a. Are interviewers instructed to probe or re-ask the presidential approval question if
respondents do not answer “approve” or “disapprove” initially?
CBS ABC CNN FOX NW
Yes – Go to question 5b
•
•
•
No – Go to question 6
•
•
5b. Are interviewers instructed to probe in all cases, or only in certain cases, such as respondents
who say “it depends” or “approve for some things but disapprove for others”?
ABC FOX
•
•
NW
•
In all cases – Go to question 5c
Only in certain cases – Go to question 5c
5c. Are interviewers instructed to probe only once, or to make several attempts, if necessary, to
try to get an answer?
ABC FOX
•
•
NW
•
Once
Make several attempts if necessary
6. Which of the following does your company call back at least once in order to complete an
interview?
CBS
•
•
•
ABC CNN FOX NW
Non-contacts (e.g. no answer, busy signal, answering machine)
•
•
•
•
Refusals
•
•
•
Neither
Other (Please specify)
CBS – Call back selected respondents who were not at home at the time of the call
(appointments); also redial all “not in service” numbers.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 39
7. What type of sample do you typically use? (Please choose one)
CBS ABC CNN FOX NW
•
•
•
•
•
Listed sample
Random digit dial
Other (Please specify)
8. Finally, please indicate below if there have been any important changes in your survey
methodology/ practices from 1997 to 2003.
CBS - Began calling back not-in-service numbers about two years ago.
CNN - Changed to most recent birthday for respondent selection following the 2000
election. Prior to election used youngest male/oldest female.
FOX - Reports experimenting with different CATI systems that vary respondent selection or
callback procedures.
Newsweek - Increased its sample size from 750 to 1,000 in January 2001.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 40
PRESIDENTIAL APPROVAL BY ORGANIZATION
ABC News/Washington Post
100
--------------------Clinton Years--------------------
--------------------Bush Years-------------------Early Oct 01
92%
Percent Approve
90
9/11 attacks 90%
80
SOU Address
Jan 27, 1998
68%
70
Near end of
Iraq War
71%
Late June 03
68%
House approves impeachment
Late Dec 98
67%
Clinton acquitted
Mid Feb 1999
68%
Saddam’s Capture
59%
60
Lewinsky scandal goes
public – late Jan 98
57%
50
40
Jan 97
Jan 98
Jan 99
Jan 00
Jan 01
Jan 02
Jan 03
Jan 04
Month and Year
CBS News
100
--------------------Clinton Years--------------------
--------------------Bush Years--------------------
89%
90
9/11 Attacks
Percent Approve
84%
80
Mid War
73%
House approves impeachment
Late Dec 98
73%
Late Feb 98
68%
Start of Iraq War
67%
End of War
67%
70
Saddam’s Capture
60%
60
50
Lewinsky scandal goes
public – late Jan 98
56%
40
Jan 97
Jan 98
Jan 99
Jan 00
Jan 01
Jan 02
Month and Year
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Jan 03
Jan 04
Presidential Approval 41
CNN/USA Today/Gallup
100
--------------------Clinton Years--------------------
--------------------Bush Years-------------------90%
90
9/11 Attacks
Percent Approve
86%
80
House approves impeachment
Late Dec 98
73%
Clinton acquitted
Mid Feb 1999
68%
SOU Address
Jan 27, 1998
70
69%
Start of Iraq War
71%
Mid-War 71%
End of War 69%
Saddam’s Capture
63%
60
50
Lewinsky scandal goes
public – late Jan 98
58%
40
Jan 97
Jan 99
Jan 98
Jan 00
Jan 01
Jan 02
Jan 03
Jan 04
Month and Year
FOX News/Opinion Dynamics
100
--------------------Clinton Years-------------------
--------------------Bush Years-------------------Nov 01
88%
Percent Approve
90
80
70
9/11 attacks
81%
Approval question
not asked in 1997
Late Feb 98
69%
Mid-War
71%
Early Dec 98
69%
Clinton acquitted
Mid Feb 1999
68%
Start of Iraq War
66%
End of War
65%
60
Saddam’s
Capture
52%
50
40
Jan 97
Jan 98
Jan 99
Jan 00
Jan 01
Jan 02
Month and Year
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Jan 03
Jan 04
Presidential Approval 42
Newsweek
100
--------------------Clinton Years--------------------
--------------------Bush Years-------------------Oct 01
88%
90
86%
Percent Approve
9/11 Attacks
82%
80
SOU Address
Jan 27, 1998
70
Mid-War
71%
70%
Clinton acquitted
Mid Feb 1999
66%
Start of
Iraq War
68%
60
50
End of War
65%
Saddam’s
Capture
54%
Lewinsky scandal goes
public – late Jan 98
54%
40
Jan 97
Jan 98
Jan 99
Jan 00
Jan 01
Jan 02
Month and Year
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Jan 03
Jan 04
Presidential Approval 43
QUARTERLY AVERAGES BY ORGANIZATION
ABC News/Washington Post
2003.4
2003.3
2003.2
2003.1
2002.4
2002.3
2002.2
2002.1
2001.4
2001.3
2001.2
2001.1
2000.4
2000.3
2000.2
2000.1
1999.4
1999.3
1999.2
1999.1
1998.4
1998.3
1998.2
1998.1
1997.4
1997.3
1997.2
1997.1
% Approve
% Disapprove
% No opinion
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
56
58
70
61
40
39
28
35
4
3
4
4
2003 Yearly Average
61
35
4
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
67
70
76
81
31
28
21
16
3
3
3
3
2002 Yearly Average
74
24
3
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
89
68
59
57
9
28
36
28
2
3
6
15
2001 Yearly Average
68
25
6
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
n/a
60
57
62
n/a
35
40
35
n/a
5
3
3
2000 Yearly Average
60
37
4
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
59
58
59
65
38
38
38
33
3
4
4
2
1999 Yearly Average
60
37
3
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
64
63
63
63
33
34
33
32
3
4
5
4
1998 Yearly Average
63
33
4
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
59
61
58
58
36
34
38
38
6
5
5
5
1997 Yearly Average
59
37
5
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 44
CBS News
2003.4
2003.3
2003.2
2003.1
2002.4
2002.3
2002.2
2002.1
2001.4
2001.3
2001.2
2001.1
2000.4
2000.3
2000.2
2000.1
1999.4
1999.3
1999.2
1999.1
1998.4
1998.3
1998.2
1998.1
1997.4
1997.3
1997.2
1997.1
% Approve
% Disapprove
% No opinion
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
55
55
67
60
37
37
26
32
9
8
7
8
2003 Yearly Average
59
33
8
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
63
67
74
81
29
24
17
12
9
8
9
7
2002 Yearly Average
71
21
8
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
86
74
55
57
8
17
32
22
6
9
13
22
2001 Yearly Average
68
20
13
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
68
60
59
59
28
35
33
33
4
6
8
9
2000 Yearly Average
62
32
7
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
57
58
58
66
35
33
34
29
8
9
8
5
1999 Yearly Average
60
33
8
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
66
63
63
62
29
31
29
30
5
6
8
8
1998 Yearly Average
64
30
7
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
59
58
58
62
31
31
32
31
11
11
10
8
1997 Year Average
59
31
10
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 45
CNN/USA Today/Gallup
% Approve
% Disapprove
% No opinion
2003.4
2003.3
2003.2
2003.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
55
57
67
61
42
39
30
34
3
4
4
5
2003 Yearly Average
60
36
4
2002.4
2002.3
2002.2
2002.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
64
68
75
81
29
26
19
14
6
5
6
4
2002 Yearly Average
72
22
5
2001.4
2001.3
2001.2
2001.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
87
63
56
58
9
29
33
25
4
8
11
17
2001 Yearly Average
66
24
10
2000.4
2000.3
2000.2
2000.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
62
59
58
61
34
37
38
35
4
4
5
4
2000 Yearly Average
60
36
4
1999.4
1999.3
1999.2
1999.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
58
60
59
66
38
36
37
31
5
4
4
3
1999 Yearly Average
61
36
4
1998.4
1998.3
1998.2
1998.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
66
63
62
63
31
33
33
32
3
4
6
6
1998 Yearly Average
64
32
5
1997.4
1997.3
1997.2
1997.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
58
60
56
59
33
32
36
33
9
8
9
8
1997 Yearly Average
58
34
9
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 46
FOX News/Opinion Dynamics (based on RVs)
% Approve
% Disapprove
% No opinion
2003.4
2003.3
2003.2
2003.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
52
57
65
60
37
34
25
30
11
9
10
11
2003 Yearly Average
58
31
10
2002.4
2002.3
2002.2
2002.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
65
68
75
80
24
21
15
12
11
11
10
8
2002 Yearly Average
72
18
10
2001.4
2001.3
2001.2
2001.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
85
63
60
55
8
25
25
19
7
13
14
26
2001 Yearly Average
66
19
15
2000.4
2000.3
2000.2
2000.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
63
64
65
65
32
32
30
29
6
5
6
6
2000 Yearly Average
64
31
6
1999.4
1999.3
1999.2
1999.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
61
62
59
65
33
32
35
30
6
7
6
6
1999 Yearly Average
62
33
6
1998.4
1998.3
1998.2
1998.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
67
64
66
68
28
29
27
26
5
6
7
6
1998 Yearly Average
66
28
6
Note: Presidential approval question not asked by FOX News in 1997.
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL
Presidential Approval 47
Newsweek
% Approve
% Disapprove
% No opinion
2003.4
2003.3
2003.2
2003.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
52
53
66
59
40
38
26
33
8
9
9
8
2003 Yearly Average
57
34
8
2002.4
2002.3
2002.2
2002.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
61
66
71
79
29
25
18
14
10
9
10
8
2002 Yearly Average
69
22
9
2001.4
2001.3
2001.2
2001.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
85
85
55
54
10
10
30
25
5
5
16
20
2001 Yearly Average
70
19
12
2000.4
2000.3
2000.2
2000.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
n/a
n/a
52
n/a
n/a
n/a
37
n/a
n/a
n/a
11
n/a
2000 Yearly Average
n/a
n/a
n/a
1999.4
1999.3
1999.2
1999.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
57
52
56
62
34
36
35
32
10
12
9
6
1999 Yearly Average
57
34
9
1998.4
1998.3
1998.2
1998.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
61
60
66
63
34
34
26
29
5
5
8
7
1998 Yearly Average
63
31
6
1997.4
1997.3
1997.2
1997.1
Quarterly Average
Quarterly Average
Quarterly Average
Quarterly Average
59
n/a
55
54
30
n/a
35
33
11
n/a
10
12
1997 Yearly Average
56
33
11
PRINCETON SURVEY RESEARCH ASSOCIATES INTERNATIONAL