International Journal of Public Opinion Research HI. g No. 4
0954—2892/97 $3.00
OPINION QUALITY IN PUBLIC OPINION
RESEARCH
Vincent Price and Peter Neijens
The following article is part of IJPOR's series on quality criteria in survey
research, based upon papers presented at the WAPOR Regional Seminar,
'Quality Criteria in Survey Research', held at Cadenabbia, Italy, in June 1996.
WJG
ABSTRACT
In recent years, a number of new techniques have been developed"—including deliberative
polls and educational surveys—that attempt to gather measures of public opinion that
is of higher quality (i.e. better informed or more deliberative) than that recorded in
typical mass opinion surveys. This paper addresses several general sets of questions.
What is meant by 'quality' in public opinion? What criteria can be enumerated by
which the quality of public opinion can be assessed? In grappling with these questions,
the paper argues that conceptions of quality in public opinion are inextricably bound
to broader conceptions of quality in democratic decision making, a complex process
involving multiple phases and collective participants. In addition, a number of important
contradictions and ambiguities underlie conceptions of quality in public opinion.
Since the inception of scientific study of public opinion, survey researchers and
democratic theorists alike have pondered the central concept of public opinion
and its relationship to mass survey data. Classical theorists of public opinion
framed it as an emergent product of widespread discussion—emanating ideally
from debate that is open to wide popular participation, free flowing and
uncensored, and well informed (e.g. Bryce, 1888; Park, 1972; also Lasswell,
1041; Blumer, 1946; Lazarsfeld, 1957; Habermas, 1989). However, early scientific
analysts (e.g. Allport, 1937) found the classical conception of public opinion as
an 'emergent product' of discussion difficult to grasp empirically and problematic
in a number of respects, and over time they came to accept mass survey data
as the only workable empirical rendering of public opinion (Key, 1961; Converse,
1987). Pioneering pollsters and survey researchers also clearly aimed at achieving
broad democratic participation by giving voice to popular opinion through mass
surveys (Converse, 1987).
© World Association for Public Opinion Research iggj
OPINION QUALITY IN PUBLIC OPINION RESEARCH
337
Yet the extent to which general population surveys render valid measures of
classically denned public opinion—grounded in public discussion and well
informed by debate—has been questioned by scholars of many stripes (Blumer,
1948; Mills, 1956; Habermas, 1989; Graber, 1982; Neuman, 1986; Ginsberg,
1986; Crespi, 1989; Entman, 1989; Fishkin, 1991; Herbst 1993). Empirical
evidence does seem to support the conclusion that opinions given to pollsters
and survey researchers can often be disorganized, disconnected, individual
responses formed outside the arena of public debate (Bishop, Oldendick,
Tuchfarber and Bennett, 1980; Devine, 1970; Neuman, 1986; Zaller, 1992).
The relationship of mass survey data to informed public opinion has re-surfaced
as a vigorous and sometimes contentious debate among contemporary researchers. Recently and with increasing frequency, 'deliberative' polls, 'educational' polls, polls of 'informed' public opinion and variants of focused
group discussions have been advanced as supplements—and in some cases
alternatives—to mass opinion surveys. These efforts are variously intended to
redress perceived problems of superficiality in mass opinion data, deficiencies
in mass communication of public affairs information, or both. Some researchers
have set about the task of gathering readings of'better-informed' public opinion,
in order to counter the tendency of mass opinion surveys to collect and
disseminate opinions that may be ill-informed 'non-attitudes' or 'pseudoopinions' developed outside of any meaningful public debate. Fishkin (1991;
1995) for example, has recently conducted several large-scale 'deliberative' polls—
televised discussions among nationally representative groups of citizens—aiming
to repair the USA's present reliance on superficial readings of mass opinion,
defects in the representativeness of legislative bodies, and the tendencies of the
news media to engage in unreflective 'sound-bite' coverage of political affairs.
These contemporary research trends have developed alongside a growing
fascination in what has become known as 'public journalism' (e.g. Rosen, 1991),
a trend that eschews the objective stance of the journalist as a disinterested and
impartial observer, criticizes the 'commodification' of opinion through the
routine collection and publication of polls, and aims to restore popular discussion
and political participation presumed lost or in serious decline to contemporary
public affairs. In a number of recent treatments (e.g. Herbst, 1993; Parkin,
1995), polls have come under harsh criticism. They are seen as manufacturing
or 'seducing' public opinion (Salmon and Glasser, 1995), as suppressing public
engagement, or as contributing in various indirect ways to the demise of public
debate and the collapse of the discursive public sphere.
Other analysts are far more sanguine about polling and the value of mass
survey data and, although welcoming the new interest in public discussion and
debate, express doubts about the supposed degeneration of the public sphere
and the utility of some recently touted deliberative techniques (e.g. Traugott,
338
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
1992). At the same time, various researchers have proposed some middle-ground
approaches, in less drastic modifications of standard survey research techniques
that attempt assessments of more informed or deliberative opinion (e.g. the
'Americans Talk Issues' program led by Alan Kay and associates in the USA
(Kay, Henderson, Steeper and Lake, 1994) or the Dutch 'Choice Questionnaire'
developed by Willem Saris and colleagues (Neijens, Ridder and Saris, 1992).
By what criteria can the quality of these various efforts to gather measures
of 'deliberative' or 'informed' public opinion be judged? This is a large and
important question for public opinion research to address, as it strikes at the
validity of measures of public opinion. Indeed it begs the larger question: By
what criteria can the quality of public opinion itself be judged? When confronted
widi a reading of public opinion deemed 'better' than some other (the standard
here being a conventional opinion poll or an election outcome), one needs to
consider very carefully the specific evaluative criteria to be applied. For this
reason, we attempt to address the following questions: What is meant by
'quality' in public opinion? What criteria can be enumerated by which the
quality of public opinion can be assessed?
Our analysis has three parts. Following Kaplan (1964), we first look at the
problems the concept of public opinion was created to deal with—problems of
collective decision making—and the ways in which it contributes to the
solution of those problems. Decision making in a democratic society is a
complex, over-time process involving several phases and implicating multiple
groups of actors: political leaders, interest groups, the journalistic community,
attentive publics, and much larger mass audiences. There are consequently
numerous distinct aspects of the process that might be judged problematic
from a quality perspective and a correspondingly wide range of options for
possible reform. In the second section of our discussion, we attempt to
enumerate the various quality concerns that have been voiced in the research
literature, with the aim of cataloging the various dimensions underlying
judgments of quality in public opinion. In the third and final section, we
discuss some of the difficulties these criteria present for researchers interested
in operationalizing various concepts of quality in measuring public opinion.
The quality criteria voiced to date in the literature, we submit, belie several
fundamental contradictions. What's more, critical ambiguities surround the
application of quality criteria to different phases or different participants in
the collective decision-making process. These contradictions and ambiguities
cloud debates over new polling and research techniques that aim at rendering
measurements of higher quality public opinion. We hope to clarify the often
opaque theoretical connections between commonly held—although rarely
explicitly defined—notions of quality in public opinion and the variety of new
operational procedures advanced in its name.
OPINION QUALITY IN PUBLIC OPINION RESEARCH
339
WHAT IS 'QUALITY' IN PUBLIC OPINION?
The concept of public opinion is part and parcel of democratic theory, created
in response to problems of collective judgment and decision making. The
social-philosophical and empirical literatures alike indicate an overriding concern
with the 'goodness' of democratic judgment, its 'proper' reflection of public
opinion, and the 'soundness' of that opinion (e.g. Madison, 1966; Bryce, 1888;
Lippmann, 1922; Dewey, 1927; Berelson, 1950; Lazarsfeld, 1957; Habermas,
1989; Yankelovich 1991; Fishkin 1995). As Price (1992, p. 2) has noted, connecting the concepts public and opinion represented an attempt by liberal
democratic philosophy to unite the 'one' and the 'many', to craft mechanisms
that create coordinated, collective action out of disparate individual choices. It
did so principally by investing in the idea of democracy: collective decision
making through deliberative communication, namely, through discussion and
debate among members of the citizenry, under conditions of openness, fairness,
mutual respect, and concern for the common welfare (e.g. see Habermas, 1982,
on the 'ideal speech situation').
COLLECTIVE DECISION MAKING
The concept of public opinion is thus inextricable from the concept of collective
decision making. Not surprisingly, many theorists of public opinion over the
last century (e.g. Bryce, 1888; Park, 1972; Foote and Hart, 1953; Davison,
1958) have thus conceptualized public opinion in terms of collective, over-time
processes unfolding through a series of phases or stages leading toward some
collective action. These stage or process models not only bear a strong resemblance to one another (Price, 1992, pp. 30-1), but also overlap to a surprising
extent with more formal models of decision making adopted in the past 30
years by social scientists working in the areas of decision analysis (e.g. Slovic,
Fischhoff and Lichtenstein, 1977; Keeney and Raiffa, 1976; Jungerman, 1980;
Humphreys, 1983) and policy analysis (e.g. Stokey and Zeckhauser, 1978;
Kuypers, 1980). These fields have focused on the ways a decision problem is
structured, methods of information acquisition and organization, the ways
alternatives are evaluated, and the manner in which optimal alternatives are
selected. Following the general outlines suggested by these various literatures,
we can reasonably describe the process of democratic decision making—in an
admittedly idealized and perhaps over-rationalized fashion—as including five
main phases: (1) the elicitation of values, (2) the development of options; (3)
the estimation of consequences, (4) the evaluation of alternatives, and (5) the
decision itself. Let us examine each phase in turn.
340
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
Elkitation of values. An essential part of the problem-solving process is the
definition of the problem itself. What exactly is at stake? What precisely do we
want? In their analysis of public opinion as collective behavior, Foote and Hart
(1953) called this the problem phase of the public opinion process, originating
out of a poorly defined but shared sense of the need for something to be done.
Because people often do not know what they want, a critical first step is the
elicitation of goals and values considered important. By the end of this phase,
the problem has crystallized into a recognized issue, and a significant number
of people have identified their (perhaps conflicting) goals. Still, they do not
quite understand the best way to achieve those aims.
Development of options. Consequently, the second phase of collective decision
making involves the formulation of possible options for dealing with the problem,
identified by Foote and Hart (1953) as the proposal phase. At this point the
multiple dimensions of the problem are sorted out, while people struggle to
identify alternative ways of resolving it. A large number of differing ideas and
potential solutions may be contributed, but many are eliminated from further
consideration as a smaller, manageable number of viable proposals come to the
fore.
Estimation of consequences. Once alternative proposals have been developed,
which is then to be chosen? In pondering that choice, decision makers confront
the need to project a set of alternative futures: What will happen if we choose
this option, or that? Decision makers thus need to estimate, as well as one is
able under conditions of uncertainty, the various intended consequences and
potential unintended consequences should various options be chosen (Keeney
and Raiffa, 1976; Kuypers, 1980). In the case of collective decision making, it
often falls upon the shoulders of technical experts to make such estimations,
which are then in the realm of scientific predictions or 'educated guesses' rather
than mere conjectures.
Evaluation of options. Next comes the evaluation of alternative proposals, what
Foote and Hart (1953) referred to as the policy phase. This is the phase that is
most clearly identified as public discourse, when advocates of each competing
policy seek to advance their cause with persuasive appeals. It is at this stage of
the public opinion process, noted Davison (1958), that even people who are
unconcerned with the issue at hand are drawn in. 'Although they may not have
formed a personal attitude about the issue', Davison observed, 'and indeed may
not be aware of its nature, they still cannot ignore the behavior of those about
them who do feel strongly about it' (p. 101). Pollsters actively monitor opinion
during this phase of the process, and the news media keep a close tally of the
debate as it ensues.
OPINION QUALITY IN PUBLIC OPINION RESEARCH
341
Decision. The evaluation phase leads eventually to a choice of one or another
proposal. This decision may take the form of a single person's choice, or, in
the case of representative democracies, it could take collective form as an
election outcome. There are then a series of additional follow-up decisions to
be reached concerning implementation of the agreed-upon choice (what Foote
and Hart, 1953, term the program phase). Finally, once action is taken, it is
sometimes later evaluated in light of its intended effects (the appraisal phase in
the conceptual scheme of Foote and Hart).
A SCHEMATIC VIEW OF THE PROCESS
Decision analysts often intend their models of decision making as prescriptive
methods for improving individual judgments, but policy analysts and public
opinion researchers are clearly interested in more complex, collective decisionmaking processes. Large-scale, democratic choices and actions are especially
complicated due to the fact that a large number of different individuals and
groups may be differently involved in different parts of the decision-making
process. Another complication is that, in practice, the phases described earlier
do not necessarily unfold as stages, in linear fashion. As numerous critics of
rational decision-making models have suggested, collective decision making is
a very ambiguous, politically charged, and haphazard affair (e.g. Edelman, 1995;
Yankelovich, 1991). March (1994, 1995) notes that often the process is actually
turned on its head in what is known as the 'garbage can' model of decision
making: political 'solutions' favored by some interest groups and political leaders
may well lie in wait of corresponding 'problems' to which they can readily be
applied (see also Kingdon, 1984).
Keeping in mind such complexities, how are we to represent the process
conceptually? Figure 1 attempts one schematic view, in matrix fashion, by
crossing each phase of decision making with each of the major groups of
participants: political leaders, technical experts, interest groups, news reporters
and editors, members of attentive publics, and members of mass audiences.
Figure 1 is intended to suggest a wide range of possible collective decisionmaking processes. In each cell of the matrix, one could identify a variety of
potential contributions from each group of participants to different phases of
the decision-making process. As public opinion researchers, we focus on the
role played by the press, by attentive members of the public, and by mass
audiences, but their activities and contributions cannot be properly understood
without reference to all activities constituting the total process. Comparisons
across the vertical dimension of the matrix capture the relative degree to which
the process is 'top-down' or 'bottom-up' in nature. We might well imagine, at
one extreme, a model of elite decision making, which would locate activity only
342
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
Elicitation of
goals/values
Development
of options
Estimation of
consequences
Evaluation of
options
Decisions
Political
leaders
Technical
experts
Intererst
groups
Reporters and
editors
Attentive
publics
Mass
audiences
FIGURE I The collective decision-making process—matrix of phases and participants
in the uppermost rows of the matrix—political leaders, technical experts, and
interest groups—across all phases of the decision process. Lippmann (1922),
for example, having identified a number of fundamental inadequacies of both
the press and the public in dealing with an increasingly complex environment,
suggested a form of technocratic rule that invested heavily in the capacities of
political leaders and technical experts to discern sensible options and to organize
public opinion for the press (p. 32). At the other extreme, a model of direct or
participatory democracy (e.g. Pateman, 1970) locates a great deal of desirable
activity in the lower cells of the matrix, aiming to involve ordinary citizens in
as active a fashion as possible, across all phases of the collective decision-making
process.
Between these extremes are a variety of representative democratic models (see
Held, 1987), including pluralistic theories of democratic decision making (e.g.
DahJ, 1971, 1985). These models place burdens for much of the developmental
work—the elicitation of values, development of options, and estimation of
consequences—on political leaders, technical experts, and organized interest
groups, who are then required during the evaluation phase to refer their favored
options to the citizenry for final decision. In some models ('polyarchicaT
democracies), periodic selection and removal of political leaders is deemed a
sufficient mechanism of public decision making, whereas other systems ('populist'
OPINION QUALITY IN PUBLIC OPINION RESEARCH
Elicitation of
goals/values
Political
leaders
Promote
goals/values; leek
input from
constituents
Development
of options
Estimation of
consequences
Develop political & Engage experts in
technical options;
estimation of
seek input from
consequences
experts
343
Evaluation of
options
Decisions
Advocate favored
option; debate
Select option;
opponents; seek
approval of publics
mentation;
reformulate
goab/values in
& mass audiences
plan imple-
Light of decision
Technical
experts
Develop technical
options for political
leaders Ac interest
groups
Estimate
consequences of
Appraise viable
options
various options
Assist in
planning
implementation
of selected
option
Interest
groups
Promote
goals/ values; seek
input from
constituent!
Develop political it
technical options;
seek input from
experts
Engage experts in
estimation of
consequences
Advocate favored
Reformulate
option; debate
opponents; seek
approval of publics
goals/values in
light of decision
& mass audiences
Reporters and
editors
Monitor activities
Monitor activities
Present political
Report decision
of above for
attentive publics
of above for
debate for publics
& mass audiences
& related
activities for
attentive publics
publics & mass
audiences
Attentive
publics
Provide tome input
Follow activities
Follow activities
to interest groups
6c political leaders
reported in press;
develop opinions
reported in press;
develop opinions
Follow debate in
press; advocate.
debate & register
Follow activities
reported in press
opinions on viable
options
Mass
audiences
Follow debate in
press; develop &
register opinion on
viable options
FIGURE
2 Illustration of one possible collective decision-making scenario
democracies) allow for more ongoing and substantial direct involvement in
decisions via policy referenda (or, less directly and in a less binding fashion,
via opinion polls; see Kelso, 1978).
For the sake of illustration, we can fill in the matrix to describe one such
possible decision-making scenario, as shown in Figure 2. Although the scenario
portrayed in Figure 2 strikes us as reasonably descriptive of processes commonly
at play in advanced Western democracies, we present it here solely for purposes
of discussion, with no claims to descriptive accuracy nor any intention of
promoting a particular model.
344
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
Note that in the process portrayed in Figure 2, different participants act as
'role players', entering and exiting the process with differentiated functions.
Technical experts, for example, mainly assist political leaders and interest groups
and are most active in the late-developmental stages of policy formation. Mass
audiences enter the process only at the evaluation phase, during which time
they follow active debate over a limited number of options favored by elites
and register their opinions as to which they prefer. More active citizens engage
in the process as members of attentive publics who follow elite activities between
such critical decision points as elections and may provide some limited input,
when invited, into the directions pursued by elected officials and interest-group
leaders (Almond, 1950). Empirical estimates in the USA would place this group
at perhaps 20 percent of the electorate (e.g. Devine, 1970; Neuman, 1986).
Members of the press serve as critical conduits for information and opinion
exchange between elites, attentive publics, and their much larger mass audiences
(Price, 1092).
The main value of viewing the process in componential terms, as displayed
in Figures 1 and 2, is that it helps to isolate particular aspects of the process
that might be judged problematic from a quality perspective. It also highlights
the range of options one might consider in reforming the process should it be
found faulty. For example, if it is indeed the case that members of the mass
audience have no engagement in the process until they are asked their opinions
in the evaluation phase (as portrayed in Figure 2), then it places heavy burdens
indeed upon the press to properly inform their audiences at this juncture. Even
then, the capacity for citizen judgment may be heavily circumscribed by the
fact that they have at their disposal little or no knowledge of alternatives that
were considered and rejected (or indeed not considered) by elites, nor perhaps
sound understanding of the consequences of various options, save what is injected
by political contestants into their persuasive appeals during the evaluation phase.
C O N C E P T U A L I Z I N G QUALITY CRITERIA
Judging the quality of public opinion, then, is inextricably bound to judging
the quality of democratic decision making. How, exactly, are such judgments
of quality to be made? In considering the many possible scenarios that might
be identified via Figure 1, what quality criteria are to be applied?
As decision theorists have acknowledged in their own efforts to examine and
improve human decision making through systematic study, there are at least
two dimensions relevant to judging the quality of a decision (e.g. Rohrmann,
1986). One can judge the quality of the decision making process employed, or
the quality of the decision outcome that is eventually attained. In principle, the
'total' quality of the process (see Figure 1) depends upon the quality of each
OPINION QUALITY IN PUBLIC OPINION RESEARCH
345
phase: one is unlikely to make a sound choice (at the final phase) without having
properly elicited the collective goals or values to be maximized (first phase),
without having developed a full range of options (second phase), or without
having properly examined the consequences (third phase), and so on. It is
possible, however, that one might find the process of collective decision making
quite good, and at the same time find the result of public judgment to have
been, in retrospect, a mistake. Some decision theorists have observed that
judging the quality of decision outcomes is thus problematic, because an
undesired outcome does not necessarily mean that one made a 'bad' choice.
Nor does a desired outcome necessarily indicate that a 'good' choice has been
made (Rohrmann, 1986; Edwards et al., 1984).
One can also apply quality considerations at both the individual level (e.g.
How well do people understand the options? Do they evaluate them rationally?
Do they hold views that are in their best interests?); as well as at the collective
level (e.g. How completely has a society examined its options? How rationally
is the collective choice determined? Is the choice in the collective interest?). As
some researchers have argued, it is possible for public opinion in its collective
form, for example as an aggregation of individual opinions, indeed for it to be
rational, stable, and a sound input to policy making, even if individual opinions
tend to be far less rational and stable (Page and Shapiro, 1992).
We can, then, evaluate the quality of public opinion along a variety of
dimensions, which we have attempted to catalogue in Table 1. The table lists
a number of different criteria that can be applied either to public opinion as an
outcome or as a decision-making process, and to either individual opinions or to
collective opinion. As with Figures 1 and 2, this table is offered for illustrative
purposes. It strikes us as fairly comprehensive and representative of concerns
that have surfaced in connection with writings on public opinion, but certainly
other, more complete, inventories are possible.
INDIVIDUAL, OUTCOME-ORIENTED QUALITY CRITERIA
Researchers of public opinion are probably most familiar with quality criteria
that have been applied to individual opinions. Among these, perhaps most
empirical attention has been given to the relative stability of opinions, or to
their consistency (Converse, 1964, 1970; Nie, Verba and Petrocik, 1976; Schuman
and Presser, 1981). The former refers to the degree to which people's opinions
remain consistent over time, whereas the latter refers to the organization of
opinions—the extent to which people's opinions are logically or ideologically
consistent with other views they hold and with their general values and attitudes.
Also of continuing concern in the field is the assessment of the conviction with
which opinions are held: How intensely a viewpoint is held, how important an
346
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
TABLE I
Possible quality criteria for evaluating democratic decision making
Quality criterion
Applied to
Individual level
Applied to
Collective level
Outcome-oriented criteria focusing on quality of opinions/decisions
Stability, lack of volatility
J
J
Consistency with values, other opinions
Conviction
Extensiveness of information base
Acceptance of consequences
Optimization of individual interests
Degree of consensus
Optimization of collective interests
Representation of collective desires
Actionability, clarity of path to action
Degree of implementation
J
J
J
J
J
J
y
J
/
/
J
J
/
y
y
Process-oriented criteria focussing on quality of opinion formation/decision making
Extensiveness of information search
J
/
Care of deliberation and analysis
J
J
Responsiveness to feelings and emotions
/
/
Extensiveness of discussion and debate
J
J
Freedom from censorship or control
J
J
Understanding of and respect for differing
/
/
viewpoints
Independence from social pressure
/
J
Enjoyment and satisfaction
/
/
Efficiency and cost effectiveness
/
/
Extensiveness of popular participation
J
Representativeness of participants
J
Generation of differing viewpoints/options
/
Degree of perceived legitimacy
/
issue is thought to be, or how confident a person is in his or her opinion
(Schuman and Presser, 1981; Krosnick, 1988a, 1988^). An even more routinely
applied quality criterion has to do with the amount of information that supports
an opinion (Lane and Sears, 1964; Bishop et al., 1980; Graber, 1982; Neuman
1986).
Other quality criteria can also be identified, even though they have received
less explicit attention in the public opinion literature. Although many have
argued that sound opinions should support policies in the collective interest, it
is nonetheless common for opinions to be judged faulty if they are understood
as being not in a individual's own best interests. For example, when members
of the working classes espouse views in favor of big business or free enterprise,
OPINION QUALITY IN PUBLIC OPINION RESEARCH
347
it is sometimes seen as evidence of domination. Recently, Yankelovich (1991)
has argued that a central quality of sound public opinion is the capacity to
understand and accept the consequences of any policy option that is endorsed.
It is fair to say that empirical researchers have given contemporary mass
opinion, at least in the USA, generally poor marks on many of these central
quality criteria: opinions are often very unstable, disorganized and inconsistent,
weakly held, and unbolstered by much relevant knowledge (e.g. Neuman, 1986).
But there are defenders of mass opinion as well. Yankelovich (1991), for example,
has questioned the commonly held information criterion, arguing that masses
can contribute sound judgment without much background knowledge, so long
as their opinions are stable, consistent with deeply held values, and if people
accept the consequences of their decisions. Others have argued that although
Americans may generally evidence a lack of stability, consistency, conviction,
and knowledge on many opinions, most American citizens hold a small number
of issues to be personally important, attend carefully to news about those
particular issues, and use that store of information to form stable and strongly
held opinions (Pomper, 1980; Krosnick, Berent and Boninger, 1994).
COLLECTIVE, OUTCOME-ORIENTED QUALITY CRITERIA
The same criteria applied to individual opinions—stability, consistency, conviction, and the like—can also be applied to collective opinion as well. Recently,
a number of analysts have investigated aggregate opinion and found it far more
stable and interpretable than individual opinions (e.g. Page and Shapiro, 190.2;
Smith, 1994)1 Of collective opinion outcomes, perhaps consensus is most often
held out as the principle goal. After all, it is the search for the 'common good'
that motivates so many theorists and practitioners of democratic decision making.
But even if consensus cannot be achieved, the goals are to optimize collective
interests through compromise and to attain the most accurate possible representation of collective desires.
We might also add to the list of desirable outcomes the clarity of the signal
it provides to those who must act on behalf of the collective. Registering a
strongly consensual opinion in support for reducing crime (which might be
appropriate at the elicitation-of-goals phase) is not nearly so useful as registering
support for restricting access to firearms, for constructing more prisons, or
spending a larger share of public funds on police activities (at the evaluation
phase). Finally, the ultimate quality of the outcome has to do with the degree
1
It is important to bear in mind that the conceptual meaning of some attributes—such as conviction—can
shift in important ways when one moves from the individual to the collective level. As Lazarsfeld and Menzel
(1961) suggested, a collectivity made up of members who hold different views and are individually very
certain of their opinions may well be, at the collective level, quite uncertain of how to proceed.
348
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
of implementation. Public opinion exerts its power only when elites managing
the social system are responsive and pay it heed (Key, 1961). In the early
1980s, for example, the Netherlands undertook a dramatic effort to invite an
unprecedented degree of popular participation in determining its future course
in national energy policy, by launching what came to be known as the 'General
Social Debate' (see Neijens, 1987). Although this effort may have been in many
respects close to an ideal democratic forum for decision making, and the
population was apparently able to register a rather thoughtful and clear choice
among alternative policy options, for a variety of political reasons national
government leaders did not ultimately act in accordance with the popular choice.
INDIVIDUAL, PROCESS-ORIENTED QUALITY CRITERIA
Whereas outcome-oriented criteria focus on the end-state achieved by an
individual or collectivity, process-oriented criteria focus on the means employed.
Behind much of the research on opinion outcomes lies concern about process.
For example, the lack of knowledge exhibited by respondents to mass opinion
surveys is commonly accepted as evidence of a weak information search process?
Typically, most analysts hold out care of deliberation and analysis as an important
process, in that opinion formation should be thoughtful and rational.
On the other hand, some have argued that such expectations for rational
deliberation are overblown, and that it is more important that opinions properly
express sentiment, or people's feelings and emotions (e.g. Yankelovich, 1991).
For many who have pondered public opinion, the key process is not so much
dispassionate private deliberation but active public engagement (e.g. Dewey,
1927; Yankelovich, 1991; Fishkin, 1995). There is a long democratic tradition,
as noted earlier, of seeking conditions that facilitate widespread discussion and
debate, under conditions that render that debate free of censorship or control, and
in a civil environment fostering respect for and understanding of differing viewpoints
(Habermas, 1989). In the end, the democratic process should instill a strong
sense of community.
Although social engagement is thus celebrated as central to proper opinion
formation, it also brings with it a variety of potential downsides in the form of
untoward social pressures—tendencies toward a tyranny of the majority (e.g.
Tocqueville, 1945), distortions due to social desirability or propriety (NoelleNeumann, 1984), susceptibility to demagoguery (Lippmann, 1925), and perhaps
1
In this connection, » e might note in passing that there is at least the possibility that people do mount
more extensive searches of news and information thin their performance on knowledge tests would suggest
This could be so, for instance, if people generally form and 'update' their opinions as they encounter new
information, but discard large amounts of the supporting information and retain only their final opinion (as
'on-line' models of political information procession posit; e.g. Lodge, McGraw and Stroh, 1989).
OPINION QUALITY IN PUBLIC OPINION RESEARCH
349
dangerous forms of'group think' (Janis, 1972). Thus, the process should ideally
be independent of social pressure, reflected in widespread use of secret ballots in
elections, creation of republican legislatures and executive offices insulated from
too much popular pressure (e.g. Madison, 1966), as well as in routine efforts
to combat interviewer and sponsor 'demand' effects in surveys and polls.
Finally, we can identify a few less commonly applied but sensible processoriented criteria, including the extent to which the process is enjoyable and
satisfying for participants. Central criteria in decision analysis are the efficiency
and cost-ejfectiveness of the decision-making process. A number of defenders of
mass opinion have argued that the same characteristics viewed as defects by
some analysts—lack of thorough information search, or of sufficient care and
deliberation, investment in discussion and debate, etc.—may well be indicative
of rational cost/benefit trade-offs on the part of ordinary citizens (e.g. Sniderman,
Brody and Tetlock, 1991; Popkin, 1994).
COLLECTIVE, PROCESS-ORIENTED QUALITY CRITERIA
Again, the same criteria applied to individual opinion formation processes—
extensiveness of information search, care of deliberation, engagement in discussion, freedom from control and from untoward social pressure, and the
like—can also be applied to collective decision making as well. And again, it is
conceivable that, even if many or most individuals fail to measure up to
normative standards, the collectivity as a whole could be judged in a more
favorable light. For example, even if most people maintain singular and narrow
perspectives, talk infrequently, and contribute relatively little information to
the process, in a heterogeneous population the collective fund of information,
discussion, and perspective might still be expansive. Certainly efficiency criteria
are sometimes applied to collective decision making. One common complaint
against strong democratic models is their reputed inefficiency, particularly in
relation to the time required for generating sufficient consensus. At lower levels
of analysis, one finds this complaint directed toward 'rule by committee', as
committees are seen—by virtue of their democratic functioning—to be inefficient, indecisive, and unrewardingly time consuming. Recent criticisms of
the Deliberative Poll have centered, in part, on the argument that the method
offers relatively little by way of popular input that conventional polls cannot
provide at a much lower cost (Kohut, 1996; Merkle, 1997).
Some criteria apply only to the collective decision-making process. Two
related but not identical concerns are that participation in the process be
widespread and that participants be representative of the affected population.
Referenda and polls may significantly advance the latter cause but not necessarily
the former. Like other trends toward more direct democracy, polls help bring
35°
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
power to the people, but they may be unlikely to bring thoughtful or deliberative
power to the people (Fishkin, 1991). Finally, the process must be perceived to
be legitimate by participants—not only by members of the mass audience who
must live with the consequences of collective choices, but also, as noted above,
by elites who are charged to act on behalf of the whole.
T H E CHALLENGE OF M E E T I N G QUALITY CRITERIA
Having thus far presented a schematic conception of democratic decision making
and enumerated a series of quality criteria, we turn in this third and final
section to various operational attempts to realize quality in public opinion and
collective decision making. A wide variety of methods have been developed to
assist society in behaving democratically, each of which can be viewed as
attempts to operationalize one or another sought-after quality in collective
decision making. Some of these—including public opinion polls—are now
considered time-honored parts of the machinery of governance in many societies.
Other techniques, like deliberative polls, public journalism, and surveys of
informed public opinion, represent more 'experimental' efforts to improve either
the processes or the outcomes of collective decision making.
CONTRADICTIONS AND AMBIGUITIES
Before discussing these efforts to put quality criteria into practice, however, we
should take note of two very important points in connection with our list of
quality criteria. First, as has been noted at various junctures, a number of the
criteria identified in Table 1 are contradictory. On the one hand, analysts
commonly elevate the rational and deliberative aspects of the decision-making
process, and express a desire for opinions and choices that are grounded in
empirical support (that is, on the best information available). On the other
hand, a variety of non-rational factors and processes—feelings and emotions,
even 'gut reactions'—are also thought to be indispensable. Yankelovich (1991),
for example, has castigated political science for its insistence on rational choice,
arguing that it is symptomatic of a broader 'culture of technology'. What is
needed is not knowledge but wisdom, Yankelovich argues, not rationally calculated
decisions but rather public judgment, formed largely out of popular sentiment
and deeply held values (see also Edelman, 1995; Boyte, 1995). The notion that
the masses rightly contribute sentiment rather than well-founded ideas about
policy is at least a century-old proposition (see Bryce, 1988, pp. 7-8). Another
fundamental contradiction turns on ambivalent attitudes toward social influence.
On the one hand, social interaction and communication are elevated as defining
features of sound public opinion (e.g. Blumer, 1948; Habermas, 1989). On the
OPINION QUALITY IN PUBLIC OPINION RESEARCH
351
other hand, the literature is replete with recurring fears of untoward social
influence, concerns about political persuasion processes that might subvert
popular opinion, and arguments that participants in collective decision making
need to be insulated from social pressure (witness, for example, the secret
ballot).
Second, there is a large degree of ambiguity surrounding the application of
criteria to different phases or different participants in the decision-making
process (see Figure 1). At what particular phase or phases of the process is a
particular quality desired? Care and deliberation might be much more essential
in the development of options and the estimation of consequences than in the
elicitation of values or the evaluation of policy options, where other qualities—
such as consistency with values or responsiveness to feelings and emotions—
might be of central concern. Similarly, one might well imagine the application
of different quality criteria to different participants. High standards of care and
deliberation might be far more applicable to political leaders and technical
experts than to members of attentive publics or mass audiences, as critiques of
the 'omni-competent citizen' suggest (see Fishkin, 1995).
TECHNIQUES FOR ATTAINING QUALITY IN PUBLIC DECISION MAKING
Traditional methods. Our present interest centers on polling and opinion survey
techniques, but these are clearly not the only, nor the most important, technologies developed to assist society in democratic decision making. Election of
representatives to legislatures, for example, can be properly understood as
a decision-making technology designed to optimize collective interests, the
representation of collective desires, popular representativeness, and perceived
legitimacy. Balancing legislative power against judicial and executive powers is
aimed at increasing stability, maximizing freedom from control and independence
from social pressure, and ensuring care of deliberation and analysis. Referenda
have many of the same goals as legislative mechanisms but extend even further
the degree of popular participation in policy making. Many of the traditional
functions and activities of journalism—surveillance of political activities, publication of editorial fora, gathering of man-on-the-street interviews—have been
introduced to maximize the information base of attentive publics and mass
audiences, to foster the generation of differing viewpoints, and to extend the
sphere of active decision making to the masses. Guarantees of freedom for the
press and assurances of broad public access to the media aim at maximizing
the free exchange of information, popular freedom from elite control, and
popular representativeness. Town meetings seek to extend and encourage
widespread debate and foster respect for differing viewpoints; and guarantees
of freedom of assembly aim at insuring that such events can occur free of
352
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
censorship or control. Political parties and interest groups exist to represent
mass or other constituencies, to help in organizing their values and opinions,
develop policies that constitute a clear path to action, and to convey those
desires upward to decision makers. More broadly, reforms in public education
have been advanced as means of developing a citizenry that is capable and
sufficiently motivated to participate in public decision making (e.g. Dewey,
1916, 1927).
Each of these institutions and practices represents an attempt to improve a
particular facet of 'quality' at a particular stage of the collective decision-making
process. Opinion polls are no exception in this regard. Scientific opinion survey
methods were developed in part to serve as tools for both basic and applied
research. But polls were also implemented to advance a particular aspect of
quality—broad participation—at a particular phase of the decision-making
process—the evaluation stage. They could serve as a popular check against the
influence of organized interest groups and other elite political actors, who were
only too willing to claim broad public support, even in its absence, for their
favored policies (Gallup and Rae, 1940). Opinion surveys were thus viewed,
and continued to be seen, as a technological means for advancing quality in
collective decision making.
Newer extensions. Many more recent trends in the conduct of democratic
decision making are in one way or another extensions of these more traditional
technologies. Randomly selected 'shadow parliaments' have been proposed to
further increase the representativeness of deliberative, legislative bodies. The
'public journalism' movement aims to foster popular discussion and more
active political participation to contemporary public affairs. Rosen (1991) has
characterized this effort as a shift of emphasis from a journalism in the service
of 'information' to a new journalism in the service of 'conversation' (p. 268).
Following Carey (1987), he argues that journalists should spend less time trying
to 'transport' information to passive audiences and more time engaging readers
to become active participants in public debate. Talk shows, electronic 'town
meetings' and computer talk groups maximize the enjoyment of the mass
audience, as well as foster popular discussion. Each of these, as with traditional
methods of democratic decision making, suffer weaknesses as well as strengths.
As representative as 'shadow parliaments' may be, they may not improve and
could conceivably weaken the care and deliberation of legislative policy formation
and evaluation; some journalists worry about the advocacy role of 'public
journalism' leading to problematic biases in news reporting; talk shows and
electronic town meetings may encourage discussion and increase responsiveness
to feelings and emotions, but they tend in practice to be far from representative.
OPINION QUALITY IN PUBLIC OPINION RESEARCH
353
Newer trends in polls and surveys. Many traditional institutions of collective
decision making are thus undergoing transformations and challenges, and this
is true of polls as well. In their discussion of the public journalism movement,
Salmon and Glasser (1995, p. 453) identify opinion polls as one prominent
example of the 'journalism of information', as activities that, notwithstanding
their stated democratic intentions, end up packaging faulty and unreflective
opinion as a product for mass consumption, ultimately hindering effective
popular deliberation. As noted at the outset, critiques of mass opinion surveys
have long been a staple of the public opinion literature (e.g. Blumer, 1948;
Mills, 1956; Ginsberg, 1986; Crespi, 1989; Fishkin, 1991; Herbst, 1993). Various
new polling and survey methods have now been developed in attempts to
remedy the perceived deficiencies of conventional opinion polls. These include
'deliberative' polls, 'educational' polls, polls of 'informed' public opinion, and
variants of focused group discussion, each advanced as alternatives or supplements to mass opinion surveys.
Since 1987, for example, the Americans Talk Issues Foundation (ATIF) has
conducted a series of 'educational' public interest surveys designed and conducted by small, politically balanced teams of issue experts and public opinion
researchers (Kay et al., 1994). Their announced goal is consensus location: 'a
search for the most widely supported proposals by testing various features of
the proposals separately and in combination' (Kay et al., 1994). The ATIF
surveys are educational in the sense that, in addition to posing opinion questions,
they also expose respondents to information: pro and con arguments, including
cost, benefits, and probable consequences of various policy proposals. Referring
back to Table 1, we can see that the ATIF surveys are focused on a variety of
desired outcomes: consensus (a collective outcome), the extensiveness of the
information base (an individual or collective outcome), the acceptance of consequences (an individual or collective outcome), as well as the optimization of
collective interests and the representation of collective desires (collective outcomes).
In that they are intended to assist policy makers, they are also aiming to improve
the clarity of the path to action (a collective outcome). In terms of processes,
the ATEF efforts aim at improving the extensiveness of the information search, at
least on the part of political leaders during the development of proposals (see
Figure 1), and the extensiveness of mass participation in the proposal development
process. That is, whereas opinion polls and surveys often assess popular response
to a narrow range of proposals favored by elites during the evaluation phase of
decision making, the ATIF program seeks ways of injecting mass input during
the earlier, developmental phase when policy options are being formulated (See
Figure 1).
More elaborate yet are the procedures employed in the Deliberative Poll
(Fishkin, 1991, 1995). 'An ordinary poll', according to Fishkin, 'models what
354
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
the electorate thinks, given how little it knows. A deliberative opinion poll
models what the electorate would think if, hypothetically, it could be immersed
in intense deliberative processes (1991, p. 81; italics in original). The basic idea
involves selecting a national, random sample of the voting-age population and
transporting them as 'delegates' to a single site for several days of debate and
deliberation. After debating issues with political leaders and with each other,
the delegates are then polled on their preferences. The exercise would combine
the thoughtfulness and depth of face-to-face politics with the representative
character of a national event (Fishkin, 1991, p. 9). Referring back to Table 1,
we can see that the Deliberative Poll is focused on most of the same goals
driving the ATIF surveys, but adds to these the goals of advancing discussion
and debate as both individual and collective processes, and to create a communicative forum that is fully representative of the population at large. Like the
ATIF surveys but in much more dramatic fashion, the Deliberative Poll also
aims at improving the extensiveness of the information search—and not just among
political leaders but among popular participants as well (see Figure 1). It seeks
to increase mass participation in the proposal development process, by creating
an event that would allow ordinary citizens to penetrate the 'hocus pocus' and
'amplification' of press-mediated political debate, in an environment that would
allow them to engage with candidates and issues free of control by 'spin doctors'
(Fishkin, 1991, p. 8). Also like the ATIF program, the Deliberative Poll seeks
ways of promoting mass input when policy options are being formulated. The
ambitious aim is to create a new institution that can advance participatory
democracy across all phases of the collective decision-making process. According
to Fishkin, the goal is nothing less than 'direct democracy', carried out by 'a
statistical microcosm of the society' that is empowered to deliberate for the
whole (1991, p. 93).
In a companion piece to this paper (Price and Neijens, 1997), we examine
in detail several prominent examples of efforts to measure more 'informed'
public opinion (including the ATIF surveys and Fishkin's Deliberative Poll).
Here we should simply note that they follow historically in a long line of
previous efforts by pollsters and survey researchers to counter the tendency of
mass opinion surveys to collect and disseminate opinions that may be ill-informed
'non-attitudes' or 'pseudo-opinions'. In fact, survey researchers and pollsters
have long attempted in a number of ways to 'qualify' their measures of opinion,
albeit in less dramatic fashion. Examples include the use of 'don't know'
response options, probes and question filters designed to reduce manufactured
opinions, or the addition of measures of opinion intensity to accompany
directional measures of opinion (e.g. Schuman and Presser, 1981; Converse and
Presser, 1986). More distinctive efforts can be found in, for example,
the application of 'likely voter' models to qualify poll results in
OPINION QUALITY IN PUBLIC OPINION RESEARCH
355
election contexts (e.g. Traugott and Tucker, 1984) or the use of a 'mushiness
index' that attempts to assess the relative stability, consistency, and thoughtfulness of mass opinion data (Yankelovich, 1991). Modifications of survey and
polling methods thus range from these sorts of attempts at filtering standard
opinion data (some of which have now become widely accepted as good
practice), to more elaborate attempts to inform or educate respondents using
conventional survey formats, to even more ambitious attempts both to inform
respondents and to engage them in lengthy programs of active deliberation
and discussion.
The contradictions and ambiguities surrounding quality criteria in opinion
research are bound to color debates over the value of any experimental procedures
aimed at measuring 'higher quality' public opinion. We propose here no immediate
solution to this problem. But a logical first step is to be as straightforward as
possible in first identifying quality objectives—a task to which we hope to have
contributed with this analysis—and then measuring various specific opinionquality concerns. Only in this way will different techniques aimed at improving
quality, for example by gathering more deliberative or informed public opinion,
be rendered subject to scientific evaluation. After taking note of the specific
qualities of the decision-making process each method seeks to improve, the
quality criteria developed here can be used to evaluate in detail the various
specific methodologies. Attempts to gather more deliberative measures of public
opinion will also need to be evaluated in terms of quality criteria that apply to
different phases and/or participants in the democratic decision-making process.
As we note in our companion review of various deliberative polling methods,
new techniques often attempt to maximize several distinct qualities at once,
making it difficult to identify specific objectives for assessing success. Little
is to be gained by arguing that a certain combination of decision-making
methods or polling technologies are simply, of themselves, intrinsically better.
First, one must articulate a theory that a particular aspect of method (of
organized debate, of questionnaire design, etc.) will produce improvements
in some particular aspect of quality. Public opinion research, through careful
experimentation and empirical analysis, may thus eventually help to resolve
debates. This hope will only be realized, however, once specific qualities
of alternative decision-making techniques have been clearly identified and
examined in relation to clearly defined and measured qualities in opinion
processes or outcomes.
ACKNOWLEDGEMENTS
Portions of this manuscript were presented at the World Association for Public
Opinion Research thematic seminar on Quality Criteria in Survey Research,
356
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
Cadenabbia, Italy, June 1996. Work on this project was supported in part by
the Marsh Center for the Study of Journalistic Performance in the Department
of Communication Studies at the University of Michigan.
REFERENCES
Allport, F. H. (1937): 'Toward a science of public opinion', Public Opinion Quarterly,
1,
7-23-
Almond, G. (1950): The Ameruan People and Foreign Policy, New York, Harcourt.
Berelson, B. (1950): 'Democratic theory and public opinion', Public Opinion Quarterly,
16, 313-30.
Bishop, G. F., Oldendick, R. W., Tuchfarber, A. J. and Bennett, S. E. (1980):
'Pseudo-opinions on public affairs', Public Opinion Quarterly, 44, 198-209.
Blumer, H. (1946): 'Collective behavior'. In A. M. Lee (ed.) New Outlines of the
Principles of Sociology, New York, Barnes and Noble, pp. 167-222.
Blumer, H. (1948): 'Public opinion and public opinion polling', American Sociological
Review, 13, 542-54.
Boyte, H. C. (1995): 'Public opinion as public judgment'. In T. L. Glasser and C. T.
Salmon, (eds.) Public Opinion and the Communication of Consent, New York, Guilford,
PP- 4I7-436Bryce, J. (1888): The American Commonwealth, Vol. 3, London, Macmillan.
Carey, J. W. (1987): 'The press and public discourse', The Center Magazine, March/April,
4-16.
Converse, J. M. and Presser, S. (1986): Survey Questions: Handcrafting the Standardized
Questionnaire, Sage University Papers series on Quantitative Applications in the
Social Sciences, series no. 07-063 Newbury Park, CA, Sage.
Converse, P. E. (1964): 'The nature of belief systems in mass publics'. In D. E. Apter
(ed.) Ideology and Discontent, New York, Free Press, pp. 206-261.
Converse, P. E. (1970): 'Attitudes and non-attitudes: Continuation of a dialogue'. In
E. R. Tufte (ed.) The Quantitative Analysis of Social Problems, Reading, MA,
Addison-Wesley, pp. 168—89.
Converse, P. E. (1987): 'Changing conceptions of public opinion in the political process',
Public Opinion Quarterly, 51, S12-S24.
Crespi, I. (1989): Public Opinion, Polls, and Democracy, Boulder, CO, Westview Press.
Dahl, R. A. (1971): Polyarchy: Participation and Opposition. New Haven, Yale University
Press.
Dahl, R. A. (1985): A Preface to Economic Democracy, Berkeley, CA, University of
California Press.
Davison, W. P. (1958): 'The public opinion process', Public Opinion Quarterly, 22,
91—106.
Devine, D. J. (1970): The Attentive Public: Polyarchical Democracy, Chicago, Rand
McNally.
OPINION QUALITY IN PUBLIC OPINION RESEARCH
357
Dewey, J. (1916): Democracy and Education: An Introduction to the Philosophy ofEducation,
New York, Macmillan.
Dewey, J. (1927): The Public and its Problems, New York, Holt, Rinehart & Winston.
Edelman, M. (1995): 'The influence of rationality claims on public opinion and policy'.
In T. L. Glasser and C. T. Salmon (eds.) Public Opinion and the Communication of
Consent, New York, Guilford, pp. 403-416.
Edwards, W., Kiss, I., Mojore, G. and Toda, M. (1984): Panel discussion on 'What
constitutes "a good decision"?' (with an introduction and postscript by C. Vlek) Acta
Psychobgica, 56, 5-27.
Entman, R. M. (1989): Democracy Without Citizens: Media and the Decay of American
Politics, Oxford, Oxford University Press.
Fishkin, J. S. (1991): Democracy and Deliberation: New Directions for Democratic Reform,
New Haven, Yale University Press.
Fishkin, J. S. (1995): The Voice of the People: Public Opinion and Democracy, New
Haven, Yale University Press.
Foote, N. N. and Hart, C. W. (1953): 'Public opinion and collective behavior'. In M.
Sherif and M. O. Wilson (eds.) Group Relations at the Crossroads, New York, Harper
& Bros, pp. 308-331.
Gallup, G. and Rae, S. (1940): The pulse of democracy, New York, Simon & Schuster.
Ginsberg, B. (1986): The Captive Public: How Mass Opinion Promotes State Power, New
York, Basic Books.
Graber, D. A. (1982): 'The impact of media research on public opinion studies'. In D.
C. Whitney, E. Wartella, and S. Windahl (eds.) Mass Communication Review Yearbook
(Vol. 3) Newbury Park, CA, Sage, pp. 555-64.
Habermas, J. (1982): 'A reply to my critics'. In J. B. Thompson and D. Held (eds.)
Habermas: Critical Debates, Cambridge, MA, MIT Press, pp. 218—83.
Habermas, J. (1989): The Structural Transformation of the Public Sphere: An Inquiry into
a Category of Bourgeois Society (T. Burger, trans.) Cambridge, MA, MIT Press.
(Original work published 1962).
Held, D. (1987): Models of Democracy, Cambridge, Polity Press.
Herbst, S. (1993): Numbered voices: How opinion polling has shaped American politics.
Chicago, University of Chicago Press.
Humphreys, P. (1983): 'Decision aids: Aiding decisions'. In L. SjSberg, T. Tyszka, and
J. A. Wise (eds.) Decision Analyses and Decision Processes 1, Lund, Doxa.
Janis, I. L. (1972): Victims of Groupthink: A Psychological Study of Foreign-Policy
Decisions and Fiascos, Boston, Houghton Mifflin.
Jungerman, H. (1980): 'Speculations about decision-thoretic aids for personal decision
making', Acta Psychobgica, 45, 7-34.
Kaplan, A. (1964): The Conduct of Inquiry, Scranton, PA, Chandler.
Kay, A. F., Henderson, H., Steeper, F. and Lake, C. (1994): Interviews with the Public
Guide us ... On the Road to Consensus, St. Augustine, Florida, Americans Talk Issues
Foundation.
Keeney, R. L. and Raiffa, H. (1976): Decisions with Multiple Objectives: Preferences and
value tradeoffs, New York, Wiley.
358
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
Kelso, W. A. (1978): American Democratic Theory: Pluralism and its Critics, Westport,
CT, Greenwood Press.
Key, V. O., Jr. (1961): Public Opinion and American Democracy, New York, Knopf.
Kingdon, J. W. (1984): Agendas, Alternatives, and Public Policies. Boston, Little, Brown
&Co.
Kohut, A. (1996): 'The big poll that didn't', Poll Watch, 4, 2-3.
Krosnick, J. A. (1988a): 'Attitude importance and attitude change', Journal of Experimental Social Psychology, 24, 240-55.
Krosnick, J. A. (1988^): 'The role of attitude importance in social evaluation: A study
of policy preferences, presidential candidate evaluations, and voting behavior',^oar»a/
of Personality and Social Psychology, 55, 196-210.
Krosnick, J. A., Berent, M. K. and Boninger, D. S. (1994): 'Pockets of responsibility
in the American electorate: Findings of a research program on attitude importance',
Political Communication, n , 391-412.
Kuypers, G. (1980): Beginselen van beleidsontmkkeling (delen A en B), Muiderberg,
Coutinho.
Lane, R. E. and Sears, D. O. (1064): Public Opinion, Englewood Cliffs, NJ, Prentice-Hall.
Lasswell, H. D. (1941): Democracy Through Public Opinion, Menasha, WI, George
Banta.
Lazarsfeld, P. F. (1957): 'Public opinion and the classical tradition', Public Opinion
Quarterly, 21, 39-53.
Lazarsfeld, P. F. and Menzel, H. (1961): 'On the relations between individual and
collective properties'. In A. Etzioni (ed.) Complex Organizations: a Sociological Reader,
New York, Holt, Rinehart & Winston, pp. 422-450.
LJppmann, W. (1922): Public Opinion, New York, Harcourt Brace Jovanovich.
Lippmann, W. (1925): The Phantom Public, New York, Harcourt Brace Jovanovich.
Lodge, M., McGraw, K. and Stroh, P. (1989): 'An impression-driven model of candidate
formation', American Political Science Review, 83, 399-420.
Madison, J. (1966): The Federalist Papers, New York, Doubleday. (Original work
published 1788).
March, J. G. (1994): A Primer on Decision Making: How Decisions Happen, New York,
Free Press.
March, J. G. (1995): Democratic Governance, New York, Free Press.
Merkle, D. M. (1997): 'Polls-review: The National Issues Convention Deliberative
Poll', Public Opinion Quarterly, 60, 588-619.
Mills, C. W. (1956): The Power Elite, Oxford, Oxford University Press.
Neijens, P. (1987): The Choice Questionnaire. Design and Evaluation of an Instrument for
Collecting Informed Opinions of a Population, Amsterdam, Free University Press.
Neijens, P., Ridder, J. A. de, and Saris, W. E. (1992): 'An instrument for collecting
informed opinions', Quality and Quantity, 26, 245-58.
Neuman, W. R. (1986): The Paradox of Mass Politics: Knowledge and Opinion in the
American Electorate, Cambridge, MA, Harvard University Press.
Nie, N. H., Verba, S. and Petrocik, J. R. (1976): The Changing American Voter,
Cambridge, MA, Harvard University Press.
OPINION QUALITY IN PUBLIC OPINION RESEARCH
359
Noelle-Neumann, E. (1984): The Spiral Of Silence: Public Opinion—Our Social Skin,
Chicago, University of Chicago Press. (Original work published 1980).
Page, B. I. and Shapiro, R. Y. (1992): The Rational Public: Fifty Yean of Trends in
Americans' Policy Preference, Chicago, University of Chicago Press.
Park, R. E. (1972): The Crowd and the Public and Other Essays (H. Eisner, Jr., ed.; C.
Eisner, trans.), Chicago, University of Chicago Press. (Original work published 1904).
Parkin, A. (1995): Forum: 'In defence of critical perspectives: Remarks on Herbst,
Peer, Goodnight, and StapeP, International Journal of Public Opinion Research, 7,
39-1-4Pateman, C. (1970): Participation and Democratic Theory, London, Cambridge University
Press.
Pomper, G. M. (1980): Elections in America: Control and Influence in Democratic Politics,
New York, Longman.
Popkin, S. L. (1994): The Reasoning Voter: Communication and Persuasion in Presidential
Campaigns, Chicago, University of Chicago.
Price, V. (1992): Public Opinion, Newbury Park, CA, Sage.
Price, V. and Neijens, P. (1997): 'Deliberative polls: toward improved measures of
"informed" public opinion?' Unpublished manuscript. Institute for Social Research,
Ann Arbor, MI.
Rohrmann, B. (1986): 'Evaluating the usefulness of decision aids: A methodological
perspective'. In B. Brehmer, H. Jungermann, P. Lourens, and G. Sevon (eds.) New
Directions in Research on Decision Making, Amsterdam, North-Holland, pp. 363—81.
Rosen, J. (1991): 'Making journalism more public', Communication, 12, 267—84.
Salmon, C. T. and Glasser, T. L. (1995): 'The politics of polling and the limits of consent'.
In T. L. Glasser and C. T. Salmon (eds.) Public Opinion and the Communication of
Consent, New York, Guilford.
Schuman, H. and Presser, S. (1981): Questions and Answers in Attitude Surveys:
Experiments on Question Form, Wording, and Context, New York, Academic Press.
Slovic, P., Fischhoff, B. and Lichtenstein, S. (1977): 'Behavioral decision theory',
Annual Review of Psychology, 28, 1-39.
Smith, T. W. (1994): 'Is there real opinion change? International Journal of Public
Opinion Research, 6, 187-203.
Sniderman, P. M., Brody, R. A. and Tetlock, P. E. (1991): Reasoning and Choice:
Explorations in Political Psychology, Cambridge, Cambridge University Press.
Stokey, E. and Zeckhauser, R. (1978): A Primer for Policy Analysis, New York, Norton.
Tocqueville, A. de (1945): Democracy in America (2 Vols.), New York, Knopf. (Original
work published 1835, 1840).
Traugott, M. W. (1992): 'The "deliberative poll" is a well-intended but flawed idea',
The Public Perspective, January/February, 27-29.
Traugott, M. W. and Tucker, C. (1984): 'Strategies for predicting whether a citizen
will vote and estimation of electoral outcomes', Public Opinion Quarterly, 48, 330-43.
Yankelovich, D. (1991): Coming to Public Judgment: Making Democracy Work in a
Complex World, Syracuse, NY, Syracuse University Press.
Zaller, J. R. (1992): The Nature and Origins Of Mass Opinion, Cambridge, Cambridge
University Press.
360
INTERNATIONAL JOURNAL OF PUBLIC OPINION RESEARCH
BIOGRAPHICAL NOTE
Vincent Price is chair of the Department of Communication Studies and faculty
associate at the Center for Political Studies in the Institute for Social Research
at the University of Michigan. He is the author of Public Opinion (Sage
Publications, 1992) and other articles and essays on mass communication, public
opinion, and persuasion.
Peter Neijens is associate professor at the Amsterdam School of Communications
Research (University of Amsterdam). He is also director of the Foundation for
Fundamental Research on Commercial Communication and deputy director of
the Dutch Press Foundation. Both foundations are affiliated with the Department
of Communication at the University of Amsterdam.
Correspondence should be addressed to Vincent Price, Department of Communication Studies and Institute for Social Research, University of Michigan,
2020 Frieze Building, 105 South State Street, Ann Arbor, MI 48106; E-mail:
[email protected] and Peter Neijens, Department of Communication and
the Amsterdam School of Communications Research, University of Amsterdam,
Oude Hoogstraat 24, 1012 CE Amsterdam, The Netherlands; E-mail:
[email protected].