PARTICIPATORY METHODS TOOLKIT
A practitioner’s manual
Deliberative Polling®
Colophon
Participatory Methods Toolkit. A practitioner’s manual
Method: Deliberative Polling®
This is an extract of the publication ‘Participatory Methods Toolkit. A practitioner’s manual’, a joint publication of the
King Baudouin Foundation and the Flemish Institute for Science and Technology Assessment (viWTA).
The full version of the manual includes:
-
Introduction about participatory methods
-
General guidelines and tips for participatory methods
-
Complete description of 13 participatory methods: 21st Century Town Meeting® ; Charrette ; Citizens Jury ; Consensus
Conference ; Deliberative Polling® ; Delphi ; Expert Panel ; Focus Group ; Participatory Assessment, Monitoring and
Evaluation ; Planning Cell ; Scenario Building Exercise ; Technology Festival ; The World Café
-
Brief descriptions of 50 methods and techniques
All publications are available free of charge, online via www.kbs-frb.be or www.viWTA.be.
Editors
Stef Steyaert (viWTA) and Hervé Lisoir (King Baudouin Foundation)
Author
Janice Elliott (Public Policy Forum, Canada)
Layout and printing
Belgian Advertising (B.AD)
Legal depot
D/2005/2893/19
ISBN
90-5130-506-0
With the support of the Belgian National Lottery
September 2005
Deliberative Polling®1
Deliberative Polling® was developed by Jim Fishkin and Bob Luskin at the Centre for Deliberative Polling at the University of
Texas in 1988. The method is available to all, but the presence and authorization of the owners is required for all events which
make explicit mention of the name of their method. In such cases this implies that all costs linked to their presence must be
covered and that the owners shall be entitled to charge a fee.
I. DEFINITION
Deliberative Polling® was developed to address two challenges in modern democracies: how to get both a representative and
an informed (deliberative) view of what the public thinks and feels about an important public issue to help guide public
decision-making. Conventional polling has a significant influence on policy makers but these polls often represent only
the public's surface impressions of sound bites and headlines. Deliberative Polling is an attempt to use television and
reveals the conclusions the public would reach if they had the opportunity to become more informed and more engaged
in public policy issues.
A Deliberative Poll® combines the characteristics of representation and deliberation by undertaking:
1.
a survey of a random representative sample of the public;
2. an informed deliberation among a random representative cohort drawn from the survey sample, and
3. a post-deliberation survey to enable an analysis of changes in the knowledge and attitudes of participants in the
Deliberative Poll.
Deliberative polls seek to address the limitations of ‘top-of–mind’ opinion surveys by giving participants the time and
support they need to learn about an issue and to move from raw uninformed opinion to more considered judgment
(Fishkin and Rosell, 2004). The creation of an informed deliberation is enabled by:
1.
Providing the opportunity to have meaningful conversations with other members of the public in a safe and
neutral space2;
2. Balanced (unbiased) information that captures the views of experts from all sides of the issue;
3. A structured methodology for working through the information with other members of the public.
This process allows the public to become more informed about an issue and to confront the critical trade-offs and
consequences that are inherent within many key public policy debates and choices. The statistically significant changes
in opinion that have been noted at the conclusion of Deliberative Polls (in comparing the pre- and post-survey results)
are representative of the broader shifts in opinion that would occur in the public at-large were they to have the opportunity to learn more about and deliberate on key public policy issues.
87
1 The author, Janice Elliott, Ph. D. (Senior Associate, Public Policy Forum) acknowledges the support of David Brook, M.A. (Former Senior Research Associate,
Public Policy Forum ; Principal, DBk Consulting) for his work with the drafting and reviewing of this article.
2 The face-to-face methodology has been adapted to an online environment. (See Designing the Deliberation Process and Creating the Agenda, Section B4)
DELIBERATIVE POLLING
public opinion research in a new and constructive way to overcome these limitations. The Deliberative Polling process
A final distinct aspect of Deliberative Polling is the involvement of media/television to cover the event. This serves three
purposes: it engages the public and policy makers in a highly visible, transparent process; it adds to the reach of the
process in relation to the broader public; and it can give a sense of empowerment to the public who is engaged in the
deliberative poll (see Media, Section B below).
II. WHEN TO USE
There are several important criteria that should be used in determining whether a Deliberative Poll would be a useful
methodology for a particular engagement:
1. THE NATURE AND STATUS OF THE ISSUE ITSELF.
The following questions should have an affirmative answer before proceeding:
DELIBERATIVE POLLING
Is the issue of significant public interest and importance?
In general, such an issue will feature a number of strongly differing viewpoints and perspectives on how best it should
be addressed. If the public is not strongly invested or if there is not sufficient public awareness of an issue, it will be
difficult to generate a sufficient level of interest and sponsorship to undertake a Deliberative Poll® and, ultimately, it is
likely that the process will not be worth the expense.
Does the issue involve significant differences in public values?
It is these value judgements that suggest the need for informed deliberation and that lend richness to the deliberation
around trade-offs and consequences.
2. THE IMPORTANCE OF A SCIENTIFIC RANDOM SAMPLE TO DECISION MAKERS, SPONSORS, AND MEDIA.
Decision makers, especially elected officials, are generally interested in the representativeness of the sample and
learning about whether or not changes have occurred in knowledge and attitudes. Among deliberative methods, the
deliberative poll is distinctive in aspiring to get informed public opinion from a scientific random sample.
3. WHETHER AND HOW THE OUTCOMES OF THE PROCESS WILL BE LINKED TO KEY DECISION MAKERS AND STAKEHOLDERS.
The goal of the Deliberative Polling process is to influence decisions and to demonstrate to the public that their
deliberations can make a difference. One of the most important features of a Deliberative Poll is the opportunity for
participants to provide feedback directly to decision-makers and key stakeholders. Key decision makers must remain
open to the outcomes of the Deliberative Poll and the views and perspectives that will emerge from participants – if
decision makers have a strong vested interest in a particular viewpoint or set of conclusions it can undermine the value
of the entire process.
88
Examples of the types of issues for which Deliberative Polls have been conducted are:
1996
USA
National Issues Convention (US Presidential election)
1999
Australia
Changing from a Monarchy to a Republic
1999
Portland, USA
Education Issues
2000
Denmark
Referendum on the Euro
2001
Australia
Reconciliation with the Aboriginals
2002
Yale, USA
Regional Economic Development
III. PROCEDURE
A. Overview
DELIBERATIVE POLLING
The basic steps involved in developing and implementing a Deliberative Poll are summarized in Figure A below:
B. Designing the Deliberative Poll
1. ENGAGING A CONVENING ORGANIZATION/PROJECT TEAM
Once the decision is made to undertake a Deliberative Poll, based on the criteria identified in section II above, the first
and most important task in preparing for the deliberation is for the project sponsor to identify and engage the project
convener.
The project sponsor is generally a significant stakeholder who has a vested interest in a particular issue or set of
issues. Examples of potential project sponsors could include a national or regional government and/or government
department, National or regional Non-Government Organizations and others. Legitimacy and impact is enhanced by
engaging project sponsors from all sides of the issue and a media partner that is seen to be credible and competent
by the public that will be engaged.
The credibility of the convener is vital to the success of the deliberation. It should be a respected non-partisan
organization that can lend legitimacy to the process. It should be seen to be unbiased and have the respect of
decision makers and the public alike.
The project team should consist of a group of seasoned practitioners who are skilled at developing and implementing citizen engagement processes. Because of the complex nature of the Deliberative Polling method, it is useful if
some members of this team have experience implementing Deliberative Polls.
One of the first tasks to be undertaken by both the convener and the project sponsor is to develop and begin to implement a work-plan and budget for the deliberation (for a sample work-plan and schedule, please see Section IV below).
89
Figure A: Developing and Implementing a Deliberative Poll®
DELIBERATIVE POLLING
Designing the Deliberative Poll (B)
1. Engaging a convening organization/project team
2. Engaging additional sponsors
3. Engaging a media partner
4. Forming a steering Group with linkages to: Decision makers, Stakeholders, The project team, Sponsors,
Deliberative Polling Practitioners, and the polling firm
5. Engaging a reputable polling firm
6. Engaging an Advisory Group for deliberative materials
7. Determining the target ‘audience’
Naming and Framing Issues and Crafting the
Deliberative Materials (C)
1. Framing and Identifying Alternatives
2. Creating deliberative materials
The Survey (D)
1. Designing and conducting a survey to
measure knowledge and perspectives on
the issue.
Designing the Deliberation Process Creating the
Agenda (F)
1. For face to face and/or online build the
agenda and identify and recruit:
a. Subject matter experts,
b. A moderator, and
c. Facilitators
Inviting and Engaging the
Public (E)
1. Inviting a random and representative
sample to participate in the deliberation
2. Ongoing follow up with participants to
secure commitment to the process
Convening a Face-to-Face and/or Online Deliberation (G)
1. A random and representative sample of the public convene at the deliberation site to:
a. Deliberate
b. Learn
c. Undertake a Post-Survey
2. Participants present outcomes to decision makers during Assured Listening
Feedback Results and Follow-up (H)
1. Immediate follow up
a. Compare pre and post survey results for changes
b. Release results to participants, decision makers, media and sponsors.
c. Provide feedback to the participants on next steps, what has happened
2. Longer term engagement with participants.
d. Follow up survey
e. Compare results with control group
f. Other follow up depending on original purpose and needs of sponsors
90
2. ENGAGING ADDITIONAL SPONSORS
Given both the scope and the expense associated with undertaking a Deliberative Poll, it is often desirable to engage
additional sponsors and stakeholders; this again, in the interests of credibility, should be the responsibility of the
convener. Additional sponsors are drawn primarily from other stakeholders who have a vested interest in the outcomes
of the Deliberative Poll. This can include other orders of government (regional, local), NGOs, private sector organizations,
associations and the media.
3. ENGAGING A MEDIA PARTNER
A unique and critical component of the Deliberative Polling methodology is the engagement of regional and/or
national media, television ideally, to provide coverage of some or all of the Deliberative Poll. Media involvement
enhances the potential impact of the poll on decision/policy makers and the public at large. It enables the Polling
exercise to reach a broader audience and helps to raise awareness about the issue and the deliberation. It is particularly
DELIBERATIVE POLLING
important when the "Target Audience" for the engagement process is the broader public. (See Sub-Section 6 below)
4. FORMING A STEERING GROUP WITH LINKAGES TO:
1.
Decision makers, stakeholders
2. The project team
3. Sponsors
4. Seasoned practitioners of the Deliberative Polling methodology
5. The public polling firm
The objective of the Steering Group is to provide guidance on the process and linkages to the key decision makers who
will be engaged at the deliberation, the project team, the media partner, the polling firm and the project sponsors in
order to ensure that they are all familiar with and support the process and materials and are open to the potential outcomes of the process.
The Steering Group will be engaged at the outset and periodically throughout the process.
5. ENGAGING A REPUTABLE POLLING FIRM
One of the first tasks of the project team is to engage a reputable public opinion polling firm to work to develop and
deliver the survey elements of the Deliberative Poll (in conjunction with the project team). The polling firm should be an
integral part of the planning process and should be represented on the project Steering Committee.
91
6. ENGAGING AN ADVISORY GROUP FOR FRAMING OF ISSUES/DELIBERATIVE MATERIALS
The project team should also engage a separate Advisory Group whose purpose is to ensure that the chosen issue is
framed in as balanced, comprehensive and unbiased manner as possible. The Advisory Group should consist of credible
subject matter experts (neutral if they exist) on the issue that is selected. Ensuring that the Advisory Group has
members that represent a balance of perspectives is essential.
This group will be asked to verify that the Deliberative Materials are balanced and accurate and to attest to this publicly
through the inclusion of their names in the materials themselves. Again, it is important to ensure that all participants
as well as the public, sponsors and stakeholders have confidence in the balance and accuracy of the materials.
7. DETERMINING THE TARGET ‘AUDIENCE’
A final task in the initial preparatory stages of the deliberation is to determine the target ‘audience’ for the Deliberative
Poll. In essence, the Steering Group must answer the following question - who is it that the event is intended to
DELIBERATIVE POLLING
influence? This could be policy-makers and decision makers or it could be the public at large (e.g. prior to an election).
There is also a significant choice to be made as to whether the deliberation will be face-to-face or online. This decision
should be made by the project convener with strong guidance from the Steering Group (see Designing the Deliberation
Process, Section B4).
C. Naming and Framing the Issues & Creating the Deliberative Materials
One of the most important and time-consuming tasks associated with the development of a Deliberative Poll is the
naming and framing of issues and the related crafting of Deliberative Materials.
1. FRAMING AND IDENTIFYING ALTERNATIVES
The overall purpose of this phase of the process is to develop materials that will encourage thoughtful deliberation
among the public. How an issue is framed often determines how it is discussed. As such, it is critical to ‘Frame’ the
chosen issue in such a way that members of the public who are participating in the deliberation can see their voices
(and values) in the material and can begin to see and understand the trade-offs and consequences of particular types of
actions. The issue framing process is not unique to the Deliberative Polling process and is an important part of many
deliberative processes.
There are three primary objectives to be achieved through the process of naming and framing the issue:
92
1.
To inform and educate the public about the issue in a fair and balanced manner,
2. To identify and describe credible, reasonable, alternative ways of addressing the issue that can serve as the basis
for deliberation about what, if anything, to do about the issue. (The deliberation will ask the representative
public to consider these choices and to try to come to a decision about what they believe should be done, if
anything), and
3. To do both of the above in a credible and legitimate way in the eyes of sponsors, media, major stakeholders and
the public.
Identifying Reasonable Alternatives to Address the Issue
The issue is framed through the development of a set of alternative courses of action. These courses of action are identified by engaging stakeholders from various sides/perspectives on the issue. Each course of action is described in terms
of:
a. A brief description of the option;
b. An overview of what the option would mean for affected communities, governments, industries and publics; and
c. A discussion of the pros and cons associated with the option written in language of the ordinary public that
DELIBERATIVE POLLING
reflects their values.
Examples of different courses of action to address air pollution might include:
Governments should set tough industry and product standards to reduce air pollution
It will take voluntary action on the part of citizens, business and industry to change our day-to-day habits and lifestyles
and reduce air pollution.
Since your tip says to have more than 2 alternatives, offer at least one more example here.
TIP: It is important to create more than 2 alternatives or choices– 3 or more scenarios enable a more robust/less
polarized discussion. In developing these choices, make sure you get perspectives from stakeholders on different
sides of an issue.
2. CREATING DELIBERATIVE MATERIALS
As noted above, the purpose of the Deliberative Materials is to:
1.
Increase knowledge and understanding about the issue by presenting unbiased, balanced, comprehensive and
accessible information; and
2. Present credible alternatives to addressing the issue.
93
Format and Accessibility
The Deliberative Materials should be written in such a way that they are as accessible as possible to the general public
(written at a grade 8 literacy level, formatted in a broadsheet or large print format, including pictures, graphs and
charts). Overcoming the literacy barrier has been an ongoing focus for practitioners of the Deliberative Polling methodology. One way to address this issue is to develop a multi-media presentation of the information so that participants can
learn through an engaging audio-video format.
The Advisory Group
The Deliberative Materials should be vetted by the Advisory Group for balance, fairness and credibility. It is not
uncommon for the Deliberative Materials to go through several drafts before they are completed.
TIP: Focus test the materials with ordinary members of the public to ensure that the alternatives are presented
in a way that makes them equally credible, as choices, to proponents of all sides of the issue.
For more information on the framing of issues please see Framing Issues for Public Deliberation a publication of the
DELIBERATIVE POLLING
Charles F. Kettering Foundation, 2001.
D. The Survey
DESIGNING AND CONDUCTING A SURVEY TO MEASURE KNOWLEDGE AND PERSPECTIVES ON THE ISSUE
One of the two key components of the Deliberative Poll is the public opinion survey which is conducted prior to the
deliberative event and at the conclusion of the process. It is designed to reflect the naming and framing of issues.
Questions must seek views on alternatives and must assess the public’s knowledge about the issue. The survey is
designed with the polling firm and conducted with a random and representative sample of the public from the entire
region that is being engaged in the Deliberative Poll. If resources permit, a control group is also used to compare with
the sample that participates in the deliberation event to ensure that any changes are not a consequence of other factors
external to the deliberation.
E. Inviting and Engaging the Public
1. INVITING A RANDOM AND REPRESENTATIVE SAMPLE TO PARTICIPATE IN THE DELIBERATION
Each citizen who completes the survey is asked at the end of the survey whether they would be willing to participate in
the Deliberative Polling event. Members of the public who agree are then enrolled as participants (contact information
is recorded) and arrangements are made for each citizen to attend and participate in the Deliberative Poll.
TIP: Regular follow up with the public is required to maintain interest and commitment to attending. This
process is labour intensive particularly when travel is involved.
94
2. ONGOING FOLLOW UP WITH PARTICIPANTS TO SECURE COMMITMENT TO THE PROCESS
Participants receive an immediate follow-up mailing confirming their participation with information on the deliberation, the convener and the project sponsors. Often, all follow-up is undertaken by the polling firm that is engaged in the
initial survey process. In advance of the event, participating members of the public are sent a balanced briefing
document (the Deliberative Materials) and a general information package. Regular follow-up is required to ensure that
the cohort of participants remains engaged and informed.
If the Deliberative Poll is longer than a single day, arrangements should be made by the project team for the entire
participant group to stay on site (in order to foster dialogue and interaction around the key issue) and to have common
meals. It is useful to ensure that local media and other respected community organizations are aware of the
Deliberative Poll, its convener and project sponsors as the public will tend to be skeptical of the event and may attempt
to verify its legitimacy by contacting trusted information sources.
DELIBERATIVE POLLING
F. Designing the Deliberation Process and Creating the Agenda
BUILDING THE AGENDA AND IDENTIFYING AND RECRUITING:
a. Subject matter experts
b. A Moderator, and
c. Facilitators
d.
Volunteers
Purpose and flow of the process
The overall intent of the process is to give the public the opportunity to learn and deliberate on the alternatives using
the following flow:
1.
Exploring information and "facts" from the deliberative materials in small groups and in plenary both with other
members of the public and with "experts";
2. Working through each of the alternative choices in small groups to explore their personal and collective
perspectives and values;
3. Developing questions for subject matter experts to clarify and to learn more about the issue and the alternatives;
4. Asking these questions to the subject matter experts panel, who are selected because of their credibility, or
which is balanced through representation of different perspectives from various sides of the issue; and
5. Reconvening in small groups to consider all of the alternatives and to see if they can come to a decision on
which alternative they prefer, if any.
Learning Orientation
In essence, a deliberation is a public-centred and learning-oriented process. The agenda that is developed, and the
interactions that take place among members of the public, should reflect these priorities. The exact agenda for each
95
deliberation will be different, but the work plan in Section IV contains a basic overview of what an agenda for a one day
session might look like. Other agendas and materials are available through the Center for Deliberative Polling.
http://www.la.utexas.edu/research/delpol/
Subject matter experts
During each of the plenary sessions throughout the Deliberation process (see Convening a face-to face or online
Deliberative Poll, Section C below) members of the public should have access to a panel of subject matter experts to
answer their questions and to provide additional information and clarification about the facts and on the Deliberative
Materials. These experts should be identified, interviewed and recruited by the project team in advance of the session to
serve as onsite resources for the members of the public. It is critical to ensure that there is a balance in terms of knowledge and perspectives amongst the experts.
TIP: The experts do not make presentations. Rather, they are there as a resource and are responsive to the learning needs of the public.
DELIBERATIVE POLLING
The Moderator and Facilitators
Two key enablers of the success of the Deliberative Poll are the moderator and the facilitators. Ideally, a well-known and
trusted personality is identified and recruited by the convener to serve as the moderator and ‘face’ of the Deliberative
Poll. The project team should also identify facilitators for the small group sessions among members of the public. The
facilitators should be interviewed and selected well in advance of the session. It is important to ensure that the
moderator and the facilitators do not have any particular attachment to a specific outcome from the session and/or do
not feel the need to express their own particular views on the issues that will be discussed.
It is critical that the facilitators and the moderator are skilled at facilitation and are well informed about the nature and
objectives of the Deliberative Poll and about the issue that is to be the focus of discussion. To this end, the project team
should craft a facilitator’s guide to inform the facilitators about the process and their roles. [You might include a link to a
facilitator’s guide available online.]
Finally, both the moderator and the facilitators should spend a half-day with the project team in advance of the session
training for the event and becoming familiar with the methodology and the Deliberative Materials.
Volunteers
Volunteers are often needed to help manage such a large group of the public. These often are university or college students who have an interest in learning more about this kind of process.
Online Deliberative Polling
A variation on the traditional face-to-face methodology for conducting Deliberative Polls that has emerged in the past
few years is the adaptation of this methodology to an online environment. This methodology replaces face-to-face
deliberation with virtual interactions amongst the public through this environment. The online approach generally uses
96
voice interactions rather than text and these interactions are enabled by the provision of computers to a sample of the
public. It takes place in weekly synchronous discussions with moderated small groups, with asynchronous answers being
distributed between sessions. Answers are provided by competing experts (selected using a similar approach to the
expert panels in the face-to-face).
Some of the potential benefits/reasons for conducting an online Deliberative Poll include:
1.
A representative sample of members of the public can be assembled online for a fraction of the cost of an
in-person gathering;
2. It is easier and more flexibility exists to engage a community of participants over time; and
3. It is easier to generate exact metrics on what participants are doing during the process.
Potential cost savings can be somewhat offset, however, by the need to provide computers and internet access for
members of the random representative sample who are not already so equipped. This additional cost will tend to be
lower in urban areas (with higher concentrations of home computer ownership and broadband access) and over time
DELIBERATIVE POLLING
(as computer ownership becomes increasingly ubiquitous in nations where Deliberative Polls are conducted).
G. Convening a Face-to-Face and/or Online Deliberation
1. A RANDOM AND REPRESENTATIVE SAMPLE OF THE PUBLIC CONVENE AT THE DELIBERATION SITE TO:
a. Deliberate
b. Learn
c. Undertake a Post-Survey
The day of the Deliberative Poll itself begins with the convening of participating members of the public at the chosen
venue. As a general rule, Deliberative Polls will last for a minimum of one day and often extend for an entire weekend.
Arrangements are made by the project team to ensure that the members of the public are able to attend the entire
session. An honorarium is provided to participants to offset some of the personal costs associated with participating.
Although there is no standard amount for the honorarium, in North America it is usually in the range of $100 – 150 for
an entire weekend event.
TIP: In a weekend long session participants should be provided with a collective dinner and an evening recreational activity in order to have the opportunity to bond with one another. This helps enable the group to deepen
their sense of interpersonal connection and enhance the quality of their deliberation .
Registration
The Deliberative Poll begins with a registration session to ensure that participants are present and that they have all the
materials that they will need. At this point participants are informed of the small group that they will be working with over
the course of the event. Inevitably there will be some last minute changes and cancellations by participants so it is important
for the project team to ensure that the small groups are balanced in terms of numbers and demographics (if possible).
97
Convening
Following registration, the participants convene in the opening plenary, where they are given an overview of the
Deliberative event and questions are answered by the moderator and by the project team. It is a good idea to ensure
that at least one member of the project team serves as a trouble shooter for the participants, answering any questions
they might have and helping them to sort out any organizational details.
Small Group Work
The deliberative aspect of the Deliberative Poll then begins as the members of the public break into small facilitated
groups to discuss the key issues. As mentioned in a previous section, the Deliberative Materials frame the issue in such a
way that they present a set of viable alternative courses of action. In the small group sessions, the participants engage in
deliberation around each of the alternatives, exploring the trade-offs and consequences, their values, and asking
questions of each other. Over the course of the small group sessions, participants are encouraged to collect any
questions that might arise so that they can be addressed to the subject matter experts, who are available during the
plenary sessions.
Plenary Sessions
DELIBERATIVE POLLING
At the conclusion of each small group session, participants return to a plenary session, where they have the opportunity
to ask questions of clarification to the panels of experts who are present. These experts, who have been identified in
advance of the session (as described above), provide answers in as balanced a manner as possible, focusing on presenting facts and not their personal opinions. Balance can be achieved in the make up of the panel as well as in responses
to questions. The moderator manages the panel and their responses to ensure questions are answered in a balanced
manner.
The purpose of the small group/plenary session format is to allow each of the participants to increase their knowledge
and to work through each choice, to explore their own values and those of the other participants and, ultimately, to
learn from the deliberation. This small group/plenary session discussion pattern is then iterated for the remainder of
the day or weekend.
TIP: Observers are another important aspect of the Deliberative Poll experience. For many, ‘seeing is believing’
that the public can actually engage around complex public policy issues. They can also attest to the power and
legitimacy of the process. To facilitate observation without disrupting the small group process, a video feed of
one group (with their permission) can be fed into a room for observers.
Post-Deliberation Survey
At the conclusion of the deliberation and before the "assured listening", the same survey that was conducted in advance
of the session is administered again to participants, in order to map if and how the public’s knowledge and opinions
have changed over the course of the session. Surveys from each participant are matched pre and post in order to
quantify the magnitude of individual change (and to account for offsetting changes in opinion). This requires the
capacity to ID participant surveys in a way that respects their anonymity. (Note: the deliberative polling process does not
seek consensus as it is designed to better understand opinion change at the individual level. The survey is administered
before the final plenary to minimize the impact of social pressure on individual opinions.)
98
2. PARTICIPANTS PRESENT OUTCOMES TO DECISION- MAKERS DURING ASSURED LISTENING
At the conclusion of the process, participants have an opportunity to address key decision makers (e.g. politicians and key
stakeholders from various relevant jurisdictions and sectors) and present the results of their deliberation. The purpose of this
session is to enable and empower members of the public to understand the impact that their deliberations can have on key
decision makers and for key decision makers to ‘receive’ the outcomes of the deliberation. The Public Policy Forum calls this
"assured listening". This process of reporting the outcomes of the session should include both areas of consensus among
participants and the views and perspectives of smaller groups of participants and of individuals.
TIP: Decision makers need to be coached in advance for receiving the results of the deliberation.
Decision makers are briefed in advance of the session and encouraged not to challenge the members of the public’s
opinions, but rather to listen carefully to the participants’ comments and perspectives. They are coached to see this
session and the public as a "rich resource" for understanding how the public views what should be done about an issue.
This is particularly important when politicians such as government Ministers are involved. They are encouraged to ask
DELIBERATIVE POLLING
questions of clarification if they do not understand a particular point or issue.
At the conclusion of the session participants are thanked for their time and arrangements are confirmed for outgoing
travelers. They are advised of what follow up to expect from the convener and, at a general level, from decision makers
and others.
Managing the Media
Special attention must be paid to working with the media during the event. The project team must deal with the media,
decision makers and the public at the same time in a way that preserves the quality of public deliberation. It is essential
to have someone assigned to managing the media on site. Media ground rules for engaging with participants should be
set in advance to minimize disruption of the deliberation.
H. Feedback Results and Follow-up Post Deliberative Poll
1. IMMEDIATE FOLLOW UP
a. Compare pre and post survey results for changes
b. Release results to participants, decision makers, media and sponsors.
c. Provide feedback to the participants on next steps, what has happened
Following the event, the polling firm matches and compares the pre- and post- surveys in order to identify and quantify
statistically significant changes in knowledge, attitudes/ opinions. On the basis of this analysis, reports are then
prepared for the participants, decision makers, and sponsors. These findings are also usually presented in press releases
to the media and public.
99
It is critical to ensure that the members of the public who participated in the project get feedback on the next steps and
other actions that will be taken as a result of the findings from the session. It is also important to release the results of
the process quickly in order to sustain media interest. In general, the project team and the project sponsor will send
follow-up letters to participants thanking them for their participation and outlining the steps (if any) that the
sponsor/decision maker intends to take to build on the outcomes.
2. LONGER TERM ENGAGEMENT WITH PARTICIPANTS.
d. Follow up survey
e. Compare results with control group
f.
Other follow up depending on original purpose and needs of sponsors
One option is for the project team to conduct the same survey one final time six months to one year post the event to
see what changes have persisted over time and what behavioral change has occurred. While it is not usually an explicit
purpose or objective of a Deliberative Poll, the potential exists to enable members of the public who participate in the
DELIBERATIVE POLLING
Deliberative Poll to remain engaged with one another, with the sponsors and/or with decision makers. Options in this
regard might include creating an online community or listserve through which participants can remain in contact with
one another. Also sponsors or decision makers may want to follow up to support or encourage action or behavior change
on the part of the engaged public.
IV. RESOURCE CONSIDERATIONS
A. Work plan
Conducting a Deliberative Poll is a time- and resource-intensive undertaking, although the costs can be significantly
reduced by moving from a face-to-face to an online methodology. Although there is no fixed schedule on when the tasks
outlined above should be completed, the following are some basic guidelines which can inform the development of a
Deliberative Polling process.
In total, the development and implementation of a Deliberative Poll can take from six months to a year to complete,
from early design and conception to the completion of the post-event polling and the final analysis of outcomes. It can
be done in a shorter time period depending on the available resources and other similar considerations. Table 1 below
outlines the significant activities that should be undertaken in preparation for the Deliberative Poll, based on a six
month lead time.
100
Table 1: Schedule for the Development and Implementation of a Deliberative Poll
Timing
Activity
Pre-project Launch
Decision to proceed
Get commitment of sponsors/key decisions makers to be open to receive a variety of
outcomes
Engage a convener
Confirm principle sponsor
Develop project budget and timeline & get funding commitment from principle sponsor
Initiate Project
Confirm key issue
Recruit project design and implementation team
Confirm timing and location of the event
DELIBERATIVE POLLING
Month 1
Engage Key Decision Makers
Confirm participation of key decision makers.
Generally, these decision makers will be associated with the project sponsor but should
also include key politicians and/or public servants who have an interest in and
responsibility for the issue.
Develop Project Steering Group including:
a. Decision makers, stakeholders
b. The project team
c. Sponsors
d. Seasoned practitioners of the Deliberative Polling methodology
e. Polling firm
Develop Advisory Group
Identify & engage key thinkers and opinion leaders relevant to the issue in question
Recruit media partner
Identify and engage local and national media sources to sponsor/cover the event
Recruit additional event sponsors (optional)
Engage additional organizations and jurisdictions that could have a vested interest in
the issue that is being addressed.
Engage reputable Public Opinion Polling Firm
Confirm timing and other issues relating to the survey
101
Begin Naming And Framing Of Issue (Developing Deliberative Materials)
Conduct exhaustive literature review on key issue and related subjects
Month 2
Confirm Venue
Must be able to accommodate the number of participants plus the project team and
related event staff and observers.
Must be able to accommodate both a single large room and a number of smaller rooms for
small group work with facilitators
Continue Deliberative Materials Development
Draw on Advisory Group and stakeholders to develop an understanding of the intricacies
and various perspectives related to the issue
Identify and Engage the Moderator for the Deliberative Poll
DELIBERATIVE POLLING
An important determinant of the success is the moderator.
Month 3
Recruit Subject Experts for Event
Identify experts in key areas
Ensure participation of experts from all sides of the issues
Recruit facilitators
Identify and engage facilitators.
Develop Draft of Deliberative Materials
Develop working draft of Deliberative Materials
If possible, an audio-visual or online version of the materials should also be developed in
order to enable as broad a learning environment as possible
Focus Group Testing of Deliberative Materials
The draft of the Deliberative Materials should be focus group tested at least once and
preferably several times over the course of their development.
Develop Survey
Develop questions for survey, working with the project sponsor, the polling firm and the
Steering Group
Confirm Survey questions
Validate through focus testing
Validate with project sponsors
Validate with Advisory Group
102
Month 4
Validate Deliberative Materials
Review drafts of Deliberative Materials with key stakeholders and subject experts
Focus is to ensure that the materials are as comprehensive, neutral and unbiased as possible.
Advisory group reviews and agrees fair and balanced
Undertake Pre-survey
Survey random representative sample
Invite random representative sample of survey to participate (minimum number of
participants should be 250-300 to ensure statistical validity)
Month 5
Confirm participants
Immediate letter of confirmation with package explaining Deliberative Polling and convener
Book lodgings
Book travel
DELIBERATIVE POLLING
TIP: To ensure commitment to attend, regular contact and follow up with public who
have agreed to attend is essential. A minimum of 3 contacts with the sample is required.
Finalize and Print Deliberative Materials
Develop final version of Deliberative Materials
Final review
Translation and review if required
Typeset and print materials (will take at least two to three weeks)
Corrections and updates to typeset materials
Final print run
Finalize Other Event Materials
Finalize agenda
Finalize roster of experts
Month 6
Confirm participant travel
Ensure that all special needs travelers have been accommodated
Distribute Deliberative Materials
Participants should have the opportunity to read the Deliberative Materials at least a week
in advance of the session.
Participants should also receive a participant package with event schedule and other
useful materials including day-of emergency contact information.
Preparation of Facilitators Guide
Development of guide for small group facilitators which outlines their roles and responsibilities
103
Facilitators also review the Deliberative Materials in advance of the session and are aware
of and familiar with the issue in question.
Week Prior to Event
Conference Call with experts
Confirm their role (balanced, only answering questions that are directed to them from
participants), no presentations or advocacy
Allows an opportunity for the different experts on each panel to meet and discuss in
advance of the session.
Allow for the free exchange of ideas on the types of questions that will be asked and who
will answer the questions.
Conference Call or Meeting with Moderator
A critical session in order to ensure that the Moderator understands role in the process.
In many ways they will serve as the participants representative or leader, asking questions
DELIBERATIVE POLLING
of clarification from experts and serving as a trusted, neutral lead for the event.
Day Prior to Event
Preparation of Venue
All signage and set up complete
Confirm Participant Travel
Ensure that members of the public are arriving as expected and that unexpected issues
are resolved.
Facilitator Training
Facilitator preparation is key to the event
The convener should plan on spending a minimum of half a day with the moderator and
facilitators in advance of the sessions
Moderator and facilitators must be familiar with the Deliberative Materials
Brief Volunteers
Event Agenda
In general, Deliberative Polls can last anywhere from a single day to a full weekend event.
The basic outline of events will be similar, however, with more group work and plenary sessions in
the longer session.
Breakfast
Registration of Participants
Participants are registered and given the information they need for the remainder of the
event
104
Participants should be pre-grouped for the small group work that they will undertake
Introduction
Introduction and welcome by the moderator and the project team
Outline of the days activities and answers to questions
Small Group Session
Each small group session is led by a trained facilitator.
As part of the small group work, each group collects then chooses which questions they
would like answered by the panel of experts during the plenary.
Plenary Session
The plenary session brings together the small groups to ask questions of clarification from
the experts.
The moderator acts as the discussant re-stating questions and ensuring that the experts
DELIBERATIVE POLLING
respond to the questions as they are presented.
It is important to ensure that the experts answer only the questions that are asked and
that they do not editorialize or express their own opinions. The goal is balance and a fair
representation of the issues.
Lunch
Small Group/Plenary (Repeat)
Preparation for Report Out and "Assured Listening"
At the conclusion of the session each small group prepares to present the outcomes of
their group work to the key stakeholders and decision makers in the final plenary. Each
group has approximately 5 minutes to share their views.
Post-deliberation Survey
Undertaken at the conclusion of the Deliberative Poll by the polling firm or self
administered in small groups.
Assured Listening
Participants express the results of their deliberation to decision makers.
Decision makers have the opportunity to ask any questions of clarification.
Conclusion
105
Post-Session
(immediate)
De-brief with Moderator and Facilitators
Review the outcomes of the process while it is still fresh in the minds of the moderators
and facilitators
Post (Month 1)
Analysis of Data and Report
Analysis of the data that is collected pre and post and the drafting of an outcomes report
for the project sponsor and participants
Potentially, the drafting of academic or other articles to capture the project outcomes.
Media/Public Release of Outcomes
Reporting out to the public/media on the outcomes of the project
Statistically significant changes are highlighted.
Thank you and Report out to Participants
Acknowledging the participation of members of the public is important in addition to
DELIBERATIVE POLLING
making them aware of the outcomes of the project.
Post Year 1
(optional) Follow-up Survey
Undertake final follow up survey of the original survey cohort and control group to see if
there has been a long term change in their behavior/opinions.
B. Budget
A Deliberative Poll is expensive to undertake. Costs are very context dependent (they depend on duration, location, and
whether participants travel and accommodations are required.) They can range anywhere from a few hundred thousand
dollars to upwards of a million dollars. This process, however, presents a unique opportunity to generate a rich and
nuanced understanding of what an informed public thinks and believes about an issue. As such, it is necessary to
balance the cost of undertaking such a process against the richness of the information that is uncovered. Conducting a
Deliberative Poll online can reduce these costs significantly. The primary cost drivers for the process include:
1. ENGAGING A CONVENER AND PROCESS DESIGN TEAM — Given the specialized nature of the Deliberative Polling
process, it is almost inevitable that a project sponsor will want to engage a seasoned project design team that can
oversee the development and implementation of the Deliberative Polling methodology.
Approximate Expense: € 40.000 - € 80.000
2. UNDERTAKING THE PUBLIC OPINION SURVEYS & ANALYSIS (PRE AND POST) — Again, given the very specific skill
sets and infrastructure necessary to undertake a large-scale public opinion poll, the convener will need to engage a
respected public opinion firm to poll and recruit participants.
Approximate Expense: € 40.000
106
3. FOLLOW UP WITH PUBLIC AND MAKING LOGISTICAL ARRANGEMENTS:
Approximate Expense: € 10.000
4. COMMUNICATIONS — Given its focus on political leaders and media involvement, there is a considerable communications component that must be addressed as part of the planning and implementation of the Deliberative Poll.
Approximate Expense: € 20.000
5. TRANSPORTATION AND LODGINGS — Depending on the nature of the Deliberative Poll it is possible that many of
the participants will need to travel to the venue in which the event is taking place. If the Deliberative Poll is being
conducted in a small region, these costs can be minimal. If, however, the Deliberative Poll is national in scope, especially in larger countries, travel expenses, lodgings and the associated coordination can be a considerable expense.
Participants are also usually paid a nominal honorarium for their time (on the order of € 100- € 150)
6. VENUE — It can be a significant challenge to locate a suitable venue to encourage and enable dialogue among a
large group of members of the public. For multi-day events, items such as meals for participants can also constitute
a significant expense.
Approximate Expense: € 10.000 - € 15.000
7. MODERATOR AND FACILITATORS — Depending on the number of participants and the availability of skilled facilitators, it can be costly to identify, recruit and train facilitators for a Deliberative Poll. Often, larger cities will have an
association of facilitators that can be asked to identify appropriate potential facilitators.
Approximate Expense: € 15.000
8. ONLINE ENGAGEMENT — Although the authors of this chapter did not employ an online component to the work
that they conducted, current Deliberative Polling practice often involves significant online elements that can generate an additional category of expense.
Approximate Expense: tbd
V. ADDITIONAL BEST PRACTICES AND POTENTIAL PITFALLS
From the sponsor’s (the organization or government that is commissioning the Deliberative Poll) perspective, it is important to not have an investment in any particular outcome for the process. Once members of the public are informed in a
balanced, unbiased way, it is up to them to guide the discussion during the process in the directions that they think
most appropriate or most useful.
107
NAME OF METHOD:
Approximate Expense: € 10.000 - € 100.000
For a Deliberative Poll to have the maximum possible impact there must be a commitment on the part of sponsors/
decision makers to listen carefully and where possible take action based on the results of the sessions. Members of the
public want to know that their time spent in deliberation will be listened to by key decision makers (through the
assured listening) and that they will receive feedback on what has happened as a result of the deliberation.
Finally, for a Deliberative Poll to be truly transformative for members of the public, it ideally will be part of a broader,
ongoing process of engagement. If one of the goals of any citizen engagement process is to inform members of the
public and give them a sense of agency and allow them to identify the actions that they can personally take in order to
help address an issue or concern, then every effort must be made by the project team and the project sponsor following
the event to support members of the public in taking action based on their learnings. Ideally, this will have been part of
the overall planning of the project.
References and Resources
DELIBERATIVE POLLING
Belcher et al (2001) Framing Issues for Public Deliberation (Charles F. Kettering Foundation)
Berkman Center for Internet & Society at Harvard Law School, Representation in Cyberspace Study
http://cyber.law.harvard.edu/rcs/fish.html
The Center for Deliberative Polling®, University of Texas
http://www.la.utexas.edu/research/delpol/bluebook/execsum.html
Centre for Deliberative Democracy, Stanford University
http://cdd.stanford.edu/
Discussion of Public Choice Theory
http://encyclopedia.laborlawtalk.com/Public_choice_theory
Fishkin, J. (Revised 2004) Deliberative Polling®: Toward a Better-Informed Democracy (Stanford University: Center for
Deliberative Democrac)
Fishkin, J.; Rosell, S. (Winter 2004) Choice Dialogues and Deliberative Polls: Two Approaches to Deliberative Democracy,
(National Civic Review, Vol 93, #4)
Fishkin, J; Ackerman, B. (2005) Deliberation Day 2004 (Yale University Press)
Lynegar, S.; Luskin, R.; Fishkin, J. (2004) Facilitating Informed Public Opinion: Evidence from Face-to-Face and Online
Deliberative Polls, (Center for Deliberative Democracy, Research Papers)
108