Campaign News and Vote Intentions in
Canada, 1993-2008
1
Marc André Bodet
Lori Young
Stuart Soroka
McGill University
Mass media play a central role in modern representative democracy; indeed, it is
difficult to imagine representative democracy without them. Media are the
principle means by which citizens learn about what their representatives are
doing, when they are doing things right, and, moreover (given what we know
about media) when they are doing things wrong. Representation between elections
requires mass media.
So too do elections themselves. The modern election campaign is hugely dependent
on media; indeed, election campaigns are almost by definition media campaigns.
Mass media are the principle if not the only source of information about leaders,
candidates, parties, policies, and the horserace.2 Journalists are highly attentive to
the campaign as well as to the state of public opinion, particularly vote intentions.
And many citizens are more attentive to media content during election campaigns.
As a consequence, the link between media and the public during an election
campaign is likely to be especially strong.
Note that we are agnostic on whether media actually lead or follow public opinion.
It is likely that at any given time mass media are doing a bit of both. Media
content can be regarded in two often empirically inseparable ways: (1) it can
reflect the issues, themes and actors that are currently prominent in public debate,
and (2) it can be a potential driver of public opinion and policy. In the former
Paper prepared for presentation at the Annual Meeting of the American Political
Science Association, Toronto ON, Sept 3-6 2009. We are grateful to Blake Andrew for
his help developing and managing the manual content analyses, to Mark Daku for his
ongoing work on both election prediction, and programming Lexicoder, and to John
Galbraith for comments on related work. This work was funded in part by the Fonds
québécois de la recherche sur la société et la culture, the Donner Canadian
Foundation, the McGill-Max Bell Strategic Initiative, and the McGill Institute for the
Study of Canada.
2 On horserace coverage in particular, see, e.g., Craig 2000; Fletcher 1991; Graber
1976; Jaimeson 1992; Mendelshon 1993; Patterson 1993; Wilson 1980.
1
1
case, mass media act simply as mirror. Media content in this view is a useful
summary indication of the more general public sphere. In the latter case, mass
media are not mirroring but affecting. Media content differs from what citizens or
politicians currently think, and has the potential to affect these actors’ attitudes.
In either case, media may shift before opinion. Journalists are highly attentive to
the campaign, and their stories are published daily. The larger public is less
attentive, and likely takes longer to react to campaign events. Measuring public
opinion also takes time. The end result is that, even if media are not leading
opinion substantively speaking, they may well lead opinion statistically speaking.
To be clear: media content is likely to register shifts in the campaign before public
polls, even if media effects are somewhat limited.
We take advantage of this possibility here, with an eye on predicting campaignperiod dynamics. This paper builds on our own recent work predicting campaignperiod vote intentions the 2004 and 2006 Canadian elections (Soroka et al. N.d.),
not to mention the vast literature on predicting election outcomes,3 alongside a
body work suggesting that campaign events matter to electoral decision-making.4
In our previous work, human-coded media content is shown to be a valuable
component of a campaign-period prediction model; indeed, media data alone are
able to account for the lion’s share of the movement in vote intentions over the
campaign. Here, we aim to both refine and add to our model in several ways.
First, we develop and test a means of automating the content analysis of tone in
election media. Automated content analysis is a powerful tool to compute relevant
indicators systematically, for large quantities of data. To date, most content
analyses in political science have been performed manually. Reliability and
homogeneity in the coding process are often problematic, particularly for
comparatively subjective codes such as “tone.” Automated content analysis goes
some way towards solving this problem. It also makes feasible the (identical)
treatment of a previously unimaginable quantity of data.
Much of this work focuses on long-term forecasts of election results using variables in
the pre-election period, e.g., Abramowitz 1988; Cambell and Garand 2000; Hibbs
1982; Lewis-Beck and Rice 1984; Wlezien and Erikson 2004. (For the most recent
examples in the US context, see volume 41 of PS: Political Science and Politics,
including, e.g., Campbell 2008, Erikson and Wlezien 2008, Lewis-Beck and Tien 2008,
Norpoth 2008.)
3
The literature is vast, but see, e.g., Holbrook 1996, Huber and Arceneaux 2007;
Johnston et al. 1992, Gelman and King 1993, Lodge et a. 1995, Shaw 1999, Wlezien
and Erikson 2002.
4
2
Once we can deal with a very large body of content analytic data, we are also able
to deal with a good number of election campaigns. We begin our predictions here
by re-analyzing the 2004 and 2006 election, now using automated data. We then
add a model for 2008, to confirm that our previous results were not the product of
two peculiar campaigns. Finally, we add the 1993, 1997 and 2000 elections. There
were relatively few opinion polls during these earlier campaigns, so our analysis in
these cases relies on a pooled model, of all six campaigns together. Analyzing the
data in this way reflects our belief that there is a generalized relationship between
media content and vote intentions. We discuss this further in the concluding
section. First, however, we introduce our approach to automated content analysis.
Automated Content Analysis
The body of content-analytic data used here relies on a comprehensive database of
all election stories – news, editorials, and opinion pieces – published in six major
daily newspapers across Canada during the six Canadian federal election
campaigns from 1993 to 2008. Data were gathered using full-text indices in Nexis.
The newspapers analyzed include the Vancouver Sun, Calgary Herald, Toronto
Star, National Post, Globe and Mail and Montreal Gazette. French newspapers are
not included, as the automated system is currently suited to English text only.
Inclusion of the Montreal Gazette achieves some regional variance, but clearly is
no substitute for French-language newspapers. We should thus keep in mind that
this sample is not entirely regionally representative. both the Ottawa Citizen and
National Post were not available for the entire time period (the National Post did
not exist), and so are not included in those year. Details of the dataset are
included in Appendix Table A1. In total, 26,898 news stories, editorial and opinion
pieces are are included in this dataset.
The automation of tone is conducted using a standard dictionary-based approach
– a simple word count of positive and negative words in a text from a pre-defined
categorical dictionary. With a comprehensive and well-defined dictionary, the
frequency of linguistic categories can provide a powerful mapping of the
underlying meaning in a text. There are a good number of established content
analytic dictionaries that focus on sentiment (or valence, or tone) in text (see, e.g.,
Whissell 1989; Mergenthaler 1996; 2008; Bradley and Lang 1999; Pennebaker et al.
2001; Strapparava and Valitutti. 2004). Sentiment lexicons arise out of diverse
methodologies; many are designed for specific purposes and/or generated for, or
from, particular genres. Consequently, they vary widely with respect to sentiment
categories and scope of coverage. There is surprisingly little overlap among
dictionaries and, to the extent that there is, codes are quite discrepant.
3
Here we use the Lexicoder Sentiment Dictionary (LSD), which was developed to
improve the scope of coverage and accuracy of valence codes specifically for
political texts (Young and Soroka, N.d.). The LSD merges and standardizes three
of the largest and most widely used affective lexical resources to generate a broad
lexicon scored for positive and negative tone. The basis of the LSD is Roget’s
Thesaurus (Roget 1911) – the only truly comprehensive word list scored for
valence. Roget’s Thesaurus is manually annotated for hundreds of categories
including, for example, positive categories such as “benevolence,” “vindication,”
“respect,” “cheerfulness” and “intelligence” and negative categories such as
“insolence,” “malevolence,” “painfulness,” “disappointment,” and
“neglect” (n=47596). The LSD also includes the General Inquirer’s (GI) (Stone et
al. 1996) two broad valence categories labeled “Positiv” and “Negativ” (n=4295);
and from the emotion class of the Regression and Imagery Dictionary (RID)
(Martindale 1975, 1990), the positive categories “positive affect” and “glory” and
the negative categories “chaos,” “aggression,” “diffusion,” “anxiety” and
“sadness” (n=1056).
Once merged, the number of positive and negative classifications was calculated
for words with multiple categories in a single dictionary, and the highest valence
score retained. Each word was then classified as positive or negative in the LSD,
so long as at least two of the three core dictionary entries listed the word as either
positive or negative. Ambiguous words were not included – that is, any entry with
one positive and one negative code. Dictionary entries were then truncated to
capture inflected variations, providing all inflected forms retained the same tone;
in cases where the tone of inflected words differed from the original entry,
truncation was not applied, and the inflected variations were individually added to
the dictionary with appropriate codes for tone. Common stop-words were removed
and alternative spellings added.
Two of the greatest challenges for the automation of tone concern (1) homonyms,
and (2) the contextual nature of tonal language. Few automated methods attempt
to disambiguate homonyms, and standard bag-of-words approaches gloss over
context entirely. To address these issues, at least minimally, LSD entries were
analyzed in context using WordStat’s KWIC (keyword in context) list. KWIC
functionality is available in many software packages. It enables content analysts to
examine different uses of the same word in a corpus. LSD entries were analyzed in
context using some 10,000 newspaper articles, enabling the authors to identify
dictionary entries with multiple word senses and tricky contextual usage.
Ambiguous entries were either dropped or disambiguated in one of two ways. For
many entries, contextual phrases were added to the dictionary to capture various
uses and senses of a word. For example, the homonym “lie” is only negative in
4
certain contexts. To capture negative senses only, the dictionary entry “lie” is
replaced with several phrases, including: “a lie” “lie to,” “they lie,” “are lies” etc.
Contextual information was also employed to generate a pre-processing module,
which formats the corpus prior to content analysis, to facilitate word sense
disambiguation and contextual analysis of tonal language. The main body of the
pre-processor removes “false hits” for dictionary entries, where false-hits are
topical, multi-toned or non-tonal instances of a dictionary entry. For example,
multi-toned phrases such as ‘good grief,’ ‘losing hope’ and ‘tears of joy’ are
processed to remove the toned words ‘good,’ ‘hope’ and ‘tears’; non-tonal phrases
such as ‘an awful lot,’ ‘crude oil’ and ‘child care’ are processed to remove the
toned words ‘awful,’ ‘crude,’ and ‘care.’
Pre-processing also standardizes punctuation and basic negation phrases.
Standardizing negation allows for a variety of negated phrases to be captured with
a limited number of dictionary entries. The pre-processor first replaces negation
words such as “no,” “never,” “neither,” “hardly,” “less,” etc. with “not.” Second,
various verbs and pronouns following negation words are removed: “not going to,”
“not feeling,” “hardly any,” “nor were they,” “no one was,” “without much,” etc. are
replaced with “not.” Standardized in this manner, the truncated dictionary entry
“not good” captures a range of negative phrases including: “not at all good”;
“hardly a good idea,” “no one good”; “nor was it good,” “without goodness,” and
many more. Some of the most nuanced entries in the dictionary use a combination
of contextual phrases, truncation, negation and pre-processing.
Contextual analysis also facilitated the addition of numerous dictionary entries –
many particular to political news reporting – that were not present in any of the
core lexicons. The final dictionary includes 6016-words scored for positive or
negative tone and the pre-processing of over 1500 words, facilitating basic word
sense disambiguation and the contextualization of many commonly used sentiment
words and phrases. The LSD performs reliably against human coders and improves
the scope of coverage for political texts relative to several other commonly used
sentiment lexicons (Young and Soroka N.d.).
Coding the tone of parties and party leaders presents an additional challenge for
automated content analysis. Full-text analysis, generating document-level tone,
does not necessarily reflect the tone of coverage of particular actors mentioned
within a text. The automation of ‘actor tone’ generally requires subdocument
analysis to determine the tone of coverage about particular actors. To automate
actor tone, we employ a proximity-based process that relies on simple lexical rules
to account for the proximity (or local co-occurrence) of sentiment words and a
‘subject’ of interest in a text, a method noted in many studies to improve
5
sentiment analysis relative to full-text analyses (see, e.g., Pang, Lee and
Vaithyanathan 2002; Mullen and Collier 2004; Tong 2001). Mentions of parties
and party leaders are captured relatively easily in automated analysis – most
standard database software can record the number of mentions of different words
in a corpus. Our automated analysis begins, then, by identifying sentences in
which major parties or leaders are mentioned. The software extracts, for instance,
all sentences including a reference to Stephen Harper, and then counts the number
of positive and negative words in those sentences.
The resulting ‘net tone’ measure is the number of positive words, minus the
number of negative words, taken as a proportion of all words in sentences
mentioning the party or leader in question. This proximity-based procedure results
in a continuous measure of polarity, or tone, for each party and/or party leader,
for each article. A score of zero reflects perfect neutrality; positive scores indicate
increasingly positive coverage; and negative scores indicate increasingly negative
coverage. Given the nature of news reports, positive and negative coverage
necessarily reflects a combination of objective content about the various issues and
events being reported on, and subjective opinions or attitudes about the content
itself. Positive coverage can result from attention to objectively positive events or
policy successes; it might equally result from avid support for, or praise of,
government policies. Likewise, negative coverage can result from attention to
objectively tragic events or the failure of a policy; it might equally reflect criticism
of government policies. Thus, tone should be considered a composite measure of
the relative negativity of the actual events or issues being covered and the
opinions and attitudes of newsmakers about those events and issues.
[Figure 1 about here]
Figure 1 shows time series of net tone for both the Liberal and Conservatives in
the 2006 election - an election for which we have comparable human-coded data as
well. Those data capture tone using positive/neutral/negative codes from human
coders; they are used and discussed in detail in Soroka et al. N.d. Here, they serve
as useful diagnostic. If our automated tone is capturing the “right” stuff, the
automated results should look much like those derived from the human-coded
data. There will be minor differences, to be sure, if for no other reason than
because the human-coded data does not include the Montreal Gazette, and does
include two French-language newspapers (Le Devoir and La Presse), and the
inclusion (or not) of these papers surely makes a difference.
6
Even so, trends in the automated data are very close to those derived from
human-coded data.5 They are by no means perfectly matched, to be sure. But the
general trends are very similar through much of the campaign. We take this to be
a first sign that the automated coding is capturing something “real.” Of course, the
proof is in the pudding: if automated coding captures tone for parties and leaders,
then the resulting measures will be significant in our prediction models. We test
that possibility below.
Predicting Election Dynamics, 1993-2008
Data and Models
Our predictions here build directly on work in Soroka et al. N.d. There,
predictions were computed separately for the Liberal Party and the Conservative
Party for the 2004 and 2006 elections. This paper goes further by including four
other elections but, more importantly, by testing the efficiency of a pooled model.
Estimating a pooled model is interesting for both substantial and practical
reasons. On the substantial side, while there may well some various over time (see
the conclusions below), we would like to test for a generalized — i.e., common
across elections — link between media coverage and vote intentions. To a certain
extent, the objective is here to estimate the systematic, or “natural” relationship
between media tone and vote intentions.
In the extreme case, variations in estimated coefficients across elections are due
only to measurement error and lack of efficiency. Pooling elections together might
then improve efficiency and offer estimates closer to true values. On a more
pragmatic level, a cross-election model allows us to overcome estimation
difficulties that arise for elections in which there is a limited body of polling data.
Pre-2004 elections featured relatively few commercial polls,6 and as a consequence
our view of campaign dynamics in this period is still somewhat limited. But future
election predictions might also benefit from a pooled model -- a model which, for
instance, would be able to use past election data to make campaign-period
We readily admit that the ideal comparison of automated and manually-coded data
combines this kind of aggregate-level analysis with an individual-level one — that is,
an analysis of the reliability with which the automation captures the same information
as a human coder in individual articles. We have completed such an analysis for
economic news coverage; an analysis using election coverage is ongoing.
6 In the 1993 campaign for example, only 11 polls were found despite roughly 7 weeks
of campaign.
5
7
predications even in the very early days following the election call, when only a
few polls have yet been conducted.
We have reviewed media measures in the preceding section. Our other body of
data — polling results — relies on all publicly available polling results for the last
six Canadian federal elections. Opinion surveys are usually conducted over a three
to four day period, so we assign dates in the following way: vote intentions are
attached to the middle field day, or, for polls in the field for an even number of
days, the day before the midway point. Whenever two observations fall on the
same day, a weighted average is computed using the number of respondents in
each survey.
Using these bodies of media and polling data, we first estimate election-specific
Ordinary Least Square (OLS) regression models of the current vote share for each
of the major two parties, for each of 2004, 2006 and 2008. The 2004 and 2006
estimations are exactly as in Soroka et al. (N.d.), though here we are of course
using automated media content analysis. The 2008 estimations are entirely new.
Each model includes an autoregressive component alongside two matrices of lagged
independent variables. The first is a series of dummy variable for recurrent polling
firms in our sample: SES, The Strategic Council, Decima, Ipsos Reid, and Nanos.
These variables are equal to 1 if a given firm has a poll on the field on a given day
and 0 otherwise; they are intended to capture ‘house’ or ‘pollster’ effects – the
tendency for different firms to capture slightly but systematically different
distributions of vote intention, due to methodological decisions relating to, for
instance, the partition of undecided voters and “don’t knows”. (See, e.g, Jackman
2005; Pickup and Johnston 2007; McDermott and Frankovic 2003.) Note that
these pollster effects can be viewed as one of several different types of error in
polling data, alongside what classical statistics labels ‘random error,’ a function of
sample size, clerical errors, etc. And while this random error is typically handled
by using least squares estimates, the error resulting from pollster effects is dealt
with by using the dummy variables outlined here.7 In any case, we do not
interpret pollster effects dummies below, but include them only as controls.
Note also that there is another aspect of error in polling results that is rarely
discussed but that is often implicitly accepted in public opinion research. Because
respondents may be in a better position to accurately express vote intention later in
the campaign, survey responses may be more reliable closer to the election date. As a
consequence, time series of vote intentions may be prone to temporal
heteroskedasticity – random error variance may not be constant across time. This
violates a critical assumption of OLS, but we do not address it here. We simply
assume temporal homoskedasticity.
7
8
The second matrix of independent variables includes four net tone measures, for
(1) the Liberal Party, (2) the Conservative Party/Reform Party/Canadian
Alliance, (3) Liberal Party Leader, and (4) Conservative Party/Reform/Canadian
Alliance Leader. ‘Net tone’ is simply the number of positive mentions for a leader
or party minus the number of negative mentions on a given day.8
Put more formally, our prediction model is as follows,
(1)
Votep,t = α +∑ (βf *Pollsterf,t) + ∑ (ωη*Toneη,t-k) [+ ωη*Votep,t-k] + εt ,
where polled vote intentions for party (p) at some time (t) is a function of a set of
dummy variables capturing pollster effects for each firm (f) at that time and a set
of net tone measures for each major party and leader (η) lagged by k time periods.
We also include, in square brackets above, the dependent variable, Votep, lagged k
time periods. (By lagging Votep by the same period by which media variables are
lagged, we produce a model that could be used during an election campaign.
Predicting four days in the future, for instance, relies on variables lagged no less
that four days.)
The square brackets in model 1 reflect the fact that we do not always include a
lagged dependent variable. While a model’s predictive capacity can be can be
limited by not taking into account a strong autoregressive (AR) process, excluding
the AR process provides a good opportunity to evaluate the predictive capacity of
media content variables. That said, any effort to produce estimates of some
variable sometime in the future would certainly include, if available, recent or
current values of that variable, and since we typically have current polling results
at any given (t-k) point in the campaign, there is nothing preventing us from
including the party’s vote share in the prediction. Indeed, including both media
and lagged vote share together provides a strong test of the degree to which media
content improves the prediction, above and beyond what we would know using
just the current vote share; and a model using just lagged vote share provides a
good baseline against which to measure the contribution of media variables. We
accordingly estimate each model three ways below: (1) vote share at t-k, (2) media
content at t-k, and (3) both vote share and media content at t-k.
Note that for ‘net tone’ we lump together all articles from all newspapers – we do
not give newspapers different weights based on audience reach, nor do we distinguish
between the potentially different content in different newspapers. While newspapers
differ in levels of tone for different parties (Soroka and Andrew, N.d.), however, they
follow very similar trends over the campaign. It is not clear that there is much to gain
by looking at newspapers separately.
8
9
The choice of lag – that is, the value of k – is driven by a combination of
pragmatic and statistical considerations. Pragmatically speaking, the further back
the lags are (the greater the value of k), the further forward we are able to predict.
Too far back, and we cannot make predications until well into the campaign. Of
course, the link between media and vote intentions will also weaken as we move
too far back in time, or too far forward. Based on preliminary tests in our earlier
work, we rely here on media content lagged by 4 to 6 days (inclusive).
The pooled model is very similar. Since polling data is missing a significant
number of days in older elections, however, it does not make sense to keep pollster
dummies on the right-hand side. (The potential impact of a single poll by a
pollster will be entirely soaked up by the pollster dummy.) The autoregressive
component is also dropped, again due to missing values. Note that the
consequence is that our pooled model is predicting vote shares based on media
content alone.
The typical pooled model would include election dummy variables. This raises
several problems for our analysis, however. First, it ensures that our predications
will be (roughly speaking) mean-centered for each party’s polls in each election.
That is, the effect of the election dummy will be to shift our mean prediction of
the Liberal vote share in 1993 to the mean of Liberal votes shares in 1993, our
mean prediction of the Liberal vote share in 1997 to the mean of Liberal votes
shares in 1997, and so on. In doing so, election dummies might give our
predictions an unfair advantage: at least, they greatly reduce the possibility that
our predictions veer far away from the actual vote shares. Perhaps more
importantly, election dummies could not be used for predictions in a new, ongoing
election, and our aim here is to produce a model that can handle not just
postdiction, but prediction as well.
There are nevertheless significant differences in levels of vote shares across
elections — differences which need not be reflected in the levels of tone in media
content. We capture those differences here by including, in each pooled model, the
first campaign-period poll for the party at hand. Doing so provides a baseline —
essentially, a starting point — for each election. It also relies on data that would
be readily available at the start of a new election. The pooled model is, in sum, as
follows:
(2)
Votep,t = α +∑ (ωη*Toneη,t-k) [+ ωη*Pollp,o] + εt .
10
Results
Results from 2004, 2006, and 2008 estimations are included in Tables 1 and 2.
With three lags of four different variables, multicollinearity abounds, leaving the
model is vastly over-specified. Coefficients can be very difficult to interpret; given
that there will be several coefficients capturing what is, substantively speaking, a
single effect. Standard errors will also be inflated by multicollinearity so
coefficients will rarely achieve common levels of statistical significance.9 This is of
course common for prediction models that seek to explain as much variance as
possible, and place little emphasis on interpreting individual coefficients. It
presents no particular problem for our work here, but it does mean that we should
not place too much weight on the individual coefficients.
[Tables 1 and 2 about here]
In an effort to make results more readily interpretable, the first rows show the
summed coefficients (and related standard error) for all six lags of tone relating to
the Conservative Party and Stephen Harper. The same is done for the Liberal
Party and its leaders in the second row. Each provides an omnibus test of the
direction and magnitude of the effects of tone for the two major parties. The
rather obvious expectation is that Conservative tone will be positively related to
Conservative vote intentions and negatively related to Liberal vote intentions,
while Liberal tone will be positively related to Liberal vote intentions and
negatively related to Conservative vote intentions.
We begin with results for the Conservatives in 2004, the first three models in
Table 1. Column 1 shows results using just pollster effects and an intercept. The
Rsq is at .28. But note that while the Rsq provides a summary of the proportion
of variance in y explained by x, it does not provide the piece of information we are
most interested in where prediction is concerned: exactly how close are the
predictions to the future values of y? To better assess this critical piece of
information, we rely here on the Mean Absolute Error (MAE), which captures the
average gap between the prediction and the actual vote intentions.10 For this first
model, the MAE is roughly 1.4. (Note that the MAE is in the same unit as the
dependent variable, so we are talking here about a prediction that on average
misses the Conservative vote share by about 1.4 percentage points.)
Not mentioning that models that exclude an autoregressive component suffer from a
serious omitted variable problem making formal discussion about inference quite
adventurous.
10 On the value of the MAE and SEE a goodness of fit measures in prediction and
forecasting, see Krueger and Lewis-Beck 2005.
9
11
Column 2 shows results for the 2004 Conservative vote share, this time adding
lagged media variables to the series of pollster dummies. First, note that
Conservative and Liberal net tone seems to not be correlated to Conservative vote
share, contrary to what we would expect; the MAE is cut by an impressive .4
percentage points, to .97. The MAE is at its lowest, .47, in the third model (see
column 3), which includes an autoregressive process (at lag 4). That said, the
improvement over the second model is rather limited. Consolidated tone effects are
of the right sign, but not significant.
Column 4 shows the base model for the 2006 general election. The Rsq of .07 is
smaller than what we observed in the 2004 base model and its MAE is quite
sizeable at 3.2. In column 5, we see again a nice improvement in the MAE (a
decrease to 2.4), and effects of the consolidated tone variables that are in line with
our expectations. The negative impact of good coverage on Conservatives’ vote
intention is rather large at -3.7. Adding the autoregressive process in column 6
improves the precision of the prediction substantially, with a decrease of more
than a point (from 2.4 to 1.4). Finally, estimates for 2008 are consistent with
those discussed in the previous paragraph. There is a constant improvement from
base model to full model and consolidated effects are of the right sign and, in some
cases, substantive.
Liberal vote intention models for 2004 and 2006 (in Table 2) offer mixed results.
Column 1 to 3 (2004) shows that adding media content to the model does cut the
MAE by half. Adding the autoregressive process also improves precision
marginally. The impact of media tone is not always as expected, with strong and
positive linear combinations for Conservative tones and weak correlation for
Liberal tones. Results are definitely better for 2006. Column 4 shows that the base
model explains 14% of the variance with an MAE of 3.4. Adding media content in
column 5 drops the MAE by close to one point of percentage with strong and
positive consolidated effect for Liberal media coverage on Liberal vote intention.
Conservative media tone is also of the right sign. Adding the autoregressive
component in column 6 improves the precision by another percentage point.
Columns 7 to 9 in each of Tables 1 and 2 show estimates for 2008. Results here
are not very different from in the previous two elections. For the Conservatives,
the MAE drops from 1.3, to 1.0 to .7; for the Liberals, .77, to .48, to .47. In each
case, media tones make a significant contribution to the precision of our estimates,
and lead to a marked improvement in model fit. For the Conservatives, both
Conservative and Liberal tone are correctly-signed. For the Liberals, Liberal tone
is correctly signed, while Conservative tone has no impact.
12
These 2008 estimations offer a first test of the generalizability of results in Soroka
et al. (N.d.). In 2008, as in previous elections, there is a strong link between
campaign-period media content and vote intentions. As before, the degree to
which media both reflect current vote shares and affect future trends is evidenced
by the strength of models including just media content. Looking across Tables 1
and 2, it is striking that we can explain around half of the variance in major party
vote intentions, and improve predicting precision, using just lagged media.
Estimates for pooled models, combining the six general elections dating back to
1993, are presented in Table 3.
[Table 3 about here]
Column 1 shows the baseline model — including just the contextual-level variable
that takes on the value of the first election poll — for the dominant right-of-centre
(RW) party in each election. This pooled RW model uses polling numbers from
the Progressive Conservatives in 1993, Reform in 1997,11 Canadian Alliance in
2000, and the Conservative Party from 2004 to 2008. It confirms that there is
indeed some campaign-period variance to explain: the estimated coefficient for the
first campaign poll is very high, at .856; even so, this explains 50% of the variance
in our dependent variable. Column 2 shows the model with the media tone
variables added. Both variables are correctly signed and significant. There is 13point increase in the percent of variance explained, and the MAE drops 11%, from
2.9 to 2.6.
Columns 3 and 4 repeat the process for the Liberal Party. Column 3 shows the
baseline model. The first campaign polls in in this case an especially strong
predictor of vote intentions through the campaign, with an estimated magnitude
of .94 and a Rsq of .75. Adding media content in column 4 does nevertheless
improve precision -- the MAE drops by 7% from 2.3 to 2.2. Favorable media
coverage of the Liberal Party and its leaders is associated with higher vote
intention across elections. There is no clear effect of RW coverage on Liberal vote
shares.
[Figure 2 about here]
The addition of media variables thus makes a small but potentially important
contribution to the pooling model of vote intentions. Figure 2 illustrates the
predictive power of both the election-specific (for 2004 through 2008) and pooled
We debated using either Reform or the Progression Conservatives for 1997. As it
turns out, using one or the other makes little difference to the efficiency of the model,
or indeed even the reliability of the RW or Liberal prediction in that year.
11
13
models. Party predictions are made using pollster dummies and media content for
election-specific models, and contextual plus media content variables for pooled
models. In both cases, then, lagged vote share is not included. In 2004, the
election-specific model clearly works better than the pooled model, but the pooled
model performs reasonably well in 2006 and 2008, where it generates predication
that are not noticeably different from the election-specific models. The model also
performs reasonably, or least not shamefully, in 1997 and 2000, where the
predictions show show consistent movement up and down in accord with actual
polling data. 1993 is clearly the worst of the bunch. The general trend in vote
shares is reflected in the predictions, but that reflection is very weak. There is
clearly more happening in 1993 than what we have captured in media content.
Conclusions
This paper has sought to make three additions/revisions to our previous efforts at
predicting campaign-period vote intentions using media content. First, we have
introduced an automated measure of media tone. There is room for improvement
in this tone measure, to be sure. But results are in line with human-coded results,
and are also systematically linked to vote intentions. We take these as signs of
success where the automation is concerned.
We have also tested the predictive power of media content in the most recent 2008
election. Our results largely confirm past work on the 2004 and 2006 elections.
Media content is systematically related to vote intentions. Whether the story is
about media as a highly responsive mirror of opinion, or a highly effective driver
of opinion, we are not sure. But the relationship exists; media content is clearly a
valuable addition to models predicting campaign-period trends in vote intentions.
Finally, we have tested the potential for a pooled model, using all six of the most
recent Canadian federal elections. The model has advantages for (a) producing
trends for past elections, in which little polling data exist, and (b) predicting
trends in the early days of future elections, when there is often little polling data
to work with. Or, at least, the model has potential advantages. At this stage, our
pooled model does reasonably well for recent elections, but not so well for past
elections. There is much more data from recent elections, of course — the mediaopinion relationship in these elections is clearly privileged in the model. And that
relationship does not broadcast especially well, particularly to the 1993 election.
It may be that there is no “generalized” relationship between media content and
opinion across elections. We do not believe this to the be case — and our models
suggest that a generalized relationship exists, albeit a weak one. But there are
14
good reasons to expect that the structure of media tone (as we capture it) and
vote intentions can change over time. The nature of reporting can change, for
instance. Journalists can use different words, affecting our tone measure.
Alternatively, a media-as-mirror account might be as follows: in an age when more
information is becoming more readily accessible, and when journalists are not just
writing articles but blogging (and reacting to blogging), and when there are many
more opinion polls to read about and analyze, the link between media content and
public opinion may be especially strong. New technologies may be important in
what may be the increasing capacity of media to capture upcoming trends in
opinion. The media-as-predictor approach suggests another possibility: in an
environment when newspapers are more homogenous, the media signal is clearer.
Citizens are increasingly responding to similar messages.
This last account does not sit easily with recent on the proliferation and diversity
of news sources in recent years. There are, to be sure, reasons to believe that the
link between media content and vote intentions might be weakening rather than
strengthening, though we see no evidence of that here. Indeed, we cannot even
really be sure that the link is increasing either. Right now, we know just that our
pooled models fare better in more recent elections — elections that are also betterrepresented in the pooled dataset.
The next step for a pooled model, then, might be to allow for variations in the
media-opinion connection over time (without completely losing the value of a
pooled model). And while we focus here on tone, recent work suggests that party
and leader “first mentions” are a powerful predictor of trends in vote intentions
(Daku et al. 2009); combining tone and “first mentions” might yield an especially
powerful model. There is also a good of space for econometric fine-tuning. Wleizen
and Erikson (2001, 2002) have outlined a number of challenges in capturing
campaign effects given various forms of survey error, which we have dealt with
only partly here. Sampling error is only roughly incorporated by weighting data
based on polls' sample sizes, for instance; Pickup and Johnston (2007) offer a
better approach using bayesian hierarchical techniques. The current prediction
model also does not include information about temporal heteroskedasticity, which
is certainly an issue. We expect these refinements will improve the predictive
power of our model, strengthening further still the case for media as a predictor of
campaign-period dynamics.
15
Bibliography
Abramowitz, Alan I. 1988. “An Improved Model for Predicting the Outcomes of
Presidential Elections.” PS: Political Science and Politics 21:843-7.
Andrew, Blake. 2007. "Above the Fold and Behind the Link: A Comparative Analysis
of Election Shortcuts in Newspapers and on the Web." Paper presented at the 4th
European Consortium for Political Research (ECPR) General Conference. Pisa,
Italy
Behr, Roy L. and Shanto Iyengar. 1985. “Television News, Real-World Cues, and
Changes in the Public Agenda.” Public Opinion Quarterly 49(1): 38-57.
Blais, Andre, and M. Martin Boyer. 1996. “Assessing the Impact of Televised Debates:
The Case of the 1988 Canadian Election.” British Journal of Political Science 26(2):
143-164.
Bradley, M.M., & Lang, P.J. 1999. Affective Norms for English Words (ANEW):
Stimuli, Instruction Manual and Affective Ratings. Technical report C-1,
Gainesville, FL. The Center for Research in Psychophysiology, University of
Florida.
Brians, Craig Leonard and Martin P. Wattenberg. 1996. “Campaign Issue Knowledge
and Salience: Comparing Reception from TV Commercials, TV News and
Newspapers.” American Journal of Political Science 40(1): 172-193.
Campbell, James E., and James C. Garand. 2000. Before the Vote: Forecasting
American National Elections. Thousand Oaks, CA: Sage.
Campbell, James E. 2008. “The Trial-Heat Forecast of the 2008 Presidential Vote:
Performance and Value Considerations in an Open-Seat Election.” PS: Political
Science and Politics 41: 697-701
Craig, Richard. 2000. "Expectations and Elections: How Television Defines Campaign
News." Critical Studies in Mass Communication 17(1): 28-44.
Daku, Mark, Adam Mahon, Stuart Soroka and Lori Young. 2009. “Media Content and
Election Campaigns: 2008 in Comparative Context.” Paper presented at the
Annual Meeting of theCanadian Political Science Association, Ottawa ON, May
2009.
Druckman, James N. 2004. “Priming the Vote: Campaign Effects in a U.S. Senate
Election.” Political Psychology 25(4): 577-59.
Erikson, Robert S. and Christohper Wlezien. 2008. “Leading Economic Indicators, the
Polls, and the Presidential Vote.” PS: Political Science and Politics 41: 703-707.
Fan, David P. and Lois Norem. 1992. “The Media and the Fate of the Medicare
Catastrophic Extension Act.” Journal of Health Politics Policy and Law 17(1):
37-70.
Fleiss, JL, B. Levin, and MC Park (2003). Statistical Methods for Rates and
Proportions (3rd ed), Chichester: Wiley
Fletcher, Frederick. 1981. "Playing the Game: The Mass Media in the 1979
Campaign." In Canada at the Polls, 1979 and 1980: A Study of the General
Elections, ed. Howard R. Penniman. Washington, D.C.: American Enterprise
Institute for Public Policy Research.
————. 1991. Reporting the Campaign: Election Coverage in Canada. Toronto:
Dundurn Press.
16
Gelman, Andrew and Gary King. 1993. “Why Are American Presidential Election
Campaign Polls so Variable when Votes are so Predictable?” British Journal of
Political Science 23:409-451.
Graber, Doris A. 1976. "Press and TV as Opinion Resources in Presidential
Campaigns." Public Opinion Quarterly 40(3): 285-303.
Hall-Jamieson, Kathleen and Joseph N. Cappella. 1998. “The Role of the Press in the
Health Care Reform Debate of 1993-1994.” Pp. 110-31 in Doris A. Graber, Dennis
McQuail and Pippa Norris, eds., The Politics of News: The News of Politics
(Washington DC: CQ Press).
Hibbs, Douglas A. Jr. 1982. “President Reagan’s Mandate from the 1980 Elections: A
Shift to the Right?” American Politics Quarterly 10:387-420.
Holbrook, Thomas. 1996. Do Campaigns Matter? Thousand Oaks, CA: Sage.
Iyengar, Shanto Iyengar. 1996. “Framing Responsibility for Political Issues.” Annals of
the American Academy of Political and Social Science 546:59-70.
Jackman, Simon (2005). "Pooling the Polls Over an Election Campaign" in Australian
Journal of Political Science 40(4): 499-517.
Jamieson, Kathleen Hall. 1992. Dirty Politics: Deception, Distraction, and Democracy.
New York: Oxford University Press.
Johnston, Richard, Andre Blais, Henry E. Brady, and Jean Crete. 1992. Letting the
People Decide: Dynamics of a Canadian Election. Kingston, Ont: McGill-Queen's
Press.
Krueger, James S. and Michael Lewis-Beck. 2005. “The Place of Prediction in
Politics.” Paper presented at the Annual Meeting of the American Political Science
Association, Washington D.C.
Krosnick, Jon A., and Donald R. Kinder. 1990. “Altering the Foundations of Support
for the President Through Priming.” The American Political Science Review 84(2):
497-512.
Lewis-Back, Michael S. and Tom W. Rice. 1984. “Forecasting Presidential Elections:
A Comparison of Naive Models.” Political Behavior 6:39-51.
Lewis-Beck, Michael S. and Charles Tien. 2008. “The Job of President and the Jobs
Model Forecast: Obama for '08?” PS: Political Science and Politics 41:687-690.
Lodge, Milton, Marco Steenbergen, and Shawn Brau. 1995. “The Responsive Voter:
Campaign Information and the Dynamics of Candidate Evaluation.” American
Political Science Review 89:309-26.
Martindale, C. 1975. Romantic Progression: The psychology of Literary History.
Washington, D.C.: Hemisphere.
Martindale, C. 1990. The Clockwork Muse: The Predictability of Artistic Change.
New York: Basic Books.
McCombs, Maxwell E. and Donald L. Shaw. 1972. “The Agenda-Setting Function of
the Mass Media.” Public Opinion Quarterly 36(2): 176-187.
McDermott Monika L., and Kathleen A. Frankovic (2003). "Horserace Polling and the
Survey Method Effects: An Analysis of the 2000 Campaign" in Public Opinion
Quarterly, 67(2): 244-264.
Mendelsohn, Matthew. 1996. “The Media and Interpersonal Communications: The
Priming of Issues, Leaders, and Party Identification.” The Journal of Politics 58(1):
112-125.
17
—————. 1994. “The Media's Persuasive Effects: The Priming of Leadership in the
1988 Canadian Election.” Canadian Journal of Political Science 27(1): 81-97.
—————. 1993. "Television's Frames in the 1988 Canadian Election." Canadian
Journal of Communication 18(2): 149-171.
Mendelsohn, Matthew and Richard Nadeau. 1997. “The Religious Cleavage and the
Media in Canada.” Canadian Journal of Political Science 30(1): 129-146.
Mergenthaler, E. 1996. Emotion-abstraction Patterns in Verbatim Protocols: A New
way of Describing Psychotherapeutic Processes. Journal of Consulting and Clinical
Psychology 64(6): 1306-1315.
Mergenthaler, E. 2008. Resonating Minds: A School-independent Theoretical
Conception and its Empirical Application to Psychotherapeutic Processes.
Psychotherapy Research 18(2): 109-126.
Mullen, A. and Collier, N. 2004. “Incorporating Topic Information into Sentiment
Analysis
Models.” In Proceedings of the 2004 Conference on Empirical Methods in Natural
Language Processing (EMNLP 2004), Barcelona, Spain.
Norpoth, Helmut. 2008. “On the Razor’s Edge: The Forecast of the Primary Model.”
PS: Political Science and Politics 41:683-686.
Pang, Bo, Lillian Lee and Shivakumar Vaithyanathan. 2002. Thumbs up? Sentiment
Classification using Machine Learning Techniques. In Proceedings of the 2002
Conference on Empirical Methods in Natural Language Processing (EMNLP-02),
Philadelphia, PA.
Patterson, Thomas E. 1993. Out of Order. New York: Knopf.
Pennebaker, James W., Martha E. Francis and Roger J. Booth. 2001. Linguistic
Inquiry and Word Count: LIWC 2001. Mahway, NJ: Erlbaum Publishers.
Pickup, Mark, and Richard Johnston (2007) "Campaign Trial Heats as Electoral
Information: Evidence from the 2004 and 2006 Canadian Federal Election" in
Electoral Studies 26:460-476.
Riffe, Daniel, Stephen Lacy and Frederick Fico. 2005. Analyzing Media Messages:
Using Quantitative Content Analysis in Research. 2nd Edition. Mahwah, N.J.:
Lawrence Erlbaum.
Roget, Peter Mark. 1911. Roget's Thesaurus of English Words and Phrases,
supplemented electronic version (June 1991). Project Gutenberg Library Archive
Foundation.
Shaw, Daron R. 1999. “A Study of Presidential Campaign Event Effects from 1952 to
1992.” Journal of Politics 61:387-422.
Soroka, Stuart. 2003. “Media, Public Opinion, and Foreign Policy,” Harvard
International Journal of Press and Politics 8(1): 27-48.
————. 2002. Agenda-Setting Dynamics in Canada. Vancouver BC: University of
British Columbia Press.
Soroka, Stuart and Blake Andrew. N.d. “Media Coverage of Canadian Elections:
Horserace Coverage and Negativity in Election Campaigns,” in Linda Trimble and
Shannon Sampert, eds., Mediating Canadian Politics (Pearson).
Soroka, Stuart, Marc André Bodet, Lori Young and Blake Andrew (N.d.). “Campaign
News and Vote Intentions.” Journal of Elections, Public Opinion and Policy.
18
Stone, Deborah. 1989. “Causal Stories and the Formation of Policy Agendas,” Political
Science Quarterly 104(2):281-300.
Stone, P.J., D.C. Dumphy and D.M. Ogilvie. 1966. The General Inquirer: A Computer
Approach to Content Analysis. M.I.T. Press: Cambridge, MA.
Strapparava, Carlo and Alessandro Valitutti. 2004. WordNet-Affect: An Affective
Extension of WordNet. In Proceedings of the 4th International Conference on
Language Resources and Evaluation (LREC 2004), Lisbon, IT.
Turney, P and M. Littman. 2003. Measuring Praise and Criticism: Inference of
Semantic Orientation from Association. ACM Transactions on Information Systems
(TOIS) 21(4).
Whissell, C. 1989. The Dictionary of Affect in Language. In R. Plutchnik and H.
Kellerman, eds. Emotion: Theory and Research. New York, Harcourt Brace,
113-131.
Wiebe, Janyce M. 2000. Learning Subjective Adjectives from Corpora. In Proceedings
of the 17th National Conference on Artificial Intelligence (AAAI-00), Austin, TX.
Wilson, R. Jeremy. 1980. "Media Coverage of Canadian Election Campaigns:
Horserace Journalism and the Meta-Campaign." Journal of Canadian Studies
15(4): 56-68.
Wagenberg, R. H., W. C. Soderlund, W. I. Romanow, E. D. Briggs. 1988. Campaigns,
Images and Polls: Mass Media Coverage of the 1984 Canadian Election. Canadian
Journal of Political Science 21(1): 117-129.
Weaver, David H. 1996. “What Voters Learn from Media.” Annals of the American
Academy of Political and Social Science 546: 34-47.
Wlezien, Christopher and Robert S. Erikson. 2004. “The Fundamentals, the Polls, and
the Presidential Vote.” PS: Political Science and Politics 37:747-751.
—————. 2002. “The Timeline of Presidential Election Campaigns.” Journal of
Politics 64:969-993.
—————. 2001. “Campaign Effects in Theory and Practice.” American Politics
Research 29:419-436.
19
Table 1: Conservative Party Vote Intentions, Annual Prediction Models
DV: Conservative Party Vote Intentions
2004
1
∑ CP
∑ Lib
t-4,5,6
t-4,5,6
2
2006
3
4
2008
5
6
-.990
.771
(1.710)
.610
(1.830)
DV
t-4
t
8
9
.866
2.743
1.929
(1.118)
(.655)
(.895)
(.904)
-3.740
-1.522
-.805
-.231
(1.138)
(.720)
(.829)
(.812)
.353
.792
(1.16)
(.097)
7
Pollster Effects
SES
1.984
2.287
2.342
2.204
1.614
.236
(.649)
(1.229)
(1.210)
(1.125)
(1.891)
(1.120)
-.877
-2.039
-.856
-1.874
-.661
-.543
(.983)
(1.426)
(.848)
(.963)
(1.342)
(1.126)
SC
Decima
Ipsos
-.562
.791
-.677
-1.032
-1.026
-.934
(1.707)
(2.435)
(1.438)
(.863)
(1.194)
(1.598)
.845
(1.076)
-.375
(1.804)
-.920
(2.730)
-1.364
-1.133
-1.614
1.040
.168
-.067
(.756)
(.977)
(1.046)
(1.162)
(1.591)
(.932)
Nanos
Constant
Rsq
N
MAE
31.165
31.047
19.495
31.330
34.376
7.071
36.501
36.348
15.260
(.413)
(2.422)
(10.242)
(.974)
(2.729)
(3.719)
(.958)
(1.754)
(6.896)
.278
.646
.692
.070
.324
.776
.151
.490
.736
31
25
25
61
51
51
36
36
32
1.381
(.986)
.970
(.579)
.894
(.558)
3.189
(1.757)
2.444
(1.775)
1.416
(1.015)
1.280
(.921)
1.008
(.621)
.705
(.507)
Cells contain OLS coefficients with standard errors in parentheses except for MAE which contains mean
values.
Table 2 : Liberal Vote Intentions, Annual Prediction Models
DV: Liberal Vote Intentions
2004
1
∑ CP
∑ Lib
DV
t-4,5,6
t-4,5,6
t
2006
2
3
5
6
8
9
2.563
1.859
-.245
-.788
.249
.115
(1.032)
(1.121)
(1.136)
(.722)
(.525)
(.571)
.328
1.098
4.166
2.499
.807
.722
(1.104)
(1.205)
(1.157)
(.767)
(.486)
(.547)
t-4
4
2008
.290
.767
(.215)
(.106)
7
Pollster Effects
SES
-.584
-.350
-.105
-1.390
-1.659
.580
(.625)
(.742)
(.736)
(1.177)
(1.921)
(1.254)
SC
Decima
Ipsos
-.881
.557
-1.113
.440
.757
1.044
(1.028)
(1.449)
(.945)
(.661)
(.787)
(.809)
-1.088
-2.119
-1.134
-.422
-1.016
-.500
(.592)
(.700)
(1.206)
-.934
(.739)
1.394
(1.058)
.520
(1.977)
(1.786)
(2.474)
(1.571)
-.789
-.623
-.782
-2.447
-1.319
.220
(.729)
(.598)
(.579)
(1.215)
(1.617)
(1.044)
Nanos
Constant
Rsq
N
MAE
33.701
30.281
21.251
35.710
32.173
6.884
27.001
23.979
18.551
(.398)
(1.461)
(6.843)
(1.019)
(2.773)
(3.921)
(.658)
(1.029)
(4.888)
.084
.614
.678
.141
.422
.775
.108
.609
.635
31
25
25
61
51
51
36
36
32
1.062
(.823)
.538
(.420)
.468
(.412)
3.408
(1.626)
2.508
(1.769)
1.546
(1.125)
.769
(.601)
.483
(.497)
.474
(.369)
Cells contain OLS coefficients with standard errors in parentheses except for MAE which contains mean
values.
Table 3 : Liberal and RW Vote Intentions, Annual Prediction Models,
Pooled Data
DV: Vote Intentions t (1993-2008)
RW
1
∑ RW
∑ Lib
t-4,5,6
t-4,5,6
Liberal
2
3
1.956
0.404
(0.525)
(0.422)
-2.621
1.844
(0.581)
First Campaign Poll
Constant
0.856
(0.067)
4
0.866
(0.076)
(0.463)
0.944
(0.043)
0.921
(0.060)
4.958
4.876
-0.443
-1.852
(2.107)
(2.588)
(1.573)
(2.188)
Rsq
0.5
0.629
0.746
0.784
N
163
142
163
142
2.893
(2.903)
2.572
(2.380)
2.333
(1.969)
2.188
(1.724)
MAE
Cells contain OLS coefficients with standard errors in parenthèses except for
MAE which contains mean values.
Appendix Table A1: Stories by Newspaper and Election
Election
Calgary Herald
Globe and Mail
Montreal Gazette
National Post
Ottawa Citizen
Toronto Star
Vancouver Sun
1993
1997
2000
2004
2006
2008
Total
908
506
941
0
0
1443
904
516
672
612
0
0
841
507
653
785
672
0
0
921
563
656
968
632
862
0
945
595
940
1455
833
1001
0
1152
737
611
828
635
443
766
1040
354
4284
5214
4325
2306
766
6342
3660
Figure 1. 2006 Net Tone, automated versus manual coding
Figure 2. 1993-2008 predictions