Treating Words as Data with Error:
Estimating Uncertainty in the Comparative Manifesto Project
Measures∗
Kenneth Benoit
Trinity College Dublin
[email protected]
Michael Laver
New York University
[email protected]
Slava Mikhaylov
Trinity College Dublin
[email protected]
September 13, 2007
Abstract
Spatial models of party competition are central to political science, but can only be tested empirically if there exist reliable and valid measurements of agents’ positions on salient policy dimensions.
To date, the main source of cross-national, time-series data on party policy positions comes from
from the content analysis of party manifestos by the Comparative Manifesto Project (CMP), cited
in hundreds of published studies since the first publication of its results in 1987. Despite the very
widespread use of these policy position estimates for studies published in the best journals in our
discipline, however, and despite the fact that estimates in these data arise from documents coded
only once by one of a handful of CMP coders, the level of uncertainty associated with each CMP
estimate has never been characterized. This greatly undermines the value of the CMP dataset as a
scientific resource, since even the CMP’s creators agree that their dataset contains several types of
error. We propose a remedy to this situation. First, we outline the process by which the CMP data
are generated, characterizing the error that arises both from documented tests of coder unreliability,
as well as from the inherently stochastic process by which latent party positions are recorded by
manifesto authors into observable texts. Second, using previously ignored coding-unit frequencies
reported by the CMP project, we reproduce the error-generating process by simulating coder unreliability and bootstrapping analyses of coded quasi-sentences to reproduce both forms of error. This
allows us to estimate precise levels of non-systematic error for every category and scale reported
by the CMP for its entire set of 3,000+ manifestos. Finally, using our estimates of these errors, we
demonstrate ways to correct otherwise biased inferences from statistical analyses of the CMP data,
and issue recommendations for future research based on CMP measures.
Key Words: Comparative Manifesto Project, mapping party positions, party policy, error estimates,
measurement error.
∗ A previous version of this paper was prepared for the 2007 Annual Meeting of the American Political Science Association,
Hyatt Regency and Sheraton Chicago, Chicago, Illinois, August 30–September 2, 2007. This research was partly supported
by the European Commission Fifth Framework (project number SERD-2002-00061), and by the Irish Research Council for
Humanities and the Social Sciences. We thank Andrea Volkens for generously sharing her experience and data regarding the
CMP, and Thomas Daubler for research assistance. We also thank James Adams, Garrett Glasgow, Simon Hix, Abdoul Noury,
and Sona Golder for providing and assisting with their replication datasets and code. An outline form of this paper was drawn
up for the Conference on the Dynamics of Party Position Taking, March 23-24, 2007, University of Binghamton.
1
Using text to measure policy positions
Political text, properly analyzed, offers extraordinary potential as a source of information about the
policies, preferences and positions of political actors, information vital to many models at the heart of
modern political science. The authors of political text may be single individuals such as candidates,
prime ministers or judges; they may be collective entities such as political parties, courts or cabinets.1
Research designed to measure policy positions using political text has, until now, largely been limited by
a lack of suitable methods for systematically extracting information from the vast amounts of suitable
text available for analysis. Recent methodological advances have made some progress in this area,
perhaps most importantly by breaking from traditional forms of content analysis to treat text, not as an
object for subjective interpretation, but very explicitly as objective data from which information about
the author can be estimated in a rigorous and replicable way (e.g. Slapin & Proksch 2007, Monroe &
Maeda 2004, Laver, Benoit & Garry 2003, Laver & Garry 2000). Treating words as data opens up the use
of well-understood methods of statistical analysis to allow inferences to be drawn from the observable
content of authored text about the fundamentally unobservable underlying characteristics (for example
policy positions) of its author. The main advantage of this statistical approach is that, relying on wellestablished procedures for rigorously characterizing patterns in data, it eliminates both subjectivity and
the propensity for human error, while making the results of text-based analysis easily replicable. One
particularly important consequence of treating texts as data and basing inferences on solid statistical
procedures, furthermore, is that measures of uncertainty accompany any resulting estimates—a feature
now recognized not just as a bonus, but as a sine qua non, for serious empirical research in the social
sciences (King, Keohane & Verba 1994, 9).
While our methodological focus in this paper is on the analysis of text in general, we do have a
more specific type of text in mind. This is the text contained in party manifestos, an enormous volume
of which has been analyzed, using human coders, by the Comparative Manifestos Project (CMP), in a
twenty-year effort representing one of the most long-running and ambitious data projects in the history
of political science. The result of this truly Herculean effort is a dataset reporting estimates of the policy
positions of a large number of national political parties in a large number of countries over the entire postwar period.2 First reported in 1987 (Budge, Robertson & Hearl 1987), a hugely expanded version of this
1 Of
course there are many alternative ways to measure political positions, including but not limited to the analysis of:
legislative roll calls; survey data on preferences and perceptions of political elites; survey data on preferences and perceptions
of voters; surveys of experts familiar with the political system under investigation; the analysis of political texts generated by
political agents of interest. Benoit and Laver (2006) review and evaluate these different approaches.
2 We also note, however, that the CMP is not the only text-based measure that is based on party manifestos: Laver & Garry
2
dataset was reported in the project’s core publication, Mapping Policy Preferences (Budge, Klingemann,
Volkens, Bara & Tanenbaum 2001, hereafter MPP), to have covered thousands of policy programs,
issued by 288 parties, in 25 countries over the course of 364 elections during the period 1945-1998. The
dataset has recently been extended, as reported in the project’s most recent publication Mapping Policy
Preferences II (Klingemann, Volkens, Bara, Budge & McDonald 2006, hereafter MPP2) to incorporate
1,314 cases generated by 651 parties in 51 countries in the OECD and central and eastern Europe (CEE).
For scholars seeking long time series of party policy positions in many different countries, the
CMP dataset is effectively the only show in town. Since, commendably, these data are freely available
as a public resource for the profession, researchers have indeed used them very widely, as evidenced
by over 800 Google Scholar citations by third-party researchers of core CMP publications.3 However,
despite near-universal recognition within the profession of vital need to measure and report uncertainty
in all social scientific data analysis, the CMP data, as used in every one of the very many pieces of
published research to date, come with no measures whatsoever of uncertainty.
Substantively, the range of research applications using the CMP data is vast, encompassing four
major areas of political science: descriptive analyses of party systems (e.g. Bartolini & Mair 1990, Evans
& Norris 1999, Mair 1987, Strom & Lejpart 1989, Webb 2000); empirically grounded analyses of party
competition (e.g. Adams 2001, Janda, Harmel, Edens & Goff 1995, Meguid 2005, Van Der Brug, Fennema & Tillie 2005); models of coalition building and government formation (e.g. Baron 1991, Schofield
1993); and measuring responsiveness of representative democracy in linkages between government programs and governmental policy implementation (e.g. Petry 1991, Petry 1988, Petry 1995). Additional
applications of the CMP data include the analysis of American political behavior (e.g. Erikson, Mackuen & Stimson 2002); evaluation of partisan effects on government expenditure (e.g. Bräuninger 2005);
identification of structure and dimensionality of the political space of European Parliament, European
Commission and Council of Ministers (e.g. Thomson, Boerefijn & Stokman 2004); evaluation of the
issue convergence in US presidential campaigns (e.g. Sigelman & Emmett 2004); analysis of the relationship between budgetary cycles and political polarization and transparency (e.g. Alt & Lassen 2006);
evaluation of partisan effects on trade policy (e.g. Milner & Judkins 2004); and establishing the effect of
endogenous deregulation on productivity in OECD (e.g. Duso & Röller 2003); The CMP data has also
been used as a means to validate other measures of parties’ policy positions, such expert surveys and
(2000), Laver, Benoit & Garry (2003), and Slapin & Proksch (2007) are also examples.
3 As of August 25, 2007. The precise number of third-party citations is hard to calculate because third-party users are likely
to cite several CMP sources in the same paper.
3
automated content analysis (e.g. Laver, Benoit & Garry 2003, Ray 1999).
Notwithstanding this range and volume of applications, the CMP measures of party policy positions come with no measure of the uncertainty surrounding their point estimates, which lie on very
precise continuous scales (0–100 or -100–100). While this problem has long been acknowledged by
the project and its critics (e.g. Benoit & Laver 2007, Klingemann et al. 2006, Ch. 5), no conclusive
solution has yet been offered to the problem of providing uncertainty measures for any of the policy
scores reported by the CMP. Reliable and valid use of the CMP data, however, mandates to use of such
measures. Without them, users of the data cannot distinguish between “signal” and “noise”, making it
impossible to tell the difference between measurement error and “real” differences in policy positions.
Compounding this problem, the overwhelming use of CMP estimates of party policy positions is as
explanatory variables. Measurement error in these CMP variables, if not taken into account, will yield
biased inferences about key causal relationships and thus misleading research findings. In short, the
unknown level of non-systematic error contained in the CMP dataset drastically undermines its primary
value for the profession, as a reliable and valid set of estimates of party policy positions across a wide
range of years, countries and policy dimensions.
In this paper, we set out to address this problem head on, and thereby to enhance the value of
the CMP data for the profession as a whole. We do this by decomposing stochastic elements in the
data generation process that leads to the reported CMP estimates and simulating the effects of these.
This enables us to obtain precise estimates of the non-systematic error associated with every reported
CMP measure of party policy positions. The result of our analyses of these stochastic processes is a set
of standard errors for the CMP data that allow scholars to make more informed judgments, when they
compare two CMP estimates of different parties at the same time or of the same party at different times,
about whether these differences are likely to be “real”, or simply the result of stochastic features of the
data generation process.
Our analysis proceeds as follows. First, we briefly describe the CMP dataset and the processes
that led to its generation. We then analyze the two main stochastic processes that are involved: manifesto
writing and manifesto coding. Based on these analyses, we show how to calculate standard errors for
each estimate in the CMP dataset. Analyzing these error estimates, we find that many CMP quantities
should be associated with substantial uncertainty. Exploring the effects of this uncertainty on substantive
work, we use show how the error estimates can be brought to bear in judging substantive change versus
measurement error in both time-series and cross-sectional comparisons of party positions. Finally, we
4
suggest ways to use our error estimates to correct analyses that do use CMP data as covariates, re-running
and correcting some prominent analyses reported in recent literature.
2
The CMP coding process
In order to characterize the error in the CMP dataset, we must begin by understanding the process
which gives rise to the observed estimates. First, researchers who spoke the language of the country
under investigation were asked to identify a “text” for a particular party at a particular election, which
was either a formal party “manifesto” or, when this was not available, some valid manifesto surrogate.
Second, the selected text was read by a trained human coder and broken down into what the CMP
calls “quasi sentences”. These are defined as self-contained text units, each expressing a distinct policy
message, which could have been written as stand-alone sentences had the manifesto been written by
a different author with different stylistic proclivities. “Language is...complex, and different forms or
styles may be used to express the same political ideas” (Klingemann et al. 2006, 166). The identification
of natural sentences only requires a coder be able to recognize a simple punctuation mark when s/he
sees it; but the identification of quasi-sentences: (a) fundamentally recognizes the inherently stochastic
nature of text-generation; (b) requires a fundamentally subjective judgment that might well vary between
coders. In practice, as we see below, the total number of quasi-sentences identified for a given manifesto
does indeed vary substantially between different coders. Despite this potential variance in even the basic
text units to be coded, all estimates reported in the CMP dataset are based on manifestos coded once,
and only once, by a single coder.
Having broken the text under analysis into quasi-sentences, the coder then assigns each quasisentence to one of the 56 policy categories that form the CMP coding scheme. A quasi-sentence such
as “...incentives for employee share ownership,...” is assigned to category 202, “Democracy: Positive”
(example taken from Klingemann et al. 2006, 177).4 If no suitable category can be identified, the quasisentence is marked as “uncoded.” The proportion of uncoded quasi-sentences in each manifesto in the
CMP dataset is typically in the range 0-5 percent. Once a manifesto has been coded in this way, the
number of quasi-sentences coded into each content category is counted, and reported as a percentage of
4 This quasi-sentence is taken from the sample coded manifesto from the British Liberal/SDP Alliance in 1983 (Klingemann
et al. 2006, 177). The CMP coding scheme is based on a 56-category coding scheme developed in the mid-1980s, building
on earlier work by David Robertson (1976). The coding of party manifestos by CMP researchers has been very extensively
described in the CMP’s core publications (MPP and MPP2). Note that here we do not evaluate the substantive validity of this
scheme, noting only that the particular 56-category CMP coding scheme that generated these data is one of many possible
coding schemes that might have been devised.
5
the total number of quasi-sentences in the manifesto. These percentages, reported by coding category,
are the fundamental quantities in the CMP data that are used by third-party researchers.
In practice, most scholars do not use the full 56-category set of CMP coding categories; rather,
they seek a low-dimensional representation of party policy positions. It is very common practice, therefore, for scholars to use the composite “left-right” scale developed and tested by the CMP. Based on
subjective judgments about the meaning of particular categories, supplemented by exploratory factor
analyses, this simple additive scale combines information from 26 CMP policy variables, 13 referring
to left wing positions and 13 referring to right wing positions. The CMP refers to the measurement
of party movement on this left-right positional scale as its “crowning achievement” (MPP, 20) and has
relied extensively on such left-right movements when establishing the face validity of its estimates. We
do not examine the substantive validity of this scale here. We do, however, devote considerable attention
to an analysis of potential errors in the estimation of this left-right scale, since it by far the CMP measure
most widely used by third party researchers.
3
The stochastic process generating CMP data
The reason to analyze manifestos is that they provide authoritative statements of party policy at the time
of elections. If we treat these texts as data, however, we must understand the processes giving rise to
these data, especially whether these processes generate systematic and/or non-systematic error. Thinking
about the analysis of party manifestos, consider the stochastic relationship between: the unobservable
party position(s) of a manifesto’s author(s); the observed text deposited in the party manifesto; and the
coded dataset generated from this text. We view this relationship as having two main elements. The first
involves manifesto authorship, the stochastic generation of observed data, in the form of text deposits
in manifestos, from non-stochastic unobserved party positions. Put simply, two different authors, or
one author at two different times (perhaps after hard disk crash), are almost certain to generate two
somewhat different manifestos, even when doing their utmost to express an identical party position.
The observed manifesto is one of many different manifestos that could have been written to express
precisely the same position. A second stochastic process involves manifesto coding, given the particular
manifesto text deposited on the record. Two different coders, or the same coder working at two different
times, are very likely to code the same manifesto in somewhat different ways. To begin with, the
guidelines for determining what constitutes the basic text unit for the CMP coding scheme—the “quasisentence”—are quite difficult for different coders to follow consistently, leading to a stochastic process
6
in the dissection of manifestos into text units, before any coding takes place.5 As a consequence of the
number of quasi-sentences in any given manifesto being a matter of subjective judgment (by a single
coder), the basic number of coded units observed for any manifesto forms a random variable. Following
this, the particular allocation of a particular text unit to a particular coding category is also subject to
variation between two coders setting out to do precisely the same thing.
In what follows, we elaborate on these two different aspects of the stochastic process generating
the observed CMP data.
3.1
From unobserved policy position to written text
Our characterization of the text generation process begins by assuming there is indeed some underlying
policy position that text authors set out to communicate. This is shown on the left hand side of Figure
1. We need not concern ourselves here with whether this position is “sincere” or “strategic”, nor with
whether or not there is internal party conflict over what the party position should be. We are interested
solely in the position that text authors set out to communicate, for whatever reason. Given this position—
fundamentally unobservable by anyone else but common knowledge to the authors of the text—a text
is generated to do this job. Our point is that many different texts could be generated to communicate
the same position. There is nothing “golden” about the actual published text, although the generation
of policy texts is a mysterious and idiosyncratic process, about which our best information is largely
anecdotal. Thus, for example, we do know that drafts of eventually-published policy texts are typically
amended many times, often right up the last minutes before publication. This means there are typically
many different actual texts, stored in word processors, filing cabinets and trash cans; but only one of
these texts is eventually published. To this we add the very many conceivable policy texts that could
have been drafted by authors earnestly setting out to communicate precisely the same position. Thus
if you and I set out to draft the same party manifesto containing precisely the same policy messages,
it would be eerie beyond belief, vanishingly improbable, for us somehow to generate two texts that
were word-for-word identical. Looking at this another way, imagine the universe of possible 10,000word texts that could be generated using a 5,000-word political lexicon. There is a gigantic number of
5 If we were to go further, we could also identify the problem of category “seepage”, which happens when a text unit
is related to several categories and therefore prone to a degree of inherent misclassification, a process likely to cause bias
whereby some true categories πi j are over-represented by empirically observed categories pi j . While this problem is widely
acknowledged in CMP work (e.g. MPP2, 116; Volkens 2001a, 38) and certainly adds additional error to the CMP dataset, here
we stick with the assumption of Equation 1 which states that whatever misclassification problems might exist in the coding
process, they do not result in systematic bias. Additional forms of systematic error, which we do not consider here, arise
from the potential for which arises because coders know, before they start coding, which party generated the manifesto under
investigation, or because certain coders tend to use certain categories more than other coders.
7
possible different texts in this universe and it would clearly be ridiculous to claim that every single one
of these communicates a unique policy position.
[FIGURE 1 ABOUT HERE]
Working within the CMP’s implicit measurement assumption that the universe of possible policy positions is spanned by a set of fixed positions 56 different issue dimensions—where differences
in position between parties “consist of contrasting emphases placed on different policy areas” (Budge
et al. 2001, 82), we characterize the fixed policy positions as a set of proportions in each coding category
(rescaled by the CMP as percentages) that directly reflect positions. This is works within the CMP view
of policy positioning, grounded in a “saliency” theory of party competition (see Budge et al. 2001, Ch.
3). For a given policy category j, we define πi j as the true, unobservable policy position of countryparty-date unit i. The j categories in this case are the 56 policy categories in the CMP coding scheme,
plus an additional category for “uncoded”, giving a total of k = 57 categories. Since, according to the
CMP’s measurement model, these true policy positions are represented by relative or “contrasting” emphases on different policy categories within the manifesto, these policy positions are relative proportions,
with ∑kj=1 π j = 1.6 For example, party i’s emphasis, for a given election, on the 20th issue category in
the CMP coding scheme (401: Free Enterprise), is represented as πi20 .
Although we never directly observe the πi j “true” policy positions of manifesto authors, it is
possible, using the CMP’s coded manifesto data, to measure the relative emphasis that each πi j is given
in party i’s manifesto, measured as p j , . . . pk , where p j ≥ 0 for j = 1, . . . , k and ∑kj=1 p j = 1. In the
absence of systematic error (bias):
E(pi j ) = πi j
(1)
In other words, the relative emphasis given in a party’s manifesto will on average reflect the true,
fixed, and unobservable underlying position πi j . Its particular manifestation in any given manifesto,
however, reflects the imperfect and stochastic process of text authorship, yielding the observed proportions pi j . Put another way, every time a manifesto is written with the intention of expressing precisely
the same underlying positions πi j , we expect to observe a slightly different values pi j .
Given the way we characterize both observed and unobservable policy positions—which directly
6 In what follows, we refer to these quantities as policy “positions.” The CMP’s saliency theory of party competition is
neither widely accepted nor indeed taken into any account by most third-party users of CMP data. However, inspection of the
definitions of the CMP’s coding categories reveals that all categories but one of the 56 are very explicitly positional in their
definitions, which refer to “favorable mentions of...”, “need for...,” etc. The sole exception is PER408 “Economic goals”, a
category which is (quite possibly for this reason) almost never used by third-party researchers. For this reason, we do not regard
it as in any way problematic that third-party users almost invariably interpret the CMP’s “saliency” codings as “positional.”
8
follows the CMP’s own assumptions—we can go further and derive a statistical distribution for the observed policy positions. If we consider that each quasi-sentence’s category is determined independently
from that of other quasi-sentences, then we can characterize the CMP’s manifesto coding scheme as
corresponding to the well-known multinomial distribution with parameters ni and πi j , where ni refers
to the total number of quasi-sentences in manifesto i. The probability for any manifesto i of observing
counts of quasi-sentences xi j from given categories j is then described by the multinomial formula:
Pr(X j = x j , . . . , Xk = xk ) =



xk
x1
n!
x j !···xk ! π1 · · · πk

 0
when ∑kj=1 x j = n
(2)
otherwise
Translating into the CMP coding scheme, (for a given manifesto) each xk represents the observed
number of quasi-sentences that occur in a given category j. In terms of the “PER” or percentage categories reported by the CMP for each manifesto, what is actually reported is xi j /ni j 100, or the estimate
of manifesto i’s “true” percentage (πi j 100) of the quasi-sentences from category j. For our purposes, we
have no additional information to suggest any reason why there would be a systematic function mapping (in a biased way) the true position to a different expected observed position—already expressed
by Equation 1.7 Our concern here is with non-systematic (unbiased) error, which is the extent to which
Var(pi j ) > 0, even though πi j is fixed at a single, unvarying point.
So far we have considered only the case of a “given” manifesto, but of course the combined CMP
dataset set deals with many such units—a total of 3,018 separate units representing different combinations of country, election date, and political parties for the combined (MPP + MPP2) datasets.8 If we
are to fully characterize the error from the stochastic process whereby texts are generated, then this will
mean estimating Var(pi j ) for every manifesto i for all k = 57 categories.
3.2
From written manifesto to CMP-coded text units
The second stochastic process which we identify here concerns the way that a manifesto, once authored,
published, and effectively “fixed”, is then translated into the CMP text units and assigned a code. Here
7 Despite the clearly observable and striking variations in manifesto length and content, in other words, we know of no
systematic expectation, based on empirical evidence or even pure theory, about the nature of the stochastic process that maps
actors’ (unobservable) political positions to observed manifesto content. The process whereby the unobserved parameters πi j
are generated, in other words, is something of a mystery that we simply assume to be revealed by the coded manifesto dataset
itself.
8 It is not quite accurate to state that the dataset represents 3,018 separate manifestos, since some of these countryelection-party units share the same manifesto with other parties (progtype=2) or have been “estimated” from adjacent parties
(progtype=3). See Appendix 3, MPP. The full CMP dataset also failed to provide figures on either total quasi-sentences or
the percentage of uncoded sentences for 141 manifesto units, limiting the sample analyzed here to 2,877 units.
9
we focus on one particular aspect of this process, the identification of the “quasi-sentences” that are
coded and counted as the basis of the policy estimates reported. The right hand side of Figure 1 shows
what happens between the generation of an observed party manifesto and the eventual characterization
of this document in the CMP dataset.
[TABLE 1 ABOUT HERE]
The task of parsing a manifesto into quasi-sentences is significant because policy texts not only
vary widely in length, but also express separate, politically relevant ideas (or “arguments” in CMP parlance) within single natural sentences. One thing clear from any investigation of the CMP dataset, for
example, is that the lengths (or ni ) of the coded manifestos vary significantly, as shown in Table 1. Although the total number of quasi-sentences identified in each manifesto is reported the CMP dataset,
(and summarized in Table 1), this quantity is almost never referred to by subsequent users of the data.
However, about 30 percent of all manifestos had less than 100 quasi-sentences, coded into one of 56 categories. Some manifestos had less than 20 quasi-sentences; some had more than 2000 quasi-sentences.
Despite this wide variation in the amount of policy information in different manifestos, all estimated
party policy positions are almost invariably treated in the same way, regardless of whether they are
derived from a coding of 20 text units, or 2000.
The total number of quasi-sentences found in any one manifesto appears to be—at least for lack of
any systematic information or prior expectations on this topic—unrelated to any of the political variables
of interest. Yet assuming that the proportions πi j remain the same irrespective of document length, then
increasing the length of a manifesto should also increase our confidence about the estimates of these
proportions. This reflects one of the most fundamental concepts in statistical measurement: the notion
that uncertainty about an estimate should decrease as we add information to that estimate.
[TABLE 2 ABOUT HERE]
The problem of short manifestos and thus of sparse data is compounded by the fact that not all
quasi-sentences can be coded by the CMP scheme. For instance, while immigration is self-evidently an
important subject of political debate these days in many different countries, there is no unambiguous
coding category in the CMP scheme in which to record statements about immigration, asylum-seekers,
or labor market restrictions to foreigners. Table 2 describes the frequencies of manifestos with varying
levels of uncodable content, as a percentage of the total number of quasi- sentences. While the median
percentage of uncoded content is quite low, at 2.1%, the top quarter of all manifestos contained 8% or
more of uncoded content, and 10 percent contained 21% or more uncoded content. The saliency theory
10
of party competition that motivated the original CMP project led to the methodological assumption that
party policy in particular areas can be measured in terms of the percentage of total manifesto content
coded into each category—in effect assuming that ∑ j πi j = 1. This means that increasing the proportion of uncodable content will necessarily decrease the estimated proportions of coded content. This
problem has been noted previously (e.g. Kim & Fording 1998) in the context of the left-right scale in
particular, but affects all policy categories. Since (unpredictably variable) uncoded content affects the
policy estimates that are calculated from coded content, it does introduce a further stochastic element in
the manifesto generation process.9
CMP researchers have long been aware that different coders can code the same quasi-sentence in
different ways, quite apart from finding a different set of quasi-sentences in the same manifesto. The
ideal way to drive towards more reliable estimates, assuming no coder is perfectly reliable, is of course to
ask many different human coders to code each text and then combine this information in a conventional
way to produce a set of estimates with associated uncertainty. The resource implications of doing this
are so enormous, however, that it is simply never going to happen; the reality is that nearly all of the
3,000+ manifestos that form the basis of the CMP dataset were coded once, and once only, by a single
coder. A second-best solution to minimizing coder unreliability, adopted by the CMP, has been to devote
increasing levels of care and attention to the training of hired coders.10 The objective of this training
has been to reduce the variance in human coders’ application of the same coding scheme to the same
“ master” text, selected by the CMP and reproduced in (Klingemann et al. 2006, Appendix 2). This
training process does also generate limited empirical evidence about levels of variation in the qualitative
judgments made about the same text by different coders. This is reported in various publications by
Andrea Volkens, who supervised this process over a long period of time (Volkens 2001a, Volkens 2001b,
Volkens 2007). Even when coders are given a second chance to code the training document, following
comparison to the gold standard “master” coded document, the best average correlation of coders is still
only 0.88; before correction, this proportion can be as low as 0.72 (Volkens 2001a, 39).
Because our concern here is limited to non-systematic forms of error in the coding process, we
focus only on the stochastic process whereby a written manifesto is (subjectively) parsed into quasi9 When
uncoded content consists of purely non-political sentences, then this still affects the political sentence codings
because of the additive assumption that position is determined by the proportion of mentions. In the context of discrete choice
models such as the multinomial logit, this problem is known as a violation of the IIA principle. When uncoded content consists
of political content that the CMP scheme cannot capture, the problem is likely to be a more serious problem of bias, although
we do not investigate that possibility here.
10 Despite the considerable attention now devoted by the CMP to the training of hired coders, it is also the case that manifestos
from early CMP publications, and much of the current CMP data associated with this, were coded by senior researchers in the
original project who were, to all intents and purposes, completely untrained.
11
sentences by the human coder. An analysis of the results from repeated codings of the training document
used by the CMP to initiate new coders by Volkens (2001b) reveals important insight into the deviation of
coders from the “correct” quasi-sentence structure. Volkens reports that average deviation from correct
quasi-sentence structure in reliability tests of thirty-nine coders employed in the CMP was around ten
percent. These results suggest that, even using the best estimates from coders who have been corrected
and given a second chance to get it right, different coders nonetheless make very different judgments
made during the first step of parsing a document into “quasi”-sentences. In the tests we have analyzed
ourselves, involving 59 different CMP coders, we found that these identified between 127 and 211 quasisentences in the same training document, with a SD of 19.17 and an IQR of (148, 173).11 As mentioned
earlier, an inter-coder variance in counting quasi-sentences in the tests is typically +/-10%, a conclusion
also reached by Volkens (2001b, 38). Because these correlations refer only to totals of quasi-sentences
identified, there is no measure of the correspondence of the actual parsed sentences themselves. This
means that there may well be many more differences between the quasi-sentence structures identified by
different researchers than those we report here. Nonetheless on the basis of these reliability test results,
we can make the assumption that
E(n̂i ) = ni
(3)
Var(n̂i ) = (0.10 ni )2
(4)
ni ∼ N(n̂i , 0.10n̂i )
(5)
where n̂i refers to the to the total number of quasi-sentences reported for each manifesto i in the CMP
dataset, and (as before) ni refers to the “true” or correct number of quasi-sentences in manifesto i. Using
this characterization of variance, combined with the the multinomial characterization of the stochastic
manifesto authorship process, we are now in a position to estimate the errors for each policy category
for each manifesto.
11 We thank Andrea Volkens for kindly sharing not only this dataset with us, but also her vast experience in dealing with
different coders over the many years of the CMP. In reporting the variance of the quasi-sentence length of the training document,
we have deliberately not reported the “correct” text length since this document is still used by the CMP for training coders.
12
4
4.1
Estimating Error in Manifesto Generation
Analytical error estimation
One way to assess the error variance of estimated percentages of text units any of the CMP’s 56 coding
categories is using analytic calculations of variance for the two error distributions we have stipulated,
the multinomial and normal distributions. For the normal distribution which describes the stochastic
distribution of total quasi-sentences, calculating the variance for any manifesto length is trivial, since
(by Equations 4) this is a direct proportion of the observed (by Equation 3) total quasi-sentences. The
discussion here focuses on the slightly more complicated case of computing the analytic variance for
the 56 policy categories. The target is to determine the variance of the percentage (“PER”) categories
reported by the CMP, which in the language described above represent π̂i j 100 for each category j and
each manifesto i. Here we are assuming no bias (i.e. Equation 1), where each πi j represents the true but
unobservable policy position of country-party-date unit i on issue j.
Returning to the definition of the multinomial distribution in Equation 2, for any multinomial
count Xi j , the variance is defined as
Var(Xi j ) = ni pi j (1 − pi j )
(6)
With a bit of algebraic manipulation (detailed in Appendix A) we can express the variance of the proportion pi j , and the rescaled percentage (used by the CMP as):
1
pi j (1 − pi j )
ni
(7)
SD(pi j 100) =
100
√
ni
(8)
SD(pi j ) ∝
1
√
ni
Var(pi j ) =
q
pi j (1 − pi j )
In part, then, the error will depend on the size of the true percentage of mentions pi j 100 for each “
PER” category j. Assuming this quantity is fixed for each party-election unit i, however, what is variable
as a result of the data generating process is the length ni of the manifesto that the party associated with i
deposits. This aspect of the error in the CMP estimates, therefore, is inversely proportional to the (square
root of the) length of the manifesto. This should be reassuring, since it means that longer manifestos
will reduce the error in any CMP category, irrespective of its proportion p j . Longer manifestos provide
13
more information, in other words, and therefore we can be more confident about codes constructed from
them.
The situation is slightly more complicated for additive measures such as the pro-/anti-EU scale
(PER108-PER110) or for RILE, the additive category obtained by summing 13 categories on the “
left” and subtracting 13 categories on the “right”. This is because for summed multinomial counts, the
covariances between categories must also be estimated, since it is a property of variance that Var(aX +
bY ) = a2 Var(X)+b2 Var(Y )+2abCov(X,Y ). There are several strong reasons—including coder practice
as well as innate relationships between mentions of related categories—to suspect that these covariances
will be non-zero. For these reasons, we do not recommend using the analytically derived errors in
practice, since the simulated errors developed in the next section will allow us to confront the unknown
covariances directly, as well as to combine the multinomial and normal variances into a single measure
of estimated error.
4.2
Estimating Error Through Simulation
Given the potential analytical problems we identify at the end of the previous sections, we also suggest
a way to assess the extent of error in the CMP estimates using simulations that recreate the stochastic
processes that led to the generation of each manifesto, based on our belief that there are many different
possible manifestos that could have been written to communicate the same underlying policy position.
This is done by bootstrapping the analysis of each coded manifesto, based on resampling from the set
of quasi-sentences in each manifesto reported by the CMP. Bootstrapping is a method for estimating the
sampling distribution of an estimator by resampling with replacement from the original sample. Bootstrapping has three principal advantages over the analytic derivation of CMP error from the previous
section. First, it does not require any assumption about the distribution of the data being bootstrapped
and can be used effectively with small sample sizes (N < 20) (Efron 1979, Efron & Tibshirani 1994).
Second, bootstrapping can be used directly on additive indexes such as the CMP “right-left” scale without making the assumptions about the covariances of these categories that would be required to derive
an analytic variance. Since the exact covariances of these categories are unknown, highly sample dependent, and very probably also influenced by coder errors, it would be speculative to make any assumptions
about these covariances for use in an analytically computed variance for the combined scales. Finally,
simulation allows us to mix error distributions, a key requirement in our case if we are to incorporate
the two types of error we have identified here, consisting of both multinomial and normal distribution
14
parameters. For all of these reasons, we always recommend the bootstrapped error variances over an
analytic solution for additive CMP measures such as the left-right scale.
The bootstrapping procedure is straightforward. Since the CMP dataset contains percentages of
total manifesto sentences coded into each category, as well as the raw total number of quasi-sentences
observed, we can convert reported percentages in each category back into raw numbers. This gives us a
new dataset in which each manifesto is described in terms of the number of sentences allocated to each
CMP coding category. We then bootstrap each manifesto by drawing 100 different random samples
from the multinomial distribution, using the pi as given from the reported PER categories, and a new n̂i
drawn from a normal distribution as per Equation 5 for each sample. Each (re)sampled manifesto looks
somewhat like the original manifesto and has the same length, except that some sentences will have been
dropped and replaced with other sentences that are repeated. We feel this is a fairly realistic simulation
of the stochastic text generation process. The nature of the bootstrapping method applied to texts in
this way, furthermore, will strongly tend to reflect the intuition that longer (unbiased) texts contain more
information than shorter ones.
One problem that is not addressed by bootstrapping the CMP manifesto codings is that, as anyone
who has a close acquaintance with this dataset will know, it is typical for many CMP coding categories
to be empty for any given manifesto—resulting in zero scores for the variable concerned. No matter how
large the number we multiply by zero, we still get zero. Thus a user of the CMP data dealing with a 20sentence manifesto that populates only 10 coding categories out of 56 must in effect assume that, had the
manifesto been 20,000 sentences long, it would still have populated only 10 categories.12 In extremis,
if some manifesto populated only a single CMP coding category, then every sampled manifesto would
be identical. We cannot get around this problem with the CMP data by bootstrapping, unless we make
some arbitrary assumptions about probability distributions for non-observed categories, and we are very
reluctant to do this.
Aside from the problem of empty CMP coding categories, the great benefit of bootstrapping the
CMP estimates in this way, to simulate the stochastic process of text generation, is that we can generate
the standard errors and confidence intervals associated with the estimated value for each coding category,
for scales generated by combining these categories, such as the CMP additive left-right scale, as well as
more specific scales measuring party positions on social, liberalism or economic policy.
12 In
this context note that any manifesto with fewer than 56 quasi-sentences is constrained to yield some empty policy
categories in the CMP’s 56-category scheme. More generally, even with more that 56 quasi-sentences, the probability of
empty coding categories increases as the number of quasi-sentences declines.
15
[FIGURE 2 ABOUT HERE]
The results of using this bootstrapping procedure are reported in Figure 2, where we show the relationship between manifesto length, measured in quasi-sentences, and the bootstrapped standard error of
both the additive CMP left-right scale, as well as the error estimated for PER501, the “pro-environment”
category.13 The first panel (a) of Figure 2 shows the effects of the bootstrapped measure for PER501,
the single category indicating a pro-environmental stance. As predicted in the previous section, its error
variance declines directly with (logged) manifesto length. The wide differences (indicated by the thick
band) reflect the very different proportions of piPER501 for different manifestos i.
In panel (c), we compare the bootstrapped error variance to the variance computed analytically
(per Equation 8) for the Environment (PER501) category. The near equivalence of these two methods of
obtaining the standard errors serves to validate both the analytical derivation of the CMP error variance,
as well as showing that bootstrapping the manifestos from their quasi-sentences is a valid way to simulate
error variance in the categories. The bootstrapped method, in other words, faithfully reproduces the error
we have identified using the analytic variance based on the multinomial distribution.
[FIGURE 3 ABOUT HERE]
When we add in the additional step of letting the number of quasi-sentences vary (according
to Equation 5), then we observe additional noise in the results of the bootstrapping procedure, arising
from varying the number of quasi-sentences by approximately +/-10%. The results are shown in Figure
3, which plots the bootstrapped error without randomly drawing the simulated ni vector, versus those
where it has been randomly drawn. As can be seen by the clustering about the unit axis, this procedure
introduces additional noise but no evident bias, since the number of quasi-sentences is not systematically
related to the policy quantities being estimated. In the analyses in the next section that apply error
estimates to specific empirical research problems, we will always use this combined bootstrapped error
with the randomized ni as our best approximation of overall non-systematic error in the CMP’s reported
estimates.
5
Using CMP Error Estimates in Applied Research
There are two main research settings in which scholars need to estimate the policy positions of political
actors. The first is cross-sectional: a map of some policy space is needed, based on estimates of different
13 Given the distribution of the data, both axes of Figure 2 uses logarithmic scales and the figure demonstrates clearly that the
level of bootstrapped error in this scale error is very strongly related to manifesto length—short manifestos have much more
potential for error of this type than the long ones.
16
agent positions at the same point in time. The second is longitudinal: a time series of policy positions is
needed, based on estimates of the same agent’s policy positions at different points in time. As we have
noted, many alternative techniques are available for estimating cross-sectional policy spaces. The signal
virtue of the CMP data is that it purports to offer time series estimates of the policy positions of key
political agents. Clearly, either cross sectional or time series estimates of policy positions convey little
or no rigorously usable information if they do not come with associated measures of uncertainty. Absent
any such measure, estimates of “different” policy positions may either be different noisy estimates of
the same underlying signal, or accurate estimates of different signals. We have no way to know which is
the case; yet we need to know this if we are to draw valid statistical inferences based on variation in our
estimates. Drawing such inferences is, of course, the entire purpose of conducting empirical research in
the first place.
5.1
Estimating valid differences
A substantial part of the discussion found in MPP and MPP2 of the face validity of the CMP data comes
in early chapters of each book, during which the policy positions of specific parties are plotted over time.
Sequences of estimated party policy movements are discussed in detail and held to be substantively
plausible, with this substantive plausibility taken as evidence for the face validity of the data. But are
such estimated changes in party policy “real”, or merely measurement noise? We illustrate how to
answer this question with a specific example taken from Germany, the country in which the CMP has
been based for many years. Figure 4 plots the time series of the estimated environmental policy positions
of the CDU-CSU (PER 501 in the CMP coding scheme). The dashed line shows the CMP estimates, the
error bars show the bootstrapped 95 percent confidence intervals around these estimates, calculated as
described above.
[FIGURE 4 about here]
We see clearly from this figure that the error bands around the CMP estimates are large and thus
that most of the estimated “changes” in the CDU-CSU’s environmental policy over time could well
be noise. Statistically speaking, we can conclude that the CDU-CSU was more pro-environmental in
the early 1990s than it was either in the early 1980s or the early 2000s, but that every other observed
“movement” on this policy dimension could well be caused by measurement error.
[TABLE 3 about here]
Table 3 reports the result of extending this anecdotal discussion in a much more comprehensive
17
way. It deals with observed “changes” of party positions on the CMP’s widely-used left-right scale
(RILE) and thus systematically summarizes all of the information about policy movements that is used
anecdotally, in the early chapters of MPP and MPP2, to justify the face validity of the CMP data.
The table reports, considering all situations in the CMP data in which the same party has an estimated
position for two adjacent elections, the proportion of cases in which the estimated policy “change” from
one election to the next is statistically significant. These results should be of considerable interest to
all third-party researchers who use the CMP data to generate a time series of party positions and show
that observed policy “changes” are statistically significant in only about 28 percent of the relevant cases,
or slightly more than one in four. We do not of course conclude from this that the CMP estimates are
invalid. But we do conclude that many of the policy “changes” that have hitherto been used to justify
their validity are not statistically significant and may well just be noise. More generally, we argue that, if
valid statistical (and hence logical) inferences are to be drawn from “changes” over time in party policy
positions, estimated from the CMP data, then it is essential that these inferences are based on valid
measures of the uncertainty in the CMP estimates, which have not until now been available.
[FIGURE 5 about here]
The other type of estimated policy difference that interests scholars analyzing party competition,
as we have seen, concerns cross-sectional differences between different parties at the same point in
time. Considering some static spatial model of party competition that is realized by estimating the positions of actual political parties at some particular time, many of the model’s implications will depend
on differences in the policy positions of the various parties. It is crucial, therefore, when estimating a
cross-section of party policy positions, to know whether the estimated positions of different parties do
indeed differ from each other in a statistical sense. Figure 5 illustrates this problem for recent elections
in Germany (top panel) and France (bottom panel), giving estimates for 2001 of party positions on the
CMP left-right scale. These show us quite clearly, for the case of Germany, that the five political parties under investigation took up only three statistically distinguishable positions. Taking account of the
uncertainty of the estimates, the Social Democrats and Free Democrats have statistically indistinguishable estimated positions on the CMP left-right scale, as do the Greens and PDS. The situation is even
“muddier” in France where four parties—Communists, Socialists, Greens and UMP—have statistically
indistinguishable estimated positions on the CMP left-right scale in 2001. Only the far-right FN had an
estimated left-right position that clearly distinguished it from all other parties.
To conclude this discussion, we can see that deriving measures of uncertainty for the CMP esti-
18
mates is no small matter. On one hand it considerably extends our confidence in the inference that two
party positions differ—cross-sectionally or longitudinally—if this difference can be shown to be significant in a statistical sense. On the other hand we see that many of the policy “differences” on which
third-party authors have hitherto relied in their empirical work may well be just noise. This stands to
reason. It has never been a secret that some of the manifestos on which the CMP estimates are based
are very short indeed, and thus convey rather little information, while other manifestos are quite long,
and thus tend to convey much more information. Notwithstanding this, users of CMP data have hitherto
treated all estimates as if they conveyed precisely the same amount of information. As we have just
seen, using the measures of uncertainty we develop in this paper makes a big difference to the statistical
inferences we draw from the CMP estimates, whether in a longitudinal or cross-sectional setting.
5.2
Correcting estimates in linear models
When covariates measured with known error are used in linear regression models, the result is bias and
inefficiency when estimating coefficients on the error-laden covariates (Carroll, Ruppert, Stefanski &
Crainiceanu. 2006). The usual assumption is that coefficients will suffer from “attenuation bias”: they
are likely to be biased towards zero, thus underestimating the relevant variables’ true effect.14
The CMP estimates of party policy positions clearly have some error, even on the CMP’s own
published accounts of inter-coder reliability. As we have shown above, it is possible to derive measures
of the effect of some of the error processes involved in generating the CMP data. Nearly all empirical
research that relies on CMP estimates as measures of party policy positions uses these estimates as
covariates in linear regression models. Of all of the studies that have used CMP data as covariates in
linear regression models, however, to our knowledge not a single one has explicitly taken account of the
likelihood of error in the CMP estimates, or even used the length of the underlying manifesto as a crude
indication of potential error. As a result, we expect many of the reported coefficients in the studies that
use CMP estimates as covariates to be biased and inconsistent.
We address this issue by replicating and correcting three recent high-profile studies that use the
CMP data, each published in a major political science journal: Adams, Clark, Ezrow & Glasgow (2006,
14 The assumption of attenuation bias in models with error-contaminated covariates has even been called the “Iron Law of
Econometrics” (Hausman 2001, 58). However, Carroll et al. (2006, 41) suggest that conclusion of attenuation bias has to be
qualified, because it depends on the relationship between true covariate, its observed proxy, and possibly other covariates in the
model. Moreover, in case of multiple variables containing measurement error it will also depend on the relationship between
measurement error components of these error-contaminated variables. Thus, in linear regression depending on whether it is a
simple or multiple regression, single or multiple covariates measured with error, and whether measurement error contains bias,
the results will range from simple attenuation bias to cases where real effects are hidden, or relationships found in the observed
data are nonexistent in true data, or the reversal of the signs of coefficients compared to the case with true data.
19
AJPS), Hix, Noury & Roland (2006, AJPS) , and Golder (2006, BJPS). Each study uses CMP-based estimates of policy as explanatory variables, and each uses the measures in a slightly different fashion. In all
three cases we obtained the datasets (and replication code) from the authors and replicated the analyses
correcting for measurement error in the CMP-derived variables. We do this using a simple error correction model known as simulation-extrapolation (SIMEX) that allows generalized linear models to be
estimated with correction for known or assumed error variance for error-prone covariates (Stefanski &
Cook 1995, Carroll et al. 2006). SIMEX is a method to estimate and reduce the effect of measurement
error on an estimator experimentally via simulations. The basic idea of the method is fairly straightforward: If a causal relationship is biased by measurement error, then adding more measurement error
should increase the degree of this bias. By adding (or simulating) successive levels of measurement
error, it is possible to estimate the relationship between the expected value of the coefficient and its true
value, and then extrapolate back to the unbiased estimate.15 Given a relatively robust set of assumptions,
the SIMEX procedure has been verified to be exactly consistent in linear and loglinear mean models,
and approximately consistent in general (Carroll et al. 2006, 151), and to provide a good approximation
to the unbiased coefficient estimates in a variety of cases, including non-linear models. SIMEX is particularly useful for models with additive measurement error. More complicated error corrections are of
course possible, but here we deliberately chose a method that is simple, applicable to a wide class of
general linear models, and for which freely available software is available that can be used with popular
statistical packages.16
5.2.1
Adams, Clark, Ezrow and Glasgow (2006)
Adams et al. (2006) are concerned with whether “niche” parties (for example Communists, Greens
or extreme-right) are fundamentally different from mainstream parties. The first model investigated
in the article focuses on the question whether such “niche” parties adjust their policies in response to
public opinion shifts. The authors operationalize party policy shifts between elections as changes in a
party’s CMP left-right scale (“Rile”) in adjacent elections. This measure is then regressed on public
opinion shifts, a dummy variable for niche party status, and the interaction of these two variables, plus
15 More
precisely, if β is the true value of the coefficient, and b̂ is the estimated (biased) coefficient, then
E(b̂) =
β
1 + σ2m /σ2p
(9)
where σm is the standard deviation (measurement error) of the predictor, and σ p is the standard deviation of the true predictor.
As σm → 0, E(b̂) → β. See Carroll et al. (2006) for full details.
16 For R, the simex package is available from CRAN. Information on SIMEX implementation in STATA can be found at
http://www.stata.com/merror/.
20
lagged policy shifts, lagged vote share change, the interaction of these two terms, and a set of country
dummies.17
In our replication of Adams et. al. Table 1, we focus on the effect of measurement error in lagged
policy shift and its interaction with lagged vote share change, since error effects in covariates are what
is usually considered in the classical measurement error model. Our estimate of error for lagged policy
shift is based on the bootstrapped standard error. For the interaction term of lagged policy shift and
lagged vote share change, we assume that change in vote share is measured without error. In this and
in the replications that follow, our error estimates for each error-prone covariate is the mean of the
in-sample average error variance from the bootstrapping procedure with randomized text lengths (and
specified in the note to each table).
The second model in Adams et al. (2006) tests whether policy adjustments tend to enhance parties’
electoral support. The authors model change in party’s vote share as a function of parties’ policy shifts
towards the center of the voter distribution and away from it, and how it differs between mainstream and
niche parties. The key explanatory variables are derived directly from CMP and thus are expected to
be error-prone: Centrist policy shift, Noncentrist policy shift, Niche Party * Centrist policy shift, Niche
Party * Noncentrist policy shift. The first variable measures the shift in party’s policy position towards
the mean voter position and is calculated as the absolute value of the change in party’s position on the
CMP left-right scale when a leftist party shifts right or rightist party shifts left, and zero otherwise.
Non-centrist shift concerns the shift away from the center from election t − 1 to t, and it is calculated
as the absolute value of the change in party’s position on the CMP left-right scale when a leftist party
shifts left or rightist party shifts right and zero otherwise. The next two variables pick up the differences
in the electoral effects for niche and mainstream parties in relation to centrist and non-centrist policy
shifts. Two additional more control variables are based on CMP measures: Party policy convergence
and Party policy convergence * Peripheral party. The former variable is constructed to capture the idea
that parties’ electoral success may be affected by policy shifts of other parties. It is operationalized as
the sum of all centrist policy moves by all parties in the system. The latter variable is an interaction of
Party policy convergence with a dummy variable for peripheral parties. In addition to these six error17 We
expect to find error contamination in both dependent variable on the left-hand side, its lagged value on the right-hand
side, and an interaction of lagged dependent variable and lagged change in vote share. In order to remain within the classical
measurement error (CME) domain we assume that error in first-differences in the dependent variable is uncorrelated with
error in second-differences in its lagged value. Within CME it is known that measurement error in the dependent variable, if
uncorrelated with other covariates, will only inflate standard error of the regression (Wooldridge 2002, 711-72). Moreover the
effect of measurement error in first-difference estimation in panel data models is much higher than in level models, which may
somewhat explain low reported R2 .
21
prone covariates, Model 2 in Adams et al. (2006) contains dummy variables for niche parties, governing
parties, coalition governments, previous change in vote share, and several economic control variables:
changes in unemployment and GDP rates and their interaction with governing party dummy.
[TABLE 4 about here]
Table 4 presents results for two models, taken from the two regression tables from Adams et al.
(2006). For each model, we report our replication of the published results, plus the SIMEX errorcorrected results.18 The SIMEX correction of Model 1 strengthens the coefficient on Previous policy
shift variable by 40%, and quadruples the coefficient on the interaction term of Previous policy shift and
Previous change in vote share. Although error correction does not alter the remaining results greatly
it also serves to illustrate a generally a turbid nature of error processes when the coefficient on a key
explanatory variable in Model 1 (Public opinion shift) weakens by 7%, and Previous change in vote
share becomes statistically significant. Neither Public opinion shift or Previous change in vote share are
derived from the CMP both are affected due to their intra-model relationship with error contaminated
covariates.
In Model 2 a quarter of all variables are measurement error prone. Due to high levels of collinearity we should expect the SIMEX error correction to have a much more pronounced effect on the results
than in Model 1. The key explanatory variable Niche Party * Centrist policy shift does not change
significantly (weakens only by 5%) and still suggests that niche parties are punished at the polls for
moving towards the center of voter distribution. The SIMEX error corrections unmasks several results
that do not appear in the original analysis. Thus the coefficient on Previous change in vote share gains
slightly and becomes barely statistically significant. However—and more importantly—the coefficient
on Party policy convergence grows by 65% and becomes statistically significant, suggesting the substantive conclusion that when party policy positions move to the center of the policy space centrist parties
are electorally penalized more than peripheral parties. Another interesting result concerns the effect of
a policy shift away from the center of voter distribution on mainstream parties. The negative and statistically significant coefficient on Noncentrist policy shift is expected within the theoretical framework
of Adams et al. (2006) but couldn’t be shown empirically. Here after the SIMEX error correction the
coefficient on Noncentrist policy shift almost doubles and becomes statistically significant. This result
of the SIMEX error correction on Noncentrist policy shift is shown graphically in Figure 6.
18 In
this replication and the ones that follow, we report our replicated estimates instead of the published estimates, which
are in each case slightly but not significantly different. This differences arose due to slight errors in data preparation in each
published analysis which we uncovered in the course of replicating their results. For full details including our own replication
code are available from http://kenbenoit.net/cmp/.
22
[FIGURE 6 about here]
Correcting for bias induced by measurement error in Noncentrist policy shift is a good illustration
of general principles behind the SIMEX method. In Figure 6, the y-axis represents the size of the
coefficient. The amount of noise added to the variance of error-prone covariate is on the x-axis. It is
in the form (1 − λ), where λ is a set of discrete values, usually: {−1, −.5, 0, .5, 1, 1.5, 2}. When λ = 0
the model is estimated without any error correction or error inflation—often called naive model. Each
node in Figure 6 represents a mean of 100 simulations of the model with each successive increase in λ,
viz. an increase in the amount of measurement error added to the model. Once a pattern is established
SIMEX extrapolates back to the case of λ = −1, i.e. no measurement error. Among several extrapolants
a quadratic function has been shown to have good numerical properties. As becomes clear from Figure
6 while the naive model produced a coefficient on Noncentrist policy shift of around -2.3 (standard
error=1.8) and increasing amounts of measurement error brings the size of the coefficient closer to 0,
however, the SIMEX correction shows the coefficient to be around -4.3 (standard error=2.3). This is
a 84% increase compared to the naive model. Overall, measurement error in linear models either in
one or several covariates cannot be simply dismissed as it affects the estimates produced in standard
normal-linear models, sometimes in a substantively significant way.
5.2.2
Hix, Noury, and Roland (2006)
Hix, Noury & Roland (2006) are concerned with the content and character of political dimensions in
the European Parliament (EP). Following an inductive scaling analysis of all roll-call votes in the EP
from 1979 and 2001, Hix, Noury & Roland (2006) attempt to validate their interpretations of these inductively derived dimensions by mapping variation in the dimensional measures to exogenous data on
nation party policy positions. The authors conclude that the first dimension is the classic left-right dimension, while the second dimension primarily concerns party positions on European integration as well as
government-opposition conflicts. Substantive interpretation of dimensions is achieved by regressing the
mean position of each national party’s delegation of MEPs on each dimension in each parliament on two
sets of independent variables. The first set of variables includes exogenous measures of national party
positions on the left-right, social and economic left-right, and pro-/anti-EU dimensions. The second set
relates to government-opposition dynamics and consists of categorical variables whether a national party
was in government and whether the party had a European Commissioner, as well as dummy variables
for each European party group, each EU member state, and each (session of) European parliament.
23
The measures of national party positions of interest to us here are either taken directly (like leftright) or constructed from the CMP data (EU, economic and social left-right). National party positions
on the EU dimension are taken as the difference between positive mentions of EU in party manifesto
(category PER108) and negative mentions (category PER110). Party positions on economic and social
dimensions follow the suggestion in Laver & Garry (2000, 628-629), where the economic dimension is
constructed as the difference between “Capitalist Economics” and “State Intervention” clusters, and social dimension is constructed as the difference between “Social Conservatism” and “Anti-establishment
views” clusters from Laver & Budge (1992, 24).19 We expect to find error contamination in all models
containing variables derived from the CMP data.
[TABLE 5 about here]
Table 5 reports our replicated estimations of four models in Hix, Noury & Roland (2006) along
with error-corrected measurements based on our bootstrapped variances with randomized text length.
In each error-corrected estimation, we used simulation-extrapolation with an error variance estimated
as the mean of the in-sample average error variance from the bootstrapping procedure with randomized
text lengths.
Although Hix, Noury & Roland (2006) present numerous estimations, here we examine only those
that used CMP-based policy measures as covariates in their regressions on the first dimension from their
inductive scaling procedure. Model 2 is the most basic model regressing their first estimated dimension
on several covariates: general left-right and EU dimensions taken from the CMP, plus categorical variables whether a national party was in government and whether the party had a European Commissioner,
and as dummy variables for each (session of) European parliament. Model 3 replaces the general leftright with two more specific left-right dimensions drawn from the CMP: economic left-right and social
left-right. Model 6 extends Model 3 to include dummy variables for each European party group. Model
9 extends Model 6 to include a set of country dummies.
The first thing that is apparent from the Table 5 is that the SIMEX error correction had only
minimal effect on most coefficients with all error-prone covariates increasing only slightly and most
other covariates decreasing slightly. By far the biggest correction is achieved on the coefficient on
national party position on EU dimension. This is to be expected, since the “pro vs anti EU” variable
constructed from the CMP data uses only two of the 56 substantive coding categories and is relatively
sparsely populated with coded data. The coefficient on Pro-/Anti-EU variable with the SIMEX error
19 Thus,
economic dimension = (per401 + per402 + per407 + per414 + per505) - (per403 + per404 + per406 + per412 +
per413); social dimension = (per203 + per305 + per601 + per603 + per605 + per606)- (per204+per304+per602+per604)
24
correction increased by 78% in Model 2, doubled in Model 3, increased by 2.5 times in Model 6, and
by 2.2 times in Model 9. In the latter two models, these large increases in the size of the estimated
coefficients means that the Pro-/Anti-EU variable becomes statistically significant, where it failed to do
so in the naive models. For example, in Model 3 it would suggest that for the mean of each national
party’s delegation on the first dimension to move from the center to extreme right position, ceteris
paribus, its position on the EU dimension would have to change by 23 and not by 45 points as in the
naive estimates. Party policy positions on EU dimension have an in-sample range of 33. Thus while such
a whole-sale move from one end of political spectrum to another seems highly unlikely, it is nevertheless
possible under error corrected results. Substantively, for an explanation of the mean position of national
party’s delegation its national party’s position on the EU dimension is more important than its position
on general left-right or substantive economic and social left-right.
Moreover, the SIMEX error correction of Pro-/Anti-EU variable and its subsequent regaining
of importance in Models 6 and 9 forces rethinking of some of the conclusions in the article. Hix,
Noury & Roland (2006) expect to find the first dimension to be explained by policy positions of political
parties on general (domestic) left-right issues, while the second dimension should be explained by policy
positions of political parties on the EU dimension.20 Indeed, the estimates from naı̈ve models, viz. not
taking account of measurement error bias in CMP derived variables, provide ample evidence to support
original theoretical expectations. Once party groups are controlled for, party positions on EU dimension
have no role to play in interpreting the first dimension. However, applying the SIMEX error correction
to the error-prone Pro-/Anti-EU covariate, not only more than doubles it in magnitude but also makes
it statistically significant. Thus, the first dimension still contains Pro-/Anti-EU and not only left-right
as claimed by the authors. This calls into question the conclusion of the article that political parties’
positions in EU policy making process are shaped predominantly by important socioeconomic positions
in domestic politics rather than preferences about European integration. Instead, the error-corrected
results suggest that there is no clear distinction between general left-right positioning and positioning
on European integration when it comes to the primary axis of contestation in the European Parliament,
and that if anything, positioning on European Union appears to be even more important than general
left-right in explaining this primary axis.
20 In
the words of Hix, Noury & Roland (2006) interpreting their results from the naive model:
EU policies of national parties and national party participation in government are only significant without the
European party group dummies. This means that once one controls for European party group positions these
variables are not relevant explanatory factors on the first dimension. (502)
25
5.2.3
Golder (2006)
Golder (2006) applied non-linear probit models to investigate the formation of pre-electoral coalitions
between parties and the forces that may influence the likelihood of such coalitions. Pre-electoral coalitions between dyads of parties in a system are modeled as functions of Ideological incompatibility,
Polarization and Electoral threshold (plus the interaction between these two variables), Coalition size
and Coalition size squared, Asymmetry, and an interaction between Coalition size and Asymmetry. We
can expect three variables to be particularly error-prone, since they are based on the CMP left-right
measure: Ideological incompatibility, Polarization and Polarization * Electoral threshold.
Ideological incompatibility measures the ideological distance between parties in a dyad. It is directly computed as the absolute value of the difference of the CMP “rile” score for parties in a dyad.
We compute error variance for the incompatibility measure combining error variances for individual
parties in a dyad utilizing the analytic definition of difference between two uncorrelated variances (see
Appendix A). Polarization measures ideological dispersion in a system and is calculated as the difference of the CMP “ rile” scores for the biggest left-wing and the biggest right-wing parties in political
system. We compute error variance for Polarization following the same procedure as for Ideological
incompatibility.
[TABLE 6 about here]
Table 6 applies the SIMEX correction to Golder’s probit model predicting the probability of preelectoral coalition formation.21 The majority of covariates experience only minimal changes after correcting for measurement error. As a result of the SIMEX error correction, the coefficient for Polarization
changes sign from positive to negative, but remains statistically indistinguishable from zero. Error correction on Incompatibility results in a 18% coefficient increase, thus making the creation of pre-electoral
coalition even less probable with the increase in ideological incompatibility of potential partners.
Although none of the substantive results changed with the SIMEX error correction, our implementation of error-correction for this model demonstrates that error correction using our estimates of
CMP error variances is straightforward to apply to non-linear models.
21 We depart from original estimation in two ways here.
First, we use newer CMP dataset made available with the publication
of MPP2, which results in some very slight differences in the dataset. Second, we use estimate only a fixed effects probit model,
rather than the additional random effects probit model as in the original piece to avoid complicating the computation. Neither
change resulted in any substantive differences in our replications versus the original results.
26
6
Concluding Remarks and Recommendations
When we treat text as data that are prone to measurement error, we can analyze bodies of text using wellknown statistical tools. The implications of this are very general, although in this paper we focus on the
text found in party manifestos and analyzed by the CMP; this is by far the largest and most significant
dataset in the profession that is based on systematic text analysis. We show how manifestos provide
systematic information about their authors, in the form of random variables deposited as text through
process of authorship that is inherently stochastic, even when the author’s underlying position is fixed. In
addition, we characterize one important aspect of the CMP coding process as stochastic: non-systematic
variation in identifying coding units, or quasi-sentences, on which all CMP measures are based. Putting
together these two stochastic processes and resampling from quasi-sentence counts reconstructed from
the CMP dataset, we show how to compute errors in a practical and theoretically supported way for the
entire set of CMP measures. We also show how these errors affect descriptive and causal inferences
based on CMP measures. Building on this method, we offer a “corrected” version of the CMP dataset
with bootstrapped standard errors for all key estimates. A full version of these corrections is available
on our website http://kenbenoit.net/cmp/.
Our task in this paper has been fundamentally constructive. The CMP dataset is potentially a
very valuable, indeed currently unique, resource dealing with party positions on many dimensions in
a cross-national time series context. We therefore regard it as essential that these data be used in a
statistically rigorous way. Hitherto, third-party authors have either ignored or swept under the carpet the
self-evident fact that the CMP data are error-prone, yet come with no estimate of associated error. The
huge volume of CMP source documents, written in many languages, were coded once only by a single
human coder and will likely never all be recoded many times by different coders. (Some of them may
even be lost). We offer a way around this problem with a method for simulating the level of error in
key CMP variables and a way to make use of these simulated errors when deploying CMP variables as
covariates in statistical analyses—by far their most common use. When applied to descriptive analyses,
we have shown that for many of the apparent “differences” in CMP estimates of party policy positions—
differences over time in the position of one party or differences between parties at one point in time—are
probably attributable to measurement error rather than to real differences in policy position. Thus only
about one quarter of all CMP-estimated “movements” in parties’ left-right policy positions over time
were assessed on the basis of our simulations to be statistically significant.
Error in the measurement of covariates is likely to bias regression coefficients, yet none of the
27
large volume of published empirical research using CMP data takes any account whatsoever of this. We
replicate three prominent recently-published articles in which error-prone CMP variables are used as
covariates, and show how to correct these using a SIMEX error correction model, parameterized by our
bootstrapped estimates of likely error. The likely systematic effect of error contaminated covariates is
attenuation bias, and this is what we tend to find in each case. Some error-corrected effects are stronger,
and more significant, than those estimated in models taking no account of error in the covariates. This
can cause substantively important reinterpretation of results in cases where inferences have relied on
the lack of significance of some covariates. A good example is what emerges as the potentially flawed
inference that national party policy positions on the EU have no influence in their EP roll-call voting
behavior, an inference that is reversed once account is taken of error-contamination in the CMP dataset’s
sparsely-populated variables measuring EU policy.
While we have taken an important first step towards providing a practical and theoretically supported means to estimate non-systematic measurement error in CMP estimates, the solution we provide
here is hardly the last word on this topic. Analyses of coder differences and/or coder error, for instance, could uncover systematic error leading to bias, which we have not considered in this paper; this
possibility deserves investigation in future work. Additional experiments with coder reliability using additional texts would also provide important information on the second type of error we identify. Finally,
other means of implementing error correction models are certainly possible, including Bayesian-MCMC
methods that can take into account the unit-specific nature of error in our error estimates. Indeed, we
hope that our focus of attention on error in the widely used CMP estimates will stimulate a broader dialogue on measurement error in many of our most commonly used measures in political science, such as
opinion survey results, expert survey measures, or other computed quantities. Given our knowledge of
measurement error and the wide availability of techniques for dealing with such error, there is no longer
any excuse for future scholars to use error-prone measures as if these were error free.
28
A
Computing the Analytic Variance of CMP “PER” Categories
Assume that E(pi j ) = πi j , where each πi j represents the true and unobservable policy position of
country-party-date unit i on issue j. For any given Xi j from the multinomial distribution, Var(Xi j ) =
ni pi j (1 − pi j ). To derive the analytic variance for the percentage quantity Var(pi j 100), we have (dropping the manifesto index i for simplicity):
E(X j ) = np j
x j = np j
Var
xj
= pj
n
1
xj
n
= Var(p j )
1
Var(x j ) = Var(p j )
n2
1
np j (1 − p j ) = Var(p j )
n2
1
p j (1 − p j ) = Var(p j )
n
Translating into the CMP’s percentage metric (p j ∗ 100):
10, 000Var(p j ) =
Var(p j 100) =
SD(p j 100) =
10, 000
p j (1 − p j )
n
10, 000
p j (1 − p j )
n
q
100
√
p j (1 − p j )
n
To compute the variance for additive categories like the pro-/anti-EU scale (PER108 minus PER110)
or the 26-category “right-left” scale, the covariances between categories must also be estimated, since it
is a property of variance that Var(aX +bY ) = a2 Var(X)+b2 Var(Y )+2abCov(X,Y ). Rather than attempt
to estimate these unknown—but certainly non-zero—covariances from finite samples, we recommend
the bootstrapping procedure instead of the analytic computation.
29
References
Adams, James. 2001. “A Theory of Spatial Competition with Biased Voters: Party Policies Viewed
Temporally and Comparatively.” British Journal of Political Science 31(1):121–158.
Adams, James, M. Clark, L. Ezrow & G. Glasgow. 2006. “Are Niche Parties Fundamentally Different from Mainstream Parties? The Causes and the Electoral Consequences of Western European
Parties’ Policy Shifts, 1976–1998.” American Journal of Political Science 50(3):513–529.
Alt, James & David Lassen. 2006. “Transparency, Political Polarization, and Political Budget Cycles in
OECD Countries.” American Journal of Political Science 50(3):530–550.
Baron, D.P. 1991. “A Spatial Bargaining Theory of Government Formation in Parliamentary Systems.”
The American Political Science Review 85(1):137–164.
Bartolini, S. & P. Mair. 1990. Identity, Competition, and Electoral Availability: The Stabilisation of
European Electorates 1885-1985. Cambridge University Press.
Benoit, Kenneth & Michael Laver. 2006. Party Policy in Modern Democracies. London: Routledge.
Benoit, Kenneth & Michael Laver. 2007. “Benchmarks for Text Analysis: A Reply to Budge and
Pennings.” Electoral Studies 26:130–135.
Bräuninger, T. 2005. “A partisan model of government expenditure.” Public Choice 125(3):409–429.
Budge, Ian, David Robertson & Derek Hearl. 1987. Ideology, Strategy and Party Change: Spatial
Analyses of Post-War Election Programmes in 19 Democracies. Cambridge: Cambridge University
Press.
Budge, Ian, Hans-Dieter Klingemann, Andrea Volkens, Judith Bara & Eric Tanenbaum. 2001. Mapping
Policy Preferences: Estimates for Parties, Electors, and Governments 1945–1998. Oxford: Oxford
University Press.
Carroll, Raymond J., David Ruppert, Leonard A. Stefanski & Ciprian M. Crainiceanu. 2006. Measurement Error in Nonlinear Models: A Modern Perspective. Number 105 in “Monographs on
Statistics and Applied Probability” 2nd ed. Boca Raton, FL: Chapman and Hall/CRC.
Duso, T. & L.H. Röller. 2003. “Endogenous Deregulation: Evidence from OECD Countries.” Economics
Letters 81(1):67–71.
Efron, Bradley. 1979. “Bootstrap Methods: Another Look at the Jackknife.” The Annals of Statistics
7(1):1–26.
Efron, Bradley & Robert Tibshirani. 1994. An Introduction to the Bootstrap. New York: Chapman and
Hall/CRC Hall.
Erikson, R.S., M.B. Mackuen & J.A. Stimson. 2002. The Macro Polity. Cambridge University Press.
Evans, G. & P. Norris. 1999. Critical Elections: British Parties & Voters. Sage Publications.
Golder, Sona N. 2006. “Pre-Electoral Coalition Formation in Parliamentary Democracies.” British Journal of Political Science 36(2):193–212.
Hausman, J. 2001. “Mismeasured Variables in Econometric Analysis: Problems from the Right and
Problems from the Left.” The Journal of Economic Perspectives 15(4):57–67.
30
Hix, Simon, Abdul Noury & Gérard Roland. 2006. “Dimensions of Politics in the European Parliament.”
American Journal of Political Science 50(2, April):494–511.
Janda, K., R. Harmel, C. Edens & P. Goff. 1995. “Changes in Party Identity: Evidence from Party
Manifestos.” Party Politics 1(2):171–196.
Kim, Heemin & Richard C. Fording. 1998. “Voter Ideology in Western Democracies, 1946-1989.”
European Journal of Political Research 33(1):73–97.
King, Gary, Robert Keohane & Sidney Verba. 1994. Designing Social Inquiry. Princeton: Princeton
University Press.
Klingemann, Hans-Dieter, Andrea Volkens, Judith Bara, Ian Budge & Michael McDonald. 2006. Mapping Policy Preferences II: Estimates for Parties, Electors, and Governments in Eastern Europe,
European Union and OECD 1990-2003. Oxford: Oxford University Press.
Laver, Michael & Ian Budge. 1992. Party policy and government coalitions. New York: St. Martin’s
Press.
Laver, Michael & John Garry. 2000. “Estimating Policy Positions from Political Texts.” American
Journal of Political Science 44(3):619–634.
Laver, Michael, Kenneth Benoit & John Garry. 2003. “Estimating the policy positions of political actors
using words as data.” American Political Science Review 97(2):311–331.
Mair, P. 1987. The Changing Irish Party System: Organisation, Ideology and Electoral Competition.
Pinter.
Meguid, B. 2005. “Competition Between Unequals: The Role of Mainstream Party Strategy in Niche
Party Success.” American Political Science Review 99(3):347–359.
Milner, H.V. & B. Judkins. 2004. “Partisanship, Trade Policy, and Globalization: Is There a Left-Right
Divide on Trade Policy?” International Studies Quarterly 48(1):95–120.
Monroe, Burt & Ko Maeda. 2004. “Talk’s cheap: Text-based estimation of rhetorical ideal-points.”
Working paper: Michigan State University.
Petry, F. 1988. “The Policy Impact of Canadian Party Programs: Public Expenditure Growth and Contagion from the Left.” Canadian Public Policy/Analyse de Politiques 14(4):376–389.
Petry, F. 1991. “Fragile Mandate: Party Programmes and Public Expenditures in the French Fifth
Republic.” European Journal of Political Research 20(2):149–171.
Petry, F. 1995. “The Party Agenda Model: Election Programmes and Government Spending in Canada.”
Canadian Journal of Political Science/Revue canadienne de science politique 28(1):51–84.
Ray, L. 1999. “Measuring party orientations towards European integration: Results from an expert
survey.” European Journal of Political Research 36(2):283–306.
Robertson, David. 1976. A Theory of Party Competition. London and New York: Wiley.
Schofield, Norman. 1993. “Political competitition and multiparty coalition governments.” European
Journal of Political Research 23(1):1–33.
Sigelman, L. & H.B. Emmett. 2004. “Avoidance or Engagement? Issue Convergence in U.S. Presidential
Campaigns, 1960-2000.” American Journal of Political Science 48(4):650–661.
31
Slapin, Jonathan & Sven-Oliver Proksch. 2007. “A Scaling Model for Estimating Time-Series Policy
Positions from Texts.” Paper presented at the annual meeting of the Midwest Political Science
Association, April 12.
Stefanski, L. A. & J. R. Cook. 1995. “Simulation-Extrapolation: The Measurement Error Jackknife.”
Journal of the American Statistical Association 90(432, December):1247–1256.
Strom, K. & J.Y. Lejpart. 1989. “Ideology, Strategy, and Party Competition in Postwar Norway.” European Journal of Political Research 17(3):263–288.
Thomson, R., J. Boerefijn & F. Stokman. 2004. “Actor alignments in European Union decision making.”
European Journal of Political Research 43(2):237–261.
Van Der Brug, W., M. Fennema & J. Tillie. 2005. “Why Some Anti-Immigrant Parties Fail and Others Succeed: A Two-Step Model of Aggregate Electoral Support.” Comparative Political Studies
38(5):537–573.
Volkens, Andrea. 2001a. Manifesto Research Since 1979. From Reliability to Validity. In Estimating
the Policy Positions of Political Actors, ed. Michael Laver. London: Routledge pp. 33–49.
Volkens, Andrea. 2001b. Quantifying the Election Programmes: Coding Procedures and Controls.
In Mapping Policy Preferences: Parties, Electors and Governments: 1945-1998: Estimates for
Parties, Electors and Governments 1945-1998, ed. Ian Budge, Hans-Dieter Klingemann, Andrea
Volkens, Judith Bara, Eric Tannenbaum, Richard Fording, Derek Hearl, Hee Min Kim, Michael
McDonald & Silvia Mendes. Oxford: Oxford University Press.
Volkens, Andrea. 2007. “Strengths and Weaknesses of Approaches to Measuring Policy Positions of
Parties.” Electoral Studies 26(1):108–120.
Webb, P.D. 2000. The Modern British Party System. Sage Publications.
Wooldridge, J. 2002. Econometric Analysis of Cross-sectional and Panel Data. The MIT Press, Cambridge, MA.
32
Total Quasi-sentences
(0,5]
(5,10]
(10,15]
(15,20]
(20,25]
(25,50]
(50,75]
(75,100]
(100,200]
(200,250]
(250,500]
(500,1000]
(1000,2000]
(2000,5000]
> 5000
Total
N
0
14
14
29
41
277
303
251
711
208
521
343
216
79
4
Cumulative
0.0%
0.5%
0.9%
1.9%
3.3%
12.5%
22.5%
30.9%
54.5%
61.4%
78.7%
90.1%
97.2%
99.9%
100.0%
3,011
Table 1: Total quasi-sentences in CMP dataset, for manifestos where total is reported.
Percentage Uncoded
0
(0,5]
(5,10]
(10,15]
(15,20]
(20,30]
(30,40]
(40,50]
(50,65]
(65,80]
(80,95]
(95,100]
N
864
1,121
375
193
119
129
87
50
30
9
7
-
Total
2,984
Cumulative
29.0%
66.5%
79.1%
85.6%
89.5%
93.9%
96.8%
98.5%
99.5%
99.8%
100.0%
100.0%
Table 2: Percentage of uncoded quasi-sentences in the CMP dataset, for manifestos where percentage
uncoded is reported.
33
Statistically
Significant
Change?
No
Yes
N/A
Total
Elections
% total
% applicable
1,509
590
778
52.5%
20.5%
27.0%
71.9%
28.1%
–
2,877
100.0%
100.0%
Table 3: Comparative over-time mapping of policy movement on Left-Right measure, taking into account statistical significance of shifts.
34
Model 1
Model 2
OLS
Replication
0.0591
(0.1469)
OLS with
SIMEX Correction
0.0836
(0.146)
OLS
Replication
2.1212
(2.0939)
OLS with
SIMEX Correction
3.7403
(2.4541)
Previous policy shift
-0.5127
(0.0772)
-0.7158
(0.1036)
Previous policy shift *
Previous change in vote share
-0.0044
(0.0184)
-0.0171
(0.0189)
1.089
(1.3607)
0.5123
(1.4056)
Noncentrist policy shift
-2.3335
(1.837)
-4.2818
(2.3389)
Niche party *
Centrist policy shift
-5.4069
(2.823)
-5.1522
(2.8178)
Niche party *
Noncentrist policy shift
2.6163
(2.6946)
3.6774
(2.8245)
Party policy convergence
-1.2513
(0.8416)
-2.0689
(1.058)
Party policy convergence *
Peripheral party
1.3238
(1.1628)
0.9724
(1.1732)
-0.1548
(0.0943)
-1.4181
(1.8592)
-0.166
(0.0939)
-1.856
(1.8926)
5.4118
(1.7649)
-3.0767
(1.7271)
0.9764
(1.1872)
-0.8533
(0.7263)
-0.3958
(0.3977)
0.1045
(1.0674)
-0.021
(0.4996)
-0.1594
(1.8179)
6.0491
0.2575
0.0832
122
6.0475
(1.8046)
-3.114
(1.7206)
1.2103
(1.2048)
-0.9707
(0.729)
-0.5087
(0.4063)
0.0079
(1.0657)
-0.0351
(0.4974)
0.4034
(1.8334)
4.4781
0.2338
0.054
122
Variable
Intercept
Centrist policy shift
Previous change in vote share
Niche party
Public opinion shift
Public opinion shift *
Niche party
Directional public opinion shift
0.0176
(0.0099)
0.1059
(0.1341)
0.8447
(0.2304)
-1.5508
(0.4433)
0.0165
(0.0102)
0.1063
(0.133)
0.9008
(0.2338)
-1.5675
(0.4488)
Governing party
Governing in coalition
Change in unemployment rate
Change in GDP
Governing party *
Change in unemployment rate
Governing party *
Change in GDP
Peripheral party
RMSE
R2
Adj R2
N
0.5908
0.3578
0.2982
154
0.6057
0.3247
0.262
154
Table 4: Results of SIMEX error correction in Adams et. al. (2006). Country dummies are included in the estimations
but not reported here. Italicized and grayed-out variables are error-corrected as follows: (Model 1) Previous policy shift=.43, interaction of
Previous policy shift and Previous change in vote share=.43; (Model 2) Centrist policy shift=0.228, Noncentrist policy shift=0.181, Party policy
convergence=0.847, interaction of Niche party and Centrist policy shift=0.072, interaction of Niche party and Noncentrist policy shift=0.048,
interaction of Party policy convergence and Peripheral party=0.390. Coefficients in bold are statistically significant at the p ≤ .1 level; SIMEX
standard errors are based on jacknife estimation.
35
Variable
Model 2
OLS
OLS
Replication
SIMEX
Model 3
OLS
OLS
Replication
SIMEX
Model 6
OLS
OLS
Replication
SIMEX
Model 9
OLS
OLS
Replication
SIMEX
Intercept
0.045
(0.0581)
0.044
(0.0584)
-0.149
(0.0596)
-0.181
(0.0604)
0.364
(0.0440)
0.339
(0.0459)
0.332
(0.0667)
0.296
(0.0693)
Pro-/Anti-EU
0.018
(0.0062)
0.032
(0.0076)
0.019
(0.0062)
0.038
(0.0079)
0.004
(0.0037)
0.010
(0.0048)
0.005
(0.0038)
0.011
(0.0049)
Left-Right
0.012
(0.0009)
0.012
(0.0010)
Social L-R
0.011
(0.0023)
0.014
(0.0025)
0.004
(0.0013)
0.005
(0.0015)
0.003
(0.0014)
0.004
(0.0015)
Economic L-R
0.024
(0.0022)
0.024
(0.0022)
0.006
(0.0015)
0.007
(0.0015)
0.007
(0.0016)
0.008
(0.0017)
0.022
(0.0304)
0.061
(0.0259)
-0.560
(0.0351)
-0.641
(0.2115)
-0.168
(0.0359)
-1.003
(0.0510)
0.077
(0.0993)
-0.820
(0.0496)
0.099
(0.0623)
-0.230
(0.0539)
-0.785
(0.0568)
0.447
(0.1244)
0.2013
0.8136
0.8029
349
0.022
(0.0304)
0.056
(0.0259)
-0.548
(0.0355)
-0.630
(0.2114)
-0.164
(0.0359)
-0.978
(0.0532)
0.071
(0.0991)
-0.786
(0.0527)
0.110
(0.0622)
-0.225
(0.0539)
-0.765
(0.0577)
0.453
(0.1242)
0.2022
0.8119
0.8011
349
0.030
(0.0315)
0.034
(0.0271)
-0.556
(0.0350)
-0.693
(0.2097)
-0.178
(0.0360)
-1.017
(0.0524)
0.119
(0.1018)
-0.797
(0.0511)
0.079
(0.0644)
-0.254
(0.0551)
-0.804
(0.0572)
0.393
(0.1245)
0.1923
0.83
0.8122
349
0.031
(0.0314)
0.030
(0.0271)
-0.544
(0.0357)
-0.671
(0.2104)
-0.173
(0.0360)
-0.990
(0.0546)
0.107
(0.1019)
-0.760
(0.0548)
0.091
(0.0646)
-0.245
(0.0552)
-0.781
(0.0582)
0.409
(0.1243)
0.1935
0.8279
0.8099
349
Commissioner
In Government
0.104
(0.0495)
0.107
(0.0434)
0.095
(0.0492)
0.086
(0.0440)
0.079
(0.0495)
0.103
(0.0434)
0.070
(0.0492)
0.077
(0.0439)
0.3585
0.4089
0.395
349
0.3621
0.3971
0.3829
349
0.3578
0.4112
0.3956
349
0.3645
0.3892
0.373
349
Socialists
Italian Communists
and allies
Liberals
Greens
British Conservatives and allies
Radical left
French Gaullists
and allies
Non-attached
members
Regionalists
Radical right
RMSE
R2
Adj R2
N
Table 5: Results of SIMEX error correction in Hix, Noury & Roland (2006, 503–504, Table 4). All
models include dummies for parliament, and Model 9 includes country dummies but these are not shown.
Italicized and grayed-out variables are error-corrected as follows: Left-Right=4.48, Social L-R=2.27,
Economic LR=2.06, Pro-/anti-EU=1.83. Coefficients in bold are statistically significant at the p ≤ .1
level; SIMEX standard errors are based on jacknife estimation.
36
Probit
Replication
Probit with
SIMEX Correction
Intercept
-2.0000
(0.2121)
-1.8220
(0.2267)
Incompatibility
-0.0146
(0.0024)
0.0014
(0.0028)
0.0003
(0.0002)
-0.0173
(0.0029)
-0.0026
(0.0034)
0.0005
(0.0002)
0.0170
(0.0067)
0.0409
(0.0088)
-0.0004
(0.0001)
0.0243
(0.2455)
-0.0226
(0.0071)
1250
3385
0.0114
(0.0072)
0.0408
(0.0088)
-0.0004
(0.0001)
0.0113
(0.2467)
-0.0223
(0.0071)
Variable
Polarization
Polarization * Threshold
Threshold
Seat Share
Seat Share2
Asymmetry
Asymmetry * Seat Share
-2*Log-Likelihood
N
3385
Table 6: Results of SIMEX error correction in Golder (2006, 203, Table 1, Model 2). Italicized and
grayed-out variables are error-corrected with standard deviations of 8.54, 8.25, and 8.25 respectively.
Coefficients in bold are statistically significant at the p ≤ .1 level; SIMEX standard errors are based on
jacknife estimation.
37
Underlying position to be
communicated by text
author; fundamentally
unobservable by others
Stochastic
text generation
process
Observed text used
as indicator
author’s position
Stochastic
text coding
process
Data characterizing
unobservable underlying
position of author in
terms of coded
observed text
Figure 1: Summary of stochastic processes involved in the generation of policy texts
38
(b)
●
●
● ●
●
●● ●●● ●●
●●
●
●●●●
●
● ●●●
●●
●●●● ●
●● ●
●
●
● ●
●
●
●
●
●
●
●
●
●●
● ●●
●
●●
●●●
●
●
●
●
●●●
●
●
●
●
● ●●
●● ●
● ●
●
● ●●
●
●
●
●
●●
●● ●
●
●
●
●●●● ●
●●
●
●●
●●
●●
●
●
●
●● ●●
●●
●●
●
●●
● ●●●
●●
●●
●
●●●
●
●
●
● ●
●●
●
●
●●
●
●●
●●
●
●
●
●●
●
●
●●
●
●
●●
●
●● ●●
●
●
●●
●
●
●●
● ●
●
●
●
●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●●
●
●
●●
● ●●
●●●
●
●
● ●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●● ●●●
●●
●
●●●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●● ●
●●●
●●
●
●
●●
●
●●
●●
●●
●
●●
●●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
● ●●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●●
●
●●
●
●
●
●
●
● ●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●●●●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●● ●
●●●
●
●
●
●●●
●●
●●
●
●●
●●
●●
●●
●●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●●●
●●
●
●●● ●
●●
●●●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●●
●
●
●
●
●
●
●
●
●●
●
●
●
●● ●
●●
●
●●
● ●●
●
●
●●
●
●●●●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
●
●●●
●
●●
●●
●●●
● ●●
●●
●●
●
●●
●●●
●
●
●●
●●●
●
●
●
●
●
●●●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●●
●
●●
●
● ●●●
●●
●
●●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●●● ●
●
●●
●
●●●●●
●
●
●●
●
●
●●●● ●●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●●●
● ●
●
●●
●●●
●
●● ●
● ● ●●●
●●
● ●
●
●
● ●
●
●
●
●
●
●●
● ● ●●
●●● ●
●
●
● ●●● ● ● ●
●●
●
●
●
●
●
●
●
●
●●
● ●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●● ● ●
●●
●
●●●●●
●●
●
●
● ●
●
●
●
● ●
●● ●●
●●
●
● ●
20.0
10.0
1.0
2.0
5.0
RILE Standard Error
●
●
●
●
●
20
50
100
● ●
●●
●
●
●
● ●
●●
●
●● ●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
●
●●
●●
●
●
●
●
● ● ●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●●●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●● ● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
● ●●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ●●●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
● ●
●●●
●
● ●●
●●
0.5
0.2
0.5
1.0
2.0
5.0
●
●
0.1
PER501 (Environment) Standard Error
10.0
(a)
500
●
2000 5000
10 20
Number of Quasi−Sentences
50
200
500
2000
Number of Quasi−Sentences
●
0.2
0.5
1.0
2.0
5.0
●●
●
● ●●●●
●
●●●●●
●●
● ●●
●●
●
●
● ●●
●●
●● ●
●
●●●●●●●●
● ●
●
●●● ●●
●
●
●
●●
●●●
●●●
●●●
●
●
●●
●●●
●
●
●
●●
●●
●
●●●
●
●●
●●●
●
●●●
●
●●
●●
●
●
●●●●
●●●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●●●●●
●
●●
●
●
●
●●
●●●
●
●●●
●●
●
●●
●
●
●
●
●
●●
●●
●●
●
●●
●
●
●● ●●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●●
●●●
●
●
●●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●●
●
●●●
●
●
●
●
●
●
●●
●
●●●
●
●●
●●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●●●
●●
●
●
●
●
●
●
●●
●
●
●●●
●
●●●
●●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●●
●
●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●●
●●
●
●●
●
●
●●
●
●●
●
●
●
●●●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●●
●●
●
●●●
●
●
●
●
●●●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●● ●
●●●
●●●
●
●
●
●●
●
●●
●
●
●
●●
●
●
●●
●
●●●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●●
●
●
●●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●
●
●●
●●
●●●●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●●
●●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●●
●
●●
●
●
●●●
●● ●
●●●
●●●●
●●●●
●
● ●
●
●●
●●
●●
●●
●●●●●
●
●
●
●
●
●● ●
●●●●
●●●●
●●
●●●
●
●
● ●●
●
●●
●●
●
0.1
Bootstrapped PER501 (Environment) Standard Errors
10.0
(c)
●
●
0.1
0.2
0.5
1.0
2.0
5.0
10.0
Analytical PER501 (Environment) Standard Errors
Figure 2: Comparing estimated error variances for Environment (PER501) and the CMP Left-Right
scale (RILE). Axes use log scales.
39
10.0
(a) Environment (PER501)
2.0
1.0
0.5
0.2
●
0.1
S.E. with random N
5.0
●
●
● ●
●
●
●● ●● ●
●●●
●
●●
●●●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
● ●●
●●●●●
●
● ● ●●
●●●●●●
●
●
●●
●
●
●●
●●●
●●
●●
●
●●
●
●
● ●
●
●
●●●●●
●
●
●
● ●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●●
●
●
●●
●
●●●
●●●
●●
●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●●●●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●● ● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●●●●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●●●●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●●●
●● ●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●● ●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●●
●
●
●●
●
●
●
●
●
●●
●
● ●●
●
●●
●
● ● ●●●●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●●
●
●
●
●●
●●
●
●
●●
●●●
●
●
●
●
●
●
●
●
●●●●
●
●●
●
●●
●●
●
●●●
●●
●●
●
●
●
● ●●
●
●
●●●
●●
●
●
●
●
●
●●●
● ●●
●
●
●● ●
●●
●●●
● ●
●●
●
●
●
●
● ●
●
●
● ●●●
●●●
● ●●
● ●
●
●
●
●
●
0.1
0.2
0.5
1.0
2.0
5.0
10.0
S.E. with fixed N
(b) RILE
10.0
5.0
2.0
●
●
● ●
● ●●●
● ●
●●
●● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●●●
●
●
● ●●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●●●●●●
●
●
●
●
●
●●
●
●
●
●
●●
●● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
●
● ●●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●
●
●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●●
●●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●●●●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●
●
●● ●●●
●
●
●
●
●
●
● ● ●● ● ●●●●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
● ●●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●
●● ●
●●
●
●
●●●
●●●
●●
●
●
●
●●●
●●
●●
●
●●
●
●
●●
●
●
● ●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ● ● ● ●
●
●
●●●
●●
●●
●
●● ●
●
●
●
●●●
●
●●●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●●● ●
●●
●●
●●
●● ●
●●
●
●
●
●
● ●●
●
●
0.5
1.0
S.E. with random N
20.0
●
●
●
0.5
1.0
2.0
5.0
10.0
20.0
S.E. with fixed N
Figure 3: Comparing bootstrapped standard errors when n is fixed to when n is randomly determined,
for Environment (PER501) and the CMP left-right scale (RILE). Axes use log scales.
40
25
20
15
10
●
●
●
5
●
●
●
●
●
●
●
●
●
●
●
−5
0
CMP Environment Dimension (PER501)
●
1950
1960
1970
1980
1990
2000
German CDU−CSU
Figure 4: Movement on environmental policy of German CDU-CSU over time. Movement of Dashed
line is % environment with 95% CI; dotted line is the number of quasi-sentences per manifesto coded
PER501.
41
(a) Germany 2002
CDU−CSU
●
SPD
●
FDP
●
Greens
●
PDS
●
−40
−30
−20
−10
0
10
20
30
●
CMP Right−Left Dimension
(b) France 2002
FN
●
UDF
●
UMP
●
Verts
●
PS
●
PCF
●
−30
−20
−10
0●
10
20
30
40
CMP Right−Left Dimension
Figure 5: Left-Right placement of German and French parties in 2002. Bars indicate 95% confidence
intervals.
42
−1.0
●
●
●
2.5
3.0
−2.0
−3.0
●
−4.0
Noncentrist policy shift
●
●
0.0
0.5
1.0
1.5
2.0
(1 + λ)
Figure 6: SIMEX plot for Noncentrist policy shift in Table 2 in Adams et. al. (2006). λ is the amount of
noise added to variance of error-prone covariate. (1 + λ) = 1 is the naive estimate with no extra noise
added. Measurement error is added in the increments of .5 to establish a pattern, and then extrapolated
back (quadratically) to the case of no measurement error (λ = −1).
43