Research Paper No. 1943
Selection Criteria for Roll Call Votes
Keith Krehbiel
Jonathan Woon
September 2005
RESEARCH PAPER SERIES
Selection Criteria for Roll Call Votes
Keith Krehbiel and Jonathan Woon
Stanford University and Carnegie-Mellon University
Prepared for the meetings of the American Political Science Association
Washington, D.C.
September, 2005
Abstract
On grounds of inclusion of undesirable votes (type I errors) and exclusion of
desirable votes (type II errors), we question the convention of selecting only finalpassage votes for roll call analysis. We propose an alternative selection method based on
the estimated salience and strategic significance of roll calls and argue that this method
reduces type I and type II errors. We demonstrate that selection of roll calls based on
alternative criteria has a major bearing on the asymmetry of partisan roll rates and, we
conjecture that its application will also be substantively significant in other modes of
inquiry.
Selection Criteria for Roll Call Votes
Keith Krehbiel and Jonathan Woon
Politicians play multiple games in multiple arenas before multiple audiences. Among the
many manifestations of this political complexity, roll call voting is one of the most thoroughly
studied. But a given roll call cast by a given legislator on a given day may be of major, minor, or
no consequence, depending upon individual and collective circumstances. As a general
proposition, the political (and hence scholarly) relevance of such behavior is proportional to the
salience and the strategic significance of the voting situation. Salience is of particular importance
from the individual’s perspective. A reelection-seeking legislator is keenly concerned about the
consequences of his roll call when media, interest groups, or constituents are watching—and less so
when they are not. Such is the essence of position-taking. Strategic significance, in comparison, is
an attribute of the inside game of collective choice and its consequences for policy outcomes. Is
the agenda structured optimally for a minimal winning coalition to capture benefits from a
minority? At what stage of the legislative process does this happen? Which votes are pivotal in
determining the crucial contested content of policy outcomes as opposed to simply date-stamping
the end of a piece of legislative business?
The challenge of systematically identifying salient and strategically significant roll calls can
be illustrated with a recent example. On July 21, 2005, the US House of Representatives chose to
consider, debate, amend, and pass HR 3199, Reauthorization of the Patriot Act. In the course of its
deliberations, the House recorded 14 roll calls, all of whose consideration was provided for by
H.Res. 369, the rule providing for consideration of the bill. The two roll call votes on the rule itself
were straight party-line votes: 224-197 on ordering the previous question, and 224-196 on agreeing
1
to the resolution. Thereafter, most of the amendment activity was successful and lopsided, some of
its seemingly trivial as well. For example, roll call vote 407, which passed 418-7, changed the term
“domestic terrorism” to “federal crime of terrorism.”
Not everything was trivial, however. In particular, a motion to recommit with instructions
that would impose sunset provisions on the bill was offered by a majority party member (Rep. Rich
Boucher of Virginia). It not only attracted unanimous Democratic support and enough Republicans
and abstentions to fail by an unusually small margin, 209-218, but also attracted considerable press
attention, such as a reference in the Wall Street Journal to Republicans’ “heavy-handed tactics.”1
Likewise, an amendment offered by Democratic Representative Sheila Jackson-Lee was adopted in
a reasonably close vote of 233-192, with unanimous minority party support (200-0), substantial
majority-party defections (45), but opposition of a majority of the majority party. In other words,
the majority party was “rolled.” These close calls vanished, however, when it came time to vote
on final passage. The amended bill was adopted 257-171.
We cite this example neither to express nor imply any substantive conclusions. Indeed, our
lack of special expertise and inside information about the reauthorization of the Patriot Act is, in
many respects, the point of the example. Analysts of roll call votes in large-N studies rarely have
special expertise or inside information about their hundreds or thousands of observations. Yet, in
this case, and undoubtedly many others like it, the convention in recent years is to include only the
relatively lopsided and often uninformative vote on final passage, to emphasize (in examples like
this one) that the roll call is an instance of the majority party rolling the minority, and to ignore the
1
“House Majority Limits Changes in Patriot Act,” Wall Street Journal, Friday, July 22, 2005, page A5.
2
earlier and closer votes that earned national coverage and in which the minority party rolls the
majority party.
Although researchers have become increasingly innovative in exploiting roll call votes for
theory-testing purposes, alongside innovations come increasing concerns about which roll calls to
use for which purposes. Whether by accident or design, and for better or for worse, the emerging
convention is to focus on final passage votes.2 Thorson (1998) singles out such votes for being
distinctively helpful for testing hypotheses about unified versus divided government. Jenkins,
Crespin, and Carson (2003), Thorson and Nitzschke (2000), and Van Houweling (2001) argue that
final passage votes are highly significant, and that they manifest systematically different forces than
earlier votes (see, however, Roberts and Smith 2003). Cox and McCubbins’s (2004) extensive
study focuses overwhelmingly on final passage votes, while Wiseman and Wright (2005) and
Krehbiel, Meirowitz, and Woon (2005) conform with the convention and consider only votes on
final passage (see, however, Krehbiel 2005). The explanation for this academic equivalent of pack
ball3 is not entirely clear, but there appear to be two forces at work: a push effect and a pull effect.
The push factor is the absence of a theory that has clear implications about how to interpret
pre-final-passage roll call voting. For instance, while models of agenda setting (Cox and
McCubbins 2004) or pivotal politics (Brady and Volden 1998, Krehbiel 1996) provide necessary
and sufficient conditions for changing a status quo policy, they are silent about the procedural
2
Definitions vary, but a common if not modal definition is Rohde’s (http://www.msu.edu/~pipc/pipcdata.htm), which
includes votes on adoption of a bill, conference report, a resolution, a joint or concurrent resolution, or passage of any
of the aforementioned under suspension of the rules (in the House).
3
Pack ball—sometimes called kids’ soccer—is a game in which unquestioning players unconsciously and
oxymoronically preserve a radius-minimizing spread as they occasionally move to, but mostly park at, some random
place on the field.
3
choices that initiate the lawmaking process and about the proposal and destination-significant
amendment behavior that transpires between procedural and policy choice stages. This fact seems
to make empirical researchers skittish about drawing inferences based on roll calls that occur in the
early and middle stages of the legislative process.
The pull factor is that, in the case of final-passage votes, the nature of the collective choice is
simple, relatively well-defined, and binary. Roberts and Smith put it succinctly: “Final passage
votes, conference report votes, and veto override votes come very close to pitting a policy
alternative against the status quo” (2003, 309). The corresponding attraction of final passage votes
may inhere in the presumption that such votes, and only such votes, are clean instances of
legislators’ making a well-defined choice between an equilibrium proposal p* and a status quo
policy q.4
Whatever the sources of the convention, the selection criterion of final passage is error-prone
in each of two ways, summarized in table 1. First, a high proportion of votes are included that
should not be, either on conceptual grounds (i.e., they are not really final passage votes) or, more
seriously, on grounds that they are not informative with respect to specific research objectives.
These are parallels of type I errors in statistics, or false positives in many other contexts. The
specifics supporting the claim that a high proportion of final passage votes are false positives stem
from each of the three words that make up the term. Final passage votes are often not final, because
when a chamber first passes a bill, there is rarely an assurance that it will not revisit the content of
the bill later in the process, say, in amendments between the chambers, after a negotiated
4
An added layer of theoretical defense is available, though not commonly cited. In a complete information setting with
a finite binary agenda, sophisticated voting (i.e., non-truthful revelation of preferences) may occur at any stage except
one: the final vote (McKelvey and Niemi 1987). It can be questioned whether most congressional roll call situations
conform with these conditions (Groseclose and Krehbiel 1993).
4
conference committee settlement, or after a presidential veto. A given piece of legislation can
therefore be represented in a dataset several times with ostensible votes on final passage.5 Final
passage votes also occasionally fail, in which case they are arguably mislabeled and of
questionable theory-testing value for theories intended to predict outcomes. The reason, of course,
is that in the presumed final matchup between the status quo q and the ostensible equilibrium policy
p*, the latter is undeserving of its *. In other words, if the proposed policy does not pass, it
evidently is not an equilibrium policy.6 Finally, final passage votes that happen to meet both the
final and the passage criteria—the above arguments to the contrary notwithstanding—can also be
diluted in value for a number of procedural and behavioral reasons. As a literal matter, the last
congressional action on many bills that becomes law is often an act of unanimous consent or a
voice vote rather than a recorded roll call vote. While such exclusion is more of a type II problem
(see below) than a type I problem, it serves to underscore a closely related problem of a falsepositive sort. Specifically, as in the case of underlying preferences expressed in voice votes, it is
well-known that coalition sizes regularly exceed those predicted by formal models. This fact
suggests, and anecdotes further support the notion, that late-stage expressions via votes are often
more acts of capitulation, rationalization, or idiosyncratic position-taking, than they are compelling
indications that the supporting legislators strictly prefer policy state p* to policy state q. In
5
To be somewhat more precise, we regard this as more of a problem in logical consistency and operationalization than
a problem of false positives. That is, if one embraces the theoretical argument that only final passage votes can be
relied upon as instances of truthful preference revelation (see previous footnote), then the fact that final passage votes
as operationalized are often not the final say of legislators is problematic. Acknowledging that it is a subjective matter,
we tend not to find the theoretical premise to be compelling (see Krehbiel and Rivers 1990).
6
Bicameralism regularly causes this problem. For example, in “Daschle’s Dead Zone,” the Wall Street Journal (July
22, A12) identifies 11 major pieces of legislation that obtained majority [ostensibly final] votes in the House of
Representatives over three-month period in early 2003. Each bill, however, was subject to a Senate filibuster, and none
of the bills obtained cloture. Therefore, none of them passed.
5
summary, indiscriminant inclusion of final passage votes almost surely attenuates or biases the
inferences that could otherwise be made from late-stage roll calls.
[Table 1]
The second source of error-proneness under the final-passage criterion is exclusion of
significant and informative votes, i.e., errors of a false-negative type. The prevalence and severity
of type II errors is to some extent research-specific. Roberts and Smith, for example, reason that “if
party structures final passage or other forms of voting, party also should structure amendment
voting” (2003: 309). If so, then the exclusion under the final-passage criterion of all votes on
amendments guarantees that there will be a high incidence of type II errors. The baby is, in effect,
thrown out with the bath water. Viewed in this light, the error-proneness of the final-passage
selection criterion is disturbing.
Another selection strategy that merits brief attention is to go to the other extreme and include
all votes in roll call studies (see, for example, the voluminous body of work by Keith Poole and
various coauthors). This approach is an improvement in one respect and a setback in another.
Summarized in the second row of table 1, the improvement, of course, is that there are no type II
errors (exclusion of useful votes) simply because almost no votes are excluded.7 The setback is
that the very high number of idiosyncratic, trivial, and uninformative votes is likely to bias the
findings or, at the very least, inject a great deal of noise into the analysis.
This aim of this paper is to reduce the error-proneness of existing vote selection conventions
by screening roll calls systematically on the basis of a richer set of roll call attributes than is
7
The exception is that unanimous (and near-unanimous) votes are excluded because the provide no (little) information
about differences in preferences.
6
conventional. Our premise is that, other things equal, closeness of roll calls is indicative of the
strategic significance of roll calls, and attention by media sources, interest groups, and Washington
insiders is indicative of the salience of roll calls. We estimate salience and strategic significance
using factor analysis and demonstrate the consequences of the resulting selection criteria.
Data and Methods
The unit of observation in our dataset is a House or a Senate roll call vote. The dataset
includes every roll call in the 102nd through 107th Congresses with the exception of approximately
ten phantom observations for which a roll call number was assigned in spite of the fact that no
votes were actually cast.8 The number of observations is 10,789, with 6,679 supplied by the House
of Representatives and 4,110 by the Senate. Data were obtained from vote descriptions from
Congressional Quarterly’s website (www.cq.com). CQ’s vote descriptions are much more detailed
and reliable than those of the ICPSR and somewhat more detailed than those available from the
Library of Congress (www.thomas.gov). We wrote and ran Visual Basic and Perl programs to
extract various quantifiable characteristics from the vote descriptions. Text searches in Microsoft
Excel and queries in Microsoft Access were used to identify and code various vote attributes.
As summarized in the introduction, we are striving for measurement of two somewhat
amorphous and possibly related concepts that bear on the desirability of roll calls for analysis: the
salience and the strategic significance of voting situations. A single dummy variable indicating
where, in the sequence a bill’s history, a roll call occurred may have a bearing on these underlying
factors, but it not likely to be the sole defining characteristic of any such standard of desirability.
So, for instance, while late-stage or so-called final-passage votes are often important, mid-stage
8
Most commonly, the ephemeral roll calls were quorum calls rescinded before recorded votes were cast.
7
votes on amendments or early-stage votes on cloture or rules are surely not uniformly unimportant.
Therefore, they sometimes attract attention of interest groups, the media, or constituents. They are
often (though not necessarily) of substantive significance and are regularly (though not always)
pivotal in the outcome, even though a near-unanimous majority eventually forms on the final
passage vote.
Factor analysis is an acceptable method for the combination of objectives, opportunities, and
constraints we face. On the positive side, we have a set of vote-attribute variables that a priori are
indicators of underlying unmeasured concepts (factors). On the negative side, we have neither an
explicit theory nor a corroborated empirical hunch about how the indicators relate to the factors. In
between, we are confident in assuming that, over a suitably large number of roll calls, close votes
are considerably more likely to be of strategic and substantive significance than near-unanimous
votes. Likewise, media coverage, interest-group attention, and/or presidential involvement, for
example, are surely reflections of salience of roll calls and possible indicators of strategic
substantive, too.
Although we are generally comfortable with these conjectures, we are not sufficiently
knowledgeable about any such relationship to devise and defend a specific weighting scheme in
which we impose the relative degree to which these relationship hold. Exploratory or confirmatory
factor analysis is therefore a way to accommodate these strengths and weaknesses. Factor analysis
takes a set of (measured) exogenous variables and estimates their effects on a smaller set of
common (unmeasured) factors. Formally, it finds q common factors that linearly reconstruct p
original variables:
y ij = z i1b1 j + z i 2 b2 j + ...z iq bqj + eij ,
8
where yij is the value of the ith observation on the jth variable, zik is the ith observation on the kth
common factor, bkj is the set of coefficients (called factor loadings), and eij is a residual, i.e., the jth
variable’s unique factor (Spearman, 1904).
We use several indicators of salience or attention that roll calls attract. Two of these are
newspaper-based. Searches of text of the New York Times and the Wall Street Journal were
conducted using an online search engine, Factiva. The search required that the word “vote” appear
in the headline or lead paragraph.9 All articles meeting these criteria were downloaded and then
parsed using a Perl script that identifies indicators of vote tallies.10 Articles were then screened and
sorted into two types. False hits consist of those that either did not refer to votes after all, that
referred to a vote in the other chamber, or that referred to votes taken more than one week prior to
the publication date of the article and, therefore, failed to constitute current coverage. Confirmed
hits consist of articles that can be linked specifically to a recent, specific roll call in the House or
the Senate. The resulting variables NYT articles and WSJ articles are, therefore, counts of the
number of articles in the respective newspapers.
The variable CQ key vote is a similar indicator of salience albeit of a distinctively more
inside-the-beltway sort. It is simply a dummy variable equaling 1 if the Congressional Quarterly
Almanac identified the roll call as one of its approximately 30 “key votes” for the chamberCongress.
9
In addition to raising the salience standard by requiring that the press gives coverage to the roll call as opposed to the
congressional action more broadly, this requirement substantially reduces the number of false positive hits on actual
congressional roll calls.
10
For example, “#-#” or “# to #” where “#” denotes an integer in [0,435].
9
The variable AAP vote does the same for the Almanac of American Politics for about half as
many votes, while ACU vote codes the 553 roll calls that the American Conservative Union
selected on which to base its annual ratings.
A final, indicator of insider significance and/or salience of roll calls is President position: a
dummy variable equal to 1 if the president took a stance specific to the roll call.11
The more challenging measurement task is to find indicators that allow for the possibility of,
and that capture the strategic significance of, votes earlier in the legislative process than final
passage and thereby address the false-negative problem, yet that do not over-select earlier,
unimportant votes, thereby exacerbating false-positive errors. To do this, we use two measures:
mid-stage votes and closeness of vote.
We single out mid-stage votes by process of elimination. First omitted are roll calls that have
no direct bearing on the passage of a specific law. These nonlegislative votes in the House are
overwhelmingly are votes to approve the Journal; in the Senate, they are split between motions to
instruct the sergeant-at-arms to request the attendance of absent senators and confirmations of
executive appointees. Next-omitted are early-stage votes, which are often bill-specific but which
precede actual consideration of the legislation in question. In the Senate, these are votes are votes
to invoke cloture; in the House, they are votes pertaining to a resolution (rule) that specifies terms
under which a bill subsequently is considered. The final omission is late-stage votes, which is our
counterpart to what the literature calls votes on final passage. The remaining votes take on values
of 1 in the dummy variable. It is important to clarify that the method of factor analysis does not
11
It bears emphasis that this is roll-call specific. That is, if the president supported passage on, say, omnibus
reconciliation, the dummy variable is not 1 for all amendment activity on the budget package. Even so, the coverage of
this variable is relatively large: the president took a position on almost 12 percent of the roll calls.
10
dictate that this variable be systematically related to a common factor (i.e., to covary with other
indicator variables) but rather merely provides an opportunity to assess whether it seems to have
something in common with other indicators (including, possibly, all of the salience indicators).
Closeness of vote is the final variable in the analysis. It is intended to reflect the
competitiveness of the legislative process at critical junctures of bills’ histories. From an individual
legislator’s perspective, we wish to tap the perceived likelihood that a legislator could, by switching
her vote, be pivotal on the given roll call. We do this as a function of the aggregate realized vote
outcome as follows. Let y and n, respectively, be the number of yea and nay voters on a given roll
call. Let q be the proportion of votes required for passage of the motion in question, e.g., q = .667
for a veto override or suspension of the rules, q = 3/5 for invoking cloture12 or passing a motion on
the House Corrections Calendar, and q = (n+1)/2 for all simple majority motions. Finally, let v be
the minimum y such that
y
≥ q , that is, the size of a minimum winning coalition given the type
y+n
of roll call and parliamentary situation in which it arises. Then a measure of closeness that has
useful comparability properties both across roll calls with different passage thresholds and across
chambers is:
⎛ 1 ⎞⎛ | y − v | ⎞
⎟⎟⎜⎜
⎟⎟ .
1 − ⎜⎜
⎝ 1 − q ⎠⎝ y + n ⎠
When a minimum winning coalition forms and the vote is, by definition, as close as possible
(subject to a possibly-needed but trivial even/odd assumption), the right-most term is (almost) 0 in
12
Technically, q is strictly 60 votes even if fewer than 100 senators vote. We account for this properly in the
implementation. For purposes of introducing and understanding the logic of the measure, assume that every senator
votes.
11
which case the measure takes on its maximum value of (almost) 1. In contrast, when a maximallyoversized, unanimous coalition is formed, the difference between actual and needed votes, |y -v|, is
such that the resulting measure is 0.13
Findings
The factor analysis is based on the eight variables describe above. Although, by default, it
generates loadings for four factors, the drop off in eigenvalues (measures of variation accounted
for) between the second and third factors is precipitous, indicating that the third and fourth factors
are essentially random noise. Table 2, therefore, presents the factor loadings only for factors 1 and
2. Somewhat to our surprise in light of the inherently coarse nature of most of the righthand-side
variables, a sensible pattern emerges. All six of the salience-intended indicators load well on the
first factor (but not the second), while the closeness of vote and mid-stage vote measures represent
significant common variation in the second factor (while sharing relatively little variance with
factor 1).
[Table 2]
13
Due to the scale factor, 1 / (1 - q), the measure is normalized between 0 and 1 for y > v and, for simple-majority
votes (q = .5), normalized also for y < v. For different types of supermajority votes (q’ > .5), however, y = 0 will
produce different negative values. This is not problematic for two reasons. First the measure is still well-behaved in
the sense that, for a given number of voters, the farther is y from the winning threshold, the lower is the roll call’s
closeness value. There is nothing special or problematic about the possibility that roll calls requiring a supermajority
may take on negative values. Second, only 24 of 10,789 votes produce negative closeness values. One example is vote
511 in the first session of the 106th Congress in the House: a motion to suspend the rules and pass the bill to adopt a
Clinton-endorsed measure to increase taxes and user fees by $19.2 billion, to nearly triple the tobacco excise tax to $8
billion, to increase food inspection fees on poultry, livestock, and eggs by $504 million and forest service fees by $20
million, to increase a number of fees related to transportation by about $3 billion. The motion was rejected 0-419 and
earned a record low closeness measure of -1.
12
The empirical finding appears to be very robust. In addition to the iterated principal
components results presented in the table, we also estimated loadings using more common
principle-component factor analysis, and garden-variety (non-iterated) principal components. The
differences in these methods are statistically arcane and, in our case at least, of no practical
consequence. Regardless of the type of estimation, and regardless of the subsequent method of
rotation—varimax (orthogonal), oblique (promax), or none—a clean, two-factor pattern of
estimates, such as that shown in table 2, results. We are therefore quite confident that the statistical
pattern represents a genuine empirical regularity.
Even so, there is an element of interpretive art that goes with the supposed science. The
interpretation we give to factor 1 is relatively straightforward and, we expect, non-controversial.
Each of the six factor loadings that are clearly nonzero is positive and pertains to coverage of, or
special attention given to, a roll call. It therefore seems defensible to label this factor salience. The
interpretation of factor 2 is more challenging but not impossible. Notice, first, that none of the
designated salience indicators bears much relationship to the second factor; their loadings are all
near zero.14 Therefore, votes with high factor scores on this dimension will not be systematically
salient and will presumably capture more of the inside-the-beltway component of strategic
significance than the more public electoral-connection game present in factor 1. Notice, second,
that the two variables with large positive loadings, while substantively quite different from one
another, can be related plausibly in a legislative context. The uncovered factor is consistent with
the intuition expressed in the introduction that, perhaps often, strategically critical votes (as
indicated by closeness here) occur in the middle of legislation’s multi-stage journey through the
14
Because the results presented are not rotated, the two factors are orthogonal by construction. However, even when
rotated without an orthogonality constraint (promax rotation), the correlation between factors 1 and 2 is -.0055.
13
process, even though they may tend figuratively to fly under the radar of broader public scrutiny.
Likewise, the second factor also fuels suspicions that focusing too much on final-passage or latestage votes—which tend not to be as close15—probably ignores a lot of outcome-consequential
legislative voting behavior. Via this interpretation, it seems defensible to label the second factor
strategic significance of the vote. This labeling, however, comes with the caveat that salient votes
as defined by factor 1 are not necessarily insignificant.
With these findings, it is a straightforward matter to define alternative selection mechanisms
for roll calls which potentially reduce the incidence of both type I and type II errors. We simply
use the factor loadings with the data to produce factor scores F1 and F2 as linear functions of the
data, just as one would compute predicted values of a dependent variable from, say, coefficient
estimates in a linear regression equation.16 This process yields two continuous-variable factor
scores, F1 = salience and F2 = strategic significance. Finally, factor scores can be dichotomized at
various arbitrary points and used as selection criteria instead of the relatively crude final-passage
criterion.17
Consequences
Acknowledging the possibility that this process may be much ado about nothing, we wish
finally to investigate whether different roll call selection methods lead to different substantive
inferences. Our application consists of reconsidering roll rates, i.e., the proportion of times a
majority of one party is on the losing side of a winning coalition. The objective is not to reiterate
15
The average closeness of mid-stage votes (.64) is twice that of late-stage votes (.32).
16
Typically, factor analysis software rescales the loadings so that the resulting factor scores are normalized with mean
zero and variance each approximately equal to 1 .
17
More sophisticated applications could weight observations by the factors. We shall not pursue this possibility here.
14
or revisit the ongoing dialogue about whether such measures represent what their proponents
purport (Cox and McCubbins 2004, Krehbiel 2005). Rather, we focus simply on the consequences
of applying roll rate measures to different subsets of roll call votes.
Table 3 confirms that the consequences can be substantial. Column 1 summarizes the finding
in the seminal work on roll rates, which is based on final passage votes in the House from the 45105th Congresses. Among these 3,134 roll calls, the minority party was rolled somewhat more than
one-fourth of the time, while the majority party was rolled fewer than two times in 100—a striking
finding that serves as the basis for strong claims about majority party dominance.
[Table 3]
Because in the present study, we use less data, more recent data, and data from the Senate as
well as the House, comparability with the Cox-McCubbins benchmark rates might be a concern.
Column 2 shows that this is not a problem. Employing our close-cousin criterion for final-passage
votes—late-stage votes—the measured rates remain in the same ballpark. If anything, our late-
stage criterion supports an even stronger claim than those in the earlier works, since the column-2
ratio of minority to majority party rolls exceeds 20:1.18
The consequences of the alternative selection criteria are shown first in column 3, which
calculates roll rates and the roll ratio using our entire dataset. The proportion of minority party
rolls goes up slightly, the proportion of majority party rolls increases five-fold, and the roll ratio,
correspondingly, goes down by a factor of four. While this change from the final-passage
18
An Appendix [in progress] explores some discrepancies between columns 1 and 2 by comparing reports of rolls in
the chamber (House) and congresses (102-105) in which out data overlaps with Cox and McCubbins’s.
15
benchmark is large, the indiscriminate inclusiveness of this criterion undoubtedly admits literally
thousands of type II errors.
The more important findings are those criteria in columns 4 and 5 that use the factor-based
selection criteria. To make the Ns in the set of selected votes approximately equal to those in the
baseline columns, we selected the top quintile of our derived measures of salience and strategic
significance and then calculated roll rates for these two sets of approximately 2200 votes. The
consequences are nearly identical using either criterion, even though they are, by construction,
uncorrelated and therefore reflect different notions of importance in roll calls. Specifically, the
minority roll rate rises again —now to at least 1 in 3. The majority roll rate, meanwhile, increases
much more dramatically to roughly 10 times what it was in the set consisting of only late-stage
votes in column 2. These effects combined reduce the roll ratio to 3.7 for each criterion.
Finally, column 6 imposes a more stringent filter, requiring that the roll calls be in the top
quintiles of both of the new criteria. The result is as expected based on the current analysis but
probably not as expected based only on prior analyses that uses the final-passage criterion. The
majority party now is rolled roughly 1 in 6 votes, and the roll ratio dips below 2:1. The comparison
between columns 1 and 6 is dramatic. Selection criteria for roll call votes appear to matter greatly.
Summary
We question the convention of heavy reliance on final passage votes for roll call analysis on
grounds that (1) often, such votes are unimportant, and (2) many non-final-passage votes are
important. Defining importance and operationalizing a corresponding selection mechanism,
however, is challenging in the absence of a theory of roll call importance. We addressed this
challenge with what might be described as a method that lets the data speak for itself, making
16
minimal and theory-neutral assumptions about likely covariates of salience and strategic
significance of roll calls and using factor analysis to uncover the specifics about these relationships.
Our tentative conclusion is that, relative to the final-passage criterion, employment of our
salience and strategic-significance criteria dramatically reduces type I errors by omitting finalpassage votes that are not salient and are of little strategic importance. Type II errors (false
negatives) are also diminished by allowing many otherwise-excluded high-scoring pre-finalpassage votes into the selection pool.
Still unknown, however, is the degree to which our method’s inclusion of close mid-stage
votes opens the door to another kind of type I errors, such as the inclusion of close but frivolous
amendments. We doubt whether this is a regular occurrence in the salience scores inasmuch as
such roll calls tend not to be covered by major newspapers, selected by interest groups, designated
as “key votes,” etc. However, if such roll calls are close, they probably pass through the current
strategic significance filter. We plan to assess the extent of this problem in future work and to finetune the factor analysis by adding other indicator variables that discriminate more finely between
roll calls.
17
References
Brady, David and Craig Volden. 1998. Revolving Gridlock. Westview Press.
Cox, Gary and Mathew McCubbins. 2004. Setting the Agenda. Manuscript: UCSD.
Groseclose, Tim and Keith Krehbiel. 1993. The Pervasiveness of Sophisticated Sincerity (with
Timothy Groseclose). In Political Economy: Institutions, Information, Competition and
Representation. W. Barnett, M. Hinich, H. Rosenthal, and N.Schofield, eds., New York:
Cambridge University Press, (1993) 247-78.
Jenkins, Jeffrey, Michael Crespin, and Jamie Carson. 2003. Parties as Procedural Coalitions in
Congress: Evidence from a Natural Experiment. Manuscript: Northwestern.
Krehbiel, Keith. 1996. Institutional and Partisan Sources of Gridlock: A Theory of Divided and
Unified Government. Journal of Theoretical Politics, 8: 7-40.
Krehbiel, Keith. 2005. Macro Politics and Micro Models: Cartels and Pivots Reconsidered. In
The Macro Politics of Congress, eds E. Scott Adler and John Lapinski. Princeton: Princeton
University Press. Krehbiel, Keith, Adam Meirowitz, and Jonathan Woon. 2005. Testing
Theories of Lawmaking (with). In Social Choice and Strategic Decisions: Essays in Honor of
Jeffrey S. Banks, eds. David Austen-Smith and John Duggan, Berlin: Springer (pp. 249-268,
2005).
Krehbiel, Keith, and Douglas Rivers. 1990. Sophisticated Voting in Congress: A
Reconsideration. Journal of Politics, 52: 548-78.
Krehbiel, Keith. 2005. Partisan Roll Rates in a Nonpartisan Legislature. Journal of Law,
Economics and Organization forthcoming.
18
McKelvey, Richard and Richard Niemi. 1978. A Multistage Game Representation of
Sophisticated Voting for Binary Procedures. Journal of Economic Theory. 18 (June): 1-22.
Poole, Keith and Howard Rosenthal. 1997. Congress: A Political-Economic History of Roll Call
Voting. New York: Oxford University Press.
Roberts, Jason and Steven Smith. 2003. Procedural Contexts, Party Strategy, and Conditional Party
Voting in the U.S. House of Representatives. American Journal of Political Science.
Spearman, Charles. 1904. General Intelligence Objectively Determined and Measured. American
Journal of Psychology. 15, 201-293.
Thorson, Gregory and Tasina Nitzschke. 2000. Politics and Policy in the 103rd and 104th
Congresses: Evaluating the Effects of Divided Government in the Postreform Era. Congress
and the Presidency.
Thorson, Gregory R., Divided Government and the Passage of Partisan Legislation, 1947-1990
Political Research Quarterly 51 (3): 751-64.
Van Houweling, Robert. 2001. Legislators’ personal policy preferences and partisan legislative
organization: special rules in the post-Reform House, Manuscript: Berkeley.
Wiseman, Alan and John R. Wright. 2005. The Legislative Median and Partisan Policy
Manuscript: Ohio State.
19
Table 1. Error-proneness in roll call selection
Type I errors
Inclusion of undesirable votes
False positives
Type II errors
Exclusion of desirable votes
False negatives
High
High
Very High
0
Selection Criterion
Final passage votes
All votes
Table 2. Factor analysis
Variables
Factor Loadings
1
2
NYT articles
WSJ articles
CQ key vote
President position
ACU vote
AAP vote
Closeness of vote
Mid-stage vote
0.476
0.298
0.484
0.383
0.412
0.405
0.221
-0.135
-0.086
-0.071
-0.034
-0.045
0.108
0.008
0.591
0.611
*Factors are orthogonal and unrotated. Eigenvalues
for factors 1-4: 1.097, .749, .136, .030. Factors 3
and 4 omitted.
20
Table 3. Roll rates under different vote-selection criteria
1
Rolls
2
Passage
3
Late-stage
All
4
5
6
Salient
Strategically
Significant
Both
Minority
0.259
0.219
0.256
0.334
0.385
0.305
Majority
0.016
0.010
0.049
0.091
0.104
0.176
16.6
21.9
5.2
3.7
3.7
1.7
3,134
2,932
10,789
2,159
2,159
364
Ratio
N
*Column 1 data are as reported in Cox and McCubbins (2004, 175) and on the House of
Representatives from the 45-105th Congresses. All other columns summarize both House and
Senate data from the 102-107th Congresses.
21