How to Measure Institutions
Stefan Voigt
Philipps-University Marburg
Talk prepared for World Bank Seminar on “The Empirics
of Governance”, May 1 - 2, 2008
1
How to Measure Institutions
I. Introduction
•
•
•
Statement „Institutions Matter“ has become commonplace.
In order to hold any water, two conditions need to be met:
– The world does not solely exist of institutions;
– Institutions can be empirically ascertained.
This talk: a number of considerations regarding measurement of
institutions, 4 main messages:
(1) Try to measure specific institutions rather than aggregate institutional
systems;
(2) Objective indicators are generally preferable over subjective ones;
(3) Measure both de jure AND de facto;
(4) Ability to measure does not imply capacity to change institutions.
Voigt: How to Measure Institutions
2
I. Introduction (2)
Structure of the Talk:
II.
Critique: Have We Been Measuring Policies all these years?
III. Defining Institutions: A Proposal
IV. Measurement Follows Theory: Assumptions and Implications
V.
Measuring Institutions: Some Pragmatic Proposals
VI. Conclusions and Outlook
Voigt: How to Measure Institutions
3
II. Critique:
Have We Been Measuring Policies all these years?
Glaeser et al. (2004): Yes !
– Institutions are supposed to
• Constrain behavior
• Be stable over time.
– Many of the frequently used measures (they cite the ICRG, the
World Bank Governance Indicators and Polity IV) would neither
capture policy constraints nor would they be stable. They would,
rather, measure outcomes, i.e. policy choices.
– The subjectivity of these measures makes them likely to be
influenced by income levels (or other outcome variables).
 Two serious points! And: Two pitfalls to evade!
Voigt: How to Measure Institutions
4
III. Defining Institutions: A Proposal
•
•
•
Attempt to treat Checks and Balances Institutions Within the NIE
more generally
Institutions := Commonly known rules used to structure recurrent
interaction situations that are endowed with a
sanctioning mechanism.
A more fine-grained delineation according to who sanctions:
(1) Convention (self-enforcing);
(2) Ethical rule (self-commitment of actor);
(3) Customs (informal societal control);
(4) Private rule (organized private enforcement);
(5) State law (organized state enforcement).
•
(1) – (4): sanctioning does not rely on state  „internal institutions“
Voigt: How to Measure Institutions
5
III. Defining Institutions (2)
Distinction between Political and Economic Institutions frequent
– E.g. Acemoglu et al (2004):
• Economic institutions „determine the incentives of and the
constraints on economic actors“;
• Political institutions „determine the constraints on and the
incentives of the key actors, this time in the political sphere“.
 Political institutions determine economic institutions.
 Implementation of political institutions precarious as it is unclear
who could sanction non-compliance  checks and balances as
one possible solution.
Voigt: How to Measure Institutions
6
IV. Measurement Follows Theory: Assumptions and
Implications
• Assumption 1: Effects of institutions due to their material content and
their factual implementation.
• Implication 1: Measure both de jure and de facto.
• Assumption 2: The constraining effect of institutions is a function of
their factual enforcement which depends on the behavior of those who
are to sanction non-compliance.
• Implication 2: Behavior of enforcers (police, prosecutors, judges,
prison staff, but also the press, lobby groups, public at large) needs to
be explicitly taken into account.
 Messy but important!
Voigt: How to Measure Institutions
7
IV. Measurement Follows Theory (2)
• Assumption 3: Factual behavior is likely to be determined by a number
of institutions.
• Implication 3: Identify all institutions that potentially constrain behavior.
• Assumption 4: The factual enforcement of political institutions is often
highly precarious and a function of internal institutions.
• Assumption 5: Internal institutions are largely exempt from intentional
modification
• Implication 4: If factual enforcement of political institutions depends on
internal institutions, then political institutions should not be entirely at
odds with them!
Voigt: How to Measure Institutions
8
V. Measuring Institutions: Some Pragmatic Proposals
(1) Before attempting any measurement, a clear concept of the
institution is needed.
(2) Predict behavior that would be observed were the institution
truthfully implemented.
 E.g. if judges are appointed for twelve years, expected average
tenure should be twelve years.
(3) „Measure“ the factually observed behavior.
 Less difficult regarding political than economic than political
(because number of actors is smaller);
 Sometimes, appropriately chosen sample will do (e.g. only highest
court judges).
 Measure behavior over appropriately chosen period of time.
Voigt: How to Measure Institutions
9
V. Measuring Institutions (2)
(4) Compare (2) with (3).
 Appropriate „measuring rod“ depends on the issue at stake: dummy
vs. continuous variable etc.;
 Observed variation might also play a role.
(5) If at all possible, rely on objective measures.
 Subjective measures often tainted by dependent variable;
 Subjective measures influenced by many diverse and implicit
theories;
 Objective measures (how many times has X happened between
1990 and 2000?) can be verified.
 Yes, objective and de facto can be mixed!
Voigt: How to Measure Institutions
10
V. Measuring Institutions (3)
(6) Make coding transparent.
 Allows other researchers to use the data for their research
questions (and recode, reweigh etc.)
(7) If aggregating institutions into indicators, use standard
techniques.
Voigt: How to Measure Institutions
11
V. Measuring Institutions (4)
Some Examples
(1) Central Bank Independence (and accountability) has been
measured drawing on objective indicators; but focus clearly on
de jure  de facto remains to be done;
 E.g. number of changes in legal bases, qualification requirements,
development of governor‘s salary …
(2) Also true for regulatory agencies (telecom, energy, water, …);
(3) Judicial Independence;
Voigt: How to Measure Institutions
12
V. Measuring Institutions (5)
Some Examples (cont.d)
(4) Most relevant aspects of the rule of law.
 Generality of rules,
 their abstractness,
 certainty of law,
 Absence of retroactive legislation;
 Expropriation without compensation;
 Habeas corpus;
 Protection of confidence;
 Principle of proportionality …
Voigt: How to Measure Institutions
13
VI. Conclusions and Outlook
• Ability to measure institutions combined with econometric
findings showing their significance for explaining variation in
output variables is not sufficient for reforming or modernizing
institutions at will.
– Beware of premature policy conclusions!
• Many (most?) of the currently available indicators suffer from
serious flaws  major data collection exercises ahead of us!
Voigt: How to Measure Institutions
14
Thank you for your attention!
Voigt: How to Measure Institutions
15