Experimental Economics: an Introduction
Dr Melis Kartal ([email protected]),
Dr James Tremewan ([email protected])
Introduction
Overview
• Why run experiments?
• What are experiments?
• What can we learn from them?
• What are the advantages and disadvantages of experiments
relative to theoretical or empirical approaches?
• Some basic design decisions and methodological issues:
• Deception.
• Framing, incentivisation, one-shot vs repeated,
Partner-matching vs stranger-matching, incentivisation,
sequential vs strategy method, within vs between subject design
etc.
2/15
Why Experiments
Types of Experiments (a rough taxonomy)1
• Conventional lab experiment: uses students as subjects, an
abstract framing, imposed set of rules.
• Artefactual field experiment: a conventional lab experiment but
with a non-standard subject pool, e.g. real traders or Papua
New Guinean villagers.
• Framed field experiment: same as artefactual field experiment
but with field context in either the commodity, task, or
information set, e.g. trading of sports cards at show (List; 2001).
• Natural field experiment: same as framed field experiment, but
subjects naturally undertake the tasks and are unaware they are
in an experiment, e.g. changing notices and seed money in
donation box.
1
3/15
Based on ”Field Experiments”, Harrison and List (2004).
Why Experiments
Why run experiments?
• Identifying causality: random assignment to treatments allows
identification of causality e.g. do high-powered incentives
increase effort?:
• can compare data on productivity of workers paid flat wage and
those paid piece rate but maybe more highly motivated workers
choose jobs paid by piece rate muddying identification.
• in the lab we can randomly assign subjects from the same pool
a flat wage or piece rate so no selection effect.
• Control: can simplify the environment to focus on specific
effects, and vary parameters as we need to to test specific
theories.
4/15
Why Experiments
Why run experiments?
• Testing theories with hard-to-measure variables: often theories
involve parameters that are hard to measure accurately, such as
marginal cost or willingess to pay for a good. In the lab we can
assign these values directly.
• Observing hidden behaviour: much economic behaviour is hard
to observe, such as corruption, but can be simulated in the lab.
• Testing mechanisms cheaply: policies that may be expensive to
test in the field, or are one-shot in nature, can be pre-tested in
the lab, e.g. auction to pay Georgian farmers to reduce irrigation
(Cummings, Holt, and Laury; 2004).
5/15
Why Experiments
Defense of common criticisms of experiments
• Most criticisms relate to ”external validity”: can we extrapolate
what we learn in the lab to the ”real world”?
• Lab environment ”not real”: but economic experiments are real
people making real decisions with real consequences. Also,
general economic theories should also apply in specific
environments: a theory that doesn’t work in the controlled
environment of the lab can be rejected as a general theory.
• Students make different decisions: students may make more
mistakes or exhibit biases not seen in experienced traders or
CEOs; students populations can be more homogenous than the
general population. True, but results using students can be
supported by studies using other subject pools.
• Theory, experimental, and empirical approaches have different
strengths and weaknesses, and ideally evidence from all three
should be used.
6/15
Design Issues
Deception
• The golden rule of experimental economics: ”Thou shalt not
•
•
•
•
7/15
deceive thy subjects!”
Why not? It is crucial that subjects believe the experimenter,
e.g. if we tell subjects they will be paid by piece rate but they
believe they will be paid a flat wage at the end then we may
misleadingly find no effect of high-powered incentives.
If subjects are deceived in one experiment, or hear or read of
studies where deception has taken place, they may have
suspicions in future experiments, e.g. MacCoun and Kerr (1987).
In experimental economics an unpolluted subject pool is seen as
a public good, and studies using deception are unlikely to be
published or permitted in economics labs.
There can be a fine line between deception and acceptable
ommission of information, but tread carefully!
Design Issues
Incentivisation
• Subjects in economic experiments should be paid according to
performance. It is highly unlikely that a non-incentivised
experiment will be published in an economic journal.
• It is a common assumption that subjects without appropriate
financial incentives will not exert effort to make optimal choices.
• Empirically this is not so clear. Some ”stylized facts”:2
• Where incentives matter, the difference is between hypothetical
and real payoffs rather than low and high payoffs.
• Improvement in mean performance occurs in boring tasks with
high marginal returns to effort.
• Incentives can reduce mean performance (e.g. induce
”over-thinking” when intuitive answer is right).
• Incentives can reduce variance in performance, which is
sometimes useful.
• Incentivised subjects are greedier and more risk averse.
2
8/15
The following is based on Camerer and Hogarth (1999).
Design Issues
Framing
• Most economic experiments are done without context (a
”neutral frame”), e.g. choose option A or B, rather than
”cooperate” or ”defect.” Remember: lab experiments are not
about re-creating reality!
• This is done to retain generality, avoid inducing particular
behaviour because of connotations of words, or ”role-playing.”
Traditionally economic theory has held the view that arbitrary
labels should not matter.
• However, researchers may use framed experiments if they are
interested specifically in the effects of context.
• Also, in some instances, not framing experiments can lead to a
loss of control: subjects may apply their own context to help
understand an abstract situation, and if different subjects apply
different contexts there will be uncontrolled heterogeneity.
9/15
Design Issues
Anonymity
• Typically subjects in economics experiments do not know the
identity of the subjects with whom they are interacting (they will
know they are interacting with someone in the room, but not
precisely who). This controls for effects related to individual
characteristics that may influence behaviour in others.
• Subjects are usually told that their name will not be connected
to data from the decisions they have made, so subjects make
more ”natural” decisions as they would if unobserved.
• Sometimes experimenters are interested in how specific elements
of identity affect decisions, so gender, class, nationality, etc may
be revealed, either directly or indirectly (e.g. by name).
• As a rule of thumb, you need a good reason to run a
non-anonymous experiment.
10/15
Design Issues
One-shot or repeated?
• A game can be played once or many times.
• Reasons for repeating games:
• It may take time for subjects to learn to understand game.
• May be interested in learning process.
• Getting more observations.
• Possible problems with repeating games:
• Dilutes incentives.
• Extra observations will not be statistically independent.
• Note: when a game is repeated, typically only one randomly
chosen round is paid. This is to avoid ”wealth effects” (once a
subject knows they have already earnt a certain amount they
may become more risk-averse or exert less effort, but we want
each game they play to be comparable.)
11/15
Design Issues
Partner or stranger-matching?
• If a game is repeated, subjects can play with the same person
(people) each time, or be randomly rematched.
• Stranger matching: randomly rematched every time. May only
interact once with a given subject (”perfect stranger matching”)
or possibly multiple times. Typically anonymity means that in
the latter case subjects will never be sure whether they have
played with a subject before or not. Used if interested in
one-shot interactions.
• Partner matching: play with the same subjects each time. This
option will be used if interested in repeated-interactions e.g. to
study reciprocity or reputation effects. Essentially just
implementing a ”repeated game.”
12/15
Design Issues
Within/Between Subject Design
• When looking for a difference in behaviour between two
treatments we can choose either a within or between subject
design.
• Within subject design: Each subject participates in both
treatments.
• Subjects may choose the same in both treatments to be
consistent.
• May make purpose of experiment obvious (”experimenter
demand effect”), or highlight treatment variable.
• Has greater statisitical power.
• May be interested in effect on individuals rather than averages.
• Between subject design: Each subject participates in only one
treatment.
• Less statistical power for same number of subjects.
• Fewer worries about order and demand effects.
• Problem with subjective judgements: can show 9 > 221!
13/15
Design Issues
Order effects
• Sometimes we want the same subjects to make multiple
decisions: the order in which subjects complete tasks may affect
their decisions.
• Examples:
• Consistency and demand effects (as in within subject design).
• Success in a task may increase risk taking in future tasks
(winner effect: related to testosterone).
• Belief elicitation prior to a task may increase understanding of
the task.
• Beliefs elicited after a task may be adjusted to be consistent
with or to justify previous decisions.
• Often experiments are ”counterbalanced” (half of subjects to
task A first then task B, half do B then A). Can then test for
order effects.
14/15
Design Issues
Strategy method
• Dynamic games can be implemented in two different ways, e.g.
the sequential prisoners’ dilemma:
• Direct-response: P1 makes a decision, P2 is informed of the
decision then makes their own decision.
• Strategy method: P1 makes a decision, and simultaneously P2
decides what they would do if P1 chose C and what they would
do if P1 chose D. The strategies of the two players are
combined and the outcome determined.
• Advantages of the strategy method:
• Obtain more data. This is especially important if, for example,
all P1s chose D: in this case we would get no information about
behaviour at P2’s second decision node.
• Can categorise individuals: with direct-response, if we see P2
chose C after P1 chooses C, we don’t know if they are altruistic
or a conditional cooperator.
15/15