Why Policymakers Should Use Field Experiments
to Solve the World’s Problems
John A. List
U. Chicago & NBER
A. Three Fundamental Empirical Challenges Policymakers Face
B. How Field Experiments Can Help
*Human Capital Formation
*Tax Compliance
C. Closing Remarks
3 Major Challenges for Policymakers
1.
Evaluating the effectiveness of an implemented policy
Did the 2012 Australian Tobacco Plain Packaging / graphic health
warning legislation curb smoking?
The Policy
Evidence: Prevalence
4
3 Major Challenges of Policymakers
2.
Projecting the likely effectiveness of a policy in
environments different from the one in which it is
experienced
Would Tobacco Plain Packaging legislation work in Europe?
3.
Forecasting the effects of a new policy, never before seen
Would behavioral economic interventions, such as leveraging loss
aversion, curb smoking?
Knightian Wisdom

Knight (1921): ‘The existence of a problem in knowledge
depends on the future being different from the past, while
the possibility of a solution of the problem depends on the
future being like the past.’
In many policy circles…..
Rather than admit knowledge deficits…..
policymakers take a “Mark Twain” approach

“All you need in this life is ignorance and
confidence; then success is sure.” ~ Mark Twain

We have been crafting public policies for
centuries, yet we still know embarrassingly
little about the fundamental structure and
makeup of good policies in many areas

The fact that we know so little about
optimal policies in key areas is troubling
A Path Forward

We must understand that each policy
intervention is not just about helping
today’s citizens, but teaching ourselves
what works and why

In this way, we should explicitly recognize
the import of building knowledge for future
generations when designing policy roll out
How Can Field Experiments Help
Close this Knowledge Gap?
Field Experiment Example 1

How can we efficiently increase human
capital formation, or how can we improve
public education?
Conventional Wisdom

We need teachers with higher degrees, we need
lower student/teacher ratios, and we need to
spend more to make public education work
Teachers are receiving more
years of education
Percentage of Teachers with a Master's Degree or Higher
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
23,5%
27,5%
1961
1971
49,6%
53,1%
56,8%
61,8%
1981
1991
2001
2006
0%
13
Student to Teacher Ratios Have Dropped
Student to Teacher Ratio
24:1
22,3 :1
22:1
20,4 :1
20:1
18,7 :1
17,9 :1
18:1
17,2 :1
17,3 :1
16:1
14:1
1970
14
1975
1980
1985
1990
1995
16,0 :1
15,6 :1
2000
2005
Spending Has Risen
$12.000
$10.508
$10.000
$8.000
$6.000
$12,116
Total Expenditure Per Pupil (2008-09 $)
$5.243
$6.049
$6.268
$8.790
$8.949
1990
1995
$11.438
$7.347
$4.000
$2.000
$0
1970
15
1975
1980
1985
2000
2005
Results Have Not Materialized
100%
75%
HS Graduates as a ratio of 17 year-olds
76,9%
73,6%
71,4%
72,4%
1975
1980
1985
73,4%
69,3%
69,8%
75,4%
50%
25%
0%
1970
16
1990
1995
2000
2005
74,7%
Results Have Not Materialized
Reading and Math Achievement of 9, 13, and 17 year-olds
325
300
9 year-olds
275
13 year-olds
250
17 year-olds
225
200
175
1971 1975 1980 1984 1988 1990 1994 1996 1999 2004 2008
17
Our Work in Education
We
have conducted field experiments to
explore the education production function
from pre-k through high school.
We
have looked at teachers, mentors,
families, communities, friend networks,
administrators, etc.
We
18
have started 2 pre-k schools
An Example Within Schools
Teacher performance pay…has
become a big hot button issue in
parts of the world.
19
U.S. Press: Merit Pay Doesn’t Work
 “Teacher
Performance Pay Alone Does Not Raise
Student Test Scores” – Vanderbilt News, September
2010
 “The
long, failed history of merit pay and how the Ed
Department ignores it” – Washington Post, September
2010
 “Why
Teacher Performance Pay Won't Work” –
Huffington Post, November 2012
Press: Merit Pay Doesn’t Work
 “$75M
teacher pay initiative did not improve
achievement” – Chalkbeat.org, March 2012
 “An
education reform without merit: Performance pay
doesn't work, so let's stop wasting money” New York
Daily News, January 2014
 “John,
teachers will never respond to merit pay
incentives—they value the kids not the money!” CPS
Teacher’s Union, dozens of times
…But, teachers do like
money!!
“Teacher's
Union Endorses Bonus Plan
Not Based on Performance“ – ABC News,
July 2014
Our Field Experiment
 Teacher incentives:
 If
their students improved standardized test performance
teachers could earn as much as $8,000.
 The
expected value of the reward was $4,000—an 8%
increase in their annual salary.
23
©John List, University of
Chicago
Field Experiment
 Framing of rewards
Gains: The reward was distributed at the end of the
school year as a bonus.
Loss: Teachers received $4,000 at the start of the
year and had to write a check to us if they earned less
than $4,000.
©John List, University of
Chicago
Field Experiment
All Teachers
Randomization
Group 1: Gain,
Traditional bonus
Group 3: Control, paid
nothing
Group 2: Loss, paid
$4000 upfront
Experimental Results: Gain Works A Little
Standard Deviations
0,35
0,3
0,25
0,2
0,15
0,1
0,05
0
Gain
Loss
Racial Achievement Gap
26
©John List, University of
Chicago
Experimental Results:
Clawback Works Well
Standard Deviations
0,35
0,3
0,25
0,2
0,15
0,1
0,05
0
Gain
Loss
Racial Achievement Gap
27
©John List, University of
Chicago
Example II:
A Tax Nudge

UK tax commission contacted us some
years ago about ‘problem’ tax payers.

Developed a partnership with UK
Behavioral Insights Team
Our First Field Experiment

UK taxpayers who had not made the correct payment by July
31, 2011, and who had not responded to the initial reminder
statement

All taxpayers had a debt of between £400 and £100,000 on
August 1, 2011


Debts below £400 and above £100,000 are subject to different
actions and hence could not be included in the sample
Sample is 101,471 individuals geographically distributed
across England, Wales and Northern Ireland
Control
Descriptive norms

Importance of psychological distance
(Cialdini, 2003; Goldstein et al., 2008; Trope
& Liberman, 2010)

The relevance of the ‘Country’ name

Basic: “9 out of 10 people pay their tax on time.”

Country: “9 out of 10 people in the UK pay their
tax on time.”

Minority: “9 out of 10 people in the UK pay their
tax on time. You are in the small minority of
people who have not yet paid their tax on time.”
(Wenzel, 2005)
Two letters, one difference
We Also Experimented with Gain/Loss
Concerns of Public Goods

Gain: “Paying tax means we all gain from vital
public services like the NHS, roads, and
schools.”

Loss: “Not paying tax means we all lose out
on vital public services like the NHS, roads,
and schools.”
What we have learned

Treatments matter
 Appealing
to social norms and losses
increases the likelihood of payment
 Smaller perceptual distance between the
norm message and its recipient produces
higher tax compliance

Pricing the effects:
 Brought
in an additional $20m+ through FE
This Lead to a New Field Experiment

FE with small businesses explores norms and
statements that non-payment is an act of omission
or commission

“Previously, we treated your lack of response as
an oversight. Now, if you do not call [telephone
number], we will treat this as an active choice.”

Acts of commission treatment turns the nudge into
a shove!
NBER 2015: The Making of Homo Honoratus: From Omission to
Commission
Policy implications

Extremely cheap policy can generate large
benefit
 Field

experiments recouped millions
Cost was researchers time
 But
negligible in future as Governments can
now use this information
Anatomy of a Partnership

2 Very Different Models of Partnerships
A. Revolutionizing existing approaches
B. Refining existing approaches
We Have Only Touched the Tip of the Iceberg
Why do women earn less than men?

Early in the job search there are key differences:
 women
shy away from merit pay incentive schemes
 in
ambiguous bargaining situations women exercise caution
whereas men (especially the low skilled types) ask for
higher wages
 data
thus far are consistent with nurture playing an
important role
Discrimination
 Why
do people discriminate?
(hatred) becoming less important, 3rd
degree price discrimination becoming more
important over time
Animus
Means
that today people are discriminating less
out of hatred and more out of profit motives
How can we enhance the
charitable pie?

Many people give primarily because of
selfish motives rather than altruism

We can increase the pool of donors and
donations if we understand why people
give
Great Big Externalities

One of the biggest problems humanity
faces is climate change

How can we induce households to adopt
green technologies?
 Social
norms work for the very first adoption,
prices for deeper adoptions
Closing Remarks

To provide insights into the 3 fundamental
challenges field experiments can help
We must go beyond correlations to causality
AND,
understand the channels of that causation
After we accomplish these we can craft
efficient public policies
In Closing: A Knightian Revision

Knight (1921): ‘The existence of a problem in knowledge
depends on the future being different from the past,
while the possibility of a solution of the problem depends
on the future being like the past.’

A Revision (today): ‘The existence of a problem in
knowledge depends on the future being different from
the past, while the possibility of a solution of the problem
depends on the future being like a field experiment.’
An Example
Using a Field Experiment to
Understand Discrimination
Why do people receive different price quotes
for the same good?
Economists have two major theories
Discrimination NFE
Study seller side discrimination following List (2004; QJE):
12 disabled and 12 non-disabled testers approached
various body shops in Chicago with different cars
(identical cars across disabled and abled) that were
in need of repair
Offer differences: disabled receive prices 30% higher than
the non-disabled receive
Complementary Evidence
Survey
 Disabled search less
 Body shop mechanics believe there are search
differences.
New Treatments
 Re-send different pairs to receive price quotes
 One treatment replicates above treatment
 Another treatment is identical except that it has both
agent types explicitly noting that “I’m getting a few price
quotes today”
Replication Treatment
New Treatment Results
700
Dollars
600
500
400
300
200
100
0
Disabled
Non-Disabled
Disabled "Few Non-Disabled
Quotes"
"Few Quotes"
Group Type
“Few Quote” Treatment
New Treatment Results
700
Dollars
600
500
400
300
200
100
0
Disabled
Non-Disabled
Disabled "Few
Quotes"
Group Type
Non-Disabled
"Few Quotes"
A “Bottom Line” Example

From “shoves” (starting your own school)
to “nudges”

2 Very Different Models of Partnerships
A. Revolutionizing existing approaches
B. Refining existing approaches