The Social Science Registry:
Philosophy, Implementations, Results
Esther Duflo, Massachusetts Institute of Technology
Abdul Latif Jameel Poverty Action Lab
March 2, 2017
https://www.socialscienceregistry.org/
A large registry in social science research!
Potential use of a registry
1) Make sure that all the studies that are
undertaken can be tracked to undertake
comprehensive meta-analysis
2) Tie author’s hands in terms of design,
outcomes, reporting
3) Give a good idea of the landscape to guide
future research/reporting
There is potentially some tensions between these
objectives
•
(1) and (3) requires something akin to a census of relevant studies:
–
Define population clearly
–
Make registration easy.
•
(2) requires more commitment from authors to report a particular design: this
may make registration more rare
•
The AEA registry’s philosophy has been to ensure something close to a census:
–
Restrict to RCTs
–
Make registration easy and give authors flexibility to do more.
–
Tie to AEA who controls the leading journals in economics, and represents the
profession
–
Make a big push to register past studies
–
Provide support to people/organizations interested in registering (e.g. MDRC)
Objectives of the AEA RCT Registry
1) Provide a searchable database of planned,
on-going, abandoned, and completed RCTs.
2) Link to data and research resulting from
registered studies.
3) Provide information on study designs in a
manner easily usable in meta-analysis.
Who it is for?
Eligibility:
• Randomized controlled trials (RCTs) in the social
sciences
• The RCT can be run anywhere in the world.
Consumers:
• Researchers in economics and other social sciences;
• Policymakers; and
• General public good.
Creating a culture: Institutional support
•
The AEA recommends registration for all papers submitted to the AEA
journals.
•
J-PAL and IPA require pre-registration for any study that has been
implemented or funded by us.
•
NBER requires pre-registration at the time of IRB review
•
Interest from other organizations: Canadian Economic Association
(CAE), Econometric society, International Growth Centre (IGC), MDRC,
DIME at the World Bank, etc.
Easy / Fast Sign Up
socialscienceregistry.org/users/sign_up
Simple Registration
Mandatory Fields Are Highlighted in RED Tabs
All Mandatory Fields
Trial Title
Country
Status of Project
Keywords
Abstract
Trial Start Date
Intervention Start
Date
• Intervention End Date
•
•
•
•
•
•
•
Trial End Date
• Sample size: planned
Outcomes
total number of
Experimental Design
observations
Randomization
• Sample size (or
Method
number of clusters)
• Randomization Unit
by treatment arms
• Treatment clustering • IRB approval status*
• Sample Size: Planned
Number of Clusters
(unit of
randomization)
•
•
•
•
* For new and ongoing projects only.
When to register?
We encourage registration as early as possible, before data
collection begins (though you may still register afterwards).
• Registrations may be edited later.
• A history will be kept of all changes/versions post-submission into
the registry.
Registration and Preregistration Icons
Preregistered Study
Non Preregistered
Study
Quality control: Review criteria
•
All draft trials are
reviewed prior to
publication of the
registry entry
•
Review completed
within 15 days
•
Criteria focuses on:
–
Registry applicability
–
Duplicate
registrations
–
Language
consistency
Citing a Registration Entry
•
•
Upon registration, an auto-generated citation will appear at the
bottom of your publicly registered trial.
Registrations can be cited as: AEA RCT Registry, RCT ID: AEARCTR000000.
–
Example: “This study is registered in the AEA RCT Registry as AEARCTR --000000.”
Advanced Search and Downloading Metadata
CSV download of data
A new
milestone
reached:
1,000+ studies
from over
100+
countries!
Total Cumulative Registrations on AEA
1200
1100
1000
900
800
700
600
500
400
300
200
100
0
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
2015Q2
2015Q3
2015Q4
2016Q1
2016Q2
2016Q3
2016Q4
2017Q1
Data from May 2013 - Feb. 2017
Individual Registrations per Quarter
160
140
120
100
80
60
40
20
0
2013Q2
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
2015Q2
2015Q3
2015Q4
2016Q1
2016Q2
2016Q3
2016Q4
Data from May 2013 – Dec. 2016
Individual Preregistrations per Quarter
45
40
35
30
25
20
15
10
5
0
2013Q2
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
2015Q2
2015Q3
2015Q4
2016Q1
2016Q2
2016Q3
2016Q4
Data from May 2013 – Dec. 2016
Total Registrations and Preregistrations
Per Quarter
All Time
140
120
100
24%
80
Registered
60
40
20
0
Preregistered
76%
Registered Analysis Plans (Public/Private) per Quarter
25
20
15
10
5
0
2013Q2
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
Private
2015Q2
Public
2015Q3
2015Q4
2016Q1
2016Q2
2016Q3
2016Q4
Data from May 2013 – Dec. 2016
Number of Studies Reporting Post Trial Info (by field)
Data from May 2013 – Nov. 2016
Conclusions: next steps
1) In very little time, the AEA registry has established itself as the go-toplace for RCTs in economics and more generally in social sciences
2) Some success in terms of tying up with other initiatives (e.g. OSF): we
would like to do more.
3) The registry is NOT recognized by medical journals which is a
problem (double registration is necessary for researchers aiming to
publish in medical journals).
4) Increasing acceptance and visibility in the profession: moving
toward requirement in journals may be on the horizon.
5) Need to keep an eye on quality of post registration follow up to
avoid the problem that has faced other registries (e.g.
clinicaltrials.gov)