Multiple Outcomes and Inference: An Example
from Analysis of Long-Term Effects of a
Scholarship Program
PopPov Methods Workshop
Willa Friedman
Center for Global Development
January 2014
Willa Friedman (CGD)
Multiple Outcomes
January 2014
1 / 15
Introduction
What’s the problem?
With 100 outcomes, on average how many would be significant at the
95% level, by chance?
So what?
Hypothesis testing standards are based on testing a single hypothesis,
not 100 hypotheses.
But sometimes we want to test a lot.
So what do we do?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
2 / 15
Introduction
What’s the problem?
With 100 outcomes, on average how many would be significant at the
95% level, by chance?
So what?
Hypothesis testing standards are based on testing a single hypothesis,
not 100 hypotheses.
But sometimes we want to test a lot.
So what do we do?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
2 / 15
Introduction
What’s the problem?
With 100 outcomes, on average how many would be significant at the
95% level, by chance?
So what?
Hypothesis testing standards are based on testing a single hypothesis,
not 100 hypotheses.
But sometimes we want to test a lot.
So what do we do?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
2 / 15
Introduction
What’s the problem?
With 100 outcomes, on average how many would be significant at the
95% level, by chance?
So what?
Hypothesis testing standards are based on testing a single hypothesis,
not 100 hypotheses.
But sometimes we want to test a lot.
So what do we do?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
2 / 15
Introduction
What’s the problem?
With 100 outcomes, on average how many would be significant at the
95% level, by chance?
So what?
Hypothesis testing standards are based on testing a single hypothesis,
not 100 hypotheses.
But sometimes we want to test a lot.
So what do we do?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
2 / 15
Introduction
Outline
Mean effects
I
Example from Education as liberation? (joint with Michael Kremer,
Edward Miguel, and Rebecca Thornton)
Standard error adjustments
Pre-registration of trials
Willa Friedman (CGD)
Multiple Outcomes
January 2014
3 / 15
Introduction
Outline
Mean effects
I
Example from Education as liberation? (joint with Michael Kremer,
Edward Miguel, and Rebecca Thornton)
Standard error adjustments
Pre-registration of trials
Willa Friedman (CGD)
Multiple Outcomes
January 2014
3 / 15
Introduction
Outline
Mean effects
I
Example from Education as liberation? (joint with Michael Kremer,
Edward Miguel, and Rebecca Thornton)
Standard error adjustments
Pre-registration of trials
Willa Friedman (CGD)
Multiple Outcomes
January 2014
3 / 15
Mean Effects
Mean Effects
Create sets of outcomes, and jointly test impacts on these groups
Straight-forward to implement
Can facilitate interpretation
Reduces concerns about data-mining with lots of outcomes
Willa Friedman (CGD)
Multiple Outcomes
January 2014
4 / 15
Mean Effects
Mean Effects
Create sets of outcomes, and jointly test impacts on these groups
Straight-forward to implement
Can facilitate interpretation
Reduces concerns about data-mining with lots of outcomes
Willa Friedman (CGD)
Multiple Outcomes
January 2014
4 / 15
Mean Effects
Origin of mean effects
Moving-to-Opportunity program
I
I
I
I
Provided housing vouchers to families in poor neighborhoods
Program could change A LOT
Kling, Jeffrey, Jeffrey Liebman, and Lawrence Katz (2007),
“Experimental Analysis of Neighborhood Effects,” Econometrica
For example: Adult mental health
F
distress, depression, anxiety, calmness, sleep
Willa Friedman (CGD)
Multiple Outcomes
January 2014
5 / 15
Mean Effects
Education as Liberation?
(Joint with Michael Kremer, Edward Miguel, and Rebecca Thornton)
Girls Scholarship Program in Western Kenya
Announced scholarship in 34 schools, with 35 control schools
Saw large short-term impacts (Kremer, Michael, Edward Miguel, and
Rebecca Thornton (2009), “Incentives to Learn,” Quarterly Journal of
Economics)
What are the long-run impacts? (What could they be?)
Willa Friedman (CGD)
Multiple Outcomes
January 2014
6 / 15
Mean Effects
Education as Liberation?
(Joint with Michael Kremer, Edward Miguel, and Rebecca Thornton)
Girls Scholarship Program in Western Kenya
Announced scholarship in 34 schools, with 35 control schools
Saw large short-term impacts (Kremer, Michael, Edward Miguel, and
Rebecca Thornton (2009), “Incentives to Learn,” Quarterly Journal of
Economics)
What are the long-run impacts? (What could they be?)
Willa Friedman (CGD)
Multiple Outcomes
January 2014
6 / 15
Mean Effects
Education as Liberation?
(Joint with Michael Kremer, Edward Miguel, and Rebecca Thornton)
Girls Scholarship Program in Western Kenya
Announced scholarship in 34 schools, with 35 control schools
Saw large short-term impacts (Kremer, Michael, Edward Miguel, and
Rebecca Thornton (2009), “Incentives to Learn,” Quarterly Journal of
Economics)
What are the long-run impacts? (What could they be?)
Willa Friedman (CGD)
Multiple Outcomes
January 2014
6 / 15
Mean Effects
Education as Liberation?
This paper focused on political and cultural beliefs
We took 50+ outcomes, grouped into 7 sets.
I
I
I
Some didn’t fit within sets. That’s okay.
e.g.: Democratic attitudes (Agreement with “We should choose our
leaders in this country through regular, open and honest elections.”
Disagreement with “Only one political party should be allowed to stand
for election and hold office,” etc.)
e.g.: Household autonomy (If married, family involved in spouse
choice; Disagree with “Men can beat their wives and children if they
misbehave,” etc.)
Test impacts of program on these groups
Willa Friedman (CGD)
Multiple Outcomes
January 2014
7 / 15
Mean Effects
Technique
1
Identify sets
2
Make sure predicted changes all go in the same direction (some
flipping needed)
3
Normalize all outcome variables (mean=0, SD=1)
I
(individual value - mean(all values in control group))/ sd(all values in
control group)
4
Average all outcomes within sets
5
Normalize this mean
6
Use this as the outcome (just as would use single outcome)
Willa Friedman (CGD)
Multiple Outcomes
January 2014
8 / 15
Mean Effects
Technique
1
Identify sets
2
Make sure predicted changes all go in the same direction (some
flipping needed)
3
Normalize all outcome variables (mean=0, SD=1)
I
(individual value - mean(all values in control group))/ sd(all values in
control group)
4
Average all outcomes within sets
5
Normalize this mean
6
Use this as the outcome (just as would use single outcome)
Willa Friedman (CGD)
Multiple Outcomes
January 2014
8 / 15
Mean Effects
Technique - Example
Lack of Autonomy in Household
I
I
I
I
I
I
Agree with “Women have always been subject to traditional laws and
customs and should remain so.” vs. “Women should have equal rights
and receive the same treatment as men do.”
“Men can beat their wives and children if they misbehave.” vs. “No
one has the right to use physical violence against anyone else.”
Ever married.
Ever married, with family involved in spouse choice.
Ever married, without family involvement in spouse choice.
Total fertility.
Notice that all go in same direction.
Willa Friedman (CGD)
Multiple Outcomes
January 2014
9 / 15
Mean Effects
Technique - Example
Normalize each one
I
I
sum var if treat==0
replace var=(var-r(mean))/r(sd)
Take the average
I
egen hhmean=rmean(var1 var2 var3 ...)
Normalize this
I
I
sum hhmean if treat==0
replace hhmean=(var-r(mean))/r(sd)
Use as outcome
I
I
reg hhmean treat ...
etc.
Willa Friedman (CGD)
Multiple Outcomes
January 2014
10 / 15
Mean Effects
Technique - Example
Normalize each one
I
I
sum var if treat==0
replace var=(var-r(mean))/r(sd)
Take the average
I
egen hhmean=rmean(var1 var2 var3 ...)
Normalize this
I
I
sum hhmean if treat==0
replace hhmean=(var-r(mean))/r(sd)
Use as outcome
I
I
reg hhmean treat ...
etc.
Willa Friedman (CGD)
Multiple Outcomes
January 2014
10 / 15
Mean Effects
Willa Friedman (CGD)
Multiple Outcomes
January 2014
11 / 15
Mean Effects
Remaining concerns
Not always logical categories
Sometimes we wouldn’t expect to see impacts on all elements of a
group of related outcomes.
I
I
I
wages and hours?
learning in different subjects?
others?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
12 / 15
Mean Effects
Remaining concerns
Not always logical categories
Sometimes we wouldn’t expect to see impacts on all elements of a
group of related outcomes.
I
I
I
wages and hours?
learning in different subjects?
others?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
12 / 15
Mean Effects
Remaining concerns
Not always logical categories
Sometimes we wouldn’t expect to see impacts on all elements of a
group of related outcomes.
I
I
I
wages and hours?
learning in different subjects?
others?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
12 / 15
Mean Effects
Remaining concerns
Not always logical categories
Sometimes we wouldn’t expect to see impacts on all elements of a
group of related outcomes.
I
I
I
wages and hours?
learning in different subjects?
others?
Willa Friedman (CGD)
Multiple Outcomes
January 2014
12 / 15
Other methods
Standard-error adjustments
One issue with multiple-hypothesis testing is that the SEs might be
wrong
Solution: adjust the SEs
I
Familywise error rate
F
F
F
I
not that kind of family
Bonferroni Correction
Holm’s method
False discovery rate
F
Benjamini and Hochberg
Willa Friedman (CGD)
Multiple Outcomes
January 2014
13 / 15
Other methods
Standard-error adjustments
One issue with multiple-hypothesis testing is that the SEs might be
wrong
Solution: adjust the SEs
I
Familywise error rate
F
F
F
I
not that kind of family
Bonferroni Correction
Holm’s method
False discovery rate
F
Benjamini and Hochberg
Willa Friedman (CGD)
Multiple Outcomes
January 2014
13 / 15
Other methods
Standard-error adjustments
One issue with multiple-hypothesis testing is that the SEs might be
wrong
Solution: adjust the SEs
I
Familywise error rate
F
F
F
I
not that kind of family
Bonferroni Correction
Holm’s method
False discovery rate
F
Benjamini and Hochberg
Willa Friedman (CGD)
Multiple Outcomes
January 2014
13 / 15
Other methods
Standard-error adjustments
One issue with multiple-hypothesis testing is that the SEs might be
wrong
Solution: adjust the SEs
I
Familywise error rate
F
F
F
I
not that kind of family
Bonferroni Correction
Holm’s method
False discovery rate
F
Benjamini and Hochberg
Willa Friedman (CGD)
Multiple Outcomes
January 2014
13 / 15
Other methods
Pre-registration of trials
State in advance what will be tested
Binds the researchers hands
Standard/required in medicine, public health
Growing in economics, political science
Range of options
I
I
Detailed pre-analysis plan with simulations
Basic trial registration
F
F
I
AEA (American Economics Association) registry:
www.socialscienceregistry.org/
EGAP (Experiments in Governance and Politics):
e-gap.org/design-registration/standards-project-registration/
Clinical registries:
F
F
F
US NIH: clinicaltrials.gov
WHO: apps.who.int/trialsearch/
Pan African Clinical Trial Registry: www.pactr.org/
Willa Friedman (CGD)
Multiple Outcomes
January 2014
14 / 15
Other methods
Pre-registration of trials
State in advance what will be tested
Binds the researchers hands
Standard/required in medicine, public health
Growing in economics, political science
Range of options
I
I
Detailed pre-analysis plan with simulations
Basic trial registration
F
F
I
AEA (American Economics Association) registry:
www.socialscienceregistry.org/
EGAP (Experiments in Governance and Politics):
e-gap.org/design-registration/standards-project-registration/
Clinical registries:
F
F
F
US NIH: clinicaltrials.gov
WHO: apps.who.int/trialsearch/
Pan African Clinical Trial Registry: www.pactr.org/
Willa Friedman (CGD)
Multiple Outcomes
January 2014
14 / 15
Other methods
Pre-registration of trials
State in advance what will be tested
Binds the researchers hands
Standard/required in medicine, public health
Growing in economics, political science
Range of options
I
I
Detailed pre-analysis plan with simulations
Basic trial registration
F
F
I
AEA (American Economics Association) registry:
www.socialscienceregistry.org/
EGAP (Experiments in Governance and Politics):
e-gap.org/design-registration/standards-project-registration/
Clinical registries:
F
F
F
US NIH: clinicaltrials.gov
WHO: apps.who.int/trialsearch/
Pan African Clinical Trial Registry: www.pactr.org/
Willa Friedman (CGD)
Multiple Outcomes
January 2014
14 / 15
Other methods
Pre-registration of trials
State in advance what will be tested
Binds the researchers hands
Standard/required in medicine, public health
Growing in economics, political science
Range of options
I
I
Detailed pre-analysis plan with simulations
Basic trial registration
F
F
I
AEA (American Economics Association) registry:
www.socialscienceregistry.org/
EGAP (Experiments in Governance and Politics):
e-gap.org/design-registration/standards-project-registration/
Clinical registries:
F
F
F
US NIH: clinicaltrials.gov
WHO: apps.who.int/trialsearch/
Pan African Clinical Trial Registry: www.pactr.org/
Willa Friedman (CGD)
Multiple Outcomes
January 2014
14 / 15
Other methods
Pre-registration of trials
eg: Casey, Glennester, and Miguel, 2012
side benefit: pre-specification could improve quality of
implementation/data collection
Time consuming
Not required, but increasingly expected
Could also use for analysis without RCT if don’t have data yet
Willa Friedman (CGD)
Multiple Outcomes
January 2014
15 / 15
Other methods
Pre-registration of trials
eg: Casey, Glennester, and Miguel, 2012
side benefit: pre-specification could improve quality of
implementation/data collection
Time consuming
Not required, but increasingly expected
Could also use for analysis without RCT if don’t have data yet
Willa Friedman (CGD)
Multiple Outcomes
January 2014
15 / 15
Other methods
Pre-registration of trials
eg: Casey, Glennester, and Miguel, 2012
side benefit: pre-specification could improve quality of
implementation/data collection
Time consuming
Not required, but increasingly expected
Could also use for analysis without RCT if don’t have data yet
Willa Friedman (CGD)
Multiple Outcomes
January 2014
15 / 15
Other methods
Pre-registration of trials
eg: Casey, Glennester, and Miguel, 2012
side benefit: pre-specification could improve quality of
implementation/data collection
Time consuming
Not required, but increasingly expected
Could also use for analysis without RCT if don’t have data yet
Willa Friedman (CGD)
Multiple Outcomes
January 2014
15 / 15
Other methods
Pre-registration of trials
eg: Casey, Glennester, and Miguel, 2012
side benefit: pre-specification could improve quality of
implementation/data collection
Time consuming
Not required, but increasingly expected
Could also use for analysis without RCT if don’t have data yet
Willa Friedman (CGD)
Multiple Outcomes
January 2014
15 / 15