3/13/2017
Confidence Intervals
in Research and Daily Life
Confidence Intervals in Bar Graphs:
Error Bars
• Alexander L. N. van Nuijs et al. 2009. “Can cocaine use
be evaluated through analysis of wastewater? A nationwide approach conducted in Belgium.” Addiction, 104,
734–741.
1
3/13/2017
Types of Error Bars
• Types of error bars:
– Confidence interval (90%, 95%, 99%)
– Standard error (or standard error times a
multiplier, typically a z-score of 2)
– Standard deviation (or standard deviation times a
multiplier)
• Recommended: confidence interval error bars
• When examining graphs with error bars, check
the notes for the type of error bars used
• Ben Anderson & Karina Tracey. 2001. “Digital Living:
The Impact (or Otherwise) of the Internet on Everyday
Life.” American Behavioral Scientist 45(3):456-475.
2
3/13/2017
The original error bars
display standard errors – I
converted them to
confidence intervals (in red;
twice the size of the SE
ones)
• Ben Anderson & Karina Tracey. 2001. “Digital Living:
The Impact (or Otherwise) of the Internet on Everyday
Life.” American Behavioral Scientist 45(3):456-475.
The two confidence intervals
overlap –> we cannot conclude
that the two groups (16-24 and
55+) are different; further
testing needed.
• Ben Anderson & Karina Tracey. 2001. “Digital Living:
The Impact (or Otherwise) of the Internet on Everyday
Life.” American Behavioral Scientist 45(3):456-475.
3
3/13/2017
Conclusions Based on Error Bars
• If error bars are confidence intervals:
– Error bars do not overlap can say that groups are
different with at least that level of confidence
– Error bars overlap cannot say that the groups are
different based on these error bars; need further tests
• If error bars are standard errors:
– Multiply error bar height by 2
confidence intervals
interpret like 95%
• Warning: error bars are not helpful for comparing
the results of pre-test and post-test of the same
people (because retests are highly correlated)
• Source: Salminen T., Strobach T. & Schubert T. 2012. “On
the impacts of working memory training on executive
functioning.” Frontiers in Human Neuroscience 6:166.
4
3/13/2017
• Source: E. Juulia Paavonen, Marjo Pennonen, Mira Roine,
Satu Valkonen, & Anja Riitta Lahikainenj. "TV Exposure
Associated With Sleep Disturbances in 5- To 6-Year Old
Children." 2006. Sleep Research 15, 154–161
More on Error Bars
• Error bar – a way to graphically display either a
confidence interval or a standard error (can’t
distinguish visually)
• Standard error – measure of variation of the sampling
distribution; for the mean, SEM: σX̅ = s/sqrt(n)
• Confidence interval – an interval estimate of the mean.
95% confidence interval = 4 SE wide (X ± 1.96*σX̅)
• The top of an error bar shows the upper limit of the
confidence interval and its bottom – the lower limit
• If an error bar displays the standard error, it can be
viewed as a 68% confidence interval = 2 SE wide
5
3/13/2017
What Makes Error Bars Longer?
• 99% confidence interval > 95% CI > 90% CI >
standard error
• Smaller sample size wider CI
• Less homogenous population wider CI
• Source: Lori Beaman, Esther Duflo, Rohini Pande, & Petia Topalova.
2012. "Female Leadership Raises Aspirations and Educational
Attainment for Girls: A Policy Experiment in India." Science 335: 582586.
6
3/13/2017
• Source:
https://familyinequality.wordpress.com/tag/employment/
• Source:
https://familyinequality.wordpress.com/tag/employment/
7
3/13/2017
Confidence Interval vs Margin of Error
• Margin of error = the "radius" (or half the
width) of a confidence interval
8
3/13/2017
Public Opinion Polls & Margin of Error
• https://www.youtube.com/watch?v=DAufq8y20d4
Confidence Intervals for Proportions
• Same as for mean but do not need a separate
estimate of standard deviation
Example. For a sample size n = 900 and an estimated
sample proportion = 0.2:
95% CI = 0.2 ± 1.96*sqrt(0.2*0.8/900) =
= 0.2 ± 0.026
Probability(.174 ≤ p ≤ 0.226)=.95
Margin of error = 0.026*100 = 2.6%
9
3/13/2017
Margin of Error in Polls
• Media usually report the maximum margin of
error for any percentage from that poll
• The margin of error for a particular percentage
usually smaller
• Maximum margin of error: for percentage=50%
(which is proportion =.5); based on sample size:
Margin of error and 95% confidence
intervals for an estimate of 50%
10
3/13/2017
Example
• Sample size n = 400
• Margin of error = 1/√n = 1/sqrt(400)=1/20=0.05
• Answer: margin of error = ± 5%
• Online calculator:
http://americanresearchgroup.com/moe.html
• But: margin of error applies only when we
consider the whole sample, not particular
subgroups (e.g., women, Republicans, etc.)
2016 Presidential Election
• https://en.wikipedia.org/wiki/Nationwide_opinio
n_polling_for_the_United_States_presidential_el
ection,_2016
11
3/13/2017
• http://fivethirtyeight.com/features/electionupdate-yes-donald-trump-has-a-path-tovictory/
• https://fivethirtyeight.com/features/the-oddsof-an-electoral-college-popular-vote-split-areincreasing/
https://fivethirtyeight.com/features/therereally-was-a-liberal-media-bubble/
Nate Silver: “Trump outperformed his national
polls by only 1 to 2 percentage points in losing
the popular vote to Clinton, making them
slightly closer to the mark than they were
in 2012. Meanwhile, he beat his polls by only 2
to 3 percentage points in the average swing
state. Certainly, there were individual pollsters
that had some explaining to do, especially in
Michigan, Wisconsin and Pennsylvania, where
Trump beat his polls by a larger amount. But the
result was not some sort of massive outlier; on
the contrary, the polls were pretty much as
accurate as they’d been, on average, since
1968.”
12
3/13/2017
Beware: Margin of Error vs Bias
Sources of Potential Bias
• Sample is not a random sample of all voters
– E.g., internet polls rely on “opt-in” panels
– “Robocall” polls can’t use cellphones
– One landline vs multiple cellphones per household
– Difficult to predict who will actually vote
• High levels of non-response
• Biased question wording
13
3/13/2017
14
3/13/2017
How Do Polls Deal with Nonresponse?
• Polls can (and do) correct for nonresponse bias by
using demographic weights
• Compare demographics of the poll to known
demographics of population
• E.g., if more educated are overrepresented
count the opinions of less educated twice
• Beyond that, we cannot know if there is still bias
in a poll; that is why only the margin of error can
be reported
• Controversy: should internet polls report margin
of error? They are not based on random samples
3/13/2017
15