PSYC 5210
Meetings:
Instructor:
Office hours:
Email:
Text:
Twitter:
Statistical Methods
Fall 2016
MW 1:40-3:00 Rogers-Stout 425 / 422
Matthew T. McBee, PhD
T 8:15-9:15am and by appointment
[email protected]
(919) 943-7204
@TunnelOfFire
Catalog description
This course includes an overview of inferential statistics including topics such as probability,
hypotheses testing, population sampling, and analysis of regression and prediction. Both
parametric and nonparametric tests are reviewed. Parametric tests include the Z-test, t-test,
Sandler A, Analysis of Variance, Analysis of Covariance, and the Newman-Keuls test.
Nonparametric tests include the Chi square test, Sign test, Wilcoxon test, Mann-Whitney test,
Kruskal-Wallis test, and the Friedman test. Students learn the purpose of these tests and their
strengths and limitations. (fall)
Additional description
This class will begin to prepare you to analyze data from psychological research and to critique
the methodology and analyses of published studies. We will strike a balance between statistical
theory and applied statistics, emphasizing the theoretical justification for decisions made during
the design and analysis of research.
This course focuses on the frequentist statistics that are the dominant paradigm in psychology,
but some Bayesian ideas and methods will be introduced, including using tools such as JASP,
JAGS, and the BayesFactor library to calculate posterior distributions, credible intervals, and
Bayes factors.
The course will, largely through readings, introduce students to what is being called the
‘reproducibility crisis’ in psychology. These readings describe problematic aspects of the
research culture and of scientific and statistical practices in psychology and the social/behavioral
sciences, and the resulting crisis of confidence that has consumed the field. Replication studies,
previously rare, are becoming much more widespread, and the results are showing that many
supposedly-established phenomena apparently do not exist or are much weaker than previously
imagined. Many people in the field (including me!) believe that fundamental changes to the way
we do science are required in order for our field to produce the reliable and cumulative evidence
that enables scientific progress.
Software. The R statistical software will be used throughout this class. R has the tremendous
advantage of being free. It is quickly becoming the dominant package for data analysis in the
social sciences. It will be to your tremendous advantage to transition to R as your primary data
1
analysis package as soon as possible. SPSS is quickly becoming an anachronism in the academic
world, as the following figure demonstrates. Using R will develop your skills, sharpen your
intuition, and make you a better scientist. The JAGS (‘Just Another Gibbs Sampler’) package for
Bayesian inference interfaces with R, as does the BayesFactor library.
By contrast, JASP is a free, menu-driven, point-and-click tool for calculating Bayes factors for a
number of common models, including t-tests, ANOVAs, and linear regression models.
Figure 1: Trends in stats
package use. Data from Google
Scholar. Image by B.
Muenchen.
Please install R and RStudio on
your personal computer and
bring it to class if possible. R
can be freely downloaded here:
https://www.r-project.org.
RStudio is a nice front-end to R
that adds organization and
features that facilitate writing R
code. It is freely downloadable
from https://www.rstudio.com.
Versions are available for
Windows, OSX, and Linux PCs. You will also need to download and install JASP. JASP can be
downloaded for free from https://jasp-stats.org.
The major topics to be covered are 1) basic probability theory, random variables, and
distributions, 2) the logic of Neyman-Pearson hypothesis tests including decision errors, effect
size, power, and confidence intervals, 3) t-tests and their variants, 4) oneway ANOVA 5)
factorial ANOVA, interactions, and simple effects, 6) repeated measures ANOVA, 7) mixed
ANOVA, 8) analysis of covariance, and 9) Bayesian versions of many of these tests.
Course Learning Outcomes
Class participants will be able to critically think about, and be able to apply in research, reading,
discussion, or writing their understanding of:








Basic statistical concepts such as random variables, distributions, sampling distributions,
null and alternative hypotheses, effect size, and power.
The t-test for comparing two groups of subjects.
The analysis of variance for comparing three or more groups of subjects, including its
factorial, repeated measures, and mixed variants.
The analysis of covariance model.
Post-hoc tests and corrections for multiple comparisons.
Bayes theorem.
Prior distribution
Posterior distribution
2







Frequentist confidence intervals and Bayesian credible intervals
Strengths and weaknesses of p-values and Bayes factors for assessing evidence.
The nature and causes of the reproducibility crisis.
Questionable research practices (QRPs), including optimal stopping and HARKing.
The Open Science movement, including preregistration, open data, open materials, and
registered reports.
ManyLabs collaboration.
Forensic tools, including the p-curve, for identifying literature likely distorted by QRPs.
Readings
Required textbooks:
Dienes, Z. (2008). Understanding psychology as a science: An introduction to scientific and
statistical inference. NY: Palgrave Macmillan.
Required papers:
Begley, C. G. & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer
research. Nature, 483(7391). 531-533. doi: 10.1038/483531a
Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive
influences on cognition and affect. Journal of Personality and Social Psychology, 100(3),
407-425. doi: http://dx.doi.org/10.1037/a0021524
Cumming, G. (2013). The new statistics: Why and how. Psychological Science, 25(1), 7-29. doi:
10.1177/095679761350496
Francis, G., Tanzman, J., & Matthews, W. J. (2014) Excess success for psychology articles in the
journal Science. PLoS ONE, 9(12): e114255. doi:10.1371/journal.pone.0114255
Haggar, M. & Chatzisarantis, N. L. D. (2016). A multilab preregistered replication of the ego
depletion effect. Perspective on Psychological Science, 11(4), 546-573. doi:
10.1177/1745691616652873
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med, 2(8):
e124. doi: 10.1371/journal.pmed.0020124
Jarosz, A. F. & Wiley, J. (2014). What are the odds? A practical guide to computing and
reporting Bayes factors. Journal of Problem Solving, 7, 2-9. doi:
http://dx.doi.org/10.7771/1932-6246.1167
John, L. K., Loewenstein, & Prelec, D. (2012). Measuring the prevalence of questionable
research practices with incentives for truth telling. Psychological Science, 23(5), 524532. doi: 10.1177/0956797611430953
Kidwell, M. C., Lazarevic, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, LS., Kennett, C., Slowik, A., Sonnleitner, C., Hess-Holden, C., Errington, T. M., Fiedler,
3
S., & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost,
effective method for increasing transparency. PLoS Biology, 14(5), e1002456.
doi:10.1371/journal.pbio.1002456
Lakens, D. (2014). Professors are not elderly: Evaluating the evidential value of two social
priming effects through p-curve analyses. [preprint]. Social Sciences Research Network,
doi: http://dx.doi.org/10.2139/ssrn.2381936
Ledgerwood, A., Soderberg, C., & Sparks, J. (2016). Designing a study to maximize
informational value. In M. C. Makel & J. A. Plucker (eds.), Doing Good Social Science:
Trust, Accuracy, Transparency. Washington, DC: APA.
Lurquin, J. H., Michaelson, L. E., Barker, J. E., Gustavson, D. E., von Bastian, C. C., Carruth, N.
P., et al. (2016) No evidence of the ego-depletion effect across task characteristics and
individual differences: A pre-registered study. PLoS ONE 11(2): e0147770. doi:1
0.1371/journal.pone.0147770
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How
often do they actually occur? Perspectives on Psychological Science, 7(6), 537-542. doi:
10.1177/1745691612460688
Masson, M. E. J. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis
significance testing. Behavior Research Methods, 43(3), 679-690. doi: 10.3758/s13428010-0049-5
McBee. M. T. & Field, S. H. (2016). Confirmatory study design, data analysis, and results that
matter. In M. C. Makel & J. A. Plucker (eds.), Doing Good Social Science: Trust,
Accuracy, Transparency. Washington, DC: APA.
Meehl, P. E. (1990). Why summaries of research on psychological theories are often
uninterpretable. Psychological Reports, 66, 195-244. doi: 10.2466/PR0.66.1.195-244
McKiernan, E. C. et. al. (2016). How open science helps researchers succeed. [preprint]. eLife
(July 7). doi: 10.7554/eLife.16800
Nuijten, M. B., Hartgerink, C. H., van Assen, M. A., Epskamp, S., Wicherts, J. M. (2015). The
prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research
Methods, 1-22. doi: 10.3758/s13428-015-0664-2
Open Science Collaboration (2015). Estimating the reproducibility of psychological science.
Science, 349, aac4716. doi: 10.1126/science.aac4716
Peterson, D. (2016). The baby factory: Difficult research objects, disciplinary standards, and the
production of statistical significance. Socius: Sociological Research for a Dynamic
World, 2, 1-10. doi: 10.1177/2378023115625071
4
Rouder, J. N., Morey, R. D., Verhagen, J., Province, J. M., & Wagenmakers, E. J. (2016a). Is
there a free lunch in inference? Topics in Cognitive Science, 8, 520-547.
10.1111/tops.12214
Rouder, J. N., Morey, R. D., & Wagenmakers, E. J. (2016b). The interplay between subjectivity,
statistical practice, and psychological science. Collabra, 2(1),1–12, doi:
http://dx.doi.org/10.1525/collabra.28
Simmons, J.P., Nelson, L.D., Simonsohn, U. (2011). False-positive psychology: Undisclosed
flexibility in data collection and analysis allows presenting anything as significant.
Psychological Science 22(11). 1359-1366. doi: 10.1177/0956797611417632
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (October 14, 2012). A 21-word solution. Social
Sciences Research Network (SSRN). Retrieved from http://ssrn.com/abstract=2160588.
doi: http://dx.doi.org/10.2139/ssrn.2160588
Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file drawer.
Journal of Experimental Psychology: General, 143(2), 534-547. doi: 10.1037/a0033242
Torfs, P. & Brauer, C. (Mar 3, 2014). A (very) short introduction to R. Retrieved from
https://cran.r-project.org/doc/contrib/Torfs+Brauer-Short-R-Intro.pdf
Wagenmakers, E. J. (2007). A practical solution to the pervasive problem of p values.
Psychonomic Bulletin & Review, 14(5), 779-804. doi:
Wagenmakers, E. J., Wetzels, R., Borsboom, D., & van der Maas, H . L. J. (2011). Why
psychologists must change the way they analyze their data: The case of Psi: Commentary
on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426-432. doi:
10.1037/a0022790.
Wagenmakers, E. J., Verhagen, J., Ly, A., Matzke, D., Steingroever, H., Rouder, J. N., & Morey,
R. (in press). The need for Bayesian hypothesis testing in psychological science.
Required Internet Readings and Resources
[1]
Aschwanden, C. (Nov 24, 2015). Not even scientists can easily explain p-values. [Web
log post]. Retrieved from: http://fivethirtyeight.com/features/not-even-scientists-caneasily-explain-p-values/
[2]
Avery, K., Carvell, T., Gondelman, J., Gurewitch, D., Haggerty, G., Mauer, J., Oliver, J.,
Sherman, S., Tracy, W., Twiss, J., & Weiner, J. (Writers) & Leddy, B. & Pennolino, P.
(Directors). (2016). [Television series episode]. In Carvell, T., Stanton, L., Taylor, J.,
Thonday, J., & Oliver, J (Executive Producers), Last Week Tonight with John Oliver.
New York: HBO. Retrieved from https://www.youtube.com/watch?v=0Rnq1NpHdmw
[3]
Bohanon, J. (March 3, 2016). About 40% of economics experiments fail replication
survey. Science. Retrieved from http://www.sciencemag.org/news/2016/03/about-40economics-experiments-fail-replication-survey
5
[4]
Brown, N. (2016, May). An outsider’s view of the incentive structure in psychology.
Presentation given at the Royal Society’s meeting on Replication and Reproducibility in
Psychology. London, England. Retrieved from https://www.youtube.com/watch?v=tTuZIEc0Eg&feature=youtu.be&t=1h50m15s
[5]
Center for Open Science. (Dec 29, 2015). ezANOVA in R: A simple function for new
users. [Video file]. Retrieved from https://www.youtube.com/watch?v=IXavu8skv9I
[6]
Center for Open Science. (n.d.). DataONE introduction to the Open Science Framework
(OSF) (~1hr) [Video file]. Retrieved from https://www.dataone.org/webinars/openscience-framework-increasing-reproducibility-across-entire-research-lifecycle
[7]
Center for Open Science (Jan 30, 2015). Getting started with the Open Science
Framework. (~2min) [Video file]. Retrieved from
https://www.youtube.com/watch?v=2TV21gOzfhw
[8]
Center for Open Science (Sep 2, 2015). The Open Science Framework: OSF 101. (~1 hr)
[Video file]. Retrieved from https://www.youtube.com/watch?v=YBFUVlor08A
[9]
Chambers, C., et al. (June 5, 2013). Open letter: Trust in science would be improved by
study pregistration. [Web log post]. Retrieved from
https://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-preregistration
[10]
Gelman, A. (April 1, 2015). Enough with the replication police. [Web log post].
Retrieved from http://andrewgelman.com/2015/04/01/enough-replication-police/
[11]
Gelman, A. (March 3, 2016) More on replication crisis. [Web log post]. Retrieved from
http://andrewgelman.com/2016/03/03/more-on-replication-crisis/
[12]
Gelman, A. (June 23, 2016). It comes down to reality and it’s fine with me ‘cause I’ve let
it slide. [Web log post]. Retrieved from http://andrewgelman.com/2016/06/23/it-comesdown-to-reality-and-its-fine-with-me-cause-ive-let-it-slide/
[13]
Gelman, A. (June 26, 2016). When are people gonna realize their studies are dead on
arrival? [Web log post]. http://andrewgelman.com/2016/06/26/29449/
[14]
Lakens, D. (June 30, 2014). Data peeking without p-hacking. [Web lob post]. Retrieved
from http://daniellakens.blogspot.com/2014/06/data-peeking-without-p-hacking.html
[15]
Lakens, D. (Jan 6, 2015). Always use Welch’s t-test instead of Student’s t-test. [Web log
post] Retrieved from http://daniellakens.blogspot.com/2015/01/always-use-welchs-t-testinstead-of.html
[16]
Lakens, D. (May 20, 2016). Absence of evidence is not evidence of absence: Testing for
equivalence. [Web log post]. Retrieved from
http://daniellakens.blogspot.nl/2016/05/absence-of-evidence-is-not-evidence-of.html
6
[17]
Lakens, D. (June 23, 2016). NWO pilot project will fund €3.000.000 worth of replication
research. [Web log post]. Retrieved from http://daniellakens.blogspot.com/2016/06/nwopilot-project-will-fund-3000000.html
[18]
Lakens, D. (July 18, 2016). Dance of the Bayes factors. [Web log post]. Retrieved from
http://daniellakens.blogspot.com/2016/07/dance-of-bayes-factors.html
[19]
Srivastava, S. (August 11, 2016). Everything is fucked: the syllabus. [Web log post].
Retrieved from https://hardsci.wordpress.com/2016/08/11/everything-is-fucked-thesyllabus/
[20]
Martin, R. (Narrator). (2015, May 24). Author says researcher faked gay marriage
opinion study. In S. L. Oliver (Producer), Weekend Edition Sunday. Washington, DC:
National Public Radio. Retrieved from
http://www.npr.org/2015/05/24/409210207/author-says-researcher-faked-gay-marriageopinon-study
[21]
Vazire, S. (February 2016). It’s the end of world as we know it… and I feel fine. [Web
log post]. Retrieved from http://sometimesimwrong.typepad.com/wrong/2016/02/end-ofthe-world.html
[22]
Vazire, S. (March 2016). Is this what it sounds like when the doves cry? [Web log post].
Retrieved from http://sometimesimwrong.typepad.com/wrong/2016/03/doves-cry.html
[23]
Warren, R. (August 3, 2016). One thing I learned by editing Sociology of Education
[Web log post]. Retrieved from https://thesocietypages.org/edsociety/2016/08/03/onething-i-learned-by-editing-sociology-of-education/
[24]
Goldacre, B. (Feb 25, 2016). How did the NEJM respond when we tried to correct 20
misreported trials? [Web log post]. Retrieved from http://compare-trials.org/blog/howdid-nejm-respond-when-we-tried-to-correct-20-misreported-trials/
Optional / Supplemental texts:

Maxwell, S.E. & Delaney, H. D. (2003). Designing Experiments and Analyzing Data: A
Model Comparison Approach. NY: Taylor and Francis. Note: This is the best book I
have ever seen for this material. I don’t require it because it is huge and expensive,
but if you want a deep understanding of these techniques, there is nothing better.

Gamst, G., Meyers, L. S., & Guarino, A. J. (2008). Analysis of variance designs: A
conceptual and computational approach with SPSS and SAS. NY: Cambridge.

Vasishth, S. & Broe, M. (2011). The foundations of statistics: A simulation-based
approach. New York: Springer.

Rutherford, A. (2011). ANOVA and ANCOVA: A GLM approach (2nd ed.). Hoboken, NJ:
Wiley
7

Vik, P. W. (2013). Regression, ANOVA, and the general linear model: A statistics
primer. Thousand Oaks: SAGE

Matloff, N. (2011). The art of R programming: A tour of statistical software
development. San Francisco: No Starch Press.

Kabacoff, R. I. (2011). R in action: Data analysis and graphics with R. Shelter Island,
NY: Manning Press.
Internet resources
Open Science Framework (OSF). http://osf.io
OSF guide:
http://help.osf.io/m/gettingstarted
Preregistration services
OSF preregistration template
AsPredicted
https://osf.io/prereg/
http:/aspredicted.org
Also see Center for Open Science
(COS) preregistration challenge.
https://cos.io/prereg/
Preprint Servers
PsyArVix
OSF prerpints
SocArVix
PeerJ
FigShare
coming soon! follow @PsyArVix on twitter for updates
coming soon! https://osf.io/preprints/
https://osf.io/view/socarxiv/
https://peerj.com
https://figshare.com
Course Expectations




All students are expected to attend class, read assigned materials and participate in class
discussions and activities.
Cheating, Plagiarism or any other form of academic dishonesty will not be tolerated.
Students are responsible for the ETSU Code of Academic Integrity.
All wireless devices, including cell phones, must be on silent during class.
Students are expected to have made an intense and good-faith effort (via experimentation
or reading) to solve programming or analysis questions prior to seeking out help from the
instructor.
Evaluation
Course evaluation will consist of eleven computer assignments, three exams, and one researcher
paper with class presentation.
Exams (2x, 40 points each, 40% of grade cumulatively)
8
I am planning to administer the midterm exam in class but reserve the right to do outside of
class. They may consist of multiple choice, short answer, and essay questions. Mathematical
operations and the debugging, writing, or understanding of computer code may be required.
Homework (10x, 10 points each, 50% of grade)
The only way to learn statistics is to practice. This course will feature frequent homework
assignments to all you to practice the ideas and methods that we will cover in class. The
assignments will be distributed in class and posted online. Homework is expected to be
completed by the beginning of class on the due date. Extra credit may sometimes be offered for
particularly thorough or incisive responses to homework questions at the instructor’s discretion.
Research Philosophy paper (10% of grade, 20 pts)
This paper is your opportunity to formally react to the reading assignments. Do you believe that
research practices in your field need to change? If so, how? What will you, specifically, do in
your research in order to maximize the trustworthiness and reproducibility of your findings?
How do you think these changes will be received by your advisor and/or lab? Do you think your
approach to doing science will facilitate or harm your career prospects? 10-20 pages.
Grade Scale
89.5% and up
77.5% to 89.4%
69.5% to 77.4%
below 69.5%
Grade
A
B
C
F
Accommodations
It is the policy of ETSU to accommodate students with disabilities, pursuant to federal law, state
law and the University's commitment to equal educational opportunities. Any student with a
disability who needs accommodations, for example arrangement for examinations or seating
placement, should inform the instructor at the beginning of the course. Faculty accommodation
forms are provided to students through Disability Services in the D.P. Culp Center, Suite A,
telephone 439-8346.
Syllabus changes
I reserve the right to make changes to this syllabus or the course schedule as needed to
accommodate the learning needs of the class. The most current syllabus will be posted on D2L
and any changes will be announced in class.
FINAL EXAM: Monday December 5th 1:20-3:20pm, Rogers-Stout 425
9
Academic Calendar
Click here for important ETSU dates such as drop dates, holidays, etc.
http://www.etsu.edu/etsu/academicdates.asp#fallcurryear
Mental Health
Students often have questions about mental health resources, whether for themselves or a friend
or family member. There are many resources available on the ETSU Campus, including: ETSU
Counseling Center (423) 439-4841; ETSU Behavioral Health & Wellness Clinic (423) 439-7777;
ETSU Community Counseling Clinic: (423) 439-4187. If you or a friend is in immediate crisis,
call 911. Available 24 hours per day is the National Suicide Prevention Lifeline: 1-800-273TALK (8255).
10
Preliminary Schedule (note: subject to change)
Bracketed numbers (e.g., [1]) indicate internet resources
Week Week of
Topics
Readings
1
8/22
Notation, random variables, distributions, R intro
Torfs & Brauer (2014); Ioannidis (2005); [2]; [12]
2
8/29
Intro hypothesis testing, power, effect size
Dienes ch 1; Begley & Ellis (2012); [4], [9]
HW 1
3
9/5 (No class on 9/5 for
Labor Day)
Power, effect size, confidence intervals, single
sample t test
Simmons et al (2011); John et al (2012); [15]; [20]; [21]
HW 2
4
9/12
Paired t-test, intro oneway ANOVA
Dienes ch 2; Makel et al (2012); Open Science Collaboration
(2015)
HW 3
5
9/19
Type-I error control
Lurquin et al (2016); Francis et al (2014); [3]; [10]
6
9/26
Contrast coding and effect size
Dienes ch 3; Peterson (2016); Haggar et al (2016); [11]
7
10/3
Intro to factorial ANOVA, simple effects
Bem (2011); Wagenmakers et al (2011); [1]; [13]; [24]
8
10/10 (No class on 10/10 for
Fall break - no class
fall break)
Simmons et al (2012); Nuijten et al (2015); [19]; [22]; [23]
Exam 1
9
10/17
3-way factorial ANOVA
McKiernan et al (2016); Kidwell et al (2016); [17]
HW 5
10
10/24
Repeated measured ANOVA, univariate and
multivariate tests
Simonsohn et al (2014); Lakens (2014); Dienes ch 4
HW 6
11
10/31 (No class on 11/2 for
NAGC)
Mixed ANOVA, simple effects, and contrasts
Cumming (2013); Ledgerwood et al (2016); [5]; [6]; [7]
HW 7
12
11/7
Analysis of covariance, ANCOVA vs gain scores
McBee et al (2016); Meehl (1990); [8]; [14]
HW 8
13
11/14
Bayesian statistics intro, Bayes theorem, role of
priors
Masson (2011); Jarosz et al (2014); [16]
HW 9
14
11/21 (No class on 11/23 for Bayes factors, posterior distributions, credible
Thanskgiving)
intervals
Dienes ch 5; Wagenmakers et al (2007); Rouder et al (2016a)
15
11/28
Rouder et al (2016b); Wagenmakers et al (in press), [18]
12/5 Final Exam (1:20-3:20)
Using JAGS, JASP, and BayesFactor package
Assignment due
HW 4
HW 10
Exam 2, Paper due
11