1
Replicability, Generalization &
Real World
Dr. Stefanie Drew
[email protected]
2
How do you judge a study’s
importance?
• Have the tools to study whether it was conducted
well?
• How do you decide if a study is important?
▫ Should it be replicable?
▫ Should it have external validity?
▫ Should it apply to the real world?
• Keep in mind…
▫ Researcher’s priorities
▫ Study’s purpose
3
Things to think about…
• Are your results a fluke or are they replicable?
• Replicable: pertains to a study whose results
have already been obtained again when study
was conducted
▫ Gives credibility!
Image adapted from www.jayhosler.com/jshblog/?p=726 on April 23, 2014
4
5
Replication Studies
• Replication studies: when researcher actually
performs study again
Replication
Studies
Direct
Replication
Conceptual
Replication
Replicationplusextension
6
Replication Studies: Direct Replication
• Direct replication (aka exact replication):
Researchers repeat original study as closely as possible
• Looking to see if original effect is outcome
Replication
Replication
Replication
• Keep in mind: Threats present in original study may
also be present in replication
7
Experimental Example
• What if seeing blue makes
people feel more sympathy?
▫ Study: Female participants
view pictures of sad people
wearing blue shirts or white
shirts, then report how much
they sympathize
Image adpated from www.clker.com/clipart-polaroid-b-s.html on April 30, 2014
Image adapted from shelliebee.blogspot.com/2011/01/whos-that-with-sad-face.html on April 27, 2013
• Researchers try to replicate as
close to original as possible
• Even in direct replication,
some small variations
▫ Different sets of participants
▫ May not be at exact same
time of year
▫ Experimenters may be
different
8
Replication Studies: Conceptual
Replication
• Conceptual replication: researchers use
different methods to address same research
question as another study
▫ E.g. Variables are the same, different
operationalizations
Image adapted from www.cf.ac.uk/socsi/contactsandpeople/harrycollins/paper-1.html on April 30, 2014
9
Experimental Example
• What if seeing blue makes people feel more sympathy?
▫ Study 1: Female participants view pictures of sad people
wearing blue shirts or white shirts, then report whether
they sympathize
▫ Study 2: Female participants sit in blue or white room and
rate their sympathy for characters in different stories.
• Looking to see if overall exposure to blue causes people
to have be more sympathetic
CONCEPTUAL
OPERATIONAL
IV:
Exposure to
Blue
Color of
shirts
Color of
Room
DV:
Sympathy
Report
Sympathy
Rate
sympathy
10
Replication Studies: Replication-plusextension
• Replication-plus-extension: researchers
replicate original study but add variables to
test additional questions
• Can introduce new
▫ Participant variables
▫ Situation variables
11
Experimental Example 1
• What if seeing blue makes
people feel more sympathy?
• Introduces a participant variable
▫ Original Study: Female
participants view pictures of
sad people wearing blue
shirts or white shirts, then
report how much they
sympathize
▫ E.g. participant gender variable
▫ Extension: Study repeated
with both male and female
participants
Image adpated from www.clker.com/clipart-polaroid-b-s.html on April 30, 2014
Image adapted from shelliebee.blogspot.com/201101/whos-that-with-sad-face.html on April 27, 2013
12
Experimental Example 2
• What if seeing blue makes
people feel more sympathy?
▫ Original Study: Female
participants view pictures of
sad people wearing blue
shirts or white shirts, then
report how much they
sympathize
▫ Extension: Study repeated
during summer and winter
Image adpated from www.clker.com/clipart-polaroid-b-s.html on April 30, 2014
Image adapted from shelliebee.blogspot.com/2011/01/whos-that-with-sad-face.html on April 27, 2013
• Introduces a situational variable
▫ E.g. changing time of year
13
Overview
Direct Replication
Uses same methods
to study same
variables as original
Image adapted from jobvanwolferen.wordpress.com/2013/01/31/replication-sample-sizes/ on April 27, 2013
Conceptual
Replication
Use different
methods to study
same variables as
original
Replication-plusextension
Replicate original
study but add
variables to test
additional questions
14
Reviewing the Literature
• Scientific literature (or just “literature”):
series of related studies conducted by different
researchers testing similar variables
• Several Different Methods for reviewing
literature
▫ Summarize the literature
Review article or Literature review
▫ Quantitative
Meta Analysis
15
Image adapted from acadiaabroad.blogspot.com/2011/07/recidivism.html on April 27, 2013
Meta-analysis
• Meta-analysis: process collecting all
studies on a topic mathematically
combining to examine overall trend
• Uses effect-size analyses to combine
results from several studies
• Can use meta analytic formula to find
average across studies
Study
2
Study
1
Study
3
▫ Similar to mean, but weights studies with
larger sample sizes heavier
• Average all of the effect sizes to find
overall effect size
▫ r
▫ d
I
if correlation
if group difference
Overall Trend
16
Image adapted from www.all-about-psychology.com on April 30, 2014
How are Meta-analyses used?
• Results of meta-analyses can
identify how studies can be
grouped into categories
• Researchers can calculate
effect sizes for different
categories
• Value of combining results
across studies for single
average
▫ Removes small differences
▫ Conceptual variables remain
• Researchers can detect new
patterns in literature
Image adapted from intergroup.uconn.edu/foels/research/pedagogy.html on April 30, 2014
17
Considerations for Meta-analyses
• Data are typically from published literature,
assuring peer review
• Caution: publication bias
▫ File drawer problem: studies with null effects
less likely to be published than those with
significant effects
▫ Solution: contact researchers for unpublished data
as well
18
Remembering r and d
Effect size d
• Group difference
• Comparing difference between
means in standard deviation
units
• Larger = greater the
difference between two
groups
Correlation Coefficient r
• Indicates strength of
relationship between two
variables
• Correlation
Cohen’s descriptions
Effect size d
Description
Compare to r
0.50
Moderate/medium
0.30
0.20
0.80
Weak/small
Strong/large
0.10
0.50
19
20
Do the following support external
validity?
• Direct replication studies?
• Conceptual replication studies?
• Replication-plus-extension studies?
• Why?
21
Image adapted from en.wikipedia.org/wiki/Generalization on April 27, 2013
Generalizing to People
• How were the participants obtained?
▫ “How” not “How many”
• What is the population of interest?
▫ “A” population not “The” population
22
Generalizing to Other Settings
• Ecological validity: (a.k.a. mundane realism)
similarity of experiment real-world contexts
Image adapted from https://twitter.com/RealWorldMTV on April 27, 2013
23
When is External Validity important?
• Two modes to consider
▫ Testing theory mode
▫ Generalization mode
24
Theory Testing Mode
• Theory testing mode: testing
association or causal claims to test
support for theory
• Often researchers design
experiments to
▫ test competing explanations
▫ confirm disconfirm hypothesis
• Not as concerned with external
validity
Image adapted from www.sodahead.com/living/do-you-know-why-cats-are-so-attracted-to-eating-mice/question-266823/ on April 16, 2013
25
Experimental Example
• Remember Harlow’s monkeys
▫ What were the two theories?
▫ Were the tested monkeys representative?
▫ Were the tested monkeys randomly selected?
Image adapted from http://psychclassics.yorku.ca/Harlow/love.htm on January 22, 2013
26
Generalization Mode
• Generalization Mode:
trying to generalize findings
from sample in study to larger
population
▫ Use probability samples with
appropriate diversity
• Very concerned about external
validity
27
So which studies do which???
• Research supporting frequency claims is in
generalization mode
• Many times association and causal claims are in
theory-testing mode
▫ But can be in generalization mode too
• Applied Research tends to be done in
generalization mode
• Basic Research tends to be done in theory-testing
mode
Participation Activity
29
Special Case: Cultural Psychology
• Cultural psychology: focus is
generalization mode
▫ Focuses on how cultural
background and environment
shapes a person
• Experimental Example
▫ Almost all North Americans
see fall for illusion, but not all
people
▫ Segall et al (1966) report
individuals exposed to right
angle visual cues (e.g. corner
of a room) in their culture
perceive illusion more
Müller-Lyer illusion
30
Image adapted from www.nationalgeographic.com/explorers/grants-programs/cre-application/ on April 28, 2013
Relation to the Real World
• Field setting: real world setting for
research study
▫ Advantage: Already applies to real
world
• Ecological validity: type of
external validity, refers to similarity
between study and the situations
encountered in everyday life
▫ Basically: real-world similarity
• Experimental realism: how much
laboratory study is designed so
participants experience authentic
emotions/motivations/behaviors
▫ Property of certain laboratory
experiments