Experimentation
Chapter 9
What are experiments?
• These are studies in which a researcher
manipulates an independent variable in order to
see the affect on some dependent measure
• “Control” is very important. Basically, this
means making everything equal between or
among the different experimental groups
– Random assignment, situation, procedures, repetition
Conducting experiments
•
•
•
•
•
•
•
•
1) select the setting;
2) select the experimental design;
3) operationalize variables;
4) decide how to manipulate the independent
variable;
5) select and assign subjects to experimental
conditions;
6) conduct a pilot study;
7) administer the experiment; and,
8) analyze and interpret the results.
Between Subjects Design
• Separate groups of subjects are exposed to
levels of
independent variable
Within Subjects Design
• The same group of subjects are exposed to
levels of
independent variable
• Sometimes referred to as “repeated measures”
design
Single Subjects Design
• Individual subjects are exposed to levels
of independent variable
Mixed Design
• Use of between subjects and within
subjects design
Notation
•
•
•
•
R= random assignment
X= independent variable
No X= control group
O= dependent variable
Randomized Two-group Design
•R X
•R
O1
O2
Compare difference between O1 and O2
Single Factor Parametric Design
R
X1
O1
R
X2
O2
R
X3
O3
R
O4
Single Factor Nonparametric
R
X1
O1
R
X2
O2
R
X3
O3
R
O4
Factorial designs
• • These are experimental designs with 2 or
more independent variables.
• • We have both main effects of the
independent variables and the interactions.
• • The main effects are the separate effects of
each independent variable.
• • The interactions occur when the effect of one
independent variable changes across levels of
another independent variable.
Eg. Of factorial design
•
•
•
newspaper/no newspaper and radio
and no radio
where a TV station manager wants to
promote a newscast and wants to know
what is relative or combined strength of
print ads versus radio
• .Influence of Story and Structure on Perceived
Story Bias and News Organization
Credibility. By: Fico, Frederick; Richardson,
John D.; Edwards, Steven M.. Mass
Communication & Society, Summer2004
Internal validity
researcher-related
• Experimenter effect: when a researcher or confederate
knows the purpose of the study and unconsciously treats
participants differently in various conditions
– Also known as Pygmalion effect or Rosenthal effect
• Observer Bias
– When a researcher who knows the purpose of the study is
biased in observing dependent variable (sees what they expect
to see)
• Researcher attribute effect
– Some attribute of the researcher that could bias results
– E.g. male vs female research confederates for different groups
Internal validity
Participant-related problems
• The Hawthorne Effect: when subjects think they
know they are being studied and thus behave
out of the ordinary
• Testing effect; knowledge of a pretest affects the
posttest
• Maturation
• Mortality
• Selection bias; when the groups being
compared in an experiment are not the same at
the beginning of the study (PBS eg.)
Participant-related
• Intersubject biases; when experimental
subjects can interact before an experiment
and contaminate the findings
• Compensatory rivalry; if subjects in a
control group believe they are being
denied something (like the stimulus..) they
may overcompensate
• Demoralization; if they (control) feel
denied they may become demoralized
Procedure-related problems
• History
• Instrumentation
• Treatment confound; sometimes the treatment is
woven together with another variable…hard to
separate relative effect
• Statistical regression
• Compensation; pressures to compensate the
control group for that which they have been
denied (ie. The treatment or independent
varible)
Pre-experimental designs
•
•
•
One- shot case study (e.g. show people with smiles after using
shampoo)
One-group pretest-posttest design (before and after commercials
Static-group comparison (compare brand X and Y)
O1
X
O
X
O1
X
O1
O2
Quasi-experimental designs
• Time-series (new teaching procedure eg.)
• Nonequivalent control group design (2
different schools eg.)
• Multiple-time-series design (traffic
accidents eg…)
Quasi-experimental designs
Time series O1
O2
O3
X
O5
O6
O7
Non-equivalent control group
O1
X
………………………………………………..
O3
O2
O4
Multiple time series
O1
O2
O3
X
O5
O6
……………………………………………….
O8
O9
O10
O11
O12
O7
O13
True experimental designs
• pretest-posttest control group design
• Posttest-only control group design
• Solomon four-group design
• Random assignment: based on random
sampling process
• Double-blind: when both researchers and
subjects do not know when they get
independent variables
• Manipulation checks; to make certain that the
treatment (independent) variable did what it was
intended (e.g. have a question on posttest that
asks if subjects remember seeing something in
the movie…the treatment)
External validity problems
• Interaction between testing setting and treatment
– Does the lab setting impact the generalizability?
• Interaction between selection and treatment
– Use of university students vs gen population
• Interaction between history and treatment
– Views toward computer use change so rapidly..make
sure the procedure matches reality
2X2 Factorial design
Design notation for 2X2 factorial design
Posttest only
• R
• R
X
O
O
Pretest-posttest
• R O
• R O
X
O
O
Solomon 4 group design
•
•
•
•
Combines pre and posttest only designs
Reduces internal validity problems
Requires more subjects
NOT a factorial design
Solomon four group design
Interaction effects and main effects
Interaction
effects
Main
Effects