Experiment Basics: Designs
Psych 231: Research
Methods in Psychology

Exam 2 a week from today


Review sessions after labs this week
(DEG13)
First draft of Class Experiment APA paper
due in Labs this week
Announcements

Some specific experimental designs.


Some bad (but common) designs
Some good designs
•
•
•
•
1 Factor, two levels
1 Factor, multi-levels
Factorial (more than 1 factor)
Between & within factors
Experimental designs

How does anxiety level affect test performance?
Random
Assignment
Dependent
Variable
Anxiety
Low
Test
Moderate
Test
test performance
participants
1 factor - 2 levels
low
moderate
anxiety


For more complex theories you will typically
need more complex designs (more than two
levels of one IV)
1 factor - more than two levels


Basically you want to compare more than two
conditions
The statistics are a little more difficult, an ANOVA
(Analysis of Variance)
1 Factor - multilevel experiments

Good design example (similar to earlier ex.)

How does anxiety level affect test performance?
• Groups take the same test
• Grp1(low anxiety group): 5 min lecture on how good grades
don’t matter, just trying is good enough
• Grp2 (moderate anxiety group): 5 min lecture on the
importance of good grades for success
• Grp3 (high anxiety group): 5 min lecture on how the
students must pass this test to pass the course
1 Factor - multilevel experiments
Random
Assignment
participants
IV:
Anxiety
Dependent
Variable
Low
Test
Moderate
Test
High
Test
1 factor - 3 levels
low
mod
high
60
80
60
test performance
anxiety
low
mod
high
anxiety
1 Factor - multilevel experiments

Advantages

Gives a better picture of the relationship
(functions other than just straight lines)
2 levels
test performance
test performance
low moderate
anxiety

3 levels
low
mod
high
anxiety
Generally, the more levels you have, the less
you have to worry about your range of the
independent variable
1 Factor - multilevel experiments

Disadvantages


Needs more resources (participants and/or
stimuli)
Requires more complex statistical analysis
(ANOVA [Analysis of Variance] & follow-up
pair-wise comparisons)
1 Factor - multilevel experiments

The ANOVA just tells you that not all of the groups
are equal.

If this is your conclusion (you get a “significant ANOVA”)
then you should do further tests to see where the differences
are
• High vs. Low
• High vs. Moderate
• Low vs. Moderate
Pair-wise comparisons

Two or more factors

Some vocabulary
• Factors - independent variables
• Levels - the levels of your independent variables
• 2 x 4 design means two independent variables, one with 2 levels and one with 4
levels
• “Conditions” or “groups” is calculated by multiplying the levels, so a 2x4
design has 8 different conditions
B1
B2
B3
B4
A1
A2
Factorial experiments
Two or more factors


Main effects - the effects of your independent variables ignoring
(collapsed across) the other independent variables
Interaction effects - how your independent variables affect each
other
• Example: 2x2 design, factors A and B
B1
• Interaction:
• At A1, B1 is bigger than B2
• At A2, B1 and B2 don’t differ
Dependent Variable

B2
A2
A1
A
Everyday interaction = “it depends on …”
Factorial experiments

Rate how much you would want to see a new
movie (1 no interest, 5 high interest):


5
Hail, Caesar! – new Cohen Brothers movie in 2016
Ask men and women – looking for an effect of gender
4.5
4
3.5
3
M en
Women
2.5
2
1.5
1
Interaction effects
Not much of a difference:
no effect of gender

Maybe the gender effect depends on whether you know
who is in the movie. So you add another factor:

Suppose that George Clooney or Scarlett Johansson
might star. You rate the preference if he were to star
and if he were not to star.
5
4.5
4
3.5
M en
Women
3
2.5
2
1.5
1
No star
Effect of gender
depends on
whether George
or Scarlett stars in
the movie or not
This is an
interaction
Clooney stars Johansson stars
Interaction effects
A video lecture from ThePsychFiles.com podcast

The complexity & number of outcomes increases:
• A = main effect of factor A
• B = main effect of factor B
• AB = interaction of A and B
• With 2 factors there are 8 basic possible patterns of results:
1) No effects at all
2) A only
3) B only
4) AB only
5) A & B
6) A & AB
7) B & AB
8) A & B & AB
Results of a 2x2 factorial design
A1
B1
B2
A2
Condition
Condition
mean
mean
A1B1
A2B1
Condition
Condition
mean
mean
A1B2
A2B2
A1 mean
Interaction of AB
What’s the effect of A at B1?
What’s the effect of A at B2?
B1 mean
Main
effect
of B
B2 mean
A2 mean
Main effect
of A
Marginal
means
2 x 2 factorial design
A1
A2
Main Effect
of B
B1
30
60
45
B2
30
60
45
30
60
B
Dependent Variable
A
Main Effect
of A
Main effect of A
Main effect of B
Interaction of A x B
B1
B2
A2
A1
A
✓
X
X
•
At A1: B1 = B2
•
At A2: B1 = B2
The effect of A doesn’t
depend on level of
B
Examples of outcomes
B1
A1
A2
Main Effect
of B
60
60
60
B
B2
30
30
45
45
30
Dependent Variable
A
Main Effect
of A
Main effect of A
X
Main effect of B
✓
Interaction of A x B X
B1
B2
A2
A1
A
•
At A1: B1 - B2 = 30
•
At A2: B1 - B2 = 30
The effect of A doesn’t
depend on level of
B
Examples of outcomes
B1
A1
A2
Main Effect
of B
60
30
45
30
60
45
45
45
B
B2
Dependent Variable
A
Main Effect
of A
Main effect of A
X
Main effect of B
X
Interaction of A x B ✓
B1
B2
A2
A1
A
•
At A1: B1 - B2 = +30
•
At A2: B1 - B2 = -30
The effect of A does
depend on level of B
Examples of outcomes
B1
A1
A2
Main Effect
of B
30
60
45
30
30
30
30
45
B
B2
Dependent Variable
A
Main Effect
of A
Main effect of A
Main effect of B
Interaction of A x B
B1
B2
A2
A1
A
✓
✓
✓
•
At A1: B1 - B2 = 0
•
At A2: B1 - B2 = 30
The effect of A doesn’t
depend on level of
B
Examples of outcomes
Let’s add another variable: test difficulty.
test performance
easy
medium
hard
low mod
anxiety
high
Test difficulty
anxiety
hard
medium
easy
low
mod
high
35
80
35
65
80
65
80
80
80
60
80
60
main effect
of anxiety
Interaction ? Yes: effect of anxiety depends on
level of test difficulty
Anxiety and Test Performance
main effect
of difficulty
50
70
80

Consider the results of our class experiment
✓ Main effect of cell

phone
✓

Main effect of site type
✓

An Interaction between
cell phone and site type
-0.78
0.04
Factorial designs
Resource: Dr. Kahn's reporting
stats page

Advantages

Interaction effects
– Can’t “see” interaction effects without factorial design
– Consider the interaction effects before trying to interpret the
main effects
–
Adding factors decreases the variability
– Because you are controlling more of the variables that
influence the dependent variable
– This increases the statistical Power (your ability to detect an
effect) of the statistical tests
–
Increases generalizability of the results
– Because you have a situation closer to the real world (where
all sorts of variables are interacting)
Factorial Designs

Disadvantages



Experiments become very large, and unwieldy
The statistical analyses get much more complex
Interpretation of the results can get hard
• In particular for higher-order interactions
• Higher-order interactions (when you have more than two interactions, e.g.,
ABC).
Action Movie
Romantic Comedy
5
5
4.5
4.5
4
4
3.5
3.5
M en
Women
3
2.5
M en
Women
3
2.5
2
2
1.5
1.5
1
1
Clooney stars
Johansson stars
Clooney stars
Johansson stars
Here there is a three way interaction: gender X who stars X type of movie
Factorial Designs

Some specific experimental designs.


Some bad (but common) designs
Some good designs
•
•
•
•
1 Factor, two levels
1 Factor, multi-levels
Factorial (more than 1 factor)
Between & within factors
Experimental designs

What is the effect of presenting words in
color on memory for those words?

So you present lists of words for recall either in
color or in black-and-white.
Clock
Chair
Cab

Clock
Chair
Cab
Two different designs to examine this question
Example

Between-Groups Factor
 2-levels
 Each of the participants is in only one level of the IV
levels
Clock
Colored
Chair
words
Cab
participants
Test
BW
words
Clock
Chair
Cab

Within-Groups Factor
 Sometimes called “repeated measures” design
 2-levels, All of the participants are in both levels of
the IV
levels
participants
Colored
words
Clock
Chair
Cab
Test
BW
words
Clock
Chair
Cab
Test

Between-subjects
designs
 Each participant
participates in one and
only one condition of
the experiment.

Within-subjects designs

All participants
participate in all of the
conditions of the
experiment.
Colored
words
participants
Test
BW
words
participants
Colored
words
Test
BW
words
Test
Between vs. Within Subjects Designs

Between-subjects
designs
 Each participant
participates in one and
only one condition of
the experiment.

Within-subjects designs

All participants
participate in all of the
conditions of the
experiment.
Colored
words
participants
Test
BW
words
participants
Colored
words
Test
BW
words
Test
Between vs. Within Subjects Designs

Clock
Colored
words Chair
Cab
Advantages:
participants
Test
BW Clock
words
Chair
Cab

Independence of groups (levels of the IV)
• Harder to guess what the experiment is about without
experiencing the other levels of IV
• Exposure to different levels of the independent variable(s)
cannot “contaminate” the dependent variable
• Sometimes this is a ‘must,’ because you can’t reverse the
effects of prior exposure to other levels of the IV
• No order effects to worry about
• Counterbalancing is not required
Between subjects designs

Disadvantages
Clock
Colored
words Chair
Cab
participants
Test
BW Clock
words
Chair
Cab

Individual differences between the people in the
groups
• Excessive variability
• Non-Equivalent groups
Between subjects designs

The groups are composed of different
individuals
participants
Colored
words
BW
words
Individual differences
Test

The groups are composed of different
individuals
participants

Colored
words
BW
words
Excessive variability due to individual
differences

Test
Harder to detect the effect of the IV if there
is one
Individual differences
NR
R
R

The groups are composed of different
individuals
participants

Colored
words
Test
BW
words
Non-Equivalent groups (possible confound)

The groups may differ not only because of the IV, but also because the
groups are composed of different individuals
Individual differences

Strive for Equivalent groups



Created equally - use the same process to
create both groups
Treated equally - keep the experience as
similar as possible for the two groups
Composed of equivalent individuals
• Random assignment to groups - eliminate bias
• Matching groups - match each individuals in one
group to an individual in the other group on relevant
characteristics
Dealing with Individual Differences
Group A
Red
Short
21yrs
Blue
tall
23yrs
Green
average
22yrs
Brown
tall
22yrs
Group B
matched
matched
matched
matched
Matching groups
Red
Short
21yrs
Blue
tall
23yrs
Green
average
22yrs
Brown
tall
22yrs

Matched groups


Trying to create
equivalent groups
Also trying to reduce
some of the overall
variability
• Eliminating variability
from the variables
that you matched
people on
Color
Height
Age

Between-subjects
designs
 Each participant
participates in one and
only one condition of
the experiment.

Within-subjects designs

All participants
participate in all of the
conditions of the
experiment.
Colored
words
participants
Test
participants
Colored
words
Test
BW
words
Test
BW
words
Between vs. Within Subjects Designs

Advantages:

Don’t have to worry about individual differences
• Same people in all the conditions
• Variability between conditions is smaller (statistical
advantage)

Fewer participants are required
Within subjects designs

Disadvantages


Range effects
Order effects:
• Carry-over effects
• Progressive error
• Counterbalancing is probably necessary to address these
order effects
Within subjects designs

Range effects – (context effects) can cause
a problem


The range of values for your levels may impact
performance (typically best performance in
middle of range).
Since all the participants get the full range of
possible values, they may “adapt” their
performance (the DV) to this range.
Within subjects designs

Carry-over effects


Transfer between conditions is possible
Effects may persist from one condition into
another
• e.g. Alcohol vs no alcohol experiment on the effects on
hand-eye coordination. Hard to know how long the
effects of alcohol may persist.
Condition 1
Condition 2
test
Order effects
How long do we
wait for the
effects to wear
off?
test

Progressive error


Practice effects – improvement due to repeated
practice
Fatigue effects – performance deteriorates as
participants get bored, tired, distracted
Order effects

Counterbalancing is probably necessary

This is used to control for “order effects”
• Ideally, use every possible order
• (n!, e.g., AB = 2! = 2 orders; ABC = 3! = 6 orders, ABCD = 4! = 24 orders, etc).

All counterbalancing assumes Symmetrical
Transfer
• The assumption that AB and BA have reverse effects
and thus cancel out in a counterbalanced design
Dealing with order effects

Simple case


Two conditions A & B
Two counterbalanced orders:
• AB
• BA
Colored
words
Test
BW
words
Test
BW
words
Test
Colored
words
Test
participants
Counterbalancing

Often it is not practical to use every possible
ordering

Partial counterbalancing
• Latin square designs – a form of partial
counterbalancing, so that each group of trials occur in
each position an equal number of times
Counterbalancing

Example: consider four conditions
Recall: ABCD = 4! = 24 possible orders
1) Unbalanced Latin square: each condition appears
in each position (4 orders)

Order 1
A
B
C
D
Order 2
Order 3
B
C
D
A
C
D
A
B
Order 4
D
A
B
C
Partial counterbalancing

Example: consider four conditions
Recall: ABCD = 4! = 24 possible orders
2) Balanced Latin square: each condition appears
before and after all others (8 orders)

A
B
C
D
A
B
D
C
B
C
D
A
B
C
A
D
C
D
A
B
C
D
B
A
D
A
B
C
D
A
C
B
Partial counterbalancing

Mixed factorial designs


Treat some factors as within-subjects
(participants get all levels of that factor) and
others as between-subjects (each level of
this factor gets a different group of
participants).
This only works with factorial (multi-factor)
designs
Mixed factorial designs