Experimental Design in the Real World
We learned what an experiment is in Chapter 5 and some of the terms that come up when
discussing experiments. We discussed lurking variables, confounding, the placebo effect, and
the fact that we want a randomized comparative experiment to control for these things. This
set of lecture notes talks about specific measures we take to avoid these pitfalls (and others)
in experiments.
The Goal: We want to generalize our results to the population of interest and be able to
defend the conclusions scientifically.
In order to meet this goal we need to have
– statistically significant results
– Realistic settings - treatments, subjects, and environment of our experiment
– Repeatability – the ability to repeat the experiment a number of times in various
settings and get the same results
As a start to meeting our goal, we’ll discuss three different types of randomized comparative
experiments. These specific types of randomized comparative experiments are referred to as
the experimental design or design of the experiment.
1. In a _________________________________ experimental design, all the experimental
subjects are allocated at random among all the treatments
Can have more than one explanatory variable
Use treatment combinations to conduct the study
(Comparison to survey design methods: this is the analog to a simple random sample – all
the names are put into one hat, then subjects are randomly drawn for treatments)
Draw a Picture:
2. BLOCK DESIGN
A _______________ is a group of experimental subjects that are known before the experiment
to be similar in some way that is expected to affect the response to the treatments
In a BLOCK DESIGN, the random assignment of subjects to treatment is carried out separately
within each block
(Comparison to survey design methods: this is the analog to a stratified random sample –
first divide the subjects into groups, called blocks, then carry out a separate randomized
comparative experiment within each block)
• Allows us to draw separate conclusions about each block
• We can include a potential lurking variable in the design and its effects can now be
accounted for – this is called a blocking variable
• A blocking variable is any variable that may have an effect on the response, but
isn’t one that we’re necessarily interested in – we just want to account for the
variability it would introduce in the results
• By forming the blocks to account for lurking variables, blocks reduce confounding
Draw a picture of a Block Design:
Examples of Blocking in an Experiment:
Chapter 5
Page 2
3. MATCHED PAIRS
The MATCHED PAIRS design is a type of block design that compares two treatments by
choosing pairs of subjects that are as closely matched as possible.
• Randomly assign a member of a pair to a treatment
• sometimes a single subject serves as the “pair” by taking both treatments
– Must randomly allocate the order of the treatments in this case!
Draw a picture of Matched Pairs:
Examples of possible experiments with matched pairs:
Chapter 5
Page 3
Clicker Questions
_____________________________________________________________________________
An experiment investigated the effects of repeated exposure to an advertising message
Subjects were randomly allocated to 1 of 6 treatments (pictured in the table below) and
viewed a 40 minute program which included ads for a digital camera.
Responses measured: recall of ad, attitude toward camera, intention to purchase
Identify the experimental design.
A. completely randomized
B. block design
C. matched pairs
_____________________________________________________________________________
Another experiment of the effectiveness of ad campaigns is pictured in the following table.
Identify the blocking variable.
A. Ads
B. Group
C. Reaction
D. Gender
_____________________________________________________________________________
In a randomized block design, where is the randomization performed?
A. When placing subjects into a block.
B. When picking the response of interest
C. When assigning treatments w/in a block.
D. When picking the blocking variable.
_____________________________________________________________________________
An experiment is conducted to determine which is the better insect repellent: Brand A or
Brand B.
Subjects are sprayed with repellent Brand A on one arm, and repellent Brand B on the other.
Choose the best answer:
This is an example of an experiment using
A. a completely randomized design
B. single blinding
C. matched pairs
Chapter 5
Page 4
The placebo effect presents a large problem to researchers, particularly in medical research.
Using a randomized comparative experiment with a good design is a good start, but we can do
more…
In a ____________________________________, neither the subjects, nor the people who
work with them, know which treatment each subject is receiving.
A ____________________________________ is one in which either the experimenter or
subject (but not both) knows which treatment is being administered
Why use a design that incorporates blinding?
For the patient (subject)
– May be a psychological effect if the patient knows which drug he/she is receiving
Treatment giver
– Patients known to get a new treatment may be treated differently than those on
standard treatment
Response measurer
– A physician who has a preconceived notion of how the new treatment should
work may be biased in his/her evaluation
Even in a well designed experiment that includes blinding (among other considerations not
discussed in this lecture), experiments can be plagued with similar problems to those we
learned about when studying sample surveys
Problems in Experiments
• Undercoverage (same idea as in surveys)
– Certain groups of subjects from the population of interest are left out
• Refusal to Participate
– Subjects may not like the treatment they are assigned
• Non-adherers
– Subjects who participate but do not follow the protocol for their experimental
treatment
• Dropouts
– Subjects who begin the experiment but do not complete it
Chapter 5
Page 5