Experiments

The Beauty of Random Assignment

How Experiments Work

Strengths and Weaknesses
The Beauty of Random Assignment

Problem: In non-experimental studies,
what determines the values that an
independent variable takes on? Often, a
confounding variable determines these
values, and affects the DV.

For instance, the confound of “Are you a
serious student” may determine where you
will sit in a class.
An Example of a Confound from
Recent Research

For example:
Does contacting registered voters through
phone calls or visits make them more likely
to turn out on election day?
 Potential Confound: Previous participation
records. Campaigns with limited resources
concentrate their mobilization efforts on
voters who have turned out in the past.

The Beauty of Random Assignment

Solution: Interrupt the causal path that leads
from the confound to the independent variable
by “randomly assigning” the values that the IV
takes on in each case.


Randomly assign seats so that there are just as
many serious students and slackers in each part of
the lecture hall.
Randomly assigned some voters to be contacted,
in order to measure the actual effects of
mobilization (which are quite weak).
The Beauty of Random Assignment

Treatment Group

All of the cases in
this group have
been assigned one
value of the IV (sit
in front, take
medicine, etc.)

Control Group

All of the cases
have been assigned
a different value of
the IV (in most
cases, left alone or
given placebo)
In all other respects (including the values of confounding
variables that they take on), these groups are similar.
How Experiments Work
Step #1: Random Assignment

Begin by splitting your cases into two or
more groups of 30 or more through a
process that is truly random.
Using something like a random number
generator is key, because many seemingly
neutral assignment processes may be
correlated with a confounding variable.
 Examples: arrival times, last names, section
times, bleeding hearts.

How Experiments Work
Step #2: (Optional) Pre-Test

To check how the random assignment
process worked, measure the value that
the DV takes on for each case before any
treatment has been applied.
Each group should average about the same
values on the dependent variable.
 Even if something went wrong, we can still
learn from the “time-series” comparison.
 Often it is hard to pre-test.

How Experiments Work
Step #3: Apply the Treatment

Change the value of the independent
variable in the treatment group.

Administer the medicine or the placebo, put
students in their seats, request that subjects
administer an electric shock.

This is where ethical issues arise.
How Experiments Work
Step #4: Post-Test

Measure the value that the DV takes on
for each case AFTER the treatment has
been applied.
Comparing values of the DV in treatment
group vs. control group tells us the effect of
the treatment, if random assignment
worked.
 Comparing shifts from pre-test to post-test
is helpful when random assignment failed.

Schematic of an Experiment
Treatment Group
(pre-test)
Random
Assignment
Control Group
Treatment
(post-test)
Compare
(pre-test)
(post-test)
Strength of Experiments:
High Internal Validity

Internal validity judges how well a
research design has tested a causal
relationship, in the cases examined.
“Among the cases in our study, do we have
reason to believe that IV #1 causes DV?
 Because random assignment takes away
our fear of confounds, experiments have
high internal validity.

Weakness of Experiments:
Low External Validity

External validity judges how confident we
can be that a causal relationship
identified in our cases can be generalized
to the outside world.

Our cases may be different than the general
population, or our cases may react differently to
treatments, or our treatments may be very artificial.

You can’t assign every treatment: gender, race.
Studying Sensitive Topics - 1
I. Race – The List Experiment
A. One half of a general population, randomly
chosen, is asked how many of three items
– the federal government increasing the
tax on gasoline, professional athletes
getting million-dollar salaries and large
corporations polluting the environment make them angry – NOT which items, just
how many. Not revealing which items
makes people more willing to express
socially undesirable answers.
Studying Sensitive Topics - 2
2. The second group gets the same three
items plus a fourth item – awarding college
scholarship based upon race.
3. If the average of the first group was 2.0 and
the average of the second group was 2.6,
then it is likely that 60% of the second
group selected awarding college
scholarships based upon race.
Studying Sensitive Topics - 3
II. Race – The Audit Experiment - Housing
A. Units being marketed for sale or rent are
randomly selected and separate teams of
white and African-American auditors
posing as home-seekers are sent to
inquire about the availability of the
advertised units and the terms under which
units might be obtained. Auditors are
assigned similar personal, social and
economic characteristics.
Studying Sensitive Topics - 4
After each encounter, the auditor fills out
a form describing the nature and
outcome. African-American auditors
were more likely than white auditors: (1)
to be told that no units were available;
(2) to be shown fewer units; and (3)
given higher down payments, rents and
interest rates.
Studying Sensitive Topics - 5
B. Phone inquiries offer another opportunity to
test for more subtle forms of discrimination.
Researchers found white real estate
agents were able to accurately identify
callers who spoke African-American
English from the messages they left. Black
“sounding” callers were significantly less
likely to (1) get a “call back”; and (2) obtain
an appointment to view a home –
especially in predominately white areas.