withstand the scrutiny of others by being valid and believable and moreover
providing its own measure of how strongly its inferences and conclusions
can be trusted.
SECTION 10.1 EXERCISES
1. Give an example of a statistical experiment.
(You can look for examples in newspapers, magazines, the internet, television, etc.)
Clearly state the goal of the experiment and
the treatment to be given.
2. Give an example of a statistical survey. Clearly
state the question that the survey tries to answer.
10.2 OBSERVATIONAL STUDIES
AND DESIGNED EXPERIMENTS
Smoking and Lung Cancer
A major ongoing public health controversy of the twentieth century has
been whether smoking causes lung cancer. In 1957 the British Medical
Journal editorialized that smoking does indeed cause lung cancer, citing
“the painstaking investigations of statisticians that seem to have closed
every loophole of escape for tobacco as the villain in the piece.” Some of
this evidence was based on data collected in 1948–1949 by Doll and Hill.ⴱ
A group of 709 people with lung cancer was compared to a group of 709
people without lung cancer, called controls (that is, members of the control
group). Using lung cancer seems a backward way to define treatment and
control groups, because we are used to thinking of lung cancer as the effect
of something (such as smoking) rather than as a treatment that could be the
cause of something. But as statisticians or medical investigators we define
treatment and control groups in the way that best suits the goals of our
study, here the groups differing in the variable of having cancer or not.
In each group there were 649 men and 60 women. Among the 1230
men from both groups who smoked (most of the men in both groups
smoked), 647 had lung cancer, this being almost 52%. However, among
the 68 nonsmoking men, only two had lung cancer, this a percentage of
just under 3%. Thus the experimental probability of lung cancer varied
enormously from smoking to nonsmoking men. The principles of good
ⴱ
See R. D. Cook, “Smoking and Lung Cancer,” in R. A. Fisher: An Appreciation, ed. S. E.
Fienberg and D. V. Hinkley (New York: Springer-Verlag, 1980), pp. 182–191; P. D. Stolley,
“When Genius Errs: R. A. Fisher and the Lung Cancer Controversy,” American Journal of
Epidemiology, vol. 133, pp. 416–425; and B. W. Brown Jr., “Statistics, Scientific Method, and
Smoking,” in Statistics: A Guide to the Unknown, ed. J. M. Tanur, F. Mosteller, W. H. Kruskal,
R. F. Link, R. S. Pieters, and G. R. Rising (San Francisco: Holden-Day, 1972), pp. 40–51. These
references discuss Fisher’s role in the smoking/lung cancer controversy, and provide many
additional references.
science require us to not presume a causal explanation here, no matter how
plausible it seems. But we can certainly say that for the men, smoking and
lung cancer seem heavily associated, at least in this study. For the women,
the rate of lung cancer among the smokers was about 59%, compared to
a 37% lung cancer rate among the nonsmokers. Again the association is
clear, but with the intriguing observation that a much higher percentage
of the nonsmoking women had lung cancer than their nonsmoking male
counterparts (a possible reason is the women may have had more exposure
to secondhand smoke, judging from today’s scientific knowledge about
secondhand smoke and lung cancer). These data, as well as data from many
other studies, clearly establish an association between smoking and lung
cancer.
Sir Ronald Fisher (1890–1962) was perhaps the most famous and influential statistician of all time (and a famous geneticist as well). In his
role as careful scientist and eminent statistician, Fisher was appalled by the
strong causal conclusions drawn by the British Medical Journal. He forcefully
pointed out (and continued pointing out for the rest of his life) that just
because smoking and lung cancer are associated, one cannot necessarily
conclude that smoking causes lung cancer. In the absence of further scientific
evidence, it could just as well be that lung cancer causes smoking, or that
a third factor causes both to tend to occur together. For example, people
in the early stages of a long-developing disease like lung cancer could
experience severe physical discomfort, and smoking could help alleviate
the discomfort. If this explanation were true, lung cancer would indeed
cause cigarette smoking. Moreover, this rather unusual causal mechanism
would produce a positive association between cigarette smoking and lung
cancer. To quote Fisher, “And to take the poor chap’s cigarettes away from
him would be rather like taking away his stick from a blind man. It would
make an already unhappy person a little more unhappy than he need be.”
As for a third factor causing both, Fisher went to great lengths to argue
that one’s genetic makeup could predispose one toward or away from
smoking, and this same genetic makeup could also affect susceptibility
to lung cancer. To understand this strange idea, imagine a gene whose
possession both causes a high rate of lung cancer and facilitates a high
rate of cigarette smoking. (Interestingly, some geneticists currently claim
that a tendency toward addictive behaviors is genetically influenced, and
certainly it is well known that certain cancers have a genetic predisposition. Any positive linkage between genetic characteristics contributing to
addictive behaviors and those contributing to cancer would make Fisher’s
claim at least somewhat true.) Other possible third factors include type of
employment (people working in coal mines, foundries, and other settings
with polluted air, may be more likely to smoke than people who work in
offices or other professional settings) or whether one lives in an urban or
rural environment (people living in an urban area may be more likely to
smoke, and also live amid more pollution).
Science is conservative in its progress and only accepts causal explanations in data-driven studies when both the statistical and scientific evidence
are very strong. Sir Ronald Fisher was in this sense acting as the good, cautious statistical scientist (even though today the evidence is overwhelming
that cigarette smoking does indeed cause lung cancer, as discussed below).
From the perspective of wanting valid statistical inferences, the big problem
with the Doll and Hill study is that it was an observational study rather
than a randomized experimental study.
Observational Studies
Fisher’s concerns are relevant for any observational study. Let’s get our
terminology clear! An observational study shares with an experimental
study the basic structure of treatments applied to units. The key distinction
is that an experimental study has the treatment of interest and the control
carefully assigned to different units, whereas an observational study has
the treatment of interest arbitrarily assigned to units, usually because
the data were collected prior to the currently posed question. Expert
judgment or, as is far better from the statistical perspective, randomization is
generally used in an experimental study to assign the treatment and control
to units. In a previously conducted observational study the investigator
has no control over which units receive which treatments. In particular,
because the relevant treatments were already assigned to units in the
past, the randomized assignment of treatments to units is impossible to
achieve. Without randomization, valid statistical inference is difficult to
impossible. Observational studies are often statistically analyzed because
the investigator is forced to use data previously collected for another
purpose than the investigator’s. Even if those data have been randomized,
they have not been randomized with respect to the current treatment of
interest! Historical data are plentiful and cheap, while the cost of careful
statistical experimentation is high. The main problem with observational
data is that the experimenter does not determine which units receive the
treatments that the study is focused on. For example, Doll and Hill decided
neither who would smoke and who would not (presumably the people
themselves decided), nor who would get lung cancer and who would not
(the treatment variable in the Doll and Hill study).
How were the people chosen for this observational study? Doll and Hill
found 709 people with lung cancer in various London hospitals. They then
searched for 709 controls, that is, people without lung cancer. They were
careful to select this control group to be in certain other important ways
similar to the treatment group, namely the people with lung cancer.
For example, there were 60 women with lung cancer, so the researchers
deliberately balanced for gender (which is another factor that could be
influencing the occurrence of lung cancer) by finding 60 women without
lung cancer. Note that although the researchers did decide who was to
be in their study and indeed did try to match the two groups in ways
that might matter, the people already either did or did not smoke, and
either had or did not have lung cancer. Further, note that any variable
not consciously controlled for by this “expert” selection of the control
group was not balanced between the two groups. By contrast, randomized
assignment of treatments to units, if it had been possible, would have had the
great advantage of simultaneously controlling for all important variables or
factors (that is, balancing the levels of each such variable in the two groups).
As Fisher claimed, there is nothing in the data themselves that rules out
lungcancercausingsmoking,orathirdfactorcausingbothsmokingandlung
cancer. To drive the point home, he looked closely at another portion of the
Doll and Hill study. Among the smokers, they categorized people as inhalers
and noninhalers, based on whether the people inhaled the smoke or not.
They found that 61.6% of the people with lung cancer inhaled, while 67.2%
of the controls (those without lung cancer) inhaled. It looks like inhaling
could help reduce the rate of lung cancer, at least if one believes here that
association implies causation. Fisher’s point is that if one is willing to believe
that smoking might cause lung cancer based on this study, one must also
accept that inhaling likely helps prevent lung cancer. The strangeness of this
last conclusion thus might put the smoking causing lung cancer viewpoint
in doubt. We will return to this strange piece of data later.
Analyzing the Key Problem
The data described in the Key Problem are also from an observational study.
Whether the women of the study smoked was in no way controlled by the
researchers. In particular, this is clearly not a case of random assignment of
treatment (smoking or not) to units (women). Clearly, smoking is positively
associated with being alive 20 years later (31% of smokers alive vs. 24%
of nonsmokers alive). But is it reasonable to conclude that the smoking is
causing longer life? There may be a third factor. In fact, there is: age in 1974.
Table 10.1 provides an expanded version of the data.
Look first at the women who were young (18–34) in 1974. Almost all
were still alive 20 years later, whether they smoked or not. There is basically
no difference in the percentage still alive. Look next at the women over
65. Most of them (about 85–86%) were dead 20 years later, again whether
they smoked or not. However, there is a difference among the women
who were middle-aged (35–64) in 1974: 26% of the smokers were dead
twenty years later, while only 18.4% of the nonsmokers were dead. Thus
smoking is not associated with death 20 years later for young and old
Table 10.1
Age in 1974
Number of Women Alive in l994
Dead
Smokers
Alive Percentage dead
Dead
Nonsmokers
Alive Percentage dead
18–34
35–64
65+
5
92
42
174
262
7
2.8%
26.0
85.7
6
59
165
213
261
28
2.7%
18.4
85.5
Total
139
443
23.9
230
502
31.4
Source: Table abstracted from table 2 in Appleton, French, and Vanderpump, “Ignoring a Covariate:
An Example of Simpson’s Paradox.”
people, but is associated with death for middle-agers. This new conclusion
seems intuitively reasonable, noting that originally these data suggested that
smoking reduces death rates! To help explain this seeming contradiction,
consider the percentages who were smokers and the percentage who died
based on the three age groups:
Percentage who
were smokers
Percentage who
were dead in 1994
18–34
35–64
45.0
52.5
2.8
22.4
65+
20.0
85.5
Age in 1974
Here we see that elderly people in 1974 were much less likely to be smokers
than younger people and that elderly people were also much less likely
to survive another 20 years. Thus these elderly people both were much
more likely to die within the next 20 years and smoked far less than the
other two age groups! These two trends together resulted in nonsmokers
being less likely to die in 20 years than smokers, but the cause was not
smoking. Rather, old age is simultaneously related to not smoking and
to dying within 20 years. We can think of age as the third factor that is
simultaneously influencing how soon one is likely to die and how likely
one is to smoke. Indeed, the influence of this third variable (age) is so strong
that, when not controlled for, it entirely hides the fact that smoking reduces
life expectancy.
Observational studies are valuable, but they cannot fail to leave doubt
about how to interpret the results. There is always the possibility of some
third factor explaining an observed association. We have seen one way to
handle such a third factor when we have identified it, namely to study the
association between the two variables of interest separately for restricted
ranges of the third factor—for example, to study the association between
smoking and life expectancy separately for the young, middle-aged, and
elderly as we did above. However, this approach is effective only if we
have been clever enough to figure out all such potentially troublemaking
third factors, a challenge we are likely to often fail at. It is important to
realize, though, that this strategy of studying the association of interest
within narrow ranges of a possibly confounding third factor is often used
effectively in statistical studies that are forced to depend on observational
data.
Fortunately there is a better way, namely the randomized statistical
experiment! It avoids the possibility of a damaging third factor by forcing
the two groups being compared to be approximately alike in every way except the treatment aspect under study. We will illustrate such a randomized
experimental study next.
Planned Experiments: The Salk Polio Vaccine
Polio is a scary disease that primarily strikes children, often leaving its
victims paralyzed. Throughout the first half of the twentieth century, it
claimed many victims, including Franklin Delano Roosevelt before he
became president. By the late 1940s and early 1950s, approximately forty
thousand people per year were contracting polio. Research revealed that
polio was caused by a virus, leading to the search for a vaccine.ⴱ In the early
1950s Dr. Jonas Salk developed a killed-virus vaccine against polio.
In 1954 it was decided to carry out a large-scale study to assess its
effectiveness.† Some observational studies were contemplated but rejected.
One such proposal was to use a given year, say 1953, to provide controls
(children not vaccinated) and the next year, 1954, as the treatment year
(vaccinated children). In the treatment year, the vaccine would be widely
dispersed, and whoever wished to use it could. One could then compare the
polio rates in the two years to see if the rate decreased. The problem here
is that the two years could be very different in ways other than just having
or not having the vaccine. For example, look at Figure 10.1. If the control
year had been 1946, and the treatment year 1947, then it would look as if
the vaccine were quite effective even if it were worthless. If the control year
ⴱ
A vaccine is a substance administered to a person that causes the body to believe it has
contracted the virus. The body reacts by developing antibodies in the blood, which are
specifically targeted to kill that virus. These antibodies remain in the body, so that if later the
person does contract the virus, the antibodies will kill it before it can cause harm. Vaccines
can be made of small amounts of the live virus, or of killed virus. The danger in live virus is
that the vaccine may cause the person to contract the disease before enough antibodies can
be aroused. The danger in killed virus is that it may not be able to fool the body into creating
the antibodies.
†
P. Meier, “The Biggest Public Health Experiment Ever: The 1954 Field Trial of the Salk
Poliomyelitis Vaccine,” in Statistics: A Guide to the Unknown, ed. Tanur, Mosteller, Kruskal,
Link, Pieters, and Rising (San Francisco: Holden-Day, 1978), pp. 2–13.
60,000
Number of cases
45,000
30,000
15,000
0
Figure 10.1
1930 32
34
36
38
40
42
44
46
48
50
52
54
56
Annual occurrences of polio in the United States (1956 value estimated).
were 1947, and the treatment year 1948, then even if the vaccine had cut the
polio rate in half, it would have appeared ineffective. This problem exists
because polio, like many other diseases, has an epidemic character and thus
can vary greatly in incidence from year to year.
Another possibility was to give the children in some regions of the
country the vaccine, leaving the rest of the country as the controls. The
problem with this experimental design is that regions are different in more
ways than just whether they have the vaccine. In particular, polio is prone
to regional epidemics.
If the vaccine worked, it was important to have evidence as strong as
possible that it worked, so that it could be widely used immediately to stop
polio cold. Ambiguous results could have delayed adoption for years. Thus,
as stressed at the beginning of this chapter, valid and believable evidence
was required! These observational approaches would not do.
NFIP Study: In 1938 President Roosevelt established the National Foundation for Infantile Paralysis (NFIP), now known as the March of Dimes Birth
Defects Foundation. (In honor of President Roosevelt and his association
with the March of Dimes, the United States placed Roosevelt on the dime
in 1946.) The NFIP planned a massive study that would deal with possible
year effects and region effects. At schools willing to participate, the second
graders would be the group getting the vaccine, and the first and third
graders would be the controls. There would be some differences between
the treatment and control groups, such as age and possibly more contagion
within certain grades than in others, but because the control ages bracketed
the treatment ages and everyone was in the same schools, those factors
were expected to be minimal. An ethical problem arose: People could not be
forced to take the vaccine. More precisely, children could not be given the
vaccine without their parents’ permission. For this reason the NFIP asked
parents of second-graders to volunteer their kids for the study. About 64%
did volunteer. Thus the NFIP plan:
䢇
䢇
Treatment group: Second-graders whose parents volunteered are given
the vaccine.
Control group: The first- and third-graders at the same schools are not
given the vaccine.
Although not randomized, this design for the statistical experiment is fairly
good, but there are still some important ways in which the treatment and
control groups differ.
The Volunteer Effect: The people in the treatment group receiving the
vaccine were all volunteers. The control group, by contrast, was made up of
people of whom some would have volunteered and some not. Volunteers
are somewhat different from nonvolunteers in ways that matter medically.
In this instance, they tended to be more well-to-do. But surprisingly, polio
has been shown to be more likely to strike the more affluent.ⴱ Thus this
design is somewhat biased against the vaccine because the treatment group,
being less affluent on average than the control group, has a greater likelihood
of polio for its members.
Change in behavior: In the NFIP study the parents knew whether their
child was vaccinated or not. This knowledge may have changed their
behavior. For example, if the parents knew their child was vaccinated, they
may have been more likely to let the child engage in more risky behavior,
like attending summer camp, where exposure to the virus may have been
more likely. This effect could also work against the vaccine.
Effect on Diagnosis: The doctors evaluating the children at the end of
the study to determine who contracted polio and who did not would know
ⴱ
One usually associates affluence with better health, but in the case of polio it works the other
way around. The explanation is that in less affluent areas the polio virus is more likely to be
present (because of poorer hygiene). Hence, while children are still young and protected by
antibodies transmitted at birth from their mothers, they are likely to contract polio when it
is relatively harmless, at which time they will develop their own polio antibodies to protect
them from contracting the disease when they are older, when it is more harmful.
who got the vaccine. Depending on the attitude of the doctor toward the
effectiveness of the vaccine, this knowledge may sway the doctor’s diagnosis
in borderline cases, perhaps with the doctor being totally unaware of being
so influenced.
Randomized Control Study: Some of the health departments that would
be involved in the study objected to the NFIP plan for the above reasons. It
was too important to obtain a clear conclusion to allow these effects to muddy
the waters. Mindful of the ethical problems but realizing that randomization
was essential, a second design was proposed with the following features:
1. Randomized controls
2. Placebo
3. Double-blind protocol
These features are aimed at preventing confounding third-factor differences
from arising between the treatment and control groups. What are these
features? How are they used? The proposal was to take a large group
of children and randomly assign about half to the vaccine and half to
the control. Randomly has a specific meaning: an objective probabilistic
mechanism assigns people to the groups. Conceptually, one places the
names of all the children in the study in a box and randomly draws half
of the names without replacement. The children whose names are drawn
receive the treatment (vaccine), and the ones whose names remain in the
box receive the controls. In practice, people use tables of random numbers
or let a computer do the randomization. The important notion is that the
subjects themselves do not decide (nor do their families or doctors) whether
to take the vaccine. Moreover, the subjective judgments of the researchers
have no effect on who receives the vaccine. In order to effect this design
for this study, people were asked to volunteer their children, and only
those volunteered children were randomly assigned to the two groups. The
parents were explicitly told that their volunteered child may or may not
receive the vaccine. Notice that this approach eliminates the volunteer effect:
both groups contained all volunteers, so on that count they were the same.
Randomization will also tend to even out other important variables, such as
overall health, age, sex, and level of affluence. Thus we simply do not have
to be concerned about the confounding influence of other health-related
variables.
To eliminate differences in behavior based on knowing whether one
received the vaccine, it was important that neither the children nor their
parents knew who was in which group. There is no way a child would
not notice being given an injection, so the plan was to give everyone in the
study a seemingly identical injection. The treatment group had the vaccine
in their injections, while the control group had plain saltwater. The saltwater
injection is an example of a placebo. A placebo is an inert (that is, having no
active medicinal ingredients) treatment that, to the recipient, looks and feels
the same as the real treatment (see the section “Those Amazing Placebos,”
below). The placebo produces a single-blind study: that is, the subjects are
blind to whether they received the treatment or control. Thus, for example,
the risk-taking behavior and the psychological state of the two groups
should be the same.
Finally, the doctors making the diagnoses of the children are not told
which children received the vaccine, and which the placebo. Hence the
single-blind study becomes a double-blind study, the second blindness
being that of the evaluators.
Thus we have the plan for the randomized control study:
䢇
Treatment group: Vaccine is given to a randomly selected half of the
volunteers.
䢇
Control group: Placebo is given to the other half of the volunteers.
The two plans were implemented, each in about the same number of
schools. The results of the studies were announced on April 12, 1955, the
10th anniversary of President Roosevelt’s death. Some of the data are in
Table 10.2. The experiment was a success! The randomized control study
showed that the polio rate among the nonvaccinated children was about 2.5
times greater than for the vaccinated ones. The NFIP study also showed that
the vaccine group did better, but not by quite as wide a margin. It appears as
if the volunteer effect, and possibly the other effects, did lessen the vaccine’s
Table 10.2
Studies
Results of NFIP and Randomized Control Polio
Number of
subjects
Number with
polio
Polio rate
per 100,000
Vaccinated
(2nd-grade
volunteers)
221,998
56
25
Control
(1st- and 3rd-graders)
725,173
391
54
Vaccinated
(volunteers)
200,745
57
28
Control
(volunteers)
201,229
142
71
NFIP study
Randomized control study
apparent effectiveness. More important, the randomized control study was
not subject to the criticisms aimed at the NFIP study, and thus its validity
and believability were much greater.
There are some other indications of the volunteer effect at work. Compare
the two studies. Notice that the polio rates in the vaccinated groups of the
two studies were very close (25 versus 28), but the polio rates in the control
groups were quite a bit different: 54 in the NFIP study compared to 71 in
the randomized control study. Neither control group received the vaccine.
The difference is likely due to the volunteer effect: the controls in the NFIP
design consisted of everyone, while those in the randomized control study
were all volunteers. Volunteers are on average more affluent and hence
more likely to contract polio.
The goal of finding convincing evidence that the vaccine worked was
achieved, so that in the subsequent year the vaccine was widely disseminated throughout the nation. Unfortunately, the dissemination was
abruptly cut short when a bad batch of the vaccine led to 79 children
contracting polio. Later the vaccine was again given out, but it was not
until new and improved vaccines were developed a few years later that
widespread use of the vaccine eventually led to the virtual eradication of
polio from the United States. As students of statistics, note the vital role
that a randomized experimental statistical study played in this nationally
important public health issue.
Back to Smoking
The studies showing association between smoking and lung cancer are
not as convincing as the randomized control Salk vaccine study. Why not
execute a similar study for smoking? Imagine such a study. A group of
people is identified (volunteers?) to be in the study, say, all about 18 years
old. Half are randomly assigned to smoke for the rest of their lives; half are
randomly assigned to not smoke. But they should not know which group
they are in, so we need a placebo. We create placebo cigarettes that look
(and taste) just like regular cigarettes, and have the same detectable effect
on people. (If they did not, then it would not be long before the subjects
would figure out which group they are in.) But these placebo cigarettes
cannot have the cancer-causing ingredients of regular cigarettes. (Do we
know really what those are?) These people would have to be followed and
supplied with the correct type of cigarettes until they die or, say, for 40 years.
Then the researchers would analyze who died of lung cancer and who did
not. Then, because of the randomization and the use of a placebo, it would
indeed be scientifically safe to conclude that smoking is a major cause of
lung cancer if in fact the rate of lung cancer was much higher among the
regular cigarette smokers than among the placebo (nontobacco) smokers.
Thus, if carried out, this study would provide powerful evidence one way
or the other. Unfortunately, for numerous reasons both practical and ethical,
such a study is impossible to carry out. Medical studies involving cigarette
smoking must be observational.
The collective force of many observational studies can be convincing.
The surgeon general appointed a committee in 1962 to review the scientific,
medical, and statistical evidence of the health effects of smoking and to
arrive at a summary conclusion. The committee surveyed a large number of
various kinds of studies, and in 1964 it issued a comprehensive report flatly
stating that “cigarette smoking is causally related to lung cancer in men; the
magnitude of the effect of smoking far outweighs all other factors.”ⴱ Taking
the studies as a whole, the idea that the relationship between smoking and
lung cancer is causative is convincing because of a number of considerations:
1. The association appears in many types of studies using many types of
2.
3.
4.
5.
subjects.
The association is so strong (the mortality due to lung cancer among
smokers is 10 to 20 times that among nonsmokers) that it is unlikely that
the effect could be totally explained by other factors.
There is a time sequence in which the suspected cause (smoking) appears before the effect (lung cancer.)
There is a plausible scientific explanation for the causation: tobacco
smoke contains substances that are known to cause cancer in animals
(established by doing randomized controlled experiments!).
Increased levels of smoking are associated with increased rates of lung
cancer. Studies have shown that people are more likely to contract lung
cancer the more they smoke per day, the earlier they start smoking, and
the more they inhale.
Thus the overwhelming and multifaceted evidence allows us to bypass the
need for the usually required randomized experiment with treatment and
control groups and yet still conclude causality. Such successful bypassing
of randomized experimentation is rather unusual, and this instance of it
required an enormous investment of scientific and medical resources.
As an interesting aside, recall that R. A. Fisher made the case that
according to some of the Doll and Hill data, noninhalers seemed more likely
to contract lung cancer than inhalers. This is an example of an association
that did not hold up under more careful examination. In fact, even Doll
and Hill had additional data that showed the difference was minimal. By
ⴱ
Smoking and Health: Report of the Advisory Committee to the Surgeon General of the Public
Health Service, U.S. Department of Health, Education, and Welfare, Public Health Service
publication no. 1103, 1964, p. 106.
contrast, the surgeon general’s report cited a large body of evidence that
showed that the more people inhaled, the higher the rate of lung cancer.
It seems likely that some subtle third-variable influence was present in the
earlier Doll and Hill data or, as happens occasionally in statistical studies,
the statistical gods of chance have conspired to fool us by capitalizing on the
natural randomness that produced the data to produce an unlikely result
(like 9 heads in 10 tosses of a fair coin).
The case against smoking was strong, but there were still doubters, in
particular the tobacco companies. (As a point of scientific logic, there could be
other third factors no one had considered, for example.) In 1979 the surgeon
general produced another (heftier) report, even more comprehensive and
even more damning for smoking.ⴱ Still, it was not until January 1998 that
even the tobacco companies had to admit, “We recognize that there is a
substantial body of evidence which supports the judgment that cigarette
smoking plays a causal role in the development of lung cancer and other
diseases in smokers.Ӡ
After so many years of statistical research, at least no one can argue with
the pundit who said, “It is now proved beyond doubt that smoking is one
of the leading causes of statistics.”
Observational studies are valuable, but it takes many more of them,
and a wide variety of types, to collect evidence equal to good randomized
control studies.
Those Amazing Placebos
An angry crowd has gathered outside the Hibbert Medical Clinic:
Crowd: We need a cure! We need a cure!
Hibbert: Ho ho ho. Why, the only cure is bedrest. Anything I
give you would be a placebo.
Woman: [frantic] Where can we get these placebos?‡
The notion that an inert substance can heal because the patient believes
it can is an old one. Such inert substances were dubbed placebos, from the
Latin “I shall please,” because a healer, lacking a truly curative substance,
would prescribe a placebo to make the patient happy. Over the years
ⴱ
Smoking and Health: A Report of the Surgeon General, U.S. Department of Health, Education,
and Welfare publication no. (PHS) 79-50066, 1979.
†
Statement before the U.S. House of Representatives Commerce Committee, January 19,
1998, by Geoffrey C. Bible, chairman and chief executive officer of Philip Morris Companies,
Inc.
‡
From the “Marge in Chains” episode of the television show The Simpsons, The Simpson
Archives, www.snpp.com.
placebos have included “usnea (moss from the skull of victims of violent
death), Gascoyne’s powder (bezoar, amber, pearls, crabs’ eyes, coral, and
black tops of crabs’ claws), triangular Wormian bone from the juncture
of the sagittal and lambdoid sutures of the skull of an executed criminal,
. . . wood lice, human placenta and perspiration,”ⴱ and many other strange
and unpleasant substances. The placebos worked in the sense that people
would often feel better after having taken them, even though now we
know these “medicines” were basically useless. This placebo effect is
real; that is, true healing can occur simply by believing the placebo will
work. Indeed psychologists and medical researchers study the influence
of mental functioning on the immune system and on other physiological
characteristics related to disease.
When a new drug or medical procedure is introduced, people often find
a high cure rate. For example, one treatment for angina (suffocating chest
pains associated with heart disease) was to tie off the mammary artery.
Two studies without any controls reported 68% and 91% of the patients
improving from the surgery. These rates seem impressively high, and this
procedure subsequently became quite popular in the years 1955–1960.
Popularity plummeted after two more experiments with controls showed
67% improvement in the treatment group, but 71% in the control group. The
controls received a placebo in the form of a skin incision that did not affect
the artery.† It appears as though the improvements people felt were based
on the placebo effect. The actual surgery did not provide the relief. Without
carefully controlled experiments comparing the treatment to the placebo,
surgeons would likely have continued performing a dangerous but useless
procedure.
Chemonucleolysis is a treatment for alleviating the pain of a slipped
disc in the spine. It involves injecting an enzyme directly into the disc.‡
Between 1963 and 1975, almost 17,000 people had this treatment, with the
studies reporting success rates from 50% to 80%. These studies did not use
controls. Before the procedure was finally approved for use in the United
States, it needed to be evaluated with a controlled experiment. In 1975
a double-blind controlled experiment using 106 patients was performed
in which the placebo was an injection without the active enzyme. The
treatment group had 60% successes, and the placebo control group had 50%
successes. Thus the treatment appears to be a little better than the placebo.
ⴱ
A. K. Shapiro and E. Shapiro, “The Placebo: Is It Much Ado about Nothing?” in The Placebo
Effect, ed. A. Harrington (Cambridge, Mass.: Harvard University Press, 1997), pp. 13–14.
†
‡
Ibid.
R. L. Sanford, “The Wonders of Placebo,” in Statistics in the Pharmaceutical Industry, ed. C.
R. Buncher and J.-Y. Tsay (New York: Marcel Dekker, 1994)
The difference of 10% turned out not to be statistically significant (that is, the
statistical evidence was weak; see Section 10.4), and the treatment failed to
be approved. A later study, again without controls, was reported in 1977 to
have a 70% success rate with the procedure. Still, no approval for its use. Two
more double-blind placebo-control studies were conducted. Combining the
three placebo-control studies, the success rate for the chemonucleolysis
treatment was 70%, and for that the placebo was 47%, enough to produce
strong statistical evidence in favor of the treatment. Finally, the treatment
was approved. This appears to be a case in which the treatment really
is effective, but again the message is clear: we only found this out by
doing careful randomized double-blind treatment-versus-control statistical
experiments.
Notice that the uncontrolled studies for both the angina surgery and
the chemonucleolysis injection showed success rates of 50% to 90%. The
angina surgery actually did slightly worse than the placebo, suggesting
that the surgery was useless at best and possibly even slightly harmful.
For the disc treatment, the placebo still did well, but not quite as well
as chemonucleolysis. Chemonucleolysis was only about 23% better than a
placebo. Both examples suggest that there is a strong placebo effect (and
also that people just had a high recovery rate in the absence of even a
placebo). Moreover, by not comparing the treatment to a placebo, one
obtains over-optimistic, or even wrong, impressions of the effectiveness of
the drug.
Lessons
In the introduction to this chapter, we mentioned three benefits to running
a well-designed experiment. The examples in this section illustrate the first
two: validity and believability.
䢇
Salk Polio Vaccine. Validity: Both studies gave evidence that the vaccine
was effective, but the randomized control study was more scientifically
valid than the NFIP study. Believability: The medical profession, and
indeed the entire country, immediately accepted the effectiveness of the
vaccine.
䢇
Smoking studies. Validity: The Key Problem illustrates the dangers of
taking observational studies at face value; any such single observational
study has little validity in arguing that smoking causes lung cancer.
Believability: Because randomized control studies were not feasible,
many and varied observational studies over decades were necessary
to achieve common acceptance (believability) of the harmful effects of
smoking.
䢇
䢇
Surgery for angina. Validity: Uncontrolled studies suggested that this
surgery was effective, but the randomized controlled studies showed
that the effectiveness could be accounted for by the placebo effect.
Hence, there is no statistical validity argument for making the surgery
the procedure of choice!
Chemonucleolysis. Validity: Uncontrolled studies showed roughly a
70% success rate for this procedure, while randomized control studies
eventually showed that the placebo success rate was about 47%, leaving
23% due to the treatment, producing a validity argument supporting
the procedure. Believability: If more controlled experiments instead of
just observational studies had been run early on (1964), then approval
for use of the procedure would likely not have taken until 1982. Poor
planning delayed approval for 18 years, denying unknown numbers of
people an effective treatment.
SECTION 10.2 EXERCISES
1. The following is a quote about a recent study
on breast cancer. Use this paragraph to answer
the following questions below.
The cancer institute’s study involved
13,338 women in the United States and
Canada, making it one of the largest
cancer prevention studies ever. Some
women were given tamoxifen, others
placebos. For those given tamoxifen
over a five-year period, one in 236 developed breast cancer. The placebo breast
cancer rate was one in 130 women. There
were significant reductions in the occurrence of both invasive and non-invasive
breast cancers in every age group, from
35–45 to the over-60 group. (Source:
http://cnn.com)
a. Identify the treatment group and the control group in this study.
b. How were placebos used in this study?
Do you think it was necessary to use
them?
c. Name three additional factors that could
influence the outcomes of this study.
d. Write down one question you would like
to ask the researchers about their study.
2. The following is a quote about a recent study
on the effect of the vitamin folic acid on heart
disease and homocysteine levels in the blood.
Use this paragraph to answer the questions
below.
In the study, researchers fed breakfast
cereal daily to 75 men and women with
heart disease at the Providence St. Vincent Medical Center in Portland, Oregon. They found that the more folic
acid there was in their cereals, the
more their blood homocysteine levels
declined. They also found that while
cereals with the standard level of fortification had little effect, adding nearly
five times as much, or a total of 665
micrograms of folic acid, cut homocysteine levels by 14 percent. (Source:
http://cnn.com)
a. Identify the treatment group and the control group in this study.
b. From the information in the paragraph, do
you think that placebos were used in the
study? Explain your answer.
c. Name three additional factors that could
influence the outcome of this study.
d. There was a large amount of press coverage
on this study, and several doctors on television were recommending increasing one’s
folic acid intake to at least 400 micrograms.
Do you think these recommendations were
reasonable given the information from the
study?
3. This following is a quote about a recent study
on the effect of smoking on the heart attack
rate of men and women. Use this paragraph
to answer the questions below.
Women who smoke have a 50 percent
higher risk of having a heart attack
than male smokers, according to a report in the British Medical Journal. Dr.
Eva Prescott and colleagues at the Institute of Preventive Medicine in Copenhagen concluded that women may be
more sensitive to the harmful effects of
cigarettes because of an interaction between components of tobacco and hormones. “There is growing epidemiological evidence that women who smoke are
relatively deficient in estrogen,” Prescott
said. Doctors have known that estrogen
deficiency is associated with cardiovascular disease, and women’s risk of having a heart attack increases after the
menopause when estrogen levels fall.
Studies of hormone replacement therapy have shown it lowers the chance of
suffering a heart attack.
The researchers studied 25,000 men
and women over a 12-year period
and compared the risk of heart attack
among smokers and non-smokers. The
women’s 50 percent increased risk did
not depend on age and was not influenced by high blood pressure, cholesterol, height, weight, exercise or alcohol
consumption. (Source: http://cnn.com)
a. Identify the treatment group and the control group in this study.
b. Is this study an observational study or a
planned experiment?
c. Could one use placebos in this study? Why
or why not?
d. Name three additional factors that could
influence the outcomes of this study.
e. Write down one question you would like
to ask the researchers about their study.
4. How would you conduct an observational
study to determine the effects of alcohol consumption on heart disease? Consider the following questions: How do you choose the
control group and the treatment group? What
information should you get from the subjects?
5. Now design a planned experiment to determine the effects of alcohol consumption on
heart disease. In the design of your experiment, consider the following questions: How
do you choose the control group and the treatment group? How do you choose these groups
to minimize the volunteer effect? Would a
placebo be necessary for this kind of experiment? If so, what would you use as a placebo
for this experiment? What limitations or obstacles would you face with this experiment?
For additional exercises, see page 728.
10.3 SAMPLING FROM A POPULATION
It is a good idea to review the Chapter 6 material on populations and
samples before beginning this section.
Preelection Polls
Let us go back to 1936, to the presidential election between Franklin
Roosevelt (the Democrat) and Alf Landon (the Republican). The Literary