American Journal of Epidemiology
Copyright © 1998 by The Johns Hopkins University School of Hygiene and Public Health
All rights reserved
Vol. 148, No. 10
Printed in U.S.A.
Bias in the Standardized Mortality Ratio when Using General Population
Rates to Estimate Expected Number of Deaths
Michael E. Jones and Anthony J. Swerdlow
Cohort studies often compare the observed number of cases arising in a group under investigation with the
number expected to occur on the basis of general population rates. The general population is taken to
represent unexposed persons, but it is almost inevitably biased in that it comprises all types of people
including exposed ones. To identify circumstances when this bias matters, the authors modeled its effect in
relation to the size of the observed standardized mortality ratio (SMR) and the prevalence of exposed
individuals in the general population. The authors found that bias may be a major problem, causing substantial
underestimation of the true relative risk, when either the prevalence of exposure in the general population or
the SMR are large. The bias can cause an apparent trend in SMRs with age when none exists. It also places
a limit on the maximum size of the observed SMR, no matter how large the true relative risk. A table is provided
showing the extent of bias in different circumstances. Cohort studies of people with common diseases or
exposures, or that find large SMRs, when using general population expectations, need to consider the extent
of bias from this source. Am J Epidemiol 1998;148:1012-17.
bias (epidemiology); mortality; standardized mortality ratio
Many cohort studies compare the observed number
of cases arising in a cohort under investigation with
the number expected to occur on the basis of general
population rates. The investigator usually assumes that
the disease rate in the general population reflects the
rate that would occur in unexposed persons. Virtually
all general populations, however, contain exposed individuals, if only because the cohort live within the
population; it is usually not possible to determine rates
for general populations excluding all exposed persons,
because the latter are not separately identifiable. There
is thus a bias in such studies (1), but its extent and the
circumstances where it is of importance or trivial do
not seem to have been investigated, and reports of
cohort studies using general population expectations
have generally not considered the problem.
We were initially concerned about this bias when
analyzing renal disease mortality in a cohort study of
patients with diabetes mellitus (2), since an appreciable proportion of renal disease in the general population is due to diabetes and there is no simple way to
derive renal disease mortality rates for the general
population excluding all people with diabetes. The
Received for publication August 13, 1997, and accepted for
publication April 10, 1998.
Abbreviations: CMF, comparative mortality factor; IDDM, insulindependent diabetes mellitus; RR, relative risk (rate ratio); SMR,
standardized mortality ratio.
From the Epidemiological Monitoring Unit, Department of Epidemiology and Population Health, London School of Hygiene and
Tropical Medicine, University of London, London, United Kingdom.
problem, however, is more general, applying to other
cohorts of persons with relatively common diseases or
congenital conditions, and to cohorts defined, for example, by occupation, social class, or ethnic group, or
a particular diet or use of a common medication, or
exposure to a prevalent but not ubiquitous pollutant.
In this paper, we model the bias in the SMR and
derive a mathematical expression for its size. We use
the mathematical expression to investigate the extent
to which the bias occurs in different circumstances,
and hence when it is of practical importance.
METHODS
The bias described above occurs in cohort studies
when the group being followed are considered to be
exposed to the risk factor under investigation and
general population rates are taken to represent the rate
that would occur in an unexposed population. The
extent of the bias depends on the prevalence of the
exposure in the general population and the size of
the true relative risk (rate ratio) in exposed compared
with unexposed persons.
To estimate the true relative risk, let us first consider
the situation where both exposed and unexposed
groups are within the same cohort (i.e., using an internal comparison group). With a small extension to
conventional notation (3), let Xei and Au, be the death
rates in the exposed (e) and unexposed (u) population
in stratum i (where, for example, the exposed popula-
1012
Bias in the Standardized Mortality Ratio
tion may be patients with diabetes and stratification is
on age, sex, and period). The relative risk (rate ratio
(RR)) in stratum i is given by
RR; = n.e
1013
Substituting for Ag,- gives:
SMR = { £ A,,, X nJ/CZ[pe,i X K,t + (1 - Pe,,)
X A J X nj.
j/\uj.
Let nei and nui be the person-years of follow-up in the
exposed and unexposed parts of the cohort in stratum
/. The expected number of deaths in stratum i, Eh is
given by
Substituting RR X Au, for A^,, and p for pei gives:
SMR = { £ RR X \ui X nj/{2[p
X RR X Akil-
+ (l-p)XAM]X«ei,
= RR X {2 ABi, X nj/{[p
Ei = K,i X ne,h
X RR + (1 - p)}
the observed number of deaths, D,, is given by
The 2AU, X nei term in the numerator and denominator cancel out, giving:
A = K,i x ne,t,
and, as usual,
SMR = RR/[p X RR + (1 - p)],
SMR = 2 D / I E, = {2 A,,, X nJ/{J, ABil- X nj,
where 2 is the summation over all i strata. Assume
RR, is constant over strata (i.e., RR, = RR for all i)
and thus,
Ae i ~ ivK. X A u (-.
Hence, by substitution it can be shown that:
SMR = RR,
(1)
which is to say that the SMR correctly estimates the
true relative risk when the expected number of deaths
is estimated from the rate in the unexposed group.
If the expected number of deaths is estimated from
general population rates, a bias is introduced into the
SMR. Let \g i be the rate in the general population (g)
in stratum i. The expected number of deaths in stratum
i, £* is given by Ef = \gi X nei, and the expression
for the observed number of deaths remains the same,
A = \e,i X "„,,-•
The general population rate, A „ is, however, an average of the rate among the exposed and unexposed
parts of the general population. If pei is the prevalence
of exposure in the general population in stratum i, and
so (1 — pe,) is the prevalence of not being exposed in
stratum i, then
A*,, = Pe,i X K,i + (1 ~ Pe,,) X K.i-
Assume again that RR, is constant over strata i (i.e.,
RR, = RR for all i), and in addition pei is constant
over strata i (i.e., pei = p for all i), or that our analysis
is limited to those strata where this is so, then the usual
combination of observed deaths divided by expected
deaths gives the observed SMR:
SMR =
Am J Epidemiol
Ef= {2 K,i X nJ/{2 A,., X nj.
Vol. 148, No. 10, 1998
(2)
which is a biased estimate of the true relative risk (rate
ratio) when p > 0. The bias is always toward the null.
The above expression also implies that there is an
upper bound to the observed SMR, which tends to lip
as the true relative risk becomes very large.
Rearrangement of the formula for the observed
SMR gives an expression for the true relative risk, as
follows:
SMR X [p X RR + (1 - p)] = RR,
hence,
SMR X (1 - p) - RR - [SMR XpX
RR],
giving
RR = SMR X (1 - p)/[l ~{pX
SMR)].
(3)
In practice, the SMR is estimated from real data containing random error and the true relative risk given by
equation 3 is not strictly "true" because it too is subject
to this random error.
The size of the bias, expressed as percent of the
SMR, may be estimated as:
Bias = {(RR - SMR)/SMR} X 100.
(4)
If the same approach is taken with the comparative
mortality factor (CMF) (3), where w, are the weights
used in the direct method of standardization (e.g., the
world standard population), we get the same result
except that the observed SMR is replaced by the
observed CMF:
CMF = S(A,,,- X Wl-)/2(Ag>I. X w,)
= RR/[(1 -p)+PX
RR];
(5)
1014
Jones and Swerdlow
and
RR = CMF X (1 - p)/[l ~(pX
CMF)].
(6)
We used these derived expressions to produce a
table of true relative risks for various values of the
observed SMR against a range of values for the prevalence of exposure in the general population. The table
is equally applicable to the bias in the CMF. We also
estimated the true relative risk and the size of the bias
for some published studies where the bias may be of
importance.
RESULTS
Table 1 shows the estimated relative risk (rate ratio)
for a given observed SMR at different values of p, the
prevalence of exposure to the risk factor under study in
the general population, if general population rates are
used to estimate the expected number of cases. For
example, if the observed SMR was 5.0 and the prevalence of exposure in the general population was 5
percent, the true relative risk would be 9.0. The lower
right hand section of the table is blank because when
the relative risk is greater than unity there is an upper
bound to the observed SMR for any given prevalence
of exposure in the general population. For instance, if
20 percent of the general population are exposed in the
same way as the cohort under study, the SMR cannot
exceed 5.0 regardless of the size of the true relative
risk. The bias is not so pronounced when the SMR is
smaller than unity.
Figure 1 shows the degree of bias for a series of
SMRs in relation to the prevalence of exposure in the
general population. If the observed SMR is modestly
raised (e.g., 1.5), the bias will only be large if a
considerable proportion (e.g., ^ 2 0 percent) of the
general population is exposed. For large SMRs, however, even a low prevalence of exposure may cause a
large bias.
As an example, consider the cohort of patients with
type 2 diabetes mellitus diagnosed at a hospital in
Osaka Prefecture, Japan (4). The SMR for renal disease was 12.4 for males and 16.4 for females when
compared with the death rate in the general population
of Osaka Prefecture. Assuming the prevalence of diabetes in Osaka Prefecture was at least 4 percent (5),
the relative risk of renal disease mortality would be 90
percent larger for men and 180 percent larger for
women than the reported SMR.
The bias may be important even for rarer conditions.
Insulin-dependent diabetes mellitus (IDDM) had a cumulative incidence to age 14 years ranging from 0.14
percent to 0.28 percent in US white or predominantly
white populations during 1965 to 1988 (6). Among the
cohort of IDDM patients diagnosed before age 17
years at the Children's Hospital of Pittsburgh, the
SMRs for renal disease as a cause of death were 556
TABLE 1. True relative risks for a range of observed standardized mortality ratios (SMRs) based on general population
expectations, by prevalence of exposure in the general population
Observed
(biased)
SMR
Prevalence of exposure in general population (%)
0.01
0.10
1.00
2.50
5.00
7.50
10.00
15.00
20.00
25.00
30.00
0.25
0.5
1.0
0.25
0.50
1.00
0.25
0.50
1.00
0.25
0.50
1.00
0.25
0.49
1.00
0.24
0.49
1.00
0.24
0.48
1.00
0.23
0.47
1.00
0.22
0.46
1.00
0.21
0.44
1.00
0.20
0.43
1.00
0.19
0.41
1.00
1.5
2.0
2.5
1.50
2.00
2.50
1.50
2.00
2.50
1.51
2.02
2.54
1.52
2.05
2.60
1.54
2.11
2.71
1.56
2.18
2.85
1.59
2.25
3.00
1.65
2.43
3.40
1.71
2.67
4.00
1.80
3.00
5.00
1.91
3.50
7.00
3.0
4.0
5.0
3.00
4.00
5.00
3.01
4.01
5.02
3.06
4.13
5.21
3.16
4.33
5.57
3.35
4.75
6.33
3.58
5.29
7.40
3.86
6.00
9.00
4.64
8.50
17.00
6.00
16.00
9.00
21.00
7.5
7.50
10.01
25.06
7.55
10.09
25.62
8.03
11.00
33.00
9.00
13.00
65.00
11.40
19.00
15.86
37.00
27.00
10
25
50
50.25
52.58
99.00
t
75
75.56
81.00
297.0
100
101.0
111.0*
* Bold italic: true relative risk is at least 10% different from the observed SMR.
t Lower right hand section of the table is blank because the bias causes an upper bound to the observed SMR for relative risks greater
than unity.
Am J Epidemiol
Vol. 148, No. 10, 1998
Bias in the Standardized Mortality Ratio
1015
100
1
•o
S
75
-•
50
-•-,
?
5 -D
II
25 --,
.55 5
CO
•|
£
0
I
-25
0
5
10
15
20
25
Prevalence of exposure in general population (%
FIGURE 1. Bias in observed standardized mortality ratio (SMR) when using general population rates to estimate the expected number of
deaths.
in white males and 561 in white females, when compared with rates in the US white population (7). If the
prevalence of IDDM was 0.14 percent during the
period under study, the actual relative risk for renal
disease death would be at least 4.5 times larger, around
2,500. Even if the prevalence was as low as 0.10
percent, the actual relative risk would be at least twice
the observed SMR.
Studies of morbidity can also be affected by this
bias. Cryptorchidism affects about 1.5 percent of boys
(8) and is a strong risk factor for testicular cancer (9).
It has not been feasible to provide a comparison group
by following a large cohort of boys who never suffered
from cryptorchidism, and cohort studies of cryptorchid
boys have compared their incidence of testicular cancer with incidence rates from population-based cancer
registries. In the largest such cohort study, the standardized incidence ratio for testicular cancer was 7.5
(9), which, given the prevalence noted above, must
have underestimated the true relative risk by about 10
percent.
The bias can also occur in cohorts defined by occupation, social class, or ethnic group. In England and
Wales, 6.9 percent of males aged 15-64 years were
classified as social class V (unskilled occupations) in
the 1970-1972 national analysis of occupational mortality (10). The published SMR for homicide was 3.4
Am J Epidemiol Vol. 148, No. 10, 1998
in this group, compared with the rate in the male
population of England and Wales. The true relative
risk would have been 4.1 if social class V had been
compared with the remainder of the male population,
an increase of 22 percent on the observed SMR. The
SMR for rheumatic heart disease among Australian
aborigines in the Northern Territory of Australia was
reported to be 39, when compared with national Australian death rates (11). Northern Territory aborigines
make up about 0.24 percent of the total Australian
population (12), so the true SMR would be 10 percent
larger if they were excluded from expectations. If, as
seems likely, the raised risk of rheumatic heart disease
is true for Australian aboriginal people overall (1.6
percent of the total Australian population (12)), rather
than just those who live in the Northern Territory, the
estimated SMR would be 162 percent larger.
The results presented so far assume that the relative
risk is constant across strata by which the SMR has
been standardized (e.g., by age, sex, and calendar
period). This is a usual assumption when calculating
SMRs (3), but if the relative risk is heterogeneous
across strata, it is arguable whether standardized summary measures such as SMRs should be used at all.
The prevalence of exposure in the general population
is also assumed to be constant across strata, but for
many chronic diseases and several other exposures this
1016
Jones and Swerdlow
latter assumption is not even approximately true. For
numerous chronic diseases, for instance, prevalence is
far greater at older than at younger ages. This changing prevalence by age (or sex or other stratification
variable) can then lead to an artefactual trend in the
SMR by age (or sex, etc.) as the extent of the bias
alters from slight, at ages when prevalence is low, to
great at ages when the prevalence is much higher. An
example is shown in table 2. The prevalence of noninsulin-dependent diabetes mellitus in US nonHispanic whites (13) varies from under 0.1 percent at
ages 20-24 years to over 20 percent at ages 70-74
years. If the true relative risk for a particular cause of
death in patients with diabetes was constant by age at
6.0, then, as shown in table 2, the observed SMR
would appear to decrease by more than 50 percent
with increasing age simply because of the bias. Cohort
studies of mortality among diabetic patients do indeed
show all-cause SMRs decreasing with increasing age
after age 30 years (2, 14-17), and although the decreases are generally larger than can be attributed to
this bias alone, the bias must account for a proportion
of the downward trend.
DISCUSSION
We have shown that there is a potential bias in the
standardized mortality or morbidity ratio when general
population rates are used to estimate the expected
number of deaths in a cohort. The same bias can affect
the comparison of directly standardized rates (CMFs),
because the only difference between indirect standard-
TABLE 2. Observed standardized mortality ratios (SMRs) with
attained age when the prevalence of exposure in the general
population increases with age and general population rates
are used to estimate the expected number of deaths:
example based on a hypothetical cohort study of mortality
among patients with diabetes mellitus with a true relative risk
of 6.0 at all ages
Age
group
(years)
Prevalence of diabetes
mellitus in US nonHispanic white
males (%) (13)
True relative risk of
death in persons
with diabetes
mellitus
Observed
SMR
20-24
25-29
30-34
35-39
40-44
45-49
50-54
55-59
60-64
65-69
70-74
<0.1
0.9
1.4
0.7
2.6
5.7
9.7
5.7
13.3
17.2
20.8
6
6
6
6
6
6
6
6
6
6
6
6.0
5.7
5.6
5.8
5.3
4.7
4.0
4.7
3.6
3.2
2.9
ization (SMRs) and direct standardization (CMFs) is
the set of weights used as the standard (18).
As we have shown, the bias is conservative, i.e.,
toward a relative risk of unity, and consequently the
observed number of deaths will appear less extreme
than it should when compared with the calculated
expected number of deaths. The bias will therefore
make p values less significant, because the p value is
usually calculated by estimating the probability that
the observed number of deaths, or a more extreme
number, could have arisen from a Poisson distribution
characterized by the expected number of deaths (3).
The ideal solution to this bias would be to follow up
an unexposed group of individuals, in addition to those
exposed. This solution is often impracticable because
of the cost of ascertaining and monitoring a sufficiently large unexposed group for an adequate length
of time, or the difficulty in obtaining an unexposed but
otherwise comparable group (for instance, for an occupational cohort). Alternatively, general population
rates could be modified by subtracting the deaths and
person-years experienced by the exposed cohort, if the
study cohort itself constituted all exposed persons in
the general population. Another possibility, if the
prevalence of exposure was known exactly, would be
to estimate the true relative risk from equation 3, then
use this to calculate the unbiased expected number of
deaths, the p value, and confidence interval.
Generally, the prevalence is not known precisely
and no exact correction for the bias can be made; the
latter approach suggested above can be regarded as a
qualitative assessment of the effect of the bias rather
than a definitive adjustment for it. Our results, however, show that where the effect of contamination by
exposed individuals is small and the SMR is modest,
the bias may be ignored. Where the SMR is large or
the exposure is not rare in the general population, and
especially where both apply, the bias can be appreciable. Furthermore, we have shown that the bias gives an
artefactual upper limit to the observed SMR in studies
using general population expectations, the magnitude
of this limit depending on the general population prevalence of the exposure. Thus, where prevalence is
high, large SMRs will not be observed, no matter how
large the true relative risk. Indeed, there may be occasions when the bias is so strong that it is inappropriate to calculate SMRs. The bias will also lead to
apparent trends in the SMR by age or other such
factors if the prevalence of exposure in the general
population varies by these factors. Studies using general population expectations therefore need to consider
the impact of the bias on their results and conclusions,
especially if the prevalence of exposure changes with
age, sex, or other analytic variables.
Am J Epidemiol
Vol. 148, No. 10, 1998
Bias in the Standardized Mortality Ratio
ACKNOWLEDGMENTS
The Epidemiological Monitoring Unit is funded by
Medical Research Council (UK) programme grant no.
G8119454. Dr. Michael E. Jones is funded by Medical
Research Council (UK) project grant no. G9533916.
8.
9.
10.
REFERENCES
1. Measures of disease frequency and association. In: Hennekens
CH, Buring JE. Epidemiology in medicine. Boston, MA:
Little, Brown & Co, 1987:54-98.
2. Swerdlow AJ, Jones ME. Mortality during 25 years of follow-up of a cohort with diabetes. Int J Epidemiol 1996;25:
1250-61.
3. Rates and rate standardization. In: Breslow NE, Day NE.
Statistical methods in cancer research. Vol n. The design and
analysis of cohort studies. (IARC Scientific Publications no.
82). Lyon: International Agency for Research on Cancer,
1987:48-79.
4. Sasaki A, Horiuchi N, Hasegawa K, et al. Mortality and causes
of death in type 2 diabetic patients. A long-term follow-up
study in Osaka District, Japan. Diabetes Res Clin Pract 1989;
7:33-40.
5. Blackard WG, Yoshiaki O, Freedman LR. Epidemiology of
diabetes mellitus in Japan. J Chronic Dis 1965;18:415-27.
6. Karvonen M, Tuomilehto J, Libman I, et al. A review of the
recent epidemiological data on the worldwide incidence of
Type 1 (insulin-dependent) diabetes mellitus. Diabetologia
1993;36:883-92.
7. Dorman JS, LaPorte RE, Kuller LH, et al. The Pittsburgh
Am J Epidemiol
Vol. 148, No. 10, 1998
11.
12.
13.
14.
15.
16.
17.
18.
1017
insulin-dependent diabetes mellitus (IDDM) morbidity and
mortality study. Mortality results. Diabetes 1984;33:271-6.
John Radcliffe Hospital Cryptorchidism Study Group.
Cryptorchidism: a prospective study of 7500 consecutive male
births, 1984-8. Arch Dis Child 1992;67:892-9.
Swerdlow AJ, Higgins CD, Pike MC. Risk of testicular cancer
in cohort of boys with cryptorchidism. BMJ 1997;314:
1507-11.
Office of Population Censuses and Surveys. Occupational
mortality. London: HMSO, 1978:37-61. (The Registrar General's decennial supplement for England and Wales, 1970-72)
(OPCS Series DS, no. 1).
Cunningham J, Condon JR. Premature mortality in aboriginal
adults in the Northern Territory, 1979-1991. Med J Aust
1996;165:309-12.
Australian Bureau of Statistics. Indigenous population by
ATSIC region, 1994. Canberra: Commonwealth of Australia,
1997. (http://www.abs.gov.au/websitedbs).
King H, Rewers M, WHO ad hoc Diabetes Reporting Group.
Global estimates for prevalence of diabetes mellitus and impaired glucose tolerance in adults. Diabetes Care 1993;16:
157-77.
Entmacher PS, Root HF, Marks HH. Longevity of diabetic
patients in recent years. Diabetes 1964;13:373-7.
Michaelis D, Jutzi E. Trends in mortality rates in the diabetic
population of the GDR. Exp Clin Endocrinol 1990;95:83-90.
Fuller JH, Elford J, Goldblatt P, et al. Diabetes mortality: new
light on an underestimated public health problem. Diabetologia 1983;24:336-41.
Kessler II. Mortality experience of diabetic patients. A twentysix year follow-up study. Am J Med 1971;51:715-24.
Standardization of rates. In: Rothman KJ. Modern epidemiology. Boston, MA: Little, Brown & Co, 1986:41-9.