American Journal of Epidemiology
Copyright © 2002 by the Johns Hopkins Bloomberg School of Public Health
All rights reserved
Vol. 156, No. 8
Printed in U.S.A.
DOI: 10.1093/aje/kwf095
PRACTICE OF EPIDEMIOLOGY
Modeling Treatment Effects on Binary Outcomes with Grouped-Treatment Variables
and Individual Covariates
S. Claiborne Johnston1, Tanya Henneman2, Charles E. McCulloch3, and Mark van der Laan2
1
Department of Neurology, University of California, San Francisco, San Francisco, CA.
Division of Biostatistics, School of Public Health, University of California, Berkeley, Berkeley, CA.
3 Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, CA.
2
Received for publication May 2, 2001; accepted for publication May 24, 2002.
During evaluation of treatment effects in observational studies, confounding is a constant threat because it is
always possible that patients with a better prognosis, not adequately characterized by measured covariates, are
chosen for a specific therapy. Ecologic analyses may avoid confounding that would be present in analysis at the
individual level because variations in regional or hospital practice may be unrelated to prognosis. The authors
used simulated data with an excluded confounder to evaluate the reliability and limitations of the groupedtreatment approach, a method of incorporating an ecologic measure of treatment assignment into an individuallevel multivariable model, similar to the instrumental variable approach. Estimates based on the groupedtreatment approach were closer to the true value than those of standard individual-level multivariable analysis in
every simulation. Furthermore, confidence intervals based on the grouped-treatment approach achieved
approximately their nominal coverage, whereas those based on individual-level analyses did not. The groupedtreatment approach appears to be more reliable than standard individual-level analysis in situations where the
grouped-treatment variable is unassociated with the outcome except via the actual treatment assignment and
measured covariates.
confounding factors (epidemiology); ecology; instrumental variable; selection bias; simulation
Abbreviation: OR, odds ratio.
The validity of any observational study is limited by the
possibility that confounders are inadequately measured or
adjusted. This is a particularly difficult problem for observational studies of treatment effects because the decision to
treat is often based on prognostic factors. This form of bias,
termed confounding by indication, is a constant threat to
validity in observational studies (1–4).
Ecologic studies may bypass confounding at the individual
level. Even when confounding obscures a true underlying
benefit of a therapy at the individual level, grouping the unit
of analysis can produce more reliable results (5–7). For
example, hospitals that use a more effective treatment more
frequently should have better overall outcomes for patients
treated there compared with another hospital that does not
offer the treatment. Regardless of the pretreatment prognosis
of those selected for the more effective therapy, outcomes in
this group will be better than if the therapy were not available. Variability in hospital or regional practice provides a
form of pseudo randomization.
On the basis of these concepts, we previously compared
traditional surgical treatment of intracranial aneurysms with
a new, minimally invasive technique in a study involving 70
academic medical centers (8, 9). Recognizing that utilization
differences among hospitals were likely primarily due to
practice variability rather than differences in patient prognosis (5, 6), we used the proportion of patients treated by the
newer technique at a given hospital (the probability of
receiving the newer treatment among patients in the analysis
Reprint requests to Dr. S. Claiborne Johnston, Department of Neurology, Box 0114, University of California, San Francisco, 505 Parnassus
Ave., San Francisco, CA 94143-0114 (email: [email protected]).
753
Am J Epidemiol 2002;156:753–760
754 Johnston et al.
treated at the hospital) as the treatment variable and did not
include in the model the actual treatment each patient
received. Rather than discarding useful information by
aggregating all data by hospital, we constructed models at
the individual level: They predicted an individual’s probability of in-hospital death. Because demographic variables
could influence both the proportion of cases treated by the
new technique at a hospital and the outcome of cases, we
included these in the model. The final model was constructed
like a standard multivariable analysis except that the actual
treatment assignment for a patient was substituted by a
grouped-treatment variable: the proportion of cases treated
by the new technique at the hospital of treatment. More
generally, a grouped-treatment variable is defined as the
probability of an individual’s receiving the treatment of
interest at a specified level of aggregation. It can be calculated by determining the proportion of cases treated by the
therapy of interest at any specified unit of aggregation, such
as a ward, clinic, hospital, region, state, or country.
For the grouped-treatment approach to produce valid estimates of the treatment effect, several assumptions are
required. The most important assumption, which we term the
basic instrumental variable assumption, is that unmeasured
confounders do not produce an association between pretreatment prognosis and the grouped-treatment variable. In other
words, conditional on the value of the actual treatment and
the modeled covariates, the grouped-treatment variable and
the outcome are statistically independent. This assumption
may be difficult to satisfy. For example, hospitals that use
the new treatment for intracranial aneurysms may receive
higher risk cases by referral or may offer other technologies
and expertise that could contribute to outcome differences.
Thus, the grouped-treatment approach is also limited by the
possibility of unmeasured confounding.
The utility of aggregating treatment assignments in
avoiding confounding has been recognized for some time by
economists, who have termed the approach the instrumental
variable method (10, 11), but it has been used infrequently
by epidemiologists and health services researchers (12).
Economists and epidemiologists tend to construct multivariable models differently, and economists rarely study dichotomous outcomes (10, 13, 14), which are frequently of
interest in health studies. The grouped-treatment approach
attempts to preserve a model structure familiar to health
researchers. Several authors have described the mathematical basis for the grouped-treatment approach (15), most
frequently in the context of illustrating the impact of omitted
covariates (16–18). However, the potential utility of the
method as a means for evaluating treatment effects in the
presence of unmeasured confounders has not been the
subject of prior reports. Furthermore, the validity of the
method has not been illustrated with terms and data familiar
to epidemiologists. In this study, we subject the method to
testing with simulated data sets in order to test its validity
and determine its limitations.
MATERIALS AND METHODS
Marginal and conditional estimates
In a multivariable model with a dichotomous outcome,
omitting a covariate that is associated with the outcome but
not with the treatment assignment does not produce
confounding, but it does reduce the apparent effect size of
the treatment. Standard multivariable models that include all
covariates produce the conditional treatment effect, which is
the expected impact of treatment in an individual while
holding all other variables constant. If covariates that are
associated with outcome are excluded from the multivariable
model, the marginal treatment effect is produced—marginal
with respect to the excluded covariates. With a logistic
model, the magnitude of the marginal treatment effect is
smaller than that of the conditional treatment effect because
the association between treatment and outcome is weakened
when a covariate is missing from the model (19): Less variability in outcome is accounted for by the treatment because
more is due to a missing covariate. The logit curves based on
averaging the outcome probabilities are closely approximated by a different logistic curve with an attenuated slope
(20). However, if a covariate is not available to include in an
analysis, it may not be available to consider in making treatment decisions, so the overall impact of treatment in a population would be more accurately reflected by the marginal
treatment effect. Randomized trials produce an estimate of
the marginal treatment effect for the studied population
unless an adjusted analysis is reported. As in a randomized
trial, we assumed that all patients could receive either of the
treatments being compared and that treatment of one patient
did not influence the outcome of another, which can occur
when other management strategies or patient selection is
affected by outcomes of previously treated patients (21).
Also similar to randomized trials, the parameter estimated by
the grouped-treatment analysis is expected to be marginal
with respect to unmeasured confounders.
Simulations
For each set of conditions tested, 1,000 patient cohorts
with N individuals were simulated using S Plus (version
2000; Mathsoft Corporation, Seattle, Washington) software.
We modeled a dichotomous treatment Ti, meant to represent
a new therapy, received at hospital Hi, where i was unique
for each individual 1, 2, 3, …, N. Individuals were distributed evenly among 50 hospitals, defined by Hj = 1, 2, 3, …,
50, with varying utilization of the treatment, as defined
below. A confounder Ci and a covariate Bi were randomly
and independently assigned a value of 0 or 1 to each patient
with 50 percent probability. The confounder was related to
both outcome and treatment according to the odds ratios
(ORs) ORCO and ORCT, respectively. The covariate was not
related to treatment but was related to outcome by ORBO.
Outcome Oi was binary, representing dead (Oi = 1) or alive
(Oi = 0). The new therapy was modeled to be effective with
ORTO = 0.75; this small treatment effect was modeled so that
confounding would bias results to produce an apparent detrimental effect in some individual-level models.
Am J Epidemiol 2002;156:753–760
Grouped-Treatment Analysis 755
To model practice variability among hospitals, the probability of receiving treatment was dependent on the hospital.
For most simulations, treatment was assigned using the
following algorithm:
Logit [probability of treatment] =
(0.04)H + [ln(ORCT)]C – 0.9,
(1)
where ln is the natural logarithm; this algorithm was chosen
to ensure a large effect from confounding while allowing a
wide range of treatment probabilities across the different
institutions. Outcome was assigned to model the independent effects of the covariate B, the confounder C, and the
treatment assignment, as follows:
Logit [probability of outcome] =
[ln(ORBO)]B + [ln(ORCO)]C + [ln(ORTO)]T + 0.05.
(2)
These equations ensure that there is no nonrandom association between the confounder and the grouped-treatment variable except through the actual treatment assignment.
In a previous analysis, we used generalized estimating
equations to account for clustering of observations within
institutions (9). To simplify these simulations and their interpretations, we assume here that there is no clustering of
observations beyond that produced by treatment assignment.
Therefore, logistic regression can be used instead of generalized estimating equations.
To test the reliability of the approach with different distributions of treatment probability among the hospitals, we
generated additional cohorts: a uniform distribution,
modeling a wide distribution of treatment frequency (probability of treatment = (0.01)H + 0.2 × C), and a split distribution, modeling frequent utilization of the treatment at only
half the hospitals (probability of treatment determined by the
uniform distribution, ranging from 0 percent to 10 percent at
half the hospitals and from 35 percent to 60 percent at the
other half). These distributions were chosen to reflect the
type of variability in clinical practice that may be encountered in epidemiologic and health services research.
A total of 1,000 cohorts, corresponding to replicates, each
with a sample size of 5,000, were generated according to a
Bernoulli distribution for each set of conditions tested using
the probabilities given by equations 1 and 2. An estimate of
the treatment effect and its 95 percent confidence interval
(by the Wald method) were calculated for each cohort using
three different logistic models:
• Model 1 (standard, individual-level analysis with known
covariate and confounder):
Logit [probability of outcome] = β1 T + β2 B + β3 C + K,
where β1 represents the conditional treatment effect when
there are no unmeasured confounders. K is a constant.
• Model 2 (standard, individual-level analysis with known
covariate and unmeasured confounder):
Logit [probability of outcome] = β4 T + β5 B + K,
Am J Epidemiol 2002;156:753–760
where β4 represents a confounded estimate of the conditional
treatment effect.
• Model 3 (unmeasured confounder and grouped-treatment
variable (grouped-treatment estimator)). A grouped-treatment variable (GT) was determined as the proportion of
patients treated by the new therapy at a given hospital:
Logit [probability of outcome] = β6 GT + β7 B + K,
where β6 represents an unconfounded estimate of the treatment effect because there is no association between GT and
outcome independent of treatment assignment T. The magnitude of β6 is an estimate of the marginal treatment effect—β4
from model 2 with ORCT set to 1—because the condition in
which all patients are treated by the new procedure (GT = 1)
differs from the condition in which no patient is so treated
(GT = 0) by the coefficient β1. If the grouped-treatment
analysis is unbiased, then β6 should be equal to β4 from
model 2 with ORCT set to 1, which was estimated by simulation and is defined as the “true” value being estimated.
For each of the models, calculated coefficients from the
logistic models listed above and the limits of the Wald 95
percent confidence interval were averaged for the 1,000
replicates. These values were converted to odds ratios. The
confidence interval for each replicate was evaluated to test
inclusion of the true underlying treatment effect: The calculated confidence interval would be expected to include the
modeled marginal treatment effect (β4 from model 2 with
ORCT set to 1) in 95 percent of the replicates if it were determined accurately. Therefore, we tallied the proportion of
replicates in which the calculated confidence interval
included the actual marginal treatment effect, termed the
percent coverage, to characterize the behavior of confidence
intervals.
A number of variables were varied to determine the limitations of the grouped-treatment method as an approach to
estimating the marginal treatment effect. Ranges of these
variables were chosen to illustrate a variety of conditions
that could occur in clinical practice and to test the method
when confounding was severe enough to reverse the direction of the apparent treatment effect.
RESULTS
For the initial simulations, the grouped-treatment variable—the proportion of cases receiving treatment T at the
hospital of treatment—varied between 42 percent and 84
percent. This distribution might reflect practice variability
for an established therapy with unclear indications. Because
we modeled a small protective effect for treatment (model 1;
ORTO = 0.75), the expected variability in outcome by
hospital was small (66–68 percent dying).
The accuracy of the grouped-treatment estimator (model
3) was tested over a range of association between the
confounder and the treatment probability (ORCT) and
compared with the standard, individual-level estimate with
the confounder “unknown” and absent from the equation
(model 2) (table 1). In these simulations, 5,000 patients and
50 hospitals were modeled, and all other associations were
held constant. The underlying conditional treatment effect
756 Johnston et al.
TABLE 1. Treatment effect estimates varying the association between the confounder and the
probability of treatment*
Individual-level estimate (model 2)
ORCT†
OR
Average
95% CI†
0.3
0.59
0.53, 0.67
0.5
0.65
0.7
0.69
1
Grouped-treatment estimate (model 3)
OR
Average
95% CI
% coverage
3.9
0.73
0.46, 1.15
96.4
0.58, 0.73
38.3
0.74
0.48, 1.14
95.7
0.61, 0.78
76.5
0.75
0.49, 1.14
95.0
0.76
0.67, 0.86
93.9
0.77
0.50, 1.16
94.6
1.5
0.83
0.74, 0.94
60.1
0.78
0.51, 1.19
93.9
2
0.89
0.78, 1.00
22.0
0.77
0.50, 1.20
94.2
3
0.96
0.85, 1.09
3.0
0.79
0.49, 1.26
93.9
% coverage
* The complete individual-level model with confounder included produces an odds ratio (OR) = 0.75, and the
“true” value being estimated, the marginal estimate of the treatment effect, is OR = 0.76. Estimates and
confidence intervals are averages of 1,000 replicates with 5,000 observations. The association between
confounder and outcome is fixed (ORCO = 2.7). Percent coverage is the percentage of replicates in which the
marginal estimate is included in the 95% confidence intervals.
† ORCT, odds ratio of the association between confounder and treatment; CI, confidence interval.
was small and protective (model 1; ORTO = 0.75). The prognosis was adversely affected by the confounder (ORCO = 2.7)
and the covariate (ORBO = 2). Under these conditions, the
underlying unconfounded marginal treatment effect is 0.76,
the individual-level estimate (model 2) when ORCT = 1, and
does not vary in the table. The association between the
grouped-treatment variable and the actual treatment assignment was relatively weak in these cohorts (R = 0.26–0.29).
The grouped-treatment estimates are generally closer to the
unconfounded marginal treatment effect than are the standard individual-level estimates with the confounder
unknown. Bias was present at the extremes, which were
smaller than in the standard individual-level model and
appeared to be in the direction of the underlying
confounding. The confidence interval of the grouped-treatment estimates included the underlying marginal treatment
effect in approximately 95 percent of replicates, as represented by the percent coverage, which suggests that the reliability of the estimate was accurately represented. This was
in sharp contrast to the percent coverages for the individuallevel estimator.
With a similar distribution of the grouped-treatment variable, the strength of the association between confounder and
outcome (ORCO) was varied (table 2). Again, the population
was set at 5,000 patients distributed among 50 hospitals with
ORBO = 2 and ORTO = 0.75. Those with the confounder were
modeled to receive treatment more frequently (ORCT = 2),
and the association between the confounder and the outcome
was varied from a strong protective effect (ORCO = 0.05) to a
strong detrimental effect (ORCO = 20.1). The unconfounded
marginal estimate of treatment effect (model 2 with ORCT =
1) varied for each level of ORCO and is given in the second
column of table 2; values include random variation because
they were calculated. Compared with the unconfounded
marginal estimates, the grouped-treatment approach generally produced less biased estimates than did the standard
individual-level model (model 2). Again, the 95 percent
confidence interval for the grouped-treatment estimator
included the underlying treatment effect in approximately 95
percent of replicates, whereas the confidence interval of the
individual-level estimator was unreliable.
To determine the sensitivity of the grouped-treatment
approach to sample size, we varied the modeled population
size from 500 to 7,000 distributed evenly among 50 hospitals
(table 3). All other variables were held constant (ORTO =
0.75, ORBO = 2, ORCO = 2.7, ORCT = 2). Under these conditions, the underlying unconfounded marginal treatment
effect (model 2 with ORCT = 1) is 0.76. Because the
confounder is associated with greater probability of treatment and is detrimental, incomplete adjustment would be
anticipated to produce an odds ratio closer to one. The
grouped-treatment approach produced less biased estimates
even with only 500 patients or 10 per hospital. Bias was
generally reduced with increasing sample size for the
grouped-treatment approach but not for the individual-level
model. The 95 percent confidence interval for both estimates
narrowed with increasing sample size; however, the confidence interval accurately reflected the reliability of the estimate for the grouped-treatment approach but not the
standard individual-level approach.
We varied the independent covariate broadly (ORBO =
0.05–20.1) to ensure that the grouped-treatment approach
produced reliable estimates when a known confounder was
included in the model (table 4). Other associations were held
constant, as in table 3, with 5,000 observations modeled. The
individual-level estimates were biased and systematically
skewed toward the direction of confounding. The groupedtreatment estimates accurately represented the marginal
treatment effect (OR = 0.76); confidence intervals were
broad but well behaved.
To model different forms of practice variability among
hospitals, we varied the distribution of treatment probability
(table 5). Regardless of the direction of confounding and the
form of the distribution of treatment probabilities across
hospitals, the grouped-treatment estimates were less biased
representations of the marginal treatment effect compared
Am J Epidemiol 2002;156:753–760
Grouped-Treatment Analysis 757
TABLE 2. Treatment effect estimates varying the association between the confounder and the outcome*
Individual-level estimate (model 2)
“True” value
OR
OR
Average
95% CI†
0.05
0.81
0.56
0.50, 0.63
0.14
0.79
0.59
0.53, 0.67
0.37
0.79
0.65
1
0.75
2.7
7.4
20.1
ORCO†
Grouped-treatment estimate (model 3)
OR
Average
95% CI
% coverage
0.0
0.77
0.49, 1.20
93.9
0.7
0.76
0.49, 1.17
95.3
0.58, 0.73
8.9
0.74
0.48, 1.12
94.4
0.75
0.67, 0.84
95.2
0.75
0.49, 1.14
94.7
0.76
0.88
0.78, 1.00
32.6
0.78
0.50, 1.21
95.3
0.78
1.03
0.91, 1.17
0.7
0.79
0.50, 1.27
96.4
0.81
1.13
0.99, 1.29
0.1
0.82
0.50, 1.33
94.6
% coverage
* The complete individual-level model with confounder included produces an odds ratio (OR) = 0.75. The “true”
value being estimated, the marginal estimate of the treatment effect, varies in each model. Estimates and
confidence intervals are averages of 1,000 replicates with 5,000 observations. The association between
confounder and treatment is fixed (ORCT = 2.0). Percent coverage is the percentage of replicates in which the
marginal estimate is included in the 95% confidence intervals.
† ORCO, odds ratio of the association between confounder and outcome; CI, confidence interval.
procedure to another. Of course, it may not be possible to
treat all patients with both procedures, and these findings
would not apply to those who are not candidates for both
(22), similar to results of randomized trials. In addition, the
relative benefit of the better procedure may not be as great in
a subpopulation of patients. This possibility can be evaluated
by testing nonlinear forms of the grouped-treatment variable.
For example, categorizing the grouped-treatment variable or
fitting splines may reveal a plateau in the benefit of a procedure at high rates of use.
In our models, the grouped-treatment variable and the
outcome are statistically independent conditional on the
value of the actual treatment. This is equivalent to the basic
instrumental variable assumption (10, 22). Practically, it
corresponds with the assumption that the proportion of cases
treated by a given procedure at an institution is not independently associated with the pretreatment prognosis of
patients. In other words, high-risk patients are not being
referred to certain hospitals because of those hospitals’ utilization of a specific procedure. For example, sicker patients
with individual-level estimates. Confidence interval
coverage was again fully in accord with the nominal 95
percent.
DISCUSSION
These simulation results suggest that, if specific assumptions are satisfied, incorporating grouped-treatment variables into models with individual-level covariates and binary
outcomes produces more reliable estimates of the treatment
effect than do standard individual-level models when a
confounder is not included in the models. In a wide variety
of situations, the coefficients produced by the grouped-treatment approach are reasonably unbiased estimates of the
treatment effect. The estimated treatment effect is the odds
ratio comparing the condition that all patients were treated
by the therapy of interest with the condition that none of the
population was thus treated after adjustment for known
covariates. This would correspond with the expected results
for an institution that converted completely from use of one
TABLE 3. Treatment effect estimates varying the number of observations*
Individual-level estimate (model 2)
Observations
(no.)
Grouped-treatment estimate (model 3)
OR
Average
95% CI†
% coverage
OR
Average
95% CI
% coverage
500
0.88
0.36, 2.15
92.1
0.86
0.23, 3.23
93.8
1,000
0.88
0.60, 1.30
87.1
0.82
0.31, 2.19
94.7
2,000
0.88
0.67, 1.16
79.1
0.80
0.35, 1.81
95.5
5,000
0.88
0.73, 1.07
62.5
0.80
0.42, 1.52
94.3
7,000
0.88
0.78, 1.00
30.7
0.77
0.50, 1.20
94.5
* The complete individual-level model with confounder included produces an odds ratio (OR) = 0.75, and the
“true” value being estimated, the marginal estimate of the treatment effect, is OR = 0.76. Estimates and
confidence intervals are averages of 1,000 replicates. Associations between confounder and outcome (ORCO =
2.7) and between confounder and probability of treatment (ORCT = 2.0) are fixed. Percent coverage is the
percentage of replicates in which the marginal estimate is included in the 95% confidence intervals.
† CI, confidence interval.
Am J Epidemiol 2002;156:753–760
758 Johnston et al.
TABLE 4. Treatment effect estimates varying the association between the independent risk factor and
the outcome*
Individual-level estimate (model 2)
ORBO†
Grouped-treatment estimate (model 3)
OR
Average
95% CI†
% coverage
OR
Average
95% CI
% coverage
0.05
0.84
0.75, 0.95
48.4
0.77
0.50, 1.18
94.5
0.14
0.84
0.75, 0.94
49.4
0.76
0.50, 1.16
94.6
0.37
0.84
0.75, 0.94
49.0
0.76
0.50, 1.15
95.1
1
0.84
0.75, 0.94
49.2
0.76
0.50, 1.15
94.5
2.7
0.84
0.75, 0.94
51.4
0.76
0.50, 1.17
95.3
7.4
0.84
0.75, 0.95
52.4
0.77
0.50, 1.19
93.4
20.1
0.84
0.75, 0.95
55.6
0.76
0.49, 1.20
93.6
* The complete individual-level model with confounder included produces an odds ratio (OR) = 0.75, and the
“true” value being estimated, the marginal estimate of treatment effect, is OR = 0.76. Estimates and confidence
intervals are averages of 1,000 replicates with 5,000 observations. We fixed the associations between
confounder and treatment (ORCT = 2.0) and confounder and outcome (ORCO = 2.0). Percent coverage is the
percentage of replicates in which the marginal estimate is included in the 95% confidence intervals.
† ORBO, odds ratio of the association between independent risk factor and outcome; CI, confidence interval.
with intracranial aneurysms do not preferentially seek treatment at those hospitals offering a new therapy. This assumption may be difficult to satisfy in practice and may limit the
utility of the grouped-treatment approach.
If there are known confounders that are clustered at certain
hospitals, such as ethnicity or older age, and these are also
associated with outcome, including these variables in the
model would be expected to eliminate the confounded association between the grouped-treatment variable and the
outcome. For example, hospitals that treat wealthier patients
may offer a new therapy more frequently. If wealth is associated with the outcome and there is no measure of wealth
available, the grouped-treatment variable would be associated with outcome and the basic instrumental variable
assumption would not be fulfilled. However, if an accurate
measure for a patient’s wealth were included in the model,
the assumption could be fulfilled. We found that adjustment
for a covariate in the model with the grouped-treatment variable was possible, at least when the covariate was equally
distributed among hospitals. Further analysis is required to
confirm that adjustment for an unequally distributed covariate is practical and to demonstrate the impact of clustering
of observations by institution due to unmeasured covariates
that are not related to treatment assignment. In addition,
adjustment for unequally distributed covariates that are
measured only at the group level may be practical, but this
requires further modeling and assumptions. Because unmeasured group-level covariates may be present, it may be advisable to use generalized estimating equations or random
effects models to account for clustering (9).
TABLE 5. Treatment effect estimates varying the association between the confounder and the outcome in three representatives of
practice variability*
Individual-level estimate (model 2)
Distribution of treatment by
hospital
Logistic
Uniform
Split
ORCO†
“True” value
OR
2.0
0.5
Grouped-treatment estimate (model 3)
OR
Average
95% CI†
% coverage
OR
Average
95% CI
% coverage
0.79
0.84
0.74, 0.94
81.1
0.77
0.50, 1.18
94.9
0.73
0.68
0.60, 0.76
75.9
0.73
0.48, 1.11
95.8
2.0
0.73
0.92
0.80, 1.05
12.9
0.77
0.37, 1.59
94.6
0.5
0.77
0.62
0.54, 0.71
12.0
0.73
0.36, 1.46
94.6
2.0
0.84
0.92
0.80, 1.06
73.6
0.79
0.39, 1.61
95.4
0.5
0.69
0.62
0.54, 0.71
64.1
0.72
0.36, 1.41
95.1
* Practice variability is reflected in the distribution of treatment probabilities among the hospitals. The logistic distribution of treatment
probability is identical to that used in other simulations. Treatment probability is proportional to hospital number in the uniform distribution. In the
split distribution, treatment probability is assigned by two uniform distributions to produce utilization rates of 0–10% in half the hospitals and 35–
60% in the other half. The complete individual-level model with confounder included produces an odds ratio (OR) = 0.75. The “true” value being
estimated, the marginal estimate of treatment effect, varies in each model. Estimates and confidence intervals are averages of 1,000 replicates
with 5,000 observations. The association between confounder and treatment is fixed (ORCT = 2.0 for the logistic distribution and described in the
methods for others). Percent coverage is the percentage of replicates in which the marginal estimate is included in the 95% confidence
intervals.
† ORCO, odds ratio of the association between confounder and outcome; CI, confidence interval.
Am J Epidemiol 2002;156:753–760
Grouped-Treatment Analysis 759
As long as the basic instrumental variable assumption is
fulfilled, the grouped-treatment estimator appears to provide
a less biased assessment of the impact of treatment than does
an estimator based on a standard individual-level model that
does not include the confounder. Therefore, at a minimum
the grouped-treatment estimator could provide a test of
residual uncontrolled confounding in the standard analysis:
A large difference between estimates based on the individual-level treatment variable and based on the groupedtreatment variable might suggest the presence of unmeasured confounding. A specific statistical test is available to
compare results from standard and instrumental variable
models (23), but a difference that is not statistically significant may still be important from a clinical or policy perspective, so a subjective comparison of the estimates may be
more useful. In this respect, the grouped-treatment approach
may have value as a sensitivity analysis.
The grouped-treatment approach is likely to provide more
accurate estimates of the treatment effect if the correlation
between the grouped-treatment variable and the actual treatment assignment is strong relative to the correlation between
the grouped-treatment variable and the unmeasured
confounder (24). The correlation between the grouped-treatment variable and actual treatment assignment increases
with greater practice variability; in this instance, the likelihood of receiving treatment varies greatly from institution to
institution, and this variation is independent of unmeasured
confounders. As the correlation decreases, the grouped-treatment estimator is expected to become more biased in the
direction of the standard, confounded estimator. Although
we modeled three distributions of treatment utilization
among the hospitals, the correlation between the actual treatment assignment and the grouped-treatment variable varied
little (R = 0.19–0.28). The grouped-treatment approach may
not perform as well when practice variability is very small.
It is possible to show that the grouped-treatment estimate
is unbiased with large samples when there is no underlying
treatment effect—equivalent to the null hypothesis in many
studies. Therefore, the approach should be reliable for
hypothesis testing when the existence, rather than the magnitude, of a treatment effect is of primary interest. Additional
theoretical research will be required to define the limits of
the method and to develop more generalized models to
accommodate nonnormal, continuous outcomes.
The confidence intervals for the grouped-treatment estimate were wider than those for the standard estimate. This
was expected because of the loss of precision in estimating
treatment by using the imperfect surrogate, the proportion of
patients receiving the treatment at a given hospital. The
confidence interval of the grouped-treatment estimate
included the actual unconfounded marginal estimate in
approximately 95 percent of replicates over a range of conditions. This was not the case for the individual-level model
with the unmeasured confounder. Therefore, the groupedtreatment approach appears to produce confidence intervals
with nominal coverage behavior.
In the example motivating these simulations and in our
previous studies, we have defined the grouped-treatment
variable as the proportion of cases receiving a certain procedure at the hospital of treatment. Other constructs for
Am J Epidemiol 2002;156:753–760
grouping the treatment are possible, such as geographic
region or relative distance to a hospital with particular characteristics (12), and are valid as long as the grouped-treatment variable is independent of outcome, conditional on
known covariates and the actual treatment assignment.
Grouping based on the hospital of treatment has certain
advantages. First, it is an intuitive unit so that the assumptions can be evaluated more directly. Second, the hospital of
treatment is a convenient variable that is frequently available
in large data sets. Third, practice variability between hospitals tends to exist whenever there is disagreement about the
indications for a given treatment (5, 6), so it is relatively easy
to demonstrate correlation between the grouped-treatment
variable and actual treatment assignment. Fourth, if there are
other hospital characteristics associated with the availability
of a given treatment, such as greater hospital treatment
volume, these can be evaluated in the models if measurable.
For example, in our previous study of intracranial aneurysm
treatment, we showed that the newer procedure was associated with lower risk of in-hospital death and that the effect
was independent of hospital volume and offering other
specialized procedures (8). If other hospital characteristics
are not measurable, the results can still be interpreted at the
hospital level: If hospitals that use the treatment more
frequently have better outcomes, regardless of whether the
treatment or other hospital characteristics are responsible,
these hospitals may be good choices for referral of patients.
However, the possibility that utilization of the treatment of
interest may be associated with other group-level characteristics must be carefully considered. Such an association
provides another potential source of unmeasured
confounding.
Although the grouped-treatment approach produces biased
estimates of the underlying marginal treatment effect, this
bias appears to be smaller than that in standard models with
only individual-level variables. In many instances, the bias
in the grouped-treatment estimate appears quite small, and
its confidence interval has good properties. Therefore, the
grouped-treatment approach may supplement standard
multivariable analyses of large data sets when the groupedtreatment variable has no independent association with the
outcome because of unmeasured confounders and when the
treatments being compared are generally options for the individuals included in the cohort.
ACKNOWLEDGMENTS
The authors thank Drs. Ira Tager, John Colford, and Alan
Hubbard for important discussions in formulating the
method.
REFERENCES
1. Byar DP. Why data bases should not replace randomized clinical trials. Biometrics 1980;36:337–42.
2. Green SB, Byar DP. Using observational data from registries to
compare treatments: the fallacy of omnimetrics. Stat Med 1984;
760 Johnston et al.
3:361–73.
3. Grobbee DE, Hoes AW. Confounding and indication for treatment in evaluation of drug treatment for hypertension. BMJ
1997;315:1151–4.
4. Miettinen OS. The need for randomization in the study of
intended effects. Stat Med 1983;2:267–71.
5. Wen SW, Kramer MS. Uses of ecologic studies in the assessment of intended treatment effects. J Clin Epidemiol 1999;52:
7–12.
6. McPherson K. The Cochrane Lecture. The best and the enemy
of the good: randomised controlled trials, uncertainty, and
assessing the role of patient choice in medical decision making.
J Epidemiol Community Health 1994;48:6–15.
7. Detels R, Munoz A, McFarlane G, et al. Effectiveness of potent
antiretroviral therapy on time to AIDS and death in men with
known HIV infection duration. Multicenter AIDS Cohort Study
Investigators. JAMA 1998;280:1497–503.
8. Johnston SC. Effect of endovascular services and hospital volume on cerebral aneurysm treatment outcomes. Stroke 2000;
31:111–17.
9. Johnston SC. Combining ecological and individual variables to
reduce confounding by indication: case study, subarachnoid
hemorrhage treatment. J Clin Epidemiol 2000;53:1236–41.
10. Kennedy P. A guide to econometrics. Cambridge, MA: MIT
Press, 1998.
11. Greenland S. An introduction to instrumental variables for epidemiologists. Int J Epidemiol 2000;29:722–9.
12. McClellan M, McNeil BJ, Newhouse JP. Does more intensive
treatment of acute myocardial infarction in the elderly reduce
mortality? Analysis using instrumental variables. JAMA 1994;
272:859–66.
13. Stefanski LA, Buzas JS. Instrumental variable estimation in
binary regression measurement error models. J Am Stat Assoc
1995;90:541–50.
14. Amemiya T. Advanced econometrics. Cambridge, MA: Harvard University Press, 1985.
15. Sheppard L, Prentice RL, Rossing MA. Design considerations
for estimation of exposure effects on disease risk, using aggregate data studies. Stat Med 1996;15:1849–58.
16. Chao WH, Palta M, Young T. Effect of omitted confounders on
the analysis of correlated binary data. Biometrics 1997;53:678–
89.
17. Neuhaus JM, Kalbfleisch JD. Between- and within-cluster
covariate effects in the analysis of clustered data. Biometrics
1998;54:638–45.
18. Hauck WW, Neuhaus JM, Kalbfleisch JD, et al. A consequence
of omitted covariates when estimating odds ratios. J Clin Epidemiol 1991;44:77–81.
19. Hauck WW, Anderson S, Marcus SM. Should we adjust for
covariates in nonlinear regression analyses of randomized trials? Control Clin Trials 1998;19:249–56.
20. Zeger SL, Liang KY, Albert PS. Models for longitudinal data:
a generalized estimating equation approach. Biometrics 1988;
44:1049–60.
21. Joseph KS. The evolution of clinical practice and time trends in
drug effects. J Clin Epidemiol 1994;47:593–8.
22. Angrist JD, Imbens GW, Rubin DB. Identification of causal
effects using instrumental variables. J Am Stat Assoc 1996;91:
444–55.
23. Hausman JA. Specification tests in econometrics.
Econometrica 1978;46:1251–71.
24. Bound J, Jaeger DA, Baker RM. Problems with instrumental
variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. J Am
Stat Assoc 1995;90:443–50.
Am J Epidemiol 2002;156:753–760