Supplemental Digital Content
Simulated Effects of Major Unobserved Confounding Variable
We conducted a Monte Carlo simulation to assess the extent to which the primary observed effect –
namely, the 11% increase in observed-to-expected in-hospital mortality ratio associated with relative
10% increments in HDR – might have been adversely influenced by important covariates unavailable to
us at the time of analysis (e.g., complexity of arterial lesions or extent of atherosclerotic burden). In this
analysis, we allowed treatment recommendations to be influenced by the presence of a simulated
binary covariate 𝑍 that had an independent effect on mortality which differed between PCI and CABG.
Below, we summarize the salient details of the analytic approach for this sensitivity analysis; a full
description is available from a section within our prior methods paper [A1.1] titled “Major Unobserved
Covariables”. We will be working with estimated odds of in-hospital mortality (as opposed to
probabilities), and remind the reader of the equivalent mathematical relations 𝑜𝑑𝑑𝑠 = (𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦) ÷
(1 − 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦) and 𝑝𝑟𝑜𝑏𝑎𝑏𝑖𝑙𝑖𝑡𝑦 = 𝑜𝑑𝑑𝑠 ÷ (1 + 𝑜𝑑𝑑𝑠).
Let 𝜃𝑃𝐶𝐼 be the predicted odds of mortality for a patient (with given values of the available covariates)
under PCI, and let 𝜃𝐶𝐴𝐵𝐺 be the predicted odds under CABG. By definition, the model-preferred
treatment is CABG when the odds under CABG are lower than the odds under PCI (or, equivalently,
when the ratio 𝜙 = 𝜃𝑃𝐶𝐼 ÷ 𝜃𝐶𝐴𝐵𝐺 is greater than 1). Likewise, the model-preferred treatment is PCI
when 𝜃𝐶𝐴𝐵𝐺 > 𝜃𝑃𝐶𝐼 (or, equivalently, when 𝜙 < 1).
Now, let 𝜁𝑃𝐶𝐼 be the odds ratio for 𝑍 under PCI, and let 𝜁𝐶𝐴𝐵𝐺 be the odds ratio for 𝑍 under CABG. Note
that these odds ratios may be different for the two procedures. For example, a four-fold differential
effect of 𝑍 between the two procedures could be represented as 𝜁𝑃𝐶𝐼 = 2 and 𝜁𝐶𝐴𝐵𝐺 = 0.5, reflecting
an increase in risk attributable to 𝑍 under PCI and a decrease under CABG. Likewise, the same four-fold
differential effect would be present in a situation where 𝑍 increases risk under both PCI and CABG, e.g.,
𝜁𝑃𝐶𝐼 = 5 and 𝜁𝐶𝐴𝐵𝐺 = 1.25. In both of the above situations, the differential effect 𝛾 = 𝜁𝑃𝐶𝐼 ÷ 𝜁𝐶𝐴𝐵𝐺 is
equal to 4. We call the parameter 𝛾 the effect ratio of 𝑍.
It can be shown that the model-preferred treatment only changes when the effect ratio of 𝑍 is large
enough to overcome the magnitude of the original odds ratio 𝜙. For example, suppose the odds of
mortality under PCI are 𝜃𝑃𝐶𝐼 = (1/10) and the odds under CABG are 𝜃𝐶𝐴𝐵𝐺 = (1/5). The odds ratio 𝜙
(prior to consideration of the unavailable covariate 𝑍) in this case is (1/10) ÷ (1/5) = 1/2, and
therefore the model-preferred treatment is PCI. With the updated odds ratio computed as the product
(𝜙 × 𝛾), only a differential effect of 𝑍 that is a magnitude of 2 or higher would result in a change in
model-preferred treatment from PCI to CABG. That is, if (as in the preceding paragraph) 𝑍 increases risk
under both PCI and CABG, with 𝜁𝑃𝐶𝐼 = 5 and 𝜁𝐶𝐴𝐵𝐺 = 1.25 the updated odds of mortality under PCI are
(5)(1/10) = 1/2, the updated odds under CABG are (1.25)(1/5) = 1/4, and therefore the new
model-preferred treatment (in the presence of 𝑍) is CABG.
To summarize the above point, changes in model-preferred treatment for a given patient after
incorporating the effects of the external variable only occur when the differential effect of that variable
is large enough to overcome the original discrepancy in risk. In this Monte Carlo simulation analysis we
modified both the prevalence of 𝑍 and the effect ratio 𝛾. Specifically, we considered prevalence values
ranging from 0% to 50% (in increments of 5%) and effect ratio 𝛾 values of 1/8, 1/4, 1/2, 2, 4, and 8.
Values of 𝛾 less than 1 decreased risk under PCI relative to CABG, and thus encouraged the model to
recommend PCI more frequently than in our main analysis. Likewise, values of 𝛾 greater than 1
increased risk under PCI relative to CABG and encouraged the model to recommend CABG more
frequently than in our main analysis.
For each combination of prevalence of 𝑍 and effect ratio 𝛾, we created 10 simulated datasets where,
first, discharges were randomly assigned to values of the binary covariate 𝑍; second, model-preferred
treatment was recomputed in light of the updated odds of mortality under each treatment (as described
in the preceding paragraphs); and third, the primary association between hospital-level discordance –
under the new model-preferred treatment assignments which incorporated the effect of 𝑍 – and the
ratio of observed-to-expected in-hospital mortality was re-estimated.
Results of individual simulations were visualized as well as averages for each combination of prevalence
and effect ratio. These results are presented in Figure S-2. In the obvious case when the incidence of the
unobserved covariate 𝑍 was 0% we obtained results equal to those obtained in our main analysis. As the
incidence of 𝑍 increased, we observed minimal sample-to-sample variability in estimated effect, and
under effect ratios less than 1, the primary effect estimate remained stable at 11% (as well as the
associated 95% confidence limits). For effect ratios greater than 1, results were less stable but not to the
extent that increased hospital-level discordance rates were no longer associated with risk-adjusted
mortality.
References
1.
Dalton JE, Dawson NV, Sessler DI, Schold JD, Love TE, Kattan MW. Empirical treatment
effectiveness models for binary outcomes. Med Dec Making. (in press).
Supplemental Figure Legend
Figure S-1
Overall distribution of propensity scores (estimated probability of receiving coronary
artery bypass grafting, given information on patient characteristics and present-onadmission diagnoses), split by actual procedure (CABG = coronary artery bypass grafting
and PCI = percutaneous coronary intervention).
Figure S-2
Results of Monte Carlo simulation analysis modeling the hypothetical impact of a major
unobserved covariate 𝑍 on the observed association between hospital discordance rates
and observed-to-expected mortality. The estimate [95% confidence interval] from our
primary analysis of 11% [5% —17%] can be seen on the left side of each panel, under
prevalence values of 0%. Estimates (black points) and confidence limits (blue points)
from individual simulated samples are shown, and solid lines display their means for
each combination of effect ratio 𝛾 and prevalence of 𝑍.
Figure S-1
Figure S-2