Supplemental Digital Content 4: Sensitivity Analyses
Part 1. Sensitivity of Results to Prior Distributions
Hyperprior distributions are placed on the hyperparameters of the multivariate normal
distribution N28 (γ, ) (see Supplemental Material, Section 3). However, the values drawn from
the multivariate normal distribution are on a logit scale. We are interested in priors on the pijs
(i.e., the values on a probability scale) that result from the hyperpriors for N28 (γ, ). It turns out
that the hyperprior distribution for  has very little impact on the priors on the pijs . The
hyperprior distribution for γ is a multivariate normal distribution N28 (m,R), where m is a 28dimension vector of 0s and R-1, the inverse of the variance-covariance matrix, has zeros in the
off-diagonal cells. We ran the model without conditioning on the data and observed the prior
distribution of the pijs, i.e., the distribution resulting from the hyperprior distributions we placed
on the parameters on the logit scale without their being any impact of the data. We assessed
uniformity visually from the density plots of the pijs. It turned out that the numbers in the
diagonal cells of R-1 had a strong influence on the induced priors on the pijs. By
experimentation, we determined that a value of .38 on the diagonals of R-1 resulted in
approximately uniform priors on the pijs; a value of .0001 on the diagonal resulted in a bimodal
prior distribution for the pijs with nearly all of the weight at 0 or 1. In the main analyses, we
used .38 in the diagonal cells of R-1. To test the sensitivity to an alternative hyperprior, we used
.0001.
As shown in Figure 4.1, with a very few exceptions the point estimates of the composite
scores are very similar when using the approximately uniform hyperprior or the bimodal
1
hyperprior for the pijs. However, in many cases there is a substantial difference in the PTQ
values.
Part 2. Sensitivity of Conclusions about PTQ versus PTH to the Skewness of the
Distribution of the Composite Scores
We used simulated data to analyze the sensitivity of conclusions about PTQ versus PTH
to the skewness of the distribution of composite scores. The steps in the simulation were the
following:
1. Generate 112 composite scores (corresponding to our 112 facilities) from a beta distribution.
We used 3 parameterizations – beta(6,6), which leads to a symmetric distribution of composite
scores; beta(1,9), which leads to a right skewed distribution; and beta(9,1), which leads to a left
skewed distribution. The left most graphs in Supplemental Material 4, Figure 4.2, show the 3
distributions of generated data. These generated data correspond to composite scores and are the
“true” probabilities of developing an adverse event at each facility;
2. Using each of the 3 beta distributions, assuming the same number of residents at each facility
as in our data set and assuming that adverse events arise from a binomial distribution, we used
WinBUGS to generate the number of adverse events at each facility;
3. Again using each of the 3 beta distributions, from the number of simulated adverse events at
each facility and the number of residents, and using the same model we used to generate the data
above, we estimated the “true” probability of developing an adverse event at each facility, as
well as PTQ and PTH.
2
The right most column in Figure 4.2 shows graphs similar to those in the paper for each
of the 3 distributions. As is apparent, the main story associated with PTQ vs. PTH is the same
regardless of the nature of the distribution for the composite scores. However, as the distribution
of composite scores changes from left skewed to symmetric to right skewed, there is somewhat
great distinction among the highest of the high performers.
Part 3. In a P4P Program, Comparing the Impact of Using PTQ with a Composite Measure
and Using PTQ with the Individual QIs
In the main analyses reported in the paper, we calculated a composite measure for each
facility and then estimated the probability the composite measure was in the top quintile (PTQ).
P4P payments were based on PTQ. In the analyses reported here, we calculated PTQ for each
individual QI in each facility and then determined QI-specific payment based on the QI-specific
PTQ using the same approach as used when considering the composite measure. Thus, for each
QI, we know the proportion of the QI-specific payment pool each facility would receive. In
order to divide the overall P4P payment pool into QI-specific payment pools, we used “overall”
opportunity-based weights. These are calculated as follows: Let Dij = the number of residents
eligible for QI i in facility j. Then, the opportunity-based weight for QI i in facility j is
Dij / Σi Dij. The overall opportunity-based weight for QI i is calculated as Σj Dij / Σij Dij. The
proportion of the overall payment pool allocated to QI i is the overall opportunity-based weight
for QI i. This approach is consistent with the way in which the composite measure was created,
i.e., facility-specific opportunity-based weights were used to combine the shrunken estimates of
the individual QI rates. The effect of this is that the percentage of the overall payment pool
allocated to each facility is a weighted average of the facility’s proportion of the QI-specific
payment pool, where the weight for each QI is the overall opportunity-based weight for that QI.
3
Figure 4.3 shows the relationship between the proportion of the payment pool going to
facilities when the composite measure is used and when individual QIs are used as the basis for
bonus payments. There are two striking features of the graphs:1) There is much less variation in
payment when bonuses are paid based on performance on the individual QIs; and 2) There is no
relationship between bonus payments when the composite measure is used and bonus payments
when individual QIs are used.
4
Figure 4.1: Comparing Composite Score and PTQ Using a Uniform Prior and a Bimodal Prior
5
Figure 4.2: Impact of the Skewness of the Simulated Distribution of the Composite Scores on the
Relationship Between PTQ and PTH
Simulated Distribution of
Composite Scores
Payment Based on PTQ (monotonically declining
line) and PTH (jagged line)
1) Beta(6,6)
2) Beta(1,9)
3) Beta(9,1)
6
Figure 4.3: Comparing P4P Payments Using a Composite Score and Using the Individual QIs
7