Results of Current HCT Center
Specific Outcomes Analysis
Brent R. Logan, Ph.D.
September 10, 2010
Patient inclusion
Unrelated
(2004-2008)
Related
(2008)
N
N left
excl.
N
N left N excl. N left
excl.
First allogeneic U.S.
transplants with
Baseline or Pre_TED
form
11600
Total
2701
14301
Excluded 11 centers
due to low follow-up
712
10888
225
2476
937
13364
Excluded cases
without follow-up
89
10799
31
2445
120
13244
Excluded Centers
 Excluded if <75% overall fu at 1 year or
<50% fu among related donor tx
All 1st allo txs
Unrelated
Related
Center
# txs
% FU
% 1yr
FU
A
4-10
70%
70%
71%
71%
67%
67%
B 141-250
89%
86%
100%
100%
22%
0%
C 141-250
83%
71%
90%
78%
54%
38%
1-3
100%
33%
100%
33%
E 141-250
76%
64%
83%
70%
50%
44%
1-3
100%
0%
100%
0%
G 141-250
83%
61%
93%
77%
71%
43%
100%
29%
100%
100%
100%
0%
21%
21%
D
F
H
4-10
100%
29%
I
11-20
82%
64%
J
1-3
100%
0%
K
>250
78%
75%
% FU
% 1yr
FU
% FU
% 1yr
FU
78%
88%
56%
85%
Histogram of 1 year fu
120
100
Frequency
80
60
40
20
0
0.00%
15.00%
30.00%
45.00%
60.00%
1 year fu, %
75.00%
90.00%
Most centers have very good (>90% follow-up)
Final study population
 157 centers
 13,244 patients transplanted
 Primary outcome: One year survival
 Overall: 59%
 Censoring:
 991 (7.5%) had less than one year of
follow-up
 Detailed demographics are given in the
report in your handouts
Statistical Analysis:
Basic Principles
 Examination of individual center
specific outcomes relative to the
overall network
 Risk Adjustment for case mix at a
given center
 Assessment of performance needs to
account for sampling variability/sample
size
 Understandable to public audience
Statistical Methods
 Comparison of observed vs. predicted one
year survival probabilities in each center
 Observed survival probability: Kaplan-Meier
estimates of one year survival, by center
 Predicted survival probability (Risk
adjustment):
 Fit a (pseudovalue) logistic regression model
for one year survival to all patients in entire
network to predict patient outcomes based on
individual patient characteristics alone
Statistical Methods
 Predicted survival outcome at a given
center is based on the average predicted
survival of patients actually transplanted
at that center
 This represents what we would have
expected to happen to the patients at
that center if they had been transplanted
at a “generic” center in the network (i.e.
no center effect)
 Need to account for sampling variability in
comparing observed and predicted
outcomes
Statistical Methods
 95% confidence interval constructed
 Range of plausible values for survival
probability, if those patients had been
transplanted at a generic center in the
network
 Constructed by resampling
pseudovalues (Logan et al, Lifetime
Data Analysis, 2008)
 If observed survival is outside interval,
the center appears to be
underperforming or overperforming
relative to the overall network
Statistical Methods
 We also provide a case mix score (1-5)
 Describes the sickness/severity of
patients transplanted at that center NOT
the center outcome itself
 Compute predicted survival outcomes
for each center by averaging across
patients at that center.
 Scores are quintiles of center predicted
outcomes
 Score=1 is 20% of centers with highest
predicted survival outcomes according
to their case mix of patient
characteristics
 Descriptive information only – not used
explicitly in center outcomes analysis
Case Mix Score
 Quintles of centers based on the predicted
survival outcomes of patients at their
center
100.00%
Expected 1 year survival
80.00%
60.00%
40.00%
20.00%
0.00%
1
2
3
case mix score
4
5
Statistical Properties
 An “average” center has a <=5% chance
that they will be incorrectly identified as
overperforming or underperforming
(Type I error)
 Type I error rate is not dependent on
 Case mix, as long as included in
regression model
 Sample size (because wider intervals for
small centers)
Risk adjustment model
 Risk factors significant in regression
model
 disease / stage;
 Recipient age;
 Donor age (Unrelated donor PBSC and
BM tx only)
 Donor Type/HLA matching/graft type;
 recipient CMV status;
 recipient race;
 co-existing disease (yes/no);
Risk adjustment model
 Risk factors significant in regression
model (cont.)
 Karnofsky / Lansky score;
 year of transplant;
 conditioning regimen intensity (NHL, HL,
PCD, other malignancy only);
 resistant disease (NHL and HL only);
 Time from dx to tx (ALL and AML not in
CR1/PIF only);
 Donor/recipient sex match
Other factors tested but not significant
 ALL T-cell lineage
 ALL Philadelphia chromosome
 Donor race
 Prior autotx
 Donor CMV
 Donor parity
Reporting Results
 Results of risk adjustment model:
 Odds ratios (95% CI’s) for one year survival
(>1 means better survival)
 For each center, we include a table with






Number of tx
Case mix score
Observed survival
Predicted survival
95% prediction interval
An indicator of whether the center is
underperforming, performing comparably to, or
overperforming the entire network
 Graphical representations can also be helpful
Center Results
 13 centers identified as underperforming
 Correction from 14 centers in draft report
 Nine previously identified as underperforming in
2009 center outcomes report
 Three previously identified as underperforming
in each of the last 6 reports
 11 centers identified as overperforming
 Seven previously identified as overperforming in
2009 center outcomes report
 One previously identified as overperforming in
each of the last 6 reports
Transplant Center Access Directory
(Text from last year)
 This center had 80 transplant recipients between
January 1, 2004, and December 31, 2008,
included in the analysis.
 The risk level of these recipients placed the
center in the medium-high risk group.
 The estimated actual one-year survival of the 80
recipients was 40.0%.
 The center-specific analysis predicted a one-year
survival for these recipients of 42.6%, with 95%
statistical confidence that the predicted survival
was between 32.5% and 52.5%.
 This center’s actual results are similar to the
predicted range.
 The national one-year survival was 56.3% for
the 9,673 recipients transplanted in the United
States.
Additional analysis
 Consideration of new variables
 Separate risk adjustment modeling of
related and unrelated donor
transplants
 Risk adjustment modeling using 3 year
vs. 5 year window of data
 Impact of related donors on center
performance
 Relationship between center size vs.
case mix or performance
Consideration of new variables
 Not collected consistently across years in this
analysis
 AML cytogenetics: Form 2000
 Distance from transplant center (recipient zip
code centroid to transplant center zip code
centroid): Form 2000 or Legacy Form
 Median household income by recipient zip
code: Form 2000 or Legacy Form
 Sorror comorbidity score: Form 2400
 Each variable added one at a time to risk
adjustment model, accounting for form
collection
AML Cytogenetics
(Form 2000 only)
Level
Odds
Ratio
n
Lower
Upper
p-value
Adverse
442
1.00
Intermediate
616
1.24
1.01
1.53
0.040
Favorable
92
2.41
1.41
4.15
0.001
Unknown
96
0.71
0.46
1.10
0.125
Median household income by zip code
(Form 2000; Legacy Form)
Level
Odds
Ratio
n
Lower
Upper
p-value
<35K
2423
1.00
35 to
45K
2837
1.00
0.88
1.12
0.952
45 to
60K
2754
1.03
0.91
1.16
0.639
>=60K
2187
1.12
0.99
1.28
0.078
539
0.97
0.79
1.19
0.794
Unknown
0.236
Distance from Transplant Center (Form
2000; Legacy Form)
Level
Odds
Ratio
n
Lower
Upper
p-value
<80
6301
1.00
80 to 180
1993
0.97
0.87
1.08
0.587
840
0.97
0.83
1.13
0.669
1292
1.23
1.07
1.41
0.003
314
1.17
0.91
1.51
0.229
180 to
300
>300
Unknown
Sorror comorbidity score
(Form 2400)
Level
Odds
Ratio
n
Lower
Upper
p-value
0
2960
1.00
1
722
0.87
0.73
1.04
0.136
2
517
0.79
0.64
0.97
0.026
3
586
0.83
0.68
1.00
0.052
4
309
0.76
0.59
0.99
0.039
>4
272
0.49
0.37
0.64 <0.001
New variables: summary
 AML Cytogenetics are strongly predictive of
outcome but requires addition to Form 2400
 Distance from transplant center is modestly
predictive but requires addition of Zip code
to Form 2400
 Is distance from transplant center really a
patient characteristic?
 Median income not predictive
 Sorror comorbidity score appears to work
well as a predictor, but further validation is
planned through CIBMTR study in RRT
working committee; already on Form 2400
Separate models for related and
unrelated donor transplants?
 Combined model with donor type as a
covariate
 Pools sample size
 Assumes common effects of covariates unless
interaction is detected
 Separate models: may have small numbers
for certain variables (disease/stage groups)
 Allows for distinct effects of covariates
 Predicted survival probabilities compared
between the two approaches
 Brier score (and R^2) computed for each
model
 combined model (R^2=9.9%)
 separate models (R^2=10.2%)
Goodness of Fit measures
 Brier Score: Average squared difference
between predicted survival probability using
model and observed outcome at one year
 Adjusted for censoring
 Rescaled as R^2:
 1-BS(model)/BS(no adjustment)
 100% if perfect prediction of patient’s one
year survival status
 0% if no better than unadjusted model
(Kaplan-Meier estimate)
 UNR+REL combined model (R^2=9.9%)
 separate models for UNR and REL
(R^2=10.2%)
Plot of predicted survival probabilities for
separate vs. combined (UNR+REL) models
Plot of predicted center survival for
separate vs. combined (UNR+REL) models
Center performance outcomes for
separate vs. combined models:
Two centers changed status
=Observed survival
=Confidence Interval
Risk adjustment based on 3 years vs. 5
years of data
 2008 Forum recommended center outcomes
be based on 3 year window
 Plan to implement next year as another year
of related donor transplants becomes
available
 Should risk adjustment model continue to use
5 years of data?
 Stabilize model particularly for small
disease/risk groups
Risk adjustment based on 3 years vs. 5
years of data
 Plot of predicted survival probabilities for each
patient receiving URD tx between 2006-2008
using 3 year or 5 year risk adjustment model
 Brier score and R^2 measure
 five-year model (R^2=10.1%)
 three-year model (R^2=10.1%)
Plot of predicted 1 year survival probabilities based on regression
model using 3 years vs. 5 years of data
Plot of predicted survival for each
center by 3 year vs. 5 year models
Impact of related donor tx on center
performance
 Repeated center outcomes analysis
excluding related donor transplants
 Decrease in sample size per center
 Is performance on related donor
transplants different than URD tx?
 6 centers with discrepant performance
for overall analysis vs. URD tx only
analysis
Center performance discrepancies
=Observed survival
=Confidence Interval
Do larger centers transplant higher
risk patients?
Primarily adult centers
0.9
Predicted survival
0.8
0.7
0.6
0.5
0.4
0.3
0
100
200
300
400
Center size
500
600
700
800
Do larger centers transplant higher
risk patients?
Primarily pediatric centers
0.9
Predicted survival
0.8
0.7
0.6
0.5
0.4
0.3
0
50
100
Center size
150
200
Center size for underperforming
and overperforming centers
800
700
600
n
500
400
300
200
100
0
Underperforming
Overperforming
Conclusions
 Addition of related donor transplants
has had
 minimal impact on risk adjustment model
 Some impact on identification of
underperforming or overperforming centers
 Accounting for related donor
transplants can be reasonably done
through simpler model with a covariate
and interactions
 3 years of data is sufficient to stabilize
the risk adjustment model
Conclusions
 AML cytogenetics and to a lesser extent
distance from transplant center are
predictive of outcome but require
modification of forms
 Sorror score appears promising but will
be validated further through CIBMTR
study in RRT committee
 Larger transplant centers transplant
similar risk patients as smaller
transplant centers, and are more likely
to be overperforming centers