Research Designs For
Evaluating Disease
Management Program
Effectiveness
Disease Management Colloquium
June 27-30, 2004
Ariel Linden, Dr.P.H., M.S.
President, Linden Consulting Group
1
What’s the Plan?
Discuss “threats to validity”
Provide methods to reduce those threats
using currently-used evaluation designs
Offer additional designs that may be
suitable alternatives or supplements to the
current methods used to assess DM
program effectiveness
2
Measurement Error
Treatment Interference
Seasonality
Loss to Attrition
Hawthorne Effect
New Technology
VALIDITY
Benefit Design
Access
Reimbursement
Selection Bias
Case-mix
Unit Cost Increases
Regression to the Mean
Secular Trends
3
Maturation
Selection Bias
Definition: Participants are not
representative of the population from
which they were drawn:
Motivation
Severity or acuteness of symptoms
Specifically targeted for enrollment
4
Selection Bias (cont’)
Fix #1: Randomization
How: Distributes the “Observable”
and “Unobservable” variation
equally between both groups
Limitations: costly, difficult to
implement, intent to treat, not
always possible
5
Selection Bias (cont’)
Pretest-posttest Control Group:
R O1 X O2
R O3
Solomon 4-Group Design:
R O X O
R O
6
O4
O
R
X O
R
O
Selection Bias (cont’)
Fix #2: Standardized Rates
How: Direct/indirect adjustment
enables comparisons over time or
across populations by weighting
frequency of events
Limitations: does not control for
“unobservable” variation
7
Age-adjusted Program Results
Pre-Program
(rate/1000)
rXP
Program
(rate/1000)
rXP
Proportion (P)
of Population
20 – 29
7.3
0.9
10.2
1.2
0.1189
30 – 39
65.2
5.7
79.9
6.9
0.0868
40 – 49
190.8
13.4
173.6
12.2
0.0703
50 – 59
277.9
21.3
226.1
17.4
0.0768
60 - 69
408.4
25.2
287.8
17.7
0.0616
70 - 79
475.8
17.7
368.8
13.8
0.0373
80 +
422.2
8.4
356.0
7.0
0.0198
Age
Group
Adjusted rate
8
92.6
76.2
Tenure-adjusted Program Results
9
Baseline Group
Compared to
inflationadjusted…
Baseline Group
Compared to
inflationadjusted…
2003 prevalent
group’s 2003
claims
2003 prevalent
group’s 2004
claims plus
2004 incident
group assumed
to have cost
2003 prevalent
group’s claims
in 2003
2002 prevalent
group’s 2003
claims
2003 prevalent
group’s 2004
claims
2003 Newly
incident
members
actual claims,
2003
2004 Newly
incident
members
actual claims,
2004
Selection Bias (cont’)
Fix #3: Propensity Scoring
What?: Logistic regression score for
likelihood of being in intervention
How: Controls for “Observable”
variation
Limitations: does not control for
“unobservable” variation
10
1st Year CHF Program Results
Intervention
(N=94)
Control Group
(N=4606)
P(T<=t)
two-tail
Age
77.4
76.6
NS
% Female
0.51
0.56
NS
% Portland
0.17
0.69
p<0.0001
Pre-Hospitalization
1.13
0.5
p<0.0001
Pre-ED
0.7
0.4
p=0.003
$8,974
p<0.0001
Pre-Costs
Post-Hospitalization
0.59
0.87
p=0.008
Post-ED
0.57
0.58
NS
$11,874
$16,036
p=0.005
Post-Costs
11
$18,287
1st Year CHF Program Results
Admits
Hospitalization Rate
1.2
1.13
1
0.87
0.8
0.6
0.4
0.59
0.5
0.2
0
Pre-Hospital Admits
Intervention Group (N=94)
12
Post-Hospital Admits
Control Group (N=4606)
ED Visit Rate
1st Year CHF Program Results
ER Visits
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.7
0.58
0.57
0.4
0
Pre-ED Visits
Intervention Group (N=94)
13
Post-ED Visits
Concurrent Control Group (N=4606)
Average Cost
1st Year CHF Program Results
Costs
$20,000
$18,000
$16,000
$14,000
$12,000
$10,000
$8,000
$6,000
$4,000
$2,000
$0
$18,287
$16,036
$11,874
$8,974
Pre-Costs
Intervention Group (N=94)
14
Post-Costs
Control Group (N=4606)
1st Year CHF Program Results
Propensity Scoring Method
Cases
(N=94)
Matched Controls
(N=94)
P(T<=t)
two-tail
0.061
0.062
NS
Age
77.4
78.2
NS
% Female
0.51
0.51
NS
% Portland
0.17
0.17
NS
Pre-Hospitalization
1.13
1.09
NS
Pre-ED
0.70
0.67
NS
$18,287
$17,001
NS
Post-Hospitalization
0.59
1.17
0.005
Post-ED
0.57
0.77
0.026
$11,874
$24,085
0.003
Propensity Score
Pre-Costs
Post-Costs
15
1st Year CHF Program Results
Propensity Scoring Method - Admits
Hospitalization Rate
1.4
1.2
1
1.13
1.09
1.17
0.8
0.6
0.59
0.4
0.2
0
Pre-Hospital Admits
cases (N=94)
16
Post-Hospital Admits
matched controls (N=94)
1st Year CHF Program Results
ED Visit Rate
Propensity Scoring Method – ED Visits
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0.77
0.7
0.67
0.57
Pre-ED Visits
cases (N=94)
17
Post-ED Visits
matched controls (N=94)
1st Year CHF Program Results
Propensity Scoring Method – Costs
$30,000
Average Cost
$25,000
$20,000
$15,000
$24,085
$18,287
$17,001
$11,874
$10,000
$5,000
$0
Pre-Costs
cases (N=94)
18
Post-Costs
matched controls (N=94)
Regression to the Mean
Definition: After the first of two
related measurements has been made,
the second is expected to be closer to
the mean than the first.
19
Regression to the Mean
CAD
Where the 1st Quintile (N=749) Went In Year 2
Fifth
quintile
17%
First
quintile
35%
Where the 5th Quintile (N=748) Went In Year 2
Fifth
quintile
27%
Fourth
quintile
16%
Third
quintile
13%
20
Second
quintile
19%
Fourth
quintile
23%
First
quintile
11%
Second
quintile
18%
Third
quintile
21%
Regression to the Mean
CHF
Where the 1st Quintile (N=523) Went In Year 2
Fifth
quintile
15%
First
quintile
35%
Fourth
quintile
17%
Third
quintile
15%
21
Second
quintile
18%
Where the 5th Quintile (N=537) Went In Year 2
Fifth
quintile
37%
First
quintile
7%
Fourth
quintile
23%
Second
quintile
14%
Third
quintile
19%
Regression to the Mean (cont’)
Fix #1: Increase length of
measurement periods
How: Controls for movement
toward the mean across periods
Limitations: periods may not be long
enough, availability of historic data
22
Regression to the Mean (cont’)
Currently-Used Method
Claims run-out periods
Baseline
Measurement
Year
1st Contract
Measurement
Year
Compare to baseline
Compare to baseline
23
2nd Contract
Measurement
Year
Regression to the Mean (cont’)
Valid Method (from Lewis presentation)
2 year “freeze” period + measurement
Historic 2-Year
Period
Baseline
Measurement
Year
2 year “freeze” period + measurement
24
1st Contract
Measurement
Year
Regression to the Mean (cont’)
Fix #2: Time Series Analysis
How: Controls for movement across
many periods (preferably > 50
observations)
Limitations: availability of historic
data, change in collection methods
25
Measurement Error
Definition: Measurements of the same
quantity on the same group of
subjects will not always elicit the same
results. This may be because of
natural variation in the subject (or
group), variation in the measurement
process, or both (random vs.
systematic error).
26
Measurement Error (cont’)
Fix #1: Use all suitables in the analysis (to
adjust for the “zeroes”)
Fix #2: Use identical data methods pre
and post (like unit claims-to-claims
comparison)
Fix #3: Use utilization and quality
measures instead of cost.
27
Alternative Designs
Survival Analysis
Time Series Analysis
Time-dependent Regression
28
Survival Analysis
Features:
Time to event analysis – longitudinal
Censoring
Allows for varying enrollment points
29
Survival Analysis
First Patient Commencement of
Contact
DM Intervention
Improvement in
clinical markers
??
??
??
30
Reduction in Utilization
and Costs
Survival Analysis
.6
.5
.4
.3
PRA Value
.2
High Risk
High-censored
.1
Low Risk
Low-censored
0.0
0
31
10
20
Time (Months)
30
Time Series Analysis
Features:
Longitudinal analysis
Serial Dependency (autocorrelation)
Does not require explanatory variables
Controls for trend and seasonality
Can be used for forecasting
32
Time Series Analysis (cont’)
60
Admits (Actual)
SES
50
DES
Admits PTMPY
ARIM A (1,0,0)
SES (15.5%)
DES (19.7%)
ARIMA (16.0%)
SES (3.6%)
Baseline
Period
1st Measurement
Period
40
30
20
10
Historical Period
Ja
n9
Ap 8
r-9
8
Ju
l-9
O 8
ct98
Ja
n9
Ap 9
r-9
9
Ju
l-9
9
O
ct99
Ja
n0
Ap 0
r-0
0
Ju
l-0
O 0
ct00
Ja
n0
Ap 1
r-0
1
Ju
l-0
1
O
ct01
Ja
n0
Ap 2
r-0
2
Ju
l-0
2
O
ct02
0
Month
33
Time-dependent Regression
Combines important elements of other
models to create a new method,
including variables such as:
Program tenure (censuring)
Seasonality (important for Medicare)
Can be used for forecasting
34
350
300
Admits/1000
250
200
150
100
50
0
1
2
3
4
5
6
7
8
9
10
11
12
Admits/1000 (Tenure)
13
14
15
16
17
18
19
20
21
Admits/1000 (Month)
Simulated hospital admissions per thousand members based on program tenure
and month-of-year (months 1-12 represent Jan – Dec of program year 1, and
months 13-24 represent Jan – Dec of program year 2).
35
22
23
24
Conclusions
Identify potential threats to validity
before determining evaluation method
Choose outcome variables that mitigate
measurement bias (e.g. all identified
members vs those with costs)
There is no panacea! Use more than one
design to validate results.
36
How does this presentation differ
from what you just saw?
• Lewis approach is the only valid prepost population-based design in use
today
• But valid = accurate. “Valid” just means
adjustment for systematic error
• These methods reduce chances of nonsystematic error to increase accuracy
37
References (1)
38
1.
Linden A, Adams J, Roberts N. An assessment of the total population
approach for evaluating disease management program effectiveness.
Disease Management 2003;6(2): 93-102.
2.
Linden A, Adams J, Roberts N. Using propensity scores to construct
comparable control groups for disease management program evaluation.
Disease Management and Health Outcomes Journal (in print).
3.
Linden A, Adams J, Roberts N. Evaluating disease management program
effectiveness: An introduction to time series analysis. Disease
Management 2003;6(4):243-255.
4.
Linden A, Adams J, Roberts N. Evaluating disease management program
effectiveness: An introduction to survival analysis. Disease Management
2004;7(2):XX-XX.
References (2)
39
5.
Linden A, Adams J, Roberts N. Evaluation methods in disease
management: determining program effectiveness. Position Paper for the
Disease Management Association of America (DMAA). October 2003.
6.
Linden A, Adams J, Roberts N. Using an empirical method for
establishing clinical outcome targets in disease management programs.
Disease Management. 2004;7(2):93-101.
7.
Linden A, Roberts N. Disease management interventions: What’s in the
black box? Disease Management. 2004;7(4):XX-XX.
8.
Linden A, Adams J, Roberts N. Evaluating disease management program
effectiveness: An introduction to the bootstrap technique. Disease
Management and Health Outcomes Journal (under review).
References (3)
9.
Linden A, Adams J, Roberts N. Generalizability of disease management
program results: getting from here to there. Managed Care Interface
2004;(July):38-45.
10. Linden A, Roberts N, Keck K. The complete “how to” guide for selecting
a disease management vendor. Disease Management. 2003;6(1):21-26.
11. Linden A, Adams J, Roberts N. Evaluating disease management program
effectiveness adjusting for enrollment (tenure) and seasonality. Research
in Healthcare Financial Management. 2004;9(1): XX-XX.
12. Linden A, Adams J, Roberts N. Strengthening the case for disease
management effectiveness: unhiding the hidden bias. J Clin Outcomes
Manage (under review).
40
Software for DM Analyses
 The
analyses in this presentation used
XLStat for Excel. This is an Excel add-in,
similar to the data analysis package that
comes built-in to the program.
 Therefore,
users familiar with Excel will
find this program easy to use without
much instruction.
41
Questions?
Ariel Linden, DrPH, MS
[email protected]
42