Adjusting Quality Measures for Race,
Ethnicity, Language and Country of
Origin (RELO): Preliminary Results
D O U G L A S W H O L E Y 1 , M I C H A E L F I N C H 2 , R O B K R E I G E R 1 , D AV I D R E E V E S 3
A PRIL 1 8 , 2 0 1 6
1 C ENTER FOR C ARE O RGANIZATION R ESEARCH AND D EVELOPMENT, H EALTH P OLICY AND M ANAGEMENT, U NIVERSITY OF M INNESOTA
2 F INCH AND K ING
3 I NSTITUTE OF P OPULATION H EALTH, U NIVERSITY OF M ANCHESTER, U NITED K INGDOM
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
1
Presentation Overview
Key Findings
Research Questions and Data
Methods
Results
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
2
Key Study Findings
The sociodemographic characteristics variables (i.e., race, ethnicity, language, country of
origin, and deprivation) showed significance in the risk adjustment models, but did not
provide much added explanatory power
◦ There is a high degree of correlation between sociodemographic characteristics and other
measures included in the risk adjustment (e.g., insurance type, rurality, clinic type)
The overall effect of sociodemographic characteristics, insurance status, and other
factors included in the model was modest
◦ Available measures may imperfectly capture social complexity and the differential effects of
poverty
Risk adjusting measures with sociodemographic characteristics caused some clinics to
move up in rank, others to move down, and more clinics to move to the middle
Complete sociodemographic characteristics was missing for many medical groups
◦ Cannot generalize to those medical groups
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
3
Research Questions and Data
03/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
4
Research Questions
1. What effect does using available patient socio-demographic factors and facility
characteristics to risk adjust clinic quality measures have on model performance and
clinic rankings?
2. How is risk adjustment performance affected by methodological and measurement
issues such as measurement completeness for socio-demographic factors and
methods for measure composition?
03/11/2016
RISK ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
5
Measures and Risk Factors to be Tested
Measures
Risk Factors
Risk Factors (cont.)
Optimal Diabetes Care
01/01/2014 – 12/31/2014 DOS
•
•
•
•
•
•
•
•
• Deprivation (inverse median
income, poverty,
unemployment, public
assistance, SNAP, single
mother families)*
• Clinic type
• Rurality
Optimal Vascular Care
01/01/2014 – 12/31/2014 DOS
Optimal Asthma Control –
Adult
07/01/2014 – 06/30/2015 DOS
Race
Hispanic ethnicity
Preferred language
Country of origin
Primary payer type
Gender
Age
Comorbidities
Colorectal Cancer Screening
07/01/2014 – 06/30/2015 DOS
Primary data source: Minnesota Community Measurement
* American Factfinder (Factfinder.census.gov) for period ending in 2014
03/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
6
Methods
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
7
Sample
Medical
Groups
Clinics
Patients
Optimal Diabetes Care 01/01/2014 – 12/31/2014 DOS
121
579
211,103
Optimal Vascular Care 01/01/2014 – 12/31/2014 DOS
119
579
84,624
Optimal Asthma Control – Adult 07/01/2014 – 06/30/2015 DOS
134
614
58,881
Colorectal Cancer Screening 07/01/2014 – 06/30/2015 DOS
133
624
997,223
Measures
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
8
Measure Definitions
Diabetes: The number of diabetes patients who met ALL of the following targets:
◦ The most recent HbA1c in the measurement period has a value less than 8.0.
◦ The most recent Blood Pressure in the measurement period has a systolic value of less than 140 and a diastolic value of
less than 90 (both values must be less than).
◦ Patient is currently a non-tobacco user.
◦ If the patient has a co-morbidity of Ischemic Vascular Disease, the patient is on daily aspirin OR an accepted
contraindication.
Vascular: The number of IVD patients who met ALL of the following targets:
◦ The most recent Blood Pressure in the measurement period has a systolic value of less than 140 and a diastolic value of
less than 90 (both values must be less than).
◦ Patient is currently a non-tobacco user.
◦ Patient is on daily aspirin OR an accepted contraindication (any date).
Asthma: The number of asthma patients who meet both of the following targets:
◦ Asthma well-controlled.
◦ Patient not at elevated risk of exacerbation: The total number of emergency department visits and hospitalizations due to
asthma, as reported by the patient, are less than two occurrences.
Colorectal: The number of patients who were up-to-date with appropriate colorectal cancer screening exams.
◦ Appropriate exams include colonoscopy, sigmoidoscopy, or fecal blood tests.
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
9
Measures
MNCM/SQRMS Measures and Clinic Registry
◦ Sociodemographic characteristics (Race, ethnicity, primary language, country of origin, deprivation)
◦ Patients could select up to four race and ethnicity categories – the models include an indicator for each category a patient selected
Clinic types (Federally Qualified Health Center (FQHC), Critical Access Hospital (CAH))
Clinic rurality indicators (Urban, Micropolitan, Small Town, Frontier)
Deprivation
◦ Constructed using data from American Factfinder (Factfinder.census.gov) for period ending in 2014
◦ Data from Zipcode Tabulation Areas (ZCTA)
◦ Measures
◦
100 - median household income as percent of maximum median household income
◦
Percent Public Assistance
◦
Percent SNAP
◦
Percent Poverty
◦
Percent Unemployed
◦
Percent Single Female with Children
◦ Psychometrics: Cronbach Alpha (standardized measures) = .79, percent of common variance explained = .51
◦ Based on ideas in Singh, Gopal K. 2003. "Area Deprivation and Widening Inequalities in Us Mortality, 1969–1998." American Journal of Public Health
93:1137-1143.
Distance from patient’s zip-code centroid to clinic’s zip-code (measures access or risk selection)
◦ Indicators for same zip code, within 10 miles, 10 to 20 miles, more than 20 miles
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
10
Methods
Missing Data Analysis
◦
◦
◦
◦
Missing data affects which clinics the results generalize to
Systematically missing data (e.g., missing data by medical group) may bias estimates
The analysis assesses the potential magnitude of these issues
Tabulation of frequency of missing RELO data by clinic and medical group
Comparable/Overlapping Groups
◦ Clinics not serving comparable can make clinic level comparisons difficult
◦ If a clinic does not have disadvantaged patients, assessing how that clinic would disadvantaged patients cannot be done
◦ Pooled data over all conditions to increase measurement reliability and minimize issue of
conflicting results by condition
◦ Cluster clinics (SAS Proc Cluster Wards Method) in terms their similarity in proportion of
patients who
◦ Had MHCP, Commercial, Medicare, Self-Pay/Uninsured insurance
◦ Were Asian, American Indian / Pacific Islander, Black, Hispanic, White
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
11
Risk Adjustment
Why do risk adjustment?
◦ Measure provider performance while reducing the effect of causes of performance outside the
control of providers
Proportion of variance explained by RELO and other factors
◦ General Linear Models (SAS Proc GLM)
Expected values (logistic regression used because dependent variable is binary)
◦ Logistic with no corrections for patients clustered by clinic or greater dispersion in the error than
expected (SAS Proc Glimmix)
◦ Logistic with no corrections for patients clustered by clinic and correction for greater dispersion in the
error than expected (SAS Proc Glimmix)
Assessing model fit
◦ Area under receiver operating characteristic curve (AUROC)
◦ Plot
◦ True positive rate (sensitivity = p(high quality | predicted high quality)) against
◦ False positive rate (1 – specificity where specificity = p(low quality | predicted low quality))
◦ 1 is perfect prediction, .5 is a coin toss
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
12
Categorizing Performance
Observations about performance measurement
◦ Compare performance of each clinic to average performance of all clinics and categorize outliers (low and high
performance)
◦
◦
◦
◦
◦ Alternative is to compare all clinic pairs
Reliability of performance measurement is a function of the number of patients observed.
Listing by percentage or caterpillar plots can result in inappropriate inferences
Risk adjust for greater dispersion in the error than expected
Spiegelhalter, David J. 2005. "Funnel Plots for Comparing Institutional Performance." Statistics in Medicine
24:1185-1202; Spiegelhalter, D J. 2005. "Handling over-Dispersion of Performance Indicators." Quality and Safety
in Health Care 14:347-351.
Issues
◦ Excluded clinics with < 30 non-missing RELO patients
◦ When performance measures do not take a standard distribution (binomial, normal, poisson), assuming that
distribution can result in incorrect categorization
◦ Recommendation is to use bootstrap techniques from population to obtain empirical confidence limites
◦ Neuburger, Jenny, David A Cromwell, Andrew Hutchings, Nick Black, and Jan H van der Meulen. 2011. "Funnel Plots for Comparing Provider
Performance Based on Patient-Reported Outcome Measures." BMJ Quality & Safety 20:1020-1026.
Confidence interval bounds match commonly used quality improvement confidence interval bounds
◦ 95% (~2 standard deviations), 99.8% (~3 standard deviations)
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
13
Results
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
14
Missing Data – Medical Group Level
ALL RACE / ETHNICITY DATA MISSING
SOME RACE / ETHNICITY DATA MISSING
#
Medical
Groups
#
Patients
#
Missing
Race
Average
Medical
Group
Size Patients
#
Medical
Groups
#
Patients
(All
Missing
Race)
Diabetes1
95
177,510
5,510
1,869
26
33,593
11,547
1,292
18.5
Vascular2
93
71,990
1,592
774
26
12,634
3,790
486
16.8
Asthma3
108
50,531
1,549
468
26
8,350
3,177
321
16.8
Colorectal4
107
846,706
22,836
7,913
26
150,517
48,512
5,789
17.4
1.
2.
3.
4.
Maximum Average TOTAL
%
Number Medical
of
Group Missing
Race
Missing
Size Patients Patients
There were also medical groups with 49% and 83% missing RELO.
There were also medical groups with 87% and 63%missing RELO.
There were also a medical group with 50%missing RELO.
There were also medical groups with 58% and 98%missing RELO.
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
15
Comparable / Overlapping Clinic
Clustering Characteristics
CLUSTER
1
CLUSTER
2
CLUSTER
3
CLUSTER
4
CLUSTER
5
N=361
N=130
N=9
N=45
9
% COMMERCIAL
41%
65%
56%
23%
17%
% MEDICARE
39%
20%
16%
30%
19%
% MHCP
14%
9%
17%
40%
36%
% SELF-PAY / UNINSURED
2%
1%
2%
4%
25%
% ASIAN
1%
3%
62%
6%
2%
% AMER. INDIAN / PAC. ISL.
1%
1%
0%
4%
11%
% BLACK
1%
4%
4%
34%
18%
% HISPANIC
1%
1%
15%
5%
31%
% WHITE
96%
91%
2%
52%
52%
AVERAGE DEPRIVATION
-0.34
-0.51
-0.23
0.77
0.34
Percent of patients in
clinic who have / are:
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
16
Comparable / Overlapping Clinics
Clinic Characteristics
CLUSTER
1
CLUSTER
2
CLUSTER
3
CLUSTER
4
CLUSTER
5
N=361
N=130
N=9
N=45
9
% FQHC
2%
0%
11%
16%
56%
% CAH
10%
1%
0%
0%
0%
% URBAN
51%
98%
100%
93%
89%
% MICROPOLITAN
11%
2%
0%
2%
11%
% SMALL TOWN
13%
0%
0%
0%
0%
% FRONTIER
25%
0%
0%
4%
0%
Percent of patients in clinic
who have / are:
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
17
Comparable / Overlapping Clinic Analysis
Interpretation
Based on clinic patient mix characterized by sociodemographic characteristics and
insurance type, clinics cluster into groups that are consistent differences among clinic
types
The non-overlapping nature of some of these patient factors makes direct
standardization not feasible so indirect standardization must be used
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
18
Proportion of Variance Explained
(Sum of Squares, Percent of Total)
Sum of Squares
Due to All
Explanatory
Variables
Sum of Squares
Due to Clinics
(Fixed Effect)
Total Corrected
Sum of Squares
R-Squared
Diabetes
2,656
781 (.02)
31,600
.08
Vascular
665
223 (.02)
11,305
.06
Asthma
1,781
1,269 (.16)
8,133
.16
Colorectal
8,633
3,917 (.03)
121,755
.07
Measure
Estimated with a general linear model and linear probability model using SAS Proc GLM
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
19
Risk Adjustment
Effects of Sociodemographic Characteristics
Effect (Race Contrast is White)
American Indian
Asian
Black
Hispanic
Pacific Islander
Other
Non English Speaking
Non US Origin
Deprivation
Bold - p < .05
4/19/2016
Diabetes Vascular Asthma Colorectal
-0.63
-0.28
-0.36
-0.36
-0.13
0.16
-0.10
0.02
-0.38
-0.35
-0.44
-0.17
-0.17
0.14
-0.05
0.10
-0.46
-0.19
-0.49
-0.35
-0.25
-0.17
-0.07
-0.37
0.12
0.35
-0.04
-0.27
0.40
0.23
-0.03
-0.18
-0.09
-0.10
-0.05
-0.08
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
20
Risk Adjustment
Comorbidity Effects
Comorbidity
Diabetes
Type 1 Diabetes
-0.39
Vascular
-0.21
Diabetes
Vascular
Asthma
Colorectal
0.13
Depression
-0.20
-0.15
-0.06
Bold - p < .05
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
21
Risk Adjustment
Insurance Status, Rurality, and Distance Effects
MHCP (Commercial is Contrast for Insurance
Status)
Medicare (Duals)
Self/Uninsured
Distance to Clinic (Same zip is contrast)
Within 10 Miles
10 to 20 Miles
More than 20 Miles
Rurality (Urban is Contrast)
Micropolitan
Small Town
Frontier
Bold - p < .05
4/19/2016
Diabetes
Vascular
Asthma
Colorectal
-0.50
-0.18
-0.45
-0.68
-0.37
-0.77
-0.48
-0.48
-0.81
-0.53
-0.11
-1.19
-0.02
-0.04
-0.12
-0.03
0.01
-0.05
0.00
0.01
-0.02
0.02
0.00
-0.12
-0.16
-0.18
-0.29
-0.14
-0.26
-0.24
-0.64
-0.91
-1.40
-0.02
-0.17
-0.44
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
22
Model Performance Assessed With Area Under Receiver
Operating Characteristic Curve (AUROC)
INSURANCE
AND SDC
+ SDC
+ INSURANCE
+ INSURANCE
AND SDC
IMPROVEMENT
BY ADDING
SDC TO
OTHER
MEASURES
Diabetes
.608
.627
.626
.636
.009
Vascular
.586
.599
.602
.609
.010
Asthma
.572
.623
.621
.639
.018
Colorectal
.552
.589
.596
.607
.011
W/O
MEASURE
SDC = Sociodemographic characteristics (race, ethnicity, language, country of origin, deprivation)
AUROC = Area under receiver operating characteristic curve (.50 = coin toss, 1=perfect prediction)
RELO = Race, Ethnicity, Language, Country of Origin, Deprivation
Why don’t sociodemographic characteristics add much explanatory power? Collinearity?
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
23
AUROC Curves
Diabetes
4/19/2016
Vascular
Asthma
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
Colorectal
24
Changes in Performance Categories: Diabetes
Unadjusted Performance to Risk Adjusted Observed / Expected Performance Comparison
Risk Adjusted Observed / Expected Performance
Unadjusted
Performance
Much Below
Average
Below
Average
Average
Above
Average
Much Above
Average
Total
Much Below Average
0
0
0
0
0
0
Below Average
53
18
23
1
0
95
Average
1
12
143
8
5
169
Above Average
0
0
18
6
7
31
Much Above Average
0
0
8
17
28
53
Total
54
30
192
32
40
348
Estimated with residual
dispersion model
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
25
Changes in Performance Categories: Diabetes
Interpretation
Did risk adjustment reduce the number of outliers (over-dispersion)? Yes.
◦ 95 worse than average without risk adjustment, 84 (54+30) worse than average with risk adjustment
(but 54 moved to much worse than average)
◦ 84 (31+53) better than average without risk adjustment 72 (32+40) better than average with risk
adjustment
◦ 169 average without risk adjustment, 192 average with risk adjustment
What was correlated with performance categorization changes?
◦ More FQHCs than expected decreased and increased; fewer FQHCs than expected stayed the
same
◦ Fewer CAHS than expected decreased and more CAHS than expected increased
◦ More urban clinics than expected to decreased. More Small Town and Frontier clinics than expected
increased.
◦ Clinics in Cluster 4 (high MHCP, high black, high deprivation) were more likely to increase
◦ Clinics in Cluster 5 (high MHCP, self-pay, uninsured, high Hispanic) were more likely to decrease
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
26
Changes in Performance Categories: Vascular
Unadjusted Performance to Risk Adjusted Observed / Expected Performance Comparison
Risk Adjusted Observed / Expected Performance
Average
Above
Average
Much
Above
Average
Total
0
0
0
0
0
23
16
17
0
0
56
Average
0
5
169
7
1
182
Above Average
0
0
13
9
1
23
Much Above Average
0
0
5
6
8
19
Total
23
21
204
22
10
280
Much
Below
Average
Below
Average
Much Below Average
0
Below Average
Unadjusted
Performance
Estimated with residual
dispersion model
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
27
Changes in Performance Categories: Vascular
Interpretation
Did risk adjustment reduce the number of outliers (over-dispersion)? Yes.
◦ 56 worse than average without risk adjustment, 44 (23+21) worse than average with risk
adjustment (but 23 moved to much worse than average)
◦ 42 (23+19) better than average without risk adjustment, 32 (22+10) better than average with
risk adjustment
◦ 182 average without risk adjustment, 204 average with risk adjustment
What was correlated with performance categorization changes?
◦ No FQHC effect
◦ Fewer CAHS than expected decreased and more CAHS than expected increased
◦ More urban clinics than expected to decreased. Fewer Small Town and Frontier clinics than
expected decreased. More Small Town and Frontier clinics than expected increased.
◦ Clinics in Cluster 4 (high MHCP, high black, high deprivation) were more likely to increase
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
28
Changes in Performance Categories: Asthma
Unadjusted Performance to Risk Adjusted Observed / Expected Performance Comparison
Risk Adjusted Observed / Expected Performance
Average
Above
Average
Much
Above
Average
Total
0
0
0
0
0
58
5
10
0
1
74
Average
0
4
55
14
13
86
Above Average
0
0
12
4
3
19
Much Above Average
0
0
18
19
32
69
Total
58
9
95
37
49
248
Unadjusted
Performance
Much Below
Average
Below
Average
Much Below Average
0
Below Average
Estimated with residual
dispersion model
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
29
Changes in Performance Categories: Asthma
Interpretation
Did risk adjustment reduce the number of outliers (over-dispersion)? Yes.
◦ 74 worse than average without risk adjustment, 67 (58+9) worse than average with risk
adjustment (but 58 moved to much worse than average)
◦ 88 (19+69) better than average without risk adjustment, 67 (58+9) better than average with risk
adjustment
◦ 86 average without risk adjustment, 95 average with risk adjustment
What was correlated with performance categorization changes?
◦ More FQHCs than expected decreased, fewer FQHCs than expected stayed the same or
increased
◦ Fewer CAHS than expected stayed the same and more CAHS than expected increased
◦ Urban clinics were less likely to increase. Micropolitan, Small Town and Frontier clinics were
more likely to increase.
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
30
Changes in Performance Categories: Colorectal
Unadjusted Performance to Risk Adjusted Observed / Expected Performance Comparison
Risk Adjusted Observed / Expected Performance
Average
Above
Average
Much
Above
Average
Total
0
0
0
0
0
131
19
27
1
4
182
Average
8
13
55
7
12
95
Above Average
0
2
15
0
5
22
Much Above Average
0
1
23
17
69
110
139
35
120
25
90
409
Unadjusted
Performance
Much Below Average
Below Average
Total
Much Below
Average
Below
Average
0
Estimated with residual
dispersion model.
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
31
Changes in Performance Categories: Colorectal
Interpretation
Did risk adjustment reduce the number of outliers (over-dispersion)?
◦ 182 worse than average without risk adjustment, 174 (139+35) worse than average with risk
adjustment (but 139 moved to much worse than average)
◦ 132 better than average without risk adjustment (22+110), 115 (90+25 better than average with
risk adjustment
◦ 95 average without risk adjustment, 120 average with risk adjustment
What was correlated with performance categorization changes?
◦ More FQHCs than expected decreased and fewer FQHCs than expected stayed the same
◦ Fewer CAHS than expected increased
◦ More urban clinics than expected increased. More Small Town and Frontier clinics than
expected increased
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
32
Discussion
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
33
Conclusions
The sociodemographic characteristics variables (i.e., race, ethnicity, language, country of
origin, deprivation) showed significance in the risk adjustment models, but did not provide
much added explanatory power
◦ There is a high degree of correlation between sociodemographic characteristics and other measures
included in the risk adjustment (e.g., insurance status, rurality, clinic type)
The overall effect of sociodemographic characteristics, insurance status, and other factors
included in the model was modest. This is consistent with levels of explanatory effects found
in the literature –
◦ Fung, Vicki PhD, Julie A. PhD Schmittdiel, Bruce M. A. Fireman, Aabed B. A. Meer, Sean M. D. Thomas, Nancy PhD Smider,
John M. D. M. B. A. Msce Hsu, and Joseph V. M. D. M. P. H. Selby. 2010. "Meaningful Variation in Performance: A Systematic
Literature Review" Medical Care 48:140-148.
Available measures imperfectly capture social complexity and the differential effects of
poverty:
◦ Explore whether measures that are more granular (e.g., geocoding to census block, person-level
income data) could add more explanatory power
◦ Explore whether measures such as housing insecurity, acculturation could add explanatory power
Risk adjusting measures with sociodemographic characteristics caused some clinics to move
up in rank, others to move down, and more clinics to move to the middle
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
34
Limitations
This presentation focuses on using sociodemographic characteristics for risk adjusting provider
performance and does not address other uses of sociodemographic characteristics data, such as for
reporting disparities related to sociodemographic characteristics
The analysis does not include measures that may measure other important outcomes prevalent in
disadvantaged populations
Missing data
Validating Outliers
◦ The analysis identifies low and high performers using a statistical methodology that is not validated by observation
of these outliers using independent measures of whether a clinic is an outlier.
Identifying Causes of More Outliers Than Expected (Overdispersion)
◦ “It could be argued that substantial over-dispersion is an adequate reason for dropping an indicator. If the indicator
is retained, then an effort should be made to understand the reasons for the variability and to adjust accordingly.
Nevertheless, situations will arise when there is still residual over-dispersion and it is desirable to have a simple
statistical method for estimating the degree of over-dispersion and adjusting the control limits.”
◦ Spiegelhalter, D J. 2005. "Handling over-Dispersion of Performance Indicators." Quality and Safety in Health Care 14:347-351.
◦ e.g., measure definition, accuracy and validity of measurement
◦ The cause that the methodology does not take into account clustering of patients within clinics is being explored by
implementing a two-stage bootstrap (patients sampled within clinics).
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
35
Thank you
Questions?
Contact: Doug Wholey, [email protected],
Health Policy & Management, School of Public Health,
University of Minnesota
4/19/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
36