Adjusting Quality Measures for Race,
Ethnicity, Language and Country of
Origin (RELO): A Proposal
D OU GL A S W H OL E Y1, M IC H A E L F IN C H 2, R OB K R E IGE R 1
M A R C H 11 , 2 0 1 6
1 C ENTER FOR C ARE O RGANIZATION R ESEARCH AND D EVELOPMENT, H EALTH P OLICY AND M ANAGEMENT, U NIVERSITY OF M INNESOTA
2 F INCH AND K ING
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
1
Presentation Overview
Research Questions and Data
Risk Adjustment Conceptual Issues
Methodological issues
Risk adjustment methodology
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
2
Research Questions and Data
03/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
3
Research Questions
1. What effect does using available patient socio-demographic factors and facility
characteristics to risk adjust clinic quality measures have on model performance and
clinic rankings?
2. How is risk adjustment performance affected by methodological and measurement
issues such as measurement completeness for socio-demographic factors and
methods for measure composition?
03/11/2016
RISK ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
4
Measures and Risk Factors to be Tested
Measures
Risk Factors
Optimal Diabetes Care
01/01/2014 – 12/31/2014 DOS
•
•
•
Optimal Vascular Care
01/01/2014 – 12/31/2014 DOS •
•
Optimal Asthma Control – Adult •
07/01/2014 – 06/30/2015 DOS •
Colorectal Cancer Screening
•
07/01/2014 – 06/30/2015 DOS
Race
Hispanic ethnicity
Preferred language
Country of origin
Primary payer type
Gender
Age
Comorbidities
Risk Factors (cont.)
• Median income*
• Clinic type
• Rurality
Data source: Minnesota Community Measurement
*Data source for median income is Primary Care Service Area Data from the Dartmouth Atlas of
Health Care (based on 2010 Census Tract) or Internal Revenue Service Individual Income Tax
Statistics - ZIP Code Data for 2013 (linked through patient zip-code)
03/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
5
Risk Adjustment: Contextualizing the
Analysis
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
6
What are the expectations for risk adjustment?
Risk adjusting mechanics
◦ Risk adjustment is not just a technical exercise, it must carefully consider the impacts to avoid
unintended consequences
◦ The adequacy of the performance of the risk adjustment may also be related to unintended
consequences
How do we interpret the effects of race, ethnicity, language, and country of origin?
How do we assess risk adjustment performance?
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
7
How can race, ethnicity, language and country of
origin affect quality?
Cumulative Complexity
◦ Patient workload-capacity imbalances are the central mechanism driving patient complexity
◦ Workload can be caused by social factors (economic insecurity, housing insecurity, food
insecurity, social support systems, medical complexity)
◦ When capacity for self-management decreases, providers will face a larger challenge in helping
patients reach care goals
◦ Shippee, Nathan D., Nilay D. Shah, Carl R. May, Frances S. Mair, and Victor M. Montori. 2012.
"Cumulative Complexity: A Functional, Patient-Centered Model of Patient Complexity Can
Improve Research and Practice." Journal of Clinical Epidemiology 65:1041-1051.
Patient Culture / Preferences
Care Organization / Bias
◦ Lack of fit of clinic capabilities (translators or social workers or behavioral health providers) with
population needs, lack of cultural fit, and bias
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
8
Understanding race, ethnicity, language and
country of origin effects
Ideally, we would have separate distinct measures for cumulative complexity, patient
culture/preferences, and care organization/bias
Race, ethnicity, language, and country of origin are potentially associated with all three
concepts
Using race, ethnicity, language, and country of origin as proxies for cumulative
complexity, patient culture/preferences, and bias/care organization risks risk adjusting for
factors both outside of and within the control of providers.
The analysis of RELO risk adjustment performance cannot address what specifically is
being risk adjusted for.
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
9
Methodological Issues in Risk Adjusting:
Issues to assess when doing risk adjustment
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
10
Missing Data and Risk Adjustment
In any profiling initiative, the quality of the data is more important than choice of
statistical models. Clear and concise definitions for data elements are exceedingly
important.
◦ Normand, Sharon-Lise T. and David M. Shahian. 2007. "Statistical and Clinical Aspects of
Hospital Outcomes Profiling." Statistical Science 22:206-226.
What is the pattern of missing data? Is it random or systematic? What are the
implications of the missing data pattern for risk adjustment RELO?
3/11/2016
RISK ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
11
Overlapping Groups
“there may be patients from one hospital for whom there are no comparable patients in the
other hospital (in causal inference parlance, there is no empirical counterfactual), and thus
no way to fairly compare performance in all patients cared for by the two hospitals. p. 96
“Thus, ‘risk-adjusted’ results derived using indirect standardisation cannot be used to directly
compare two hospitals unless their patient mix has been demonstrated to be similar (eg,
overlapping propensity score distributions).” p. 96
“Covariate imbalance is a common problem in profiling using observational data because
patients are not randomized (the method used to achieve covariate balance in clinical trials).
Standard regression-based adjustment may not completely address bias when there is
substantial lack of covariate balance.” P. 97
Shahian, David M and Sharon-Lise T Normand. 2015. "What Is a Performance Outlier?" BMJ
Quality & Safety 24:95-99. p. 96
How may non-overlapping groups be related to risk adjustment for RELO?
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
12
Case Mix Bias
“Despite excellent patient-level risk adjustment, substantial case mix bias (eg, due to
marked differences in the distributions of high and low risk cases between hospitals)
may be present and may impact performance estimates and outlier status.”
◦ Shahian, David M and Sharon-Lise T Normand. 2015. "What Is a Performance Outlier?" BMJ
Quality & Safety 24:95-99.
Case mix bias may exist because of risk adjustment has not adequately included
covariates or because of unobserved factors that affect the distribution of risk in clinics.
How may case mix bias affect RELO risk adjustment?
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
13
Overdispersion
Outliers and Over-Dispersion - “A problem can arise when a performance indicator
shows substantially more variability than would be expected by chance alone, since
ignoring such ‘‘over-dispersion’’ could lead to a large number of institutions being
inappropriately classified as ‘‘abnormal’’.”
◦ Spiegelhalter, D J. 2005. "Handling over-Dispersion of Performance Indicators." Quality and
Safety in Health Care 14:347-351.
Overdispersion results in type 1 errors (false positives) – clinics being identified as
performance outliers when they may not be
Overdispersion can occur because of omitted variables or unobserved risk.
Is there over-dispersion in measures that are not risk adjusted? How much does risk
adjustment reduce over-disperson? Is there over-dispersion remaining after risk
adjustment?
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
14
Small Sample Size
“Small sample sizes are common in provider profiling, and this makes it difficult to
reliably differentiate hospital performance and classify outliers.”
Sample size affects the reliability and precision of quality measurement.
◦ Shahian, David M and Sharon-Lise T Normand. 2015. "What Is a Performance Outlier?" BMJ
Quality & Safety 24:95-99.
◦ Spiegelhalter, David J. 2005. "Funnel Plots for Comparing Institutional Performance." Statistics
in Medicine 24:1185-1202.
How may small sample sizes affect risk adjustment for RELO and identifying outliers?
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
15
Methods for Risk Adjusting for Race,
Ethnicity, Language, and Country of Origin
and Other Factors
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
16
Proposed Models and Analysis
Direct standardization
◦ By RELO and insurance type; Age and gender categories can be added; Diabetes type for diabetes.
Indirect Standardization
◦ Regress quality on RELO, insurance, age, gender, comorbidities
◦ Regress quality on RELO, insurance, age, gender, comorbidities with a correction for over-dispersion
◦ Regress quality on RELO, insurance, age, gender, comorbidities with a random effect for clinic to
correct for over-dispersion
Compare Observed and Expected
◦ Observed / Expected
◦ Explore Observed – Expected to examine the degree that measure construction affects overdispersion (Thomas, J. W., K. L. Grazier, and K. Ward. 2004. "Economic Profiling of Primary Care
Physicians: Consistency among Risk-Adjusted Measures." Health Serv Res 39:985-1003)
Assess over-dispersion and performance outlier patterns
◦ Explore “shrinkage” methods for assessing performance
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
17
Thank you
Questions?
Contact: Doug Wholey, [email protected],
Health Policy & Management, School of Public Health,
University of Minnesota
3/11/2016
ADJUSTING QUALITY MEASURES FOR RACE, ETHNICITY, LANGUAGE AND COUNTRY OF ORIGIN
18