Methodological
Issues
Presentation by Ian Brownwood for the
meeting of Health Promotion, Prevention
and Primary Care Subgroup 22 October,
2009, Paris
The Indicators
•
•
•
•
•
•
Asthma*
Chronic Obstructive Pulmonary Disease*
Congestive Heart Failure*
Angina
Hypertension*
Diabetes
– Short -term complications*
– Long -term complications
– Lower -extremity amputation*
– Uncontrolled
* To be published in the 2009 edition of OECD Health at a Glance
Their Nature
• Based on data from routine hospital
administrative information systems.
• Conditions are considered amenable or
sensitive to care provided in primary care.
• Lower values are intended to be reflective
of effective primary care systems.
• Does not imply that indicator values of
zero can be achieved.
Their Definition
• AHRQ definitions except:
– Aged 15+ instead of 18+ years
– Exclusion of day cases.
– Mapping between ICD9 and ICD10.
– Procedure coding
• Based on PDX codes, except for:
– Asthma (i.e. cystic fibrosis and anomalies of the
respiratory system)
– Lower-extremity amputation (i.e. trauma)
– COPD (i.e. qualification of bronchitis)
Validity and Comparability
Key issues considered:
1.
2.
3.
4.
5.
Potential confounding factors
Data coverage of the hospital sector
Reporting of meaningful composites
Indicator instability in small populations
Indicator-specific issues:
•
•
•
Technical specification of indicators
Cross walk of ICD 9 and ICD 10 codes
Country variations in coding practice/capacity
Confounding Factors
• Potential confounding factors
– Demand  Prevalence
– Supply  Hospital Expenditures, Staff, Beds,
Bed days and Admissions
• Potential explanatory factors
– GP per 100,000
• Preliminary examination during HCQI
data collection for 2008-09.
WE DID FIND…..
Demand
Demand
Supply
Supply
Explanatory
Explanatory
BUT THEN WE ALSO
FOUND…
Demand
Supply
Explanatory
Conclusions
• Some evidence of relations but not strong
or consistent.
• Data limitations:
– Invalidated country reported prevalence rates.
– Small sample size
– Temporal issues for diabetes long-term
complications.
• Some of the key findings reflected in 2009
edition of Health at a Glance.
Key Issues
1.
2.
3.
4.
5.
Potential confounding factors
Data coverage of the hospital sector
Reporting of meaningful composites
Indicator instability in small populations
Indicator-specific issues:
•
•
•
Technical specification of indicators
Cross walk of ICD 9 and ICD 10 codes
Country variations in coding practice/capacity
Hospital Coverage
• Denominator =Population not Admissions
(i.e. not same unit of measurement).
• Sensitive to coverage of hospital system.
• Not all countries provided the additional
data (e.g. UK could be 10% understated).
• Adjustment of rates may result in slight
overestimation in some cases (i.e. 40% of
countries indicated they thought the
public rate would be higher).
Adjusted Rates
Conclusions
• Without adjustment potential exists for
underestimation of rates in some countries.
• Incomplete additional data for 2009-10.
• Unadjusted rates reported in 2009 edition
of Health at a Glance.
• Reconsider application of adjusted rates
based on enhanced data in 2010-11.
Key Issues
1.
2.
3.
4.
5.
Potential confounding factors
Data coverage of the hospital sector
Reporting of meaningful composites
Indicator instability in small populations
Indicator-specific issues:
• Technical specification of indicators
• Cross walk of ICD 9 and ICD 10 codes
• Country variations in coding
practice/capacity
Composite
What is the objective?
1. To explain the variation in the individual
indicators through the use of one or more
composite indicators?
2. To indicate the performance of the whole
system through the aggregation of indicators
that relate to parts of the system?
Some countries currently report composite
indicators (e.g. Canada, Australia).
AHRQ Findings
•
•
•
•
Indicators are positively correlated

Positive factor loadings on the 1st factor 
1st factor explains 94% of variation

Indication for separate diabetes composite
(high factor loading on the 2nd factor)

HCQI Findings
• Not all indicators are positively correlated
(e.g. angina and diabetic amputations)
×
• Factor loading for amputation is negative ×
• 1st factor only explains 31% of variation ×
• No clear indication for a 2nd factor
×
Preliminary HCQI analysis does not
provide clear support for use of composite.
Correlation
| Asthma
COPD
CHF
Angina
Hyper
Diab ST Diab LT Diab UC DiabLEA
-------------+-----------------------------------------------------------------------------------Asthma
|
1.0000
COPD
|
0.1812
1.0000
CHF
|
0.1559
0.0152
1.0000
Angina
| -0.1658
0.4965
0.1733
1.0000
Hypertension |
0.0627
0.2765
0.4532
0.3830
1.0000
Diabetes ST |
0.5346
0.2123
0.2886 -0.2476 -0.0903
1.0000
Diabets LT
|
0.1367
0.1675
0.0892 -0.0779
0.5676
0.1674
1.0000
Diabetes UC |
0.1403
0.2929
0.1552
0.2508
0.8065
0.0226
0.5757
1.0000
Diabetes LEA |
0.0692 -0.2891
0.3810 -0.3284 -0.1404
0.2650
0.2201 -0.3302
1.0000
Rotated factor loadings (pattern matrix) and unique variances
------------------------------------------------Variable | Factor1
Factor2 |
Uniqueness
-------------+--------------------+-------------Asthma
|
0.1936
0.5950 |
0.6085
COPD
|
0.5645
-0.1366 |
0.6627
CHF
|
0.3596
0.4773 |
0.6429
Angina
|
0.5136
-0.5182 |
0.4677
Hypertension |
0.9054
0.0088 |
0.1802
Diabetes ST |
0.0829
0.7547 |
0.4235
Diabetes LT |
0.6062
0.3689 |
0.4965
Diabetes UC |
0.8721
-0.0298 |
0.2385
Diabetes LEA | -0.2611
0.6924 |
0.4524
------------------------------------------------Proportion
0.3080
0.2284
Key Issues
1.
2.
3.
4.
5.
Potential confounding factors
Data coverage of the hospital sector
Reporting of meaningful composites
Indicator instability in small populations
Indicator-specific issues:
•
•
•
Technical specification of indicators
Cross walk of ICD 9 and ICD 10 codes
Country variations in coding practice/capacity
Indicator Stability
• Inherent instability in annual indicator
values (high random error) associated with
low numerator values  indicates use of
multi- year averaging or aggregation of
numerator cases.
• Particular issue for smaller states (e.g.
Iceland and Luxembourg).
• Use of the 3-year rolling average would
facilitate comparison with other countries.
Key Issues
1.
2.
3.
4.
5.
Potential confounding factors
Data coverage of the hospital sector
Reporting of meaningful composites
Indicator instability in small populations
Indicator-specific issues:
•
•
•
Technical specification of indicators
Cross walk of ICD 9 and ICD 10 codes
Country variations in coding practice/capacity
Asthma
• Differential diagnosis of asthma/COPD.
• Code coverage issue between ICD 9 and ICD 10:
– Coding guidelines indicate that ICD 9 4932 equates
to ICD 10 J448.
– Supplementary data indicates that chronic
obstructive asthma (4932) accounts for 10-50% of
asthma numerator cases (n=4).
– Plan to remove J448 from COPD and add to Asthma
codes to align with ICD 9 code coverage for future
data collections.
Asthma and COPD
Asthma
Other Specified
COPD (ICD 10 J448)
COPD
(Chronic
Bronchitis)
Chronic Obstructive
Asthma (ICD 9 4932)
COPD
(Pulmonary
Emphysema)
COPD
• Alternative calculation suggested by
Denmark:
– Aged 30+ years with J44 as PDX, and
– J44 as SDX and one of the following PDX:
• J96 (Respiratory Failure not elsewhere classified)
• J13-18 (Pneumonia)
– Reflect common diagnosis of patients with
COPD (i.e. PDX = Pneumonia).
Potential Issues
• Use of SDX is potentially problematic.
• Create inconsistency in the numerator and
denominator across indicators  impact
on ability to aggregate indicators.
Angina
• Calculation based on principal diagnosis.
• Supplementary data was collected during
HCQI data collection for 2008-09 to
explore variations in coding practices:
– Cases identified in SDX (any PDX).
– Cases identified in SDX (PDX of chest pain).
PDX and SDX
Explanation
• Secondary diagnosis coding
– Limited data availability for this analysis:
• Italy had the lowest number of SDX codes and
highest proportion of PDX cases.
• However, this relationship did not hold for other
countries.
• Other potential factors at play?
– E.g. Saver et al. (2009) attribute a sharp decline
in angina admissions in the 1990’s to more
aggressive diagnosis of coronary atherosclerosis.
Secondary Diagnosis
Discharge Trends
Saver et al. in Medical Care 2009:47, pp.1106-1110
Angina DX on Admission and
Discharge DX
Saver et al. in Medical Care 2009:47, pp.1106-1110
Hypertension
• Benign hypertension included in ICD 10
code list but not ICD 9 code list (i.e. 4011)
• Questions over validity  clinical grounds
for PDX (The Netherlands, NZ).
Lower Extremity Amputation
• Inclusion of toe amputation
– Interpretation in relation to quality of care
– Performed in hospital and ambulatory service
settings and on a day case and inpatient basis.
• Use of secondary diagnoses
– Up to 50% cases identified in SDX codes
– Availability of secondary diagnosis codes?
• Trauma code exclusions
– Adequacy of code list
Toe Amputation
• Reflects tertiary prevention
– Can reflect good quality of care as it prevents
further complications (OECD 2004)
– Failed primary and secondary prevention in
primary care system  retain in code set?
• Only multi-day stays reflected in data.
– If ambulatory care = day cases  retain in
code set?
Secondary Diagnosis
Secondary Diagnosis
Consider further data collection to locate where
cases are identified in SDX codes, similar to PSI
in 2008-09.
Options:
1. Limit calculation to:
– countries able to report above a lower threshold
number of SDX codes , and/or
– an upper threshold number of SDX codes
2. Apply adjustment methodology, similar to PSI
3. Limit the calculation to PDX only.
Trauma Exclusions
• Currently includes traumatic amputation
of toe, foot and leg (ICD 9 codes 895-897)
• There may be justification for extending
the exclusion code list.
– The Netherlands suggest inclusion of:
•
•
•
•
Fracture of specified bones of lower limbs (820 to 828)
Late effect of muscle & skin injury (905,906)
Crushing injury of lower limb & other sites (928, 929)
Burns of lower limbs & other sites (945, 946)
Uncontrolled Diabetes
• Potentially problematic for ICD 10 countries.
• 60% (11) countries reporting data in 2008-09
could identify uncontrolled cases:
– Just<50% by using ICD 9
– Just >50% by using ICD 10 markers.
• Austria, Canada, Finland, Korea, UK, Norway,
Poland and Sweden could not identify
uncontrolled cases.
• Canada establishing a marker for 2009+
Suggested Approach to Key Issues
1. Continue to examine potential confounding and
explanatory factors and report key findings with the
indicators in OECD publications.
2. Collect further data on hospital coverage from all
countries in 2010-11 and review approach.
3. Further analysis required before proceeding with the
use of composite indicators.
4. Present 3-year averages for Iceland and Luxembourg
in the future to address indicator instability.
5. Consider indicator-specific issues on a case by case
basis  next slide.
Indicator-Specific Issues
Indicator
Suggested Approach
Asthma
Revise ICD 10 code list (add J448)
Chronic Obstructive Pulmonary Disease
For discussion
Congestive Heart Failure
No action
Angina*
For discussion
Hypertension
Revise ICD 9 code list (add 4011)
Diabetes : Short –term complications
No action
Diabetes: Long –term complications*
For discussion
Diabetes: Lower -extremity amputation
For discussion
Diabetes: Uncontrolled*
For discussion
* Not currently considered for publication in 2009.
The Subgroup is invited to:
–comment on these methodological
issues
–endorse plans for further analysis and
refinement of the indicators in
preparation for the next round of data
collection in 2010-11.