Covariate Selection and Balance in
Propensity Score Methods
M Sanni Ali
University Medical Center Utrecht, the Netherlands
Outline
 Confounding
 Propensity Score (PS) Methods
 Covariate Selection
 Balance Diagnostics and their Applications
 Balance Assessment
 PS Methods in Time-varying Treatment
 Reporting of PS Analysis
Confounding
 Non-intermediate common cause of exposure
and outcome
 Unblocked backdoor path in causal diagrams
 Observed, unobserved, unknown
Propensity Score Methods
 Propensity Score
- Summary Score – Variables included?
- Balancing Score – Covariate balance?
 Often derived using logistic regression
- Selection of variables and forms
 Ensemble methods
- Automated – Interactions/polynomials
 Matching, stratification, weighting, and covariate
adjustment
Rubin & Rosenbaum 1983; Westreich et al 2010; Lee 2010
Advantages of PS
 Confidence on the causal inference
 Transparency/Easy to communicate
 Design a study separate from analysis
 Summary Scores – rare outcome
Covariate Selection
 Clinical knowledge - Sufficient?
 Variable with Instrumental Variable like properties
- Exacerbate the imbalance in unmeasured
confounders = Amplify the bias
 When could bias be amplified
- Unmeasured Confounding – Strong!!!
- Strong IV !!!
- Linear models
 Balance diagnostics are powerful tools
- Absolute standardized difference is robust
Bhattacharya & Vogt 2007; Pearl 2010&2011; Myers et al 2011; Austin 2009;
Belitser et al 2011; M S Ali et al 2014
Covariate Selection
 A simulation study using binary covariates,
treatment, and outcome.
 In different scenarios, confounding variables, risk
factors, instrumental variables, their interaction and
square terms were considered.
X9
X3
X8
A
Y
X6
X7
X1, X2 , X5
X4
M S Ali et al. 2014
Covariate Balance
 PS model versus covariate sets (balance)
 Four sets of covariates
- Full PS Model
- True PS Model
- Outcome PS model (Prognostic score)
- Confounder PS model
 Main terms versus Main + Interaction/squares
9
Covariate Selection
Description of the Different Propensity Score Models
Covariate Selection
 Treatment effects estimates (risk ratios) were
derived using Poisson models.
 PS model selection was made based on the balance
achieved on different sets of covariates, their
interaction/square terms.
 Covariate balance was assessed using the absolute
standardized difference.
 Covariate sets in balance assessment were
compared with respect to bias and precision of the
treatment-outcome relation as well as the PS model
selected.
11
Covariate Selection - Balance
Table 1. Balance of Covariates Measured Using Standardized Difference
When Different Sets of Covariates Were Included in the PS Model
Data was unmatched* and matched using Full PS model (PS_2), True PS model (PS_18), Risk Factor PS model (PS_25),
Confounder PS model (PS_11), Confounder PS model omitting confounder, X4 (PS_15), Confounder PS model omitting confounder,
X4, but risk factors were included (PS_29), Confounder PS model omitting confounder, X4, but instrumental variables were included
(PS_22), Confounder PS model omitting confounder, X4, but risk factors and instrumental variables were included (PS_8).
12
Covariate Selection - Balance
Table 2. Balance of Covariates Measured Using Standardized Difference
When Different Sets of Covariates Were Included in the PS Model
Data was unmatched* and matched using Full PS model (PS_2), True PS model (PS_18), Risk Factor PS model (PS_25),
Confounder PS model (PS_11), Confounder PS model omitting confounder, X4 (PS_15), Confounder PS model omitting confounder,
X4, but risk factors were included (PS_29), Confounder PS model omitting confounder, X4, but instrumental variables were included
(PS_22), Confounder PS model omitting confounder, X4, but risk factors and instrumental variables were included (PS_8).
13
Covariate Selection: balance
Table 3. Balance of Covariates Measured Using Standardized Difference
When Different Sets of Covariates Were Included in the PS Model
Data was unmatched* and matched using Full PS model (PS_2), True PS model (PS_18), Risk Factor PS model (PS_25),
Confounder PS model (PS_11), Confounder PS model omitting confounder, X4 (PS_15), Confounder PS model omitting confounder,
X4, but risk factors were included (PS_29), Confounder PS model omitting confounder, X4, but instrumental variables were included
(PS_22), Confounder PS model omitting confounder, X4, but risk factors and instrumental variables were included (PS_8).
Table 4.
Median (Interquartile Range, IQR) of Estimated Treatment
Effect Using Different PS Models in Different Scenarios
(True RR=1.75)
Model
All Covariates
were Independent
X4 and X5 were
Correlated
X4 and X7 were
Correlated
RR (IQR)
RR (IQR)
RR (IQR)
Crude
0.93
0.15
0.89
0.17
0.91
0.14
Full PS Model
1.73
0.40
1.78
0.68
1.72
0.59
True PS Model
1.71
0.47
1.73
0.63
1.67
0.58
Outcome PS Model
1.71
0.39
1.77
0.48
1.72
0.51
Conf PS Model
1.77
0.40
1.79
0.56
1.71
0.50
Omit.Conf PS Model 1*
1.55
0.34
1.57
0.40
1.46
0.35
Omit.Conf PS Model 2**
1.54
0.32
1.59
0.39
1.45
0.34
Omit.Conf PS Model 3†
1.50
0.36
1.50
0.50
1.51
0.47
Omit.Conf PS Model 4††
1.50
0.34
1.55
0.52
1.52
0.45
15
Covariate Selection
 PS matching improved balance of measured
covariates included in the PS model
 It exacerbated the imbalance in the unmeasured
covariate that was unrelated to measured covariates
 In choosing covariates for a PS model, the pattern of
association among covariates has substantial impact
on other covariates’ balance and the bias of the
treatment effect estimate.
Table 5. Median (IQR) Treatment Effect When Propensity Score Model
was Chosen Based on Balance Calculations on Different Sets of
Covariates (True RR =1.75)
Calliper
All
Covariates
Confounding
Factors
Confounding
and Risk
Factors
Confounding
and Treatment
Related Factors
0.05
1.51
0.42
1.77
0.64
1.75
0.62
1.48
0.43
0.10
1.50
0.43
1.75
0.63
1.75
0.58
1.48
0.42
0.15
1.50
0.44
1.73
0.67
1.76
0.65
1.48
0.43
0.20
1.48
0.42
1.75
0.62
1.75
0.60
1.46
0.43
0.25
1.46
0.43
1.75
0.61
1.72
0.58
1.43
0.42
0.30
1.47
0.43
1.74
0.62
1.72
0.60
1.43
0.42
0.35
1.44
0.41
1.69
0.61
1.69
0.57
1.41
0.39
0.40
1.40
0.41
1.68
0.59
1.67
0.61
1.37
0.39
0.50
1.38
0.36
1.68
0.54
1.63
0.50
1.35
0.35
0.60
1.33
0.37
1.58
0.51
1.55
0.52
1.31
0.34
*Balance was assessed on main terms as well as interaction/square terms
Table 6. Median (IQR) Treatment Effect When Propensity Score Model
was Chosen Based on Balance Calculations on Different Sets of
Covariates (True RR =1.75)
Calliper
All Covariates
Confounding
Factors
Confounding and
Risk Factors
Confounding and
Treatment Related
Factors
0.05
1.73
0.59
1.74
0.47
1.73
0.51
1.77
0.67
0.10
1.76
0.64
1.78
0.52
1.75
0.54
1.75
0.62
0.15
1.72
0.68
1.75
0.50
1.73
0.54
1.75
0.63
0.20
1.74
0.60
1.74
0.50
1.74
0.45
1.74
0.61
0.25
1.70
0.65
1.74
0.52
1.71
0.50
1.72
0.61
0.30
1.71
0.59
1.72
0.53
1.72
0.53
1.71
0.58
0.35
1.67
0.56
1.70
0.48
1.68
0.47
1.68
0.58
0.40
0.50
1.67
1.60
0.60
0.53
1.70
1.65
0.49
0.49
1.68
1.62
0.48
0.48
1.66
1.61
0.56
0.55
0.60
1.54
0.48
1.65
0.45
1.64
0.46
1.53
0.50
*Balance was assessed only main terms not interaction/square terms
Table 7. Median Number of Subjects Included When Propensity Score
Model was Chosen Based on Balance Calculations on Different
Sets of Covariates (True RR =1.75, n=2000)
Confounding and
Risk Factors
Confounding and
Treatment Related
Factors
Calliper
All Covariates
Confounding
Factors
0.05
1388
1388
1388
1388
0.10
1390
1389
1389
1390
0.15
1391
1391
1391
1391
0.20
1393
1392
1392
1393
0.25
1394
1394
1394
1394
0.30
1397
1396
1396
1397
0.35
1397
1398
1398
1398
0.40
1401
1401
1401
1401
0.50
1408
1408
1406
1408
0.60
1415
1417
1415
1415
*Balance was assessed main and interaction/square terms
Table 8. Median Number of Subject Included When Propensity Score
Model was Chosen Based on Balance Calculations on Different
Sets of Covariates (True RR =1.75, n=2000)
Confounding and
Risk Factors
Confounding and
Treatment Related
Factors
Calliper
All Covariates
Confounding
Factors
0.05
1388
1481
1479
1388
0.10
1390
1481
1479
1389
0.15
1392
1479
1478
1391
0.20
1393
1479
1479
1392
0.25
1393
1479
1478
1393
0.30
1396
1480
1480
1396
0.35
1398
1482
1481
1397
0.40
1401
1484
1484
1401
0.50
1407
1488.5
1489
1407
0.60
1415
1496
1496
1415
*Balance was assessed only main terms not interaction/square terms
PS Model Selection
 When PS model was chosen based on balance
calculation on confounding variables, the confounder
PS is often selected followed by the Risk factor PS
models.
 When balance calculation involves all covariates or
treatment related covariates, the full PS and true PS
model are often selected compared to the
confounder PS and Risk factor PS models.
PS Model Selection

In selecting PS model based on covariate balance,
the choice of covariates, interaction/square terms
in balance calculation has substantial impact on
bias and precision of the treatment effect.

PS model selection based on the balance achieved
on confounding variables, risk factors and
important interaction terms among confounders
and risk factors is optimal approach.
Balance Diagnostics
Quantitative Falsification of Instrumental Variable Assumptions
 Instrumental Variable Methods
three Basic IV assumption
- Associated with Exposure
- Independent of Confounders
- Independent of outcome except through exposure
 Association of IV with observed confounders could be
quantified with balance metrics
M S Ali et al 2014
IV Methods and assumptions
PS and time-varying treatment
 In the presence of treatment switching or
channeling - “Intension to treat analysis”?
 Time-varying Cox model
or
 Time-varying propensity score
or
 Inverse probability weighting or others
Time-varying treatment
Example
Associations Between Current SSRI Use and
the Risk of Hip Fracture
 Study using two EU databases (Mondriaan and
BIFAP)
 A cohort of patients with a first prescription for
Antidepressant (SSRI or tricyclic AD, TCA) period
2001-2009
 Data sources (GP databases) :
- the Dutch Mondriaan
- the Spanish BIFAP
M S Ali et al 2015
Time-varying treatment

Cox models with time-varying coefficients to
control for time-varying nature of covariates
SSRIt-1
Benzot-1
SSRIt
Benzot
HF
Time-varying treatment

When time-varying confounders are themselves
affected by the previous treatment (SSRI use at
t-1), conventional time-varying Cox model gives
biased estimate
Adjusting-away part of treatment effect?
SSRIt-1
Benzot-1
SSRIt
Benzot
HF
Time-varying treatment

In the presence of unmeasured common causes
of confounders and outcome (U):
collider-stratification bias?
SSRIt-1
Benzot-1
SSRIt
HF
Benzot
U
Time - varying treatment
Table 9. Associations Between SSRI Use and the Risk of Hip Fracture
Using Time-Varying Cox Models
Mondriaan
Adjusted for
BIFAP
HR
95% CI
HR
95% CI
Crude
1.75
1.12, 2.72
2.09
1.89, 2.32
Gender
1.73
1.10, 2.69
2.07
1.87, 2.30
Gender +Age
2.36
1.51, 3.68
1.51
1.37, 1.68
Gender +Age + TCAt
2.59
1.63, 4.12
1.56
1.40, 1.73
Gender +Age + TCAt + Benzot
2.60
1.63, 4.16
1.54
1.38, 1.71
All Confounders*
2.62
1.63, 4.19
1.52
1.37, 1.69
Time-varying treatment
Table 10. Associations Between SSRI Use and the Risk of Hip Fracture
Using Propensity Score Based Cox Analyses
Mondriaan
Adjusted for
HR
95% CI
HR
95% CI
1.75
1.12, 2.72
2.09
1.89, 1.71
Quintiles
2.64
1.63, 4.25
1.54
1.39, 1.71
Deciles
2.72
1.63, 4.54
1.53
1.38, 1.70
2.82
1.63, 4.25
1.61
1.45, 1.78
Crude
PS*
Stratification
PS* Adjustment
BIFAP
Time-varying treatment
Table 11. Associations Between SSRI Use and the Risk of Hip Fracture
Using Propensity Score Based Cox Analyses
Mondriaan
Adjusted for
Without
Accounting
for
Censoring*
Accounting
for
Censoring**
BIFAP
HR
95% CI
HR
95% CI
Crude
1.69
1.05, 2.67
2.14
1.91, 2.39
Gender
1.68
1.06, 2.67
2.11
1.89, 2.38
Gender +Age
2.46
1.55, 3.99
1.54
1.37, 1.72
Crude
1.73
1.08, 2.77
2.05
1.83, 2.30
Gender
1.71
1.07, 2.74
2.03
1.81, 2.28
Gender +Age
2.47
1.53, 3.98
1.51
1.35, 1.70
* Only inverse probability of treatment weights were used
** Combined inverse probability of treatment and censoring weights were used
+ Trimming at 1% and 99%
Time-varying treatment
 Differences between the various methods to adjust
for time-dependent confounding (i.e., time-varying
Cox and PS as well as inverse probability weighting of
marginal structural models) were small.
 The observed differences in treatment effects
estimates between the datasets are likely attributable
to different confounding information in the datasets.
 Adequate information on (time-varying) confounding.
Conduct and Reporting PS Analysis:
Literature review
388 articles
PubMed search:
92 Articles were excluded:
296 Articles available for review
63
20
6
2
1
Non clinical
Methodological
Other language
Systematic reviews
Editorials/letters
Drug-related intervention: 108 (36.5%)
Clinical*
50 (16.9%)
Surgical intervention:
138 (46.6%)
Figure 1. Flow chart of abstracts or articles extraction for the systematic review
M S Ali et al 2015
Reporting PS Analysis
 PS matching is the commonly used approach.
 Balance is often checked with PS matching followed
by inverse probability weighting
 P-values are the most commonly used tests.
 Absolute standardized difference was used only in
25% of the studies in which balance was assessed.
M S Ali et al 2015
Reporting PS Analysis
 Transparency in conducting the analysis
 What needs to be reported??
 Guidelines: STROBE or ENCePP
M S Ali et al 2015
Variable Selection for PS Model
- Method used for variable selection
- Whether empirical knowledge was considered
- Whether variable’s association with treatment
and/or outcome
Propensity Score Estimation
-
Method used to estimate the PS, e.g. logistic
regression
Variables and/or interaction terms included in the
PS
PS Matching
- Matching algorithm, caliper width, and
matching ratio used.
- Whether matching was with replacement.
and matched nature of the data accounted
in the analysis.
- Number of patients at the start and after
matching.
- Number and characteristics of excluded
patients (versus matched ones)
- The distribution of baseline characteristics
between treated and control patients
in the matched and starting population.
-Balance of Covariates between matched
groups
Applying the PS methods
♠
-
Type of PS method employed.
PS Adjustment
-
Overlaps of the PS distribution between
treatment groups. *
Whether linear relationship was checked
between outcome and the PS.
Balance Assessment
-
- Balance measure used.
Quantifying the balance and whether imbalance on
covariates was detected after the final PS model.
Treatment Effect Estimation
- Statistical method used.
- Whether additional adjustment was made for
covariates.
- Whether sensitivity analysis was performed.
Interpretation of Effect estimate
-
The interpretation of the effect estimates in
relation to the research question, target
population & type of PS method used (ATE,
ATT)
PS Stratification
- The quantile used for stratification.
- The overlap of PS with quantiles of PS using
plots or balance measures.
- Balance of covariates with in quintiles of the
PS.
PS Weighting
- The range (mean, max, min) of unstabilzed
and stabilized weights.
- Variables included in the PS models for
-
both numerator and denominator of the
weights.
Whether weights were truncated and method
used.
Balance of Covariates in the weighted
sample groups.
M Sanni Ali 2015
Limitations of PS
 Unmeasured Confounding
 Treatment Modification/Interaction
 Multilevel Treatment Modeling
 Dynamic Prescription Patterns/rare Exposure
 Pooling Data on PS? Treatment guidelines
Alternatives: Disease Risk Score?
Acknowledgement






Olaf H. Klungel, PharmD, PhD1,2
Anthonius de Boer, MD, PhD1
Rolf HH Groenwold, MD, PhD1,2
Arno W. Hoes, MD, PhD2
Svetlana V. Belitser, MSc1
Kit C.B. Roes, PhD2
1 Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical
Sciences, Utrecht University, Utrecht.
2 Julius Centre for Health Science and Primary Care, University Medical Centre Utrecht, Utrecht.
Thank You!