Online Supplement
Supplement 1: Prescription placed in patient chart ’11-’12, printed on yellow paper
1
Supplement 2: Laminated reminder placed in patient chart ’12-‘13
2
Supplement 3: Pink sticker placed on clinic visit medication reconciliation form ’12-‘13
3
Supplement 4: Influenza vaccination reminder placed in public patient areas
4
Supplement 5
A. Provider survey ’12-‘13
1) Did you find the laminated reminder for flu shot included in the patients’ charts/clinic desk helpful in
terms of prescribing the flu shot?
2) Did you find the laminated reminder for flu shot included in the patients’ charts/clinic desk helpful in
terms of addressing whether the patient has been vaccinated against influenza in the current flu season?
3) Were you aware of the Best Practice Alerts regarding influenza vaccination in MiChart?
4) Did you find the Best Practice Alerts embedded in MiChart helpful in terms of prescribing the flu shot?
5) Did you find the Best Practice Alerts embedded in MiChart helpful in terms of addressing whether the
patient has been vaccinated against influenza in the current flu season?
6) Do you believe that we should continue these intervention(s) at the next flu season, in order to prompt
providers to address influenza vaccination?
Supplement 6
C: Logistic model for effect of the intervention
The unit of analysis is each patient, and we posit a model for the probability of a previously unvaccinated
patient being vaccinated after his/her first appointment of the flu season. Make the following definitions.
Let V be the outcome indicating vaccination status. Let Yr be the year beginning each flu season (Yr =
2005,2006, … ,2012). Let Mo be a length-4 vector of indicators for the month of the patient’s first
appointment of the flu season, with Mo = (1, 0, 0, 0)T denoting September, Mo = (0, 1, 0, 0)T denoting
October, Mo = (0, 0, 1, 0)T denoting November, and Mo = (0,0,0,1)T denoting the remaining months of
the flu season (December, January, February, and March). Let Age be a length-3 vector of indicators of
20-year age groups, beginning at age 17, with the youngest group being the reference category. So, an
adult who is younger than 37 during the current flu season has Age = (0, 0, 0)T, an adult at least 37 but
younger than 57 has Age = (1,0,0)T , etc. The oldest age group contains all ages over 77. Let H1N1 be
an indicator variable for Yr = 2009, which was the H1N1 flu season. Finally, let Int indicate the presence
or not of the intervention, i.e. Int =1 when an eligible patient’s first appointment was after October 3, 2011
in the ’11-’12 flu season or after September 24, 2012 in the ’12-’13 season, which are the start dates of
the intervention (see Methods), and Int =0 for appointments before those dates as well as for all previous
flu seasons. A logistic model relating the covariate pattern
X = {Yr, Mo T, Age T, H1N1 , Int }T to the probability of vaccination is given by
Pr(𝑽 = 1|X )
log {
}
Pr(𝑽 = 0|X)
= 𝛼Yr + 𝜇 T 𝑴𝒐 + 𝛿 T 𝑨𝒈𝒆 + 𝛾 T 𝑴𝒐 × 𝑯𝟏𝑵𝟏 + 𝜃 T 𝑴𝒐 × 𝑰𝒏𝒕
≡ 𝛽 T 𝑿.
(1)
The coefficient α assumes a constant yearly change in the log-odds of being vaccinated. The vector µ
allows for the odds of vaccination to differ by month of an eligible patient’s first appointment of the
season. The intercept in (1) is implicitly in the vector µ, because there is no reference category in Mo.
The vector δ allows the odds of vaccination to change depending on age. The vector γ allows the odds of
vaccination to differ in each month of the H1N1 season. The vector θ is of primary interest and
determines whether there is any additional month-by-month change in the log-odds of being vaccinated
during the intervention period of the ’11-’12 and ’12-’13 flu seasons. Importantly, because the intervention
began on October 3 and September 24 for the ’11-’12 and ’12-’13 flu seasons, respectively, we
parameterized the model accordingly, and θ may only differentially affect the probability of vaccination
with appointments that were after these dates. A hypothesis test of H0: θ = (0, 0, 0, 0) T corresponds to a
formal test for an effect of the intervention.
5
Fitting (1) to our data, we have 𝜃̂ = (7.21, 1.64, −0.06, 1.76), which suggests that the intervention did
indeed increase the rate of vaccinations. The p-values for the first, second and fourth components of 𝜃̂,
corresponding respectively to patient’s first appointment being in September, October, or
December/January/February/March, are all less than 0.0001, and the p-value for the third component,
November, is 0.618. A joint test of θ = (0, 0, 0, 0) T is highly significant, with p<0.0001. Thus, we estimate
that the log-odds of vaccination increased during the ’11-’12 and ’12-’13 flu seasons during the months of
September, October, and December/January/February/March, in comparison to previous flu seasons,
and the probability of being vaccinated in the most recent two flu seasons is larger than what would be
expected if there were no effect of the intervention.
Supplement 7
D: Estimates and standard errors of the relative and absolute increase in vaccination rate
̂ (𝑽 = 1|𝑿, ) = [1 + exp(−𝛽̂ T 𝑿)]−1 , the model-based estimate of the probability an eligible
Define 𝑃(𝑿) = Pr
̂ (𝑽 = 1|𝑿 = 𝑿𝟎 ) , where 𝑿𝟎 is
patient with covariate pattern X being vaccinated. Also, define 𝑃0 (𝑿) = Pr
the vector 𝑿 with the intervention indicator Int constrained to be zero. In other words, 𝑃0 (𝑿) is the
estimated probability of an eligible patient with covariate pattern 𝑿 being vaccinated, had the intervention
not occurred. The quantities 𝑃(𝑿) and 𝑃0 (𝑿) are the primary ingredients for constructing Figure 3 of the
manuscript. Let 𝑛(𝑿) be the number of eligible patients with covariate pattern X. We estimate two
quantities. First, R11 is the relative change in the 2011 vaccination rate as a result of the intervention and
is be estimated by 𝑅̂11 = (∑𝑿𝒊:𝒀𝒓𝒊=2011 𝑛(𝑿𝒊 )𝑃(𝑿𝒊 ) / ∑𝑿𝒊:𝒀𝒓𝒊=2011 𝑛(𝑿𝒊 ) 𝑃0 (𝑿𝒊 )) − 1. The summation is over
all possible values of the covariate pattern X, constrained to the year 2011.Second, A11 is the absolute
increase in the 2011 vaccination rate as a result of the intervention and is estimated by 𝐴̂11 =
∑𝑿𝒊 :𝒀𝒓𝒊=2011 𝑛(𝑿𝒊 ) × (𝑃(𝑿𝒊 ) − 𝑃0 (𝑿𝒊 )) / ∑𝑿𝒊 :𝒀𝒓𝒊=2011 𝑛(𝑿𝒊 ). Similarly for 2012, we have 𝑅̂12 =
(∑𝑿 :𝒀𝒓 =2012 𝑛(𝑿𝒊 )𝑃(𝑿𝒊 ) / ∑𝑿 :𝒀𝒓 =2012 𝑛(𝑿𝒊 ) 𝑃0 (𝑿𝒊 )) − 1 and 𝐴̂12 = ∑𝑿 :𝒀𝒓 =2012 𝑛(𝑿𝒊 ) × (𝑃(𝑿𝒊 ) − 𝑃0 (𝑿𝒊 )) /
𝒊
𝒊
𝒊
𝒊
𝒊
𝒊
∑𝑿𝒊:𝒀𝒓𝒊=2012 𝑛(𝑿𝒊 ).
Standard errors can be estimated using the delta method and the multivariate central limit theorem. For
𝑅̂11, we estimate the confidence interval on the log-scale and then transform back to the natural scale.
We have
T
𝜕 ln(𝑅̂11 + 1)
𝜕 ln(𝑅̂11 + 1)
̂ (ln(𝑅̂11 + 1)) ≈ (
̂ (𝛽̂ ) (
Var
) Var
)
𝜕𝛽
𝜕𝛽
=(
∑𝑿𝒊:𝒀𝒓𝒊=2011 𝑃(𝑿𝒊 )[1 − 𝑃(𝑿𝒊 )] 𝑿𝒊 ∑𝑿𝒊 :𝒀𝒓𝒊 =2011 𝑃0 (𝑿𝒊 )[1 − 𝑃0 (𝑿𝒊 )] 𝑿𝒊
−
)
∑𝑿𝒊 :𝒀𝒓𝒊=2011 𝑃(𝑿𝒊 )
∑𝑿𝒊:𝒀𝒓𝒊=2011 𝑃0 (𝑿𝒊 )
T
−1
[𝑽(𝑿𝒊 ) − 𝑛(𝑿𝒊 )𝑃(𝑿𝒊 )]2
1
×
∑
(∑ 𝑛(𝑿𝒊 )𝑃(𝑿𝒊 )[1 − 𝑃(𝑿𝒊 )]𝑿𝒊 𝑿𝐓𝒊 )
#{𝑿𝒊 } − dim(𝛽)
𝑛(𝑿𝒊 )𝑃(𝑿𝒊 )[1 − 𝑃(𝑿𝒊 )]
𝑿𝒊
𝑿𝒊
∑𝑿 :𝒀𝒓 =2011 𝑃(𝑿𝒊 )[1 − 𝑃(𝑿𝒊 )] 𝑿𝒊 ∑𝑿𝒊:𝒀𝒓𝒊 =2011 𝑃0 (𝑿𝒊 )[1 − 𝑃0 (𝑿𝒊 )] 𝑿𝒊
×( 𝒊 𝒊
−
)
∑𝑿𝒊:𝒀𝒓𝒊=2011 𝑃(𝑿𝒊 )
∑𝑿𝒊 :𝒀𝒓𝒊=2011 𝑃0 (𝑿𝒊 )
The quantity 𝑽(𝑿𝒊 ) is the observed number of vaccinations at covariate pattern 𝑿𝒊 , and #{𝑿𝒊 } is the
̂ (𝛽̂ ) includes an estimate of an
number of unique covariate patterns. The expression for Var
overdispersion parameter to account for variability in the data beyond what is expected from the binomial
6
distribution.1 Using this approximation, a 95% confidence interval for 𝑅̂11 is given by exp {ln(𝑅̂11 + 1) ±
̂ (ln(𝑅̂11 + 1))} − 1. The calculation is analogous for 𝑅̂12. For, 𝐴̂11 , we have
1.96√Var
̂ (𝐴̂11 ) ≈ (
Var
T
𝜕𝐴̂11
𝜕𝐴̂
̂ (𝛽̂ ) ( 11 )
) Var
𝜕𝛽
𝜕𝛽
∑𝑿 :𝒀𝒓 =2011 𝑛(𝑿𝒊 )(𝑃(𝑿𝒊 )[1 − 𝑃(𝑿𝒊 )] − 𝑃0 (𝑿𝒊 )[1 − 𝑃0 (𝑿𝒊 )]) 𝑿𝒊
=( 𝒊 𝒊
)
∑𝑿𝒊 :𝒀𝒓𝒊=2011 𝑛(𝑿𝒊 )
T
−1
[𝑽(𝑿𝒊 ) − 𝑛(𝑿𝒊 )𝑃(𝑿𝒊 )]2
1
×
∑
(∑ 𝑛(𝑿𝒊 )𝑃(𝑿𝒊 )[1 − 𝑃(𝑿𝒊 )]𝑿𝒊 𝑿𝐓𝒊 )
#{𝑿𝒊 } − dim(𝛽)
𝑛(𝑿𝒊 )𝑃(𝑿𝒊 )[1 − 𝑃(𝑿𝒊 )]
𝑿𝒊
×(
𝑿𝒊
∑𝑿𝒊:𝒀𝒓𝒊 =2011 𝑛(𝑿𝒊 )(𝑃(𝑿𝒊 )[1 − 𝑃(𝑿𝒊 )] − 𝑃0 (𝑿𝒊 )[1 − 𝑃0 (𝑿𝒊 )]) 𝑿𝒊
).
∑𝑿𝒊:𝒀𝒓𝒊 =2011 𝑛(𝑿𝒊 )
̂ (𝐴̂11 ). The
Using this approximation, a 95% confidence interval for 𝐴̂11 is given by 𝐴̂11 ± 1.96√Var
calculation is analogous for 𝐴̂12 .
REFERENCES
1.
McCullagh P NJ. Generalized linear models. 2nd ed. Boca Raton: CRC press; 1989.
7