SUPPLEMENT ARTICLE
Optimizing Research Methods Used for the
Evaluation of Antimicrobial Stewardship
Programs
Jessina C. McGregor and Jon P. Furuno
Department of Pharmacy Practice, Oregon State University/Oregon Health and Science University College of Pharmacy, Portland
Antimicrobial stewardship programs (ASPs) are an increasingly common intervention for optimizing antimicrobial therapy in healthcare settings. These programs aim to improve patient care and limit the emergence and
spread of multidrug-resistant organisms by supporting prudent antimicrobial use. However, pressure from the
current reimbursement climate necessitates that ASPs operate as cost-cutting programs rather than focus on
patient outcomes. This has forced the research that is evaluating ASP interventions to concentrate heavily on
economic outcomes. As the science of antimicrobial stewardship advances, it is essential that well-conducted
evaluations, focused on patient and microbial outcomes, serve as the evidence base that directs optimal ASP
intervention design and implementation. In this review, we provide guidance and recommendations for the design of studies to evaluate the impact of ASP interventions on patient and microbial outcomes.
Keywords.
antibiotic stewardship programs; methodology; outcomes; evaluation.
Use of antimicrobial stewardship programs (ASPs) to
optimize antimicrobial therapy in healthcare settings
is rapidly becoming commonplace. These programs
typically operate by implementing multiple interventions aimed at facilitating appropriate and prudent antimicrobial prescription to support effective patient care
while minimizing collateral damage. Because of differences in patterns of antimicrobial use, patient characteristics, and healthcare resources, the complexity of
ASP interventions and anticipated outcomes vary widely among institutions. Regardless of these variations,
evaluations of these programs should draw on core
methodological principles to determine their effectiveness. Here, we describe how to optimize epidemiological methods to design and conduct rigorous evaluations
of ASPs, with an emphasis on patient and microbial
outcomes. We discuss the predominant study designs
Correspondence: Jessina C. McGregor, PhD, Department of Pharmacy Practice,
OSU/OHSU College of Pharmacy, 2730 SW Moody Ave, CL5CP, Portland,
OR 97201 ([email protected]).
Clinical Infectious Diseases 2014;59(S3):S185–92
© The Author 2014. Published by Oxford University Press on behalf of the Infectious
Diseases Society of America. All rights reserved. For Permissions, please e-mail:
[email protected].
DOI: 10.1093/cid/ciu540
and the factors that should be weighed in selecting the
most appropriate design. We also emphasize the need
to explicitly define the intervention and outcomes
of interest before ASP implementation. In this review,
we aim to provide methodological guidance for conducting such evaluations and representative examples
rather than systematically review the literature on ASP
evaluation.
OUTCOMES OF ANTIMICROBIAL
STEWARDSHIP PROGRAMS
Theoretically, the first step in selecting an ASP intervention for implementation should be the identification of
a target outcome that is informed by baseline data or a
needs assessment. In reality, multiple factors drive the
selection of these program interventions and associated
target outcomes. Still, before designing the approach
to evaluate an intervention, it is critical to identify the
outcomes of interest. An explicitly defined outcome
enables accurate assessment of the intervention’s effectiveness. Consequently, we describe the outcomes often
targeted and assessed by ASP interventions and outcomes for which more data are needed. These outcomes
can be divided into the following 3 general categories:
Methodology for Evaluating Antimicrobial Stewardship Programs
•
CID 2014:59 (Suppl 3)
•
S185
clinical outcomes, microbial or resistance outcomes, and
process measures. Although not discussed in this review,
researchers may also assess satisfaction with ASPs, such as
user satisfaction regarding ASP informatics or educational
intervention.
Improvement in clinical outcomes is the goal for ASPs, as
with any healthcare intervention. Therefore, these outcomes
are most important when they are being implemented and evaluated [1]. Potential clinical outcomes of ASPs include clinical
cure or failure, mortality, hospital length of stay, and readmission to the hospital, as well as potential adverse events associated with antimicrobial therapy, including adverse drug effects
and drug–drug interactions. Each outcome can be more precisely defined depending on the specific interest of the investigators
and expected outcomes of the intervention. For example, a mortality outcome can be further specified by type (eg, all-cause or
infection-associated mortality), quantified using temporal parameters (eg, 30-day mortality), setting (eg, in-hospital mortality), or a combination of these characteristics (eg, in-hospital
mortality caused by ventilator-associated pneumonia). It is important to recognize that many ASP interventions are not expected to improve or worsen clinical outcomes; rather, they
are expected to reduce collateral damage (eg, decrease multidrug-resistant organism [MDRO] acquisition) without changing clinical outcomes. In this scenario, studies should be
planned to have sufficient statistical power to show that clinical
outcomes do not differ before and after ASP interventions, similar to noninferiority trials for assessing new antimicrobial therapies. One limitation of the current evidence base is that most
studies that show no significant difference in these outcomes
do not have sufficient power to ensure that the interventions
do not inadvertently affect patient care. Despite their importance in implementing and evaluating ASPs, a relatively small
proportion of previous studies have assessed clinical outcomes.
Additionally, much of the resulting data may be biased or underpowered because of suboptimal study design [2, 3]. Davey
et al recently updated their Cochrane Collaboration systematic
review and metaanalysis of studies evaluating interventions
aimed at improving antimicrobial prescription in hospitalized
patients. The authors concluded that interventions aimed at
improving antimicrobial prescription for pneumonia among
hospitalized patients are associated with significant reductions
in mortality rate [2]. Furthermore, interventions aimed at decreasing excessive antimicrobial use did not significantly increase mortality rate [2]. However, more recent studies also
did not find significant differences in mortality rate, number
of readmissions, or length of stay after implementation [4, 5].
Although these data provide evidence supporting the effectiveness of some stewardship interventions, they touch on only a
narrow scope of the range of interventions currently implemented in practice. Therefore, more work is needed to evaluate
S186
•
CID 2014:59 (Suppl 3)
•
McGregor and Furuno
the impact of different ASP interventions on these clinical
outcomes.
Microbial or resistance outcomes are strongly tied to ASP
goals. They measure the incidence or prevalence of colonization
or infection by an organism and may be assessed at the individual or population level. For example, rates of acquisition or
healthcare–associated infection rates of important MDROs
(eg, methicillin-resistant Staphylococcus aureus, vancomycinresistant enterococci, extended-spectrum β-lactamase–producing
gram-negative bacteria) are essential for measuring the effect of
ASP interventions. For example, in one randomized control
trial (RCT), investigators were able to attribute a 20% reduction
in antimicrobial resistance, superinfections, or both to the implementation of a short-course empiric therapy regimen for patients with pulmonary infiltrates in the intensive care unit [6].
Although a primary goal of antimicrobial stewardship is limiting the spread of MDROs, the literature regarding ASP intervention evaluation does not reflect this because most studies
have focused on economic or process measure outcomes. Although some studies have shown the potential for ASP interventions to decrease MDRO outcomes [7, 8], more effort is
needed to identify best practices for achieving this goal [2]. Additionally, as a frequent, drug-associated adverse effect, rates of
Clostridium difficile infection represent an important target for
ASPs, and this outcome is being increasingly used in stewardship research. Recent studies have identified significant decreases in the incidence of C. difficile infection after implementation
of ASPs [4, 5, 9]; however, a metaanalysis of 5 previous studies
suggests a more attenuated effect [2]. These conflicting findings
underscore the need for additional studies that evaluate the impact of interventions on C. difficile infection rates using optimal
research methods.
Process measures, also referred to as quality or performance
measures, assess the quality of healthcare delivery and are frequently used in studies of the effectiveness of ASPs [3, 10–
12]. Process measures, such as antimicrobial days of therapy
(DOT), represent the intermediate outcomes that ASP interventions directly influence. Process measure data are also typically
less subjective and more easily attainable than clinical outcome
measures. However, the association of these measures with clinical outcomes should be validated to demonstrate their value as
an intermediate outcome. Valid process measures can be useful
in understanding study findings that show limited intervention
effects on clinical outcomes. Common process measures in
evaluations of ASP interventions include overall or specific antimicrobial use, frequency and time to appropriate empiric or
definitive antimicrobial therapy, and unnecessary or excess
drug use. Several recent reviews describe the use of process measures in evaluating ASP interventions [3, 11, 12]. Cost-based or
economic outcomes are another commonly assessed process
measure. Reducing antimicrobial expenditures was the impetus
for many early ASPs and is still used to justify continued funding for these programs, especially given the inability to directly
bill for these services in the United States [4, 5, 13–15]. Despite
the relevance of economic outcomes, the importance and necessity of ASPs can be better proven by showing the impact of their
interventions on clinical outcomes, as reduced cost does not always equate to improved patient care.
Regardless of the type of outcome selected for study, multiple
factors influence the outcome rate and can result in considerable “background noise” that would complicate identification
of the impact of an ASP. Thus, appropriate design or analytical
approaches must be used to adjust for confounding factors and
rule out alternate possible causes for any observed effects. These
approaches are described further below.
SPECIFYING THE INTERVENTION
Once the outcomes of interest have been determined, investigators should clearly specify the interventions to be evaluated so
that outcomes and study design can be selected and developed
in a manner that best suits the intervention(s) under evaluation.
Even in cases in which the desire is to assess the effect of the
program as a whole, ASPs are often composed of multiple interventions. Although many ASPs consist of similar interventions
across different institutions, substantial variation might exist.
Consideration for what interventions are being implemented,
how they are being implemented, and in what populations
they are being implemented is critical for identifying the expected impact and what factors might attenuate that effect.
A common initial focus of new ASPs is the transition of patients taking parenteral antibiotics to oral antibiotics. One leading reason for the focus of this intervention is its large potential
cost savings for pharmacy, though it does not generally aim to
reduce DOT. However, transitioning patients to oral antibiotics
can lead to earlier removal of central indwelling catheters, thereby possibly reducing patient risk for central line–associated infection and other adverse events—all of which could serve as
potential outcomes of interest in a study evaluating parenteralto-oral conversions. Additionally, the ability to perform such
conversions and the risk of these outcomes are influenced by
other patient-level factors, such as use of other medications,
which should be accounted for as potential confounders.
Many stewardship interventions also focus on reducing unnecessary continuation of empirically prescribed agents (eg,
vancomycin), excessive duration of therapy, or unnecessary
double coverage. Ultimately, these interventions aim to reduce
the evolutionary selective pressures that drive increasing
MDROs. Studying the association between these interventions
and reduction in MDRO acquisition and spread can require
large sample sizes or longer follow-up periods; ultimately,
such studies are essential for supporting the central tenets of
stewardship. Furthermore, the effectiveness of these interventions may vary by both patient (eg, disease severity) and provider
characteristics (eg, medical specialty).
Many stewardship interventions also include a direct and active education program intended to alter clinician behavior, for
example, provider training coupled with new antimicrobial
order set for community-acquired pneumonia. Still, other interventions may have an indirect educational effect such as a change in the electronic order entry system that requires use of a new
extended infusion protocol but is implemented without simultaneous provider education efforts. In this scenario, providers
may seek out the rationale for the extended infusion; if the electronic intervention is removed, use of extended infusion may
continue. Thus, studies of interventions that are directly or indirectly educational should consider how expected outcomes
might be impacted. For example, when implementing a new
order set for community-acquired pneumonia, the expected
changes in antibiotic ordering may occur gradually as teams
are trained and use of the new order set increases. The duration
of the study should then be planned to appropriately capture the
effect of the intervention.
STUDY DESIGN
Study design must be balanced between both theoretical ideals
and ethical and practical considerations. An understanding of
these ideals and the principles surrounding the ability of each
design to contribute to causal inference facilitates conduct of
a study with a high likelihood for strong internal validity, that
is, a study that provides unbiased and accurate evidence of the
relationship between the intervention and the outcome. Ideally,
study design should be determined a priori to allow for integration of design and analysis features that can further strengthen
the evaluation of an ASP intervention. When designing a study,
it is important to consider holistically the methodology (eg, inclusion/exclusion criteria, control group selection, or analysis
plan). Key aspects that should be considered in designing evaluation studies of ASPs are listed in Figure 1. The following sections describe the common and appropriate study designs for
evaluation of ASP interventions, strengths and limitations of
each, and design features that can further support the internal
validity of the study; these designs are also summarized in
Table 1.
Randomized Control Trials
RCTs can provide strong evidence for attributing observed effects to ASP interventions because the process of randomization
is generally highly effective at removing the effect of confounding factors that threaten the internal validity of nonexperimental studies. In determining the appropriateness and feasibility of
an RCT for evaluating an ASP intervention, specification of the
Methodology for Evaluating Antimicrobial Stewardship Programs
•
CID 2014:59 (Suppl 3)
•
S187
Figure 1.
Important considerations in design and analysis studies that evaluate antimicrobial stewardship interventions.
Table 1. Overview of Common Study Designs for the Evaluation of Antimicrobial Stewardship Interventions
Characteristic
Randomized
Control Trial
Cluster Randomized
Control Trial
Cluster-Randomized
Crossover
Nonrandomized
Controlled Trial
Interrupted Time Series
General
description
Intervention
randomized to
patients or
clinicians;
outcomes
compared
between
intervention and
control groups
Intervention
randomized to
different clusters
(eg, units or
hospitals);
outcomes
compared
between
intervention and
control clusters,
controlling for
correlation and
cluster sizes
Two-phase design in which
clusters are randomized
to receive either
intervention in phase 1
and standard care in
phase 2 or vice versa;
outcomes compared
between intervention and
standard of care across
clusters
Intervention is
assigned to
patients or
clinicians without
randomization;
outcomes are
compared between
intervention and
control groups,
controlling for
known potential
confounders
Intervention is assigned
to either all study
patients or specific
intervention groups;
pre-intervention and
post-intervention
outcome rates are
analyzed to identify
immediate and
gradual changes
following
implementation of
intervention; control
groups may be
included
Unit of
intervention
allocation
Individual
Cluster (eg, hospital
or unit)
Cluster (eg, hospital or unit)
Individual
Cluster (eg, hospital
or unit)
Major
advantage
Randomization
generally controls
for known and
unknown
confounders
Randomization
generally controls
for known and
unknown
confounders at the
cluster level
Randomization generally
controls for known and
unknown confounders;
cluster serves as own
control
Interventions may be
allocated in a
manner that
considers logistical
or practical
implementation
issues
Allows for identification
of immediate and
gradual changes in
outcomes over time
compared with
baseline rates and
trends; typically lower
cost/effort to
implement
Potential
limitations
May be logistical or
ethical barriers to
implementation
Requires larger
numbers of
patients than RCT;
may be difficult to
standardize
intervention
across clusters;
differences in
cluster
characteristics
may be difficult to
fully control for in
analysis
Requires larger numbers of
patients than RCT; may be
difficult to standardize
intervention across
clusters; differences in
cluster characteristics
may be difficult to fully
control for in analysis;
interventions may have
lasting effects even after
removed; may
necessitate washout
periods between phases
Potential confounding
effects need to be
controlled for using
design or analytical
methods; cannot
control for
unknown
confounders
Requires longer followup periods;
controlling for
confounders more
difficult
Abbreviation: RCT, randomized control trial.
S188
•
CID 2014:59 (Suppl 3)
•
McGregor and Furuno
intervention and outcome is critical. We previously used an
RCT design to evaluate a computerized clinical decisionsupport system (CDSS) for ASPs [16]. An active ASP was in
place but required labor-intensive chart review to identify opportunities to optimize antimicrobial use. During the study,
randomization was used to determine which patients would
trigger automated CDSS alerts to guide the ASP team interventions. In this scenario, an RCT design was feasible and ethical.
While the intervention was randomly allocated at the patient
level in our study, stewardship interventions are commonly implemented at the provider level and may have direct or indirect
educational effects. Therefore, it is often not feasible to randomize at the patient level because the intervention might have also
inadvertently affected the control group. However, randomization at the provider level might still be performed. In this scenario, it is critical to recognize that when the outcome of
interest is at the patient or population level, randomization
might not sufficiently remove confounding effects because the
patient case mix seen by each provider is associated with that
provider’s characteristics (eg, discipline, specialty training,
ward/unit of service, shift time). Weighted or blocked randomization schemes may help to remove these effects by purposefully taking these provider differences into consideration; however,
doing so might be difficult depending on the size of the institution and number of providers to be enrolled. Alternatively, standard analytical techniques such as multivariable regression
might be used to control for known and suspected confounders.
Cluster Randomized Trials
Cluster randomized trials are a variation of the RCT design in
which the unit of randomization is a naturally occurring group
such as a hospital, clinic, or other healthcare facility. The
strength of this design lies in the use of randomization to reduce
threats to internal validity and the use of multiple healthcare
sites, which can increase external validity (ie, generalizability).
Despite the methodological similarities among standard patient-level RCTs, design and statistical analysis of cluster randomized trials are more complex. Although detailed statistical
guidance for sample size determinations and statistical analysis
exists elsewhere [17–19], we provide relevant considerations
here for analysis of cluster randomized trials of ASP interventions. For studies that evaluate effects on patient-level outcomes,
the correlation structure of the data must be assessed and, if
necessary, must be accounted for in the analysis by use of appropriate modeling techniques. It might also be necessary to
weight observations in studies in which cluster sizes differ. Additionally, for studies evaluating intervention effects on patientlevel outcomes, adjustment for confounding might still be
necessary. Recently, several groups successfully used this approach to evaluate stewardship interventions in non–acute
care settings. In these settings, stewardship interventions have
the same goals of supporting prudent antibiotic use, but the interventions themselves are not typically implemented through a
stewardship team. A major focus in outpatient settings has been
the reduction of unnecessary antibiotic prescription for viral
acute respiratory tract infection (ARTI). Recent cluster randomized trials of primary care clinics have evaluated different strategies to reduce unnecessary prescription for ARTIs, from
educational interventions coupled with audit/feedback [20] to
clinical decision-support systems [21, 22] to shared decisionmaking aids [23]. Cluster randomized trials have also been
used to evaluate stewardship interventions in emergency departments [24] and long-term care facilities [25, 26]. However,
this approach has been underused to evaluate stewardship interventions in acute care settings. Because ASP interventions are
already in place in many acute care facilities, willingness to participate in a study that randomly allocates an intervention might
be limited. Parallels can be drawn to infection control research,
where, despite the commonplace nature of many interventions,
cluster randomized trials [27, 28] have shown an ability to drive
the evidence-based refinement of best practice. Because the evidence base for many common ASP interventions is limited,
similar work is needed to identify stewardship best practices.
Cluster-Randomized Crossover Studies
The basic cluster-randomized crossover study is one in which
groups (eg, units or hospitals) are randomized to receive either
the intervention in phase 1 and standard care in phase 2 or the
reverse—with standard care in phase 1 and the intervention in
phase 2. The design can be modified to include additional phases to compare multiple interventions or to include washout periods to account for transient effects of removed interventions.
As with the RCTs and cluster randomized trials, the primary
analysis approach should be intention to treat in order to retain
the benefits of randomization (ie, control for known and unknown confounders). Van Werkhoven and colleagues recently
published an overview of the cluster-randomized crossover
study in which they used their own study as an example [29].
The authors designed a 3-phase cluster-randomized crossover
trial to compare beta-lactam monotherapy, beta-lactam and
macrolide combination therapy, and fluoroquinolone monotherapy as empiric treatment regimens for hospitalized patients
with community-acquired pneumonia. While the trial results are
still pending, the published methods paper by van Werkhoven
et al provides details regarding study design and analysis specific
to cluster-randomized crossover studies [29].
Nonrandomized Controlled Studies
Nonrandomized controlled studies, also known as clinical controlled trials, include both intervention and control groups, and
outcomes can be compared between groups. The intervention is
not randomly assigned. While the approach to this design is
Methodology for Evaluating Antimicrobial Stewardship Programs
•
CID 2014:59 (Suppl 3)
•
S189
similar to that of observational studies such as the cohort study,
the key differentiating factor is the assignment of the intervention, albeit not randomly, to patients. Examples of this study design for the evaluation of antimicrobial stewardship
interventions include a study by Toltzis et al in which investigators compared an assigned antibiotic cycling schedule in one
neonatal intensive care unit with another neonatal intensive
care unit that retained standard unrestricted use to determine
the impact on colonization rates with antibiotic-resistant
gram-negative bacilli [30]. The lack of randomization leaves
the study vulnerable to confounding effects. Design and analytical techniques for controlling known confounders can be used,
similar to cohort studies. These include strategies such as
matching, stratification, and multivariable regression.
Quasi-Experimental Studies
Quasi-experimental studies are a family of study designs used to
evaluate the effect of a group/population-level intervention.
Often referred to as pre/post or before-and-after studies, this design is commonly used for quality improvement projects in
acute care settings. In quasi-experimental studies, the outcome
of interest is measured at the group/population level (eg, DOT
or infection rates); for data analysis, the time point at which the
intervention is introduced should be specifically identified. Similar to experimental or observational studies, the strength of
quasi-experimental studies in providing unbiased evidence of
the effect of an intervention varies but can be meaningfully improved through the addition of higher-order design features
(described below).
The different types of quasi-experimental study designs are
described in detail elsewhere [31–33]. Here, we focus on the
strongest design for causal inference—interrupted time series
(ITS). ITS studies require repeated measurements of the outcome across evenly spaced time intervals before and after the
intervention. For example, Aldeyab et al used the ITS design
to evaluate an intervention to reduce use of antimicrobials associated with a higher risk of subsequent C. difficile infection [9].
Monthly rates of C. difficile infection were recorded in the preintervention and post-intervention periods and rates and trends
in rates were compared [9]. The strength of this approach lies in
the ability to identify both immediate and gradual effects of an
intervention. Yet, this strength is lost if the data are not appropriately analyzed. As we have shown [34], the effect of an intervention might be either over- or underestimated if statistical
regression models are not appropriately parameterized to detect
changes in intercept and slope terms in the post-intervention
period. In other words, the regression model must include not
only an indicator variable to identify the pre- vs post-intervention
periods but must also include a variable to indicate the time
since study initiation and time since intervention initiation.
These variables are all necessary to allow the model to identify
S190
•
CID 2014:59 (Suppl 3)
•
McGregor and Furuno
immediate changes and/or gradual changes in the outcome following intervention implementation. Furthermore, the collected data will be correlated, and failure to account for this
correlation will result in biased statistical tests. These more advanced statistical principles for analysis of time series data are
beyond the scope of this review and described elsewhere [31,
34, 35]. However, we wish to emphasize a few principles for designing an ITS study such that the design supports the intended
analysis. It is recommended that a minimum of 10 data points
(ie, outcome measurements) be collected before and after the
intervention. If seasonal or other secular trends are anticipated,
additional data points might be necessary to appropriately
model such trends. For studies in which the outcome is expressed as a rate (ie, events per time unit), consideration of
the width of the time intervals used in analysis is important.
Although using smaller time intervals might create more data
points before and after the intervention, it can also result in periods with very low event rates—or even periods with zero observed events. To balance the need for sufficient data points
with stability in rate estimates, outcome data should be collected
with sufficient granularity to allow periods to be collapsed or
divided, if necessary. While the study is being conducted, attention should be given to recording any coincident events that
might affect the outcome, such as antibiotic shortages, outbreaks, or facility expansion. If the timing of these coincident
events is recorded, their potential effect can be adjusted for if
there is sufficient statistical power.
As with other study designs, alternate explanations must be
ruled out in order to attribute any observed change in outcome
to the intervention. The use of higher-order design features
facilitates this goal. To those accustomed to the more traditional
experimental and observational study designs, the most apparent design feature is the addition of a control group. In an ITS
study, the ideal control group is one in which the intervention is
not implemented but is subject to influence by the same external forces (eg, drug shortages, staffing changes) as the intervention group. It is also possible to leverage the common practice of
rolling out an intervention to compare different groups in
which the intervention is implemented at the different phases.
Ideally, rolling out of the intervention would be sufficiently
staggered to allow outcome trends to be tracked in each group
at each phase of implementation. The expected pattern would
be that similar intervention effects are observed in different
groups (eg, wards or units) with each subsequent implementation of the intervention. The strength of this approach lies in
the complexity of the expected pattern. With rolling out an intervention, if the expected pattern were achieved, it would be
difficult to attribute that pattern to alternative causes. A similarly complex pattern of intervention effects would be expected
from introducing the intervention and then subsequently removing it. Demonstrating the effect and subsequent reversal
of effect using this design feature provides strong evidence that
the observed pattern can be attributed to the introduction and
removal of the intervention. Another design feature is the use of
a nonequivalent dependent variable, which is an alternate dependent outcome not subject to influence by the intervention
but by any coincident events or forces. Selection of a nonequivalent dependent variable when evaluating an ASP intervention
can be challenging because these interventions often have
broader indirect effects on antimicrobial use. When possible,
consultation with an epidemiologist and/or statistician may be
advantageous in developing the study design and analytic
approach.
CONCLUSIONS
A strong evidence base is critical to ensuring that ASPs are successful at increasing prudent antimicrobial use and limiting the
development and spread of MDROs. The nature of ASP interventions and the outcomes of importance to patients can complicate
study design selection and methodology. However, well-designed
studies aimed at evaluating ASP impact on MDRO control and
patient outcomes are critical for maintaining efficient, highquality patient care and promoting prudent antibiotic use. Randomization of patients may not always be appropriate because
interventions are typically applied at the provider level, whereas
outcomes of interest are at the patient level. Furthermore, randomization at the clinician or institution level could still result
in residual confounding effects. Antimicrobial stewardship programs also often implement multiple interventions simultaneously. Although this might necessitate assessment of multiple
relevant outcomes, interventions can have global effects on patterns of antimicrobial use and, therefore, affect simultaneously
enacted interventions. Even an intervention that seems to be
single might consist of a bundle of elements, often incorporating an educational component or integrating infection control
and ASP practices. In this case, it is important to recognize that
the observed intervention effect results from the implementation of the entire bundle and that the effect attributable to
any one component cannot be ascertained. Regardless, this
bundled approach is often the best method to increase the initial
likelihood of obtaining the desired effect.
The ability to assess the effect of an ASP intervention is also
often limited by external forces beyond the control of investigators. Drug shortages, outbreaks, changes in personnel, and expansion of facilities or services can have major influence on
outcomes. It can be difficult to isolate the true intervention effect from these secular influences. It is also important to recognize that many ASPs do not have data management or statistical
support for programmatic evaluation. Thus, the lack of these resources can be a major barrier to the conduct of high-quality
evaluations of ASPs.
There is a specific need to evaluate different ASP interventions in a broad array of clinical settings. Although the overarching goals of prudent antimicrobial prescribing are the
same in all healthcare settings, the approach to supporting prudent prescribing should be tailored to each setting. As described
previously herein, efforts have been made to curtail inappropriate antimicrobial use in primary care settings, emergency departments, long-term care facilities, and other non–acute care
settings. These interventions fall under the broad umbrella of
stewardship; whenever possible, we should learn from the experience gained from studies in all healthcare settings. Practicing
prudent antimicrobial use in all healthcare settings will best attain the overarching goals of improving patient outcomes and
limiting MDROs.
Conduct of rigorous and methodologically sound evaluations
of ASP interventions can inform best practice and policy. The
findings of several recent studies have shown the positive impact
that ASPs can have on MDRO transmission [36–39] and patient
outcomes [2, 40, 41]; however, more work is needed to further
inform the field by identifying effective and efficient stewardship interventions. To identify the most effective interventions
and implementation strategies, studies evaluating ASP interventions should be carefully planned and conducted to minimize
bias and maximize internal validity. Evaluations in a breadth
of patient settings and populations are also necessary to ensure
generalizability. Therefore, continued effort is needed to show
the value of ASPs beyond simple cost savings and to increase
the evidence base that informs best practices for stewardship
in all healthcare settings.
Notes
Acknowledgments. Editorial support was provided by ApotheCom
ScopeMedical (Yardley, Pennsylvania) and funded by Cubist Pharmaceuticals.
Supplement sponsorship. This article appears as part of a supplement
titled “Antimicrobial Stewardship: Patients Over Process,” sponsored by
Cubist Pharmaceuticals.
Potential conflicts of interest. Both authors: No reported conflicts.
Both authors have submitted the ICMJE Form for Disclosure of Potential
Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.
References
1. Dellit TH, Owens RC, McGowan JE Jr, et al. Infectious Diseases Society
of America and the Society for healthcare Epidemiology of America
guidelines for developing an institutional program to enhance antimicrobial stewardship. Clin Infect Dis 2007; 44:159–77.
2. Davey P, Brown E, Charani E, et al. Interventions to improve antibiotic
prescribing practices for hospital inpatients. Cochrane Database Syst
Rev 2013; 4:CD003543.
3. McGowan JE. Antimicrobial stewardship—the state of the art in 2011:
Focus on outcome and methods. Infect Control Hosp Epidemiol 2012;
33:331–7.
4. Malani AN, Richards PG, Kapila S, Otto MH, Czerwinski J, Singal B.
Clinical and economic outcomes from a community hospital’s antimicrobial stewardship program. Am J Infect Control 2013; 41:145–8.
Methodology for Evaluating Antimicrobial Stewardship Programs
•
CID 2014:59 (Suppl 3)
•
S191
5. Nowak MA, Nelson RE, Breidenbach JL, Thompson PA, Carson PJ.
Clinical and economic outcomes of a prospective antimicrobial stewardship program. Am J Health Syst Pharm 2012; 69:1500–8.
6. Singh N, Rogers P, Atwood CW, Wagener MM, Yu VL. Short-course
empiric antibiotic therapy for patients with pulmonary infiltrates in
the intensive care unit. A proposed solution for indiscriminate antibiotic prescription. Am J Respir Crit Care Med 2000; 162:505–11.
7. Yong MK, Buising KL, Cheng AC, Thursky KA. Improved susceptibility
of gram-negative bacteria in an intensive care unit following implementation of a computerized antibiotic decision support system. J Antimicrob Chemother 2010; 65:1062–9.
8. Dortch MJ, Fleming SB, Kauffmann RM, Dossett LA, Talbot TR, May
AK. Infection reduction strategies including antibiotic stewardship protocols in surgical and trauma intensive care units are associated with
reduced resistant gram-negative healthcare-associated infections. Surg
Infect (Larchmt) 2011; 12:15–25.
9. Aldeyab MA, Kearney MP, Scott MG, et al. An evaluation of the impact
of antibiotic stewardship on reducing the use of high-risk antibiotics
and its effect on the incidence of Clostridium difficile infection in hospital settings. J Antimicrob Chemother 2012; 67:2988–96.
10. Agency for Healthcare Research and Quality. Selecting process measures for clinical quality measurement. Rockville, MD. Available at:
http://www.qualitymeasures.ahrq.gov/tutorial/ProcessMeasure.aspx.
Accessed 14 April 2014.
11. Khadem TM, Ashley ED, Wrobel MJ, Brown J. Antimicrobial stewardship:
A matter of process or outcome? Pharmacotherapy 2012; 32:688–706.
12. File TM Jr, Solomkin JS, Cosgrove SE. Strategies for improving antimicrobial use and the role of antimicrobial stewardship programs. Clin Infect Dis 2011; 53(suppl 1):S15–22.
13. Beardsley JR, Williamson JC, Johnson JW, Luther VP, Wrenn RH, Ohl
CC. Show me the money: Long-term financial impact of an antimicrobial
stewardship program. Infect Control Hosp Epidemiol 2012; 33:398–400.
14. Standiford HC, Chan S, Tripoli M, Weekes E, Forrest GN. Antimicrobial stewardship at a large tertiary care academic medical center: Cost
analysis before, during, and after a 7-year program. Infect Control
Hosp Epidemiol 2012; 33:338–45.
15. Sick AC, Lehmann CU, Tamma PD, Lee CK, Agwu AL. Sustained savings from a longitudinal cost analysis of an internet-based preapproval
antimicrobial stewardship program. Infect Control Hosp Epidemiol
2013; 34:573–80.
16. McGregor JC, Weekes E, Forrest GN, et al. Impact of a computerized clinical decision support system on reducing inappropriate antimicrobial use:
A randomized controlled trial. J Am Med Inform Assoc 2006; 13:378–84.
17. Reich NG, Myers JA, Obeng D, Milstone AM, Perl TM. Empirical power
and sample size calculations for cluster-randomized and cluster-randomized crossover studies. PLoS One 2012; 7:e35564.
18. Campbell MK, Elbourne DR, Altman DG. C group, Consort statement:
extension to cluster randomised trials. BMJ 2004; 328:702–8.
19. Donner A, Klar N. Design and analysis of cluster randomization trials in
health research. New York, NY: Arnold; co-published by the Oxford
University Press, 2000.
20. Gerber JS, Prasad PA, Fiks AG, et al. Effect of an outpatient antimicrobial
stewardship intervention on broad-spectrum antibiotic prescribing
by primary care pediatricians: a randomized trial. JAMA 2013; 309:
2345–52.
21. Samore MH, Bateman K, Alder SC, et al. Clinical decision support and
appropriateness of antimicrobial prescribing: a randomized trial. JAMA
2005; 294:2305–14.
22. Gonzales R, Anderer T, McCulloch CE, et al. A cluster randomized trial
of decision support strategies for reducing antibiotic use in acute bronchitis. JAMA Intern Med 2013; 173:267–73.
23. Legare F, Labrecque M, Cauchon M, Castel J, Turcotte S, Grimshaw J.
Training family physicians in shared decision-making to reduce the
overuse of antibiotics in acute respiratory infections: a cluster randomized trial. CMAJ 2012; 184:E726–34.
S192
•
CID 2014:59 (Suppl 3)
•
McGregor and Furuno
24. Metlay JP, Camargo CA Jr, MacKenzie T, et al. Cluster-randomized
trial to improve antibiotic use for adults with acute respiratory infections treated in emergency departments. Ann Emerg Med 2007;
50:221–30.
25. Monette J, Miller MA, Monette M, et al. Effect of an educational intervention on optimizing antibiotic prescribing in long-term care facilities.
J Am Geriatr Soc 2007; 55:1231–5.
26. Pettersson E, Vernby A, Molstad S, Lundborg CS. Can a multifaceted
educational intervention targeting both nurses and physicians change
the prescribing of antibiotics to nursing home residents? A cluster
randomized controlled trial. J Antimicrob Chemother 2011; 66:
2659–66.
27. Harris AD, Pineles L, Belton B, et al. Universal glove and gown use and
acquisition of antibiotic-resistant bacteria in the ICU: a randomized
trial. JAMA 2013; 310:1571–80.
28. Derde LP, Cooper BS, Goossens H, et al. Interventions to reduce colonisation and transmission of antimicrobial-resistant bacteria in intensive
care units: an interrupted time series study and cluster randomised trial.
Lancet Infect Dis 2014; 14:31–9.
29. van Werkhoven CH, Postma DF, Oosterheert JJ, Bonten MJ. Antibiotic
treatment of moderate-severe community-acquired pneumonia: design
and rationale of a multicentre cluster-randomised cross-over trial. Neth
J Med 2014; 72:170–8.
30. Toltzis P, Dul MJ, Hoyen C, et al. The effect of antibiotic rotation on
colonization with antibiotic-resistant bacilli in a neonatal intensive
care unit. Pediatrics 2002; 110:707–11.
31. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton
Mifflin, 2001.
32. Harris AD, Lautenbach E, Perencevich E. A systematic review of quasiexperimental study designs in the fields of infection control and antibiotic resistance. Clin Infect Dis 2005; 41:77–82.
33. Harris AD, McGregor JC, Perencevich EN, et al. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med
Inform Assoc 2006; 13:16–23.
34. Shardell M, Harris AD, El-Kamary SS, Furuno JP, Miller RR, Perencevich EN. Statistical analysis and application of quasi experiments to antimicrobial resistance intervention studies. Clin Infect Dis 2007;
45:901–7.
35. McDowall D, McCleary R, Meidinger EE, Hay RA Jr. Interrupted time
series analysis. Beverly Hills, Calif: Sage Publications, 1980.
36. Garey KW, Lai D, Dao-Tran TK, Gentry LO, Hwang LY, Davis BR. Interrupted time series analysis of vancomycin compared to cefuroxime
for surgical prophylaxis in patients undergoing cardiac surgery. Antimicrob Agents Chemother 2008; 52:446–51.
37. Lafaurie M, Porcher R, Donay JL, Touratier S, Molina JM. Reduction of
fluoroquinolone use is associated with a decrease in methicillin-resistant
staphylococcus aureus and fluoroquinolone-resistant pseudomonas aeruginosa isolation rates: a 10 year study. J Antimicrob Chemother 2012;
67:1010–5.
38. Jump RL, Olds DM, Seifi N, et al. Effective antimicrobial stewardship in
a long-term care facility through an infectious disease consultation service: keeping a lid on antibiotic use. Infect Control Hosp Epidemiol
2012; 33:1185–92.
39. Knudsen JD, Andersen SE, Bispebjerg G. Intervention, A multidisciplinary intervention to reduce infections of esbl- and ampc-producing,
gram-negative bacteria at a university hospital. PLoS One 2014; 9:
e86457.
40. Bauer KA, West JE, O’Brien JM, Goff DA. Extended-infusion cefepime
reduces mortality in patients with Pseudomonas aeruginosa infections.
Antimicrob Agents Chemother 2013; 57:2907–12.
41. Pogue JM, Mynatt RP, Marchaim D, et al. Automated alerts coupled
with antimicrobial stewardship intervention lead to decreases in length
of stay in patients with gram-negative bacteremia. Infect Control Hosp
Epidemiol 2014; 35:132–8.