A Tale in 4 Acts:
Sustaining Evidence- Based &
Promising Practices
Phyllis C. Panzano
University of South Florida
Decision Support Services
Global Implementation Conference
August 15-17, 2011
Washington, DC.
Primary Goal

To explore the question of what
accounts for the long-term ‘success’
of healthcare provider
organizations’ efforts to implement
evidence-based and promising
practices (EBPPs)
ODMH
CLIFFNotes
IDARP*: The Tale
Mike Hogan, Director
Dee Roth & staff,
Kraig Knudsen & staff
MacArthur Network
Howard Goldman, MD,
Pamela Hyde, JD &
Colleagues
Ohio State University
Robert Billings
Paul Nutt
Robert Backoff
CCOEs
Innovation Diffusion & Adoption Research Project
ODMH & MacArthur Network on MH Policy
Patrick Boyle
Patrick Kanary
William Rubin
Four Acts: Take-home Messages?
IDARP Start
1st Contact
2001
2002
2nd Contact
2003
2004
4th Contact
3rd Contact
2005
2006
2007
2008
2009
EBPP Initiative
Kickoff
ACT 1
Adoption
ACT 2
Climate
ACT 3
De-Adoption
ACT 4
Sustained Use
Outline

Research Context

Cross-cutting Assumptions and Paradigms

IDARP: Metaphors, Methods, and Findings
Act 1:
 Act 2:
 Act 3:
 Act 4:


The adoption decision: A risk-based model
Implementation success and project climate
The de-adoption decision: Fit
Sustainability and Implementation Success
Reflections & Wrap-up
Research Context
ODMH’s Quality Agenda
Outcomes
Key Factors Considered by
Policymakers
Political Salience
Initial 8 Coordinating Centers
of Excellence (CCOEs)
SAMI-IDDT
MST
Family Psycho-education
Cluster-Based Planning
OMAP
MH/Criminal Justice
MH/Schools
Advance Directives
Political Salience
Structure of Coordinating Centers of
Excellence (CCOE)
o University or local partnership
o One Best Practice per CCOE
o Statewide service area
Role of CCOEs
o Promotion of Best Practices
(voluntary adoption)
o Training, TA, Problem Solving
o Capacity Development
o Fidelity measurement
o Cross-system sharing
Innovation Diffusion &
Adoption Research Project
(IDARP)
Major Research Questions:
 What factors explain the adoption of
Best Practices (EB & PPs) by behavioral
healthcare organizations (BHOs) in this
system context?
 What factors explain implementation
success among adopter BHOs in this
system context?
Cross-cutting Assumptions
& Paradigms
Context as Caveat

Context impacts:
 Particular questions posed
 Theory & paradigms seen most relevant
 Generalizability of findings
ODMH Context:
 Statewide mental health system initiative
 Voluntary adoption
 Promoting specific set of ‘BEST” practices
 “TA” Resource: CCOE
Key factors potentially impacting relevance
Innovations & Evidence


Focal EB & PPs = innovations

Internal standards: seen as ‘new’ by the relevant unit
of adoption regardless of newness in the marketplace
(e.g., Zaltman, 1973; Nord & Tucker, 1987)… and

External standards: recency (the state of the art) or
“differentness” (significant departures from status
quo, prior forms) (e.g., Rye & Kimberly, 2007)
Scientific evidence and adoption? It may be neither
necessary nor sufficient (e.g., Abrahamson, 1998; Denis et
al, 2002; Duffy & Farley, 1992)
Strategic Decisions

Innovation adoption & implementation decisions: organizationally
important, consequential and merit attention by top management
At the level of individual [healthcare] provider organizations, the direct
and indirect costs of acquiring and implementing [healthcare] innovations
are often substantial, and the adoption and use of innovations typically
present highly consequential financial and managerial challenges”
(Rye & Kimberly, 2007, p. 236)

The strategic value of an innovation to an organization:

May be a function of economic concerns (e.g., effectiveness),
sociological concerns (e.g., legitimacy, power) or both (e.g.,
Abrahamson, 1991; Denis, 2002; Kennedy, 2010)

Is likely to vary across organization ….and even within
organization over time (e.g., Helfrich, 2007; Klein & Sorra, 2001;
Rye & Kimberly, 2007)
Organizations Think
Organizations do not have mechanisms separate from
individuals to set goals, perceive the environment etc.
Top management team (TMT) members interpret
developments and form organizational-level interpretations
which set the stage for organizational decisions & actions (e.g.,
innovation adoption decisions)
The organizational interpretation process is something more
than individual processes because organizations have missions,
stakeholders, histories, memories and intelligence that are
preserved although individual managers may come and go.
(e.g., Daft & Weick, 1984; Hambrick & Mason, 1984; Kennedy & Fiss, 2010)
Organizations Think
Organizations do not have mechanisms separate from
individuals to set goals, perceive the environment etc.
Member of Top Management Teams (TMT) interpret
developments and form organizational-level interpretations
which set the stage for organizational decisions & actions (e.g.,
innovation adoption decisions)
The organizational interpretation process is something more
than individual processes because organizations have missions,
stakeholders, histories, memories and intelligence that are
preserved although individual managers may come and go.
(e.g., Daft & Weick, 1984; Hambrick & Mason, 1984; Kennedy & Fiss, 2010)
Interpretations Depend on Vantage Point
System Developments
Strategic Issue Agenda
Upper Echelon
CEO
CCO
COO
CFO
CIO
CCOE
INPATIENT
OUTPATIENT
CHILDREN
ADULT SERVICES
SMI
SAMI
Name
Title
CHILDRENS
SERVICES
Name
Title
Name
Name
Title
Title
Name
Title
EBP LEAD
FOSTER CARE
PROGRAM
MST THERAPIST 1
MST THERAPIST 2
MST THERAPIST 3
MST THERAPIST 4
Focal Subunit
(e.g., Mathieu & Chen, 2011)
How do you spell S-U-C-C-E-S-S?


Multiple ways to define and measure implementation success

Decision-to-adopt (e.g., Damanpour, 1991; Rye & Kimberly,
2007)

Fidelity/strategic accuracy of use (e.g., Dusenbury)

Outcomes (objective & perceived)

Duration of use/continued use (e.g., Hickson, 2003; Nutt,
2004; Kennedy & Fiss, 2010)

Decision or implementation stage achieved (e.g.,
Chamberlain, 2010; Meyer & Goes, 1988; Yin, 1979)
Meaningful measures of implementation success (with
implications for sustained use) may be conceptually different
for EB & PPs compared to other innovations.
(e.g., Fichman, 2005; Real and Poole, 2005)
The 4 Acts:
Methods,
Metaphors,
& Findings
The Four Acts
1st Contact
2001
2002
2nd Contact
2003
ACT 1
Adoption
Research Question 1
2004
ACT 2
Climate
4th Contact
3rd Contact
2005
2006
2007
ACT 3
De-Adoption
2008
2009
ACT 4
Sustained Use
Research Question 2
Design, Methods, Measures

Observational Field Study; longitudinal; < 4 Contacts

Focal Practices: 4 of the 8 practices/CCOEs
 Primary = IDDT & MST;
 Secondary = OMAP & CBP
 Maximize variability on key innovation attributes

Recruitment: CCOE  Provide key organizational contact
to research team; voluntary participation by CCOEs and
organizational participants (>90% - 75% across time period)

Methods: interviews (structured, process reconstruction
protocol); surveys (organization & CCOE; forms varied by
role, project status, and time period); archival data
Design, Methods, Measures

Interview Questions & Survey Measures: Theory-based;
adapted from existing protocols and scales; needed to be generic
to accommodate the study of multiple innovations

Key informants: Top decision-makers (e.g., CEO, CCO, CFO),
implementation managers, front-line implementers; CCOE
Project liaisons/consultants

Sample tracked: 84 adoption decisions (& decision processes)
and 50+ implementation efforts followed for < 4 contact pts.

Analyses: measurement reliability and agreement prerequisite to
aggregating for analyses; use multiple sources of measures when
possible; use measures from multiple time frames when feasible
Participation Across Contact Point1
Category
A. Focal Efforts
1st
2nd
3rd
4th
Contact Contact Contact Contact
84
50
44
44
B. Org.
Interviews
C. Org. Surveys
251
140
128
68
252
117
108
70
D. CCOE
Surveys
Total Data
Sources*
(Avg. per project)
91
59
46
34
594
(7.1)
316
(6.3)
282
(6.4)
168
(3.8)
Row
Total
222
(dup)
587
(2.6)
547
(2.5)
230
(1.0)
1360
(6.1)
* Archival measures not included; most survey and interview
questions adapted from existing measures and interview protocols
Act 1: The Adoption Decision
The Card Shark Model
The decision to adopt
depends on calculated risk;
the size of your chip stack
does matter!
Some relevant topics and readings





Decision-making under risk (e.g., Sitkin & Pablo)
Prospect theory (e.g., Tversky & Kahneman)
Threat rigidity hypothesis (e.g., Staw, Sandelands, Dutton)
Unified Theory (e.g., Venkatesh et al)
Innovation adoption by healthcare provider orgs (e.g., Rye, 2007)
A Decision Under Risk
A
N
T
E
C
E
D
E
N
T
S
Perceived
Risk of
Adopting
Capacity to
Manage or
+
Propensity
••RAISE
IMPLEMENTATION
UNDERWAY
•• CALL
JUST DECIDED TO
ADOPT
Absorb Risk
Risk-taking
Success:
Decision Stage
+
•• CHECK
STILL CONSIDERING
NEVER WILL ADOPT
•• FOLD
For more detail: Panzano and Roth, 2006
TMT Perspective
A
N
T
E
C
E
D
E
N
T
S
Perceived
-.50
Risk of
Adopting
Success:
Decision Stage
• IMPLEMENTATION
Capacity to
Manage or
UNDERWAY
.40
ADOPT
Absorb Risk
Risk-taking
Propensity
• JUST DECIDED TO
.28
• STILL CONSIDERING
• NEVER WILL ADOPT
Panzano & Roth (2006) Psychiatric Services
A bit more ….
Capacity To
Perceived
Manage Risk
Risk
Past Propensity to
Take Risks
Decision Stage
ACT 1:
1. Success: Decision to Adopt
2. Decision made in consideration of
risk to the organization
Act 2: Climate for Implementation
The Dilbert Model
Projects can rise and fall
depending on how
soundly they’re managed
Some relevant topics:
 Levels issues in climate research (e.g., Klein & Kozlowski, 2004)
 Climate for implementation (e.g., Klein et al, 2001; Holahan, 2004;
Helfrich, 2007)
 Implementation drivers (e.g., Fixsen et al, 2009)
Climate for Implementation

Definition: Targeted employees shared perceptions of
the extent to which their use of a specific innovation is
rewarded, supported and expected within their
organization (Klein & Sorra, 1996, p. 1059).

Climate dimensions in BHO & CCOE Surveys








Top management support
Goal Clarity
Dedicated resources
Access to training & TA
Rewards/recognition for implementing
Removal of obstacles
Performance monitoring
Freedom to express doubts
Focal Entity for Climate for Implementation
CEO
CCO
INPATIENT
COO
OUTPATIENT
CHILDREN
ADULT SERVICES
SMI
SAMI
CFO
CIO
FOSTER CARE
PROGRAM
Name
Title
CHILDRENS
SERVICES
Name
Title
Name
Name
Title
Title
Name
Title
MST TEAM LEAD
MST THERAPIST 1
MST THERAPIST 2
MST THERAPIST 3
MST THERAPIST 4
Focal Entity: Team
The definition of targeted employees for the measurement
of climate for implementation is likely to vary by innovation
Definition of Implementation Success
Implementation effectiveness:
The accurate, committed & consistent use
of a practice by targeted employees (e.g.,
fidelity)
Innovation effectiveness:
Benefits that accrue to an organization and its
stakeholders as a result of implementation
Klein & Sorra, 1996; 2001
CLIMATE AND SUCCESS
Time 1
Climate for
Implementation
Time 2
.75
.45
IMPLEMENTATION
EFFECTIVENESS
INNOVATION
EFFECTIVENESS
For more details: Vaidyanathan, 2004; Panzano et al, 2005
CLIMATE AND SUCCESS
Time 1
Climate for
Implementation
Time 2
+++
IMPLEMENTATION
EFFECTIVENESS
+++
NS
INNOVATION
EFFECTIVENESS
ACT 2:
1. Practice-specific implementation climate is important
2. Implementation effectiveness (e.g., fidelity) 
innovation effectiveness
Act 3: De-Adoption Decision
If the Glove Still Fits,
Keep-wearing-it-model
External and Internal
Developments Influence
Goodness-of-Fit
Some relevant topics and readings:
 Disengagement or abandonment of innovation, Rye & Kimberly, 2007
 Managerial fads and fashions: The diffusion and rejection of
innovations, Abrahamson, 1991
De-innovation, Kimberly & Evanisko, 1981
The De-Adoption Decision1




Un-anticipated
Exploratory – data from BHO key informant
interviews & surveys
12 matched pairs of practices/orgs
Compared Sustainers with “De-adopters”
1For
more detail: Massatti, Sweeney, Panzano & Roth, 2008
Key differences: Sustainers vs De-adopters

Better Fit





Stronger project management & resources




Compatibility with org mission & values
Support from external organizations to continue
Ongoing support from top management
Positive attitudes about practice among staff
Access to TA during implementation
Availability of resources (current & projected)
Know – how and skill at implementing
More convincing effectiveness evidence
Recap: Acts 1, 2 & 3
ACT 1: Adoption
& Org. Risk
ACT 2: Climate &
Effectiveness
ACT 3: De-adoption,
fit & effectiveness
Gains > Losses
Stronger
Project
climate,
better
execution
& results
+ Fit, + Results
Losses > Gains
- Fit; - Results
Act 4: Sustained Use & Assimilation
GLOVE FITS
DILBERT
Contact 4: Sample & Methods





43/44 sites still implementing (roughly 6 years, on average)
Success: Assimilation
Informants:
 BHOs: TMT members & EBPP managers
 CCOEs: primary consultant/liaison
Methods: Surveys (different forms); interviews (BHOs)
All projects represented between BHO + CCOE data:
 35: organization survey and fidelity interview
 34: CCOE surveys & fidelity data
 25: Common/overlapping
Sustainability Model: Core Elements
FIT
Strategic
Prospects/
Frame
Operational
IMP.
CLIMATE
IMPLEM.
INNOV.
EFFECTIVE
NESS
EFFECTIVE
NESS
EBI-specific:
Goal Clarity
Champion
Leader Supt
Dedicated $
Perf. Monitor
Reinforce
Training/TA
Rem_Obstcl
Express Dbt
Accuracy
Impact on:
Commitment
Clients
Consistency
Organization
Other
Stakeholders
ASSIMILA
TION
Embeddedness
Permanence
Operational Measures: In
CCOE & Organizational Surveys
FIT
Strategic
Prospects/
Frames
Operational
IMP.
CLIMATE
IMPLEM.
INNOV.
EFFECTIVE
NESS
EFFECTIVE
NESS
EBI-specific:
Goal Clarity
Champion
Leader Supt
Dedicated $
Perf. Monitor
Reinforce
Training/TA
Rem_Obstcl
Express Dbt
Accuracy
Impact on:
Commitment
Clients
Consistency
Organization
Other
Stakeholders
ASSIMILA
TION
Embeddedness
Permanence
Operational Measures/Scales
#_Alpha
Scale
Strategic Fit
Sample Item
EBI (name) fits well with the mission and
overall goals of this organization
Climate
Top administrators strongly support the
implementation of EBI at this organization.
CCOE: 9_.87
Org: 25_.91
Implementation
Effectiveness
Innovation
Effectiveness
EBI is being implemented at this
organization as prescribed by experts.
This organization’s overall effectiveness has
improved as a result of implementing EBI.
CCOE: 4_.90
Org: 4_.83
Assimilation
EBI is part of the way we do business at this
organization.
EBI will be part of the way this organization
does business for the forseeable future.
CCOE: 5_.81
Org: 3_.83
CCOE: 8 _.87
Org: 4 _.80
CCOE: 8_.83
Org: 11_.91
Major Bi-variate Linkages in the
Sustainability Framework
IMP.
CLIMATE
FIT
IMPLEM.
INNOV.
EFFECTIVE
NESS
EFFECTIVE
NESS
Focus
ASSIMILA
TION
CCOE*
.54
.59
.49
.61
ORGS*
.75
.52
.58
.67
•CCOE and Org ‘r’ values shown have 1-tailed p < .01
•Absolute rs with Assimilation range from .43 to .77
Assimilation

The extent to which an innovation is
seen as or has become a regular part of
organizational procedures.
(e.g., Yin, 1979; Zmud & Apple, 1992)
Organization & CCOE Scale Scores*
for 25 Common Projects
7
6
5
4
3
2
1
Climate
Fit
Fidelity
Agency
Effectv
CCOE
Assiml
* Color contrast: Paired
T-Test, 2-tailed p < .05
Does Climate Mediate (carry) the
influence of Strategic Fit to
Assimilation?
Mediator = Climate
IV = Fit
DV = Assimilation
Climate* significantly (and fully) carries the
influence of Strategic Fit to Assimilation
Beta .75 (.08); R2.56
Beta .54 (.17); R2.29
IV = Fit
Mediator = Climate
Beta .59 (.13)
Beta .45 (.23)
Sobel Test Significant:
Org Sample (n = 35; common n = 25)
CCOE Sample (n=34; common n = 25)
Beta .79 (.16); R2.44
Beta .80 (.16); R2.63
DV = Assimilation
Does Fidelity Mediate (carry) the
influence of Climate to Assimilation?
Mediator = Fidelity
IV = Climate
DV = Assimilation
Fidelity* significantly (and partially) carries
the influence of Climate to Assimilation
Beta = .73(.21)
Mediator = Fidelity
Climate
Beta = .79 (.15)
Beta = .55 (.11)
Assimilation
Sobel Test significant for 3 of 4 tests (Org. Survey, n = 35 and
n = 25; CCOE Survey, n = 25)
Org Survey (n = 35 shown); Sobel = 2.85, 2-tailed p < .01)
Exploratory Stepwise Regressions

Independent Variables:
 Fit
 Climate
 Fidelity – Implementation Effectiveness
 Overall Impact – Innovation
Effectiveness

Dependent Variable: Assimilation
DV: Assimilation, T4
Fidelity,
13
Unexpln,
31
Unexpln,
34
Strat. Fit,
0
Strat.Fit, 0
Fidelity
Climate
Climate,
44
Strat. Fit
Unexpln
Organization T4 (n = 35)
Fidelity
Climate
Fidelity, 9
Climate,
57
Strat.Fit
Unexpln
CCOE T4 (n = 34)
Do CCOEs Add Value (Variance)?
DV: Assimilation, T4
Unexpln,
24
Fidelity, 5
T4_IOR
Qual, 18
Climate,
53
Fit, 0
Fidelity
Climate
Fit
T4_IOR Qual
Unexpln
Contact 4: Organization Survey (n = 35)
Stepwise: F(3.31) = 32.42, p < 01.
Do parallel variables from T3
explain assimilation @ T4?
Exploratory: 2 Stepwise Regressions
T3 Org. Survey (2 yrs. prior)
CLIMATE FOR IMPLEMENTATION
IMPLEM. EFFECTIVENESS (fidelity)
T4Assimilation
Assimilation
INNOV. EFFECTIVENESS (impact)
ASSIMILATION
CCOE- IOR:
Communication Quality
.29*
.29*
CCOE
SURVEY
.21*
.21*
ORG.
SURVEY
CCOE-IOR: Relationship Quality
(e.g., instrumentality; trust)
* Variance explained R2,
Recap & Reflections
Recap: Methods

Upper echelon paradigm: top decision makers as critical key
informants

Small informant ‘n’ …yet very typical for strategic decision
making paradigms

Good reliability and convergence of measures; hearty variance
explained

Small provider ‘n’ limits sophisticated analysis: Tale is simply a
narrative account of possible links among separate but related
analyses which occurred over time

Benefit of having CCOE data at each contact pt, multiple
informants, surveys and interviews – helped to deal with shared
method variance and supported triangulation

Despite limitations, methods and measures may be useful for
other provider-level adoption and implementation studies.
Recap: Sustained Use & Assimilation

Compared to De-adopters, Sustainer BHOs viewed:




the risks of adopting as lower,
EBPPs as more effective
EBPPs fitting better with the organization’s needs, both
operationally and strategically (Act 1 & Act 3)
Among sustainer organizations (at T4), views about
assimilation were positively associated with:




strategic fit,
climate for implementation,
implementation effectiveness (fidelity), and
good working relationships with purveyor
organizations/CCOEs.
Recap: Sustained Use & Assimilation

Patterns of findings based on CCOE surveys
echoed those based on BHO surveys; in both
cases, climate for implementation explained
substantial variance in assimilation

Fidelity (implementation effectiveness) appears
to be an important mediator of the influence of
climate on other indicators of implementation
success (e.g., innovation effectiveness,
assimilation)
Just a few Reflections…

Do patterns of data related to Act 4/Assimilation represent
implicit models held about the way the world should work or
the way the world does work?

Are independent variables (e.g., climate, fidelity) predictors
of assimilation or proxies for assimilation? For example, is
faithful delivery seen as an indicator or a condition for
claiming that a practice has taken hold?

What array of value do external purveyor organizations bring
to the assimilation or sustained use equation?

Does or should assimilation signal the removal of the
innovation from the strategic decision making or
management agenda?