Models, Methods, & Measures:
Examining the adoption, implementation
and sustained use of innovations
in the Ohio Mental Health System
Phyllis C. Panzano, Ph.D.
University of South Florida
Decision Support Services, Inc.
September 20-21, 2010
Improving Implementation Research Methods for Behavioral and Social Science
CLIFFNotes
IDARP*
* Innovation Diffusion and Adoption Research Project, Panzano & Roth
ODMH; MacArthur Network on Mental Health Policy
For More Details:
•
ODMH Publications:
–
•
•
New Research in Mental Health Volume 15 - 17,
– IDARP Bulletins 1-7
Journal Articles:
– Panzano, P.C., and Roth, D. (2006). The decision to adopt evidence-based
and other innovative mental health practices: Risky Business? Psychiatric
Services. Vol. 57, pp. 1153 – 1161.
– Seffrin, B, Panzano, PC, & Roth, D (2009). What gets noticed: How
barrier and facilitator perceptions relate to the adoption and
implementation of innovative mental health practices, Community Mental
Health Journal., On-line version currently available through Springer
Science and Business Media, LLC 2008.
– Massatti, RM, Sweeney, HA, Panzano, PC, and Roth, D. (2008). The deadoption of innovative mental health practices (IMHP): Why organizations
choose not to sustain an IMHP; Administration and Policy in Mental Health
and Mental Health Services Research, 35:50 – 65.
– Carstens, C, Panzano, PC, Massatti, RM, Sweeney, HA, and Roth, D. (2007).
A naturalistic study of MST dissemination in thirteen Ohio Communities;
Journal of Behavioral Health Services and Research.
Dissertation:
– Vaidyanathan, V. (2004). Looking beyond the adoption decision in
innovation research: Investigating innovation implementation. The Ohio
State University, Columbus, OH.
ODMH Research Context

History: Policy (S & R), Strategic Initiative (Hospital Closing), Law
(MH Act of 1988) Implementation & Impact Studies

IDARP Catalyst: Funding of CCOEs to facilitate adoption &
assimilation of effective & salient practices
IDDT
MST
Family Psycho-education
Cluster-based Planning
OMAP
MH/Criminal Justice
MH/Schools
Advance Directives
Political Salience
ODMH Research Question
What factors and processes influence the adoption and
assimilation of evidence-based (EBPs) and promising
practices by behavioral healthcare organizations*?
T1: Adoption
T2: Implementation
n = 85
n = 50
2001
2002
2003
T3: Implementation
T4: Implementation
n = 38
2004
2005
n = 34
2006
2007
2008
2009
ODMH funds 8 CCOEs
* User-based model
Extensive Relevant
Literature
Core Research Streams:

Innovation development, diffusion, adoption, implementation
(e.g., Damanpour; Fixsen et al; Frambach & Schillewaert; Greenhalgh;
Hickson et al; Real & Poole; Rogers; Van de Ven; Yin)

Strategic decision making; decision making under risk
(e.g., Dutton & Jackson; March & Shapira; Panzano & Billings; Sitkin &
Weingart; Staw et al, Tversky & Kahneman)

Health care, innovation & public sector planning
(e.g, Meyer & Goes; Nutt; Yin)
Prominent Paradigm

Upper Echelon Theory (Hambrick & Mason)
Design, Methods, Measures
1.
Design: Longitudinal (up to 4 rounds), primarily
concurrent and prospective; observational field study
2.
Focal entities: 85 Innovation decision processes & 50
implementation efforts involving 4 possible practices
3.
Practices:
a.
All 8 CCOEs volunteered; resources ltd study to 4
b.
Selection: structured decision process involving
OSU faculty, ODMH Director’s and Medical
Director’s offices, IDARP researchers
c.
Variability on key innovation-level variables (e.g.,
evidence, complexity, cost); salience also important
Design, Methods, Measures
4.
Selected 2 team-delivered EBPs (IDDT and MST)
as primary; 2 individually-delivered PPs (CBP and
OMAP) as secondary
5.
Recruitment of Sites for IDARP
6.
Methods: interviews (structured; process
reconstruction), surveys (organization & CCOE),
archival data
7.
Key informants: Top decision-makers (CEO, CCO,
CFO), implementation managers, primary CCOE
liaisons
IDARP: Guiding Models and Key
Sources for Measures
1. Adoption Decision: 1 Model
 Card Shark Model
 DVs: Adopt/not Adopt; Decision Stage (e.g., Yin)
2. Implementation: 4 Models
 Launch; Russian Doll, Dilbert; and Glove Models
 DVs:
 Continued Use vs De-adoption
 Implementation Effectiveness (e.g., fidelity,
commitment)
 Innovation Effectiveness (e.g., satisfaction, outcomes)
 Decision stage: extent assimilated; plans to persist
Model 1: The Adoption Decision
The Card Shark Model
The decision to adopt
depends on calculated risk;
the size of your chip stack
does matter!
 Decision making under risk (e.g., Prospect theory)
 Strategic Issue Diagnosis
 Climate and Leadership for Innovation
Under Risk A Decision
A
N
T
E
C
E
D
E
N
T
S
Perceived
-.50
Risk of
Adopting
DECISION STAGE
• IMPLEMENTATION
Capacity to
Manage or
UNDERWAY
.40
ADOPT
Absorb Risk
Risk-taking
Propensity
• JUST DECIDED TO
.28
• STILL CONSIDERING
• NEVER WILL ADOPT
Panzano & Roth (2006) Psychiatric Services
Adoption as Decision Under Risk:
Some Key Sources for Measures
1.
Survey Scales






2.
Risk & Antecedents: Sitkin and Pablo; Sitkin and
Weingart; Panzano & Billings; Bourgeois; Khandwalla
Expectancies: Dutton & Jackson; Thomas & McDaniel
Innovation Attributes:
Moore & Benbaset; Mathieson and Davies; Venkatesh &
Davis; Rogers; Tornatsky & Klein
Climate & Culture: Amabile; Bass; Jung et al, Klein &
Sorra: Makri et al; Marsick & Watson; Siegel
Attitudes: Aarons; Chatman & O’Reilly; Dunham
Interview Questions

Decision Stage: Nutt, Meyer & Goes; Yin
Model 2: Implementation Success
The Launch Model
Initial conditions … prior to and
at takeoff… have important
impacts on the course of events.
 Organizational change
 Implementation strategy
 Planning Process frameworks
Model 2: Factors from earlier stages
impact success
INITIATION
Time 1
Decision
Time 1
Time
IMPLEMENTATION
Success
Time 2
Initiation-Phase Effects
Expected Benefits
+++
Relative Advantage +++
Results Demonstrability +++
Trust in CCOE (purveyor) +++
* Assimilation scale; Global positive outcome scale
S
U
C
C
E
S
S *
Decision-Phase Effects
Objective decision
+++
Information access +++
Internal influence
+++
Commitment
+++
* Assimilation scale; Global positive outcome scale
S
U
C
C
E
S
S
*
Model 3: Implementation Success
The Russian Doll Model
Surrounding conditions and
circumstances influence
implementation success.
 The Meso Paradigm
 Levels Issues in Organizational Research
 Social Ecology Theory
Level 5: Environment
Level 4: Inter-organizational
Level 3: Adopting organization
Level 2: Project level
Level 1: Innovation level
Dependent Variables:
• Implementation effectiveness
• Innovation effectiveness
Time 1
Environment
IOR – Quality of communication (R2 = .13)
Org – Learning culture (R2 = .23)
Project – Leadership Commitment (R2 = .38)
EBP
Time 2
Positive outcomes R2 = .38
Model 4: Implementation Success
The Dilbert Model
Projects can rise and fall
depending on how soundly
they’re managed.
 Climate for implementation
 Project management
Climate for Implementation:








Top management support
Goal Clarity
Dedicated resources
Access to training & TA
Rewards/recognition for implementing
Removal of obstacles
Performance monitoring
Freedom to express doubts
Holahan et al; Klein, Conn and Sorra; Vaidyanathan, 2004
CLIMATE AND SUCCESS
Time 1
Climate for
Implementation
Time1
.75
.45
Vaidyanathan, 2004
IMPLEMENTATION
EFFECTIVENESS
INNOVATION
EFFECTIVENESS
CLIMATE AND SUCCESS
Time 1
Climate for
Implementation
Time 2
+++
IMPLEMENTATION
EFFECTIVENESS
+++
NS
INNOVATION
EFFECTIVENESS
Panzano et al, 2006
Model 5: Sustained Use
If the Glove Still Fits,
Keep-wearing-it-model
External and Internal
Developments Influence
Goodness-of-Fit.
 Strategic Issue Diagnosis & Management
 Project Management
Sustained Use Model
Fit
Use History
Climate
Such as…
Compatibility
Success
Continued Use
vs De-adoption
Capacity
Developments
Perceived
Risk
Degree
Assimilated
Implementation
Effectiveness
Innovation
Effectiveness
Panzano and Roth, 2007
Top 2 Reasons for De-adopting1







1
Financial resources
Community & network issues
Staffing
Tx Practice Compatibility
Effectiveness
Purveyor (CCOE) Barriers
Technology integration problems
12 matched pairs of sustainers vs deadopter sites from organizational
surveys and interviews; Massati, Sweeney, Panzano and Roth, 2008
Fit and climate measures differentiate
sustainers from de-adopters
 Support from external organizations to continue
 Degree of ongoing support from top management
& organization as a whole
 Compatibility of practice with org values
 Positive attitudes about practice among staff
 Capacity: Know – how and skill at implementing
 Access to TA during implementation
 Current & projected resource availability
Sustainer Model:
CCOE-based
Tentative Revision*
Fit
Implementation
Effectiveness
(e.g. fidelity)
Innovation
Effectiveness
Sustain/
Assimilate
(e.g., outcomes)
Climate
T3 and T4 CCOE Surveys; n = 34 projects still underway at T4;
Panzano & Knudsen et al, 2010
Implementation Models:
Some Key Sources of Measures
Interview Protocols and Structured Questions
 Yin, 1979
 Hickson et al, 1986
 Nutt, 2004
 Van de Ven et al’s survey from the Minnesota
Innovation Studies (2000)
Some Key Sources of Measures
1. Survey Scales (organization and CCOE)

Innovation Attributes: see citations for Adoption Model

Attitudes: see citations for Adoption Model plus Dooley et
al; Shore et al

Inter-organizational relationships: Oliver; Ring and Van de
Ven, Granner and Phillips; El Ansari

Organizational structure, size, resources: Hall; Kimberly;
Sutcliffe

Environmental uncertainty: Sutcliffe; Milliken

Politicality: Thomas, Shankster, Mathieu; Dean and
Sharfman
Some Key Sources of Measures
1. Surveys (cont’d)

Leadership: Bass; Makri et al and Championship: Howell
and Higgins

Culture and/or climate: Glisson; Fixsen et al; Holahan et
al; Klein and Sorra

Fidelity and reinvention: Dusenbury; Rice and Rogers; Van
de Ven at al; practice-specific measures

Implementation outcomes: Hickson; Linton; Nutt; Real and
Poole, Van de Ven; Yin;
2. Archival Measures

ODMH databases (e.g., Medicaid)

Agency association database
Concluding Thoughts
 Models
 Value
 Messages
 Design, Methods, Measures
Strengths
Weaknesses
Alternatives