None of the
interventions are
evidence-based but
think positive – you get
a new and untested
intervention
IMS APPROACH TO
GOOD PRACTICE
Knut Sundell
Jenny Rehnman
Mari Forslund
www.evidens.nu
IMS director
Development & Evaluation
(assessment tools and
interventions)
20 employees
30 projects lasting 3-4
years
Systematic reviews
(assessment tools and
interventions)
10 employees
15 projects lasting 1-2
years
Additional external
researchers
Board of IMS
(12 stake-holders)
Directorgeneral
NBHW
Support & Dissemination
(administrative support &
dissemination of all that is
produced within IMS)
10 employees
15 projects lasting less
than 1 year
Sweden
• Sweden is the third largest country in Western Europe with
nine million inhabitants
• Social care service delivery involve agencies of the 290
municipalities, and 20 regional county councils. Median size of
municipalities = 15.250 inhabitants
• National agencies and research councils, deals with research
and development, supervision of service quality and safety,
and the delivery of institutional care.
• The voluntary sector is considerable
• Social welfare states (e.g., low rate of unemployment, poverty,
drug use, violence)
• General trust among Swedes in collective solutions and in
government authorities
Research on interventions for women
with experience of partner violence
(Anttila et al, 2007)
5 926 articles
Research on interventions for women
with experience of partner violence
(Anttila et al, 2007)
Effect
studies 99
5 926 articles
Research on interventions for women
with experience of partner violence
(Anttila et al, 2007)
Good
evidence 6
5 926 articles
0,1 per cent
Important
dates studies on social work
Swedish outcome
reports of the Swedish
articles in Sweden’sOfficial
largest
The
first
national
guidelines
National
action
plan
for
preventing
government
–
“Evidence-based
An
inquiry
initiated
at
the
NBHW
on
how
to
morning paper by the Director-General
onNational
socialpractice
work
(substance
alcohol
harm
&
action
social
work
– favouring
30of the Swedish
RCT related
Quasiexp
support
EBP
by
National Board
of
abuse
treatment)
plan
on
drugs
(lasting
to
2007).
clients”
National
guidelines,
research
General
election
and a new
Health and Welfare claiming that social reviews
25
100
million
SEKregional
on
research
National
and
knowledge
government
work agency
managers
did
not
have
Agreement
betweentransfer
the government
Training
of
local
prevention
workers
Developing criteria
for Swedish
local-follow
up and of
services
and the
Association
20any idea whether social care
open
comparisons
Addiction
Severity
made
any
difference
toIndex
clients’
lives.
IMS is inagurated
LocalisAuthorities
and regions on
introduced
in Sweden
15
implementing
national guidelines
Initiative to
clarify
the concept the
of EBP
on substance
abuse treatment
among governmental
authorities
35Two
10
5
0
1990
1992
1994
1996
1998
2000
2002
2004
2006
2008
Looking for an evidence-based
practice
Random sample of local authorities (2/3)
Managers responsible for budget & staff
Autumn 2007
Electronic questionnaire
Response rate 88% (n = 834)
INTEREST IN EVIDENCE-BASED
METHODS (N = 834)
Definitely
Partly
No
PREVALENCE OF EVIDENCE-BASED
METHODS (SOMETIMES)
100
24 per cent
80
60
%
40
20
0
Children
Youth
Substance
abuse
Disabled
Social
assistance
Elderly
PREVALENCE OF EVIDENCE-BASED
METHODS (MOSTLY)
100
7 per cent
80
60
%
40
20
0
Children
Youth
Substance
abuse
Disabled
Social
assistance
Elderly
To
<1
2
0
0
0
00
00
-2
5
00
n
e
ow
si
d
>2
5
12
w
n
w
n
n
w
n
ow
tr
ia
lt
w
n
ity
rb
s
to
rt
un
tr
y
us
To
To
rg
e
ub
u
m
ut
er
co
om
In
d
C
La
S
C
PREVALENCE OF EVIDENCE-BASED
METHODS (SOMETIMES)
100
80
60
%
40
20
0
PREVALENCE OF STANDARDISED
ASSESSMENT TOOLS (SOMETIMES)
100
63 per cent
80
60
%
40
20
0
Children
Youth
Substance
abuse
Disabled
Social
assistance
Elderly
PREVALENCE OF STANDARDISED
ASSESSMENT TOOLS (REGUAL)
100
14 per cent
80
60
%
40
20
0
Children
Youth
Substance
abuse
Disabled
Social
assistance
Elderly
Sweden: What are the political issues that
need to be addressed?
Keywords: Ethics – Evidence – Transparency
not the technical issues concerning internal
/ external validity
Knowledge transfer / dissemination of
innovations need extensive support at a regional
level
Bring forward good examples
Stockholm evidence-based
Clearinghouse for social work
•
”Evidence-based medicine is the conscientious, explicit and judicious
use of current best evidence in making decisions about the care of
individual patients” (Sackett et al., 1996)
•
The general aim of “the Stockholm evidence-based Clearinghouse for social
work” is to build a bridge between professionals and research on “what
works”
•
It was launched
on 1st September, 2008
18
Stockholm evidence-based
Clearinghouse for social work
•
Web-based service (at IMS’ homepage):
•
Provides information on interventions, assessment tools and
general knowledge (e.g., mechanisms)
•
•
•
Target group: Professionals, politicians and policy makers
•
Methodology of the systematic review (e.g., Higgins & Green, 2008)
http://www.socialstyrelsen.se/Amnesord/socialt_arbete/IMS/Metodguiden_index.htm
Not recommendations
Arranged in a simple, straightforward format reducing the need to
conduct literature searches
Topics Currently Available on the Website
Target groups
• Child/family
• Social assistance (for welfare recipients)
• Disability
• Addiction
• Elderly
Subheadings
• Interventions (18 descriptions/5 final evaluations)
• Assessment tools (46 descriptions/13 final evaluations)
• General knowledge
• IMS projects
20
The evaluation process
•
•
•
•
•
Selection of interventions to evaluate – starts with discussion in
IMS research council: Which interventions should we evaluate?
Which outcomes are important? What databases should be
searched?
All relevant studies with a RCT or QE (including data at baseline)
are reviewed
Use of a guide/protocol to assist the review process
Studies are evaluated for to their internal validity (i.e., can we trust
the result?)
Two independent reviewers evaluate each study. Any
disagreements are settled based on consensus with help from a
coordinator
22
Judging of internal validity
•
Selection bias
e.g., Are there any important differences between the groups?
•
Performance bias
e.g., Are researchers, participants and data collectors ”blind”?
•
Attrition bias
e.g., Any differences in size and type of attrition between the groups?
•
Detection bias
e.g., Are the outcome measures measured in the same way in all groups?
Potential bias is evaluated within and across each domain of bias
•
•
•
Low risk: plausible bias unlikely seriously alter the results
Unclear risk: plausible bias that raises some doubt about the results
High risk: plausible bias that seriously weakens confidence in the results
(from Higgins & Green, 2008)
23
Grading of evidence
• The scientific raiting scale is a modified version of a scale
developed at California Evidence-Based Clearinghouse for Child
Welfare (http://www.cachildwelfareclearinghouse.org/scientific-rating/scale)
• Based on statements in Flay et al., 2005 (Society for Prevention Research,
SPR)
• The scale is devided into five grades - A lower score indicates
effectiveness and a higher level of research support
• Grading of evidence is based on:
– Number of studies with a certain degree of internal validity (i.e., risk of
bias)
– If the practice is evaluated in usual care
– If the effect are sustained over time
– If the practice may cause harm
– If the practice is replicable
24
1. Effective practice with well-supported research evidence
•
•
•
•
•
At least two studies with low risk of bias, in different usual care or
practice settings, have found the practice to be superior to treatment
as usual (TAU).
If multiple effectiveness studies have been conducted, the overall
weight of the evidence supports the benefit of the practice.
In at least one study with low risk of bias, the practice has shown to
have a sustained effect at least one year beyond the end of
treatment.
There is no theoretical or empirical basis indicating that the practice
constitutes a substantial risk of harm to those receiving it.
The practice has a book, manual, and/or other available writings that
specify components of the service and describes how to administer
it.
2. Effective practice supported by research evidence
• At least one study with low risk of bias has found the practice to be
superior to treatment as usual (TAU).
• If multiple outcome studies (at least with medium risk of bias) have
been conducted, the overall weight of evidence supports the benefit
of the practice.
• In at least one study with low risk of bias, the practice has shown to
have a sustained effect of at least six months beyond the end of
treatment.
• There is no theoretical or empirical basis indicating that the practice
constitutes a substantial risk of harm to those receiving it.
• The practice has a book, manual, and/or other available writings that
specifies the components of the practice protocol and describes how
to administer it.
3. Practice with promising research evidence
• At least one study with medium risk of bias has established the
practice's benefit over no intervention (or placebo or waiting list) or is
found to be comparable to or better than treatment as usual (TAU).
• If multiple effectiveness studies with at least medium risk of bias have
been conducted, the overall weight of evidence supports the benefit of
the practice.
• There is no theoretical or empirical basis indicating that the practice
constitutes a substantial risk of harm to those receiving it.
• The practice has a book, manual, and/or other available writings that
specify the components of the practice protocol and describe how to
administer it.
4. Practice where the evidence fails to demonstrate effect
• At least two studies with low risk of bias have found that the practice
has not resulted in improved outcomes compared to no intervention
(e.g., placebo or waiting list), or that the practice is shown to be less
effective when compared to treatment as usual.
• If multiple effectiveness studies have been conducted, the overall
weight of evidence does not support the benefit of the practice.
• There is no theoretical or empirical basis indicating that the practice
constitutes a substantial risk of harm to those receiving it.
• The practice has a book, manual, and/or other available writings that
specify the components of the practice protocol and describe how to
administer it.
5. Concerning practice
• At least one study of low or medium risk of bias shows that the
intervention can cause serious harm, and/or there is a reasonable
theoretical basis suggesting that the practice constitutes a risk of
harm to those receiving it.
• The practice has a book, manual, and/or other available writings that
specify the components of the practice protocol and describe how to
administer it.
Beyond the five grades the scientific scale includes an additional category
(not rated - no number is given).
Practice with unknown effect
• There is lack of studies with a medium or low risk of bias.
• There is no theoretical or empirical basis indicating that the practice
constitutes a substantial risk of harm to those receiving it.
• The practice has a book, manual, and/or other available writings that
specifies the components of the practice protocol and describes how
to administer it.
Conclusions and summary Is there a sufficiently robust evidence base to identify good practice?
• What are the strength and weaknesses?
• Are there significant gaps that should be addressed?
• Is there an agreed approach to deciding what counts as evidence?
• Is there an agreed approach for judging the quality of the evidence
base?
• Is there a scope for working internationally to strengthen the
evidence-base?
IMS practical delivery
mechanisms to
promote the adoption
of good practice
IMS want to reach:
•
•
•
•
•
•
Social workers
Politicians and decision makers
Authorities
Universities and students
Researchers
Clients and others who are interested
Target groups
• Social work agency managers
• Government officials
Parts of our
communication strategy
• Value of direct, face-to-face communication
• Web-based services
• Tailored products
• Short courses for social work agency managers about EBP
• Start kit for municipalities for working with EBP
IMS STRATEGI
Target groups
Goals
Activities
Social workers
Increase the interest for
EBP
Lectures
Provide support for
decision-making
Readiness for change
ETICS
Politicians and decision
makers
Governments
INFORMATION
Handbook in leading
EBP
Implementation
TOOLS
FOR CHANGE
knowledge
Support to managers
-------Universitys
Clearinghouse
Long-term partnership
Pilot-project
Train the trainers
NOT COACHING Ask a researcher
SPECIFIC METHODSStart-up-package
IMS network for EBP
Organizational
Readiness for
Change
Anonymous survey
to co-workers, managers
and clients
Dwayne Simpson
Pilot-project Södermalm
- Trying different mechanisms to promote EBP
•
•
•
•
A steering group and 24 IMS-coodinatiors with special drive
Seminars about EBP and implementation
ORC-survey
Focusing on the managers with seminars about leading EBP
– the importance of the managers
– management research
– tools from the handbook
• Local seminars about how to find research and assessment tools
Training trainers in ASI
Addiction Severity Index
• Big difference when local support in the new method is given and when
the managers get’s support and owns the implementation
• 2 persons at IMS, 25 local trainers, 12 county administrative boards and
194 municipalities'
• Combination of IMS knowledge in implementation research and the
method ASI and the local knowledge about the practice and support to
the managers
• Success factors:
–
–
–
–
National support
Local competence
Managers in focus
Networking
Start-up- package
Focused on Managers
Study material as an
introduction to EBP
Content:







EBP – concepts and implications
Evidence, ethics and affectivity
Risk- and protective factors
Assessment tools
Research reviews
Dissemination
EBP and cost effectiveness
Including:
 DVD-films
 The book Evidence based practice in
social work
 The book To change social work
 PowerPoint material
 Glossary
 Information about IMS website
Publications
Collaboration with a
publishing house (Gothia
förlag)
Adapted to our target group
Reading guidances
IMS-nytt
New layout + more pages
4 numbers annually
Special feature issues
Research and interviews
with professionals and clients
Challenges
Challenges
1. Lack of interest among the Swedish social work academia to
support an EBP, and train the future generations of social
workers accordingly
2. Lack of (inter)national agreement on how to grade evidence
3. Questions on the transportability of evidence-based
interventions between countries
4. Lack of reliable and valid measures of quality of services at a
local level (in order to motivate change)
5. Lack of an infrastructure for diffusion of innovations, and
knowledge transfer
AIDA – acronym used in marketing
A
Attention
I
Interest
D
Desire
A
Action