Downloaded from http://bmjopen.bmj.com/ on June 17, 2017 - Published by group.bmj.com
PEER REVIEW HISTORY
BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to
complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf) and
are provided with free text boxes to elaborate on their assessment. These free text comments are
reproduced below.
ARTICLE DETAILS
TITLE (PROVISIONAL)
AUTHORS
The Green Shoots of Recovery: a realist evaluation of a team to
support change in general practice.
Bartlett, Maggie; Basten, Ruth; McKinley, Robert
VERSION 1 - REVIEW
REVIEWER
REVIEW RETURNED
GENERAL COMMENTS
Sam Porter
Dept Social Sceince and Social WOrk
Bournemouth Unoiversity
UK
27-Sep-2016
This is a well-written paper which adds new knowledge to an
important but under-researched aspect of service improvement. The
adoption of realist evaluation methodology is appropriate for the
focus of the research.
I have one significant concern about the paper as it stands. This
relates to outcomes (I think that both context and mechanisms are
handled well). The authors state that they used realist methodology
‘which seeks to explain how, why and when an intervention works by
considering configurations of contexts, mechanisms and outcomes
… rather than attempting to measure success against previously
defined outcomes’ (p7). I am not at all sure that this rejection of
previously developed outcomes is part and parcel of the realist
approach, but can accept the authors’ more pragmatic rejection of
extant performance indicators because of their instability and
unreliability (p.7). What is not clear is what is put in their place, yet
some very strong conclusions about outcomes are presented: ‘the
principle outcomes for the practices were better skill mix and
deployment of clinicians giving more appointments for patients,
better workflow arising from better deployment of administrative
staff, better communication within practice teams and increased
morale’ (p.19). However, when we look at the evidence, as
displayed in Table 2, there is scant support for most of these claims.
The table concentrates almost exclusively on psychological factors
that are amenable to qualitative evidence, such as morale and the
perception of communication quality.
From what is presented, I am not sure how judgements about the
quality of skill mix were made, and hence what would constitute a
better or worse mix; nor do I see any solid evidence about whether
or by how much those mixes improved between baseline and
completion of data gathering. I cannot see any empirical evidence
that the intervention led to more appointments for patients. Such a
clear numerical claim about rates requires equally clear numerical
demonstration. There are similar definitional and evidential gaps in
relation to workflow and staff deployment. Even the communication
finding rests on fairly fragile foundations.
Downloaded from http://bmjopen.bmj.com/ on June 17, 2017 - Published by group.bmj.com
While process in the form of contextual factors, intervention resource
mechanisms and human reasoning are well described, the
consequences of those processes need to be clearly articulated to
make the findings meaningful. As the paper stands, I do not believe
that this requirement has been fulfilled. The two options I can see
are either that a lot more solid evidence is included in the paper to
back up outcome claims, or that the claims are reduced to those
(such as morale) that can be supported by the evidence already
presented (or a combination of these two strategies).
A minor points re Q13 in reviewers’ checklist. I am assuming from
the reference list that the authors used Rameses guidelines, but
perhaps this could be made more explicit.
REVIEWER
REVIEW RETURNED
GENERAL COMMENTS
Dr John Frain
Division of Medical Sciences and Graduate Entry Medicine of the
University of Nottingham, United Kingdom.
None in relation to this study or area of study. I have been a coeditor on a book 'The ABC of Clinical Reasoning' (Wiley, 2016) to
which Maggie Bartlett contributed a chapter.
30-Sep-2016
This is a pertinent and timely study which will attract the interest of
the expert and general reader in primary care and commissioning
groups. The title may benefit from revision perhaps along the lines of
'Contexts, mechanisms and outcomes for an effective intervention of
a team supporting change in general practice using a realist
evaluation'. This would more clearly convey what the article is about
as the general reader may expect 'measuring success against
predetermined criteria'. This may then assist practices to decide if
such an intervention is for them.
Strengths and limitations of the study are well described. There is
discussion at the end of the paper on the authors' own experiences
of general practice but it would help to read more about the
investigators reflexivity regarding the topic including how this may
have evolved over the course of the study.
The introduction was well structured, easy and helpfully placed the
research question into context. The iterative process of developing
the methodology was transparent and the rationale for eventual
choices clear. Ethical issues are appropriately described.
The study population is well described and easy to follow including
the reasons for participation. The latter will assist the credibility of
the study for the general reader. The development of the SCGP's
journey through the process is balanced and clearly described
leading to a credible programme theory. Table 1 is clearly displayed
and easy to follow with a good balance of positive and negative
comments. Likewise with Table 2. Table 3 contains appropriate and
useful information but would benefit from revision of its layout as the
text, for a table, appears overly dense and crowded thus preventing
the immediacy of understanding evidence in Tables 1 and 1. The
programme specification is clearly explained. Figures 1 and 2
supplement and summarise the relevant text very well.
A well written paper of interest and value to those engaged in similar
Downloaded from http://bmjopen.bmj.com/ on June 17, 2017 - Published by group.bmj.com
processes already or considering becoming so.
VERSION 1 – AUTHOR RESPONSE
Reviewer: 1
Reviewer Name: Sam Porter
Institution and Country: Dept Social Sceince and Social WOrk, Bournemouth Unoiversity, UK
Please state any competing interests or state ‘None declared’: None declared
Please leave your comments for the authors below
This is a well-written paper which adds new knowledge to an important but under-researched aspect
of service improvement. The adoption of realist evaluation methodology is appropriate for the focus of
the research.
I have one significant concern about the paper as it stands. This relates to outcomes (I think that both
context and mechanisms are handled well). The authors state that they used realist methodology
‘which seeks to explain how, why and when an intervention works by considering configurations of
contexts, mechanisms and outcomes … rather than attempting to measure success against
previously defined outcomes’ (p7). I am not at all sure that this rejection of previously developed
outcomes is part and parcel of the realist approach, but can accept the authors’ more pragmatic
rejection of extant performance indicators because of their instability and unreliability (p.7). What is
not clear is what is put in their place, yet some very strong conclusions about outcomes are
presented: ‘the principle outcomes for the practices were better skill mix and deployment of clinicians
giving more appointments for patients, better workflow arising from better deployment of
administrative staff, better communication within practice teams and increased morale’ (p.19).
However, when we look at the evidence, as displayed in Table 2, there is scant support for most of
these claims. The table concentrates almost exclusively on psychological factors that are amenable to
qualitative evidence, such as morale and the perception of communication quality.
From what is presented, I am not sure how judgements about the quality of skill mix were made, and
hence what would constitute a better or worse mix; nor do I see any solid evidence about whether or
by how much those mixes improved between baseline and completion of data gathering. I cannot see
any empirical evidence that the intervention led to more appointments for patients. Such a clear
numerical claim about rates requires equally clear numerical demonstration. There are similar
definitional and evidential gaps in relation to workflow and staff deployment. Even the communication
finding rests on fairly fragile foundations.
While process in the form of contextual factors, intervention resource mechanisms and human
reasoning are well described, the consequences of those processes need to be clearly articulated to
make the findings meaningful. As the paper stands, I do not believe that this requirement has been
fulfilled. The two options I can see are either that a lot more solid evidence is included in the paper to
back up outcome claims, or that the claims are reduced to those (such as morale) that can be
supported by the evidence already presented (or a combination of these two strategies).
A minor points re Q13 in reviewers’ checklist. I am assuming from the reference list that the authors
used Rameses guidelines, but perhaps this could be made more explicit.
Reviewer: 2
Reviewer Name: Dr John Frain
Institution and Country: Division of Medical Sciences and Graduate Entry Medicine of the University of
Nottingham, United Kingdom.
Please state any competing interests or state ‘None declared’: None in relation to this study or area of
study. I have been a co-editor on a book 'The ABC of Clinical Reasoning' (Wiley, 2016) to which
Maggie Bartlett contributed a chapter.
Comment [MB1]: The wording has been
changed in the abstract and methods sections in
response to this comment. We have removed the
statement relating to previously defined outcome
Comment [MB2]: This is explained in the pape
– we have used the outcomes expected/desired b
the NHS England staff and the SCGP team, and
reported by the practices, and these were identifi
during the work, rather than being defined by the
evaluators based on their own perception of wha
the outcomes should be. This is in keeping with th
Realist approach.
Comment [MB3]: We have changed this to
clarify that the outcomes were those perceived b
practice staff rather than being measured using
‘quasi-experimental’ methods. The fact that pract
staff are seeing their work more positively is in its
evidence of increase morale.
Comment [MB4]: Staff perception is that
communication is better and there are multiple
statements of this. We feel that the strength of th
conclusion is reasonable. The main outcome that
were looking for was ‘the green shoots of recover
and these perceptions are in line with this.
Comment [MB5]: We have reduced the stren
of the claims as advised. We have also amended F
2 to reflect this.
Comment [MB6]: A clarifying statement has
been added to address this.
Downloaded from http://bmjopen.bmj.com/ on June 17, 2017 - Published by group.bmj.com
Please leave your comments for the authors below
This is a pertinent and timely study which will attract the interest of the expert and general reader in
primary care and commissioning groups. The title may benefit from revision perhaps along the lines
of 'Contexts, mechanisms and outcomes for an effective intervention of a team supporting change in
general practice using a realist evaluation'. This would more clearly convey what the article is about
as the general reader may expect 'measuring success against predetermined criteria'. This may then
assist practices to decide if such an intervention is for them.
Strengths and limitations of the study are well described. There is discussion at the end of the paper
on the authors' own experiences of general practice but it would help to read more about the
investigators reflexivity regarding the topic including how this may have evolved over the course of the
study.
The introduction was well structured, easy and helpfully placed the research question into
context. The iterative process of developing the methodology was transparent and the rationale for
eventual choices clear. Ethical issues are appropriately described.
The study population is well described and easy to follow including the reasons for participation. The
latter will assist the credibility of the study for the general reader. The development of the SCGP's
journey through the process is balanced and clearly described leading to a credible programme
theory. Table 1 is clearly displayed and easy to follow with a good balance of positive and negative
comments. Likewise with Table 2. Table 3 contains appropriate and useful information but would
benefit from revision of its layout as the text, for a table, appears overly dense and crowded thus
preventing the immediacy of understanding evidence in Tables 1 and 1. The programme specification
is clearly explained. Figures 1 and 2 supplement and summarise the relevant text very well.
A well written paper of interest and value to those engaged in similar processes already or
considering becoming so.
VERSION 2 – REVIEW
REVIEWER
REVIEW RETURNED
Sam Porter
Bournemouth University
England
18-Nov-2016
GENERAL COMMENTS
I think that the authors have dealt with my worries about
unsupported claims in relation to outcomes in an elegant fashion by
removing objective claims and referring instead to participants’
perceptions. I am therefore happy to recommend publication on the
proviso that the editor regards identification of outcomes using
perceptions rather than objective markers as sufficient. My own view
would be that a couple of sentences in ‘strengths and limitations’
clarifying the type and strength of evidence might be useful.
REVIEWER
Dr John Frain
University of Nottingham, United Kingdom.
REVIEW RETURNED
I know all of the authors and co-edited a book to which Maggie
Bartlett contributed a chapter.
13-Nov-2016
Comment [MB7]: We have considered this
suggestion carefully and consider that the use of
phrase ‘green shoots of recovery’ implies someth
less definite than numerically measurable
outcomes, and we state that it is a realist evaluat
which would alert readers to the type of work it is
and the way in which the work of the team is
evaluated. We would therefore prefer to keep the
title as it is.
Comment [MB8]: This discussion is a stateme
about conflicts of interests rather than being part
the main paper. We have added a statement to th
Strengths and Limitations section to acknowledge
that RKM and MB needed to think reflexively duri
the study because of their involvement in the loca
medical community in which the supporting chan
team works, and that bias may arise as a result of
informal knowledge of the team’s work. While th
can be reduced by self -awareness, it cannot be
eliminated completely.
Comment [MB9]: We revised the layout of th
table to make it easier to read. We have reduced
the density of the text by making it fit better in to
the boxes, changed to a bold font for sub-heading
which identify the sections on contexts,
mechanisms and outcomes, and removed the
narrative style which had been used in some
sections to make it more consistent and less word
Downloaded from http://bmjopen.bmj.com/ on June 17, 2017 - Published by group.bmj.com
GENERAL COMMENTS
The paper, figures and tables read well and are easy to follow. The
reviewers comments have all been assimilated and suitable
amendments made.
VERSION 2 – AUTHOR RESPONSE
With regard to the concerns of reviewer 1 about adding something to the 'strengths and limitations'
section about the type and strength of the evidence, we have inserted some extra information,
however, we felt that it has been made clear throughout the paper that the data is qualitative, and we
believe that this is neither a strength nor a weakness in itself. In a realist evaluation, what is being
considered is what works for whom, in what circumstances, and why, and this is based on the
perceptions of change of those concerned rather than an experimental or statistical process. Again,
this is not a strength or a limitation, but an inherent part of the methodology. We have explained that
we used the RAMESES II reporting standards to give rigour to the work and it is information about this
that we have added. We wish to avoid making any implication that qualitative data cannot be
considered to be strong evidence, as we do not believe that to be the case.
I hope we have addressed the reviewer's concern,
Downloaded from http://bmjopen.bmj.com/ on June 17, 2017 - Published by group.bmj.com
Green shoots of recovery: a realist evaluation
of a team to support change in general
practice
Maggie Bartlett, Ruth Basten and Robert K McKinley
BMJ Open 2017 7:
doi: 10.1136/bmjopen-2016-014165
Updated information and services can be found at:
http://bmjopen.bmj.com/content/7/2/e014165
These include:
References
This article cites 16 articles, 6 of which you can access for free at:
http://bmjopen.bmj.com/content/7/2/e014165#BIBL
Open Access
This is an Open Access article distributed in accordance with the Creative
Commons Attribution Non Commercial (CC BY-NC 4.0) license, which
permits others to distribute, remix, adapt, build upon this work
non-commercially, and license their derivative works on different terms,
provided the original work is properly cited and the use is
non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Email alerting
service
Receive free email alerts when new articles cite this article. Sign up in the
box at the top right corner of the online article.
Topic
Collections
Articles on similar topics can be found in the following collections
General practice / Family practice (629)
Medical management (221)
Notes
To request permissions go to:
http://group.bmj.com/group/rights-licensing/permissions
To order reprints go to:
http://journals.bmj.com/cgi/reprintform
To subscribe to BMJ go to:
http://group.bmj.com/subscribe/