Does therapeutic writing help people with long term
conditions? Systematic review, realist synthesis and
economic modelling
1. Background
Writing as a form of therapy to improve physical or mental health has a long history 1
and can take many formats including those from a psychotherapeutic background
such as therapeutic letter writing 2, specific controlled interventions such as
emotional disclosure/ expressive writing3, to more recent approaches such as
developmental creative writing 4 and other epistolary approaches such as blogging. 5
With the development of UK organisations such as LAPIDUS (Association for
Literary Arts in Personal Development) dedicated to the promotion of therapeutic
writing based on the premise that it has health benefits, it is important to evaluate the
effectiveness of a variety of different approaches within this field including:
• Emotional disclosure/expressive writing6
• Creative writing such as poetry4
• Reflective diaries
• New media such as blogging
• Other writing used in a health context7
The most evaluated form of therapeutic writing is the expressive writing intervention
as described by Pennebaker and colleagues. 3 This was evaluated extensively by
one of the CIs on this bid (CM) within her PhD.8-10 Narrative analysis within the
healthcare setting has also been extensively researched by another of the CIs
(TG).11-15
Expressive writing is a technique whereby people are encouraged to write (or talk
into a tape recorder) in private about a traumatic, stressful or upsetting event, usually
from their recent or distant past. They write for 15-30 minutes typically for 3-4 days
within a relatively short time period such as consecutive days or within 2 weeks.
Participants are encouraged to write about their deepest thoughts and feelings and
write about an event or experience they have not talked with others about in detail.
The format has been relatively consistent since the earliest RCTs3, 9 but more recent
studies have varied length of time, number of sessions and topic of writing, including
positive events and thoughts and feelings about illnesses. 16 The control group may
be no treatment, waiting list, or written control with a non-emotional topic.
RCTs of expressive writing have been conducted in a wide variety of participants
including healthy students, people undergoing psychological stressors such as
bereavement or being in a care-giving role or in people with long term physical
conditions such as rheumatoid arthritis and asthma. Some also include disclosure in
front of a listener who can be a confederate, a researcher or a doctor, which would
not be therapeutic writing. The presence of a listener is likely to affect outcome,
since it adds a counselling dimension. However, narrative research is moving
rapidly from the writing itself (narrative as noun) to a study of the process of narrating
for a specific audience and an analysis of the teller-listener relationship (narrative as
verb).17
1
Following expressive writing study participants can be followed up for a variety of
outcomes including physiological and immunological outcomes, objective and
subjective measures of physical health, performance and psychological outcomes.
If expressive writing affects physical health then there may be physical mechanisms
which would become apparent around the time of the intervention such as changes
in physiological, haematological and immunological parameters or a change in
health behaviours such as different eating habits, drinking patterns and smoking
behaviour which may subsequently impact on health outcomes. In part these
measures are reflective of the mechanisms through which expressive writing has
been hypothesised to work and have been summarised extensively by a variety of
authors including Esterling et al. 18 The basis of expressive writing is emotional
expression which is thought to be beneficial for health in contrast to emotional
inhibition which is thought to have detrimental effects. 19 This effect may occur
through several pathways. Cognitively it is suggested that the act of writing could
lead to more developed cognitive representations of the written-about event, which
may reduce the underdeveloped feeling states that can occur after trauma, and
hence also positively impact on emotions. 18 Alternatively Cresswell et al. 20
suggested that it is self-affirmation in contrast to cognitive processing or discovery of
meaning that could mediate any putative effect on physical symptoms. At an
immunological level, studies have shown that individuals participating in expressive
writing possibly had increased antibody levels against hepatitis B 21 and improved
cellular immune function to Epstein-Barr Virus.18 Yet other studies suggest the
mechanism to be behavioural e.g. impact on sleep22 but the relatively poor quality of
some reporting make conclusions tentative. In short, it is currently unclear whether
expressive writing does improve physical health, or by which mechanisms this might
occur.
Expressive writing does appear to heighten negative mood to a small degree for the
first few hours or days after the intervention but this effect quickly fades.23 It was also
widely reported that volunteers found that expressive writing and taking part in RCTs
of expressive writing had been beneficial to them and helped them try to understand
or come to terms with emotional difficulties from the past.23 However, this may be a
sampling effect, people who have something to disclose and are ready to disclose it
may be the ones who volunteer for the trials.
Expressive writing is thought to be beneficial for longer term health effects and as
such is now frequently referred to in general psychology textbooks as a potentially
beneficial intervention 24. However, some have argued that this intervention is too
brief to have any long lasting effects. An early BMJ editorial 13 on an emotional
disclosure/expressive writing RCT published in JAMA 25 suggested possible reasons
for the positive findings such as self selected populations being aware of their
allocation to intervention and control group and that the outcomes measured were
potentially open to recruitment and assessment bias. Since then there has not been
published much critique specifically on expressive writing.
Meads and colleagues (2005) 9 suggested possible reasons for an apparent
beneficial effect found within meta-analyses of emotional disclosure/expressive
writing RCTs including:
2
•
•
•
•
Poor allocation concealment
Lack of blinding of the intervention
Intention-to-treat analysis not being used
Selective reporting bias, ie RCTs where numerous outcomes were measured
but only reporting those where a significant improvement was found
Given that there has been a considerable number of RCTs published since this
systematic review was completed, there is a renewed need to evaluate the
effectiveness of this intervention.
Those within the field of developmental creative writing however consider that
Pennebaker’s expressive writing paradigm may be more a starting point in learning
to release emotion through writing, but that added benefit may occur with more ‘free
writing’ which could allow for development and shaping of the material, which leads
to a ‘a new relationship with aspects of self-experience’. 4 It is this connection with a
core sense of self from which creative writing is said to derive benefit. 4 With newer
forms of writing, such as blogging, association with increased perceived social
support has been demonstrated. 5 Given the importance of social support to wellbeing this suggests yet further mechanisms by which writing in its various formats
may improve health.
It is a popular assumption that creative writing helps overcome life’s stresses, and
some professional writers have noted this. For example, as Lowe (2006) notes,26
Virginia Woolf acknowledged the importance of her writing of To the Lighthouse in
working through her mental health issues. On the other hand, professional writers
with mental health problems have not necessarily attributed any therapeutic effect to
writing itself (e.g. Sylvia Plath, Stephen Fry). Notwithstanding the epistemological,
methodological and ethical challenges of studying the impact of creative writing on
mental health, it seems appropriate to evaluate whether, the wider field of
therapeutic writing might help with mental health issues in a clinical population and
also whether it may assist those with chronic physical conditions.
2. Work leading to the proposal:
2.1 Preliminary review of reviews, systematic reviews and meta-analyses
We undertook a scoping search to examine the existing literature on therapeutic
writing in a medical context. There have been several systematic reviews and metaanalyses on emotional disclosure/expressive writing, one of which was undertaken
by our own team.9, 23, 27-29 On the Cochrane Library, there is a protocol for the
evaluation of written emotional disclosure for asthma (Theadom et al., 2009). Our
preliminary scoping searches did not find any previous economic evaluations of any
forms of therapeutic writing. None of the systematic reviews or meta-analyses
incorporate a qualitative systematic review or investigate context as in a realist
review.
One of the most recent28 searched up to 2004 and included 146 randomised studies
of which 32 were in participants with chronic (physical and mental health) conditions,
the remainder being in healthy students and people under stressful situations such
as caregiving and bereavement. A number of the included studies were grey
literature such as PhD theses and manuscripts submitted for publication. Effect sizes
3
per study were calculated using averaging across different outcome measures.
Large numbers of meta-analyses (approximately 170) were performed on subgroups
of the papers, using standardised mean difference and a random effects model and
20% (approximately 34) were found to give statistically significant results. For those
with physical health selection criteria (no definition given) the overall effect size was
r=0.090 (p (one tailed) = 0.14). This meta-analysis included any emotional disclosure
interventions so some included studies did not have participants writing but speaking
into a tape recorder instead. The way the results were handled meant that it would
not be possible to distinguish patterns of results across single outcomes. For
example, it was not possible to tell if emotional disclosure alleviated depression in
participants who had been diagnosed with depression. Depression and anxiety were
accumulated with stress in a psychological health category. The physiological
functioning category included immune parameters, heart rate and liver functioning,
amongst other symptoms and signs, but did not include self-reported physical
symptoms which was placed in a different category of reported health. The results of
this meta-analysis are very difficult to interpret.
The other systematic review and meta-analysis from 2006 29 specifically looked at
RCTs of expressive writing that measured health care use. Thirty RCTs were
included of which six were in participants with pre-existing medical conditions and
the RCT with prison inmates with psychiatric conditions was placed in a different
category of stressed or psychologically defined participants. Only English language
papers were included. Meta-analysis was by using standardised mean differences
rather than weighted mean differences and used a random effects model. The
combined effect size for medical participants was 0.21 (95%CI -0.02 to +0.43).
The meta-analysis by Frisina and colleagues (2004) 27 focused on emotional
disclosure/expressive writing in clinical populations and included nine RCTs in
participants with breast, prostate and renal cancer, asthma and rheumatoid arthritis,
PTSD and psychiatric prison inmates. Meta-analyses were conducted using
techniques first described in the 1980s, and an overall effect size was calculated
using a standardised mean difference but a fixed effects model was used in spite of
the clinical heterogeneity. The overall effect size for health outcomes was d=0.19
(p<0.05) but this is very difficult to interpret. It is possible that some RCTs were
missed in the searches 10 but this may not have made any difference to their
estimate of aggregated effect size.
The systematic review by Meads (2005) used techniques more similar to those
commonly found in the medical literature. 9 The searches were conducted in 2002-3
so included fewer RCTs than the later meta-analyses. When participant types and
outcomes were categorised into homogenous groups, no clear trend of improvement
with emotional disclosure was shown, particularly for clinical populations and
objective measures of health outcome. The other meta-analysis 23 is too early to be
as relevant so is not discussed here.
With regard to creative writing, most of the studies identified so far are of
uncontrolled case series design, with examination of social and psychological factors
after the writing workshops and the written texts themselves rather than on any
potential impact on physical or mental health. A relatively recent review of the healthrelated effects of creative and expressive writing26 did not discuss any controlled
4
studies of creative writing, (for example writing poetry) where participants were
followed up for health outcomes. However, this literature is important to investigate
and may be harder to find because it could be published within humanities journals
rather than medical journals and also in grey literature. Given the large number of
expressive writing RCTs, there is a risk that other therapeutic writing interventions
could be overlooked.
3. Objectives
Our proposal will follow the key steps involved in establishing whether there are any
benefits of therapeutic writing for people with long-term mental and physical
conditions and will meet the commissioning brief by addressing the following
questions:
1. What are the different types of therapeutic writing that have been evaluated in
comparative studies? What are their defining characteristics? How are they
delivered? What underlying theories have been proposed for their effect/s?
2. What is the clinical effectiveness of the different types of therapeutic writing
for long term health conditions compared to no writing or other suitable
comparators?
3. How is heterogeneity in results of empirical studies accounted for in terms of
patient and/or contextual factors, and what are the mechanisms and
moderators responsible for the success, failure or partial success of
interventions (i.e. what works for whom in what circumstances and why)?
4. What is the cost effectiveness of one or more types of therapeutic writing in
one or more representative chronic conditions where there is sufficient
information on the intervention, comparator and outcomes to conduct an
economic evaluation.
Components of the review questions are stated in Table 1 (see below).
4. Relevance to commissioning brief
The HTA commissioning brief (11/70) states that ‘Although improvements in health
outcomes have been demonstrated in both healthy and clinical populations, findings
are sometimes contradictory’. We concur with this statement and suggest that our
project should take the direction of an inclusive review of all relevant evidence with
considerable quality assessment to discover whether this intervention does work or
whether any apparent effectiveness is a result of the way the research to date has
been conducted. We have found much RCT evidence on one aspect of therapeutic
writing namely an intervention known as emotional disclosure or expressive writing
but it is also important to evaluate other forms of therapeutic writing.
Recommendations for research can be made with more confidence once an
overview of the effectiveness, realist synthesis and cost-effectiveness of different
types of therapeutic writing has been conducted. We will also focus on the most
effective aspects of a therapeutic writing intervention in order to suggest ways in
which this might be beneficial to people with long term conditions.
5
5. Research Methods
5.1 Design: Systematic review of effectiveness, realist review and economic
modelling
We will carry out our two interlinked reviews simultaneously (see Table 1 for
inclusion criteria). For some of the therapeutic writing interventions we will use
controlled observational studies, because of lack of randomised evidence. We will
take care to evaluate risk of bias carefully (see quality assessment section below).
The effectiveness systematic review will establish if any of the different forms of
therapeutic writing are shown to be effective and the realist synthesis, by asking
“what works for whom in what circumstances?” is well placed to explain
heterogeneity in the findings of comparative effectiveness studies. Economic
evaluation methods are described in section 5.5. Using the results of the systematic
reviews we will estimate cost effectiveness of therapeutic writing if there are
sufficient well-powered RCTs with homogenous outcomes in the same disease area
or areas. Realist synthesis is described in section 5.6.
5.2 Design: Definition of long-term conditions
The Department of Health estimates that around 15 million people in England
(including half of all those aged over 60 years) are living with at least one long term
condition.45 There is, however, no definitive list of long term conditions and the
potential range of diseases of interest is both extensive and diverse. For the
purposes of this review we will adopt the UK Department of Health definition of a
long term condition: "Long term conditions are those conditions that cannot, at
present, be cured, but can be controlled by medication and other therapies. They
include diabetes, asthma, and chronic obstructive pulmonary disease." Our working
definition of long term conditions also includes mental health problems, including
eating disorders, chronic infections such as HIV and cancer where the condition is in
remission, or effectively in remission. There is a debate around whether obesity in
the absence of any co-morbidity is a disease46 and we will exclude studies in people
with uncomplicated overweight and obesity.
Table 1: Review question components
Question
Systematic review
components Inclusion criteria
Population
Any long term condition as per DoH
definition.
Exposure /
Any form of therapeutic writing
Intervention including emotional disclosure/
expressive writing, poetry, diaries
etc
Comparison
Non writing, waiting list,
inexpressive writing, any control
thought to be inactive.
Outcome
Any relevant clinical outcomes
including both disease specific
outcomes and generic outcomes
Exclusion criteria
Acute conditions, stress,
bereavement etc.
Talking to a listener,
counselling, psychotherapy,
talking into a tape recorder,
mobile phone or similar,
expressive drama, dance,
filmmaking.
Any active or possibly active
control including therapeutic
writing or talking into a tape
recorder or mobile phone.
Intermediate physiological
outcomes such as salivary
cortisol, immune parameters.
6
Study
designs
such as quality of life. Health
service use, psychological
outcomes, behavioural outcomes,
social functioning, adverse events,
adherence to therapies, costs.
Any comparative studies including
RCTs, cohort or case control
studies. Economic evaluations.
Single case reports, case
series, studies where results
for intervention and control
groups not presented
separately.
5.3 Search Strategy
For this project a database of published and unpublished literature will be assembled
from searches using a comprehensive search strategy, as well as hand searching
and contact with experts in the area. One of the challenges we will face relates to the
number of controlled studies which have been conducted as theses and
dissertations (for example, 48 of 146 included studies in the Frattaroli 2006 metaanalysis) and only available as conference abstracts (a further 14 studies in Frattaroli
2006). It is important to include grey literature because of the possibility that effect
size estimates may have been overestimated due to selective reporting bias and
unpublished studies are known to be less likely to have statistically significant results
compared to published studies.47
The following databases will be searched: MEDLINE, EMBASE, PsychInfo, CAB
Abstracts, PEDro, PILOTS, Zetoc, Science Citation Index, Social Sciences Citation
Index, Linguistics and Language Behavior Abstracts, Periodicals Index Online,
Applied Social Sciences Index and Abstracts (ASSIA), ERIC, AMED, Cochrane
Central Register of Controlled Trials (CENTRAL) and Database of Abstracts of
Reviews of Effects (DARE) will be searched for primary studies and the Cochrane
Database of Systematic Reviews (CDSR), Health Technology Assessment Database
(HTA) and the Campbell Library will be searched for systematic reviews and
economic evaluations. In addition, information on studies in progress and
unpublished research or research reported in the grey literature will be sought by
searching a range of relevant databases including the Inside Conferences, Systems
for Information in Grey Literature (OpenSIGLE), Dissertation Abstracts, Current
Controlled Trials database and Clinical Trials.gov. Internet searches will also be
carried out using specialist search gateways (such as OMNI:
http://www.omni.ac.uk/), general search engines (such as Google:
http://www.google.co.uk/) and meta-search engines (such as Copernic:
http://www.copernic.com/). Language restrictions will not be applied to searches.
Citations identified by the search will be selected for inclusion in the review in a twostage process using predefined and explicit criteria regarding populations,
interventions, comparators, outcomes and study design. First, a master database of
the literature searches will be constructed by amalgamation of all the citations from
various database sources. The citations will be scrutinised by two reviewers. Copies
of full manuscripts of all citations that are likely to meet the selection criteria will be
obtained. Two reviewers will then independently select the studies that meet the
predefined criteria. These criteria will be pilot tested using a sample of papers and
7
agreement between reviewers will be sought. Disagreements will be resolved by
consensus and/or arbitration involving a third reviewer.
Once the final sample for the systematic review has been identified, each paper will
be tracked in Google Scholar and titles screened independently by two reviewers to
identify sister papers of these documents.
5.4 Systematic review of effectiveness
As before, studies’ findings will be extracted in duplicate using pre-designed and
piloted data extraction forms, which we have already developed in our previously
completed reviews. Any disagreements will be resolved by consensus and/or
arbitration involving a third reviewer. Individual studies will be described by study
type, intervention, numbers taking part, population denominator and study quality.
Missing information will be obtained from investigators if it is crucial to subsequent
analysis. To avoid introducing bias, unpublished information will be coded in the
same fashion as published information. In addition to using multiple coders to
ensure the reproducibility of the overview, sensitivity analyses around important or
questionable judgements regarding the inclusion or exclusion of studies, the validity
assessments and data extraction will be performed.
The quality of the selected primary RCTs and observational studies on effectiveness
of therapeutic writing will be assessed based on accepted contemporary standards4850
. To assess quality, we will consider the risk of bias (internal validity), i.e. the extent
to which design, methods, execution and analysis did not control for bias in
assessment of effectiveness 51. We will use validated tools appropriate to the study
design such as the Newcastle Ottawa scale for cohort and case control studies 52. In
addition to using study quality as possible explanations for differences in results
(heterogeneity), the extent to which primary research met methodological standards
is important per se for assessing the strength of any conclusions that are reached
about effectiveness. The GRADE methodology 53-55 will guide us when assessing the
quality of the evidence overall and summarising the results. We have previously
used the GRADE methodology in our reviews 56 and have developed graphic
representations for easier interpretation57.
Meta-analyses will be conducted using standard software packages such as STATA
which will allow data analytic flexibility that may potentially be needed. A special
problem that we are likely to face is very little RCT evidence, which is why
observational studies will be included. Separate analyses will be performed on
randomised and non-randomised data. We have expertise in combining results of
categorical and continuous measures for the same outcomes (eg odds ratios and
mean differences). Any heterogeneity of results between studies will be statistically
and graphically assessed, including use of funnel plots. We will explore causes of
the heterogeneity and proceed to perform meta-analysis if appropriate 58. To explore
causes of heterogeneity subgroup analyses will be planned a priori to see whether
variations in clinical factors e.g. populations, interventions, outcomes or study quality
affect the estimation of effects. Individual factors explaining heterogeneity will also
be analysed using meta-regression to determine their unique contribution to the
heterogeneity if possible 59 and we have experience in this technique. Conclusions
regarding the typical estimate of an effect size of the intervention will be interpreted
8
cautiously if there is significant heterogeneity. If necessary, we will use indirect
comparisons to inform the economic model.
5.5 Health economic evaluation
This is a broad term to describe a variety of approaches that can be used to illustrate
the economic consequences of a therapeutic strategy. In a project such as this, it is
important to be flexible with regards to planning an economic evaluation because the
depth of complexity of any economic modelling, for example, will be driven partially
by the data available. For example, if no health-related quality of life information was
found in the extensive systematic review searches, it may not be appropriate to state
in advance that a cost utility analysis will be conducted. However, value for money is
an important consideration in the current economic climate so any information on
costs and cost effectiveness compared to an appropriate comparator will be
presented.
If there are well-powered RCTs with homogenous outcomes in the same disease
area or areas, we will then associate improvements in outcomes with gains in health
related quality of life where possible. This may include use of decision-analytic
modelling either by using an existing disease-specific model available in the
literature or by constructing a de-novo model. The aim will be to estimate the
incremental cost-effectiveness of incorporating therapeutic writing into the currently
recommended NHS treatment regimen for the particular disease area. Results will
be presented in terms of cost per quality adjusted life year (QALY) gained, with the
uncertainty in both the RCT evidence and the modelling incorporated. If the
information in the literature found in the systematic review is not sufficient then we
will carry out more general forms of economic evaluation such as costconsequences or cost minimisation analyses. All economic evaluations will follow the
reference case used by the National Institute for Health and Clinical Excellence
(NICE) as far as the available evidence permits, for example discounting costs and
benefits at 3% per annum, and using the perspective of the NHS and social services.
The economic modelling component of the work is likely to be challenging. Some of
these challenges are likely to include the following: (i) the intervention is likely to
affect a wide range of disease areas and clinical outcomes, which cannot be
captured in a single disease model; (ii) there may be a need to combine evidence
that is heterogeneous in terms of quality (mixture of observational studies, trials and
qualitative studies) as well as the nature of the intervention and population studied;
(iii) there may not be sufficient disease natural history and quality of life information
in the literature to conduct a cost-utility analysis to the specifications of the NICE
reference case.
To address these challenges, we have deliberately left the precise nature of the
health economic evaluation open ended until after the literature reviews are
completed. Whenever possible, we will conduct a “gold standard” cost-utility
analysis, using a decision analytic model to examine the impact of the intervention
on disease progression, with its parameters informed by a synthesis of the highest
quality RCT evidence, and with outcomes presented in terms of costs per QALY.
However, we envision that the opportunities for this kind of analysis are likely to be
slim. In most cases, we will make the most of any evidence available, for instance
9
conducting cost-consequences analyses if there is no quality of life information or if
there are a range of different outcomes that cannot be captured within a single
model. Such analyses will still provide useful information to guide decision making,
and will also highlight the gaps in evidence that can be addressed by future studies
5.6 Realist review
Realist reviews ask “what works for whom in what circumstances?” and considers
the interaction between context, mechanism and outcome (sometimes abbreviated
as C-M-O). i.e. how particular contexts have ‘triggered’ (or, conversely, interfered
with) mechanisms to generate the observed outcomes 60 . The philosophical basis is
realism, which assumes the existence of an external reality (a ‘real world’) but one
that is ‘filtered’ (i.e. perceived, interpreted and responded to) through human senses,
volitions, language and culture. Such human processing initiates a constant process
of self-generated change in all social intuitions, a vital process that has to be
accommodated in evaluating social programmes.
In order to understand how outcomes are generated, the roles of both external reality
and human understanding and response need to be incorporated. Realism does this
through the concept of mechanisms, whose precise definition is contested but for
which a working definition is ‘…underlying entities, processes, or structures which
operate in particular contexts to generate outcomes of interest’ 61. Different contexts
interact with different mechanisms to make particular outcomes more or less likely –
hence a realist review produces recommendations of the general format “In
situations [X], complex intervention [Y], modified in this way and taking account of
these contingencies, may be appropriate”. This approach, when done well, is widely
recognised as a robust methodology which is particularly appropriate when seeking
to explore the interaction between C-M-O in a complex intervention. (See for
example Berwick’s editorial in JAMA explaining why experimental (RCT / metaanalysis) designs may need to be supplemented (or perhaps in some circumstances
replaced) by realist studies aimed at elucidating C-M-O configurations 62).
A realist approach is particularly useful for this project because therapeutic writing is
a complex intervention which could be useful in a variety of patient groups and
currently it is unclear whether it is effective for all or some. Realist methodology has
been shown to add particular value in the six situations (alone or in combination) 63,
of which three are particularly pertinent to this review:
1) When RCTs (or non-randomised comparative trials) of particular
interventions have produced inconsistent estimates of efficacy and
there is no consensus on when, how and with whom to use these
interventions;
2) When the existing research on a particular intervention consists
mainly of disparate qualitative studies and ‘grey literature’ accounts
(e.g. internal evaluations, PhD theses) that do not lend themselves
to statistical synthesis but provide a rich source of qualitative data;
3) When new interventions are being trialled in order to identify how
and for whom they are effective.
What will be the added value of a realist approach to assess the C-M-O
configurations in therapeutic writing papers? The most important thing to note is the
potential for this approach to generate data about the mechanisms by which
10
therapeutic writing may achieve impact in different settings and subgroups. To
illustrate this, we quote from our recent paper on realist methods in medical
education 64:
“Imagine, for example, asking learners on a widening-access course for
medicine to write free text responses to the question: ‘How have you changed
as a result of coming on this course?’ Students might variously respond: ‘I
started to think more deeply about scientific problems’; ‘I met real medical
students and saw that you could still have fun whilst learning medicine’; ‘I
started to believe in my own ability’, or ‘I made some friends and we are
planning to keep in touch on Facebook as we prepare for the UKCAT test’.
These responses give an inkling of the complex outcomes that might be
generated by a widening-access course and also suggest potential
mechanisms by which the opportunities provided by the course might improve
students’ likelihood of applying to medical school and their competitiveness
for places, thereby widening access. Expressed at a slightly higher level of
abstraction, these mechanisms might be described as ‘promoting reflection
and deep learning’, ‘increasing motivation through vicarious experience’,
‘building confidence’ and ‘providing mutual support’.
“This example illustrates that the mechanisms by which educational
interventions ‘work’ are often multiple, that some mechanisms are obvious
and correspond to those intended by the course designers, and that some are
less obvious and are unanticipated by the designers. It also illustrates that a
mechanism is not inherent to the intervention, but is a function of the
participants and the context.”
This would certainly be true with therapeutic writing because the meanings that
patients make of the writing itself and the processes and ideas that come to them
whilst writing would be very difficult to capture in numerical results of conventional
outcomes of RCTs.
Realist review involves a process of interpretation and judgement: the central task is
to identify the mechanisms (defined above) by which an intervention might achieve
its impact, and consider how these mechanisms play out differently with different
actors in different contexts. Some people with chronic illness might respond well to a
particular therapeutic writing intervention while others might respond better to a
different intervention or not respond at all. A realist review begins with an initial
‘approximate’ theory of change and uses each document as an opportunity to refine
this, using the following specific analytic techniques:
• juxtaposing (e.g. when one study provides process data which help make
sense of outcomes noted in another study)
• reconciling (identifying differences which explain apparently contradictory sets
of findings)
• adjudicating between studies (e.g. in terms of quality of research)
• consolidating (producing multi-faceted explanations of success, failure and
partial success)
• situating (this mechanism in context A, that one in context B)
11
The realist review will cover the papers in the above sampling frame (see Table 1)
along with “sister papers” i.e. any qualitative or mixed-method studies linked to these
index papers, but published as a separate paper. The steps by which we will apply
the realist approach are briefly summarised below 63.
“Realist … uses interpretive cross-case comparison to understand and
explain how and why observed outcomes have occurred in the studies
included in a review. The working assumption behind realist review is that a
particular intervention (or class of interventions) will trigger particular
mechanisms somewhat differently in different contexts. In realism, it is
mechanisms that trigger change rather than interventions themselves and
thus realist reviews focus on ‘families of mechanisms’ rather than on ‘families
of interventions’. An explanation and understanding of the interplay between
context, mechanism and outcomes are then sought. The reviewer constructs
one or more middle-range theories to account for the findings.
“The realist reviewer moves iteratively between the analysis of particular
examples, an emerging picture of the over-arching programme theory, and an
exploratory search for further examples to test particular theories or subtheories.
“The pursuit of rigour in realist research reflects principles usually seen in
qualitative research, although it may draw on qualitative, quantitative or mixed
methods. Much rests on achieving immersion (i.e. reading and re-reading
papers to really understand what was done and why), thinking reflexively
about findings, developing theory iteratively as emerging data are analysed,
seeking disconfirming cases and alternative explanations, and defending
one’s interpretations to researchers within and outside one’s own team.”
Within the community of realist researchers there is much disagreement on the fine
detail of the approach. Two co-applicants (TG&GW) are currently leading an
international project, the RAMESES study, to develop guidance for realist review.
This involves a systematically recruited group of international experts who are now in
the final stages of producing methodological guidance and publication standards.
The methodology for producing this guidance has been published as a detailed
open-access methods paper which is already ‘highly accessed’ 65; it builds on and
extends the methodology used for producing the CONSORT and QUORUM
statements. All the empirical work for producing the RAMESES statement (including
three rounds of a Delphi panel) has been completed, and high levels of agreement
have now been reached on the final versions of each recommendation. The paper
describing the standards will be submitted by end June 2012 and we anticipate that
a fast-track publication will be available open access from September 2012 at the
latest. We will ensure that the new RAMESES standards (which we anticipate will
become the gold standard for realist review internationally) are strictly followed in the
realist component of this study.
12
5.7 Narrative synthesis of the systematic review of effectiveness, the realist
review and the health economic evaluation
There are two published examples of combining conventional systematic review
methods with realist review methods, both of which were led by Prof Greenhalgh.
One addressed the question 'What is the impact of school feeding programmes on
growth and educational achievement in deprived children, and what explains
variation in the findings across studies?' 66,67. This review considered randomised
trials and other experimental designs. The other review addressed the question
'What are the components and statistical properties of different risk scores for type 2
diabetes, and what explains whether and how these scores were used?' 63. This
review considered longitudinal and cross-sectional population cohort studies and
papers describing case studies of attempts to implement the risk score.
The way the systematic and realist methodologies were applied was slightly different
in each case, but essentially each review consisted of two separate reviews – one
systematic and one realist, which were presented separately with some joining text
and commentary ('narrative synthesis'). In the school feeding review, the sample of
studies for the realist review was identical to the sample for the systematic review of
experimental trials. We decided to do a realist review because of the heterogeneity
in design of the primary studies and also the heterogeneity of the findings. In short,
some school feeding interventions appeared extremely effective and some appeared
ineffective. The authors of each study had often speculated on why they got the
results they did, and had also described the practical details of their interventions.
This provided us with data on context and mechanism, which allowed us to make
sense of the qualitative data in the published studies. So really, all we were doing
was using a systematic and theory-driven method for pulling together data from the
discussion sections of RCTs. The Cochrane review answered the question 'what is
the effect size and how does this vary (statistically) across studies?'; the realist
review allowed us to add a more nuanced commentary: 'by what mechanism might
school feeding produce growth and/or educational gains, and what might explain
lack of growth or lack of educational gain in such programmes?'.
In the diabetes risk score example a review of the statistical properties of published
risk scores undertaken using the MOOSE guidelines identified over 140 such scores,
but it was evident that few if any of these scores were in widespread use. This was
an unexpected (and surprising) interim finding. Research on diabetes risk scores
seemed to have become a sort of race between statistically-oriented research teams
to produce the score with the 'best' statistical properties (e.g. best area under ROC
curve) but in the pursuit of this goal, it also seemed that they were producing scores
that were impractical, expensive and based on tests that clinicians in the real world
didn't actually do. This finding prompted us to use citation-tracking methods to see
which papers (if any) had subsequently cited the risk score validation study, and this
confirmed that of the 140+ scores, only a handful had ever been cited, except by
other statistical teams. To explore this finding further, we collected all the papers we
could find on the IMPLEMENTATION and USE of diabetes risk scores, and did a
realist review on that, to address the question 'what explains use and/or non-use of
diabetes risk scores?'. Again, the realist review was not part of the original plan, but
was a good match with the unexpected finding that emerged as the quantitative
review was being undertaken.
13
In both these examples, we linked the two reviews using an over-arching narrative
and we will conduct a similar interdigitation between the two reviews in this project.
This 'narrative summary' or 'narrative synthesis' is a very well established and robust
way of linking two sets of review findings, especially when those sets of findings are
philosophically incommensurable and/or address different research questions within
a single study (hence do not lend themselves to a 'technical' approach to combining
findings). The technique of narrative summary is listed by Mary Dixon-Woods et al in
their overview of different techniques for summarising and/or integrating mixed
evidence 68. To quote from their paper:
"Narrative summary typically involves the selection, chronicling, and ordering
of evidence to produce an account of the evidence. Its form may vary from the
simple recounting and description of findings through to more interpretive and
explicitly reflexive accounts that include commentary and higher levels of
abstraction. Narratives of the latter type can account for complex dynamic
processes, offering explanations that emphasise the sequential and
contingent character of phenomena. Narrative summary is often used in
systematic reviews alongside systematic searching and appraisal techniques
[examples]. Narrative summary can ‘integrate’ qualitative and quantitative
evidence through narrative juxtaposition – discussing diverse forms of
evidence side by side – but, as a currently largely informal approach, is likely
always to be subject to criticism of its lack of transparency. However, under
the UK ESRC Methods Programme, methodological guidance on the conduct
of narrative summaries is being developed, which will inform future good
practice in this area."
The ESRC-funded methodology study to which Dixon-Woods et al refer was
undertaken by Popay et al and published as an ESRC methods paper with detailed
approaches and techniques for different types of primary data 69. Table 2 in that
monograph lists two contrasting tasks for which narrative summary might be used –
effectiveness reviews, such as the one we will undertake for this proposal, and
implementation reviews.
We will also be integrating the results of the economic evaluation with the realist and
conventional systematic reviews. Combining economic evaluation with systematic
reviews is very commonly done for HTA reports so needs no further explanation
here. However, what is frequently not discussed is how the clinical context is
combined with the economic evaluation to enable the HTA report to have clinical
credibility. This is frequently done by ensuring that the clinicians on the project are
present when discussing the economic model, fully understand it and can appreciate
the implications of the assumptions made. Their insights into the patient experience
often result in the structure of the model needing to be changed or different
numerical inputs being used to reflect clinical reality. In our project we will be fully
involving the patient representative in all aspects of the project. However, there is no
clinical speciality of therapeutic writing in the same way that there is for, say,
psychiatry or gastroenterology. An equivalent input in the therapeutic writing project
will be from the realist review which will be able to ground the other aspects of the
project on the meanings and contexts of the patients. We will also endeavour to find
clinicians skilled in the use of therapeutic writing who could provide useful input to
the project.
14
5.8 Expected Output of Research
Based on a comprehensive search, systematic reviews and statistical syntheses of
the available data we will be able to determine whether there is sufficiently good
evidence to confirm or refute whether therapeutic writing is beneficial in a clinical
context and to indicate conditions and/or subgroups and/or settings in which different
types of therapeutic writing might be particularly helpful or unhelpful. A major
strength of our team is that we have no vested interests in establishing whether any
one writing intervention is effective or not (ie we have no personal non-pecuniary
interests). We also have strong community engagement in having a patient advocate
as one of our CIs. We will also be working with the consumer advisory panel at the
William Harvey Research Institute at Queen Mary University of London with
consumers with chronic conditions recruited from the large and very active consumer
group. We also have strong links with the Pelvic Pain Support Network.
In addition we will endeavour to estimate the cost-effectiveness of therapeutic writing
for one or more long term conditions. This work will put us into a position to make
detailed recommendations for future research and will also be orientated towards
directly influencing healthcare policy and practice.
Our findings will be communicated to the public and the medical profession using
established lines of communication, such as peer reviewed journals, the publications
of the NIHR and patient interest groups such as the Pelvic Pain Support Network.
We will also use our established connections with the lay media and work through
the Queen Mary University of London and HTA press offices to ensure that the
release of findings are accompanied by targeted press releases to all groups with a
potential interest in them.
We have developed and refined literature searching, data extraction techniques and
analytic strategies for the topic being commissioned. We welcome the opportunity to
bid for this call for proposals, which will allow us to consolidate and advance the
work we have already undertaken in this field.
6. Project timetable
The project is expected to run over an 18-month period. Figure 2 shows the project
timetable and milestones for the effectiveness and realist reviews and economic
evaluation.
6.1 Milestones
5 months: All relevant studies for inclusion into the reviews are identified
6 months: submission of progress report to HTA programme
8 months: all data extracted and quality appraised for systematic review
12 months: submission of progress report to HTA
12 months: both reviews and evidence synthesis completed
17 months: all analysis and economic model completed, report completed ready for
internal peer review.
15
Figure 2. Timetable
Months
1
2
3
4
5
Protocol development
X
Protocol peer review
X
Study selection
X
X
X
X
Effectiveness review
X
X
X
X
9
10
X
X
X
Evidence synthesis
X
X
X
X
Economic modelling
X
X
X
Realist Review
Report production
Internal peer review
6
7
8
X
X
X
X
X
X
11 12
13
14
X
X
X
X
X
X
15
16
17
X
X
X
X
18
X
16
References
1.
Pearson L. The Use of Written Communications in Psychotherapy. Springfield, IL:
Charles C Thomas; 1965.
2.
Jolly M. What I never wanted to tell you: therapeutic letter writing in cultural context. J
Med Humanit. 2011;32(1):47-59.
3.
Pennebaker JW, Beall SK. Confronting a traumatic event: toward an understanding
of inhibition and disease. J Abnorm Psychol. 1986;95(3):274-81.
4.
Nicholls S. Beyond expressive writing: Evolving models of developmental creative
writing. J Health Psychol. 2009;14(2):171-80.
5.
Baker JR, Moore SM. Blogging as a social tool: A psychosocial examination of the
effects of blogging. CyberPsychology & Behavior. 2008; 11(6):747-9.
6.
Smyth J, Nazarian D. Arigo D. . Expressive writing in the clinical context. In:
Vingerhoets A, Nyklicek I. Denollet, J. , eds. Emotion regulation Conceptual and clinical
issues. Tilberg: Springer; 2008.
7.
Wikipedia. Writing therapy. 2011 Sept 2011
http://en.wikipedia.org/wiki/Writing_therapy accessed Sept 2011
8.
Meads CA. Emotional disclosure (expressive writing) and health. Saarbrucken: VDMVerlag; 2009.
9.
Meads C, Nouwen A. Does emotional disclosure have any effects? A systematic
review of the literature with meta-analyses. Int J Technol Assess in Health Care.
2005;21(02):153-64.
10.
Meads C, Sheffield D. Issues Regarding Systematic Review Reports and Trial
Heterogeneity. J Nervous and Mental Dis. 2005; 193(6): 424-5.
11.
Greenhalgh T. Storytelling should be targeted where it is known to have greatest
added value. Med Educ. 2001;35(9):818-9.
12.
Greenhalgh T. What seems to be the trouble? Stories in illness and healthcare.
Oxford: Radcliffe; 2006.
13.
Greenhalgh T. Writing as therapy. Effects on immune mediated illness need
substantiation in independent studies. BMJ. 1999;319(7205):270-1.
14.
Greenhalgh T, Collard A, Campbell-Richards D, Vijayaraghavan S, Malik F, Morris J,
et al. Storylines of self-management: narratives of people with diabetes from a multiethnic
inner city population. J Health Serv Res Policy. 2011; 16(1): 37-43.
15.
Greenhalgh T, Hurwitz B. Narrative based medicine : dialogue and discourse in
clinical practice. London: BMJ Books; 1998.
16.
Smyth JM, Pennebaker JW. Exploring the boundary conditions of expressive writing:
In search of the right recipe. Br J Health Psychol. 2008;13(1):1-7.
17.
Reissman C. Narrative analysis. 2nd ed. London: Sage; 2009.
18.
Esterling BA, L'Abate L, Murray EJ, Pennebaker JW. Empirical foundations for
writing in prevention and psychotherapy: Mental and physical health outcomes. Clin Psychol
Rev. 1999; 19(1): 79-96.
19.
Larson DG, Chastain RL. Self-concealment: Conceptualization, measurement, and
health implications. J Soc Clin Psychol. 1990; 9(4): 439-55.
20.
Creswell JD, Lam S, Stanton AL, Taylor SE, Bower JE, Sherman DK. Does SelfAffirmation, Cognitive Processing, or Discovery of Meaning Explain Cancer-Related Health
Benefits of Expressive Writing? Personality and Social Psychol Bull. 2007;33(2):238-50.
21.
Petrie KJ, Booth RJ, Pennebaker JW, Davison KP, Thomas MG. Disclosure of
trauma and immune response to a hepatitis B vaccination program. J Consult and Clin
Psychol. 1995;63(5):787-92.
22.
Harvey AG, Farrell C. The Efficacy of a Pennebaker-Like Writing Intervention for
Poor Sleepers. Behavioral Sleep Med. 2003; 1(2): 115-24.
23.
Smyth JM. Written emotional expression: effect sizes, outcome types, and
moderating variables. J Consult & Clin Psychol. 1998;66(1):174-84.
24.
Ogden J. Health psychology: A textbook. Maidenhead, BRK, England: Open
University Press; 1996.
17
25.
Smyth JM, Stone AA, Hurewitz A, Kaell A. Effects of writing about stressful
experiences on symptom reduction in patients with asthma or rheumatoid arthritis: a
randomized trial. JAMA. 1999;281(14):1304-9.
26.
Lowe G. Health related effects of creative and expressive writing. Health Educ.
2006;106(1):60-70.
27.
Frisina PG, Borod JC, Lepore SJ. A meta-analysis of the effects of written emotional
disclosure on the health outcomes of clinical populations. J Nerv Ment Dis. 2004;192(9):62934.
28.
Frattaroli J. Experimental disclosure and its moderators: a meta-analysis. Psychol
Bull. 2006;132(6):823-65.
29.
Harris AH. Does expressive writing reduce health care utilization? A meta-analysis of
randomized trials. J Consult Clin Psychol. 2006;74(2):243-52.
30.
Greenhalgh T, Campbell-Richards D, Vijayaraghavan S, Collard A, Malik F, Griffin M,
et al. New models of self-management education for minority ethnic groups: pilot
randomized trial of a story-sharing intervention. J Health Serv Res Policy. 2011;16(1):28-36.
31.
Foster G, Taylor SJ, Eldridge SE, Ramsay J, Griffiths CJ. Self-management
education programmes by lay leaders for people with chronic conditions. Cochrane
Database Syst Rev. 2007;(4): CD005108.
32.
Griffiths C, Foster G, Ramsay J, Eldridge S, Taylor S. How effective are expert
patient (lay led) education programmes for chronic disease? BMJ. 2007; 334(7606):1254-6.
33.
Greenhalgh T. Patient and public involvement in chronic illness: beyond the expert
patient. BMJ. 2009;338:b49.
34.
Newman S, Steed L, Mulligan K. Self-management interventions for chronic illness.
Lancet. 2004;364(9444):1523-37.
35.
Clark C, Haines MM, Head J, Klineberg E, Arephin M, Viner R, et al. Psychological
symptoms and physical health and health behaviours in adolescents: a prospective 2-year
study in East London. Addiction. 2007; 102(1):126-35.
36.
Steed L, Cooke D, Newman S. A systematic review of psychosocial outcomes
following education, self-management and psychological interventions in diabetes mellitus.
Patient Educ Couns. 2003;51(1):5-15.
37.
Greenhalgh T, Wong G, Westhorp G, Pawson R. Protocol - realist and metanarrative evidence synthesis: Evolving Standards (RAMESES). BMC Med Res Methodol.
2011;11:115.
38.
Greenhalgh T, Wengraf T. Collecting stories: is it research? Is it good research?
Preliminary guidance based on a Delphi study. Med Educ. 2008;42(3):242-7.
39.
Greenhalgh T. Can 'anecdote' ever be research? Fam Pract. 2005;22(1):1.
40.
Greenhalgh T, Collard A, Begum N. Sharing stories: complex intervention for
diabetes education in minority ethnic groups who do not speak English. BMJ. 2005;
330(7492):628.
41.
Hurwitz B, Greenhalgh T, Skultans V. Narrative research in health and illness.
Malden, Mass. ; Oxford: BMJ Books; 2004.
42.
Uthman OY, I. Pennant, M. Bayliss, S. Aveyard, P. Jit, M. Barton, P. Meads, C.
Chen, Y-C. School based interventions to prevent the uptake of smoking among children
and young people: effectiveness review. 2009 29th
http://www.nice.org.uk/nicemedia/live/12827/47624/47624.pdf accessed Sept 2011
43.
Jit M, Aveyard P, Barton P, Meads CA. Predicting the life-time benefit of schoolbased smoking prevention programmes. Addiction. 2010;105(6):1109-16.
44.
Jit MS, H.R. Aldridge, R.W. White, P.J. Abubakar, I. Dedicated outreach service for
hard to reach patients with tuberculosis in London: observational study and economic
evaluation. BMJ. 2011;343:d5376.
45.
(UK). DoH. Ten things you need to know about long term conditions. 2011
http://www.dh.gov.uk/en/Healthcare/Longtermconditions/tenthingsyouneedtoknow/index.htm
accessed Sept 2011
18
46.
Heshka S, Allison DB. Is obesity a disease? Int J Obes Relat Metab Disord.
2001;25(10):1401-4.
47.
Egger M, Dickersin K, Davey-Smith G. Problems and limitations in conducting
systematic reviews. In: Egger M, Davey Smith G. Altman, D.G., editor. Systematic reviews in
healthcare, Meta-analysis in context. London: BMJ Books; 2001.
48.
Alderson P GS, Higgins JPT (eds). Cochrane Reviewers Handbook 4.2.2. 2005.
49.
Moher D, Schulz K, Altman D. The CONSORT statement: revised recommendations
for improving the quality of reports of parallel group randomized trials. Ann Intern Med.
2001;(134):657-62.
50.
Downs S, Black N. The feasibility of creating a checklist for the assessment of the
methodological quality both of randomised and non-randomised studies of health care
interventions. J Epidemiol Community Health. 1998;(52):377-84.
51.
Khan K, ter Riet G, Glanville J, Sowden A, Kleijnen J. Undertaking Systematic
Reviews of Research on Effectiveness. CRD's Guidance for Carrying Out or Commissioning
Reviews. 2nd ed. York: NHS Centre for Reviews and Dissemination, University of York;
2001.
52.
Wilson A, Lissauer D, Thangaratinam S, Khan KS, Macarthur C, Coomarasamy A. A
comparison of clinical officers with medical doctors on outcomes of caesarean section in the
developing world: meta-analysis of controlled studies. BMJ.342:d2600.
53.
Guyatt G, Oxman A, Vist G, Kunz R. GRADE: an emerging consensus on rating
quality of evidence and strength of recommendations. BMJ. 2008;336:924-6.
54.
Guyatt G, Oxman A, Vist G, Kunz R, Falck-Ytter Y, Schunemenn H. GRADE: What is
"quality of evidence" and why is it important to clinicians? BMJ 2008;336:995-8.
55.
Guyatt G, Oxman A, GE V, R K. Rating quality of evidence and strength of
recommendations GRADE: an emerging consensus on rating quality of evidence and
strength of recommendations BMJ 2008;336:924-6.
56.
Latthe PM, Foon R, Khan K. Nonsurgical treatment of stress urinary incontinence
(SUI): grading of evidence in systematic reviews. BJOG. 2008;115(4):435-44.
57.
Khan KS, Borowiack E, Roos C, Kowalska M, Zapalska A, Mol BW, et al. Making
GRADE accessible: a proposal for graphic display of evidence quality assessments. Evid
Based Med. 2011;16(3):65-9.
58.
Khan KS, Kunz R, Kleijnen J, Antes G. Systematic Reviews to Support EvidenceBased Medicine: How to Review and Apply Findings of Healthcare Research London Royal
Society of Medicine Press; 2005.
59.
Sterne JAC, Egger M. Regression methods to detect publication and other bias in
meta-analysis. Chichester: Wiley; 2005.
60.
Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of
systematic review designed for complex policy interventions. J Health Serv Res Policy.
2005;10 Suppl 1:21-34.
61.
Astbury B, Leeuw FL. Unpacking Black Boxes: Mechanisms and Theory Building in
Evaluation. American Journal of Evaluation. 2010;31(3):363-81.
Berwick DM. The Science of Improvement. JAMA. 2008;299(10):1182-84.
62.
63.
Noble D, Mathur R, Dent T, Meads C, Greenhalgh T. A statistical feast but an impact
famine: Systematic review of risk scores for type 2 diabetes. BMJ 2011;343:d7163
64.
Wong G, Greenhalgh T, Westhorp G, Pawson R. Realist methods in medical
education research: what are they and what can they contribute? Med Educ. 2012;46(1):8996.
65.
Greenhalgh T, Wong G, Westhorp G et al. (2011) . Protocol--realist and metanarrative evidence synthesis: evolving standards (RAMESES). BMC Med Res Methodol vol.
11, 10.1186/1471-2288-11-115
66.
Greenhalgh T, Kristjansson E, Robinson V. Realist review to understand the efficacy
of school feeding programmes. BMJ. 2007;335(7625):858-61.
19
67.
Kristjansson EA, Robinson V, Petticrew M, MacDonald B, Krasevec J, Janzen L, et
al. School feeding for improving the physical and psychosocial health of disadvantaged
elementary school children. Cochrane Database Syst Rev. 2007;(1):CD004676.
68.
Dixon-Woods M et al. Synthesising qualitative and quantitative evidence: A review of
possible methods. JHSRP 2005; 10: 45-53.
69.
Popay J et al. Guidance on the Conduct of Narrative Synthesis in Systematic
Reviews. ESRC 2006. Available in full text
on http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.178.3100&rep=rep1&type=pdf
70.
Ali S, Stone MA, Peters JL, Davies MJ, Khunti K. The prevalence of co-morbid
depression in adults with Type 2 diabetes: a systematic review and meta-analysis. Diabet
Med. 2006; 23(11):1165-73.
71.
Best Research for Best Health: A new national health research strategy
The NHS contribution to health research in England. Department of Health London 2006.
20