Family Practice
© Oxford University Press 1999
Vol. 16, No. 1
Printed in Great Britain
Critical appraisal using the READER method:
a workshop-based controlled trial
Domhnall MacAuley and Evelyn McCrum
MacAuley D and McCrum E. Critical appraisal using the READER method: a workshop-based
controlled trial. Family Practice 1999; 16: 90–93.
Background. Critical reading is an important skill for those trying to practice evidence-based
medicine. There are a number of recognized structures for critical reading, including the READER
model. These methods should be subjected to high-quality studies.
Objectives. We aimed to evaluate the READER method in a practical teaching setting using the
highest quality research methodology.
Methods. We carried out a modified randomized controlled trial. Two groups of GP trainers
were invited to appraise critically the two articles using either the READER acronym or a semistructured free appraisal.
Results. Of 99 participants in the workshop, 92 completed the study. One-third of participants
(33.7%) read more than five articles per month and those who had been in practice the longest read
fewer articles (P < 0.05). Both groups attributed the lowest score to paper 2. The median total
scores were higher using the READER method, although only significant for paper 2 (P < 0.05).
The median score attributed to the methodology was lower using the READER method than the
free appraisal, although this difference was only significant for paper 1 (P < 0.05). Overall, 51%
(70% of the READER group) believed that taking part in the exercise would encourage them to
be more critical of published articles in the future.
Conclusion. Those using the READER method attributed a higher total score, but were more
critical of the methodology than those using a free appraisal. Participants found the study useful
and believed that it would be of help in future critical appraisal. The study raises interesting
questions about the relative importance to GPs of methodological rigour compared with other
factors when evaluating research papers.
Keywords. Controlled trial, critical reading, family practice.
The READER method,12 first described in 1994, has
been used by doctors in a number of different environments.13–15 A recent randomized controlled trial demonstrated how this model may be used by GPs reading
research articles in their own time.16 The aim of this study
was to evaluate the READER method in a practical
teaching setting using the highest quality research
methodology. The objectives were to compare the scores
using both methods, to examine factors which influence
the scores achieved using these methods, and to enter
into dialogue with participants in order to improve the
method. The null hypothesis was that the READER
method was no better than a free appraisal. This paper
compares assessments using the READER method
with a free appraisal in a controlled trial. The particular
strengths are in the methodology, which is based on the
randomized controlled trial, and in the large number of
GP participants.
Introduction
Critical reading is recognized as an important skill for
general practice, endorsed by academic bodies in the
United Kingdom,1 Canada2,3 and Australia.4 Many doctors cite reading as their main method of keeping upto-date,5 so it is essential that GPs be able to read and
appraise research findings appropriately in order to
uphold the principles of evidence-based practice. There
are a number of published texts6–8 and guides9–11 available, but these are rather complex, and there is a need
for a simple, easy to apply guide for GPs.
Received 19 May 1998; Accepted 7 October 1998.
Division of Epidemiology, The Queen’s University of Belfast,
Mulhouse Building, Royal Victoria Hospital, Belfast BT12 6BJ,
UK.
90
Critical appraisal using the READER method
Method
The study population consisted of those GPs attending
the Devon and Cornwall Annual Residential Trainers’
Workshop who attended the critical reading component.
Four groups were chosen for the study using a two-stage
random selection. The first stage was selection by
seniority. Participants were then randomly distributed
into four groups so that each group contained a similar
number of those who had been trainers for 5 years or
more. Of these four groups (A, B, C and D), two groups
(A and B) acted as intervention groups and two groups
(C and D) as controls. Groups A and B, the intervention
groups, attended a 30 minute teaching session on the
READER method of critical appraisal and then performed a critical appraisal of one of two papers using a
free appraisal. Validity studies have shown the READER
method to be both accurate and repeatable.16 The control groups, C and D, each appraised one of the same two
papers using a free appraisal. They had had no prior training or guidance on critical reading and their instructions
were simply to rate the papers under two categories:
(1) the scientific quality (on a scale of one to 10) and (2)
their overall importance to everyday practice (on a scale
of one to 25). We also requested some personal and practice details from all participants which we anticipated
could possibly be confounding factors. Participants had
no prior knowledge of the study before taking part.
We chose the papers (for details, see footnote to Table 3)
used in this study from a previous appraisal of the
December 1995 edition of the British Journal of General
Practice (unpublished). We undertook a pilot study using
the proposed methodology among 10 GP registrars
attending a half-day release course (unpublished).
The chi-square test was used to compare the personal
characteristics between groups, and differences in the
scores between the two methods were examined using
the Mann–Whitney U test.
Results
Of 104 participants in the residential workshop,
99 attended the critical reading component, of whom
92 completed the study. There were 79 males and 13
females (Table 1). We compared the groups in order to
identify possible confounding factors, but there were no
significant differences. The four groups were similar in
TABLE 1
Males
Females
Participants in groups using the READER method
(A and B) and the free assessment
Group A
Group B
Group C
Group D
Total
23
19
21
16
79
2
4
1
6
13
91
respect of length of time in practice and practice type,
and, although one of the four groups reported that they
read more articles from academic journals per month
than the others, there was no significant difference overall between the intervention (READER) and control
groups. One-third of participants (33.7%) reported that
they read more than five articles per month (Table 2),
but there was no relationship between the number of
articles read per month and either gender or practice type,
although those who had been in practice the longest
read fewer articles (P < 0.05). The groups ranked the two
articles in the same order, based on all components of
the two scoring methods (Tables 3 and 4). Both groups
attributed the lowest scores to paper 2.
Both papers attracted a higher median total score
using the READER method than using the free
appraisal, which was statistically significant (P < 0.05)
for paper 2 but not for paper 1.
Comparing the Discrimination component of the
READER model with the Scientific quality in the free
appraisal, both papers attracted a lower score using
the READER method, although this difference was
only significant for paper 1 (P < 0.05).
Most participants (75%) stated that they enjoyed
taking part. In the READER group, 70% believed that
they would be more critical in their future reading, compared with 31% of the free appraisal group (P < 0.001) .
Discussion
There are a number of published methods of critical
reading, but, although widely accepted as valid, few have
been subjected to objective evaluation. This study used
a high-quality methodology to assess the READER
method in a practical setting. It included a large number
of participants with a high response rate (93%). All
participants were GP trainers, but it is possible that this
group of trainers may not be representative of all trainers
or indeed all GPs.
There was a marked difference in the scores attributed
to the two papers, and paper 1 scored higher with both
groups than did paper 2. Although the median total score
was higher for both papers using the READER method,
the difference was not statistically significant for paper 1,
but was significant for paper 2 (P < 0.05). Similarly,
although both papers attracted a lower median score
for their methodology using the READER method,
the difference was significant only for paper 1. Thus, the
READER method attributed a higher total, but enabled
participants to be more critical of the methodology than
a free appraisal. The null hypothesis was void.
These findings do, however, raise an interesting
point about the READER method and the importance
that GPs attribute to methodological rigour. While the
READER method discriminated between the methodologies, the overall score was not discriminatory, and
92
Family Practice—an international journal
TABLE 2
‘How many articles from academic journals
(e.g. BMJ, Brit J Gen Pract, etc.)
would you usually read fully each month?’
Fewer than 5 articles, n (%)
The number of articles read each month
Group A
Group B
Group C
Group D
21 (84)
15 (65)
15 (68)
10 (46)
61 (66)
7 (32)
12 (55)
31 (34)
22
92
More than 5 articles, n (%)
4 (16)
Total
8 (35)
25
TABLE 3
23
22
Total
Distribution of the scores using the READER method
Intervention group
Paper/Author
Paper 1a/Group A
Paper
2b/Group
B
Relevance
Education
Applicable
Discriminate
Total
5
3
4
4
16
3
1
2
2
6
Scores shown are the median.
Paper 1: Brown A, Armstrong D. Telephone consultations in general practice: an additional or alternative service? Brit J Gen Pract 1995;
45: 673–676.
b Paper 2: Andres E, Temme M, Raderschatt B, Szecsenyi J, Sandholzer H, Kochen M. COOP-WONCA charts: a suitable functional status
screening instrument in low back pain? Brit J Gen Pract 1995; 45: 661–664.
a
TABLE 4
Distribution of the scores using the free appraisal
Control group
Paper/Author
Scientific quality
Total
Paper 1/Group C
5
15
Paper 2/Group D
2.5
5
Scores shown are the median; for details of papers 1 and 2, see
footnote to Table 3.
indeed the overall score was higher for the paper that
scored lowest on methodology. This raises the issue
of the relative importance in general practice of the
methodological score versus overall score. For GPs it
may be that factors other than the methodology have
greater significance. Methodological purists would,
however, disagree, as they emphasize the importance of
the hierarchy of study design. This highlights one of the
controversies in critical reading of the literature. Clearly,
GPs consider other factors to be as important, if not
more so, than methodological rigour.
Paper 1, exploring the use of telephone consultations,
was considered to be highly relevant. Neither paper scored
highly on Education, which according to the READER
model, means that they would not encourage participants
to change what they do. The papers did not score highly
under the heading ‘Discrimination’, but as neither of the
papers used high quality methodology, this limits their
potential score using the READER scoring grid.
Unlike most other studies, the participants had an
opportunity to discuss the papers and their appraisal in a
plenary session following the exercise. This revealed an
interesting source of pre-test bias at the lower end of the
range of scores. For papers that score exceedingly low,
the READER method cannot score less than a total of 4.
Those using the free appraisal were able to allocate 4 or
less. This limits the mathematical comparison of papers
that score very low, but is purely a theoretical problem,
as papers of this quality are unlikely to make any practical
impact.
The original paper on the READER method suggests
four ways of reacting to a paper so that a score of 24+
indicates that it is a classic paper that should make an
immediate impact on practice, a score of 20–23 indicates
that it is a paper of value which may be retained for
access, a score of 15–19 means that while not fulfilling
all criteria the paper may be of interest, while a paper
scoring less than 15 should be ignored. In the discussion,
participants felt that there should be only two categories
and that one should take note of a paper only if it scores
in excess of 20 overall.
In conclusion, the objectives of the study were achieved
and the null hypothesis was void. Both the READER
method and the free appraisal discriminated between
Critical appraisal using the READER method
the papers, but there were differences in the scores attributed using the two different methods. Those using the
READER method attributed a higher total score but
were more critical of the methodology than those using
a free appraisal. This may reflect the difference weight
attributed by GPs when assessing a paper to factors other
than methodology. Participants found the study useful
and believed that it would be of help in future critical
appraisal.
5
6
7
8
9
10
Acknowledgement
11
This research was only possible with the considerable help
and enthusiasm of the Devon and Cornwall trainers.
12
13
References
1
2
3
4
RCGP Critical reading question (CRQ). Occasional paper. R Coll
Gen Pract 1990; 46: 25–26.
Herbert CP. Critical thinking and navel gazing. Can Fam Physician
1988; 34: 777.
Perkin RL. Organised curiosity. Can Fam Physician 1988; 34: 779.
Holmwood C. Critical thinking within the RACGP Training program and the college examination. Austr Fam Physician 1995;
24: 1673–1675, 1678–1679.
14
15
16
93
Smith R. What clinical information to doctors need? Br Med J 1996;
313: 1062–1068.
Sackett DL, Haynes RB, Tugwell P. Clinical Epidemiology, A Basic
Science for Clinical Medicine. Boston: Little Brown, 1985.
Jones R, Kinmonth A-L. Critical Reading for Primary Care.
Oxford General Practice Series. Oxford: Oxford University
Press, 1995.
Crombie IK. The Pocket Guide to Critical Appraisal. London: BMJ
Publishing Group, 1996.
Haynes RB, McKibbon KA, Fitzgerald D, Guyatt GH, Walker CJ,
Sackett DS. How to keep up with the medical literature: 1. Why
try to keep up and how to get started. Ann Int Med 1986; 105:
149–153.
Fowkes FG, Fulton PM. Critical appraisal of published research:
introductory guidelines. Br Med J 1991; 302: 1136–1140.
Haynes RB, Sackett DL, Tugwell P. Problems in the handling of
clinical and research evidence by medical practitioners. Arch Int
Med 1983; 143: 1971–1975.
MacAuley D. READER: An acronym to aid critical reading in
general practice. Br Gen Pract 1994; 44: 83–85.
MacAuley D, Sweeney KS. A new model for critical reading. Forum
(Journal of the Irish College of General Practitioners) 1994; 10:
52–54.
MacAuley D. Critical reading using the READER acronym at an
international workshop. Fam Pract 1996; 13: 104–105.
MacAuley D, Sweeney K. Critical reading using the READER
acronym by general practitioner registrars in the Republic of
Ireland, and Northern Ireland, and experienced general
practitioners. Irish J Med Sci 1997; 166: 21–23.
MacAuley D, McCrum E, Brown C. ‘Randomised controlled trial of
the READER method of critical appraisal in general practice.’
Br Med J 1998; 316: 1134–1137.