Health information materials in GP practices in Stoke-on-Trent:
A multi-stage readability assessment
Emee Vida Estacio, Jo Protheroe and Sirandou Saidy-Khan
BACKGROUND
This project forms part of a wider set of activities on health literacy in Stoke-on-Trent. It aims to
assess the readability of current health information materials available in GP practices. Materials in
GP practices were prioritised since patients usually refer to GP practices as their first point of contact
when it comes to healthcare. A multi-method approach was used in this study and was divided into
three stages.
Stage 1: Flesch-Kincaid Readability Test
Method (Stage 1). GP practices across Stoke-on-Trent were randomly selected, with a total of 17
practices visited by the research team. A copy of each information material in the waiting rooms
that could be taken by the public was collected. After sorting and removing duplicates, a total of
345 unique leaflets were collated. To create structure, the materials were then organised according
to the following categories proposed by Rudd (2007): 1) Health promotion; 2) Health protection; 3)
Disease prevention; 4) Healthcare and management; and 5) Systems navigation. Sections from the
leaflets were transcribed verbatim using MS Word 2007 to calculate readability scores using the
Flesch Reading Ease and Flesch-Kincaid Grade Level.
Flesch Reading Ease
This test rates text on a 100-point scale.
The higher the score, the easier it is to
understand the document.
Technical note
Flesch Reading Ease
206.835 – (1.015 x ASL) – (84.6 x ASW)
Flesch-Kincaid Grade Level
(.39 x ASL) + (11.8 x ASW) – 15.59
Flesch-Kincaid Grade Level
This test rates text on a U.S. school
grade level. For example, a score of
8.0 means someone from 8th grade
can understand the document.
ASL = average sentence length
(the number of words divided by the number of
sentences)
ASW = average number of syllables per word
(the number of syllables divided by the number of
words)
The lower the score, the easier it is
to understand the document.
ASL = average sentence length
(the number of words divided by the
number of sentences)
ASW = average number of syllables per
word
(the number of syllables divided by the
number of words)
1
The Flesch-Kincaid scores
suggest that on average,
health information
materials in GP practices in
Stoke-on-Trent have
readability levels suitable
for 13- to 15-year-olds.
Results (Stage 1). Flesch reading ease scores of the materials
collected ranged from 6.3 to 95.4, with an overall mean of
61.09 (SD=12.07). Flesch-Kincaid grade levels ranged from 1.30
to 14.20, with an overall mean of 8.24 (SD=2.09).
Table 1 provides a statistical summary of the readability
measures per category. Tables 2 and 3 provide a description
of what these scores mean in real terms.
Table 1. Descriptive statistics on readability measures
Sentences
per
paragraph
Category
Words per
sentence
Characters
per word
Passive
sentences
Flesch
Reading Ease
FleschKincaid
Grade Level
Health promotion
Mean
1.8000
17.0500
4.4222
.0517
69.9556
7.3000
(N=18)
SD
.62872
4.52850
.40082
.07414
10.66756
1.35256
Health protection
Mean
1.8184
13.2755
4.6184
.0982
63.0204
7.7061
(N=49)
SD
1.82148
4.22066
.36210
.11267
12.99853
2.19890
Disease prevention
Mean
1.6354
14.4810
4.5949
.1373
61.1342
8.3304
(N=79)
SD
.43942
2.92790
.28052
.11627
10.80384
1.91869
Healthcare and
Mean
1.5529
15.8529
4.6235
.1088
60.6029
8.6706
management (N=34)
SD
.50945
2.83936
.33670
.10002
10.70311
1.82550
Systems navigation
Mean
1.4642
14.0503
4.7012
.0879
59.6376
8.3721
(N=165)
SD
.49571
4.51528
.50750
.11878
12.39623
2.20738
TOTAL
Mean
1.5800
14.3730
4.6429
.1009
61.0942
8.2414
(N=345)
SD
.82834
4.09288
.42729
.11539
12.06625
2.08754
Table 2. Flesch reading ease values
Score
90.0–100.0
60.0–70.0
0.0–30.0
Notes
easily understood by an average 11-year-old
easily understood by 13- to 15-year-old
best understood by university graduates
Table 3. Flesch-Kincaid Grade Level
About 1 in 5 of the materials
collected will require a minimum
readability level of a 16 year old
school leaver.
There are leaflets with a FleschKincaid Grade level as high as
14.2 -- this is equivalent to a
readability level of a second year
University student.
Grade level
Age
Grade level
Age
Preschool
3-4
6th Grade
11-12
Pre-kindergarten
4-5
7th Grade
12-13
Kindergarten
5-6
8th Grade
13-14
1st Grade
6-7
9th Grade
14-15
2nd Grade
7-8
10th Grade
15-16
3rd Grade
8-9
11th Grade
16-17
4th Grade
9-10
12th Grade
17-18
5th Grade
10-11
2
Stage 2: Jargon search, lay-out and reader-friendliness
Method (Stage 2). Considering the vast amount of materials collected, the team decided to focus on
sexual health materials on this occasion. Six members of a patient and public involvement group
were recruited to help the team to identify jargon from a sample of the materials collected.
Results (Stage 2). Examples of potentially difficult to understand words are listed in Box 1.
Box 1. Potentially difficult to understand words found in sexual health information materials
400 mcg folic acid
Bacterial vaginosis
Clotting factors
Complementary
therapies
Cryotherapy
Cytology
Genital warts
Genitourinary
Hormonal
Hysterectomy
Long-acting reversible
Lubrications
Microplasma genitalium
Notoriously
Oncologists/therapists
Parabens/ paraben-free
Pelvic inflammatory
disease
Penetrative
Participants also provided suggestions on how technical jargon
can be simplified. Examples are listed in Box 2.
Comments concerning general lay-out and reader-friendliness
were also sought. Comments concerning examples of good
practice are listed below:
Simple, few facts presented in short sentences
Long words avoided
Colloquial words in brackets, e.g. to pass urine (peeing)
Modern, attractive illustrations
New or unfamiliar words explained
Different fonts and colour used
Clear contact numbers on a list
Explanation included concerning confidentiality to
encourage participation
Progesterone hormone
Royal College of
Obstetricians and
Gynecologists
Sonographer
Toxicological
Tubal occlusion
Urea plasma
Vaginal insertions
Box 2. Suggestions to simplify jargon
‘Who experience pregnancy’
(pregnant)
Any time in the menstrual cycle
(any time of the month)
Cytotoxicity (poison/cancerous)
Discomfort (pain)
Gynecological/gynaecologist
(women’s doctor)
Holistic person-centred model of
support (support and counselling)
Intrauterine system or device (coil)
Mimics (like)
Mucous membrane (lining)
Odourless (without smell)
Vaginal environment (inside the
vagina)
In contrast, some materials have limitations, as follows:
Text and booklet too small for readers
Too much detail and too much information – trying to fit everything in a small booklet
Some leaflets have too many contact details, without explaining the purpose of the services
Too many unfamiliar terms which can be intimidating, off-putting and frightening
Use of long sentences
Use of background and text colour that makes texts difficult to read
Use of unattractive images (e.g. swirly ink blot in some sexual health materials)
Booklets full of text without pictures
Use of scientific diagrams without lay explanations
3
Below are some recommendations to improve the reader-friendliness of sexual health information
materials:
Abbreviations such as pH, CASH clinic, GSOH, FPA, GUM clinic will need explanation
Some people may only have time to scan through materials. Thus, information should be
easy to find without having to read the entire booklet
Offer information in bite-size chunks
Services can be listed instead of presenting in a paragraph
It may be useful to have an index or glossary in some booklets to help with unfamiliar
words
Materials for adults could be written similarly as materials designed for teenagers (i.e.
short sentences, easy words)
There is a fine line between making the materials too simple and patronising patients –
health information designers need to be careful with this
There is a need to consider ethnic/cultural appropriateness of materials
Materials should be quality/readability assessed by lay readers before printing and
distributing to the public
Some patients may feel intimidated to pick-up information materials in the first place.
There is a need to think of ways on how to make patients feel comfortable about taking
information (e.g. consider where leaflets are located)
Stage 3: Can readers understand sexual health information materials?
Method (Stage 3). This stage aims to assess whether readers could correctly answer knowledgebased questions using information from sexual health information materials. Considering the target
group of the materials, 20 young people aged 16-22 years old (M=19.05, SD=1.605) were recruited; 6
males, 14 females; 2 GCSE students, 3 A-level students, 15 university undergraduate students. Four
leaflets were selected based on the calculated Flesch readability levels from Stage 1 (i.e. low,
medium, high).
Table 4. Sexual health materials used and readability levels
Material
FRE
FKG
Love S.T.I.ngs
74.1
4.8
The C-Card
64.5
7.7
Genital herpes (HSV)
66.3
7.8
Gonorrhoea
42.8
11
* FRE = Flesch Reading Ease * FKG = Flesch Kincaid Grade Level
4
Results (Stage 3). Participants were given 10-15 minutes to browse through the materials. When
ready, they were asked 20 knowledge-based questions on sexual health and contraception. They
were advised that the answers must be based on the information from the leaflet and NOT from
their own knowledge. They were allowed to read the leaflets to look for the answers to the
questions. They were also allowed to say ‘PASS’ if they can’t find the answers there. They were
given 10-15 minutes to answer all 20 questions (see Box 3).
Box 3. Questions asked from stage 3 participants
LOVE S.T.I.NGS: A beginner’s guide to sexually transmitted infections
1.
2.
3.
4.
5.
People can only catch sexually transmitted infections (S.T.I.s) if they have multiple sexual partners.
True or False? ANSWER: False
How long does it usually take for S.T.I. symptoms to show? ANSWER: Within 2-14 days, but can
take up to 4 weeks or longer
People can catch crabs or scabies just by skin contact. True or False? ANSWER: True
Most S.T.I.s can be treated during pregnancy without causing harm to the mother or baby. True
or False? ANSWER: True
The HPV vaccine can protect girls from getting pregnant. True or False? ANSWER: False
Genital herpes (HSV)
1.
2.
3.
4.
5.
The herpes virus that infects the mouth or the nose is classified as Type 1 or Type 2? ANSWER:
Type 1
Herpes can be passed on by kissing. True or False? ANSWER: True
A sample of urine (pee) will be taken to test if someone has herpes. True or False? ANSWER: True
How long does it usually take to get herpes test results from the lab? ANSWER: within 2 weeks
Herpes can make women infertile (not able to have babies). True or False? ANSWER: False
The C Card
1.
2.
3.
4.
5.
The C-Card allows anyone under 24 years old to get free condoms. True or False? ANSWER: True
Where can people register to get the C-Card? ANSWER: any CASH young person’s clinic or at
specialists sites around Stoke
How long does it take to register? ANSWER: about 10 minutes
The C-Card will have people’s name on it. True or False? ANSWER: False
Where can people get free condoms using the C-Card? ANSWER: Any place where you see the CCard logo
Gonorrhoea: Looking after your sexual health
1.
2.
People can get gonorrhoea through unprotected oral sex. True or False? ANSWER: True
People will be immune from gonorrhoea (can’t have it again) if they had it before. True or False?
ANSWER: False
3. How much do people have to pay to get tested for gonorrhoea? ANSWER: £0
4. The gonorrhoea test can tell how long someone had the infection. True or False? ANSWER: False
5. Pregnant or breastfeeding women can take antibiotics to treat gonorrhoea. True or False?
ANSWER: True
5
Raw data were coded and analysed using SPSS v.19. Results from this exercise indicate that young
people in general are able to understand information from sexual health materials. However, levels
of understanding may vary. As shown in Table 5 below, the number of correct answers for each
material ranged from 1 to 5. This means that while some participants may be able to answer all
questions correctly, there were others who struggled.
Table 5. Correct answers from participants (N=20)
Correct answers
Leaflet
Minimum Maximum
Mean
Standard
Deviation
Love S.T.I.NGS
1
5
3.95
1.317
Herpes
3
5
4.45
.759
The C Card
3
5
4.65
.671
Gonorrhoea
2
5
4.15
1.089
A Pearson r test was carried out to assess whether the level of education and the number of correct
responses were correlated. Although only one material showed a statistically significant positive
correlation between these variables [Herpes leaflet, r(18)=.532, p=0.16], the cross tabulations
showed early indications that those with less formal education may misinterpret information from
the materials (see Tables 6-9). More data will need to be collected to balance the disproportionately
higher number of University students in this sample to accurately test correlations with education.
Table 6. Education x Love S.T.I.NGS Cross-tabulation (N=20)
LOVE S.T.I.NGS CORRECT ANSWERS
Education
1
2
3
4
5
Total
GCSE
0
1
0
1
0
2
A level
0
0
1
0
2
3
Undergraduate
1
2
1
3
8
15
1
3
2
4
10
20
Total
Table 7. Education x Herpes Cross-tabulation (N=20)
HERPES CORRECT ANSWERS
Education
3
4
5
Total
GCSE
1
1
0
2
A level
1
1
1
3
Undergraduate
1
3
11
15
3
5
12
20
Total
Table 8. Education x The C Card Cross-tabulation (N=20)
C CARD CORRECT ANSWERS
Education
Total
3
4
5
Total
GCSE
0
0
2
2
A level
0
0
3
3
Undergraduate
2
3
10
15
2
3
15
20
6
Table 9. Education x Gonorrhoea Cross-tabulation (N=20)
GONORRHOEA CORRECT ANSWERS
Education
2
3
4
5
Total
GCSE
0
1
1
0
2
A level
0
0
0
3
3
Undergraduate
2
3
2
8
15
2
4
3
11
20
Total
An analysis of variance was also carried out to test whether there are statistical differences between
the number of correct answers between the materials. The ANOVA showed statistically significant
differences between the scores from the selected leaflets, F (3) = 2.27, p=0.49. It was assumed that
those with better Flesch-Kincaid readability scores will generate more correct answers. However,
the results suggest that although having a poor readability score may yield less number of correct
answers from participants; having a good Flesch readability score
Having a good readability
will not necessarily guarantee that readers will be able to find or
score does not necessarily
understand the information they need. As shown in Figure 1 below,
mean readers will be able
the leaflet with the best Flesch-Kincaid score (i.e. Love S.T.I.NGS)
to find or understand the
generated the least number of correct responses from the
information they need.
participants. Feedback from participants suggests that although
the material itself was easy to read, its lay-out and design made it
difficult for readers to find the information they needed.
Figure 1. Average correct answers from participants (N=20)
4.8
4.6
4.4
4.2
4
3.8
3.6
Love S.T.I.NGS
FRE=74.1
FKG=4.8
Herpes
FRE=64.5
FKG=7.7
The C Card
FRE=66.3
FKG=7.8
Gonorrhoea
FRE=42.8
FKG=11
7
SUMMARY
This project aimed to assess the readability of current health information materials in GP practices in
Stoke-on-Trent. Using a multi-stage process, findings suggest that on average, the materials
collected have readability levels suitable for 13- to 15-year olds. While some materials may have
good readability scores, this may not necessarily mean that readers will be able to find or
understand the information provided. The use of jargon, lengthy sentences and poor lay-out are
some factors that make the materials more difficult to read. It was suggested that health
information materials need to be quality and readability assessed by lay persons before printing and
distributing to the public. It would also be useful to map the findings of this exercise with the results
of the Stoke-on-Trent health literacy baseline assessment to gauge the appropriateness of
information materials with the capabilities of its target audience.
8