Family Practice
© Oxford University Press 1999
Vol. 16, No. 5
Printed in Great Britain
Use of multiple methods to determine factors
affecting quality of care of patients with diabetes
Kamlesh Khunti
Khunti K. Use of multiple methods to determine factors affecting quality of care of patients with
diabetes. Family Practice 1999; 16: 489–494.
Background. The process of care of patients with diabetes is complex; however, GPs are playing a greater role in its management. Despite the research evidence, the quality of care of patients
with diabetes is variable. In order to improve care, information is required on the obstacles faced
by practices in improving care. Qualitative and quantitative methods can be used for formation
of hypotheses and the development of survey procedures. However, to date few examples exist
in general practice research on the use of multiple methods using both quantitative and qualitative techniques for hypothesis generation.
Objectives. We aimed to determine information on all factors that may be associated with
delivery of care to patients with diabetes.
Methods. Factors for consideration on delivery of diabetes care were generated by multiple
qualitative methods including brainstorming with health professionals and patients, a focus
group and interviews with key informants which included GPs and practice nurses. Audit data
showing variations in care of patients with diabetes were used to stimulate the brainstorming
session. A systematic literature search focusing on quality of care of patients with diabetes in
primary care was also conducted.
Results. Fifty-four potential factors were identified by multiple methods. Twenty (37.0%) were
practice-related factors, 14 (25.9%) were patient-related factors and 20 (37.0%) were organizational factors. A combination of brainstorming and the literature review identified 51 (94.4%)
factors. Patients did not identify factors in addition to those identified by other methods.
Conclusion. The complexity of delivery of care to patients with diabetes is reflected in the
large number of potential factors identified in this study. This study shows the feasibility of
using multiple methods for hypothesis generation. Each evaluation method provided unique
data which could not otherwise be easily obtained. This study highlights a way of combining
various traditional methods in an attempt to overcome the deficiencies and bias that may occur
when using a single method. Similar methods can also be used to generate hypotheses for other
exploratory research. An important responsibility of health authorities and primary care groups
will be to assess the health needs of their local populations. Multiple methods could also be
used to identify and commission services to meet these needs.
Keywords. Diabetes mellitus, methodology, qualitative, quantitative.
management. A recent systematic review concluded that
computerized central recall, with prompting by GPs, can
achieve good standards of care.1 Unstructured care of
patients with diabetes in the community is associated
with poorer follow-up, greater mortality and worse
glycaemic control in comparison with hospital care.1
Despite the research evidence, the level of performance
in primary care can be variable.2,3
In order to improve care, information is needed about
the characteristics of practices that offer different levels
of care, and the obstacles faced by practices in improving
care. Knowledge of factors that influence quality of care
Introduction
Although the process of care of patients with diabetes
is complex, GPs are playing a greater role in its
Received 24 November 1998; Revised 26 April 1999; Accepted
13 May 1999.
Clinical Governance Research and Development Unit,
Department of General Practice and Primary Health Care,
University of Leicester, Leicester General Hospital, Gwendolen
Road, Leicester LE5 4PW, UK.
489
490
Family Practice—an international journal
in diabetes may therefore be helpful in implementing
quality improvement programmes. Qualitative and
quantitative methods can be used for formation of hypotheses and the development of survey procedures. To
date, few examples exist in general practice research on
how these methods can be used for hypothesis generation. In order to gather information on all factors that
may be related to good quality care of patients with
diabetes, information from as many sources as possible is
required to improve the knowledge on the subject. The
opportunity of investigating these methodological issues
arose as part of a study investigating factors associated
with quality of care of patients with diabetes in primary
care. Hypotheses had to be formulated to study the
potential factors that may be associated with good
quality of care. The aim of this paper is to describe how
multiple methods were used for hypothesis generation
of potential factors associated with delivery of care to
patients with diabetes. The effectiveness of individual
methods in generating hypotheses is also discussed.
Method
The proposed study is being conducted in primary
care comprising over 300 general practices with over
1000 GPs from three different Health Authorities in
England.
Hypothesis generation
A great deal of attention has recently been given to the
potential value of qualitative methods of research in
health care.4 A qualitative approach was therefore
determined to be necessary for generating hypotheses
on factors associated with good quality care of patients
with diabetes. Factors for consideration in management
of diabetes were generated by multiple methods:
literature review, followed by brainstorming, then focus
group and then key informant technique with GPs and
practice nurses. National primary care audit data on the
process of diabetes care were presented to participants
in the brainstorming session. The qualitative sessions of
brainstorming, focus group and key informant techniques were used only for identifying potential factors
that may be related to delivery of quality diabetes care.
As these methods were used only to generate a list of
factors relating to quality of care, these sessions were not
transcribed for detailed thematic analysis. All qualitative
sessions were facilitated by the author of this article, and
data were recorded on standard recording sheets which
were entered into a spreadsheet.
Literature review
A systematic search focused on quality of care of
patients in primary care with diabetes was conducted.
The search was conducted using Medline, including
years 1987–1997 and confined to studies involving
humans and published in English. In view of the large
number of articles, and the time and resources available,
Medline searches were conducted only on medical
subject headings and free text using the following terms:
diabetes, quality of care, process and outcome of care.
The references were sifted on the basis of article titles
only. The retrieved papers were assessed to determine
previous research evidence of factors associated with
good quality care of diabetes. Cross-referencing from
articles identified further studies for consideration.
Factors that may influence the quality of care of patients
with diabetes were identified from the retrieved
articles.1,5–23 The literature review identified many
articles that reported similar factors relating to good
diabetic care. Only key articles have been referenced for
this study. The literature searching, retrieval and assessment of the papers required approximately 16 hours
of work.
Brainstorming
Brainstorming sessions, if conducted properly, may offer
many advantages and are a quick means of focusing
attention on possible solutions for problems.24 Because
of their interactions with each other, several people in a
group setting, by building upon each other’s contributions, will be able to generate more ideas about a subject
than could the individuals by themselves.24 The individuals in the group must have a comfortable level of
trust with each other and sufficient time for the session
must be allowed. The brainstorming session was conducted in a University Department of General Practice
and lasted 30 minutes. There were 15 participants, which
included academic GPs and research staff. One member
of the research staff had diabetes. Prior to the brainstorming, participants were presented with audit data
relating to the care of diabetes in 17 Health Authorities
in England and Wales.3 The audit had shown deficiencies
in care and wide variations in the process of care of
people with diabetes.3
Focus groups
Focus groups enhance the validity of existing questionnaire by highlighting those concerns held by users and
providers that would otherwise be neglected.25 Although
brainstorming is a rapid and convenient way of collecting information from several sources simultaneously,
focus groups explicitly use group interaction as part of
the method.26 Focus groups can permit formulation of
new hypotheses and inform further study.27 This method
allowed more detailed information to be obtained
including themes identified by brainstorming. One focus
group was held and included two diabetic consultants, an
epidemiologist with an interest in diabetes, a practice
nurse with an interest in diabetes and a public health
consultant. This session lasted 25 minutes. Specific factors that may contribute to good and for quality of care
were identified.
491
Multiple methods for assessing quality of care
The key informant technique
The key informant technique is a qualitative research
method where an expert source of information is asked
to provide deeper insight into what is going on around
them.28 The aim of this part was to interview key people
currently delivering care to patients with diabetes in
primary care. The key informants included two practice
nurses who ran diabetic clinics and two GPs with
experience in providing care to patients with diabetes.
Some factors identified by the literature review, brainstorming and the focus groups were explored in more
detail, with key members responsible for delivering
diabetes care in general practice. Face-to-face interviews
lasted 20–30 minutes.
Patients
Consulting the public is an important component of
commissioning health services and it has been suggested
that health care workers wanting to know the values
people attach to health services should adopt qualitative
approaches.29 There is increasing interest in incorporating lay perspectives, especially people with experience
of the disorder, in health services research with the potential to influence the research priorities and identification
of problems.30 The importance of incorporating quality
assurance and the needs and perspectives of people with
diabetes has been well recognized.31,32 Factors relating
to quality care of patients with diabetes for patients
was obtained by brainstorming at a meeting of the local
branch of the British Diabetic Association (BDA).
Attendees at the BDA meeting included patients with
both insulin and non-insulin-dependent diabetes and
their carers.
TABLE 1
Practice factors that may be associated with quality of
diabetes care
Method of identification
Partner/s with an interest in diabetes
a, b, c, e
General practice diabetes education
a, b, c
Vocationally trained doctors
Practice nurse with an interest in diabetes
a
a, b, c, e
Young partners
a, b, c
Training practice
a, b, c
Teaching practice
b, c
Practice size (number of patients)
a, b
Number of partners
Number of practice nurses
Practice manager
b, c, d
a, b, c, d
a, b
Appointment system
b
Personal list
b
Computerized practice
Attachment of a practice manager
Practice workload
a, b, c, d
b
a, b, c, d
Fundholding
c
Teamwork in practice
c
Practice motivation
a, b, c, d
Type of consultation
a, b
a
Literature review.
Brainstorming.
c Key informant technique.
d Focus group.
e Patients.
b
Results
A wide range of potential factors were identified. The
five methods identified 54 potential factors that may
be related to good diabetic care. Twenty (37.0%) were
practice-related factors (Table 1), 20 (37.0%) were organizational factors (Table 2) and 14 (25.9%) were patientrelated factors (Table 3). Brainstorming identified
89.2% (33/37) of the factors identified by the literature
review. Table 4 shows the proportion of potential factors
identified by each of the five methods and the yield
of combining different methods. Only 5.6% (3/54) of
factors were identified by all five methods.
There was overlap of factors identified by the various
methods; however, the focus group and the key
informants identified specific and detailed enablers and
obstacles to quality diabetes care. For example, in the
brainstorming session, availability of individual members of the diabetes team were identified as broad themes.
However, the key informants found that teamwork
among these individuals was also essential for highquality care. Although only a few factors were identified
by patients and carers, they were generally very critical
of the care received in primary care. During the session
with patients, it was apparent that they felt that easy
access to a primary care diabetes service and delivery of
care by individuals who were interested in diabetes was
essential for good quality care.
Discussion
The proportion of patients with diabetes being reviewed
in general practice has increased since 1990,33 but the
level of care is variable.2,3 A potential explanation is that
the delivery of care to patients with diabetes is complex,
with many factors influencing the care.5 Previous
research by Pringle and colleagues has shown that only a
few factors out of 25 potential factors explained a small
proportion of variation in diabetes care.5 The present
study found a further 29 factors, of which only 12 factors
have been reported in the literature. The complexity of
492
TABLE 2
Family Practice—an international journal
Organizational factors that may be associated with quality
of diabetes care
Method of identification
Presence of a diabetic register
a, b, c, d
Presence of a recall system
a, b, c, d
Structured care in a diabetic clinic
a, b, c, d, e
Special arrangements to see diabetic patients
Attachment of a diabetes nurse/health visitor
Access to a chiropodist
Access to an optician/optometrist
Access to a dietician
Presence of a glucometer
Approved for chronic disease management
Proportion of patients self-monitoring
Prevalence of diabetes
Good local secondary care diabetic services
GP diabetes education
b, e
a, b, c, d
a, b, c, d, e
a, b, c, e
a, b, c, d, e
a, b, c
a, b
a, b, c, e
b, d
a, b, d
a, b, c, e
Practice nurse diabetic education
b, c, d
Involved in diabetes audit/quality
assurance programme
a, b
Presence of diabetes protocol/guideline
Development of practice protocol/guideline
a, b, c, d
b, d
Presence of decision support system
a
Introduction of change as a result of audit
b
For key, see footnote to Table 1.
TABLE 3
Patient factors that may be associated with quality of
diabetes care
Method of identification
Patient education
Sex of patient
Psychological factors
Type of treatment
Type of diabetes
Patient self-monitoring diabetes care
Length of illness
Mobile population
a, b
a
a
a, b
a, b, d
a, b, c, e
a
b
Frequency of attendance
a, b, c, e
Deprived patients
a, b, c, d
Unemployed patients
b, d
Elderly patients
b, d
Patients with chronic illness
Ethnicity
delivery of care to people with diabetes is therefore
reflected in the large number of potential factors identified in this study.
The methods in this study were used to identify themes
rather than produce a list of specific interventions
that may be related to good quality diabetes care. A
qualitative approach was employed for generating the
hypotheses given the exploratory nature of the study.
Qualitative techniques offer an alternative approach
especially to identify how people feel about the services
they deliver or receive and to explore issues in greater
depth.
b
c, d
Implications
A questionnaire study can be designed to identify
potential factors related to quality care in diabetes;
however, these surveys depend on response rates which
have been low in recent surveys.34 Multiple methods can
therefore be useful for obtaining a variety of information
on the same issue and to overcome the deficiencies of
single-method studies.
A comprehensive literature review is important in
early stages of planning any research study. Qualitative
methods were used to explore additional factors from
users and providers of diabetes care. Furthermore,
quantitative data, such as audit data, can be used to
stimulate hypothesis generation. Literature searching
and brainstorming were the most effective methods of
identifying the potential factors and together identified
nearly 95% of the factors. The combination of literature
searching and brainstorming may be an acceptable alternative for hypothesis generation, and other researchers
may wish to adopt this pragmatic approach if there
are limitations on time or resources. Although there is
increasing interest in incorporating lay perspectives in
health services research and in exploration of barriers to
effective care,35 in this study the patients or their carers
did not identify factors in addition to those identified by
other methods.
Findings from such analyses can support the relevance
and identification of factors. It has been recognized
that for health care evaluation, a wide range of research
methods is warranted.36 Each evaluation method provides
unique data. Although multiple methods are common
in nursing37 and social science research,38 there are only
a few published general practice studies that have
reported similar methods in the UK. These have been
used in rapid appraisal for planning primary care
services.39,40
Varying approaches for hypothesis generation lead to
identification of different aspects for the same topic,
leading to completeness. This study shows the feasibility
of linking qualitative and quantitative methods for
hypothesis generation. The study also demonstrates how
the exploration of related factors (practice, organization
of care and patient) not addressed by a single method
may be aided by multiple methods. Thus, multiple methods
493
Multiple methods for assessing quality of care
TABLE 4
Number (%) of potential factors that may affect quality of care of patients with diabetes identified by multiple methods
Literature
review (a)
Brainstorming
(b)
Key
informant
technique (c)
Focus
group (d)
Patients
(e)
Combination
of a+b
Combination
of b+c
Practice factors (20)
13 (65.0)
17 (85.0)
13 (65.0)
5 (25.0)
2 (10.0)
18 (90.0)
19 (95.0)
Organizational factors (20)
15 (75.0)
19 (95.0)
12 (60.0)
11 (55.0)
7 (35.0)
20 (100.0)
19 (95.0)
9 (64.3)
10 (71.4)
4 (28.6)
5 (35.7)
2 (14.3)
13 (92.9)
11 (78.6)
37 (68.5)
46 (85.2)
29 (53.7)
21 (38.9)
11 (20.4)
51 (94.4)
49 (90.7)
Patient factors (14)
Total (54)
enable a complete picture which may have been missed
if only one approach had been applied. A major strength
of using these qualitative methods is the proximity of the
researcher to key members responsible for delivering
care: users of services provide greater insight into the
topic being researched. Qualitative methods therefore
allow more personalized insight into those involved in
delivery of services. Furthermore, involvement of all key
stakeholders, including patients clinicians and policy
makers, minimizes risk of unrepresentative sampling.
The multiple methods used are complementary, especially the literature review and the brainstorming which
guided further in-depth analysis of enablers and barriers
in the focus group and key informant sessions. The list of
potential factors generated can also be used by other
researchers.
Limitations
Several problems are, however, inherent in using
multiple methods for hypothesis generation. Major
barriers to conducting such multiple methods include
time and cost and skills of the researcher. The literature
search required a substantial amount of time (approximately 16 hours). Brainstorming, key informant technique and focus groups required an additional 3 hours.
However, this does not include the time required of the
participants and for arranging the meetings. Furthermore, for research purposes the list of potential factors
identified by these methods may be so large that it would
be difficult to test all these factors in one study. In
addition, this present study did not attempt to prioritize
the identified factors on the basis of importance.
There are other difficulties in using multiple methods
in research because of its complexity. Use of multiple
methods also requires careful planning. The researcher
may have limited methodological training that may be
limited in certain areas. The researcher also needs to be
aware of the strengths and weaknesses of each method.
This method also relies on the description and interpretation by the researcher. Furthermore, the research
question needs to be clearly focused for participants. In
addition, combining and interpretation of data obtained
by multiple methods can be difficult.
Conclusions
This study has shown that multiple methods can be a
useful tool in exploratory research in primary care. A
broad range of issues relating to care of patients with
diabetes in primary care have been identified. The
factors emerging from this work confirm some hypotheses derived from previous research and suggest new
ones for exploration. This study has developed a way of
combining various traditional methods in an attempt to
overcome the deficiencies and bias that may occur when
using a single method. Similar methods can therefore
be used to generate hypotheses for other exploratory
research. Furthermore, multiple methods may also be
useful for health authorities and primary care groups to
assess the health needs of their local populations and to
commission services to meet these needs.41 The next
stage of the study will explore the relationship between
some of the identified factors and the quality of care of
patients with diabetes. Some of these factors have been
identified in the literature and other new ones identified
by qualitative methods. Further research needs to be
conducted on how patients can be used in identifying
factors relating to good quality care (primary and secondary) and on issues of priority setting.
References
1
2
3
4
5
6
Griffin S. Diabetes care in general practice: meta-analysis of
randomised control trials. Br Med J 1998; 317: 390–396.
Bennet IJ, Lambert C, Hinds G, Kirton C. Emerging standards for
diabetes care from a city-wide primary care audit. Diabet Med
1994; 11: 489–492.
Khunti K, Baker R, Rumsey M, Lakhani M. Quality of care of
patients with diabetes: collation of data from multi-practice
audits of diabetes in primary care. Fam Pract 1999; 16: 54–59.
O’Brien K. Using focus groups to develop health surveys: an
example from research on social relationships and AIDSpreventive behaviour. Health Educ Q 1993; 20: 361–372.
Pringle M, Stewart-Evans C, Coupland C, William SI, Allison S,
Sterland J. Influences on control in diabetes mellitus: patient,
doctor, practice, or delivery of care? Br Med J 1993; 306: 630–
634.
Carney T, Helliwell C. Effect of structured postgraduate medical
education on the care of patients with diabetes. Br J Gen Pract
1995; 45: 149–151.
494
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Family Practice—an international journal
Day J, Humphries H, Alben-Davies H. Problems of comprehensive
shared diabetes care. Br Med J 1987; 294: 1590–1592.
Mellors JG, Samanta A, Blandford RL et al. Questionnaire survey
of diabetes care in general practice in Leicestershire. Health
Trends 1985; 17: 61–63.
Whitford DL, Southern AJ, Braid E, Roberts SH. Comprehensive
diabetes care in North Tyneside. Diabet Med 1995; 12: 691–695.
Hiss RG. Barriers to care in non-insulin-dependent diabetes
mellitus. The Michigan experience. Annu Intern Med 1996; 124:
146–148.
Jacques CHM, Jones RL. Problems encountered by primary care
physicians in the care of patients with diabetes. Arch Fam Med
1993; 2: 739–741.
Hawthorne K, Tomlinson S. One-to-one teaching with pictures—
flashcard health education for British Asians with diabetes.
Br J Gen Pract 1997; 47: 301–304.
Wilson W, Ary DV, Biglan A, Glasgow RE, Toobert DJ, Campbell
DR. Psychological predictors of self-care behaviors (compliance) and glycaemic control in non-insulin dependent diabetes
mellitus. Diabet Care 1986; 9: 614–622.
Blaum CS, Velez L, Hiss RG, Halter JB. Characteristics related to
poor glycaemic control in NIDDM patients in community care.
Diabet Care 1997; 20: 7–11.
Butler C, Smithers M, Stott N, Peters J. Audit enhances district-wide
primary care for patients with diabetes. Eur J Gen Pract 1997;
3: 23–27.
O’Conner PJ, Rush WA, Peterson J et al. Continuous quality
improvement can improve glycemic control for HMO patients
with diabetes. Arch Fam Pract 1996; 5: 502–506.
Feder G, Griffiths C, Highton C, Eldridge S, Spence M, Southgate L.
Do clinical guidelines introduced with practice based education
improve care of asthmatic and diabetic patients? A randomised
controlled trial in general practice in east London. Br Med J
1995; 311: 1473–1478.
Vaughan NJA, Potts A. Implementation and evaluation of a decision
support system for type II diabetes. Computer Methods
Programmes Biomed 1996; 50: 247–251.
Lobach DF, Hammond WE. Computerized decision support based
on a clinical practice guideline improves compliance with care
standards. Am J Med 1997; 102: 89–98.
Benbow SJ, Walsh A, Gill GV. Diabetes in institutionalised elderly
people: a forgotten population? Br Med J 1997; 314: 1868–1869.
Ward J, Lin M, Heron G, Lajoie V. Comprehensive audit of qualityof-care and quality-of-life for patients with diabetes. J Qual Clin
Pract 1997; 17: 91–100.
Kamien M, Ward AM, Mansfield F, Fatovich B, Mather C, Anstey K.
Management of type 2 diabetes in Western Australian
metropolitan general practice. Diabetes Res Clin Pract 1994;
26: 197–208.
Kinmonth AL, Spiegal N, Woodcock A. Developing a training
programme in patient-centred consulting for evaluation in a
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
randomised controlled trial; diabetes care from diagnosis
in British primary care. Patient Educ Counselling 1996; 29:
75–86.
McMurray AR. Three decision making aids. J Nursing Staff Devel
1994; 10: 62–65.
Powell RA, Single HM, Lloyd KR. Focus groups in mental health
research: enhancing the validity of user provider questionnaires.
Int J Soc Psyth 1996; 42: 193–206.
Kitzinger J. Introductory focus groups. Br Med J 1995; 311: 299–302.
Powell RA, Single HM. Focus groups. Int J Qual Health Care 1996;
8: 499–504.
Marshall MN. The key informant technique. Fam Pract 1997; 12:
92–97.
Bowie C, Richardson A, Sykes W. Consulting the public about
health service priorities. Br Med J 1995; 311: 1155–1158.
Entwistle VA, Renfrew MJ, Yearley S, Forrester J, Lamont T. Lay
perspectives: advantages for health resaerch. Br Med J 1998;
316: 463–466.
Duchin SP, Brown SA. Patients should participate in designing
diabetes educational content. Patient Educ Counsel 1990; 16:
255–267.
Anderson LA, Satterfield D, German R, Anderson RM. Using
quantitative and qualitative methods to protest the publication
Take Charge of Your Diabetes: A Guide for Care. The Diabetes
Educator 1996; 22: 598–604.
Goyder EC, McNally PG, Drucquer M, Spiers N, Botha JL. Shifting
of care for diabetes from secondary care to primary care, 1990–
5: review of general practices. Br Med J 1998; 316: 1505–1506.
McAvoy BR, Kaner EF. General practice postal surveys: a
questionnaire too far? Br Med J 1996; 313: 732–733.
Quill TE. Recognising and adjusting to barriers in doctor-patient
communication. Ann Int Med 1989; 111: 51–57.
Black N. Why we need observational studies to evaluate the
effectiveness of health care. Br Med J 1996; 312: 1215–1218.
Corner J. In search of more complete answers to research questions.
Quantitative versus qualitative research methods: is there a way
forward? J Adv Nursing 1991; 16: 718–727.
Halvorson HW, Pike DK, Reed FM, McClatchey MW, Gosselink
CA. Using qualitative methods to evaluate health service delivery
in three rural Colorado communities. Eval Health Professions
1993; 16: 434–437.
Murray SA, Tapson J, Turnbull L, McCallum J, Little A. Listening to
local voices: adapting rapid appraisal to assess health and social
needs in general practice. Br Med J 1994; 308: 698–700.
Dale J, Shipman C, Lacock L, Davies M. Creating a shared vision of
out of hours care: using rapid appraisal methods to create an
interagency, community oriented, approach to service development. Br Med J 1996; 312: 1206–1210.
Pearson N, O’Brien J, Thomas M, Ewings P, Gallier L, Bussay A.
Collecting morbidity data in general practice: the Somerset
morbidity project. Br Med J 1996; 312: 1517–1520.