International Journal for Quality in Health Care 1999; Volume 11, Number 1: pp. 5–12
Using antecedents of medical care to
develop valid quality of care measures
YVONNE M. COYLE AND JAMES B. BATTLES
1
Division of General Internal Medicine and 2 Office of Medical Education, The University of Texas, Southwestern Medical Center at
Dallas, Dallas, TX, USA
Abstract
Objective. To present a new model for using the antecedents of medical care in outcomes assessment to develop valid
quality of care measures.
Methods. The pertinent literature describing the history of outcomes assessment and the influence of patient and
environmental risk factors on health status were reviewed.
Results. Past outcomes assessment studies have not consistently demonstrated a correlation between the processes and the
outcomes of care. The use of the model described in this article indicates that the lack of correlation between process and
outcome is probably because past outcomes assessment studies lacked the inclusion of medical care antecedents (primarily
patient and environmental risk factors) that had a significant influence on the outcomes measured. Included is a description
of a study that tests the utility of incorporating the antecedents of medical care into outcomes assessment to develop valid
quality of care measures.
Conclusion. The model presented in this article advances quality of care measure development by using: (i) qualitative
research to characterize the pertinent antecedents of medical care; and (ii) as many of the pertinent antecedents of medical
care as possible to develop risk adjustment models for measuring outcomes that are more likely to identify the true linkages
between the processes and outcomes of care. Knowing the linkages between the processes and outcomes of care would
provide the information needed to develop valid quality of care measures, so that quality can be measured accurately and
the groundwork for its improvement can be laid.
Keywords: health care, health services research, outcome and process assessment, outcome assessment, quality indicators,
risk assessment
Health care quality is a topic of concern for employers and
consumers [1]. Quality of health care has also been the focus
of recent articles in the medical literature, and in March of
1998 the issue was addressed by the Advisory Commission
on Consumer Protection and Quality in the Health Care
Industry in response to a USA Presidential Executive Order
[2]. As a result, many groups are now developing quality of
care measures that will be used to judge the quality of
care delivered by health care organizations and physicians.
However, because very few of the quality of care measures
that have been developed are valid, this suggests the need to
establish models for developing valid quality of care measures
[3].
The Institute of Medicine (IOM) has suggested that the
validity of quality of care measures be ‘based on outcomes
studies or other scientific evidence of effectiveness’ [3]. In
other words, for a quality of care measure derived from
process data to be valid, variations in the process of care
related to the process data measured must lead to a change
in the outcomes. Similarly, if outcomes criteria for quality
of care are to be valid, one must demonstrate differences
in the outcomes measured if the related process of care
is changed [4]. Unfortunately, many outcomes assessment
studies have not demonstrated a correlation between the
processes of care and outcomes of care [5]. A methodological criticism of these studies could be that their
conceptual framework lacked the inclusion of variables
that had a significant influence on the outcomes measured,
such as the antecedents of medical care (primarily patient
personal characteristics and their environmental context)
[4,5]. To assist in developing valid process and outcome
quality of care measures, the authors suggest expanding
Address correspondence to Yvonne M. Coyle, Assistant Professor, Internal Medicine, Division of General Internal Medicine,
The University of Texas, Southwestern Medical Center at Dallas, 5323 Harry Hines Boulevard, Dallas, TX 75235–8889,
USA. Tel: +1 214 648 2992. Fax: +1 214 648 2087. E-mail: [email protected]
 1999 International Society for Quality in Health Care and Oxford University Press
5
Y. M. Coyle and J. B. Battles
outcomes assessment to include as many as possible of
the pertinent antecedent variables of medical care that
significantly influence the outcomes of care. By using this
expanded approach to outcomes assessment one could
develop a risk-adjustment model for measuring the outcomes
of interest and thus be more likely to identify the true
linkages between the processes and outcomes of care [6].
Knowing the connections between the processes and
outcomes of care would provide the information needed
to develop valid quality of care measures. This article
discusses the rationale for incorporating the antecedents
of medical care into the conceptual framework for outcomes
assessment to develop quality of care measures, with an
example of its application in the clinical setting.
Structure–process–outcome model
Donabedian’s structure–process–outcome model, developed
over 30 years ago, continues to serve as a unifying
conceptual framework for health services and outcomes
assessment. Our suggested approach for expanding outcomes assessment to develop quality of care measures
builds on this model, by considering as many as possible
of the antecedents of medical care that affect outcomes.
Donabedian’s definitions for structure, process and outcome
were: (i) structure – the physical and organizational
properties of settings in which care is provided; (ii) process
– what is done for patients; and (iii) outcome – what is
accomplished for patients [7].
Antecedents of medical care and health
status
The idea of studying the influence of an individual’s
personal characteristics and environmental context on health
is not new. As early as the middle 1800s, industrialization
and its attendant social problems led various European
investigators to study the effects of these factors on health
[8]. The socioeconomic factors have been the most widely
studied: educational attainment, income level, employment
status, insurance status, housing status, psychological stress,
life events, social networking, control over one’s work
situation, social mobility, occupational and environmental
hazards (to name but a few); all of these affect health
status [8–13]. There is also some evidence that social–cultural
differences influence patient health promotional behavior
[14].
Evans and Stoddard argued that a framework for looking
at the relationship between environment and health status is
necessary. They stressed that health care decision makers must
have this additional information to develop comprehensive
health policies [15]. Others such as Starfield and later the
IOM also recommended that health services research, which
now includes outcomes assessment, recognize the pertinent
variables and events outside of the health care system when
6
evaluating the outcomes of medical care [16,17]. The IOM
defined health services research as ‘a multi disciplinary field
of inquiry, both basic and applied, that examines the use,
costs, quality, accessibility, delivery, organization, financing,
and outcomes of health care services to increase knowledge
and understanding of structure, processes, and effects of
health services for individuals and populations’ [18]. The IOM
indicated that health services research could be organized
into four general levels that include factors affecting patient
outcomes: clinical, institutional, systemic or environmental.
These levels proceeded from the core of clinical practice to
the relationships between the health care delivery system and
the larger social, political and economic environment that
affect them [17].
Furthermore, it has been suggested that when evaluating
the effects of health services, the research plan should
have many of the same requirements as for epidemiological
research. The issues faced in epidemiological research are:
what are the study objectives? what are the outcome
measures? what are the strongest methods for conducting
the study? how are the risks for not reaching conclusions
minimized? Regarding the last question, in past studies it
has often been difficult to determine whether outcomes
were due to the interventions or to patient factors. These
observations support the need to develop models that seek
to explain the influence that patient factors have on the
outcomes of care [19].
Antecedents and program evaluation in
education
If we examine disciplines other than health care, we find that
there are evaluation approaches that use antecedents to assess
outcomes. In the field of education, program evaluation
was originally developed to examine educational program
outcomes in the context of the related environment. In the
early 1970s, Stufflebeam et al. developed a program evaluation
model to make better decisions about educational interventions. These authors defined four stages for the evaluation process (context, input, process, and product or CIPP)
[20]. Determining the context of an evaluation involves
defining the environment in which the program exists and
includes documenting the conditions (antecedents) that existed prior to the advent of the program. Most often, contextual data help to adapt an existing program into a local
context. Rutman also emphasized the importance of using
antecedent information in the evaluation process [21]. The
input portion of an evaluation consists of the program
developer’s perceived needs and objectives for the program.
Process evaluation involves gathering data on the procedures
used to implement the project. Product evaluation involves
documenting the outcomes of the project. Later, Guba and
Lincoln recommended that program evaluation use qualitative
research methods, when appropriate, to identify the pertinent
contextual variables [22].
Antecedents for quality measures
Table 1 Using antecedents in outcomes assessment to develop valid quality of care measures
Antecedents
Structure
Process
Outcome
..................................................................................................................................................................................................
Environment
System characteristics
Technical style
Clinical end points
Cultural
Provider characteristics
Interpersonal style Functional status
Social
Patient
General well-being
Political
characteristics
Satisfaction with care
Personal
Physical
Health professions
Patient personal characteristics
Using antecedents to develop quality of
care measures
Using Donabedian’s structure–process–outcome model and
the CIPP model as guides, a framework for developing quality
of care measures was developed that includes as many as
possible of the pertinent antecedents of medical care [7,
20]. This framework assesses the effects of the antecedent,
structural, and medical care process variables on the outcomes
of care (Table 1).
Antecedents for medical care
Antecedents are factors that affect the structure, process and
outcomes of medical care. Therefore, antecedents could have
the greatest affect on outcomes. Antecedents involve the
environmental context of an individual and an individual’s
personal characteristics (i.e. genetics, socio-demographics,
health habits, beliefs and attitudes, and preferences). Environmental factors may be cultural, social, political, personal,
physical, or related to the health professions.
Structure of medical care
Structure describes the setting in which medical care takes
place. Its elements include the health care system’s characteristics (i.e. organization, personnel, specialty mix, financial
incentives, patient volume, access, facilities, and equipment),
provider characteristics (i.e. socio-demographics, specialty
training, economic incentives, beliefs and attitudes, preferences, and job satisfaction) and patient characteristics (diagnosis, severity of illness, and comorbidity) [7].
Process of medical care
The process of medical care is what is done for patients.
This involves a provider’s technical and interpersonal style.
Technical style refers to the specific services used and the
way in which providers manage the episode of care, which
includes continuity of care and its coordination. We measure
what we do for patients in terms of visit rates, tests ordered,
medications prescribed, referrals made, hospital rates, and
revenue generated. Interpersonal style refers to the way
providers relate to patients. This involves the provider’s
interpersonal manner, communication ability and style, and
whether providers counsel patients about their health habits
and the extent to which providers include patients in medical
decision making [7].
Outcomes of medical care
Outcomes are the ultimate test of the effectiveness of medical
care. Patient outcomes are clinical endpoints (i.e. laboratory
values, morbidity, and mortality), functional status (physical,
mental, social, and role), general well-being (health perception,
energy and fatigue, pain and life satisfaction), and satisfaction
with medical care (access, convenience, financial coverage,
quality, and general) [7].
Applying the use of antecedents for developing
quality of care measures
Both quantitative and qualitative research methods may be
required to characterize and study the relationships between
the antecedent, structural, medical care process, and outcome
variables. Qualitative research seeks to understand a social
or human problem by building a complex holistic picture
formed with words obtained from the detailed reports of key
informants. This type of research does not use hypothesis
testing, whereas quantitative research involves inquiry into a
social or human problem, based on testing a hypothesis
composed of variables, measured with numbers, and analyzed
with statistical procedures to prove whether or not a predictive
generalization holds true [23]. Qualitative methods such as
observation, in depth interviews, and focus groups provide
an understanding of a situation or behavior. Furthermore,
qualitative research methods can be used to improve the
understanding of quantitative research studies [24,26].
The research methods used to incorporate the antecedents
of medical care into the framework outcomes assessment,
for the purpose of developing quality of care measures, will
vary depending on the context of the situation, as well as
the medical condition and population studied. Ten steps can
be used to develop valid quality of care measures: steps 1–8
involve planning the evaluation in the following order:
1. selecting the medical condition and treatment for study;
2. reviewing the literature;
3. selecting the processes of care to be linked to the
outcomes studied;
7
Y. M. Coyle and J. B. Battles
4.
5.
6.
7.
8.
defining the study population;
developing the conceptual model for the study;
estimating the sample size for the study;
specifying the study time period;
planning the data collection.
In step 5, the investigator develops the conceptual model for
the study by identifying the pertinent antecedent, structural,
and process variables and defining their relationships to the
outcomes of interest. The creation of a conceptual model
for outcomes assessment is a critical step for developing
valid quality of care measures [26]. The conceptual model
for outcomes assessment indicates what is believed to cause
the outcome. More specifically, the conceptual model should
identify which variables are pertinent to the study being
undertaken and their relationships to the outcomes of interest.
In addition, the conceptual model identifies what variables
need to be controlled in the final analysis or what is sometimes
referred to as ‘risk-adjusting’ the outcomes of interest. Riskadjustment attempts to account for all patient factors that
could affect outcomes. The remaining variation in patient
outcomes is thought to be due to the reliability of the data
sources used for data collection, random variation itself, and
the quality of care that the patients received. The conceptual
model for developing the quality of care measures evaluates
the potential relationships between the different dimensions
of patient risk to the outcomes of interest. Establishing the
presence of these relationships provides what has been called
the ‘medical meaningfulness’ for an outcomes risk-adjustment
model. Establishing a ‘medically meaningful’ risk-adjustment
model for measuring outcomes is important, especially with
respect to subpopulations that may be at high risk for a
medical condition and/or its complications related to their
environmental and/or personal characteristics. It is important
to emphasize that some patient factors such as sex and
race or ethnicity may affect the outcomes, not because of
physiology but because of the differences in the way people
are treated. Therefore, risk-adjusting outcomes can mask the
quality of care received by certain patient subgroups [27].
However, because the purpose of the model presented in
this article is to use an outcomes assessment framework for
developing valid quality of care measures, not to measure
quality of care, masking the differences in the way that people
are treated is not an issue. Steps 9–10 are the execution
steps that include the data collection and documentation of
outcomes. The 10 steps presented above to develop valid
quality of care measures determines whether the variations
in outcomes are explained by the differences in antecedents,
and uses qualitative research methods whenever possible
to identify the antecedents of medical care that influence
significantly the outcomes of care.
An example of quality of care measure
development using antecedents
In an effort to test the utility for using the antecedents of
medical care in outcomes assessment to develop valid quality
8
of care measures, Coyle et al. are undertaking the project
‘Developing and Testing Asthma Quality of Care Measures’
funded by The Agency for Health Care Policy and Research
(Table 2) [28]. A brief description of this project follows to
demonstrate how one can apply this approach in a clinical
setting. The site for the project’s studies is Parkland Health
and Hospital System (Parkland). Parkland is a public hospital
system, and the primary teaching hospital system for the
University of Texas Southwestern Medical Center at Dallas
(UT Southwestern).
The primary purpose of the project is to develop and test
process quality of care measures for acute asthma exacerbation. One of the major reasons that the investigators
chose asthma as the ‘tracer condition’ is that an individual’s
personal characteristics and their environmental context have
a significant impact on asthma outcomes [29]. Adults with
asthma are the target population, with the level of analysis
being at the provider level. The providers are faculty and
physicians in training at the UT Southwestern. The project
consists of two phases. During phase I, the investigators
identified 10 process of care criteria for the management
of acute asthma exacerbation using the National Asthma
Education and Prevention Program (NAEPP) clinical practice
guidelines to conduct an effectiveness study [30]. The 10
process of care criteria are as follows:
1. inhaled short-acting B2-agonists for all patients;
2. inhaled anti-cholinergics for severe asthma exacerbations and patients with impending or actual
respiratory arrest;
3. oxygen for patients with an oxygen saturation<90%;
4. systemic corticosteroids for patients with moderate or
severe asthma exacerbations, or patients with mild
asthma exacerbations that have recently been treated
with systemic corticosteroids;
5. inhaled B2-agonists at discharge for all patients;
6. inhaled corticosteroids (double the current dose for
7–10 days) for patients with mild asthma exacerbations
currently using inhaled corticosteroids at discharge;
7. systemic corticosteroids at discharge for patients with
moderate and severe asthma exacerbations, or for
patients who were recently treated with systemic
corticosteroids at discharge;
8. medical facility follow-up visit within 5 days for asthma
care after discharge from the emergency room or clinic
for all patients;
9. patient education on metered-dose inhaler use technique for all patients at discharge;
10. patient education on a written asthma medication plan
for all patients at discharge.
Subsequently, the project’s studies will identify which of
the process of care criteria for acute asthma exacerbation
and the associated antecedents of medical care (patient sociodemographics, smoking – active and passive, recent history
of an upper respiratory tract infection, asthma knowledge
[31], air quality – ozone, type and location of residence,
asthma medication adherence [32], asthma severity [30], current allergy to aeroallergens [33], comorbidity, medical facility
Antecedents for quality measures
Table 2 An example of how to develop valid quality of care measures: the ‘Developing and Testing Asthma Quality of
Care Measures’ study
Antecedents
Structure
Process
Outcomes
............................................................ ...........................................................................................
Variable
Patient personal Patient’s
Patient
Medical facility Physician
Medical care Patient
categories
characteristics environmental
context
................................................................................................................................................................................................................................................
Study
SocioAir quality
Asthma severity Site of care
Demographics Asthma
Patient inhaler
variables
demographics (ozone and
Aeroallergen
(emergency
(age, sex, race process of
use technique
(age, gender,
aeroallergen
sensitivity
room, or clinic) and ethnicity) care criteria
Lung function
ethnicity or
exposure)
Comorbidity
Patient to staff Professional
for acute
test
race, income,
Residence type Asthma
ratio
characteristics asthma
measurements
number of
Residence
medication
(specialty, USA exacerbation on air flow
dependents,
owned or
adherence
or foreign
limitation
health insurance rented
Upper
medical school
Patient
status,
Smoking in
respiratory
graduate, and
satisfaction with
education,
residence
infection in past
training level)
medical care
employment,
month
Asthma quality
occupation, and
of life
marital status)
Medical facility
Smoking
utilization for
Asthma
acute asthma
knowledge
exacerbation
Asthma
mortality
Study
Check Your
Ozone and
NAEPP
Self-reported
NAEPP
Lung function
measures
Asthma
aeroallergen
guidelines for medication
guidelines for testing (preand
‘I.Q.’ [31]
measurements
assessing
adherence [32]
acute asthma and postprocedures
asthma severity Allergy skin
exacerbation bronchodilator
[30]
testing [33]
management FEV1) [34]
[30]
McMaster
Asthma Quality
of Life
Questionnaire
[35]
Patient Visit
Rating
Questionnaire
[36]
Data
Research clinic Research clinic Medical facility Medical facility Medical facility Medical
Research clinic
collection
visit
visit
Research clinic
facility
visit
sites
Selected sites
visit
within Dallas
County (ozone
and aeroallergen
measurements)
Data
SelfSelfSelfMedical record Medical facility Medical
Selfcollection
administered
administered
administered
review
physician
record review administered
procedure
and interviewer- and interviewer- and interviewer- Medical facility credential
and intervieweradministered
administered
administered
administrative record review
administered
questionnaires questionnaires
questionnaires record review
questionnaires
Documents and Medical record
Research clinic
records review review
visit records
Research clinic
review
visit record
review
9
Y. M. Coyle and J. B. Battles
patient to staff ratio, and physician characteristics) are predictors of outcome 2 weeks after treatment for an acute
asthma exacerbation. Study subjects are determined to have
a current allergy to an aeroallergen based on a positive allergy
skin test for the aeroallergen and exposure to the aeroallergen
during the 2 weeks prior to measuring the outcomes studied.
The outcomes of care for the study are: (i) lung function
measurements for airflow limitation (pre- and post-bronchodilator FEV1) [34]; (ii) asthma quality of life [35]; (iii) satisfaction with medical care [36]; (iv) medical facility utilization
for asthma exacerbation; and (v) asthma mortality.
Next, the investigators will use the process of care predictors to develop candidates for process quality of care
measures. A pilot study will follow to assess preliminarily the
validity and reliability of these measures, as well as to develop
a data retrieval system for obtaining their data elements.
Phase II involves a second effectiveness study that will
assess the same outcomes as the first effectiveness study in
phase I. However, in this study the investigation team will
use the non-process of care predictors to risk-adjust the
outcomes measured, and to concurrently identify the data
elements for the process quality of care measure candidates
using the data retrieval system. Using the process of care
candidates that predict the risk-adjusted outcome measures,
they will determine how often these measures detect inadequate care (sensitivity) and pass over adequate care (specificity) based on both the process (whether processes of care
are present, present and misused, or absent) and the outcomes
(desirable versus undesirable) of care. They will select the
process quality of care measures for asthma based on their
sensitivity and specificity, as well as the feasibility and cost
for identifying them.
Asthma is a good example of a medical condition in which
one must take into account a patient’s personal characteristics
and environmental context to evaluate effectively the outcomes of care (Table 2). Preliminary results from the first
173 cases (the sample size targeted for this study with 40
predictors is 400 cases) in the first effectiveness study during
phase I, indicated that increasing asthma severity (measured
using the NAEPP clinical practice guidelines [30]) correlated
with reduced lung function; older age and non-white race
showed a trend for correlating with reduced lung function
[34]. Several patient environmental factors correlated with
higher asthma quality of life scores (asthma knowledge [31],
younger age, lower comorbidity, lower asthma severity [3],
higher family income, higher educational level, and employment); whereas current allergy [33] correlated with lower
asthma quality of life scores [35]. It is generally believed that
age, sex, comorbidity, asthma severity, socioeconomic status,
and possibly race or ethnicity are the most important measures
for risk-adjusting asthma outcomes, and should be considered
for inclusion in risk-adjusting asthma outcomes [37–40].
However, as our results indicate, there are other patient risk
factors that may significantly affect asthma outcomes, such
as current allergy to aeroallergens. The importance of allergens, such as house-dust mites, animal dander, mold spores,
and cockroach allergen in affecting asthma outcomes has
also been suggested by previous studies [41,42]. In addition,
10
other patient risk factors that have been proposed to affect
asthma outcomes based on epidemiological data are poor air
quality and psychosocial problems [43,44]. Therefore, it seems
that we should have a better opportunity to develop valid
quality of care measures for acute asthma exacerbation, if we
use as many of the antecedents of medical care as possible
for asthma outcome risk-adjustment. Furthermore, one could
use the antecedents of medical care to develop quality of care
measures for other medical conditions where an individual’s
personal characteristics or environmental context may affect
significantly the outcomes studied.
Conclusion
Using a conceptual model for developing quality of care
measures that includes the antecedents of medical care provides one with a better opportunity to develop valid quality
of care measures. This approach advances quality of care
measure development by using: (i) qualitative research to
characterize the pertinent antecedents of medical care; and
(ii) as many as possible of the pertinent antecedents of
medical care to develop risk-adjustment models for measuring
outcomes. Developing a risk-adjustment model that takes
into consideration as many of the pertinent antecedents of
medical care as possible should greatly increase our chances
of identifying the true linkages between the processes and
outcomes of care. Knowing the linkages between the processes and outcomes of care would provide the information
needed to develop valid quality of care measures, so that we
can accurately measure quality and lay the groundwork for
its improvement.
Acknowledgments
The project described in this publication is supported by
number: U18 HS09461–01 for the period 9/30/96–09/29/
99, and its contents are solely the responsibility of the authors
and do not necessarily represent the official views of the
Agency for Health Care Policy and Research.
References
1. US General Accounting Office, Health, Education, and Human
Services Division. Health Care. Employers and Individual Consumers
Want Additional Information on Quality. Washington, DC: US
General Accounting Office; 1995. Publication GAO/HEHS95–201.
2. The President’s Advisory Commission on Consumer Protection
and Quality in the Health Care Industry. Quality First: Better
Health Care for All Americans. Washington, DC: US Government
Printing Office, 1998.
3. Institute of Medicine. Medicare. A Strategy for Quality Assurance.
Vol I. Washington, DC: National Academy Press, 1990.
Antecedents for quality measures
4. Brook RH, McGlynn EA, Cleary PD. Part 2: Measuring quality
of care. New Engl J Med 1996; 13: 966–970.
reach: an introduction to qualitative methods in health and
health services research. Br Med J 1995; 311: 42–45.
5. Brook RH, Davies-Avery A, Greenfield S et al. Assessing the
quality of medical care using outcome measures: an overview
of the method. Med Care 1977; 15: Suppl., pp. 5–15.
26. Kane RL. Understanding Health Care Outcomes Research. Gaithersburg, MD: Aspen Publishers Inc., 1997.
6. Iezzoni LI. (ed) Risk Adjustment for Measuring Healthcare Outcomes.
Chicago, IL: Health Administration Press, 1997.
7. Donabedian A. Explorations in Quality Assessment and Monitoring.
The Definition of Quality and Approaches to its Assessment, Vol. I.
Ann Arbor, MI: Health Administration Press, 1980.
8. Rosen G. A History of Public Health. New York, NY: MD
Publications, Inc., 1958.
9. Antonovsky A. Social class, life expectancy and overall mortality.
Milbank Q 1967; 45: 31–73.
10. Feinstein JS. The relationship between socioeconomic status
and health: a review of the literature. Milbank Q 1993; 71:
279–322.
11. Frank JW, Mustard JF. The determinants of health from a
historical perspective. Daedalus 1994; 123: 1–19.
12. Hahn B, Flood AS. No insurance, public insurance: do these
options contribute to differences in general health? J Health
Care Poor Underserved 1995; 6: 41–59.
13. Last JM, Wallace RB. (eds) Public Health and Preventive Medicine.
13th edn. Norwalk, CT: Appleton and Lange (a publishing
division of Prentice Hall), 1992.
14. Palank CL. Determinants of health-promotive behavior. A
review of current research. Nursing Clin North Am 1991; 26:
815–832.
15. Evans GE, Stoddard GL. Producing health, consuming health
care. Soc Sci Med 1990; 31: 1347–1363.
16. Starfield B. Health services research: a working model. N Engl
J Med 1973; 289: 132–136.
17. Institute of Medicine. Report on Health Services Research. Washington, DC: National Academy of Science, 1979.
18. Institute of Medicine. Health Services Research. Work Force and
Educational Issues. Washington, DC: National Academy Press,
1995.
19. Armenian HK, Shapiro S. (eds) Epidemiology and Health Services.
New York, NY: Oxford University Press, 1998.
20. National Study Committee on Evaluation – Phi Delta Kappa.
Educational Evaluation & Decision Making. Itasca, IL: Peacock
Publishers, 1971.
21. Rutman L. Evaluation Research Models, A Basic Guide. Beverly
Hills, CA: Sage Publications, Inc., 1977.
22. Guba EG, Lincoln YS. Effective Evaluation. San Francisco, CA:
Josey-Bass, 1982.
23. Creswell JW. Research Design. Qualitative and Quantitative Approaches.
Thousand Oaks, CA: Sage Publications, Inc., 1994.
24. Kelner MJ, Taylor KM. Strengthening medical research through
an integrated approach. Academic Med 1995; 70: 566–567.
25. Pope C, Mays N. Reaching the parts other methods cannot
27. Iezzoni LI. The risks of risk adjustment. J Am Med Assoc 1997;
278: 1600–1607.
28. Coyle YC, Gruchalla RS, Khan DA et al. Developing and Testing
Asthma Quality of Care Measures. Rockville, MD: Funded by Grant
Number 1H18HS09461–01, The Agency for Health Care Policy
and Research, 1996–1999.
29. Kessner DM, Kalk CE, Singer BA. Assessing health quality –
the case for tracers. N Engl J Med 1973; 288: 189–194.
30. National Heart, Lung and Blood Institute. National Asthma
Education and Prevention Program Clinical Practice Guidelines. Guidelines
for the Diagnosis and Management of Asthma. NIH Publication No.
97–4051. Bethesda, MD: National Institutes of Health, 1997.
31. National Heart, Lung and Blood Institute. Check Your Asthma
‘I.Q.’ NIH Publication No. 92–1128. Bethesda, MD: Department
of Health and Human Services, 1992.
32. Morisky DE, Green LW, Levine DM. Concurrent and predictive
validity of a self-reported measure of medication adherence.
Med Care 1986; 24: 67–74.
33. Ten RM, Klein JS, Frigas E. Allergy skin testing. Mayo Clin Proc
1995; 70: 783–784.
34. Enright PL, Lebowitz MD, Cockroft DW. Physiologic measures:
pulmonary function tests. Asthma outcome. Am J Respir Crit
Care Med 1994: 149: S9–S18.
35. Juniper EF, Guyatt GH, Willa A, Griffith LE. Determining a
minimal important change in a disease-specific quality of life
questionnaire. J Clin Epidemiol 1993; 47: 81–87.
36. Rubin HR, Gandek B, Rogers WH et al. Patient’s ratings of
outpatient visits in different practice settings. Results from the
medical outcomes study. J Am Med Assoc 1993; 270: 835–840.
37. Marder D, Targonski P, Orris P et al. Effect of racial and
socioeconomic actors on asthma mortality in Chicago. Chest
1992; 101: S426–S429.
38. Weiss KB, Wagener DK. Changing patterns of asthma mortality:
identifying target populations at high risk. J Am Med Assoc 1990;
264: 1683–1687.
39. Coultas DB, Gong Jr H, Grad R et al. Respiratory diseases in
minorities of the United States. Am J Respir Crit Care Med 1994:
149: S93–S131.
40. Pierce N, Beasley R, Burgess C, Crane J. Asthma Epidemiology.
Principles and Methods. New York, NY: Oxford University Press,
1998.
41. Rosenstreich DL, Eggelston P, Kattan M et al. for the National
Cooperative Inner-City Asthma Study. The role of cockroach
allergy and exposure to cockroach allergen in causing morbidity
among inner-city children with asthma. N Engl J Med 1997;
336: 1356–1363.
11
Y. M. Coyle and J. B. Battles
42. Call RS, Smith TF, Morris E et al. Risk factors for asthma in
inner city children. J Pediatr 1992; 121: 862–866.
43. Evans II R, Mullally DI, Wilson RW et al. National trends in
the morbidity and mortality of asthma in the US: prevalence,
hospitalization and death from asthma over two decades: 1965–
1984. Chest 1987; 91: S65–S74.
12
44. Birkhead G, Attaway NJ, Strunk RC et al. Investigation of a
cluster of deaths of adolescents from asthma: evidence implicating inadequate treatment and poor patient adherence with
medications. J Allergy Clin Immunol 1989; 84: 484–491.
Accepted for publication 1 September 1998