Methodological quality assessment
of observational studies
Nicole Vogelzangs
Department of Psychiatry
& EMGO+ institute
Outline
• Observational study designs
• Methodological quality assessment:
– Bias and confounding
– Important aspects of quality assessment
– Using methodological quality in meta-analysis
• Causality
Observational I-2
Research designs
• Etiology
– Onset of symptoms or disease
– Risk factors
• Diagnosis
– Assessment of (severity) of symptoms or disease
• Prognosis
– Course of symptoms or disease
– Prognostic factors (includes treatment!)
Observational I-3
Reasons for observational design
• Study of the natural course of disease
• Follow-up of large groups of persons
• Long follow-up necessary
• Randomization not ethical
• Randomization not possible
– e.g. work related exposure
• Practical problems (efficiency, costs)
• Rare diseases or events
– e.g. side effects
Observational I-4
Design (general)
Determinant
Outcome
Confounders
Examples:
I. Does use of Cox-2 inhibitors increase the risk of myocardial
infarction as compared with non-selective anti-inflammatory drugs
(NSAIDs)?
II. Is there an association between stressful events during childhood
and the onset of chronic pain?
III. Does the risk of a chronic course of depression increase because of
the presence of somatic diseases?
Observational I-5
Cross-sectional research
• All determinants and the outcome are
simultaneously assessed
Determinant
Outcome
Examples:
I. Patients with joint pain in 50 general practices who have used
Cox-2 inhibitors or NSAIDs in the past 5 years? Medical file
research: yes/no infarction?
II. Assessment of stressful events in the past and chronic pain
complaints
III. Assessment of depressive symptoms in the past 2 years and the
presence of somatic diseases
Observational I-6
Cohort study
•
•
•
•
Select cohort
Assess exposure to determinant
Follow cohort in time
Identify disease cases
Determinant
Outcome
Example I:
- Patients with joint pain from 50 GPs, who start medication
- (Repeated) assessment of type of painkiller
- Register myocardial infarction cases during 5 years
- Compare incidence infarction with Cox-2 inhibitors vs. NSAIDs
Observational I-7
Patient - control design
• Select disease cases
• Select a (healthy) control group
• Assess exposure to determinant (in the past)
Determinant
Outcome
Example I:
- All (new) infarctions during 1 year in 10 centers
- Select controls (e.g. through general practitioners)
- Compare type of painkiller between patients and controls (OR)
Observational I-8
Outline
• Observational study designs
• Methodological quality assessment:
– Bias and confounding
– Important aspects of quality assessment
– Using methodological quality in meta-analysis
• Causality
Observational I-9
Methodological quality assessment
• Diversity in design: no standard checklist
• Existing checklists differ
• Adjust to topic of review
• Assess presence and degree of possible bias
and confounding
– Selection bias
– Information bias
– Confounding
Observational I-10
Selection bias
• Inadequate selection of participants
• Chance to be selected depends on outcome
(example: safety belt)
• Association between determinant and outcome
is different in the study population compared
with the (theoretical) source population
Observational I-11
Assessment of risk of selection bias
• Does the study population correctly reflect the
source population?
– Clear description setting, selection procedure,
selection criteria
– PC: Do controls and patients stem from the same
source population?
– High response?
• Cohort: are persons ‘at risk’ being selected?
Observational I-12
Information bias
• Incomparable information in patients and controls:
– of the determinant
– of the disease
• Measurements are not assessed in the same way
• Measurements are influenced by knowledge on
determinant and/or disease status (e.g. childhood
events in persons with chronic pain)
Observational I-13
Assessment of risk of information bias
• Are determinant and disease measured in a
standardized way?
– Same method for all participants?
– Definition of cut-off points and diagnostic criteria?
• Are good (=valid & reliable) measurements being
used?
• Are determinant and disease assessed
independently?
– Blinding
– Independent assessment
Observational I-14
Confounding
• The association between determinant and
disease is (partly) explained by other (nonmediating) determinants => confounders
• Confounder is associated with determinant and
with outcome (and is not in the causal pathway)
Determinant
Outcome
Confounder
Observational I-15
Assessment of risk of confounding
• Are potential confounders measured?
• Does the design address confounding (e.g.
restriction, matching)
• Are the statistical analyses well conducted?
– Stratified analyses
– Multivariable analyses (is the model described?)
Observational I-16
Follow-up (cohort design)
• Drop-out of participants can distort results
– High drop-out during follow-up?
– Selective drop-out (drop-out related to exposure)?
• Is duration of follow-up sufficient?
Observational I-17
Summary quality checklist
• Adequate selection procedure?
• High response?
• PC: patients and controls from same source
population?
• Determinant and outcome similarly measured in all
persons?
• Independent measurement of determinant and
outcome?
• Limited drop-out during follow-up?
• Adequate duration of follow-up?
• Design deals with confounding?
• Analyses adjusted for confounding?
Observational I-18
Using methodological quality in meta-analysis
Use total score of checklist:
• Weighted for total score (pooling)
– RR unweighted:
– RR weighted for quality:
1.38 [1.01-1.87]
1.46 [1.29-1.64]
• Stratified analyses (pooling)
– RR studies low quality:
– RR studies high quality:
1.07 [0.89-1.29]
1.91 [1.56-2.35]
Chlorination of drinking water and cancer
Morris et al. Am J Publ Health 1992;82:955-63
Observational I-21
Using methodological quality in meta-analysis
• Study the influence of specific aspects
– Stratified analyses (subgroup analyses)
– Meta-regression analyses (addressing several
aspects simultaneously)
• Aspects of methodological assessment, e.g.
– Studies with high vs. low response
– Cohort study vs. patient-control study
– Studies with vs. without blinding
Observational I-22
Subgroup analyses
Intermittent sunlight
and melanoma
Saturated fat intake
and breast cancer
OR
RR
2.5
1.6
2.0
1.4
1.5
1.2
1.0
0.8
1.0
0.5
12 case-control
studies
6 cohort
studies
7 studies with
blinding
9 studies
no blinding
Egger et al. BMJ 1998;316:140-4
Observational I-23
Outline
• Observational study designs
• Methodological quality assessment:
– Bias and confounding
– Important aspects of quality assessment
– Using methodological quality in meta-analysis
• Causality
Observational I-24
Association or causal relationship?
• Necessary and sufficient causes
I. factor F  Disease
II. factor X 
factor F  Disease
factor Y 
III. factor P 
factor Q  Disease
factor R 
Observational I-25
Criteria for causality (adapted from Hill)
• Timing: temporal relationship determinant - outcome
1 prospective cohort study
2 patient-control study
3 cross-sectional study
•
•
•
•
•
Strength of the association / dose-response
Consistency of study results
Validity (methodological quality)
Adequate consideration to possible confounding
Plausible explanation for association
Observational I-26
Levels of evidence (example)
• Strong: consistent associations found in ≥ 2 high
quality cohorts
• Moderate: consistent associations found in ≥ 1
high quality cohort and ≥ 1 low quality cohort
• Weak: consistent associations found in ≥ 1 high
quality cohort or in ≥ 3 low quality cohorts
• Inconclusive: association found in < 3 low quality
cohorts
• Inconsistent: inconsistent findings irrespective of
study quality
Observational I-27
But …
• Associations are often weak (RR 0.5-2.0):
precise, but spurious …
• Observational studies are in general quite
heterogeneous
• It is impossible to fully exclude bias and
confounding in observational research
• Possibly greater chance of publication bias
Observational I-28
Spurious results …
Observational I-29
MOOSE
Meta-analysis Of Observational Studies in Epidemiology
• Guideline for reporting of systematic reviews of
observational studies
• CONSORT (RCTs) and QUORUM (Sys reviews)
• Checklist for editors and authors
–
–
–
–
–
Hypothesis
Search strategy
Methodological assessment
Analyses
Discussion and conclusions
Stroup et al. JAMA 2000; 283: 2008-12
Observational I-30
In sum
• Prospective cohort studies (in principal) are
preferred when studying causal relationships
• Observational studies are sensitive for bias
• Caution with conclusions about presence or
strength of causal relationship
• Methodological quality assessment is largely
customized
Observational I-31
Methodological quality assessment of
observational studies
THE END
Observational I-32