Making sense of results
- a workshop for healthcare librarians
Dr Amanda Burls
2nd UK Clinical Librarian Conference
Objectives
•
•
•
•
•
To look at how results can be presented
To understand what a meta-analysis is
To be able to interpret a “blobbogram”
To be able to make sense of tests for “statistical significance”
To explore how uncertainty in results can be summarised and
understand:
• P-values
• Confidence intervals
• To have fun!
Making sense of results
• How are results summarised?
How many of you have attended a
critical appraisal skills workshop?
• What sort of study design were you
appraising?
• What are the key things you remember?
Critical appraisal of any study
design must consider
• Validity
– Can the study (results) be trusted?
• Results
– What are the results and how are they (or can they
be) expressed?
• Relevance
– Do these results apply to the local context?
Warning!
• Everything I say from now onwards assumes
that the results being considered come from an
unbiased study!
How are results summarised?
• Most useful studies compare at least two
alternatives.
• How can the results of such comparisons be
expressed?
Expressing results:
What did the study show?
• Patients with backache:
– 100 randomised to receive a firm mattress
– 100 randomised to receive a medium mattress
• After 3 months:
– 80 get better in the firm mattress group
– 20 get better in the medium mattress group
• How would you summarise this for a friend?
Summarise
• 80 out of 100 (80%) better in firm mattress
group
• 20 out of 100 (20%) better in the medium
mattress group
• 4 times as likely to get better with a firm
mattress
• An extra 60% of people get better with a
firm mattress
How were the results summarised?
• There are two basic ways to
summarise results of studies that
compare two or more groups:
1. Difference (take them away)
2. Ratio (divide)
The blobbogram!
Blobbogram
Line of no difference
between treatments
less
more
Blobbogram - Difference (taking away)
Line of no difference
between treatments
less
0
more
Blobbogram - ratio (dividing)
Line of no difference
between treatments
less
1
more
A randomised placebo-controlled trial
Well conducted RCT – no bias
• Five people with
backache received Potters
• Five people received
placebo
• 4 out of 5 with Potters got
better
• 2 out of 5 with placebo
got better
Number in
treatment arm
5
10
Responders in
treatment arm
4
8
0.8
0.8
Number in
control arm
5
10
Responders in
control arm
2
4
0.4
0.4
Proportion
responding in
treatment arm
Proportion
responding in
control arm
No backache at 3 months
(Results of our Potters tablet
versus placebo trial)
Potters
Placebo
Favours placebo
Favours Potters
No backache at 3 months
(Results of our Potters tablet
versus placebo trial)
Potters
Placebo
Favours placebo
Favours Potters
No backache at 3 months
(Results of our Potters tablet
versus placebo trial)
Potters
Placebo
Favours placebo
Favours Potters
No backache at 3 months
Do you think this study proves Potters works?
Potters
Placebo
Favours placebo
Favours Potters

It could be due to chance!
• What if there had 1000 people in each arm and
800 got better with Potters and only 200 got
better on placebo?
• Would you believe Potters works now?
• So how many people would you want in each
arm to believe the trial?
P-value in a nutshell
The Null Hypothesis
0
Impossible
1
Absolutely
certain
• p = 0.5
quite likely - evens chance - 50:50 - 1 in 2
• p = 0.001
very unlikely - 1 in 1000
• p = 0.01
unlikely - 1 in 100
• p = 0.05
fairly unlikely - 1 in 20 - 5 times in 100
Odds ratio (12b)
100
90
Percentage
80
70
5
60
4
50
3
40
2
30
1
20
10
0
Pre and Post Workshop Scores
MAAG (9b)
100
90
Percentage
80
70
5
60
4
50
3
40
2
30
1
20
10
0
Pre and Post Workshop Scores
Moral:
Any observed difference between two groups, no
matter how small, can be made to be “statistically
significant” - at any level of significance - by
taking a sufficiently large sample.
• Question: How can we express
uncertainty due to chance?
• Answer: the p-value
• But is there a better answer?
Introduction to confidence intervals
• CIs are a way of showing the uncertainty
surrounding our point estimate.
No backache at 3 months
(Results of our Potters tablet
versus placebo trial)
Potters
Placebo
Favours placebo
Favours Potters
No backache at 3 months
(Results of our Potters tablet
versus placebo trial)
Potters
Placebo
Favours placebo
Favours Potters
No backache at 3 months
(Results of our Potters tablet
versus placebo trial)
Potters
Placebo
Favours placebo
Favours Potters
No backache at 3 months
(Results of our Potters tablet
versus placebo trial)
Potters
Placebo
Favours placebo
Favours Potters
Hypothermia vs. control
In severe head injury
Mortality or incapacity (n=158)
Clifton 1993
Clifton 1992
Hirayama 1994
Marion 1997
RR 0.63 (0.46, 0.87)
Total (95%CI)
.1
.2
1
RR
5
10
Hypothermia vs. control
In severe head injury
Mortality or incapacity (n=158)
Clifton 1993
Clifton 1992
Hirayama 1994
Marion 1997
RR 0.63 (0.46, 0.87)
Total (95%CI)
.1
.2
1
RR
5
10
Hypothermia vs. control
In severe head injury
Mortality or incapacity (n=158)
Clifton 1993
Clifton 1992
Hirayama 1994
Marion 1997
RR 0.63 (0.46, 0.87)
Total (95%CI)
.1
.2
1
Favours intervention RR
5
10
Favours control
Hypothermia vs. control
In severe head injury
Mortality or incapacity (n=158)
Clifton 1993
Clifton 1992
Hirayama 1994
Marion 1997
Total (95%CI)
RR 0.63 (0.46, 0.87)
.1
.2
1
Favours intervention RR
5
10
Favours control