Reporting quality in preclinical
studies
Emily S Sena, PhD
Centre for Clinical Brain Sciences, University of Edinburgh
@camarades_
CAMARADES: Bringing evidence to translational medicine
CAMARADES
• Collaborative Approach to Meta-Analysis and Review
of Animal Data from Experimental Studies
• Look systematically across the modelling of a range
of conditions
• Data Repository
–
–
–
–
30 Diseases
40 Projects
25,000 studies
from over 400,000 animals
CAMARADES: Bringing evidence to translational medicine
Hypotheses
• In the life sciences there are perverse incentives
(publication, funding, promotion) to produce
positive results with little attention paid to their
validity
• In the use of animal disease models, pressure to
reduce the number of animals (cost, time, ethics,
feasibility) results in studies either being
underpowered or of unknown power
• These factors combine to compromise the utility of
animal models and contribute to translational failure
CAMARADES: Bringing evidence to translational medicine
1,026 interventions in
experimental stroke
In vitro and in vivo - 1026
Tested in vivo - 603
Effective in vivo - 374
Tested in clinical trial - 97
Effective in clinical trial - 1
O’Collins et al, 2006
CAMARADES: Bringing evidence to translational medicine
Translational failure
CAMARADES: Bringing evidence to translational medicine
What happens when pharma tries to
replicate academic findings?
• Bayer, Berlin
• 67 in-house
projects over 4
years
Prinz et al, Nature Reviews
Drug Discovery, 2011
CAMARADES: Bringing evidence to translational medicine
Reproducibility crisis
CAMARADES: Bringing evidence to translational medicine
Experimental data
•
•
Hypothermia: a systematic
search identified 222
experiments in 3353 animals
Better
•
There are huge amounts of
often confusing data
Systematic review can help to
make sense of it
If you select extreme bits of the
evidence you can “prove” either
harm or substantial benefit
Investigating the sources behind
this variation may be helpful in
translation
Worse
•
Van der Worp et al Brain 2007
CAMARADES: Bringing evidence to translational medicine
Potential sources of bias in
experimental studies
• Internal validity
Problem
Addressed by
Selection Bias
Randomisation
Performance Bias
Allocation Concealment
Detection Bias
Blinded outcome assessment
Attrition bias
Reporting drop-outs/ ITT analysis
• Construct validity
• External validity
• Publication bias
CAMARADES: Bringing evidence to translational medicine
Zubin Mehta
Conductor of the LA Symphony (1964-1978) and NY Philharmonic
(1978-1990) credited with saying, “I just don’t think women
should be in an orchestra.”
Blind
Auditions
Introduced
www.curt-rice.com
Goldin & Rouse (2000) American Economic Review 90: 715
Blind auditions explain ca. 30% of the increase in the female proportion of "new
hires“ at major symphony orchestras in the US
CAMARADES: Bringing evidence to translational medicine
Non-random sample
• N=2671
• Most
published
in vivo
research is
at high
risk of bias
CAMARADES: Bringing evidence to translational medicine
Improvement over time…..
CAMARADES: Bringing evidence to translational medicine
Internal Validity:
Lessons from NXY-059
Infarct Volume
– 11 publications, 29 experiments, 408 animals
– Improved outcome by 44% (35-53%)
Efficacy 
•
Randomisation
Blinded conduct
of experiment
Blinded
assessment of
outcome
Macleod et al, 2008
CAMARADES: Bringing evidence to translational medicine
Random sample from PubMed
CAMARADES: Bringing evidence to translational medicine
Good ‘quality’ journals
CAMARADES: Bringing evidence to translational medicine
The ‘best’ institutions – RAE
1173
CAMARADES: Bringing evidence to translational medicine
Reporting of randomisation
across 3 datasets, 2009-10
CAMARADES: Bringing evidence to translational medicine
The umbrella of reporting bias
Not all outcomes and a priori analyses are reported
• Publication bias
– Neutral and negative studies
– Time lag/remain unpublished
– Less likely to be identified
• p-hacking
– Selective analysis
– Selective outcome reporting
CAMARADES: Bringing evidence to translational medicine
0.5
0.5
0.4
0.3
0.3
Precision
Precision
0.4
0.2
0.2
0.1
0.1
0.0
0.0
-150
-0.1
-100
-50
50
0
Effect Size
100
150
-10
-5
0
5
10
15
20
25
Effect Size/Standard Error
CAMARADES: Bringing evidence to translational medicine
Overall efficacy was reduced from;
32% (95% CI 30 to 34%) to 26% (95% CI 24 to 28%)
CAMARADES: Bringing evidence to translational medicine
Publication bias in
experimental stroke
• Trim and Fill suggested 16% of experiments remain
unpublished
• Best estimate of magnitude of problem
– Overstatement of efficacy 31%
• Only 2% publications reported no significant
treatment effects
CAMARADES: Bringing evidence to translational medicine
Different patterns of publication
bias in different fields
Disease
models
Toxicology
model
outcome
observed
corrected
improvement
40%
30%
Less
improvement
harm
0.32
0.56
More harm
Harm
Benefit
CAMARADES: Bringing evidence to translational medicine
Key messages
• In vivo studies which do not report simple measures to avoid
bias give larger estimates of treatment effects
• Most in vivo studies do not report simple measures to reduce
bias
• Publication and selective outcome reporting biases are
important and prevalent
• You cannot assume rigour, even in Journals of “impact”
• You can only find these things out by studying large numbers
of studies
• Any experimental design can be subverted; what’s important
is knowing how to recognise when this has happened!
CAMARADES: Bringing evidence to translational medicine
Today’s exercise
• IICARUS Training
– Reporting of in vivo research published in a major journal
– RCT of mandating authors to submit a completed ARRIVE
checklist
– Outcome: proportion of studies fully compliant with
ARRIVE
– Status: randomisation complete, outcome ascertainment
in progress
https://ecrf1.clinicaltrials.ed.ac.uk/iicarus/
CAMARADES: Bringing evidence to translational medicine
Training in critical appraisal of
publications in biomedicine
https://ecrf1.clinicaltrials.ed.ac.uk/iicarus/Training
CAMARADES: Bringing evidence to translational medicine
CAMARADES: Bringing evidence to translational medicine
Thanks to...........
CAMARADES: Bringing evidence to translational medicine