The DELTA2 Study:
Summary of Methodology
and Results
Quantifying Effect Sizes
William Sones
Tuesday 16th May 2017
PSI 2017 Conference
The Grange Tower Bridge Hotel, London
Introduction
Target difference estimation
• Target difference (effect size) estimation has been
established as a key area of uncertainty in RCTs
• Although research has been performed examining
techniques applied, little guidance exist for practical
implementation
• DELTA - Guidance for a standard superiority two-arm parallel
group trial (Cook at al. 2015)
• Research has been performed to determine techniques applied
(but this may not cover best approaches)
Target difference estimation
• The MRC and NIHR kindly offered funding to:
• Determine scope and design required
• Write and publish guidance
• The DELTA2 study aims to establish guidance
required and publish guidance on target difference
estimation
2
DELTA
The
Study
The DELTA2 Study
• Review existing guidance
• UK and International funding programmes and regulatory
bodies
• Identify key methodological developments and changes
• Systematic review of recent methodological developments
• Consensus process
• Delphi study
• Consensus meetings
• Draft guideline feedback
• Construction and dissemination of final guidelines
Review of existing
guidance
Review of existing guidance
• Funding bodies, regulatory authorities, and advisory
groups reviewed include:
• Arthritis Research UK, BHF, HRA, MRC, RDS, Cancer
Research UK, Wellcome Trust, NIHR, FDA, Health Canada,
European Commission (H2020), NIHR Statistics Group, CIHR,
PCORI, NIH, NHMRC, and AHRQ
• Majority of sites contained little, if any guidance
• Guidance observed generally provided information on
the role of the target difference, and little on
implementation
Key methodological
developments and
changes
Key methodological
developments and changes
• Systematic review was performed upon 22 Journals
between the dates 2011/01/01 and 2016/03/31
• 1395 publications were returned using search terms:
• sample size, target difference, effect size, important difference,
detectable difference, power calculation, value of information, value of
perfect information, value of partial perfect information, value of
sampling information, expected net gain
• Review of title and abstract reduced the number to 85
• Full text investigation showed 41 publications
containing information of relevance
Consensus
process
Consensus process
Delphi study
Consensus
meetings
Stakeholder
engagement
• Multiple online surveys
• 2-day consensus • Session at SCT
(core stakeholders with
meeting of
with potential
interest and expertise in
experts
scope and
the design of RCTs)
representing
content of
• Outcome of each survey
different
guidance
is fedback to
aspects of RCTs
stakeholders, influencing
opinion
• Establish scope • Session at PSI
• Establish consensus
presenting draft
(guidance scope and
guidance
level of contents)
Delphi survey
Delphi survey
• Aimed to determine a wide opinion upon the required
scope of guidance
• Invitation to participate was sent to:
• Funding panel Chairs/Directors, CTU directors, support groups
(e.g. Hubs, NIHR stats group, RDS), statisticians
• 162 invitations were sent. 78 (48%) participated
• The online survey investigated topics including:
• Type of study, methods for specifying target difference,
approaches to consider, complex designs, value of existing
guidance
Delphi results - Participant details
• Your role in RCTs (select all that apply):
Delphi results - Participant details
• Your role in RCTs (select all that apply):
Delphi results - Participant details
• Primary RCT related affiliation:
Delphi results - Participant details
• Primary RCT related affiliation:
Delphi results - Participant details
• Where do you work? If you work across Europe or Internationally
please choose the category in which the majority of your work is
performed.
Delphi results - Participant details
• Where do you work? If you work across Europe or Internationally
please choose the category in which the majority of your work is
performed.
Delphi Results - Types of studies
• Guidance for specifying the target difference for a phase III/IV
(often called definitive or confirmatory) trial needs to be dealt with
separately from early phase, pilot or feasibility trials.
Delphi results - Methods for
specifying target difference
• Should the following approaches be considered a formal method
and covered within the guidance?
Delphi results - Methods for
specifying target difference
• Should the following approaches be considered a formal method
and covered within the guidance?
Delphi results - Special topics
• Degree of coverage required for the following special topics
Delphi results - Special topics
• Degree of coverage required for the following special topics
Delphi results - Special topics
• Additional comments under special topics include:
• Additional topics of interest
• E.g. absolute vs relative risks
• Different perspectives
• E.g. provider and regulatory agency perspectives
Delphi results - Complex designs
• Degree of coverage for trials with more complex designs
Delphi results - Complex designs
• Degree of coverage for trials with more complex designs
Delphi results - Existing guidance
• The existing paper (DELTA; Cook at al. 2015) is useful
Delphi results - Existing guidance
• Criticism of DELTA guidance:
• Guidance style (e.g. more examples/example to provide
greater coverage of trials/different approach-less descriptive
and more practical guidance)
• Increase coverage for different forms of trials
• Improvement of guidance within 2-arm superiority
Delphi results - Further suggestions
• Views covered to greater extent/importance include:
• More complicated designs beyond those addressed exist
• Wider range of guidance/Support for a wider audience
• Will DELTA2 provide greater support for what is wanted rather
than trials performed?
Delphi survey
Summary
Summary
• Majority of respondents had statistical experience,
worked in academia, and were UK based
• Suggestions:
• Phase III/IV should be reviewed separately from early phase,
pilot or feasibility
• Standardised effect size should be covered in guidance
• VoI should be covered (perhaps to a lesser extent)
• Alternative research questions and multiple primary outcomes
deserved greater attention than other topics
• Support for a wider audience
Summary (cont.)
• Suggestions (cont.):
• Adaptive designs and cluster randomised trials deserved
greater attention than within-subject paired and cross-over
designs
• Existing DELTA guidance is useful
Acknowledgments
• The project was jointly funded by the Medical Research
Council (MRC) and National Institute for Health
Research (NIHR)
• Survey was performed using BOS, University of Bristol
Email:
[email protected]