Prior Elicitation: Teaching Old Dogs New Tricks
PSI Annual Conference
15th – 17th May 2017
Kimberley Hacquoil, Statistics, Programming and Data Strategy GSK
Maria Costa, Statistics, Programming and Data Strategy GSK
Outline
 Introduction
 Background
 Prior Elicitation eLearning Overview
 eLearning Development Process
 eLearning Demonstration
 Challenges & Learnings
 Conclusion
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
2
Introduction
 Prior elicitation is the process through which expert knowledge about a quantity of interest (e.g.,
treatment effect) is elicited from a subject matter expert and represented through a probability distribution
 In order to participate, experts need to understand basic probability concepts

What is a subjective probability?

How can uncertainty be represented through a probability distribution?

Which subjective judgements should be made and how to assess these?
 For a prior elicitation to be effective it is important to train experts (typically non-statisticians) in these
concepts
 This training component is a challenging and time-consuming task
 In this talk we will share GSK’s experience in developing and implementing an eLearning to train experts
in the fundamental statistical concepts underlying a prior elicitation

Objective: “To create a fit-for-purpose eLearning which would train experts for prior elicitation using Roulette
and Quartile methods”

In collaboration with Tony O’Hagan and Grifo Multimedia
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
3
Background
 Since 2014 GSK have conducted ~ 35
Prior Elicitation sessions
Problem definition (project team)
Limited /conflicting
evidence;
high uncertainty
Decision problem
or statistical model
Pre-elicitation phase (project statistician & physician + facilitator)
Decision to
conduct
elicitation
Frame
problem
Select
experts
Select
method
Prepare
evidence
dossier
Elicitation phase (experts + facilitator)
Carry out
elicitation
 Currently, training happens in a separate
~45 min meeting that occurs prior to the
actual elicitation session
 This can be challenging as experts are
unlikely to be available at the same time
(particularly if in multiple time zones)
leading to multiple training sessions being
required
Training
 Development of eLearning streamlines
training process and allows “just in time”
approach to training
Post-elicitation phase (facilitator)
Documentation
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
4
Prior Elicitation eLearning Overview
The eLearning curriculum is composed of 4 modules...
Module 0:
Introduction
Module 1:
Probabilities
Module 2:
Elicitation
Module 3:
Assessment
• Introduction to eLearning and the concept of expert judgement elicitation
• Overview of eLearning modules
• How subjective probabilities can be used to represent knowledge and uncertainty about
events of interest
• How uncertainty can be described by a probability distribution and how to interpret it
• Methods for expert elicitation and overview of structure of elicitation session
• How the Quartile and Roulette methods work – which subjective judgements are required?
• Group elicitation: the consensus prior and the role of the rational impartial observer
• Assessment: is the expert able to make the necessary judgements using either the Quartile
or Roulette methods and does he/she understand the implications of those judgements?
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
5
eLearning Development Process
Planning and Design
Storyboards
Scope of work
Development
Create Modules
Test Modules
Integration
UAT
GSK Systems
Roll out and Embedding
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
Maintenance
PSI 2017
6
eLearning Development Process
Planning and Design
Storyboards
Scope of work
Development
Create Modules
Test Modules
Integration
UAT
GSK Systems
Roll out and Embedding
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
Maintenance
PSI 2017
7
eLearning Development Process
Planning and Design
Storyboards
Scope of work
Development
Create Modules
Test Modules
Integration
UAT
GSK Systems
Roll out and Embedding
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
Maintenance
PSI 2017
8
eLearning Development Process
Planning and Design
Storyboards
Scope of work
Development
Create Modules
Test Modules
Integration
UAT
Server hosting the learning
module (mylearning)
GSK Systems
MATCH Server
Roll out and Embedding
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
Maintenance
PSI 2017
9
Server hosting the learning
module (mylearning)
eLearning Development Process
Planning and Design
Storyboards
Scope of work
Development
Create Modules
Test Modules
Integration
UAT
GSK Systems
Roll out and Embedding
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
Maintenance
PSI 2017
10
eLearning Development Process
Planning and Design
Storyboards
Scope of work
Development
Create Modules
Test Modules
 Communication of new eLearning to target
audience
Integration
UAT
GSK Systems
 Documentation of eLearning completion to
ensure compliance
 Gather feedback on content and usability
Roll out and Embedding
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
Maintenance
PSI 2017
11
eLearning Development Process
Planning and Design
Storyboards
Scope of work
Server hosting the learning
module (mylearning)
Development
MATCH Server
Create Modules
Test Modules
Integration
UAT
Server hosting the learning
module (mylearning)
MATCH Server
GSK Systems
Roll out and Embedding
Implement
Prior Elicitation: Kimberley Hacquoil & Maria Costa
Maintenance
PSI 2017
12
eLearning Demonstration
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
13
Challenges & Learnings
Challenges
Learnings
 Teaching in eLearning environment
 Project management
 Balancing teaching, examples, assessment
 Regular meetings/interactions
 Balancing length of eLearning and information
 Talking to the right people
overload
 Capture comments in a more structured way
 Ideal vs too hard/long to implement/program
 Coordination issues

Integrating different systems

Understanding what people need and when
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
14
Conclusion
Where we are...
 Currently going through User Acceptance Testing stages of implementation

Selected 2 clinical experts: one with and one without experience in prior elicitation

Also included experienced statisticians (experienced facilitator) as part of UAT
 Plan is to roll out eLearning by end of May/ early June (depending on UAT outcome)
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
15
Acknowledgements
 Tony O’Hagan
 Bill O’Shea, Denise Bird, Kate Foster
 Nigel Dallow, Nicky Best, Tim Montague
 Antonio De Girolamo
 Duncan Richards, Rajnish Saini
 Livio Melfi
 Roberta Memeo
 Ed Morris
 Jeremy Oakley
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
16
Thank You
Back Up
18
Screen Shots
Module 0 - Introduction
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
19
Screen Shots
Module 1 – Subjective Probabilities
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
20
Screen Shots
Module 1 – Subjective Probabilities
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
21
Screen Shots
Module 1 – Representing Knowledge through Probability Distributions
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
22
Screen Shots
Module 1 – Interpreting Probability Distributions
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
23
Screen Shots
Module 1 – Assessing Understanding of Probability Distributions
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
24
Screen Shots
Module 2 – Elicitation Methods
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
25
Screen Shots
Module 2 – Example to Illustrate Elicitation Methods
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
26
Screen Shots
Module 2 – Quartile Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
27
Screen Shots
Module 2 – Quartile Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
28
Screen Shots
Module 2 – Roulette Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
29
Screen Shots
Module 2 – Roulette Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
30
Screen Shots
Module 2 – Structure of an Elicitation Session
Facilitator then leads group discussion: Objective is
to reach a consensus prior, representing the views
of a Rational Impartial Observer
Session starts with individual elicitation: Each
expert is asked to represent their prior knowledge
through a distribution
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
31
Screen Shots
Module 3 – The Quantity of Interest
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
32
Screen Shots
Module 3 – The Evidence Dossier
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
33
Screen Shots
Module 3 – Assessing the Quartile Method using MATCH
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
34
Screen Shots
Module 3 – Assessing the Quartile Method using MATCH
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
35
Screen Shots
Module 3 – Assessing the Roulette Method
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
36
Screen Shots
Module 3 – Feedback
Prior Elicitation: Kimberley Hacquoil & Maria Costa
PSI 2017
37