CENTRE FOR APPLIED
ECONOMIC RESEARCH
WORKING PAPER
(2006/07)
Expected Value of Information and Decision Making
in HTA
By S. Eckermann and A. Willan
ISSN 13 29 12 70
ISBN 0 7334 2330 2
www.caer.unsw.edu.au
Expected Value of Information and Decision Making in HTA
Simon Eckermann PhD
Flinders University
Andrew R. Willan PhD
Hospital for Sick Children and University of Toronto
Corresponding author: Simon Eckermann
A/Professor in Health Economics
Flinders Centre for Clinical Change and Health Care Research
Repatriation General Hospital Daw Park, Adelaide SA 5041 Australia
[email protected]
ph: +618 8275 1296
fax: +618 8275 1130
e-mail: [email protected]
1
Abstract:
Decision makers within a jurisdiction facing evidence of positive but uncertain
incremental net benefit of a new health care intervention have viable options where no
further evidence is anticipated to:
(1) adopt the new intervention without further evidence;
(2) adopt the new intervention and undertake a trial; or
(3) delay the decision and undertake a trial.
Value of information methods have been shown previously to allow optimal design of
clinical trials in comparing option (2) against option (1), by trading off the expected value
and cost of sample information. However, this previous research has not considered the
effect of cost of reversal on expected value of information in comparing these options.
This paper demonstrates that, where a new intervention is adopted, the expected value of
information is reduced under optimal decision making with costs of reversing decisions.
Further, the paper shows that comparing expected net gain of optimally designed trials
for option (2) vs. (1) conditional on cost of reversal, and (3) vs. (1) conditional on
opportunity cost of delay allow systematic identification of an optimal decision strategy
and trial design.
Keywords: HTA; value of information; decision making; opportunity cost of delay; cost
of reversal; optimal trial design.
2
1. INTRODUCTION
In health technology assessment, decision makers frequently face situations where having
reviewed best current evidence of the costs and effects of a new health care intervention
relative to existing practice, the new intervention has positive but uncertain net benefit at
a decision maker’s threshold value for health effects. This situation is shown in Figure 1.
Assuming no further evidence from other jurisdictions is expected, decision makers in
this situation face viable options to:
(1) adopt the new intervention without further research;
(2) adopt the new intervention and undertake a trial (assuming the decision is
reversible);
(3) delay the decision and undertake a trial.
This paper addresses the question of how value of information methods can be used to
inform decisions between these options.
Previous applications of value of information methods to health technology assessment
have established that trials can be optimally designed by considering the expected value
relative to expected cost of trial (sample) information [1-6]. For evidence of positive but
uncertain incremental net benefit at a point in time, the expected value per patient from
completely reducing uncertainty can be estimated as the expected loss from bad decisions
avoided with perfect information [7-8]. This is referred to as the expected value of perfect
information (EVPI), assuming that in the presence of perfect information this loss is
costlessly avoidable.
3
The expected value of sample information (EVSI) per patient is calculated as the
difference between the EVPI at the time of decision making (time 0) and the expected
posterior EVPI at a time t>0, when additional information from a trial of known size is
expected to update current evidence. The EVSI is positive since at time 0 the EVPI is
expected to reduce when evidence is updated with additional information. Note that this
does not imply that uncertainty or the value of losses associated with uncertainty will
decrease with further information, but that at time 0 the EVPI is expected to reduce with
further information. Figure 2 illustrates EVSI per patient, given a prior density of
positive but uncertain incremental net benefit and the expected posterior density of net
benefit with evidence from a planned trial of a given size (n).
New trial evidence has value in reducing the expected value of losses from decisions
made under uncertainty. An assumption made in calculating EVSI of trial information
based on reduction in expected value of losses avoided (EVPI) is that the avoiding of
losses with perfect information is costless. This assumption is valid where decisions have
been delayed while trials are undertaken. However, where a decision is made at time 0 to
adopt the new intervention and undertake the trial, reversal is not costless. Costs of
reversing decisions are faced, reducing the expected likelihood and value of changing
decisions and hence the EVSI of trials undertaken with adoption. .
Reduction in the value of information with costs of reversing decisions becomes most
obvious in the case where the costs of reversal are high enough that the decision to adopt
4
becomes irreversible. Bernanke [9:86] defines individual investment projects as
economically irreversible if “once constructed they cannot be ‘undone’ or made into a
radically different project without high costs”. Tirole [10:308] clarifies this further as
“...the cost of being freed from the commitment within the period is sufficiently high that
it does not pay to be freed.” Under this definition, decisions become irreversible where
given costs of reversal, it would always be better to live with the decision rather than
reverse it. Hence, if a decision to a adopt a new intervention has costs of reversal high
enough that it is irreversible in this sense, then there is no expected value of sample
information once the new intervention is adopted, and hence EVSI should be 0, as
discussed in Eckermann and Willan [11].
More generally, the expected value of sample information for a trial of given size is less
where the decision has been made to adopt rather than delay, due to costs of reversal.
However, in delaying decisions there is an expected opportunity costs for patients treated
with standard intervention equal to the prior mean INB. Therefore, in considering
optimal trial design where there is positive but uncertain INB, the expected costs and
value of information of planned trials within a jurisdiction are conditional on whether the
new intervention is adopted or not while such trials are undertaken. The expected cost
and value of trial information depends on the simultaneous decision to adopt or delay as:
1. where the new intervention is adopted the expected value of sample information
from a trial is reduced, since costs of reversal are faced; and
5
2. where the decision is delayed there is an additional expected opportunity cost for
patients receiving the standard intervention outside the trial until trial information
updates evidence.
Where there is a trade-off between the value and costs of information, conditional on the
decision context (delay or adopt) and further research may or may not be justified, more
than one comparison is required in identifying an optimal strategy. To find the optimal
strategy and trial design requires identifying optimally sized trials and their expected net
gain in comparison of both:
1. delay the decision and undertake a trial (DT) versus adopt the new intervention
without further research (AN), allowing for opportunity costs of delay; and
2. adopt the new intervention and undertake a trial (AT) versus adopt the new
intervention without further research (AN), allowing for costs of reversal.
The model used to identify optimum strategies and trial design for these comparisons is
outlined in Section 2. The expected net gain of optimal trials in the comparison of DT
versus AN is modelled in Section 3 and the comparison of AT versus AN is modelled in
Section 4. An example illustrating the methods is given in Section 5. The policy
implications and the relative merits of the proposed methods over previous methods are
discussed in Section 6, and Section 7 concludes.
6
2. THE MODEL
Let b0 be the current estimate of mean incremental net benefit (INB), with associated
variance v0, where b0 > 0 and v 0 > 0 . Let ENGD be the expected net gain in the
comparison of DT and AN, i.e. ENGD is the difference between the value of the trial
(sample) information (assuming delay) minus the cost of the trial. Let ENGA be the
expected net gain in the comparison of AT and AN, i.e. ENGA is the difference between
the value of the trial (sample) information (assuming adoption) minus the cost of the trial.
If ENG*D and ENG*A are the corresponding maximum values with respect to trial sample
size, then the following are optimal decision rules.
ENG*D < 0 and ENG *A < 0 ⇒ AN.
ENG*A < ENG *D > 0 ⇒ DT .
ENG*D < ENG *A > 0 ⇒ AT.
In what follows, let
•
k = the incidence rate of the condition in question,
•
a = the accrual rate in to the trial,
•
n = the number of patients per arm recruited to the trial,
•
b̂ = the estimate of INB from the trial data,
•
Nt = the number of patient that can benefit from the new health care intervention
at time t, where the current time is 0,
•
Cf = the fixed cost of doing the trial,
•
Cv = the incremental variable cost per patient of being on trial relative to the same
treatment outside of trial (assumed to be the same by treatment arm), and
7
•
Cr = the cost of reversing a decision to adopt.
(
)
Assuming normality, then b̂ ~ N b 0 , v 0 + 2σ 2 n , where σ 2 is the between-patient
⎛b
nbˆ ⎞
variance of INB. Following the trial, the updated estimate of b is b1 = v1 ⎜ 0 + 2 ⎟
⎜v
⎟
⎝ 0 2σ ⎠
−1
⎛ 1
n ⎞
with variance v1 = ⎜ + 2 ⎟ , where b1 will be normally distributed. Methods for
⎝ v 0 2σ ⎠
determining ENG*A and ENG*D are given in the next two sections.
3. THE VALUE AND OPPORTUNITY COST OF DELAY (DT vs. AN)
Delaying the decision allows collection of further evidence. This additional evidence has
expected value in reducing the opportunity loss of making the net benefit maximising
decision. The expected value of sample information for DT versus AN is not affected by
the cost of reversal. However, there is an expected opportunity cost of delay for patients
not in trial as well as those on trial receiving standard intervention. Delaying a decision to
time t has an expected opportunity cost at time 0 of b0 (the expected INB at time 0) for
these patients. This opportunity cost is additional to the direct costs of trial information.
There is no cost of reversal to consider with DT versus AN and hence the opportunity
loss function, as illustrated in Figure 2, is given by:
L(b) = 0
if b ≥ 0
L(b) = −b if b < 0.
The expected value of sample information at time t is given by:
0
⎪⎧
⎪⎫
EVSI D = N t ⎨ ∫ − b {f 0 (b) − f1 (b)} db ⎬ ,
⎪⎩ −∞
⎪⎭
8
where f i (⋅) is the probability density function for a normal distribution with mean bi and
variance vi, i = 0, 1.
Let t = τ + ta, where ta is the duration of patient accrual and τ is the duration of time
from when accrual finishes and the results are ready for the decision maker.
Therefore N t = N 0 − tk = N 0 − (τ + t a )k = N 0 − ( τ + 2n a ) k . Making this substitution for
Nt and taking the expected value of EVSID with respect to b̂ , the expected gain, denoted
by EGD(n), is a function of n and is given by:
EG D (n) = E bˆ EVSI D
⎧⎪ ⎡ 0
⎤ ⎫⎪
= E bˆ ⎨ N t ⎢ ∫ −b {f 0 (b) − f1 (b)} db ⎥ ⎬
⎥⎦ ⎭⎪
⎩⎪ ⎢⎣ −∞
0
⎡0
⎪⎧
⎪⎫⎤
= { N 0 − (τ + 2n a)k} ⎢ ∫ − bf 0 (b)db − E bˆ ⎨ ∫ − bf1 (b)db ⎬⎥ ,
⎢⎣ −∞
⎪⎩ −∞
⎭⎪⎥⎦
0
where
∫
{
}
(
)
− bf 0 (b)db = {v 0 (2π)} 2 exp − b 02 (2v 0 ) − bi Φ − b0 v02 and Φ (⋅) is the
1
−∞
1
cumulative distribution function for a standard normal random variable.
⎧⎪ 0
⎫⎪
A formulation for E bˆ ⎨ ∫ − bf1 (b)db ⎬ is given in Willan and Pinto [6].
⎩⎪ −∞
⎭⎪
Accounting for the patients from time 0 to time t on standard intervention not in the trial,
in addition to those receiving standard intervention in the trial, the total cost equals
TCD (n) = Cf + 2nCv + ( tk − n ) b0 = Cf + 2nC v + {( τ + 2n a ) k − n} b0 ,
and ENGD(n) = EGD(n) – TCD(n).
9
Given b0, v0, k, a, N0, Cf and Cv, ENGD(n) can be maximized with respect to n > 0, and
ENG*D = ENG D (n *D ) = max n >0 ENG D (n) , where n *D is the optimal sample size for DT
versus AN if ENG*D > 0 . If ENG*D < 0 , the optimal sample size is 0.
The decision rule for DT versus AN can be seen graphically in Figure 3. The TCD(n) line
has intercept Cf + τkb0 and slope 2C v + {( 2k a ) − 1} b0 . If the EGD(n) curve falls below
the TCD(n) line then ENG*D < 0 and the optimal sample size is 0.
4. ACCOUNTING FOR COST OF REVERSAL (AT vs. AN)
In considering the expected value of further trial evidence with adoption of the new
therapy (AT versus AN), the initial question to be addressed is how to allow for cost of
reversal. Expected costs of reversal include costs of reversing information flows (e.g.
public health messages) as well as sunk cost of the intervention such as specific
equipment or training. As Pindyck [12:1111] notes, industry specific investment in fixed
costs are sunk to the extent that, where an investment turns out to be bad, fixed capital
commands commensurately lower resale value.
Where the new intervention is adopted with a trial, decision makers face costs of reversal
in using trial information. In comparing AT and AN, for trial information expected to
arrive at time t, decision makers face costs of reversal Cr in attempting to avoid expected
losses associated with negative INB in the remaining population ( N t ).
10
For values of INB (b) between 0 and − Cr N t , the cost of reversal (Cr) is greater than the
cost of living with negative INB ( Nt ×−b ), and it is optimal not to reverse. Hence, for
INB between − Cr N t and 0 the expected value of information is 0. For INB less
than − Cr N t , the optimal decision is reversal to the standard intervention, since the cost
of reversal Cr, is less than the expected loss of not reversing, − N t b1 . However, while the
expected net benefit of reversal is positive for INB less than − Cr N t , the expected value
of information is reduced in comparison to that of delay by this cost of reversal per
patient. The combined effect of costs of reversal on expected value of information with
adoption rather than delay can be modelled with the opportunity loss function shifted to
the left by Cr N t , as shown in Figure 4.
The opportunity loss function for adopting the new intervention conditional on costs of
reversal is given by:
L(b) = 0
if b ≥ −Cr N t
L(b) = − ( b + Cr N t ) if b < −Cr N t
The expected value of sample information (EVSIA) at time t is given by:
⎧⎪ −Cr N t
⎫⎪
EVSI A = N t ⎨ ∫ − ( b + Cr N t ){f 0 (b) − f1 (b)} db ⎬
⎪⎩ −∞
⎪⎭
⎧⎪ 0
⎫⎪
= N t ⎨ ∫ −b f 0A (b) − f1A (b) db ⎬ ,
⎪⎭
⎩⎪−∞
{
}
(4.1)
where f iA (⋅) is the probability density function for a normal distribution with mean
bi + Cr N t and variance vi, i = 0, 1. Recall that N t = N 0 − tk = N 0 − ( τ + 2n a ) k .
11
Making this substitution and taking the expectation of Equation 4.1 with respect to b̂ , the
expected gain at time t, denoted EGA(n), as a function of n, is given by:
0
⎡0
⎪⎧
⎪⎫⎤
EG A (n) = { N 0 − (τ + 2n a)k} ⎢ ∫ − b f 0A (b) db − E bˆ ⎨ ∫ − b f1A (b) db ⎬⎥ ,
⎢⎣ −∞
⎪⎩−∞
⎭⎪⎥⎦
where
0
∫
−∞
{
}
(
)
− bf 0A (b)db = {v 0 (2π)} 2 exp − ( b 0 + Cr N t ) (2v0 ) − bi Φ − {b0 + Cr N t } v02 . A
1
2
1
⎧⎪ 0
⎫⎪
formulation for E bˆ ⎨ ∫ −b f1A (b) db ⎬ is derived by substituting b0 + Cr N t for b0 in the
⎪⎩−∞
⎭⎪
Appendix of Willan and Pinto [6].
The total cost of the trial equals TCA(n) = Cf + 2nCv + nb0, which is the sum of the
financial cost and the opportunity cost for the n patients in the trial who receive the
standard intervention. Thus ENGA(n) = EGA(n) – TCA(n) and, given b0, v0, k, a, N0, Cr,
Cf and Cv, ENGA(n) can be maximized with respect to n > 0, and
ENG*A = ENG A (n *A ) = max n >0 ENG A (n) , where n*A is the optimal sample size for AT
vs. AN if ENG*A > 0 . If ENG*A < 0 , the optimal sample size is 0. The decision rule for
AT versus AN can be seen graphically in Figure 5.
The TCA(n) line has intercept Cf and slope 2C v + b0 . If the EGA(n) curve falls below the
TCA(n) line for all n, then ENG*A < 0 and the optimal sample size is 0.
12
5. CHOOSING BETWEEN AN, DT AND AT
As illustrated in the previous sections where decision makers face positive but uncertain
INB, the following are optimal decision rules:
ENG*D < 0 and ENG*A < 0 ⇒ AN.
ENG*A < ENG*D > 0 ⇒ DT .
ENG*D < ENG*A > 0 ⇒ AT .
In general, identifying the optimal strategy requires optimum sample size calculations
for:
(1) delay and trial versus adopt with no trial
(2) adopt and trial versus adopt with no trial.
In choosing between potential viable strategies to adopt with a trial (AT), adopting with
no trial (AN) and delaying the decision while a trial is undertaken (DT) there are
characteristic trade-offs between the expected value and costs of information in deciding
whether to research or not and; the expected cost of reversal in adopting the new strategy
and the expected opportunity cost in delaying the decision. It is therefore an empirical
issue as to which strategy is optimal.
5.1 Example –The Early ECV Trial
In a pilot study (Hutton et al.[13]), 232 pregnant women presenting in the breech position
were randomized between early (34 weeks with new intervention) versus late (37 weeks
with standard intervention) external cephalic version (ECV). ECV is an attempt to
manipulate the foetus into a cephalic presentation. Elective caesarean section is accepted
13
practice for breech presentation, and the primary outcome for the trial was a noncaesarean delivery. In the early ECV arm 41 of 116 (35.3%) patients had a noncaesarean delivery and in the late ECV arm the corresponding numbers were 33 of 116
(28.4%). Based on this data, the investigators designed a larger trial of 730 patients per
arm to have an 80% probability of rejecting the null hypothesis of no treatment effect, if
the treatments differed by eight percentage points, using a two-sided Type I error of 0.05.
Suppose, for sake of argument, that society is willing to pay $1000 to achieve a noncaesarean delivery in these patients. This reflects the cost savings and the preference for
a non-caesarean birth. Suppose further that, apart from the possible cost savings from
preventing a caesarean delivery, there is no difference in cost between early and late
ECV. Therefore, b = Δ e1000 , where Δ e is the probability of a non-caesarean delivery
for early ECV minus the probability of a non-caesarean delivery for late ECV. The prior
distribution for b, given the pilot data, is assumed normal with mean
b0 = (41/116 − 33 /116)1000 = 68.97
and variance
⎧ 41/116(1 − 41/116) 33 /116(1 − 33 /116) ⎫
2
v0 = ⎨
+
⎬1000 = 3724.78 .
116
116
⎩
⎭
Using an overall non-caesarean delivery of (41 + 33)/(116 + 116) = 74/232, an estimate
of the between patient variance is σ2 = 74/232(1 − 74/232)10002 = 217, 227 .
Assuming a North American incidence rate of 50,000 per year and a time horizon of
twenty years, then N0 = 1,000,000. Based on a total budget for the planned trial of
$2,836,000, it is assumed that the fixed cost of setting up the trial is Cf = $500,000 and
14
the variable cost per patient is Cv = $1,600. Cv is the incremental variable cost per patient
of being on the trial relative to the same intervention outside the trial. In Claxton [1],
incremental variable cost allowed for differences in cost of treatment by treatment arm.
However, the usual costs of an intervention should not be included in Cv since they are
already implicitly included in the expected net benefit.
Using these values for b0, v0, σ 2 , N0, Cf and Cv, an accrual rate of 500 per year (i.e.
a=k/100) and allowing for a six-month period following accrual to collect and analyse the
data, the optimal sample size for DT versus AN is 0. This is due primarily to the
opportunity cost incurred by the large number of patients who would receive the standard
intervention during the accrual and the six-month data collection/analysis period.
However, even if all patients are accrued to the trial (i.e. a = k), the optimal sample size is
still 0, as illustrated in Figure 6.
The cost lines for a = k/100 and a = k in Figure 6 have the same intercept, but the
coefficient of b0 in the slope term is 199 times larger for the a = k/100 line due to the
larger number of patients on standard therapy for each trial patient. It should be noted
that in Figure 6 the two accrual rates have different expected gain curves. This is because
the duration of accrual will be longer for a = k/100, than a = k, and hence there would be
fewer patients remaining to benefit from the information and a reduction in expected gain
despite the same expected value of information per patient. Therefore an increased
accrual rate both reduces the opportunity cost and increases the value of information from
a trial of a given size. Despite this, even for a = k, the expected gain curve representing
15
expected value of information from trials of size n falls below the cost line for all n. This
is because the six-month design and data collection/analysis period adds a fixed
opportunity cost of τ× k × b0 = 0.5 × 50000 × 68.97 > $1.7 million for the DT option.
Therefore, even for a = k, the data collection/analysis period would have to be reduced to
less than 11 weeks for ENG to be positive and DT be preferred to AN, with any delay
either due to accrual less than the rate of incidence (a<k) or data collection/analysis
resulting in an opportunity cost of more than $9,400 a day. Given early ECV is at 34
weeks and the effect of interest is at birth, the data collection period alone is a minimum
of 8 weeks. Trial design, accrual, data collection and analysis within 11 weeks is
therefore infeasible. AN is clearly preferred to DT for any feasible trial.
While AN is preferred to DT for the case of ECV, we need to compare adopt and trial
(AT) versus adopt no trial (AN), conditional on costs of reversal. Comparing AT vs. AN
with an expected cost of reversal of $2 million, ENGA(n) is maximized at a sample size
of 284 per arm, with expected net gain of $361,442, corresponding to an EVSI of
$1,798,882, a financial cost of $1,408,800 and an opportunity loss of $19,586. This is
illustrated in Figure 7, which is drawn to the same scale as Figure 6.
The expected gain curve for AT vs. AN is lower than that for the equivalently sized trial
for DT vs. AN, due to costs of reversal ($2 million). However, whilst ENGD(n) is
negative for all values of n, ENGA(n), is positive for values of n from 122 to 530. The
intercept for the cost line for AT vs. AN, compared to that for DT vs. AN, is smaller by
$1.7 million. This represents the opportunity cost avoided by AT for patients who
16
receive the early ECV (new intervention) during the data collection/analysis period
(τ×k×b0). The coefficient of b0 in the slope term of the cost line for AT vs. AN, is 199
smaller than that for DT vs. AN, due to the larger number of patients on standard therapy
for each trial patient.
While AT with 284 patients per arm is the preferred option for costs of reversal of $2.0
million, at some higher level of costs of reversal, AN will be preferred to AT. EVSI per
patient decreases with costs of reversal for AT versus AN at any n and hence the position
of the expected gain curve rotates towards the x axis about n=0 as its slope reduces at any
n. For ECV a cost of reversal of $5.0 million is sufficient to lower the expected value of
information curve for AT versus AN below the cost line for all n. Consequently, AN
would therefore be preferred to AT for costs of reversal of $5.0 million or more.
In summary, for the ECV example, AN is preferred to DT for any feasible trial and AT is
preferred to AN for costs of reversal below $5 million, but otherwise AN is the optimal
strategy. For an expected cost of reversal of $2 million the optimal treatment strategy is
to adopt and undertake a trial of 284 patients per arm.
While it is clearly an empirical issue as to what is the optimal strategy and trial design,
some key characteristics are in general terms predictive of which strategy is preferred:
1. AT is characteristically preferred where expected costs of reversal per patient are
small relative to the expected distribution of net benefit below 0;
17
2. AN is characteristically preferred where there is little uncertainty of positive INB
and large costs of trials and reversal;
3. DT is characteristically preferred where there is significant uncertainty in INB
and a long time horizon (increasing the value of information from delay) and;
small estimate of prior mean incremental net benefit, high accrual rate and short
data collection and analysis (reducing the opportunity cost of delay).
6. DISCUSSION
Previous applications of value of information methods to informing trial design in HTA
with evidence of positive but uncertain net benefit of a new intervention have framed the
decision context as being whether to undertake a trial or not based on value and costs of
information, and if so, a trial of what size. An implicit context in identifying optimal trial
design has been to adopt the intervention due to the opportunity cost of patients
remaining on standard intervention with delay.
In this paper costs of reversal have been shown to reduce the expected value of sample
information when adopting a new intervention while undertaking a trial, as the threshold
for reversing decisions shifts below an INB of 0. Hence the expected value, as well as
cost of trial information is conditional on whether the decision is made to either adopt the
new intervention while undertaking a trial or delay the decision while undertaking a trial.
Where expected INB is positive the cost of the trial increases with the option to delay,
given opportunity costs in patients outside of trial, while the value of the trial is reduced
with costs of reversal where the new intervention is adopted. Therefore, an explicit
18
ordering of preferences between options to delay and research, adopt and research and
adopt with no research has been shown to require estimation of expected net gain and
optimal trial design in comparison of both:
1. adopt and research versus adopt with no further research allowing for cost of
reversal; and
2. delay and research versus adopt without research allowing for opportunity costs.
This approach enables a simultaneous solution to preferred strategy and optimal trial
design.
While this paper represents the first framework to explicitly allow modelling of
preferences between viable strategies (DT, AT and AN) and optimal trial design for DT
and AT, previous studies have provided some insight. Claxton, Sculpher and Drummond
[3:711] suggested there were four options faced by reimbursement authorities:
1. adopt based on existing information;
2. adopt but demand further information;
3. reject on the basis of current information;
4. reject and demand further research.
More importantly they noted (op cit: 712) that “..the decision to demand further research
to support adoption (or rejection) must be made simultaneously.” and suggested that the
option to delay may be appropriate where “..reversal of an initial adopting decision may
be difficult or costly or adoption of a technology may make further experimental research
difficult”. They further identified that “…delaying adoption would have opportunity
costs for current patients “ (op cit.:713) and suggested that “if the benefits of delay
19
exceed the costs, then the reimbursement authority should withhold approval of the
technology until the additional evidence is made available.”
The advice provided in Claxton, Sculpher and Drummond [3] reflects the decision
making framework and modelling of costs of reversal (sunk costs) initially proposed in
Claxton [1]. The decision context with positive expected prior incremental net benefits is
assumed as adoption, given “the decision may be revised and switched back from t2 (new
intervention) to t1 if the sample (posterior) mean is less than the critical value (zero)
when the trial is completed” [1:359]. This framework considers the option to delay as
preferable only if sunk costs are large enough relative to opportunity costs of delay,
represented by the “expected benefit to the population of patients not participating in the
trial, for the duration of the trial” [1:359-60].
Specifically, the condition for preference of delay in Claxton [1:359-360] was modelled
as where sunk costs (costs of reversal) multiplied by the probability of mean posterior
INB being less than 0 (implicitly for an ‘optimally’ sized trial of AT vs. AN ignoring
costs of reversal) outweigh opportunity costs in patients treated outside of trial (again
implicitly for an optimally designed trial of AT vs. AN). This is equivalent to DT being
compared with AT where costs of reversal are modelled ex-post rather than within EVSI
and the opportunity costs of DT are identified at the size of trial optimal for AT versus
AN.
20
6.1 Advantages of the proposed approach
The framework in this paper only suggests comparing DT with AT, if DT is preferred to
AN and AT is preferred to AN, conditional on costs of reversal. If this were the case then
comparison of DT versus AT would be at the optimal trial design for each where their
respective expected net gain in comparison with AN are maximised. This approach
improves on the previous approach in:
(i).
allowing optimal trial design for delay, which was not considered previously in
comparing DT vs. AT, despite recognition of the simultaneous nature of the
decision whether to trial and adopt or delay;
(ii). optimal trial design for AT versus AN allowing for costs of reversal, where
previously expected net gain and trial design for AT vs. AN remain the same
regardless of reversal costs;
(iii). comparing AT versus DT at the ‘optimal’ trial design for each, rather than at the
optimal size trial for AT versus AN from (ii);
(iv). allowing optimal decision making in the situation where it is better to live with
a negative incremental net benefit than incur costs of reversal, whereas
previously full costs of reversal were assumed where mean INB was negative.
In general, comparison of ENG for AT versus AN and DT versus AN acts as a circuit
breaker to consider the simultaneous decisions of whether to trial and which intervention
to adopt in comparing viable strategies of AT, AN and DT with optimal trial design.
The proposed approach models ENG of AT relative to AN conditional on costs of
reversal, whereas previously this was ignored. The key insight here is that costs of
21
reversal are faced with in attempting to use additional information with adopt and trial
and hence reduce the EVSI. Therefore, situations will be identified where AT was
previously preferred to AN but should not have been given costs of reversal. Failing to
considering costs of reversal in comparison of AT versus AN clearly biases preferences
in favour of AT. In considering optimal trial design for AT versus AN, allowing for costs
of reversal reduces the slope of the EGA curve at any given n. Therefore, where AT is still
preferred the optimal trial size will reduce from that of the previous framework and the
expected net gain for AT versus AN will be less.
In comparison of AT versus DT, comparing DT at the optimal sample sized trial for AT
vs. AN rather than DT vs. AN also biases in favour of AT. This bias can only be
overcome by considering ENG of DT versus AN and AT versus AN to allow optimal
trial design in comparing adopt and delay. Hence, where DT, AT and AN are viable
alternatives (INB is positive and uncertain) only the proposed framework can reflect
optimal decision making based on optimal trial design. However, while comparing DT at
the optimal size for AT biases in favour of AT, failing to reflect optimal decision making
where it is better to live with negative INB than face costs of reversal biases against AT.
It will therefore be an empirical issue as to the direction of net bias in comparison of AT
and DT. In general the proposed approach clarifies that AT should only be compared
with DT where AT is preferred to AN and DT is preferred to AN.
To illustrate differences in policy advice, we contrast the advice to decision makers in the
ECV example from Section 5 with that which would have been provided under the
22
approach in Claxton [1]. Following Claxton [1:359-360], costs of reversal are ignored in
calculating EVSI and optimal sample size for AT vs. AN. This leads to an optimal
sample size calculation of 296 per arm (rather than 284 where a $2 million costs of
reversal is modelled in EVSI), and the probability of the expected posterior mean INB
being less than 0 is 6.93% at this sample size.
The threshold value for cost of reversal at which delay becomes preferred is estimated as
$83.8 million (bk{(2n/a)+τ}/P(posterior INB<0)), which is above the expected cost of
reversal of $2 million, implying AT is preferred to DT. The advice generally would be
that AT is preferred to AN regardless of costs of reversal, with an optimal sample size of
296 and that AN is preferred to DT for costs of reversal less than $83.8 million.
However, as shown is Section 5, since AN is preferred to DT for any feasible trial, the
appropriate comparison is between AT and AN, where AN is preferred to AT if the cost
of reversal is greater than $5million. A threshold value for cost of reversal for
equivalence between AT and DT should only be considered where there is a positive
expected net gain for DT versus AN, and then only at optimal designs for DT and AT
(conditional on cost of reversal).
This illustration makes it clear that decision making is significantly improved by
modelling the impact of costs of reversal on EVSI in comparisons between AT and AN
and only considering DT vs. AT for respective optimally sized trial where both are
preferred to AN.
23
7. CONCLUSIONS
Decision makers within a jurisdiction facing evidence of positive but uncertain
incremental net benefit have viable options, where no further evidence is anticipated, to
adopt the new intervention with a trial (AT), adopt the new intervention with no trial
(AN), or delay the decision and trial (DT). This paper has shown that, in designing
optimal trials to inform decision making, value of information methods should be used
conditional on decision context as:
1. where the decision is to adopt, costs of reversal reduce the expected value of
sample information (EVSI) from the trial; and
2. where decisions are delayed while undertaking trials there are expected
opportunity costs for patients not in the trial.
The optimal strategy and trial design has been shown to be identifiable by comparing
expected net gain for:
1.
adopt with trial versus adopt without trial allowing for cost of reversal in
estimating EVSI; and
2. delay with research versus adopt without trial, allowing for the expected
opportunity cost of standard intervention.
The approach for comparing net benefit of strategies outlined in this paper improves on
previous approaches in identifying how to allow for costs of reversal in calculating EVSI
for the option to adopt the new intervention while undertaking a trial and in providing a
method to explicitly comparing net benefit of all viable strategies with optimal trial
design for delay and adoption.
24
Using the proposed approach decision makers can be systematically informed of which
strategy and trial design maximises expected net benefit. Empirically, allowing for the
effect of costs of reversal on EVSI results in greater likelihood of AN being the optimal
strategy and smaller trials and expected net gain where AT is preferred, than previous
approaches suggested. Additionally considering the expected net gain of DT strategies
relative to AN allows systematic comparison of DT, AN and AT and optimal trial design
where delay is preferred, unlike previous approaches.
ACKNOWLEDGEMENTS
The authors wish to thank Ms Susan Tomlinson for meticulous proof reading. Of course,
all remaining errors are the authors’ own responsibility. ARW is funded through the
Discovery Grant Program of the Natural Sciences and Engineering Research Council of
Canada (grant number 44868-03); however the opinions expressed are those of the
authors and should not be attributed to any funding agency.
25
References
1. Claxton K. The irrelevance of inference: a decision-making approach to the stochastic
evaluation of health care technologies. Journal of Health Economics 1999; 18: 341364.
2. Claxton K, Thompson K. A dynamic programming approach to efficient clinical trial
design. Journal of Health Economics, 2001; 20: 797-822.
3. Claxton K, Sculpher M, Drummond M. A rational framework for decision making by
the National Institute for Clinical Excellence. Lancet, 2002; 360: 711-715.
4. Claxton K, Ginnelly L, Sculpher M, Palmer S, Philips Z. Pilot study of using value of
information analysis to set priorities in research and development. Health Technology
Assessment, 2004; 8 (31).
5. Claxton K, Ades AE, Lu G. Expected value of sample information in medical
decision modeling. Medical Decision Making, 2004; 24 2: 207-228.
6. Willan AR, Pinto EM. The Value of Information and Optimal Clinical Trial Design.
Statistics in Medicine 2005; 24:1791-1806.
7. Schlaiffer R. Probability and Statistics for Business Decisions. New York:McGraw
Hill; 1958.
8. Raiffa H, Schlaiffer R. Apllied Statistics Decision Theory. New York. Wiley
Interscience; 1967.
9. Bernanke BS. Irreversibility, Uncertainty and Cyclical Investment. Quarterly Journal
of Economics 1983; 98(1): 85-106.
10. Tirole J. The Theory of Industrial Organisation. MIT Press, Cambridge
Massachusetts 1988; 307-308.
11. Eckermann S, Willan AR. The Option Value of Delay in Heath Technology
Assessment. 2006 CAER working paper no.6.
12. Pindyck R.S. Irreversibility, Uncertainty, and Investment. Journal of Economic
Literature 1991; 29(3): 1110-1148.
13. Hutton EK, Kaufman K, Hodnett E, Amankwah K, Hewson SA, McKay D, Szalai JP,
Hannah ME. External cephalic version beginning at 34 weeks’ gestation versus 37
weeks’ gestation: a randomized multicenter trial. American Journal of Obstetrics
and Gynecology, 2003; 189: 245-254.
26
f 0 (b)
0
Figure 1. Positive but uncertain INB (b) at time of decision making
b
E ( f1 (b) | n )
L(b) = − b : b < 0
f 0 (b)
0
L(b) = 0 : b ≥ 0
Figure 2. Determination of the EVSI of trial of size n for DT versus AN
E ( f1 (b) | n ) is exp ected (at time of decision) probability density of INB
at time of new evidence from a trial of size n
b
TCD(n)
$
slope = 2C v + {( 2k a ) − 1} b0
NG D (n)
Cf + τkb0 + ENG*D
Cf + τkb0
n*D
Figure 3. Optimal trial design for DT versus AN
a ( N 0 − τk ) 2k
n
E ( f1 (b) | n )
L(b) = −(b + C r N t ) :
f 0 (b)
b < − Cr N t
−C r 0
Nt
L(b) = 0 : b ≥ − C r N t
Figure 4. Determination of the EVSI of trial of size n for AT versus AN
E ( f1 (b) | n ) is exp ected (at time of decision) probability density of INB
at time of new evidence from a trial of size n
b
$
slope = 2C v + b0
TCA(n)
Cf + ENG*A
NG A (n)
Cf
n*A
Figure 5. Optimal trial design for AT versus AN
a ( N 0 − τk ) 2k
n
$
a = k/100
19
9b
0 )n
10,000,000
+
9,000,000
τk
b
0
+(
2C
v
8,000,000
C
f
+
7,000,000
a=k
D (n
)=
6,000,000
TC
5,000,000
(2C v
kb 0 +
τ
+
C
(n ) = f
4,000,000
+ b 0)n
TC D
3,000,000
a=k
EGD(n)
a = k/100
2,000,000
1,000,000
0
0
100
200
300
400
500
600
700
800
900
1000
n
Figure 6. Expected gain and total cost for DT vs. AN for the ECV example
for two accrual rates: all patients (a = k) and 1% of patients (a = k/100)
$
10,000,000
9,000,000
8,000,000
7,000,000
6,000,000
5,000,000
4,000,000
C
+ (2 v
C
f
=
)
TC A(n
3,000,000
2,000,000
+ b 0)n
EGA(n)
1,000,000
n*A = 284
0
0
100
200
300
400
500
600
700
800
900
1000
n
Figure 7. Expected gain and total cost for AT vs. AN for the ECV example,
with a cost of reversal of $2 million