Two bootstrapping routines for obtaining uncertainty measurement
around the nonparametric distribution obtained in NONMEM VI
1
Paul G. Baverel 1, Radojka M. Savic 2 and Mats O. Karlsson 1
Division of Pharmacokinetics & Drug Therapy, Uppsala University, Sweden
2 INSERM U73 , University Paris Diderot Paris 7, Paris, France
8
N Quantifying uncertainty in parameter estimates is essential to support
N Overall, the trend and the magnitude of the 95% CI derived with the full and
the simplified nonparametric bootstrapping methods (N=100 and N=500)
matched the true 95% CI in all distributional cases and regardless of
individual numbers in original data.
decision making throughout model building process.
N Despite providing enhanced estimates of parameter distribution,
nonparametric algorithms do not yet supply uncertainty metrics.
N The simplified version induced slightly less bias in quantifying uncertainty
(prediction errors of 95% CI width) than the full version. This is expected as
the former methodology derives uncertainty from the original data.
To provide quantitative measures of uncertainty around
nonparametric distribution (NPD) and nonparametric population
parameter estimates
UNDERLYING BIMODAL CLEARANCE DISTRIBUTION
200 IDs
Q1
Q2
50 IDs
Q3
Q4
Q1
1.0
Q3
200 IDs
Q4
1.0
0.05
CL
Cumulative density function
To assess the appropriateness of parameters distributional shape by
means of two resampling-based diagnostic methods aiming at
quantifying uncertainty around NPD
Q2
Prediction errors
0.8
Q1
Q2
Q3
Q4
0.00
0.8
-0.05
0.6
0.6
0.05
True
Parametric
95% CI true
95% CI simplified (N=100)
95% CI full (N=100)
0.4
0.2
True
Parametric
95% CI true
95% CI simplified (N=100)
95% CI full (N=100)
0.4
0.00
CL
0.2
-0.05
Two different permutation methods automated in PsN [1] were developed to
quantify uncertainty around NPD (95% confidence interval) and nonparametric
estimates (SEs and variance-covariance matrix):
o
The full method [2] relies on N bootstraps of the original data and a
re-analysis of both the preceding parametric as well as the
nonparametric step
o The simplified method relies on N bootstrap samples of the
vectors of individual probabilities associated with each unique
support point of the NPD
 Full nonparametric bootstrapping method: 7-step procedure (CPU time: ca 4 hr.)
1
2
BOOTSTRAP*
N times
original data DJ
 Bootstrapped
data B1...BN
PARAMETRIC
ESTIMATION:
B1... BN
 N sets (θ,Ω,σ)
each defined at
<J support points
4
NONPARAMETRIC
ESTIMATION:
DJ given (θ,Ω,σ)
PARTITIONING
NPDN into J
individual probability
densities IPDN
 N matrices MN (JxJ)
of individual
probabilities:
Row entries
Column entries
J individuals
J support points
BOOTSTRAP*
IPDN according to
sample scheme
B1...BN
 NxN matrices
MbootN of
bootstrapped IPD<J
6
RE-ASSEMBLING
bootstrapped IPD<J
7
 N sets NPDnewN
defined at J support
points of NPDN
 From NPDnewN construct
nonparametric 95% CI around NPD
 Derive SEs and correlation
matrix of nonparametric estimates
5
 Simplified nonparametric bootstrapping method: 5-step procedure (ca 2 mn.)
3
1
2
NONPARAMETRIC
ESTIMATION:
Original data DJ
PARTITIONING
NPD into J
individual probability
densities IPDJ
BOOTSTRAP
IPDJ
N times
RE-ASSEMBLING
bootstrapped IPDJ
 Matrice M (JxJ) of
individual probabilities:
N matrices MbootN
of bootstrapped
IPDJ
 A single set of
NPDnewN defined at J
support points of NPD
 NPD defined at J
support points
Row entries
Column entries
J individuals
J support points
20
30
4
Figure 1: Sequential steps of the operating procedure of both the full and simplified
nonparametric bootstrapping methods intended for nonparametric estimation methods.
N The true uncertainty was derived by standard nonparametric bootstrapping
(N=1000) [3] of the true individual parameters and used as reference for
qualitative and quantitative assessment of the uncertainty measurements
derived from both techniques.
40
50
60
10
20
CL distribution
30
50 IDs
40
CL distribution
UNDERLYING HEAVY-TAILED CLEARANCE DISTRIBUTION (200 IDs)
N=100
Q1
Q2 Q3
N=500
Q4
Q1
1.0
Q2 Q3
Prediction errors
N=100
Q4
Q1
1.0
Q2
Q3
Q4
0.05
0.8
0.8
CL
0.00
0.6
0.6
-0.05
True
Parametric
95% CI true
95% CI simplified (N=100)
95% CI full (N=100)
0.4
0.05
True
Parametric
95% CI true
95% CI simplified (N=500)
95% CI full (N=500)
0.4
0.2
0.00
0.2
0.0
CL
5
10
15
20
25
30
35
CL distribution
Simplified
Full
-0.05
0.0
5
10
15
20
25
30
35
N=500
CL distribution
Figure 2. On the left : 95% confidence intervals obtained based on the full and simplified
nonparametric bootstrapping methodologies in case of various underlying distributions of
CL. The true 95% CI around the true parameter distribution is also represented for
comparison, as well as the parametric cumulative density function.
On the right: Prediction errors of the 95% CI width are displayed for each quartile of
parameter distribution, the true uncertainty being taken as reference.
100 Stochastic Simulations followed by Estimations (SSE)
Simplified nonparametric
bootstrap version
True empirical
FOCE
Asymptotic ($COV)
FOCE
True empirical
FOCE-NONP
(N=100)
FOCE-NONP
Θ CL
0.022
0.022
0.021
0.021
ΘV
0.024
0.022
0.024
0.02
Ω CL
0.01
0.009
0.01
0.008
0.007
0.006
0.011
0.008
5
3
 N sets NPDN
defined at J support
points for <J
individuals in B1...BN
0.0
10
Cumulative density function
N Six informative datasets of 50 or 200 individuals were simulated from an
IV bolus PK model in which CL and V conformed to various underlying
distributional shapes (log-normal, bimodal, heavy-tailed). Residual
variability was set to 10% CV.
N Re-estimation was conducted assuming normality under FOCE, and NPDs
were estimated by applying FOCE-NONP method in NONMEM VI.
0.0
Simplified
Full
-0.10
SE
Ω CL,V
ΩV
0.011
0.01
Table 1. SEs of parameter estimates obtained from 100 stochastic simulations followed
by estimations given the true model under FOCE, FOCE-NONP and the analytical
solution under FOCE ($COVARIANCE) in NONMEM VI. The simplified methodology was
applied to each simulated dataset; SEs were computed and average SEs were reported
for
comparison
with
SSE.
N SEs obtained with the simplified methodology matched the ones obtained by SSE.
N Two novel bootstrapping routines intended for nonparametric estimation
methods are proposed. Their evaluation with a simple PK model in the
case of informative sampling design was performed when applying FOCENONP in NONMEM VI but it is easily transposable to other nonparametric
applications.
N These tools can be used for diagnostic purpose to help detecting
misspecifications with respect to the distribution of random effects.
N From the sampling distribution obtained, standard uncertainty metrics,
such as standard errors and correlation matrix can be derived in case
reporting uncertainty is intended.
References:
[1]. Perl-speaks-NONMEM (PsN software): L. Lindbom, M. Karlsson, N. Jonsson. http://psn.sourceforge.net
[2]. Savic RM, Baverel PG, Karlsson MO. A novel bootstrap method for obtaining uncertainty around the
nonparametric distribution. PAGE 18 (2008) Abstract 1390.
[3]. Efron B. Bootstrap methods: another look at the jackknife. Ann Stat 1979; 7:1-26.