Reliability Engineering and System Safety 87 (2005) 223–232
www.elsevier.com/locate/ress
Validation of reliability computational models using Bayes networks
Sankaran Mahadevan*, Ramesh Rebba
Department of Civil and Environmental Engineering, Vanderbilt University, Box 1831-B, Nashville, TN 37235, USA
Received 22 September 2003; accepted 28 April 2004
Abstract
This paper proposes a methodology based on Bayesian statistics to assess the validity of reliability computational models when full-scale
testing is not possible. Sub-module validation results are used to derive a validation measure for the overall reliability estimate. Bayes
networks are used for the propagation and updating of validation information from the sub-modules to the overall model prediction. The
methodology includes uncertainty in the experimental measurement, and the posterior and prior distributions of the model output are used to
compute a validation metric based on Bayesian hypothesis testing. Validation of a reliability prediction model for an engine blade under
high-cycle fatigue is illustrated using the proposed methodology.
q 2004 Elsevier Ltd. All rights reserved.
Keywords: Bayes network; Bayesian statistics; Model validation; Reliability prediction
1. Introduction
The high cost of full-scale testing makes it uneconomical
for estimating the reliability of large complex systems.
Therefore, model-based computational methods play an
important role in the reliability assessment of such systems.
However, physical, statistical and model uncertainties make
it difficult to have high confidence in model-based reliability
prediction. Hence, there is an important need to validate
model predictions using test data. Even if it is the case that
for large systems, full-scale testing may be infeasible or
uneconomical, it may be possible to obtain test data to
validate smaller modules (subsystem and component level
models) of the overall reliability computational model. This
paper therefore develops a methodology for deriving
validation measures for reliability computational models
based on validation information for the sub-modules.
Validation involves the comparison of model prediction
with experimental data [1]. The key challenges in the
validation process relate to the definition of validation
metrics and design of validation experiments [2]. Validation
metrics are measures of agreement between model
* Corresponding author. Tel.: C1-615-322-3040; fax: C1-615-3223365.
E-mail addresses: [email protected] (S. Mahadevan),
[email protected] (R. Rebba).
0951-8320/$ - see front matter q 2004 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ress.2004.05.001
prediction and experimental observation. The requirements
for such metrics are discussed in later sections. In this paper,
we do not focus on the details of validation experiments but
develop methodology and metrics to compare model-based
prediction with the experimental result, when there is
uncertainty in both. There are several sources of physical,
statistical and model uncertainties that affect the model
prediction, apart from the various sources of error. One way
to describe these uncertainties is through probability
distributions. When the system properties and load conditions are random variables, the output also becomes a
random variable. Similarly, the uncertainty in the validation
experiment data may also be represented by random
variables. Thus validation under uncertainty, within a
probabilistic context, requires quantification of the model
output in terms of a statistical distribution and then
effectively comparing it with experimental data that also
follow a statistical distribution.
Several studies are investigating the fundamental concepts and terminology for validation of large-scale computational models, such as by the Advanced Simulation and
Computing Initiative (ASCI) program of the United States
Department of Energy [3,4], American Institute of Aeronautics and Astronautics [1], American Society of Mechanical Engineers Standards Committee (ASME PTC#60) on
verification and validation of computational solid mechanics, and the Defense Modeling and Simulation Office [5]
224
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
of the US Department of Defense. Model validation has
been pursued with respect to computational fluid dynamics
codes for aerospace systems [6–8], environmental quality
modeling [9,10], etc. An explicit treatment of uncertainty is
missing or treated cursorily in these studies.
The currently used method of ‘graphical validation’ (i.e.
by visually comparing graphs of prediction and observation
either by overlays or by side-by side comparisons) is
inadequate. With a qualitative graphical comparison, one
does not explicitly analyze the numerical error or quantitative uncertainties in the predictions and experimental
results. Such a comparison also gives little indication of how
the agreement varies over space and time. A scatter plot of
experimental values against predicted response with a linear
fit has been used for validation purposes in a simple example
[11]. A statistical inference of model validity, based on
confidence bounds, was made in this reference to evaluate
whether the line thus fit has zero intercept and a slope of
one. Also various statistical tests can be performed on the
residuals for the normality. However, standards pointing out
clear and accepted procedures for model validation under
uncertainty are yet to be developed [12,13]. A method to
validate reliability prediction models based on Bayesian
hypothesis testing has recently been proposed when
reliability testing is feasible [14]. Some researchers [15]
have attempted to assess the reliability of the parameters
used in the large-scale models using Bayesian updating and
available test data. A hierarchical model validation method
was developed to refine or update the computational models
for better prediction.
Model-based reliability analysis generally uses a demand
vs. capacity format, corresponding to a desired performance
criterion. Suppose R is the capacity and S is the demand,
then a performance function g is constructed as
g Z R KS
(1)
Failure is defined to occur when g is less than zero, and the
corresponding failure probability pf is computed as P(g!0),
knowing the statistics of R and S. Monte Carlo simulation
methods as well as analytical first-order and second-order
approximations have been developed for this computation,
based on either actual computational models (e.g. finite
element models) or response surface approximations, for a
large variety of engineering problems [16]. In Eq. (1), the
variable R could be a function of system properties (e.g.
material and geometric properties), while S could be a
function of various loads and system properties. The
variables R and S might not be evaluated directly in a
single step and instead might involve several intermediate,
smaller module calculations. Under physical, statistical and
model uncertainties, the quantities g and pf follow statistical
distributions (prior distributions). The confidence in such
limit state-based model predictions (either pf or g) can be
assessed by conducting actual validation experiments in the
laboratory. Bayesian statistics can be used to update
the distributions of pf and g, based on the experimental
data. By comparing the updated and prior distributions,
inferences are made regarding the validity of the reliability
prediction model.
A metric derived from Bayesian hypothesis testing is
used in this paper to compare prior and posterior
distributions. The metric is referred to as Bayes factor,
which is the ratio of posterior and prior density values at the
predicted value of g corresponding to a given set of input
data. The same input data are used in the model as well as in
the validation experiment, so that the predicted output may
be compared with the measured output.
The Bayes factor metric in this discussion is computed
using only model prediction and experimental data, and thus
is only related to the overall difference between the two
quantities. The difference may arise from many sources:
model form error, numerical errors related to solution
convergence and model resolution, stochastic analysis
errors, and measurement errors in model input and system
output. Each type of error has to be addressed separately for
calibration purposes; the discussion in this paper is limited
to the overall difference between prediction and
observation.
The validation metric proposed in this paper requires
Bayesian updating of the model output distribution using
validation test data. This updating requires complex
analytical integration. A Markov Chain Monte Carlo
simulation (MCMC) technique is implemented as an
efficient alternative for this purpose. Since full-scale testing
may not be feasible in many engineering applications,
experimental data on smaller or intermediate modules must
be used to update the prior reliability prediction distribution.
If the entire system of equations that are involved in the
formulation of the limit state can be represented by a
network showing the relations among the basic random
variables, intermediate quantities, and the overall output,
the updating process would be much easier. The Bayes
network approach is well suited for this purpose.
The Bayes network is a causal network, used as an
inference engine for the calculation of beliefs or probability
of events given the observation/evidence of other events in
the same network. Bayes networks have been used in
artificial intelligence [17], engineering decision strategy
[18], safety assessment of software-based systems [19], and
model-based adaptive control [20,21]. Bayes networks have
also been applied to the risk assessment of water distribution
systems, as an alternative to fault tree analysis [22].
Recently, the Bayes network concept was extended for
structural system reliability reassessment by Mahadevan,
Zhang, and Smith [23] by including multiple failure
sequences and correlated limit states. Both forward and
backward propagation of uncertainty among the components and the system were accomplished.
The attractive feature of the Bayes network is the ability
to update the statistical information for all the nodes, given
an observation for one node. This feature is very valuable in
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
the context of model validation, when experimental
observation may not be available on the final model output
but may be available on one or more intermediate quantities.
This paper therefore proposes the use of Bayes networks to
validate large-scale computational models. Given validation
experimental data on one or more nodes, the predicted
distributions of the intermediate and final quantities are
updated using the MCMC technique, and the prior and
posterior distributions of the model output are compared
through the validation metric. The proposed methodology is
illustrated for the validation of a reliability prediction model
of an engine blade subjected to high cycle fatigue.
225
the Bayes factor is
BZ
PðdatajH0 : x Z x0 Þ
PðdatajH0 : x sx0 Þ
(3)
Two types of testing, or quantities of interest, may be
considered with respect to reliability models. In the first
type, multiple tests are conducted and the results are
reported simply as failure or success with respect to specific
performance criteria. In the second type, a response quantity
such as strain, displacement, etc. is measured, not just
failure or success. The Bayes factor computation is
developed for both cases below.
2.1. Case 1: Multiple pass/fail tests
2. Bayesian model validation
One way to address model validation is through classical
hypothesis testing, a well-developed statistical method of
choosing between two competing models of an experimental outcome. It also allows one to statistically determine the
confidence in the prediction, taking into account the amount
of information (number of observations). However, it has
been argued that this method is difficult to interpret and
sometimes misleading [24]. An alternative method is to use
Bayesian hypothesis testing, which uses assumptions on the
prior distribution for the hypothesis that the model is
incorrect [25]. One important difference between the two
approaches is that, the Bayesian approach focuses on model
acceptance whereas classical hypothesis testing focuses on
model rejection. In the latter, not having enough evidence to
reject a model is not the same as accepting the model. The
differences between classical and Bayesian hypothesis
testing have been discussed in detail in Refs. [26–29].
Consider two models (or hypotheses) Mi and Mj. Their
prior probabilities of acceptance are denoted by P(Mi) and
P(Mj). Using the Bayes theorem, when an event/data y is
observed, the relative posterior probabilities of the two
hypotheses are obtained as:
PðMi jdataÞ
PðdatajMi Þ PðMi Þ
Z
(2)
PðMj jdataÞ
PðdatajMj Þ PðMj Þ
The term in the first set of square brackets on the right
hand side is referred to as ‘Bayes factor’ [25]. If the Bayes
factor is greater than 1.0 then it may be inferred that the data
favors the model Mi more than model Mj. If only a single
model is proposed, the model could be either accepted or
rejected. When an observation is made, the Bayes factor
estimates the ratio of relative likelihoods of the null
hypothesis (i.e. data supports the proposed model) and
alternate hypothesis (i.e. data does not support the proposed
model). For example, let x0 and x be the predicted and actual
values, respectively, of a quantity of interest. The predicted
value x0 may be considered as a point null hypothesis (H0:
xZx0). To estimate the Bayes factor in Eq. (2), we need to
constitute an alternative hypothesis (H1: xsx0). Then
Let x0 and x be the predicted failure probability and true
failure probability, respectively, of an engineering system.
As discussed above, the point null hypothesis is (H0: xZx0),
and the alternative hypothesis is (H1: xsx0).
If n identical and independent experiments are undertaken, and k failures are observed, the probability of
observing the data given that the true probability is equal
to x comes from a binomial distribution as
Pðkjx; nÞ Z Ckn xk ð1 K xÞnKk
(4)
where
Ckn Z
n!
:
ðn K kÞ!k!
Note that the expression in Eq. (4) is also the likelihood
function of x. Under the null hypothesis, the probability
P(datajH0: xZx0) can be estimated by simply substituting x0
in Eq. (4). Assume that there is no prior information about x
under the alternative hypothesis. Therefore, the uniform
distribution between [0, 1] is assumed for f(xjH1), the prior
distribution under the alternative hypothesis. Then the
Bayes factor may be computed as
Bðx0 Þ Z
PðdatajH0 : x Z x0 Þ
PðdatajH1 : x sx0 Þ
Z Ð1
0
Ckn xk0 ð1 K x0 ÞnKk
Ckn xk ð1 K xÞnKk f ðxjH1 Þdx
Z xk0 ð1 K x0 ÞnKk ðn C 1ÞCkn
(5)
When B(x0) is greater than unity, the data is said to favor the
null hypothesis that the failure probability predicted in fact
is true and provides us some confidence in accepting the
model prediction. However, if B(x0)!1.0, it may be
inferred that the data does not support the hypothesis that
xZx0. Thus the Bayes factor may be employed as a metric
to compare the data and prediction which is essentially the
purpose of model validation. The larger the Bayes factor,
the greater the confidence in the model.
226
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
The confidence in the model may be measured by the
posterior probability of the null hypothesis P(H0jdata). This
can be derived from Eq. (2) as
PðH0 jdataÞ Z
Bðx0 ÞPðH0 Þ
PðH1 Þ C Bðx0 ÞPðH0 Þ
(6)
where P(H0) and P(H1) are prior probabilities of the null
and alternate hypotheses, respectively. (Note that P(H1)Z
1KP(H0).) Before conducting experiments, one may
assume P(H0)ZP(H1)Z0.5 in the absence of prior knowledge. In that case, Eq. (6) simplifies as
PðH0 jdataÞ Z
Bðx0 Þ
1 C Bðx0 Þ
(7)
Thus Eqs. (6) and (7) quantify the confidence in the model,
based on validation test data. From Eq. (7), B(x0)Z0
indicates 0% confidence in the model, and B(x0)/N
indicates 100% confidence.
2.2. Case 2: System response value measurement tests
The Bayes factor derived in Eq. (5) is applicable only for
the case with uniform prior and binomial pass/fail data. For
tests conducted in other situations, only a response quantity
may be measured. In such cases, it is valuable to derive a
more general expression for the Bayes factor, by using the
prior and posterior PDFs of the predicted response. The
updating of the prior PDF in that case also requires the
likelihood function, which is a function of the predicted
response. For a discrete distribution, the likelihood function
of the parameter is the probability of observing the data
given the parameter. For a continuous distribution however,
the likelihood function is [30]
LðxÞ Z PfY 2ðy K 3=2; y C 3=2Þjxg
ð yC3=2
Z
yK3=2
f ðYjxÞdY z3f ðyjxÞ
(8)
where 3 is a small positive number, x is the predicted value
with PDF f(x), and y is the observed data. Suppose the
computational model predicted a value x0. Then the
probability of observing the data under the null hypothesis,
PðyjH0 : xZ x0 ÞZ Lðx0 ÞZ 3f ðyjx0 Þ: Similarly, the probability of observing
Ð the data under
Ð the alternate hypothesis
P(yjH1:xsx0) is LðxÞfa ðxÞdx or 3f ðyjxÞfa ðxÞdx; where fa(x)
is the prior density of x under the alternate hypothesis. Since
no information on fa(x) is available, one possibility is to
assume fa(x)Zf(x). Then, using Eq. (2), the Bayes factor is
computed as
Bðx0 Þ Z
Multiplying and dividing by f(x0),
Bðx0 Þ Z
f ðyjx0 Þf ðx0 Þ
Ð
f ðx0 Þ f ðyjxÞf ðxÞdx
(10)
Using the Bayes theorem,
f ðxjyÞ Z Ð
f ðyjxÞf ðxÞ
:
f ðyjxÞf ðxÞdx
Thus Eq. (10) becomes
Bðx0 Þ Z
f ðxjyÞ f ðxÞ xZx0
(11)
Thus, the Bayes factor simply becomes the ratio of posterior
to prior PDFs of the response at the predicted value x0 when
fa(x)Zf(x). If fa(x)sf(x), then the Bayes factor is computed
using Eq. (9) with fa(x) instead of f(x) in the denominator.
Fig. 1 shows the posterior and prior densities of model
prediction x.
If xtrue is the true solution and x is the model output, then
the following equations hold:
xtrue Z x C 3pred
(12a)
xtrue Z y C 3exp
(12b)
where 3pred is the model prediction error and 3exp is the
measurement error. If there is no prediction error,
the observed value will simply be yZxK3exp. From this
relation and Gaussian experimental error assumption, we
obtain f ðyjxÞ wNðx; s23exp Þ: It should be noted that nonGaussian experimental errors may also be considered. The
likelihood function L(x) can be created using f(yjx). If there
is only one observed value of y, then L(x)Zf(yjx), ignoring
the proportionality constant 3, which gets cancelled in
Eq. (9). If multiple data are observed, the likelihood is
constructed as a product of f(yjx) values evaluated at each y.
The predicted response x could be a system-level model
output derived using a set of smaller subsystem models. In
that case, it is possible that the observed response y may not
be available corresponding to x, since full-scale testing may
be too expensive or infeasible. In such situations, experimental observations may be available corresponding to
subsystem model outputs, facilitating Bayesian updating
and model validation at the subsystem level. The Bayes
network approach helps to propagate the validation
PðdatajH0 : x Z x0 Þ
Lðx0 Þ
ZÐ
PðdatajH1 : x sx0 Þ
LðxÞfa ðxÞdx
ZÐ
f ðyjx0 Þ
f ðyjxÞf ðxÞdx
(9)
Fig. 1. Validation metric as a ratio of posterior and prior density values.
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
227
inference from subsystem to system-level, as discussed in
Section 3.
3. Bayes networks in model validation
Model-based reliability analysis for large systems
involves the calculation of various intermediate quantities
during the computational process. Physical and statistical
uncertainties can be propagated through all the steps to
compute the (prior) statistics of the overall performance
function g. The various steps in the computational process
can be represented as nodes in a Bayes network. Once
experimental observation is available on one or more nodes,
the posterior statistics of all the nodes can be computed
using Bayesian updating as discussed in this section. The
posterior and prior statistics can then be used for reliability
model validation.
Bayes networks are directed acyclic graphical representations (DAGs) with nodes to represent the random
variables and arcs to show the conditional dependencies
among the nodes. Each node has a probability density
function associated with it. The arc emanates from a parent
node to a child node. Each child node thus carries a
conditional probability density function, given the value of
the parent node. The entire network can be represented
using a joint probability density function. This section
applies this concept to predict the statistics of the
performance function g defined in Eq. (1), by considering
each level of computation as a component node, and
develops a model validation approach.
The basic random variables and the intermediate
quantities used in the computation of g can be represented
as different nodes of a Bayes network. Since g is also a
random variable, it also can be added as a node. The
network also facilitates the inclusion of new nodes that
represent the observed data and thus the updated densities
can be obtained for all the nodes.
Consider the Bayes network U with seven nodes a to g as
shown in Fig. 2. Thus UZ{a,b,.,g}. Each node is assigned
a probability density function as f(a), f(bja), f(cja), f(djc),
f(ejb,d), f(f) and f(gje,f). In the context of this paper, the
variables or nodes a, b, etc. may correspond to input random
variables as well as quantities computed at each step of the
computational process. The joint PDF of the entire network
Fig. 3. Updated Bayes network with additional data node.
is the product of PDFs of various nodes in the network, i.e.
f ðUÞ Z f ðaÞ !f ðbjaÞ !f ðcjaÞ !f ðdjcÞ !f ðejb; dÞ
!f ðf Þ !f ðgje; f Þ
(13)
Note that for nodes b, c, d, e and g, only the conditional
densities are defined and included in the joint PDF in
Eq. (13). The marginal PDF of b (for example) can be
obtained by the integration of the joint PDF over all the
values of the remaining variables. This integration is
conveniently done using Gibbs sampling as described in
Section 4 later.
The joint probability density function for the network can
be updated using the Bayes theorem when data are
available. Assume that some evidence or test data m for
node b is available. A new node m is now added to the
network (see Fig. 3); this new node is associated with a
conditional density function f(mjb). Then the joint PDF
f(U,m) for this new network is
f ðU; mÞ Z f ðaÞ !f ðbjaÞ !f ðcjaÞ !f ðdjcÞ !f ðejb; dÞ
!f ðf Þ !f ðgje; f Þ !f ðmjbÞ
(14)
With this new joint density, the posterior marginal densities
of each of the nodes can be estimated by integrating the joint
density over the range of values of all other nodes.
With reference to the model validation method developed in Section 2 earlier, assume that test data on node g is
not available, but that data on one or more of the other nodes
are available. The posterior density of g can be calculated
using integration of the joint density in Eq. (14) with respect
to variables a to f. The Bayes factor is then computed as the
ratio of posterior and prior densities at the predicted value of
g, corresponding to the values of the input variables same as
used in the experiment. Thus a system-level Bayes factor is
derived based on experimental data for subsystem-level
modules, and may be used as a metric for system-level
validation. However, note that this is only a partial
inference, since no direct system-level data is used.
4. Implementation of the proposed method
Fig. 2. Bayes network.
It is quite difficult to obtain an exact analytical solution
for the integral of f(U,m) defined in Eq. (14). Therefore, an
228
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
MCMC approach is used in this paper for obtaining
the posterior density of g. In MCMC methods, Monte
Carlo estimates of probability density functions and
expected values of the desired quantities are obtained
using samples generated by a Markov chain whose limiting
distribution is the distribution of interest. Thus one can
generate samples of multiple random variables from a
complicated joint probability density function without
explicitly evaluating or inverting the joint cumulative
density function.
Several schemes such as Metropolis-Hastings algorithm,
Gibbs sampling, etc. are available to carry out MCMC
simulations [31]. This paper utilizes Gibbs sampling due to
its simplicity in the implementation. Let x denote a vector of
k random variables (x1,x2,.,xk), with a joint density
function fX(x). Then let xKi denote a vector of kK1
variables, without the ith variable, and the full conditional
density for the ith component is defined as fX(xijxKi). To
sample quantities from the full conditional density of the ith
variable, the following relationship is used:
fX ðxi jxKi Þ Z Ð
f ðx ; x Þ
ÐX i Ki
/ fX ðxi ; xKi Þdxi
(15)
Gibbs sampling can then be used to sequentially generate
samples from the joint PDF using the full conditional
densities, as below:
Initialize x0 Z fx01 ; x02 ; .; x0k g;
Repeat for iZ1 to N (number of runs):
iK1
iK1
{Generate xi1 wfX ðx1 jxiK1
2 ; x3 ; .; xk Þ
iK1
xi2 wfX ðx2 jxi1 ; xiK1
3 ; .; xk Þ
.
xik wfX ðxk jxi1 ; xi2 ; .; xikK1 Þ
iZ iC 1g
Gibbs sampling has been shown to have geometric
convergence of order N (number of runs) [32]. Exact full
conditional densities may not always be available. In such
cases, the Gibbs sampling procedure is supplemented with
the Metropolis algorithm or a rejection sampling technique
[31]. During each run, the full conditional density function
fX(xijxKi) is constructed by taking the product of terms
containing xi in the joint probability density function. A
rejection sampling technique is then used to obtain a sample
xi from fX(xijxKi). A large number of samples of all the
random variables can be repeatedly generated using these
full conditional density functions. The marginal density
function for any random variable xi can be obtained by
collecting the samples of that particular random variable.
In the context of the Bayes network in Fig. 2, a Gibbs
sample for a node, say b, is generated using the full
conditional density function of b, which is proportional to
the product of all the terms containing b in the joint PDF.
Thus a sample for node b is generated using f(bja)!f(ejb,d).
Similarly, a sample for node c is generated from the function
f(cja)!f(djc), and a sample for node g is generated using
f(gje,f).
Next, in the context of Fig. 3 with the joint probability
density function as in Eq. (14), the samples for node b are
drawn from a different full conditional density function
f(bja)!f(ejb,d)!f(mjb). Since the samples of node b are
now different, they affect the samples of other downstream
nodes, due to the sequential nature of Gibbs sampling. The
posterior density of g is obtained by collecting the g values
from all the runs. This posterior density is then used in the
computation of the validation metric proposed in Section 2.
The implementation of Gibbs sampling is available in the
software WinBUGS [33], which is used in the numerical
example in Section 5 below.
The steps of the proposed methodology for validating a
reliability prediction model can now be summarized as
follows:
1. Break down the reliability computational model into
smaller modules, consistent with feasible validation
experiments.
2. Represent all the modules through a Bayes network, and
compute the prior densities of the quantities computed at
each node.
3. Conduct validation experiments for one or more of the
smaller modules that compute the intermediate
quantities.
4. Use Bayesian updating through the MCMC technique to
compute the posterior densities for the quantities
computed at each node.
5. Use the proposed Bayes factor metric to validate the
reliability prediction model, using the prior and posterior
densities of the model output g.
Note that the methodology, as formulated above, is
suitable in the context of Monte Carlo simulation to estimate
reliability. The samples are used to construct prior and
posterior densities of the performance function g. The
computed failure probability pf Z Pðg! 0Þ is a single
number. If validation is desired in terms of pf, the ratio of
posterior and prior values of pf may be used. This is
particularly necessary if analytical first-order/second-order
reliability methods are used [34]. However, if statistical
uncertainty (i.e. randomness in the distribution parameters
of the basic random variables) is considered, then pf is a
random variable, and the prior and posterior densities of pf
may be compared through the Bayes factor as described
above.
Several additional issues need to be considered when
validation experiments are conducted for multiple nodes.
Some of the intermediate nodes might be functions of the
same basic random variables and hence may be statistically
correlated. The concept developed by Mahadevan, Zhang,
and Smith [23] for including correlated nodes in Bayes
networks may be investigated to consider validation data on
correlated nodes. Also, depending on the influence of any
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
229
node on the performance function g, the validation data may
or may not change the prior density of g significantly. Hence
it is advisable that the nodal sensitivities of g be computed
before validation experiments are conducted, to avoid
wasting resources on insignificant nodes. Probabilistic
sensitivity analysis of the stochastic output with respect to
the various upstream quantities is well established in the
literature [35,36]. For example, in analytical reliability
methods, one definition of probabilistic sensitivity factors
with respect to the random variables xi is
Fig. 4. Bayes network for the limit state of the blade.
vg
vxi
s xi
ai Z rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2
Pn
vg
jZ1 vxj sxj
(16)
where vg/vxi is the gradient of g with respect to xi and sxi is
the standard deviation of xi. As is well known in reliability
analysis, sxi is replaced with the equivalent normal standard
deviation sNxi when xi is non-normal.
5. Numerical example
High cycle fatigue (HCF) can cause severe damage to
the rotor blades of aircraft engines. When the applied
dynamic loading and material properties of the blade are
random variables, the failure probability of a single blade
under HCF can be estimated by a suitable limit statebased reliability prediction model. The validity of such a
model in assessing the reliability of the blade depends on
the selection of a proper performance function and
precise statistical distributions for the individual variables
present in the performance function. The blade is
assumed to have failed when the actual maximum
displacement under dynamic loading exceeds the design
or allowable maximum displacement. The blade may be
modeled as a single degree of freedom oscillator and its
dynamics may be described by a differential equation
consisting of mass, spring, dashpot and with displacement
x [37] as
F sin ut Z m€x C cx_ C kx
(17)
where F is the magnitude of the external harmonic load,
u is the applied load frequency, m is the mass of the
oscillating body, c is the damping constant, and k is the
stiffness of the spring.
The displacement is computed as
xZ
F
1
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2 2
k
½1 K ðu=un Þ C ½2ðc=cc Þðu=un Þ2
(18)
where un is the natural frequency and cc is the critical
damping factor. The Campbell diagram [38] with a slope
and a function of rotor speed is used to depict the resonance
condition. The performance function for the failure of the
blade is thus a function of the natural frequency, damping,
load factor and engine speed (all random variables):
Y
DF
ADDP
LF !MSF
g Z1K
100
DFnominal Dallow
(19)
where LF is load factor, MSF is modal shape factor, DF is
the amplification factor, ADDP is the allowable design
displacement or the nominal allowable displacement
(15 mm in this case), Dallow is the allowable displacement
and Y is percentage of nominal allowable displacement. The
limit state is denoted by the condition gZ0 (see Fig. 4).
The dynamic amplification factor is described by the
equation
1
DF Z pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
(20)
2 2
ð1 K b Þ C ð2hbÞ2
where bZud/un the frequency ratio, and hZc/cc, the
damping ratio. Also, udZ13N/60, where N is the rotary
speed in rpm. The natural frequency is computed from the
Campbell diagram using the equation
un Z unom C slopeðN K 8800Þ
(21)
The statistics of the variables used in the above set of
equations are shown in Table 1.
The series of quantities involved in computing the
performance function g can be modeled as different nodes in
a Bayes network. Given the statistics of each node, the joint
probability density function of the entire network can be
estimated. With the availability of any experimental
validation outcome of any node, the statistical distribution
function associated with all the nodes in the network,
including g, could be updated as described in Section 4.
Different cases of Bayesian updating with validation data on
single and multiple nodes are discussed below.
Table 1
Blade problem: statistics of the variables
Variable
Type
Mean
SD
unom (rpm)
N (rpm)
Slope
h
LF
MSF
Dallow (mm)
Y
ADDP (mm)
Normal
Normal
Normal
Lognormal
Normal
Normal
Normal
Deterministic
Deterministic
2194
8800
0.1
0.002
1
1
15
25
15
105
50
0.005
0.0005
0.25
0.05
0.75
–
–
230
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
Table 2
Validation data on intermediate quantities
No.
Validation test data
Bayes factor for g
1
2
3
4
5
6
7
8
bZ0.6
bZ0.9
unZ2220 rpm
unZ3316 rpm
bZ0.87, unZ2220 rpm
bZ0.6, unZ2220 rpm
bZ0.87, unZ3316 rpm
bZ0.6, unZ3316 rpm
0.00001
5.46
2.11
0.00001
1.6
1.2
0.00001
0.00001
Model prediction unZ2220 rpm, bZ0.9.
Fig. 5. Prior and posterior distributions of g with single node validation
data.
Consider a case when validation test result on the natural
frequency un is available. The observed value can be used to
update the distribution of the overall performance function g
as well as the probability of failure. Suppose the measured
value for the natural frequency of the blade is 3316 rpm
while the predicted value of this quantity, with the same
input data, is 2220 rpm. A data node, with a suitable
likelihood function associated with it, is added to the
network and the posterior probability densities of all the
nodes are computed. Then the prior and posterior densities
of g can be compared as shown in Fig. 5.
Consider the case where validation tests are performed
for multiple nodes. These observations can be used to
update the distribution of the response g similar to the
previous case. Suppose, measured values for two quantities
un and b are 3316 rpm and 0.6, respectively, which are
significantly far from their predicted values (2200 rpm and
0.9, respectively). In this case, the prior and posterior
densities of g are shown in Fig. 6. Various combinations of
validation test information, and the corresponding Bayes
factor values for the overall reliability model prediction, are
presented in Table 2.
In Table 2, the Bayes factor is greater than 1 when the
predicted and observed values are close, as expected.
Comparing the results of data sets 5–8 in Table 2, the natural
frequency un has more influence on g than the frequency
ratio b. Also, in this particular example, the intermediate
quantities DF, ud, b and un have the sensitivity factors (see
Eq. 16) K0.07, K0.0008, 0.62, and 0.78, respectively, at
the most probable failure point, using first-order reliability
analysis. Since b and un have high sensitivity factors, the
observed data for the nodes b and un affect the validation
significantly, as reflected in the Bayes factor values in
Table 2. Thus the Bayes factor metric appears to give
consistent results for reliability model validation. If
validation data on two nodes give two conflicting Bayes
factor inferences, then the sensitivity factors in Eq. (16) may
provide guidance for overall validation influence.
Note that the two quantities b and un above are
correlated, but the correlation is not considered in this
example. It may be possible to account for correlation
effects through the methodology developed by Mahadevan,
Zhang, and Smith [23]; this needs to be investigated in
future work. It is also possible that measurement errors for
the two quantities may be correlated. Such correlation also
needs to be included in future work.
In the example discussed here, the logical and stochastic
relations between various quantities are well established.
But in most cases, the relationship between some of the
quantities may be through implicit computer codes (e.g.
finite element models), and the transfer of data between
nodes may involve multiple quantities. This may increase
the computational effort, but the methodology is essentially
the same.
6. Conclusion
Fig. 6. Prior and posterior distributions of g with validation data on multiple
nodes.
A Bayesian methodology was developed in this paper for
reliability model validation from comparisons between
model prediction and experimental data, both of which have
uncertainty. A large complex computational model may be
decomposed into smaller modules. When full-scale testing
is infeasible, the validity of the overall reliability prediction
model may be partially assessed by propagating sub-module
validation information through a Bayes network approach.
The statistical distribution of the overall performance
function is updated using validation test data for individual
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
sub-modules. The ratio of posterior and prior densities at the
predicted values of the performance function is used to
validate the model prediction.
The Bayes network concept may be extended to the case
of system-level reliability prediction involving multiple
limit states. The methodology needs to be extended to
include correlated quantities in the intermediate nodes and
the effect of validation experiments at these nodes on the
overall metric for the reliability prediction model. Future
work in this direction also needs to include correlations
among experimental measurement errors for the various
quantities being validated.
Acknowledgements
The research reported in this paper was supported by
funds from Sandia National Laboratories, Albuquerque,
NM (Contract No. BG-7732), under the Sandia-NSF Life
Cycle Engineering Program (Project Monitor: Dr Steve
Wojtkiewicz). The support is gratefully acknowledged.
Sandia is a multi-program laboratory operated by Sandia
Corporation, a Lockheed Martin Company, for the US
Department of Energy under Contract DE-AC0494AL85000.
References
[1] American Institute of Aeronautics and Astronautics. Guide for the
verification and validation of computational fluid dynamics simulations. AIAA-G-077-1998. Reston, VA; 1998.
[2] Trucano TG, Easterling RG, Dowding KJ, Paez TL, Urbina A,
Romero VJ, Rutherford BM, Hills RG. Description of the Sandia
validation metrics project Report No. SAND2001-1339. Albuquerque, NM: Sandia National Laboratories; 2001.
[3] Ang JA, Trucano TG, Luginbuhl DR. Confidence in ASCI scientific
simulations Report No. SAND98-1525C. Albuquerque, NM: Sandia
National Laboratories; 1998.
[4] Trucano TG. Aspects of ASCI code verification and validation.
Presented at Quality Fourm 2000. Sponsored by the Department of
Energy, Santa Fe, NM; April 2000.
[5] Defense Modeling and Simulation Office. DoD verification, validation, and accreditation (VVandA) recommended practices guide.
Office of the Director of Defense Research and Engr. www.dmso.mil/
docslib, Alexandria, VA; April 1996.
[6] Aeschliman DP, Oberkampf WL. Experimental methodology for
computational fluid dynamics code validation. AIAA J 1998;
36(5):733–41.
[7] Oberkamf WL, Trucano TG. Verification and validation in computational fluid dynamics. Prog Aerospace Sci 2002;38(3):209–72.
[8] Roache PJ. Recent contributions to verification and validation
methodology. Proceedings of the Fifth World Congress on Computational Mechanics (WCCM V), Vienna, Austria; July 2002.
[9] Beck MB. Water quality modeling: a review of the analysis of
uncertainty. Water Resour Res 1987;23(8):1393–442.
[10] Tsang CF. A broad view of model validation. Proceedings of the
Symposium on Safety Assessment of Radioactive Waste Repositories,
Paris, France; 1989. p. 707–16.
231
[11] Hills RG, Trucano TG. Statistical validation of engineering and
scientific models: background Report No. SAND99-1256. Albuquerque, NM: Sandia National Laboratories; 1999.
[12] Thacker BH, Huyse LJ. Role of non-determinism in validation of
computational mechanics models. Proceedings of the Fifth World
Congress on Computational Mechanics (WCCM V), Vienna, Austria;
July 2002.
[13] Thacker BH. The role of nondeterminism in verification and
validation of computational solid mechanics models. Reliability and
robust design in automotive engineering, SP-1736, SAE International
Paper 2003-01-1353, Proceedings of the SAE 2003 World Congress
and Exposition, Detroit, MI; March 2003.
[14] Zhang R, Mahadevan S. Bayesian methodology for reliability model
acceptance. Reliab Eng Syst Saf 2003;80(1):95–103.
[15] Hasselman TK, Wathugala GW. A hierarchical approach for model
validation and uncertainty quantification. Proceedings of the Fifth
World Congress on Computational Mechanics (WCCM V; Vienna,
Austria); July 2002.
[16] Haldar A, Mahadevan S. Probability, reliability, and statistical
methods in engineering design. New York: Wiley; 2000.
[17] Heckerman D, Geiger D, Chickering D. Learning Bayesian networks:
the combination of knowledge and statistical data. Proceedings of the
10th Conference on Uncertainty in Artificial Intelligence, Seattle,
WA; 1994. p. 293–301
[18] Jensen FV, Jensen FB. Bayesian networks and decision graphs. New
York: Springer; 2001.
[19] Dahll G. Combining disparate sources of information in the safety
assessment of software-based systems. Nucl Eng Des 2000;
195:307–19.
[20] Friis-Hansen A, Friis-Hansen P, Christensen CF. Reliability analysis
of upheaval buckling-updating and cost optimization. Proceedings of
the 8th ASCE Specialty Conference on Probabilistic Mechanics and
Structural Reliability, NotreDame; 2000.
[21] Treml CA, Dolin RM, Shah NB. Using Bayesian belief networks and
an as-built/as-is assessment for quantifying and reducing uncertainty
in prediction of buckling characteristic of spherical shells. Proceedings of the 8th ASCE Specialty Conference on Probabilistic
Mechanics and Structural Reliability, Notre Dame; 2000.
[22] Castillo E, Sarabia JM, Solares C, Gomez P. Uncertainty analysis in
fault trees and Bayesian networks using FORM/SORM methods.
Reliab Eng Syst Saf 1999;65(1):29–40.
[23] Mahadevan S, Zhang R, Smith N. Bayesian networks for system
reliability reassessment. Struct Saf 2001;23:231–51.
[24] Berger JO, Sellke T. Testing a point null hypothesis: the
irreconcilability of P values and evidence. J Am Stat Assoc 1987;
82(397):112–22.
[25] Jeffreys H. Theory of probability, 3rd ed. London: Oxford University
Press; 1961.
[26] Berger JO, Delampady M. Testing precise hypotheses. Stat Sci 1987;
2:317–52.
[27] Birnbaum A. On the foundations of statistical inference (with
discussion). J Am Stat Assoc 1962;57:269–306.
[28] Carver R. The case against statistical significance testing. Harvard
Educ Rev 1978;48:378–99.
[29] Hwang JT, Casella G, Robert C, Wells MT, Farrell RH. Estimation of
accuracy in testing. Ann Stat 1992;20(1):490–509.
[30] Pawitan Y. In all likelihood: statistical modeling and inference using
likelihood. New York: Oxford Science Publications; 2001.
[31] Gilks GR, Richardson S, Spiegelhalter DJ. Markov chain Monte Carlo
in practice. Interdisciplinary statistics. London: Chapman & Hall;
1996.
[32] Roberts GO, Polson NG. On the geometric convergence of Gibbs
sampler. J R Stat Soc, Ser B 1994;56:377–84.
[33] Spiegelhalter DJ, Thomas A, Best NG. WinBUGS version 1.4 user
manual. Cambridge, UK, MRC Biostatistics Unit; 2002.
232
S. Mahadevan, R. Rebba / Reliability Engineering and System Safety 87 (2005) 223–232
[34] Hohenbichler M, Gollwitzer S, Kruse W, Rackwitz R. New light
on first and second-order reliability methods. Struct Saf 1987;
4:267–84.
[35] Hohenbichler M, Rackwitz R. Sensitivity and importance measures in
structural reliability. Civil Eng Syst 1986;3:203–9.
[36] Madsen HO, Krenk S, Lind NC. Methods of structural safety.
Englewood Cliffs, NJ: Prentice Hall; 1986.
[37] Annis C. Modeling high cycle fatigue with Markov chain Monte
Carlo—a new look at an old idea. Proceedings of the 43rd
AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics,
and Materials Conference, AIAA-2002-1587, Denver CO; April
2002.
[38] Prohl MA. A general method for calculating critical speeds of flexible
rotors. ASME J Appl Mech 1945;67:A142–A148.