Brigham Young University
BYU ScholarsArchive
All Theses and Dissertations
2011-07-29
Efficient, Accurate, and Non-Gaussian Error
Propagation Through Nonlinear, Closed-Form,
Analytical System Models
Travis V. Anderson
Brigham Young University - Provo
Follow this and additional works at: http://scholarsarchive.byu.edu/etd
Part of the Mechanical Engineering Commons
BYU ScholarsArchive Citation
Anderson, Travis V., "Efficient, Accurate, and Non-Gaussian Error Propagation Through Nonlinear, Closed-Form, Analytical System
Models" (2011). All Theses and Dissertations. Paper 2675.
This Thesis is brought to you for free and open access by BYU ScholarsArchive. It has been accepted for inclusion in All Theses and Dissertations by an
authorized administrator of BYU ScholarsArchive. For more information, please contact [email protected].
Efficient, Accurate, and Non-Gaussian Statistical Error Propagation
Through Nonlinear, Closed-Form, Analytical System Models
Travis V. Anderson
A thesis submitted to the faculty of
Brigham Young University
in partial fulfillment of the requirements for the degree of
Master of Science
Christopher A. Mattson, Chair
David T. Fullwood
Kenneth W. Chase
Department of Mechanical Engineering
Brigham Young University
Thesis Completed July 2011
c 2011 Travis V. Anderson
Copyright All Rights Reserved
ABSTRACT
Efficient, Accurate, and Non-Gaussian Statistical Error Propagation
Through Nonlinear, Closed-Form, Analytical System Models
Travis V. Anderson
Department of Mechanical Engineering, BYU
Master of Science
Uncertainty analysis is an important part of system design. The formula for error
propagation through a system model that is most-often cited in literature is based on a
first-order Taylor series. This formula makes several important assumptions and has several
important limitations that are often ignored. This thesis explores these assumptions and
addresses two of the major limitations. First, the results obtained from propagating error
through nonlinear systems can be wrong by one or more orders of magnitude, due to the
linearization inherent in a first-order Taylor series. This thesis presents a method for overcoming that inaccuracy that is capable of achieving fourth-order accuracy without significant
additional computational cost. Second, system designers using a Taylor series to propagate
error typically only propagate a mean and variance and ignore all higher-order statistics.
Consequently, a Gaussian output distribution must be assumed, which often does not reflect reality. This thesis presents a proof that nonlinear systems do not produce Gaussian
output distributions, even when inputs are Gaussian. A second-order Taylor series is then
used to propagate both skewness and kurtosis through a system model. This allows the system designer to obtain a fully-described non-Gaussian output distribution. The benefits of
having a fully-described output distribution are demonstrated using the examples of both a
flat rolling metalworking process and the propeller component of a solar-powered unmanned
aerial vehicle.
Keywords: uncertainty analysis, statistical error propagation, system modeling, Taylor series
expansion, variance propagation, skewness propagation, kurtosis propagation
ACKNOWLEDGMENTS
I would like to thank Dr. Christopher Mattson for supporting me in this research.
He has helped open my eyes to the world of research. He has pushed me to explore the field
of engineering design and find solutions to some problems in statistical uncertainty analysis.
He has been a great mentor to me.
I would like to thank Dr. David Fullwood for providing his insights and lending his
mathematical expertise to furthering this work. Without his help, the equations upon which
this research is based could not have been derived.
I would like to thank Dr. Kenneth Chase for his expertise in uncertainty analysis,
particularly in the field of tolerance analysis. His previous work has been a great reference
and source of ideas for this research.
I would like to thank Brad Larson for his work in compositional system models, for
his advice, and for his patience and willingness in answering my incessant questions.
I would like to thank BYU’s Department of Mechanical Engineering for its financial
support of this research.
Lastly, I would like to thank my parents for teaching me the value of hard work, my
Heavenly Father for inspiring me at every step in this work, and my wife, Lacee, for her
never-ending patience, support and love.
TABLE OF CONTENTS
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vi
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
NOMENCLATURE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
x
Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1
Chapter 2 Literature Survey . . . . . . . . . . . .
2.1 Error Propagation via Taylor Series Expansion
2.2 Non-Deterministic Analysis via Brute Force .
2.3 Deterministic Error by Model Composition . .
2.4 Error Budgets . . . . . . . . . . . . . . . . . .
2.5 Univariate Dimension Reduction . . . . . . . .
2.6 Interval Analysis Methods . . . . . . . . . . .
2.7 Bayesian Inference . . . . . . . . . . . . . . .
2.8 Response Surface Methodologies . . . . . . . .
2.9 Anti-Optimization Techniques . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
3
3
4
5
5
6
6
6
6
7
Chapter 3 Propagation of Variance . . . . . . . . . . . . . . . . . . . . . . . . .
3.1 Fundamental Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.1.1 Mathematical Expectation . . . . . . . . . . . . . . . . . . . . . . . .
3.1.2 Central Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Propagation of Variance Using First Order Taylor Series . . . . . . . . . . .
3.2.1 First-Order Formula Derivation . . . . . . . . . . . . . . . . . . . . .
3.2.2 First-Order Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Propagation of Variance Using Higher-Order Taylor Series . . . . . . . . . .
3.3.1 Second-Order Formula Derivation . . . . . . . . . . . . . . . . . . . .
3.3.2 Second-Order Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Taylor Series Error Propagation Assumptions and Limitations . . . . . . . .
3.5 Higher-Order Accuracy With First- or Second-Order Cost . . . . . . . . . .
3.5.1 Propagation of Variance Using Higher-Order Terms . . . . . . . . . .
3.5.2 Reducing Truncation Error . . . . . . . . . . . . . . . . . . . . . . . .
3.5.3 Correction Factors for Other Nonlinear Functions . . . . . . . . . . .
3.5.4 Model Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Example: Compositional System Model of a Dual Propeller, Three Degreeof-Freedom Helicopter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.7 Example: Kinematic Model of a Mechanism that Simulates Flapping Flight
Wing Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.8 Comments on Variance Propagation . . . . . . . . . . . . . . . . . . . . . . .
9
9
9
10
11
11
13
14
15
16
17
18
18
20
22
24
Chapter 4 Higher-Order Statistics . . . . . . . . . . . . . . . . . . . . . . . . .
33
iv
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
24
28
30
4.1
4.2
4.3
4.4
4.5
4.6
Motivation for Propagating Higher-Order Statistics . . . . . . . . . . . . . .
Propagation of Skewness . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.1 Definition of Skewness . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.2 Skewness Propagation Formula Derivation . . . . . . . . . . . . . . .
4.2.3 Skewness Propagation Assumptions and Limitations . . . . . . . . . .
4.2.4 Proof: Nonlinear Functions Produce Non-Gaussian Output Distributions
Propagation of Kurtosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 Definition of Kurtosis . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.2 Definition of Excess Kurtosis . . . . . . . . . . . . . . . . . . . . . . .
4.3.3 Kurtosis Propagation Formula Derivation . . . . . . . . . . . . . . . .
Example: Solar-Powered Unmanned Aerial Vehicle Propeller Thrust . . . . .
4.4.1 The Analytical Model of Thrust . . . . . . . . . . . . . . . . . . . . .
4.4.2 Statistical Component Model Output Prediction . . . . . . . . . . . .
Example: Flat Rolling Metalworking Process . . . . . . . . . . . . . . . . . .
4.5.1 The Analytical Model . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.2 Statistical System Model Output Prediction . . . . . . . . . . . . . .
4.5.3 Ramifications of Neglecting Higher-Order Statistics . . . . . . . . . .
Comments on Propagation of Higher-Order Statistics . . . . . . . . . . . . .
33
35
35
36
37
38
39
39
39
40
41
42
44
45
47
49
50
51
Chapter 5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
53
REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
55
Appendix AStep-by-Step Solution to Dual-Propeller, Three Degree-of-Freedom
Helicopter Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
A.1 Variance Propagation Through Component 1 . . . . . . . . . . . . . . . . . . 60
A.2 Variance Propagation Through Component 2 . . . . . . . . . . . . . . . . . . 60
A.3 Variance Propagation Through Component 3 . . . . . . . . . . . . . . . . . . 62
A.4 Variance Propagation Through System-Level Compositional Model . . . . . 63
v
LIST OF TABLES
3.1
3.2
3.3
3.4
Partial Derivatives Required for Taylor Series Variance Propagation . . .
Truncation Error Correction Factors for Common Nonlinear Functions .
Design Parameters for Dual-Propeller, 3-DOF Helicopter . . . . . . . . .
Mean Values and Standard Deviations of the 16 Model Input Parameters
.
.
.
.
.
.
.
.
19
23
26
29
4.1
4.2
4.3
4.4
4.5
4.6
Comparison of Positive and Negative Skew . . . . . . . . . . . . . . . . .
Statistical Properties of the Angular Velocity Distribution of a Propeller
Central Moments of the Distribution of Angular Velocity, ω . . . . . . . .
Friction Conditions in Mechanical Devices and Metalworking Processes .
Statistical Properties of the Coefficient of Friction Distribution . . . . . .
Central Moments of the Distribution of the Coefficient of Friction, µf . . .
.
.
.
.
.
.
.
.
.
.
.
.
35
43
44
47
48
49
A.1 Design Parameters for Dual-Propeller, 3-DOF Helicopter . . . . . . . . . . .
59
vi
vii
LIST OF FIGURES
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
3.10
3.11
3.12
3.13
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
Relative Error in Output Variance Using a First-Order Taylor Series Expansion for the Function y = 1000 sin(x) . . . . . . . . . . . . . . . . . . . . . .
Relative Error in Estimate of Variance Propagation Using a Second-Order
Taylor Series Expansion for the Function y = 1000 sin(x) . . . . . . . . . . .
Relative Error in Variance Propagation Using Taylor Series Approximations
Relative Computational Cost of Taylor Series Approximations, as Determined
by MATLAB Execution Time . . . . . . . . . . . . . . . . . . . . . . . . . .
Relationships Between the Second-Order Bias and both Input Standard Deviation σx and Input Variance σx2 for a sin Function . . . . . . . . . . . . . .
Relative Error in Estimations of Variance Propagation . . . . . . . . . . . .
Mean Relative Error in Variance Estimations . . . . . . . . . . . . . . . . . .
Dual-Propeller, Three Degree-of-Freedom Helicopter Used in the Design of
Unmanned Aerial Vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Relative Error in Estimations of Variance Propagation . . . . . . . . . . . .
Mean Relative Errors that Indicate the Compositional Model with a Correction Factor Predicts Output Variance with About Half the Error as the
First-Order Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Mechanism Used by the BYU Flapping Flight Research Team to Simulate
Three-Degree-of-Freedom Motion of a Flapping Wing . . . . . . . . . . . . .
Computational Time to Predict Output Distributions Using Various Error
Propagation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Relative Error in Predictions of Output Variance Obtained from Various Orders of a Taylor Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Predicted Gaussian Output Distribution Obtained from Propagating Mean
and Variance Only (a) Compared with Actual System Output (b) . . . . . .
Predicted Non-Gaussian Output Distribution Obtained from Propagating Mean,
Variance, Skewness, and Kurtosis (a) Compared with Actual System Output
(b) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Examples of Negative (Left) and Positive (Right) Skewness . . . . . . . . . .
Excess Kurtosis of Various Common Statistical Distributions . . . . . . . . .
Power and Propulsion System of a Solar-Powered Unmanned Aerial Vehicle
(UAV) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Compositional System Model of a Solar-Powered Unmanned Aerial Vehicle
(UAV) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Distribution of the Angular Velocities of a Propeller in a Solar-Powered UAV
Predicted Gaussian Output Distribution of Thrust Obtained by Propagating
a Mean and Variance Only (a) Compared with Actual Thrust Measurements
(b) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Predicted Non-Gaussian Output Distribtuion of Thrust Obtained by Propagating a Mean, Variance, Skewness, and Kurtosis (a) Compared with Actual
Thrust Measurements (b) . . . . . . . . . . . . . . . . . . . . . . . . . . . .
viii
14
17
19
20
21
22
22
25
27
27
28
29
30
34
34
36
40
42
43
44
45
46
4.10 Flat Rolling Manufacturing Process Whereby Plates or Sheets of Metal are
Made—Material is Drawn Between Two Rollers, which Reduces the Material’s
Thickness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.11 Distribution of the Coefficient of Friction in a Flat Rolling Metalworking Process
4.12 Predicted Gaussian Output Distribution Obtained from Propagating a Mean
and Variance Only (a) Compared with Actual System Output (b) . . . . . .
4.13 Predicted Non-Gaussian Output Distribution Obtained from Propagating a
Mean, Variance, Skewness, and Kurtosis (a) Compared with Actual System
Output (b) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
ix
46
48
50
50
NOMENCLATURE
N
n
x
xi
xj
x̄
σx
σx2
σx2i xj
γ1
γ2
β2
κi
µk
µk,i
∂1
∂2
y
ȳ
σy
σy2
e
Ck
E[ ]
Number of data points in a distribution
Number of inputs to a system model
An input distribution to a system model
The i-th system input distribution
The j-th element of a system input distribution
The mean value of input x
Standard deviation in the input distribution x
Variance in the input distribution x
Covariance in between input distributions xi and Xj
Skewness in a distribution
Kurtosis in a distribution
Excess kurtosis in a distribution
The i-th cumulant of a distribution
The k-th central moment of a distribution
The k-th central moment of the i-th input distribution
First partial derivative of the system model, evaluated at the mean
Second partial derivative of the system model, evaluated at the mean
System output
Mean system output value
Standard deviation in output distribution y
Variance in output distribution y
Relative error in a model’s prediction compared with actual output
Correction factor
The k-th component of a compositional system model
indicates the expectation operator performed on [ ]
Example: Dual Propeller, Three Degree-of-Freedom Helicopter
θ
Helicopter pitch angle (rad)
θ̈
Angular acceleration in pitch (rad/s2 )
m1
Mass of helicopter motors, propellers, and circuitry (kg)
m2
Mass of helicopter counterbalance weight (kg)
Lk
Length of helicopter segment k (m)
Jy
Mass moment of inertia about the pitch axis of rotation (kg-m2 )
uL
Motor command given to left motor
uR
Motor command given to right motor
km
Motor calibration constant
Example: Kinematic Model of Flapping Flight Wing Motion
t
Simulation time (s)
ω
Flapping frequency (Hz)
φ
Positional angle (deg)
θ
Elevation angle (deg)
α
Feathering or attack angle (deg)
x
Axx
Bxx
Fourier series coefficients for cos terms
Fourier series coefficients for sin terms
Example: Solar-Powered Unmanned Aerial Vehicle Propeller Thrust
T
Thrust generated by the propeller (N)
Ct
Unitless calibration coefficient of thrust
ρ
Density of air (kg/m3 )
ω
Propeller angular velocity (rad/s)
D
Propeller Diameter (m)
Example: Flat Rolling Metalworking Process
∆H
The change in material thickness with each pass (m)
∆Hmax The maximum change in material thickness attainable in a single pass (m)
µf
Coefficient of friction
R
Roller radius (m)
xi
CHAPTER 1.
INTRODUCTION
In the system design process, designers frequently experience significant uncertainty
in predicting whether a proposed design will meet the design objectives. Design decisions
often cannot be validated until a physical prototype is built and tested. If the proposed
design proves to be faulty, it can lead to a very costly design iterations. The inability of the
system designer to verify design decisions early in the design process and to determine if a
proposed design will accomplish design objectives is a limiting obstacle in system design [1].
Consequently, system behavioral modeling is an important part of system design. If
a system behavioral model could be obtained and its accuracy quantified, it would enable
the designer to verify design decisions early in the design process. This would greatly reduce
the risk of a creating a failed system design.
Determining system model accuracy is difficult, especially when the system is still
theoretical and actual system behavior is not known [2]. In many situations, a statistical
distribution of possible outputs is more meaningful than a simple max/min error bound [3].
If the system input distributions are known, such an output distribution can be obtained
for closed-form differentiable models using a Taylor series expansion to propagate input
distributions through a system model. This is typically accomplished with the first-order
Taylor series approximation, but there are two major limitations of using this method.
First, the first-order Taylor series takes a derivative-based weighted sum of independent input variances to estimate the variance in the output distribution. Since all higherorder terms in the series are neglected, this predicted output variance may be wrong by one
or more orders of magnitude for nonlinear functions [4]. While higher-order terms clearly
improve the accuracy of the approximation, they also require greater computational cost.
This thesis shows that the truncation error in lower-order approximations can be predicted
1
and accounted for. Consequently, fourth-order accuracy in error propagation can be obtained
without significant additional computational cost.
Second, system designers typically use the Taylor series to only propagate a mean and
variance. All higher-order statistics are neglected. As all non-Gaussian distributions cannot
be fully described with a mean and variance alone, the output distribution is typically
assumed to be Gaussian. This often is an erroneous and costly assumption. This thesis
proves that nonlinear functions do not produce Gaussian output distributions, even when
inputs are Gaussian. A second-order Taylor series expansion can be used to propagate higherorder statistical properties through a system model without incurring significant additional
computational cost.
This thesis explores error propagation through nonlinear system models using a Taylor
series expansion. Chapter 2 provides an overview of nine common methods of uncertainty
analysis. Chapter 3 shows how high accuracy in variance propagation can be obtained with
low-order computational cost. Chapter 4 presents a method for propagating the higher-order
statistical properties skewness and kurtosis through a closed-form system model. Chapter 5
concludes this thesis and suggests areas for future research.
2
CHAPTER 2.
LITERATURE SURVEY
Many methods are currently in use or being researched that quantify the accuracy
of a system model by propagating error through the system. These methods include the
following:
• Error propagation via Taylor series expansion
• Non-deterministic analysis via brute force (Monte Carlo, Latin hypercube, etc.)
• Deterministic model composition
• Error budgets
• Univariate dimension reduction
• Interval analysis
• Bayesian inference
• Response surface methodologies
• Anti-optimizations (sub-optimizations)
While this thesis focuses on error propagation via Taylor series expansion, the other
uncertainly analysis techniques listed above are also briefly described in the remainder of
this chapter.
2.1
Error Propagation via Taylor Series Expansion
A Taylor series can be used to propagate variation in system inputs through a system
model to produce an estimate of the variation in system outputs. This derivatives-based
method can be very simple and fast, and is the method most-often cited in literature [5, 6].
3
Nevertheless, it may produce results that are inaccurate by one or more orders of magnitude [4], particularly for nonlinear systems. This inaccuracy is a result of linearizing the
system model, as the higher-order derivatives are truncated from the Taylor series. This
linearization also means accuracy decreases as input variances increase.
Typically, this method is used only to propagate variance. Data is typically assumed
to be independent and Gaussian, as variable interactions and higher-order statistics are
ignored.
2.2
Non-Deterministic Analysis via Brute Force
Uncertainty analysis methods are usually classified as either deterministic or non-
deterministic, and non-deterministic analysis methods are further grouped into two main
categories: 1) reliability-based design methods [7, 8, 9], and 2) robust-design-based methods [10, 11, 12, 13, 14, 15].
A deterministic model always produces the same output values from the same input
values, but a non-deterministic model does not. Consequently, a non-deterministic model,
executed repeatedly with the same input values, can produce a statistical output distribution
instead of a single nominal value.
However, due to the complexities of non-deterministic modeling, it is more common to
represent uncertainty with probabilistic methods and propagate these uncertainties through a
deterministic model [16]. This approach can also result in a statistical output distribution by
executing the model repeatedly. This brute-force approach to non-deterministic uncertainty
analysis commonly uses Monte Carlo, quasi Monte Carlo [17, 18], Monte Carlo hybrid [19],
Latin hypercube, Latin supercube [20], or some other sampling technique.
This technique does not need to assume a Gaussian output distribution. Consequently, an estimate of the fully-described output distribution can be obtained. However,
this approach comes at great computational cost, which only grows exponentially with an
increase in the number of statistically distributed inputs. Furthermore, the entire simulation must be executed again each time the model or any input value changes. This can be
prohibitive in an iterative design process.
4
2.3
Deterministic Error by Model Composition
Current research is being done [21] to deterministically propagate error through a
compositional system model to obtain max/min error bounds. In order to do this, the
system model error must be included in the model itself. Augmenting the compositional
system model with component error models allows the component error interactions and
propagations to be evaluated together with the model. System model accuracy can be
determined by comparing the results of the regular system model with those from the erroraugmented system model [2]. Monte Carlo simulations or optimization routines can be
executed to find the max/min error bounds of the system model.
With this approach to error propagation, the accuracy of even complex systems and
interactions can be obtained. Errors do not need to be independent. Component models do
not have to be mathematical or closed-form functions, but rather can include dynamic models, nonlinear models, non-differentiable models, lookup-tables, software programs, digital
logic, CAD models, finite-element models, and any other model capable of mapping inputs
to outputs.
Deterministic error analysis requires known component error models and relationships. In some situations, the max/min error bounds obtained from this method are so large
that they are not helpful. In many practical applications, a statistical error distribution is
more useful than a max/min error envelop [3].
2.4
Error Budgets
The method of error budgets involves propagating the error of each component
through the system separately, and resolving each component’s error to the contribution
it makes on the total system error [22, 23]. This is done by perturbing one error source at a
time and observing the effect this has on the total system error. Consequently, this method
requires either that component errors be independent or that a separate model showing
component error interactions be developed, which typically is not done [24]. If the error
sources are not actually independent, this method will not necessarily describe the full range
of possible model error.
5
2.5
Univariate Dimension Reduction
Univariate dimension reduction methods transform data from a high-dimensional
space to a lower-dimensional space. In some situations, data analysis may even be more
accurate in the reduced space than in the original space [25].
2.6
Interval Analysis Methods
Interval analysis methods bound rounding and measurement errors in mathematical
computation. Arithmetic can then be performed using intervals instead of a single nominal
value [26]. These techniques can be used to propagate error envelopes, or intervals, through a
system model. These methods, however, are typically limited to basic arithmetic operations.
Currently there are many software languages, libraries, compilers, and data types
that implement interval arithmetic. These include XSC, Profil/BIAS, Boost, Gaol, Frink,
and a MATLAB extension named Intlab. There is also a working group currently developing
an IEEE Interval Standard (P1788).
2.7
Bayesian Inference
Bayesian inference is a method of statistical inference whereby the probability that a
hypothesis is true is inferred based on both observed evidence and the prior probability that
the hypothesis was true [27]. It combines common-sense knowledge with observational evidence in an attempt to eliminate needless complexity in a model by declaring only meaningful
relationships [28] and disregarding the influences of all other variables on system outputs.
2.8
Response Surface Methodologies
Response surface methodology is commonly used in design of experiments. It is a
modeling technique whereby an n-dimensional response surface showing the relationship
between n-input variables is created, typically from using empirical data [29, 30]. A few
common experimental design setups include full factorial, partial factorial, central composite,
Plackett-Burman, Box-Behnken, and others.
6
2.9
Anti-Optimization Techniques
Anti-optimization techniques allow the designer to find the worst-case scenario for a
given problem. This results in a two-level optimization problem, where the uncertainty is
anti-optimized on the lower level and the overall design is optimized on a higher level [31].
7
8
CHAPTER 3.
PROPAGATION OF VARIANCE
This chapter demonstrates how a Taylor series is used to propagate variance by estimating a distribution’s central moments. The mathematical expectation operator and
central moments are first discussed. The derivations of the first- and second-order Taylor
series error propagation formulas are then presented along with an in-depth discussion of
their accuracies. The assumptions and limitations of using a Taylor series to propagate error
through a system model are then summarized.
This chapter then demonstrates how fourth-order accuracy can be obtained in error
propagation with only first- or second-order computational cost. This is accomplished by
applying a correction factor to the lower-order estimate, which accounts for the higher-order
truncation error in the Taylor series. These correction factors were determined empirically
for several common nonlinear engineering functions.
Lastly, this chapter demonstrates the effectiveness of this method using the model of
a dual-propeller, three degree-of-freedom helicopter, and the kinematic system model of a
flapping wing.
3.1
Fundamental Concepts
Before continuing, it is essential to understand two statistical concepts that are funda-
mental to this research. These are the mathematical expectation operator and the statistical
central moment property, both of which are described in this section.
3.1.1
Mathematical Expectation
The mathematical expectation operator, E[ ] denotes the calculation of a weighted
average of all possible values, as shown in Eq. (3.1). The weights correspond to the probability of a particular value. When all possible values have an equal probability, the expected
9
value is equal to the arithmetic mean, which is also equal to the limit of the sample mean
as the sample size increases to infinity.
E[X] = x1 p1 + x2 p2 + ... + xn pn
(3.1)
where X is some population, xi are the possible values, pi are their respective probabilities,
and all pi add to 1. Some common expectation properties are presented below:
E[a] = a
E[X + a] = E[X] + a
E[X + Y ] = E[X] + E[Y ]
E[a · X] = a · E[X]
E[aX + b] = a · E[X] + b
E[aX + bY ] = a · E[X] + b · E[Y ]
where X and Y are statistical distributions and a and b are constants.
3.1.2
Central Moments
A central moment is a statistical property commonly used in statistical analysis [32,
6, 33]. The k-th central moment of a distribution is defined in Eq. (3.2).
µk = E (x − x̄)k
=
N
1 X
(xj − x̄)k
N j=1
(3.2)
where x represents some distribution of N values, x̄ represents the input mean, and E is
the mathematical expectation operator. The central moments of a population can easily be
estimated using any appropriate population-sampling technique.
10
The zeroth central moment is always equal to one, the first central moment is always
equal to zero [34], and the second central moment is equivalent to the variance. The third
and fourth moments are used in the calculation of higher-order statistics, such as skewness
and kurtosis. Propagation of these higher-order statistics is the subject of Chapter 4.
3.2
Propagation of Variance Using First Order Taylor Series
Generally, a distribution of input values propagates through a given system to pro-
duce a distribution of output values. While many sources of error might be present, including
modeling error (unmodeled behavior, emergent behavior) and measurement error, this research focuses on the output distribution caused by variation in system model inputs only.
The analytical formula most-often cited in literature that estimates this variance
propagation is based on a first-order Taylor series expansion. It makes several important assumptions that are limiting in many practical situations. These assumptions and limitations
cannot be fully understood (much less overcome) without understanding the derivation of
this first-order variance propagation formula.
The author of this thesis has found the complete derivation of this formula to be
absent in textbooks and archival journal literature. Consequently, the derivation is presented
in this section in order to more fully explain the assumptions, limitations, and accuracy of
this error propagation technique. The assumptions and limitations mentioned are discussed
in Section 3.4.
3.2.1
First-Order Formula Derivation
Let y be some function of n inputs xi . The first-order Taylor series approximation
expanded about the input means x̄i is shown in Eq. (3.3) [35].
n
X
∂f
y ≈ f (x̄1 , ..., x̄n ) +
(xi − x̄i )
∂xi
i=1
(3.3)
where the partial derivatives are evaluated at the mean xi = x̄i . An approximation of the
output mean ȳ is given in Eq. (3.4).
11
ȳ = E[y]
"
#
n
X
∂f
≈ E f (x̄1 , ..., x̄n ) +
(xi − x̄i )
∂x
i
i=1
n
X
∂f
µ1,i
≈ f (x̄1 , ..., x̄n ) +
∂xi
i=1
≈ f (x̄1 , ..., x̄n )
(3.4)
where E is the expectation operator and µk,i is the k-th central moment for the i-th input,
as given previously in Eq. (3.2). (Recall that the first central moment is equal to zero.)
Subtracting Eq. (3.3) from Eq. (3.4) produces Eq. (3.5).
n
X
∂f
y − ȳ ≈
(xi − x̄i )
∂xi
i=1
(3.5)
Squaring and taking the expectation of Eq. (3.5) produces Eq. (3.6).
"
#
2
n
n
X
X
∂f
∂f ∂f 2
σ
E[(y − ȳ)2 ] ≈
σx2i + 2
∂xi
∂xi ∂xj xi xj
i=1
j=i+1
= σy2
(3.6)
where σy2 and σx2 are the variances in y and x, respectively. Recall that variance σ 2 is the
second central moment, which is defined in Eq. (3.7).
σx2 = µ2
= E[(x − x̄)2 ]
N
1 X
=
(xj − x̄)2
N j=1
(3.7)
The second term in Eq. (3.6) is the covariance term, where σx2i xj is the covariance between
inputs xi and xj . Covariance is defined in Eq. (3.8).
12
σx2i xj = E [(xi − x̄i )(xj − x̄j )]
(3.8)
When inputs are independent, the covariance term is equal to zero, and Eq. (3.6) reduces to
Eq. (3.9).
σy2
≈
2
n X
∂f
i=1
∂xi
σx2i
(3.9)
This simplifying assumption of independence is typically made, both in literature and in
practice. Consequently, Eq. (3.9) is the formula typically given for statistical error propagation through an analytical system model [36, 37, 38, 4].
3.2.2
First-Order Accuracy
Clearly, Eq. (3.9) is an approximation only and can be wrong by one or more orders
of magnitude. This is especially evident when dealing with nonlinear functions [4].
For example, let y be modeled by the function y = 1000 sin(x). An estimation of the
output variance σy2 obtained from Eq. (3.9) is given in Eq. (3.10).
2
σy,1st
≈ 106 cos2 (x̄)σx2
(3.10)
Throughout this entire thesis, “actual” output variance was determined using a Monte
Carlo simulation. While convergence in statistical properties was obtained for every model
in this thesis after 1 million executions, each system’s “actual” output is the result of 10
million executions.
The relative error of Eq. (3.10) can be calculated using Eq. (3.11) and the result is
plotted as a function of input mean x̄ in Figure 3.1.
=
2
|σy,
2
predicted − σy,M onte Carlo |
2
σy,M
onte Carlo
(3.11)
As illustrated in Figure 3.1, the first-order approximation of variance propagation is
fairly accurate for most values of x̄. However, for certain input values, the approximation
can be wrong by one or more orders of magnitude, as indicated by the 100% jump in relative
13
Relative Error in 1st Order Variance Prediction
Relative Error (%)
100
80
60
40
20
0
0
0.5
1
1.5
2
Input Mean
2.5
3
Figure 3.1: Relative Error in Output Variance Using a First-Order Taylor Series Expansion
for the Function y = 1000 sin(x)
error at x̄ = π2 . This spike in relative error occurs because the sin function is nonlinear and
the higher-order terms in the Taylor series used to derive Eq. (3.9) were neglected.
3.3
Propagation of Variance Using Higher-Order Taylor Series
As shown in the preceding section, Eq. (3.9) is based on a first-order Taylor series.
For nonlinear and higher-order polynomial functions, Taylor series truncation error becomes
significant and Eq. (3.9) can become extremely inaccurate (i.e., wrong by one or more orders
of magnitude [4]).
As expected, the accuracy of this estimate of variance propagation through a system
can be improved by including higher-order terms in the Taylor series. In situations where
increased accuracy is required, a second-order Taylor series is sometimes used to propagate
statistical error. Similar to the first-order approximation, some simplifying assumptions are
commonly made in the derivation of the second-order approximation, which are presented
in Section 3.4.
This section presents the derivation of the second-order error propagation formula
and then discusses the accuracy of this formula.
14
3.3.1
Second-Order Formula Derivation
For the sake of brevity, the second-order derivation is presented in this section for
a monovariable function, y = f (x). Extending this derivation to multivariate functions is
trivial, as it follows the same derivation steps.
The second-order Taylor series taken about the input mean x̄ is given in Eq. (3.12),
where the partial derivatives are again evaluated at the mean, x = x̄.
y ≈ f (x̄) +
∂f
1 ∂ 2f
(x − x̄) +
(x − x̄)2
∂x
2 ∂x2
(3.12)
The second-order approximation of the output mean ȳ is given in Eq. (3.13).
ȳ = E[y]
1 ∂ 2f
∂f
2
(x − x̄) +
(x − x̄)
≈ E f (x̄) +
∂x
2 ∂x2
1 ∂ 2f
≈ f (x̄) +
µ2
2 ∂x2
(3.13)
Subtracting Eq. (3.13) from Eq. (3.12) gives Eq. (3.14).
y − ȳ ≈
∂f
1 ∂ 2f
1 ∂ 2f
2
(x − x̄) +
(x
−
x̄)
−
µ2
∂x
2 ∂x2
2 ∂x2
(3.14)
Squaring and taking the expectation of Eq. (3.14) produces Eq. (3.15).
E (y − ȳ)2 ≈ µ2
∂f
∂x
2
∂f ∂ 2 f
1
2
+ µ3
+
µ
−
µ
4
2
∂x ∂x2 4
≈ σy2
∂ 2f
∂x2
2
(3.15)
If x is Gaussian, all odd moments (µk where k is odd) are zero and Eq. (3.15) reduces to
Eq. (3.16).
15
σy2
∂f
∂x
2
∂f
∂x
2
≈
≈
1
µ2 +
4
∂ 2f
∂x2
2
1
+
4
∂ 2f
∂x2
2
σx2
(µ4 − µ22 )
(µ4 − σx4 )
(3.16)
Furthermore, if x is Gaussian, the substitution µ4 ≈ 3σ 4 can be made [39], which eliminates
the need to know the input’s higher-order moments. This substitution is made in Eq. (3.17).
σy2
≈
∂f
∂x
2
σx2
1
+
2
∂ 2f
∂x2
2
σx4
(3.17)
If y is a function of multiple independent inputs, the generalized form of Eq. (3.17) is given
in Eq. (3.18).
σy2
≈
2
n X
∂y
i=1
∂xi
n
σx2i
n
1 XX
+
2 j=1 i=1
∂ 2y
∂xi ∂xj
2
σx2i σx2j
(3.18)
Equation (3.18) is the second-order formula most often cited in literature [40, 36] for analytical statistical error propagation. Note that the covariance terms in Eq. (3.18) have been
neglected.
3.3.2
Second-Order Accuracy
Continuing with the same function y = 1000 sin(x), the relative errors obtained from
the second-order approximations in Eq. (3.15) (full approximation) and Eq. (3.16) (Gaussian)
are plotted as a function of input mean x̄ in Figure 3.2.
Figure 3.2 illustrates the noise introduced by assuming x is Gaussian when in reality
the distribution is never perfect. Nevertheless, in both cases the second-order approximation
successfully filters the large spikes in relative error present in the first-order approximation.
However, the second-order approximation still overestimates the actual variance propagation,
which could degrade system performance and lead to failure or infeasibility [41]. This bias
(about 4%, in this case) is a result of truncating the higher-order terms in the Taylor series.
16
Relative Error in 2nd Order Approximation
Relative Error (%)
4.5
4
Gauss.
Full
3.5
0
0.5
1
1.5
2
Input Mean
2.5
3
Figure 3.2: Relative Error in Estimate of Variance Propagation Using a Second-Order Taylor
Series Expansion for the Function y = 1000 sin(x)
3.4
Taylor Series Error Propagation Assumptions and Limitations
Though common in engineering literature and academia, Eqs. (3.9) and (3.18) have
many significant limitations. Often, designers use these equations without knowing or understanding these limitations. The following list summarizes these assumptions and limitations:
1. Only variance is propagated and higher-order statistics are neglected. Consequently,
the output distribution is generally assumed to be a Gaussian distribution, which is
not the case in many practical applications.
2. The system model y must be representable as a closed-form, differentiable, mathematical equation.
3. Taking the Taylor series expansion about a single point (x̄) causes the approximation
to be of local validity only [36, 19]. Consequently, the accuracy of the approximation
generally decreases with an increase in the input variance σx2 .
4. The approximation is generally more accurate for linear models.
5. All inputs xi are assumed be Gaussian. Consequently, only means and variances are
used to fully describe the input distributions. When inputs are not Gaussian, the
non-Gaussian terms (i.e., odd moments) cannot be neglected.
17
6. The input means and standard deviations must be known.
7. All inputs xi are assumed to be independent. When inputs are not independent, the
covariance terms cannot be neglected [37, 42, 4].
The first assumption listed above is addressed in detail in Chapter 4.
3.5
Higher-Order Accuracy With First- or Second-Order Cost
While an improvement over the first-order formula, the second-order equation given
by Eq. (3.18) is far from perfect. This section shows that accuracy can be improved by
adding higher- and higher-order terms. However, the resulting computational cost quickly
becomes prohibitive. This is a common problem in engineering analysis and design [43].
This section then presents a solution to that problem. This is accomplished by predicting the higher-order truncation error and applying a resultant correction factor to the
lower-order approximation. This can yield fourth-order accuracy in the estimation of error
propagation with first- or second-order computational cost. These correction factors were
empirically determined for trigonometric, logarithmic, and exponential functions.
3.5.1
Propagation of Variance Using Higher-Order Terms
As expected, adding higher-order terms reduces the second-order bias. This is shown
in Figure 3.3. With an infinite number of terms, the Taylor series approximation eventually
converges to zero error.
However, computational cost grows exponentially as higher-order terms are included.
This growth in computational cost is accelerated at an exponential rate with an increase in
the number of system inputs, as shown in Table 3.1. Furthermore, higher-order terms also
require the calculation of higher-order moments and covariance terms for the system inputs.
This exponential growth in cost causes higher-order terms to quickly become prohibitively
expensive for complex systems.
For the purposes of this research, the computational costs of propagating error were
determined using the MATLAB execution time. Figure 3.4 shows the relative computational
18
Relative Error Using Higher−Order Taylor Series
6
O1
Relative Error (%)
5
O2
4
O3
3
O4
2
1
0
0
0.5
1
1.5
2
Input Mean
2.5
3
Figure 3.3: Relative Error in Variance Propagation Using Taylor Series Approximations
Table 3.1: Partial Derivatives Required for Taylor Series
Variance Propagation
Inputs
2
4
6
8
10
O1
2
4
6
8
10
O2 O3 O4 O5
O6
3
6
10
15
21
10 20 35
56
84
21 56 126 252 462
36 120 330 792 1716
55 220 715 2002 5005
O7
28
120
792
3138
11146
cost associated with including higher-order terms in the calculation of variance propagation.
(The second-order approximation with a correction factor seen at the end of Figure 3.4 is
presented and discussed in Section 3.5.2.)
Furthermore, computational cost grows exponentially with an increase in the number
of system inputs and with an increase in model complexity. As this high cost can be prohibitive, higher-order terms are usually not considered. The author of this thesis has shown
that higher-order truncation error can be predicted. Consequently, a correction factor can
be applied to the lower-order formula, yielding higher-order accuracy without significantly
increasing computational cost.
19
Computation Cost
Computation Cost of Higher−Order Terms
1st Full
2nd Full
3rd Full
4th Full
2nd Corrected
Figure 3.4: Relative Computational Cost of Taylor Series Approximations, as Determined
by MATLAB Execution Time
3.5.2
Reducing Truncation Error
As Figure 3.2 illustrates, the truncation error in the second-order approximation for
a sin function is essentially a constant bias for all x̄. Furthermore, this bias has a linear
relationship to the input variance, σx2 , as shown in Figure 3.5.
Consequently, the second-order truncation error can be easily be estimated empirically. A
correction factor e corresponding to this truncation error can be calculated using Eq. (3.19).
e=
1
1 + 1.022σx2
(3.19)
This correction factor can then be applied to the second-order approximation of variance propagation, as shown in Eq. (3.20), which reduces the higher-order Taylor series truncation error.
2
σy,CF
≈ σy2 e
(3.20)
By way of example, consider the same function y = 1000 sin(x). The corrected
second-order approximation produces an estimate of error propagation with fourth-order
accuracy, but with only second-order computational cost. Figure 3.4 (already displayed in
20
2nd Order Mean % Err.
10
Mean % Error
8
6
4
2
0
0
0.05
0.1
0.15
0.2
Input Standard Deviation (σx)
0.25
0.3
2nd Order Mean % Err.
10
Mean % Error
8
6
4
2
0
0
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08
2
Input Variance (σx)
Figure 3.5: Relationships Between the Second-Order Bias and both Input Standard Deviation σx and Input Variance σx2 for a sin Function
Section 3.5.1) compares the computational costs of the uncorrected Taylor series approximations with the corrected second-order approximation.
Figure 3.6 compares the relative errors in variance estimations. Note that the fourthorder and second-order corrected approximations are almost identical in Figure 3.6.
Figure 3.7 compares the mean relative error averaged over the range 0 ≤ x̄ ≤ π,
showing that fourth-order accuracy can be obtained by adding a correction factor to the
second-order formula for error propagation.
21
Relative Error in Variance Estimation
Relative Error (%)
5
4
O2
3
O4
2
CF
1
0
0
0.5
1
1.5
2
Input Mean
2.5
3
Figure 3.6: Relative Error in Estimations of Variance Propagation
Accuracy Summary of Variance Propagation Formulas
Mean Relative Error (%)
5
2nd Order
4th Order
2nd Corrected
4
3
2
1
0
Estimation Method
Figure 3.7: Mean Relative Error in Variance Estimations
3.5.3
Correction Factors for Other Nonlinear Functions
Table 3.2 gives the correction factors for some common nonlinear functions. Note
that the cyclical nature of trigonometric functions and derivatives require a second-order
approximation before the higher-order truncation error can easily be determined, but the
exponential and logarithmic functions only require a first-order calculation. The correction
factors for y = ln(x), y = exp(x), and y = bx are given in Eqs. (3.21), (3.22), and (3.23),
respectively.
Correction factor for y = ln(x), where r = ln
22
x̄
σx
:
Table 3.2: Truncation Error Correction Factors for
Common Nonlinear Functions
Func.
Ord.
y = sin(x)
2nd
y = cos(x)
2nd
y = ln(x)
1st
y = exp(x)
1st
y = bx
1st
e=
Correction Factor (e)
Eq. (3.19)
Eq. (3.19)
Eq. (3.21)
Eq. (3.22)
Eq. (3.23)


exp (−1.9772r + 0.9128) + 1
if r >= 0
(3.21)

−3.88r3 − 4.9835r2 − 1.5704r + 1.3302 if r < 0
Correction factor for y = ex , where e is constrained to a maximum value of 2 and a minimum
value of 1:
e = max 1, min 2, 0.3375σx2 + 0.4937σx + 0.959
(3.22)
Correction factor for y = bx , where e is constrained to a maximum value of 2 and a minimum
value of 1:

h
i

max 1, min 2, X · (Z1 · B)T + 1 if b < 1
e=
h
i

max 1, min 2, X · (Z2 · B)T + 1 if b > 1
where
X=
h
1
B=
h
σx2
σx
1
b
23
σx3
2
b
b
3
σx4
iT
i
(3.23)

0.224


 −0.992
Z1 = 

 −0.037

0

0.012
0
0.076
1.544e4
0
7.803
−2.061e4
7.135
0
0
23.556
−8.612


1.182
1 

Z2 =

1000  0.101

−0.002
3.5.4
−2.570e3
−897.1
0.018
0




0


−38.093 

61.532
0.006
0.084
986.2
−0.002
−6.891
54.93
−240.9
0.089
−0.276
−1.743
0.005



−0.008 


26.52 

11.06
Model Composition
It should be noted that the correction factors given in Table 3.2 are only pertinent to
a particular function. If a system model contains this function along with other operators,
the system should be decomposed into components (with sin(x) being a single component,
for example). The error should then be propagated through each component individually.
The variances in each component’s output can then be propagated through the rest of the
system model. This process of model decomposition is demonstrated in Section 3.6 with the
example of a dual-propeller, three degree-of-freedom helicopter.
3.6
Example: Compositional System Model of a Dual Propeller, Three Degreeof-Freedom Helicopter
Consider the dual-propeller, three degree-of-freedom helicopter shown in Figure 3.8.
This helicopter is a useful tool for the design of unmanned aerial vehicles (UAVs). Among
other benefits, it allows the system designer to 1) test pitch, roll, and yaw control systems
without endangering actual aircraft, 2) observe the effects of changing the payload and
component weights and positions, and 3) calibrate the conversion factor (given as km below)
that converts a motor’s throttle command into units of thrust. A UAV’s position and
orientation are the inputs to its control system. An efficient prediction of the error in
the UAV’s position allows the designer to easily create a robust control system capable of
accomplishing mission objectives.
24
Figure 3.8: Dual-Propeller, Three Degree-of-Freedom Helicopter Used in the Design of Unmanned Aerial Vehicles
The system model for the longitudinal (pitch) acceleration as a function of the system’s design parameters and inputs is given in Eq. (3.24).
θ̈ =
(m2 L2 − m1 L1 )g
L1 km (uL + uR )
cos θ +
2
2
m1 L1 + m2 L2 + Jy
m1 L21 + m2 L22 + Jy
(3.24)
where the masses mi and lengths Li are indicated in Figure 3.8, Jy is the mass moment of
inertia about the pitch axis of rotation, g is the acceleration of gravity, uL and uR are the
throttle commands (0-100) given to the left and right motors, respectively, and km is a motor
calibration constant that converts throttle commands to units of thrust.
For this demonstration, the assumed mean and standard deviation of the dependent
and independent system design parameters are given in Tab. 3.3. Three system inputs are
also required: uL and uR , which were each fixed at 45, and the pitch angle θ. The mean
value of θ was varied from −45◦ to 45◦ , with a standard deviation of 1◦ .
The first-order approximation in Eq. (3.9) can be used to propagate error through this
system (see Figures 3.9-3.10). However, the nonlinearity of the cos function causes significant
error in the results. Consequently, the model should be decomposed into its components,
as shown in Eq. (3.25). Error can then be propagated through each component. Once the
variance in each component is obtained, each component can then be treated as a variable
25
Table 3.3: Design Parameters for Dual-Propeller, 3-DOF Helicopter
Design Parameter
Mean
m1
0.891 (kg)
m2
1.000 (kg)
L1
0.850 (m)
L2
0.3048 (m)
Jy
0.0014 (kg · m2 )
km
0.0546
Std. Dev.
10−4 (kg)
10−4 (kg)
10−4 (m)
10−4 (m)
10−5 (kg · m2 )
10−5
and its variance propagated through the system-level compositional model. A full step-bystep solution is presented in Appendix A.
θ̈ = C1 C2 + C3
(m2 L2 − m1 L1 )g
C1 =
m1 L21 + m2 L22 + Jy
C2 = cos θ
L1 km (uL + uR )
C3 =
m1 L21 + m2 L22 + Jy
(3.25)
The first-order Taylor series approximation given in Eq. (3.9) is used to propagate
error from the input parameters through components C1 and C3 . The second-order approximation with a correction factor is used to propagate error through the nonlinear trigonometric component C2 . Figure 3.9 compares the results of this compositional model with
a correction factor applied with the results of the first-order approximation from Eq. (3.9)
applied to the full system model in Eq. (3.24). These results are shown for −45◦ ≤ θ̄ ≤ 45◦ .
Figure 3.10 shows the relative error for each of these methods, averaged across the
entire input domain −45◦ ≤ θ̄ ≤ 45◦ .
The computational cost required to determine the output variance using the compositional system model was approximately three times less than the cost using the first-order
approximation. Thus, twice the accuracy was obtained with one-third of the computational
26
Relative Error
Relative Error (%)
40
30
Compositional
1st Order
20
10
0
−40
−20
0
20
θ Mean (deg)
40
Figure 3.9: Relative Error in Estimations of Variance Propagation
Mean Relative Error
Mean Relative Error (%)
2
1.5
1
0.5
0
1st Order
Approximation
Compositional
Model
Estimation Method
Figure 3.10: Mean Relative Errors that Indicate the Compositional Model with a Correction
Factor Predicts Output Variance with About Half the Error as the First-Order Approximation
cost. This efficiency in error propagation allows the system designer to more easily create a
robust system capable of accomplishing mission objectives.
27
3.7
Example: Kinematic Model of a Mechanism that Simulates Flapping Flight
Wing Motion
Consider the flapping flight wing mechanism shown in Figure 3.11. The kinematic
model used in the design and optimization of this mechanism is the Fourier series in Eq. (3.26) [44,
45].
Figure 3.11: Mechanism Used by the BYU Flapping Flight Research Team to Simulate
Three-Degree-of-Freedom Motion of a Flapping Wing






φ(t)
A
B
2  φn 

 X
 φn 






 θ(t)  =
 Aθn  cos(nωt) +  Bθn  sin(nωt)

 n=0 



α(t)
Aαn
Bαn
(3.26)
where φ is the positional angle (deg), θ is the elevation angle (deg), α is the feathering or
attack angle (deg), the As and Bs are Fourier series coefficients (deg), ω is the flapping
frequency (Hz), and t is time (s). This three-output system model has 16 Gaussian inputs
(time does not vary), which are statistically described in Table 3.4 [45].
Various orders of a Taylor series were used to estimate the variance in the three
output wing angles based on the variance in these system inputs. The system model was
then decomposed (as demonstrated in the previous example) and the trigonometric correction
28
Table 3.4: Mean Values and Standard Deviations of
the 16 Model Input Parameters
Aφ0
Aφ1
Aφ2
Aθ0
Aθ1
Aθ2
Aα0
Aα1
Aα2
ω
Mean
-20
-4
8
0
43
17
12
0
-2
0.298
StdDev
0.1
1.5
3.0
0.1
0.75
0.5
4.0
0.1
0.75
0.1
Bφ1
Bφ2
Bθ1
Bθ2
Bα1
Bα2
Mean
44
33
0
0
50
0
StdDev
1.5
8.0
0.1
0.1
0.1
0.1
factors given in Table 3.2 were then applied. Figure 3.12 shows the computational cost of
error propagation (in minutes) using these different methods. These costs are summed over
the time interval 0s≤t≤5s at time steps of 0.001s for all three output angles. The cost of
calculating input moments and input covariances from 10k input samples is also included.
Computational cost was determined using MATLAB execution time.
Computational Time of Variance Propagation Methods
Execution Time (min)
80
60
O1
O2
O3
O4
CF
40
20
0
Estimation Method
Figure 3.12: Computational Time to Predict Output Distributions Using Various Error
Propagation Methods
29
The fourth-order prediction took approximately 70 minutes to execute, where the
corrected second-order approximation executed in approximately 4 minutes—a significant
reduction in computational cost.
The relative error in only one of these output angles, φ, is shown in Figure 3.13,
as the other two output angles have similar results. The root-mean-square of the relative
error over the time interval shown (0s≤t≤5s) for the second-order approximation is 40.97%,
third-order is 11.18%, fourth-order is 1.32%, and a second-order with a correction factor is
1.96%.
Relative Error in φ
Relative Error (%)
150
O2
O3
O4
CF
100
50
0
0
1
2
3
4
5
Time (s)
Figure 3.13: Relative Error in Predictions of Output Variance Obtained from Various Orders
of a Taylor Series
Figures 3.12 and 3.13 illustrate that the correction factors presented in this paper
can achieve near fourth-order accuracy in error propagation through this model with near
second-order computational cost.
3.8
Comments on Variance Propagation
Using a first-order Taylor series to estimate error propagation through a closed-form
model is common practice. Frequently, this approximation is used without a full appreciation
of its limitations and the assumptions upon which it is based. This results in estimations that
may be substantially inaccurate. Furthermore, the author has been unable to locate the full
30
derivation of this formula either in textbook or archival journal literature. Consequently,
a useful contribution of the research presented in this chapter is the derivation and the
discussion of the limitations and assumptions of the first- and second-order Taylor series
approximations for variance propagation.
Additionally, the novel contribution of this chapter is the introduction of generic
correction factors that account for some of the Taylor series truncation error for common
nonlinear functions encountered in engineering. These correction factors are predictable,
easy to calculate, and don’t require significant computational cost. A system designer can
use these correction factors to predict error propagation through trigonometric, logarithmic,
and exponential functions with greater accuracy without greater computational cost. This
enables the designer to better verify design decisions, which reduces the risk of developing a
design that does not meet design objectives.
Future research may focus on the development of predictable correction factors for
other nonlinear models, such as differential equations and state-space models. The author
believes this can be accomplished using the same methods used in this chapter.
31
32
CHAPTER 4.
HIGHER-ORDER STATISTICS
For closed-form, analytical models, a statistical error distribution is usually obtained
by propagating variance from system inputs to system outputs using a Taylor series. This has
already been demonstrated in Chapter 3. As noted in Section 3.4, the variance propagation
formulas given in Eqs. (3.9) and (3.18) assume all inputs are Gaussian. Since all higher-order
statistics (e.g., skewness, kurtosis, etc.) are ignored, outputs are also typically assumed to
be Gaussian. This assumption is often erroneous and does not accurately reflect reality, as
proved later in Section 4.2.4.
In order to truly perform an accurate statistical error analysis, outputs cannot be
assumed to be Gaussian. However, since all non-Gaussian distributions cannot be fully
described by a mean and standard deviation alone, higher-order statistics (e.g., skewness
and kurtosis) must also be used. Fortunately, this is relatively simple using the method and
formulas presented in this chapter.
This chapter first demonstrates the need for propagating higher-order statistics. A
second-order Taylor series is then used to show how skewness and kurtosis can both be
propagated through a system model. Having a mean, variance, skewness, and kurtosis, the
system designer can then fully describe the output distribution. The benefits of obtaining
a fully-described output distribution are demonstrated using two examples: the propeller
component of a solar-powered unmanned aerial vehicle, and a flat rolling metalworking
process.
4.1
Motivation for Propagating Higher-Order Statistics
Consider the simple quadratic function, y = x2 . Assume the input x is a Gaussian
distribution with a mean x̄ and a standard deviation σx both equal to 1. Equation (3.9)
can be used to propagate this input distribution through the system model and predict the
33
Gaussian output distribution shown in Figure 4.1a. This predicted output is very different
from the actual system output distribution, shown in Figure 4.1b.
Actual Output Distribution
0.4
0.3
0.3
Prob. (%)
Prob. (%)
Output Predicted from Mean/Var. Only
0.4
0.2
0.1
0
0.2
0.1
−5
0
5
Output Value (y)
0
10
(a) Gaussian Prediction Using Mean and Variance
Only
−5
0
5
Output Value (y)
10
(b) Actual System Output
Figure 4.1: Predicted Gaussian Output Distribution Obtained from Propagating Mean and
Variance Only (a) Compared with Actual System Output (b)
However, if skewness and kurtosis are also propagated through the system model, the
predicted output distribution resembles actual system output much more closely [46]. This
is illustrated in Figure 4.2.
Actual Output Distribution
0.4
0.3
0.3
Prob. (%)
Prob. (%)
Output Predicted from Mean, Var., Skew., & Kurt.
0.4
0.2
0.1
0
0.2
0.1
−5
0
5
Output Value (y)
0
10
(a) Non-Gaussian Prediction Using Mean, Variance,
Skewness, and Kurtosis
−5
0
5
Output Value (y)
10
(b) Actual System Output
Figure 4.2: Predicted Non-Gaussian Output Distribution Obtained from Propagating Mean,
Variance, Skewness, and Kurtosis (a) Compared with Actual System Output (b)
A proof that nonlinear systems produce non-Gaussian outputs even when inputs are
Gaussian is presented later in Section 4.2.4.
34
4.2
Propagation of Skewness
As previously noted, non-Gaussian distributions cannot be fully described with a only
mean and standard deviation. Consequently, higher-order statistics, such as skewness and
kurtosis, must also be used to describe non-Gaussian distributions. This section considers the
definition of skewness and derives a formula for propagating skewness through an analytical
system model.
4.2.1
Definition of Skewness
The first-order statistic of a distribution is its mean, the second-order statistic is its
standard deviation, and the third-order statistic is its skewness. Skewness is a measure of a
distribution’s asymmetry. Skewness (denoted γ1 ) is defined in Eq. (4.1).
"
γ1 = E
=
x − x̄
σ
3 #
µ3
σ3
(4.1)
where E is the expectation operator, µ3 is the third central moment, σ is the standard
deviation.
Table 4.1 and Figure 4.3 illustrate some characteristics and terminology of positivelyand negatively-skewed distributions. A skewness of zero indicates a symmetric distribution.
Table 4.1: Comparison of Positive and Negative Skew
Sign
Left/Right Mean vs. Median [47]
Negative Left-skewed Mean is typically (though not always) less than the median
Positive Right-skewed Mean is typically (though not always) greater than the median
Skewness is an important defining characteristic of statistical distributions. A measure of skewness is required to fully describe any asymmetric distribution. Traditional uncertainty propagation, however, only propagates a mean and variance. With no skewness
35
Figure 4.3: Examples of Negative (Left) and Positive (Right) Skewness
information available, skewness is neglected (assumed equal to zero) and a Gaussian distribution is assumed.
4.2.2
Skewness Propagation Formula Derivation
Using a first-order Taylor series to propagate skewness through a system model results
in an output skewness equal to the input skewness. It has already been demonstrated that
this often does not reflect reality, even for simple nonlinear functions, and a proof is presented
later in Section 4.2.4.
Consequently, a second-order Taylor series will be used to derive a formula for skewness propagation.
The second central moment of output y has already been given in
Eq. (3.15). The third central moment is given by the dot-product in Eq. (4.2).




3
E (y − ȳ) ≈ 



 
µ3
∂13

 

  2 
 ∂1 ∂2 
(µ4 −
·

  2
3
3
 ∂1 ∂2 
µ − 2 µ2 µ3
4 5
 

1 3
1
3
3
µ − 8 µ2 µ4 + 8 µ6
∂2
4 2
3
2
µ22 )
(4.2)
where µk is the k-th central moment of input x, and ∂1 and ∂2 respectively represent the
partial derivatives
∂f
∂x
and
∂2f
,
∂x2
evaluated at the mean x = x̄. The third moment is a cubic
function, and consequently it has four terms. Equation (4.2) has both first and second partial
derivatives, because it is based on a second-order Taylor series. If a higher-order Taylor series
were used, Eq. (4.2) would contain higher-order partial derivatives.
36
The second moment from Eq. (3.15) and the third moment from Eq. (4.2) can be
used with the definition of skewness given by Eq. (4.1) to estimate the skewness in output
y. This output skewness estimation is given in Eq. (4.3).
γ1
E (y − ȳ)3
= 1.5
E (y − ȳ)2
 


µ3
∂13
 


  2 

3
2
 ∂1 ∂2 

(µ4 − µ2 )
2
·


  2

3
3
 ∂1 ∂2 

µ − 2 µ2 µ3
4 5
 


1 3
3
1
3
µ − 8 µ2 µ4 + 8 µ6
∂2
4 2
≈ 1.5
µ2 ∂12 + µ3 ∂1 ∂2 + 41 (µ4 − µ22 ) ∂22
(4.3)
Equation (4.3) estimates output skewness using the input central moments, µk . If
input skewness, kurtosis, and higher-order statistics are known instead of input moments,
these statistics can easily be substituted into Eq. (4.3) in place of these moments.
For the sake of brevity, the skewness propagation formula has only been derived for
mono-variate functions. However, this derivation can easily been extended to multivariate
functions as desired.
It should be noted that the most computationally expensive part to propagating skewness is calculating first and second derivatives. However, these have already been calculated
in order to propagate variance if a second-order Taylor series was used, and consequently
the additional cost to also propagate skewness is minimal.
4.2.3
Skewness Propagation Assumptions and Limitations
The following five assumptions and limitations apply to the method just presented to
propagate skewness:
1. Equation (4.3) is based on a second-order Taylor series. Consequently, it will predict
output skewness perfectly for second-order (or lower) functions. Accuracy decreases
with increasing non-linearity.
37
2. While not a problem for most analytical engineering models, it should be noted that
this method of skewness propagation requires the system model to be represented as a
closed-form, twice-differentiable, mathematical equation. This is due to the dependence
of this method on a Taylor series.
3. The system model y must be representable as a closed-form, differentiable, mathematical equation.
4. Taking the Taylor series expansion about a single point (x̄) causes the approximation
to be of local validity only [36, 19]. Consequently, the accuracy of the approximation
generally decreases with an increase in the input moments µk .
5. The statistical input distribution must be known (at least up to the 6th central moment).
4.2.4
Proof: Nonlinear Functions Produce Non-Gaussian Output Distributions
Consider the propagation of Gaussian error. With a Gaussian distribution, the fol-
lowing expressions are true:
• All odd moments (µk , where k is odd) are equal to zero
• The fourth moment is equal to three times the second moment squared[39] (µ4 = 3µ22 )
• The sixth moment is equal to fifteen times the second moment cubed (µ6 = 15µ32 )
Consequently, Eq. (4.3) reduces to Eq. (4.4) when inputs are Gaussian.
γ1 ≈
3σx ∂12 ∂2 + σx3 ∂23
1.5
∂12 + 12 ∂22 σx2
(4.4)
where σx is the standard deviation of the input distribution. Equation (4.4) proves that
nonlinear functions (i.e., the second partial derivative is non-zero) produces a skewed nonGaussian output, even with Gaussian inputs. Consequently, the commonly-made assumption
that outputs are Gaussian is generally erroneous for nonlinear functions.
38
4.3
Propagation of Kurtosis
Any statistical property that is propagated through a system model improves the
accuracy of the predicted output distribution. For example, propagating both a mean and a
variance is more accurate (and useful) than propagating a mean alone. In a similar manner,
propagating skewness (as shown in Section4.2) in addition to a mean and variance also
improves the accuracy of the predicted output distribution.
Further improvements in accuracy can be obtained by also propagating kurtosis, the
fourth-order statistic. This section defines kurtosis and excess kurtosis, and derives a formula
for propagating kurtosis through an analytical system model.
4.3.1
Definition of Kurtosis
The fourth-order statistic is kurtosis. Kurtosis is a measure of a distribution’s “peaked-
ness,” or the thickness of the distribution’s tails. Kurtosis (denoted β2 ) is the fourth standardized moment, and is defined in Eq. (4.5).
"
β2 = E
=
x − x̄
σ
µ4
σ4
4 #
(4.5)
The kurtosis of a Gaussian distribution is equal to 3.
4.3.2
Definition of Excess Kurtosis
In statistical analysis, “excess kurtosis” (denoted γ2 ) is often used more than kurtosis.
In practice, the term “kurtosis” more often refers to excess kurtosis instead of the fourth
standardized moment. To avoid confusion, this thesis uses the definition of kurtosis presented
above and defines excess kurtosis as the fourth cumulant divided by the square of the second
cumulant, as indicated in Eq. (4.6). Cumulants (denoted κi ) are statistical properties similar
to moments, where κ2 = µ2 , κ3 = µ3 , and κ4 = µ4 − 3µ22 . While it can be more convenient to
39
use cumulants instead of moments in some statistical analyses, the propagation of uncertainty
using a Taylor series is much simpler and easier using central moments.
Since a Gaussian distribution has a kurtosis of three, the “minus 3” in Eq. (4.6) causes
a Gaussian distribution to have zero excess kurtosis.
κ4
κ22
µ4
=
−3
σ4
γ2 =
(4.6)
Figure 4.4 shows the excess kurtosis of several common types of distributions.
Figure 4.4: Excess Kurtosis of Various Common Statistical Distributions
4.3.3
Kurtosis Propagation Formula Derivation
A second-order Taylor series will also be used to propagate kurtosis through a system
model. The third central moment of output y has already been given in Eq. (4.2), and the
fourth moment is given in Eq. (4.7).
40






4
E (y − ȳ) ≈ 




µ4
 
∂14


 
  3 
 ∂1 ∂2 

 
  2 2
3
3
 · ∂1 ∂2 
(µ2 − 2µ2 µ4 + µ6 )
2

 

 
3
1
3
2
 ∂1 ∂2 
(µ2 µ3 − µ2 µ5 + 3 µ7 )
2

 
1
4
2
4
∂2
(6µ2 µ4 − 3µ2 − 4µ2 µ6 + µ8 )
16
2(µ5 − µ2 µ3 )
(4.7)
The excess kurtosis γ2 in the output distribution y is given by Eq. (4.8).
γ2 = β2 − 3
(4.8)
where kurtosis β2 is given by Eq. (4.9).
β2
E (y − ȳ)4
= 2
E (y − ȳ)2


 
4
µ4
∂

  1 

  3 

 ∂1 ∂2 
2(µ5 − µ2 µ3 )


 

  2 2
3
3
·




∂
∂
−
2µ
µ
+
µ
)
(µ
2 4
6
2
2

  1 2


 
3

 ∂1 ∂23 
(µ22 µ3 − µ2 µ5 + 31 µ7 )
2


 
1
4
2
4
∂2
(6µ2 µ4 − 3µ2 − 4µ2 µ6 + µ8 )
16
≈
2
µ2 ∂12 + µ3 ∂1 ∂2 + 41 (µ4 − µ22 ) ∂22
(4.9)
An estimate of the kurtosis of an output distribution can be obtained using Eq. (4.9)
and a known input distribution. The input central moments µk can be estimated using any
appropriate population sampling technique.
4.4
Example: Solar-Powered Unmanned Aerial Vehicle Propeller Thrust
Consider the power and propulsion system of the solar-powered unmanned aerial
vehicle (UAV) shown in Figure 4.5.
41
Figure 4.5: Power and Propulsion System of a Solar-Powered Unmanned Aerial Vehicle
(UAV)
Larson (see Reference [21]) and the author of this thesis have previously created
a compositional system model for this system (as shown in Figure 4.6), deterministically
propagated a max/min error envelop through the model, and validated the results using real
data. As this thesis focuses on statistical, non-deterministic error analysis, a full description
of this model and the deterministic approach used to propagate error will not be replicated
in this thesis. However, the propeller component of this system and its corresponding data
will be used to demonstrate the increased accuracy obtained from propagating higher-order
statistics through a system model.
4.4.1
The Analytical Model of Thrust
The thrust T that the propeller produces is given by Eq. (4.10)[48].
T = Ct ρω 2 D4
42
(4.10)
Figure 4.6: Compositional System Model of a Solar-Powered Unmanned Aerial Vehicle
(UAV)
where Ct is a unitless coefficient of thrust calibrated to be 3.458e-3, ρ is the density of air
assumed to be constant at 1kg/m3 , and D is the propeller diameter measured to be 0.1778m.
The angular velocity of the propeller, ω, is provided in rad/s, and is determined by the motor
speed, which in turn is determined by a motor throttle command. Since a given throttle
command does not consistently produce an exact motor speed, there is some variation in
the angular velocity of the propeller. This distribution in angular velocities is described in
Table 4.2 and shown in Figure 4.7.
Table 4.2: Statistical Properties of the
Angular Velocity Distribution
of a Propeller
Statistical Property
Mean (ω̄):
Variance (σω2 ):
Skewness (γ1 ):
Excess Kurtosis (γ2 ):
43
Value
526.7
9.383
-1.289
4.896
Input Distribution
0.2
Prob. (%)
0.15
0.1
0.05
0
510
515
520
525
530
Angular Velocity ω, rad/s
535
540
Figure 4.7: Distribution of the Angular Velocities of a Propeller in a Solar-Powered UAV
4.4.2
Statistical Component Model Output Prediction
Based on this distribution of ω, the first eight central moments were calculated using
Eq. (3.2). These moments, given in Table 4.3, are used to propagate statistical properties
from the angular velocity distribution to predict a distribution for thrust output by the
propeller at a given motor throttle command.
Table 4.3: Central Moments of the Distribution
of Angular Velocity, ω
Central Moment Value
1st Moment (µ1 ):
0
2nd Moment (µ2 ):
9.355
3rd Moment (µ3 ):
-36.34
4th Moment (µ4 ):
6.617e2
5th Moment (µ5 ):
-1.063e4
6th Moment (µ6 ):
2.726e5
7th Moment (µ7 ):
-8.678e6
8th Moment (µ8 ):
3.308e8
44
The second-order prediction of the mean thrust output T̄ was calculated to be 0.9587N
using Eq. (3.13). Equation (3.15) predicts a variance of 1.232e-4N2 . Typically, higherorder statistics are not propagated and a Gaussian output distribution is assumed, which
is compared with actual thrust measurements in Figure 4.8. This prediction only accounts
for 65% of the actual system output distribution (i.e., 65% overlap in the area under the
predicted and actual probability density functions).
Predicted Output Distribution from Mean/Var. Only
Actual Output Distribution
0.8
0.5
Prob. (%)
Prob. (%)
0.6
0.4
0.3
0.4
0.2
0.2
0.1
0
0.88
0.9
0.92
0.94
0.96
Propeller Thrust, N
0.98
0
1
(a) Gaussian Prediction Using Mean and Variance
Only
0.9
0.92
0.94
0.96
Propeller Thrust, N
0.98
(b) Actual Thrust Measurements
Figure 4.8: Predicted Gaussian Output Distribution of Thrust Obtained by Propagating a
Mean and Variance Only (a) Compared with Actual Thrust Measurements (b)
The methods presented in this chapter to propagate skewness and kurtosis, using
Eqs. (4.3) and (4.9), respectively, results in a more accurate prediction of the system output.
This predicted output, shown in Figure 4.9a, accounts for 79% of actual system outputs—a
22% improvement in accuracy compared with propagating a mean and variance alone.
4.5
Example: Flat Rolling Metalworking Process
Consider the manufacture of steel plates or sheets via flat rolling, where material is
fed between two rollers (called working rolls). The gap between the working rolls is less than
the thickness of the incoming material. As the working rolls rotate in opposite directions,
the incoming material elongates as its thickness is reduced. This process, illustrated in
45
Predicted Output Distribution from Mean, Var., Skew., & Kurt.
Actual Output Distribution
0.8
0.5
Prob. (%)
Prob. (%)
0.6
0.4
0.3
0.4
0.2
0.2
0.1
0
0.88
0.9
0.92
0.94
0.96
Propeller Thrust, N
0.98
0
1
(a) Non-Gaussian Prediction Using Mean, Variance,
Skewness, and Kurtosis
0.9
0.92
0.94
0.96
Propeller Thrust, N
0.98
(b) Actual Thrust Measurements
Figure 4.9: Predicted Non-Gaussian Output Distribtuion of Thrust Obtained by Propagating
a Mean, Variance, Skewness, and Kurtosis (a) Compared with Actual Thrust Measurements
(b)
Figure 4.10, can be done either below the recrystallization temperature of the material (cold
rolling) or above it (hot rolling).
α
R
h0
v0
vf
hf
Figure 4.10: Flat Rolling Manufacturing Process Whereby Plates or Sheets of Metal are
Made—Material is Drawn Between Two Rollers, which Reduces the Material’s Thickness
46
4.5.1
The Analytical Model
The manufacturer desires to use its flat rolling equipment more efficiently by reducing
overall rolling time for each plate. Consequently, the manufacturer desires to minimize
the number of passes required to achieve final plate thickness. The maximum amount of
deformation that can be achieved in a single pass is a function of the friction at the interface
between the rolls and the material. If the intended change in thickness is too great, the
rolls will merely slip along the material without drawing it in [49]. The maximum change in
thickness attainable in a single pass (∆Hmax ) is given in Eq. (4.11) [50].
∆Hmax = µ2f R
(4.11)
where µf is the coefficient of friction between the rolls and the plate, and R is the radius
of the rolls. In this example, the radius of the rolls is measured to be 1m. Determining the
coefficient of friction in a metalworking process is more difficult, however. The conditions
surrounding friction in a metalworking process are very different from those in a mechanical
device [49], as shown in Table 4.4.
Table 4.4: Friction Conditions in Mechanical Devices and Metalworking Processes
Typical Mechanical Devices
-Two surfaces of similar material and
strength
-Elastic loads and no change in shape
-Wear-in cycles produce surface
compatibility
-Low-to-moderate temperatures
-Friction force depends on contact
pressure
Metalworking Processes
-One very hard tool and one softer
material
-Plastic deformation occurs in material
-Each set of rollers makes a single
pass on material
-Contact area constantly changes
under deformation
-Often elevated temperatures
-Friction force depends on material
strength
Furthermore, lubrication is often used both to reduce friction and consequent tool
wear, and to act as a thermal barrier to help regulate tool temperature [51]. All these
47
factors and others (e.g., rolling speed, material properties, surface finishes, etc.) combine
to create variation in the friction experienced in the flat rolling metalworking process. This
variation can inhibit the manufacturer’s ability to specify a gap width (and the resulting
change in material thickness, ∆H) for each pass.
While many empirical and mathematical formulas have been presented as methods
to predict the coefficient of friction in flat rolling processes, these will not be addressed in
this thesis. For the purposes of this example, it is sufficient to assume that some appropriate
technique has been employed to determine the distribution of friction coefficients. This
distribution is described in Table 4.5 and shown in Figure 4.11.
Table 4.5: Statistical Properties of the
Coefficient of Friction Distribution
Statistical Property
Mean (µ̄):
Variance (σ 2 ):
Skewness (γ1 ):
Excess Kurtosis (γ2 ):
Value
0.35
9e-4
0.7
0.2
Input Distribution
Prob. (%)
0.1
0.05
0
0.25
0.3
0.35
0.4
0.45
Coefficient of Friction, µf
0.5
Figure 4.11: Distribution of the Coefficient of Friction in a Flat Rolling Metalworking Process
48
4.5.2
Statistical System Model Output Prediction
Based on this distribution of µf , the first eight central moments were calculated using
Eq. (3.2). These moments, given in Table 4.6, are used to propagate statistical properties
from the friction coefficient distribution to predict a distribution for the maximum change
in thickness attainable with a single pass.
Table 4.6: Central Moments of the Distribution
of the Coefficient of Friction, µf .
Central Moment
1st Moment (µ1 ):
2nd Moment (µ2 ):
3rd Moment (µ3 ):
4th Moment (µ4 ):
5th Moment (µ5 ):
6th Moment (µ6 ):
7th Moment (µ7 ):
8th Moment (µ8 ):
Value
0
8.99e-4
1.88e-5
2.58e-6
1.45e-7
1.45e-8
1.21e-9
1.2e-10
The second-order prediction of the mean maximum reduction in thickness ∆H¯max was
calculated to be 12.5cm using Eq. (3.13). Equation (3.15) predicts a variance of 0.136cm2 .
Typically, higher-order statistics are not propagated and a Gaussian output distribution is
assumed. This Gaussian prediction is compared with actual system output in Figure 4.12.
This prediction only accounts for 53% of the actual system output distribution (i.e., 53%
overlap in the area under the predicted and actual probability density functions).
The methods presented in this chapter to propagate skewness and kurtosis, using
Eqs. (4.3) and (4.9), respectively, results in a more accurate prediction of the system output.
This predicted output, shown in Figure 4.13a, accounts for 93% of actual system outputs—a
large improvement over propagating a mean and variance alone.
49
Actual Output Distribution
0.2
0.15
0.15
Prob. (%)
Prob. (%)
Predicted Output Distribution from Mean/Var. Only
0.2
0.1
0.05
0.1
0.05
0
−0.05
0
0.05 0.1 0.15 0.2 0.25
Maximum Change in Thickness Per Pass, ∆ Hmax
0
−0.05
0
0.05 0.1 0.15 0.2 0.25
Maximum Change in Thickness Per Pass, ∆ Hmax
(a) Gaussian Prediction Using Mean and Variance
Only
(b) Actual System Output
Figure 4.12: Predicted Gaussian Output Distribution Obtained from Propagating a Mean
and Variance Only (a) Compared with Actual System Output (b)
Predicted Output Distribution from Mean, Var., Skew., & Kurt.
0.2
Actual Output Distribution
0.2
0.15
Prob. (%)
Prob. (%)
0.15
0.1
0.05
0.1
0.05
0
−0.05
0
0.05 0.1 0.15 0.2 0.25
Maximum Change in Thickness Per Pass, ∆ Hmax
0
−0.05
0
0.05 0.1 0.15 0.2 0.25
Maximum Change in Thickness Per Pass, ∆ Hmax
(a) Non-Gaussian Prediction Using Mean, Variance,
Skewness, and Kurtosis
(b) Actual System Output
Figure 4.13: Predicted Non-Gaussian Output Distribution Obtained from Propagating a
Mean, Variance, Skewness, and Kurtosis (a) Compared with Actual System Output (b)
4.5.3
Ramifications of Neglecting Higher-Order Statistics
Using the traditional approach where only a mean and variance are propagated and
a Gaussian distribution is assumed, the manufacturer would have only been able to reduce
the material thickness by a maximum of 3cm per pass, in order to achieve a 99.5% success
rate. However, using the methods presented in this chapter to also propagate higher-order
statistics, the manufacturer can confidently reduce the material thickness by 7.9cm per pass.
50
This reduces the number of passes required to achieve the desired plate thickness by over
two and a half times.
As this example clearly indicates, the benefits of propagating higher-order statistics
through a system model can be substantial.
4.6
Comments on Propagation of Higher-Order Statistics
The variance in a system’s output can easily be predicted using a first-order Taylor
series and knowledge of the input variance. However, having only a mean and variance and
lacking any additional information about the output distribution, system designers often
make the erroneous assumption that the output is Gaussian. This chapter has shown how
inaccurate that assumption can be, even for very simple functions. By following the methods
shown in this chapter, system designers can more fully describe an output distribution by
also propagating higher-order statistics, such as skewness and kurtosis, though a system
model.
While sufficient for many physical systems, the approach to higher-order statistical
error propagation presented in this chapter may not work for all types of system models,
such as state-space models, Laplace transforms, and differential equations. Additional work
is required to adapt the method presented for use with these types of models.
51
52
CHAPTER 5.
CONCLUSION
Error propagation and uncertainty analysis is a complex but important part of the
system design process. Accurately predicting a statistical probability density function for
system outputs can mean the difference between a successful system design and one that fails
to meet design objectives. The methods provided in this thesis allow the system designer to
accurately propagate fully-described statistical distributions through a system model without
incurring significantly greater computational costs.
The correction factors presented in this thesis for the efficient and accurate propagation of variance have only been determined for trigonometric, exponential, and logarithmic
functions. Additional work is required to determine correction factors for other nonlinear,
closed-form functions.
The author has based this research on the assumption that a closed-form, analytical
equation is necessary to use the methods presented in this thesis. However, it is the author’s
belief that this same approach can be applied to any model (closed-form or not) where outputs can be obtained from given inputs and derivatives can be numerically evaluated. While
non-closed-form models are outside the scope of this research, the author still believes the
correction factors presented in Chapter 3 can be determined for non-closed-form equations
if the truncation error is predictable, outputs are attainable, and derivatives can be determined. Additional work is required to determine what characteristics the model must have
in order to have a predictable truncation error.
If these methods could be used with models that are not closed-form, it would greatly
increase the quantity and variety of models this research applies to. This would allow the
designer to use these methods for more real-world engineering problems, including finiteelement models, dynamic models, transfer functions, state-space models, differential equations, software packages, “black-box” models, and others.
53
The formulas for propagating skewness and kurtosis that have been developed in this
thesis have produced very accurate estimates of actual output skewness and kurtosis in the
examples used in this thesis. However, it is possible that these estimates may not be perfectly
accurate for other nonlinear system models. If this is true, then additional work is required
to determine if correction factors can be determined that achieve higher-order accuracy in
skewness and kurtosis propagation without increased computational cost.
Lastly, future work could make the correction factors and skewness and kurtosis
propagation formulas more accessible to design engineers. This could involve creating a
table to be published online and/or included in textbook appendices.
54
REFERENCES
[1] Murphy, B., 2009. “Early verification and validation using model-based design.” EDN,
54, July, pp. 39–41. 1
[2] Poolla, K., Khargonekar, P., Tikku, A., Krause, J., and Nagpal, K., 1994. “A timedomain approach to model validation.” IEEE Transactions on Automatic Control,
39(5), May, pp. 951–959. 1, 5
[3] Hamaker, H. C., 1995. “Relative merits of using maximum error versus 3(sigma) in
describing the performance of laser-exposure reticle writing systems.” Proc. SPIE, 2440,
May, p. 550. 1, 5
[4] Vardeman, S. B., 1994. Statistics for Engineering Problem Solving. ISBN: 0-534-92871-4.
PWS Publishing Company, Boston, MA. 1, 4, 13, 14, 18
[5] Koch, P. N., 2002. “Probabilistic design: Optimizaing for six sigma quality.” In 43rd
AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Paper Number AIAA-2002-1471. 3
[6] Glancy, C., 1999. “A second-order method for assembly tolerance analysis.” In Proceedings of the 1999 ASME Design Engineering Technical Conferences, no. DAC-8707
in DETC99, ASME/DETC. 3, 10
[7] Kiureghian, A. D., 1996. “Structural reliability methods for seismic safety assessment:
A review.” Engineering Structures, 18(6), June, pp. 412–424. 4
[8] Thanedar, P. B., and Kodiyalam, S., 1992. “Structural optimization using probabilistic constraints.” Structural and Multidisciplinary Optimization, 4, pp. 236–240
10.1007/BF01742750. 4
[9] Melchers, R. E., 1999. Structural Reliability: Analysis and Prediction. Ellis Horwood
Series in Civil Engineering. John Wiley & Sons, New York. 4
[10] Parkinson, A., Sorensen, C., and Pourhassan, N., 1993. “A general approach for robust
optimal design.” Journal of Mechanical Design, 115(1), March. 4
[11] Chen, W., Wiecek, M. M., and Zhang, J., 1999. “Quality utility - a compromise programming approach to robust design.” Journal of Mechanical Design, 121, June. 4
[12] Chen, W., Sahai, A., Messac, A., and Sundaraj, G. J., 2000. “Exploring the effectiveness
of physical programming in robust design.” Journal of Mechanical Design, 122(2), June,
pp. 155–163. 4
55
[13] Su, J., and Renaud, J. E., 1997. “Automatic differentiation in robust optimization.”
AIAA Journal, 35(6), pp. 1072–1079. 4
[14] Taguchi, G., 1992. Taguchi on Robust Technology Development: Bringing Quality Engineering Upstream. Asme Press Series on International Advances in Design Productivity.
ASME Press, New York, August. 4
[15] Messac, A., and Ismail-Yahaya, A., 2002. “Multiobjective robust design using physical
programming.” Structurla and Multidisciplinary Optimization, 23(5), pp. 357–371. 4
[16] Oberkampf, W. L., DeLand, S. M., Rutherford, B. M., Diegert, K. V., and Alvin, K. F.,
2002. “Error and uncertainty in modeling and simulation.” Reliability Engineering &
System Safety, 75, March, pp. 333–357. 4
[17] Halton, J. H., 1960. “On the efficiency of certain quasi-random sequences of points
in evaluating multi-dimensional integrals.” Numerische Mathematik, 2, pp. 84–90
10.1007/BF01386213. 4
[18] Hammersley, J. M., 1960. “Monte carlo methods for solving multivariate problems.”
Annals of the New York Academy of Sciences, 86, May, pp. 844–874. 4
[19] Hutcheson, R. S., and McAdams, D. A., 2010. “A hybrid sensitivity analysis for use in
early design.” Journal of Mechanical Design, 132(11), November, p. 111007. 4, 17, 38
[20] Owen, A. B., 1998. “Latin supercube sampling for very high-dimensional simulations.”
ACM Transactions on Modeling and Computer Simulation, 8(1), January. 4
[21] Larson, B., Anderson, T. V., and Mattson, C. A., 2010. “System behavioral model
verification for concurrent design and modeling.” In 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference Proceedings, no. 9104 in AIAA-2010,
AIAA/ISSMO. 5, 42
[22] Evans, J. W., Zawadzki, R. J., Jones, S. M., Olivier, S. S., and Werner, J. S., 2009.
“Error budget analysis for an adaptive optics optical coherence tomography system.”
Optics Express, 17(16), August, pp. 13768–13784. 5
[23] Hamaker, H. C., 1995. “Improved estimates of the range of errors on photomasks using
measured values of skewness and kurtosis.” Proc. SPIE, 2621(198 (1995)), March. 5
[24] Oschmann, J., 1997. Gemini system error budget plan, January. 5
[25] Samet, H., 2005. Foundations of Multidimensional and Metric Data Structures (The
Morgan Kaufmann Series in Computer Graphics and Geometric Modeling). Morgan
Kaufmann Publishers Inc., San Francisco, CA, USA. 6
[26] Hayes, B., 2003. “A lucid interval.” American Scientist, 91(6), November-December,
pp. 484–488. 6
[27] Box, G. E., and Tiao, G. C., 1992. Bayesian Inference in Statistical Analysis. ISBN:
9780471574286. Wiley, Hoboken, NJ, April. 6
56
[28] research.microsoft.com, 2011.
http://goo.gl/rSJCD, July. 6
Basics of bayesian inference and belief networks
[29] Box, G. E. P., and Wilson, K. B., 1951. “On the experimental attainment of optimum
conditions.” Journal of the Royal Statistical Society. Series B (Methodological), 13(1),
pp. 1–45. 6
[30] Chen, W., Baghdasaryan, L., Buranathiti, T., and Cao, J., 2004. “Model validation via
uncertainty propagation and data transformations.” AIAA Journal, 42(7), p. 14061415.
6
[31] Lombardi, M., and Haftka, R. T., 1998. “Anti-optimization technique for structural
design under load uncertainties.” Computer Methods in Applied Mechanics and Engineering, 157(1-2), pp. 19–31. 7
[32] Jackson, P. S., 1982. “A second-order moments method for uncertainty analysis.” IEEE
Transactions on Reliability, R-31(4), Octobber, pp. 382–384. 10
[33] Shapiro, S. S., and Gross, A. J., 1981. Statistical Modeling Techniques. Marcel Dekker,
Inc., New York and Basel. 10
[34] Walwyn, R., 2005. Moments - Encyclopedia of Statistics in Behavioral Science., Vol. 3
John Wiley and Sons, Ltd, 111 River St. Hoboken New Jersey United States. 11
[35] Taylor, B., 1715. Methodus Incrementorum Directa et Inversa [Direct and Reverse Methods of Incrementation]. Proposition VII, Theorem 3, Corollary 2. London Translated
into English in D. J. Struik, A Source Book in Mathematics 1200-1800 (Cambridge,
Massachusetts: Harvard University Press, 1969), pages 329-332. 11
[36] Mattson, C. A., and Messac, A., 2002. “A non-deterministic approach to concept
selection using s-pareto frontiers.” Proceedings of ASME DETC, 2(DETC2002/DAC34125), September, pp. 859–870. 13, 16, 17, 38
[37] Tellinghuisen, J., 2001. “Statistical error propagation.” Journal of Physical Chemistry
A, 105b(15), March, pp. 3917–3921. 13, 18
[38] Lindberg, V., 2000. Uncertainties and error propagation - part i of a manual on uncertainties, graphing, and the vernier caliper Internet, July. 13
[39] Julier, S., Uhlmann, J., and Durrant-Whyte, H. F., 2000. “A new method for the
nonlinear transformation of means and covariances in filters and estimators.” IEEE
Transactions on Automatic Control, 45(3), March, pp. 477–482. 16, 38
[40] Putko, M. M., III, A. C. T., Newman, P. A., and Green, L. L., 2002. “Approach for
uncertainty propagation and robust design in cfd using sensitivity derivatives.” Journal
of Fluids Engineering, 124(1), March, pp. 60–69. 16
[41] Hamel, J., Li, M., and Azarm, S., 2010. “Design improvement by sensitivity analysis
under interval uncertainty using multi-objective optimization.” Journal of Mechanical
Design, 132(8), August, p. 081010. 16
57
[42] Goodman, L. A., 1960. “On the exact variance of products.” Journal of the American
Statistical Association, 55(292), December, pp. 708–713. 18
[43] Zhang, J., and Du, X., 2010. “A second-order reliability method with first-order efficiency.” Journal of Mechanical Design, 132(10), October, p. 101006. 18
[44] Aono, H., Shyy, W., and Liu, H., 2009. “Near wake vortex dynamics of a hovering
hawkmoth.” Acta Mechanica Sinica, 25(1), pp. 23–36. 28
[45] George, R. B., 2011. “Design and analysis of a flapping wing mechanism for optimization.” Master’s thesis, Brigham Young University, Provo, Utah, June. 28
[46] Mekid, S., and Vaja, D., 2008. “Propagation of uncertainty: Expressions of second and
third order uncertainty with third and fourth moments.” Measurement, 41(6), July,
pp. 600–609. 34
[47] von Hippel, P. T., 2005. “Mean, median, and skew: Correcting a textbook rule.” Journal
of Statistics Education, 13(2). 35
[48] Garner, W. B., 2009.
Model airplane propellers
rc.org/pdf/Model%20Propellers%20Article.pdf, March. 42
http://www.dc-
[49] Degarmo, E. P., Black, J. T., and Kohser, R. A., 2003. Materials and Processes in
Manufacturing., 9 ed. ISBN: 0-471-65653-4. Wiley. 47
[50] Tyfour, W. R. Rolling Internet (http://www.freewebs.com/tyfour/Rolling.doc). 47
[51] Lenard, J. G., 2007. Primer on Flat Rolling. ISBN: 0080453198. Elsevier Science. 47
58
APPENDIX A.
STEP-BY-STEP SOLUTION TO DUAL-PROPELLER, THREE
DEGREE-OF-FREEDOM HELICOPTER PROBLEM
Consider the compositional system model for a dual-propeller, three degree-of-freedom
helicopter discussed in Section 3.6. This model, given initally in Eq. (3.25), is repeated below
in Eq. (A.1).
θ̈ = C1 C2 + C3
(m2 L2 − m1 L1 )g
C1 =
m1 L21 + m2 L22 + Jy
C2 = cos θ
L1 km (uL + uR )
C3 =
m1 L21 + m2 L22 + Jy
(A.1)
Recall that variation exists in variables m1 , m2 , L1 , L2 , Jy , and km , and θ. The means
and standard deviations for these variables are repeated in Table A.1.
Table A.1: Design Parameters for Dual-Propeller, 3-DOF Helicopter
Design Parameter
Mean
m1
0.891 (kg)
m2
1.000 (kg)
L1
0.850 (m)
L2
0.3048 (m)
Jy
0.0014 (kg· m2 )
km
0.0546
θ
0
59
Std. Dev.
10−4 (kg)
10−4 (kg)
10−4 (m)
10−4 (m)
10−5 (kg· m2 )
10−5
1◦
A.1
Variance Propagation Through Component 1
All first-order partial derivatives of component C1 , evaluated at the input means, are
given in Eq. (A.2).
∂C1
∂m1
∂C1
∂m2
∂C1
∂L1
∂C1
∂L2
∂C1
∂Jy
−gL1 (m1 L21 + m2 L22 + Jy ) − gL21 (m2 L2 − m1 L1 )
= −5.4096
(m1 L21 + m2 L22 + Jy )2
gL2 (m1 L21 + m2 L22 + Jy ) − gL22 (m2 L2 − m1 L1 )
= 4.8085
(m1 L21 + m2 L22 + Jy )2
−gm1 (m1 L21 + m2 L22 + Jy ) − 2gm1 L1 (m2 L2 − m1 L1 )
= 0.5020
(m1 L21 + m2 L22 + Jy )2
gm2 (m1 L21 + m2 L22 + Jy ) − 2gm2 L2 (m2 L2 − m1 L1 )
= 18.2601
(m1 L21 + m2 L22 + Jy )2
−g(m2 L2 − m1 L1 )
= 8.1501
(m1 L21 + m2 L22 + Jy )2
=
=
=
=
=
(A.2)
The first-order Taylor series approximation from Eq. (3.9) is used to propagate variance through this component, as given in Eq. (A.3).
2
σC1
≈
2
n X
∂C1
i=1
∂xi
2
σx2i
∂C1
2
≈
σm1
+
∂m1
≈ 3.8673e-6
∂C1
∂m2
2
2
σm2
+
∂C1
∂L1
2
2
σL1
+
∂C1
∂L2
2
2
σL2
+
∂C1
∂Jy
2
2
σJy
(A.3)
The mean value for component C1 is -6.0152, which was determined by evaluated C1
at the input means.
A.2
Variance Propagation Through Component 2
All first- and second-order partial derivatives of component C2 , evaluated at the input
means, are given in Eq. (A.4).
60
∂C2
= − sin(θ) = 0
∂θ
∂ 2 C2
= − cos(θ) = −1
∂θ2
(A.4)
The second-order Taylor series approximation from Eq. (3.17) is used to propagate
variance through this component, as given in Eq. (A.5). Note that the variance must be
expressed in radians.
2
σC2
2
2
∂C2
1 ∂ 2 C2
2
≈
σθ +
σθ4
∂θ
2 ∂θ2
≈ 4.6396e-8
(A.5)
A correction factor e is then calculated using Eq. (3.19, as shown in Eq. (A.6).
1
2
1 + 1.022σC2
= 0.9997
e =
(A.6)
The resulting variance in component C2 is given by Eq. (A.7).
2
2
≈ σC2
e
σC2,CF
≈ 4.6381e-8
(A.7)
The mean value for component C2 is 1, which was determined by evaluated C2 at the
input means.
61
A.3
Variance Propagation Through Component 3
All first-order partial derivatives of component C3 , evaluated at the input means, are
given in Eq. (A.8).
∂C3
∂m1
∂C3
∂m2
∂C3
∂L1
∂C3
∂L2
∂C3
∂Jy
∂C3
∂km
=
=
=
=
=
=
−L31 km (uL + uR )
= −5.5401
(m1 L21 + m2 L22 + Jy )2
−L1 L22 km (uL + uR )
= −0.7124
(m1 L21 + m2 L22 + Jy )2
km (uL + uR )(m1 L21 + m2 L22 + Jy ) − 2m1 L21 km (UL + uR )
= −4.9566
(m1 L21 + m2 L22 + Jy )2
−2m2 L1 L2 km (uL + uR )
= −4.6744
(m1 L21 + m2 L22 + Jy )2
−L1 km (uL + uR )
= −7.6680
(m1 L21 + m2 L22 + Jy )2
L1 (uL + uR )(m1 L21 + m2 L22 + Jy )
= 103.6514
(A.8)
(m1 L21 + m2 L22 + Jy )2
The first-order Taylor series approximation from Eq. (3.9) is used to propagate variance through this component, as given in Eq. (A.9).
2
σC3
≈
2
n X
∂C3
i=1
∂xi
2
σx2i
2
2
2
∂C3
∂C3
∂C3
∂C3
2
2
2
2
≈
σm1 +
σm2 +
σL1 +
σL2
∂m1
∂m2
∂L1
∂L2
2
2
∂C3
∂C3
2
2
+
+
σJy
σkm
∂Jy
∂km
≈ 1.8564e-6
(A.9)
The mean value for component C3 is 5.6594, which was determined by evaluated C3
at the input means.
62
A.4
Variance Propagation Through System-Level Compositional Model
All first-order partial derivatives of the system model, evaluated at the input means,
are given in Eq. (A.10).
∂ θ̈
= C2 = 1
∂C1
∂ θ̈
= C1 = −6.0152
∂C2
∂ θ̈
= 1
∂C3
(A.10)
Now that the variance and mean of each component has been determined, the firstorder Taylor series approximation from Eq. (3.9) is used to propagate variance through the
system-level compositional model, as given in Eq. (A.11).
σθ̈2 ≈
∂ θ̈
∂C1
!2
2
σC1
+
∂ θ̈
∂C2
≈ 7.4019e-6
!2
2
σC2
+
∂ θ̈
∂C3
!2
2
σC3
(A.11)
Actual variance in θ̈, as determined from a Monte Carlo simulation with 10 million
data points, is 6.5979e-6, meaning this compositional system model with a correction factor
has 12% relatvie error. By comparison, the estimate of variance propagation using strictly a
first-order Taylor series (without the compositional model or a correction factor) is 4.4853e-6
(32% relative error).
As explained in Chapter 3, this increase in accuracy comes with very little additional
computational cost.
63