Faculty of Engineering,
The University of Guilan
Robust Pareto Design of GMDH-type Neural
Networks for Systems with Probabilistic Uncertainties
N. Nariman-zadeh, F. Kalantary, A. Jamali, F. Ebrahimi
Introduction
•System identification techniques are applied in many fields in
order to model and predict the behaviors of unknown and/or very
complex systems based on given input-output data
•GMDH is a self-organizing approach by which gradually
complicated models are generated based on the evaluation of their
performances on a set of multi-input-single-output data
•In order to obtain more robust models, it is required to consider all
the conflicting objectives, namely, training error (TE), prediction
error (PE) in the sense of multi-objective Pareto optimization
process
•For multi-objective optimization problems, there is a set of
optimal solutions, known as Pareto optimal solutions or Pareto
front
Modelling Using GMDH-type Networks
•System Identification Techniques Are Applied in Many Fields in
order to Model and Predict the Behaviors of Unknown and/or Very
Complex Systems Based on Given Input-Output Data.
•Group Method of Data Handling (GMDH) Algorithm is SelfOrganizing Approach by which Gradually Complicated Models are
Generated Based on the Evaluation of their Performances on a set
of Multi-Input-Single-Output Data Pairs (i=1, 2, …, M)
X1
Y1
X2
.
.
Xn
Ym
Modelling Using GMDH-type Networks
The classical GMDH algorithm can be represented as set of neurons
in which different pairs of them in each layer are connected through
a quadratic polynomial and thus produce new neurons in the next
layer.
Hidden Layer(s)
Input Layer
X1
G1
G4
X2
G6
G2
Output Layer
X3
X4
G3
G5
A Feedforward GMDH-Type Network
Application of Genetic Algorithm in the Topology Design of GMDH-type NNs
a
ad
adbc
b
a d b c b c b c
c
bc
d
A Generalized GMDH Network Structure of a Chromosome
Length of
Neuron.  2
Hidden Layer
Application of Genetic Algorithm in the Topology Design of GMDH-type NNs
a
ad
adbc
b
c
bc
d
a d b c d d d d
Crossover operation for two individuals in GS-GMDH networks
Application of Singular Value Decomposition
to the Design of GMDH-type Networks
SVD is the method for solving most linear least squares problems that
some singularities may exist in the normal equations A a  Y
The SVD of a matrix, A  RM N, is a factorization of the matrix into the
U  R M N
product of three matrices, matrix
, diagonal matrix
with non-negative elements (Singular Values), and
W  R N N
orthogonal matrix V  R N N such that :
A  U .W .V T

1  T
a  V .diag( ).U .Y
w j 

Genetic Algorithms and Multi-objective Pareto Optimization
Genetic algorithms are iterative
and stochastic optimization
techniques.
In the optimization of complex real-world
problems, there are several objective functions
to be optimized simultaneously.
There is no single optimal solution as the best
because objectives conflict each other.
There is a set of optimal solutions, well known
as Pareto optimal solutions or Pareto front.
Prediction error
Multi-objective optimization
Modelling error
Prediction error
Multi-objective optimization
Modelling error
Robust optimal solution
Optimal solution
Design Variable
Feasible
Objective Function
Infeasible
Difference between robust optimization and traditional optimization
Stochastic Robust Analysis
FX x   Pr X  x    f X x dx
x
p

EX  
 2 X  


 xdFX x   f X xdx


x  EX  f X x dx
f X x 
PDF
1.00
CDF
0.75
0.50
For the discrete sampling:
1
EX  
N
N

xi
i 1
N
1
xi  EX 2
 X  
N  1 i 1
2

0.25
Random variable
•Modelling and prediction of soil shear strength, Su , based on 5 input
parameters, namely, SPT number (Standard Penetration Test) N′, effective
overburden stress s/0, moisture content percent W , LL liquid limit, and PL
plastic limit of fine-graded clay soil
•The data used in this study were gathered from the National Iranian
Geotechnical Database, which has been set up in the Building and Housing
Research Centre (BHRC)
•The database has been established under a mandate from the Management
and Planning Organization (MPORG), which supervises the professional
activities of all of the consultancy firms in Iran
Training set
Prediction set
Comparison of actual values with the evolved GMDH model corresponding to
optimum point C (nominal table)
Objective functions and structure of networks of different optimum design points
Point
Network’s structure
TE
PE
Mean of TE
Mean of PE
Variance of TE
Variance of PE
A
bbaebcacbcaeacee
133.12
48.49
323.76
161.49
174862.64
42019.59
B
bcaebacdbcbbadde
79.20
260.15
73785.2
17844.7
3.8e11
3.3e9
C
bcaebccdbdbcaccd
89.79
75.30
28366.5
709.8
3.7e10
2.6e6
Objective functions and structure of networks of different optimum design points
Point
Network’s structure
TE
PE
Mean of
TE
Mean of PE
Variance of TE
Variance of PE
A
bbaebcacbcaeacee
133.12
48.49
323.76
161.49
174862.64
42019.59
B
bcaebacdbcbbadde
C
bcaebccdbdbcaccd
79.20
89.79
260.15
75.30
73785.2
28366.5
17844.7
709.8
3.8e11
3.7e10
3.3e9
2.6e6
D
abeecddd
132.79
237.59
234 .61
248.03
178.77
1174.283
Y1
Y3
Y5
Y2
Y4
Point C
Point D
The structure of network corresponding to point C and D
Y1=-5.94+ 0.65 N’ + 0.76 σ0’ -0.0083 N’2 - 0.0019 σ0’2 + 0.0013 N’ σ0’
Y2= 25.42 - 2.76w + 1.86LL - 0.019w2 - 0.045LL2 + 0.11w(LL)
Y3= 16.99 + 0.82Y2 - 1.27LL - 0.0015Y22 + 0.016(LL)2 + 0.015(Y2)(LL)
Y4= 10.16 + 0.74Y1 - 0.22PL - 0.019Y12 - 0.034PL2 + 0.056(Y1)(PL)
Y5= 16.12 + 0.83Y4 - 0.64Y3 - 0.0004Y42 + 0.0060Y32+ 0.0036(Y4)(Y3)
Conclusion
• A multi-objective genetic algorithm was used to optimally
design GMDH-type neural networks from a robustness point
of view in a probabilistic approach.
• Multi-objective optimization of robust GMDH models led to
the discovering some important trade-off among those
objective functions.
• The framework of this work is very promising and can be
generally used in the optimum design of GMDH models in
real-world complex systems with probabilistic uncertainties.
Thanks for your attention…