System Identification using Delaunay
Tessellation of Self-Organizing Maps
Janos Abonyi and Ferenc Szeifert
University of Veszprem, Department of Process Engineering, P.O. Box 158,
H-8201, Hungary // www.fmt.vein.hu/softcomp, [email protected]
Abstract. The Self-Organizing Map (SOM) is a vector quantization method which
places prototype vectors on a regular low-dimensional grid in an ordered fashion.
A new method to obtain piecewise linear models of dynamic processes is presented.
The operating regimes of the local linear models are obtained by the Delaunay
tessellation of the codebook of the SOM. The proposed technique is demonstrated
by means of the identification of a pH process.
1
Introduction
Artificial neural networks have successfully been used to build models directly
based on process data. The Self-Organizing Map is one of the most popular
neural network models. The SOM algorithm performs a topology preserving
mapping from high dimensional space onto a two dimensional grid of neurons
so that the relative distances between data points are preserved. The net
roughly approximates the probability density function of the data and, thus,
serves as a clustering tool [1]. It also has the capability to generalize, i.e.
the network can interpolate between previously encountered inputs. As SOM
provides a compact representation of the data distribution, it has been widely
applied in analysis and visualization of high-dimensional data [1].
In this paper a new method to use SOM for the identification of dynamic
systems is presented where SOM is used to partition the input space of piecewise linear models. As the input–output map of such a model is piecewise
linear, local linearization and inversion algorithms can be easily derived and
the model is computationally cheap to evaluate [2]. This partitioning is obtained by the Delaunay tessellation of the neurons (also called codebook) of
the SOM. A similar approach based on the application of Voronoi diagrams of
SOM has already been suggested in the context of time series prediction [3].
Furthermore, the Delaunay-based multivariable spline approximation from
scattered samples of an unknown function has proven to be an effective tool
for classification [4]. Recently, Delaunay Networks were introduced to represent interpolating models and controllers [5]. Usually, the identification of
these models is divided into two tasks: structure identification that generates the partition of the input space and parameter identification of the local
models. To obtain a suitable partition iterative model building algorithms
have been worked out [4,6], where in every iteration an additional node is
inserted into the model by a forward selection algorithm and the parameter
vector is re-estimated by standard least-squares method.
The aim of the paper is to present a new, one-step approach for the identification of such piecewise linear models. The main idea is to quantize the
available input-output data to get a set of neurons and use the obtained
neurons (codebooks) as model parameters. The paper is organized as follows. Section 2 presents the structure of the problem of the identification
of piecewise linear models based on Self-Organizing Maps (SOM). Section 3
presents the Delaunay tessellation based identification of the model. A detailed case study is given to illustrate the proposed approach on the modeling performance in Section 4. The process under study is a pH neutralization reactor. Conclusions are given in Section 5. The proposed approach has
been implemented in MATLAB and the software can be downloaded from
www.fmt.vein.hu/softcomp.
2
Self-Organizing Map of Dynamical Systems
The Nonlinear AutoRegressive with eXogenous input (NARX) model establishes a nonlinear relation between the past inputs and outputs and the pre
dicted output: y(k + 1) = f y(k), . . . , y(k − ny ), u(k − nd ), . . . , u(k − nu ) .
Here, ny and nu denote the maximum lags considered for the output, and
input terms, respectively, nd < nu is the discrete dead time, and f represents the mapping of the model. Our goal is to develop a data-driven algorithm for the identification of this model in the form yk = f (z k ), where
yk = y(k + 1), z k = [zk,1 , . . . , zk,n ]T is based on the measured input and
output data,z k = [y(k), . . . , y(k − ny ), u(k − nd ), . . . , u(k − nu )], and the yk
output of the model is identical to the one-step ahead prediction of the output of the system, yk = y(k + 1), where k = 1, . . . , N denotes the index of
the k-th input-output data-pair, xk = [zTk , yk ].
The piecewise linear models are special case of operating regime based
models. If we denote the input space of the model by T : z ∈ T ⊂ Rn ,
the piecewise linear model consists of a set of operating ranges T1 , T2 , . . . , Ts
which satisfy T1 ∪ T2 ∪ · · · ∪ Ts = T and Tj ∩ Ti = when i 6= j. Hence, the
model can be formulated as If z k ∈ Ti then yk = [z Tk 1]θ i where θ s denotes
the parameter estimate vector used in the ith local model.
The SOM algorithm performs a topology preserving mapping from high
dimensional space onto map units so that relative distances between data
points are preserved. The map units, or neurons, form usually a two dimensional regular lattice. Each neuron i of the SOM is represented by an
l-dimensional weight (l = n + 1), or model vector mi = [mi,1 , . . . , mi,l ]T .
These weigh vectors of the SOM form a codebook. The neurons of the map
are connected to adjacent neurons by a neighborhood relation, which dictates the topology of the map. The number of the neurons determines the
granularity of the mapping, which affects the accuracy and the generalization
capability of the SOM.
SOM is a vector quantizer, where the weights play the role of the codebook
vectors. This means, each weigh vector represents a local neighborhood of
the space, also called Voronoi cell. The response of a SOM to an input x
is determined by the reference vector (weight) m0i which produces the best
match of the input i0 = mini kmi − xk, where i0 represents the index of
the Best Matching Unit (BMU). During the iterative training of SOM, the
SOM forms an elastic net that folds onto ”cloud” formed by the data. The
net tends for approximate the probability density of the data: the codebook
vectors tend to drift there where the data are dense, while there are only a
few codebook vectors where the data are sparse. The training of SOM can be
(k+1)
(k)
accomplished generally with a competitive learning rule as mi
= mi +
(k)
ηΛi0 ,i (x − mi ) where Λi0 ,i is a spatial neighborhood function and
η is the
learning rate. Usually, the neighborhood function is Λi0 ,i = exp
kr i −r 0i k2
2σ 2(k)
where kr i −r 0i k represents the Euclidean distance in the output space between
the i-th vector and the winner. Both the neighborhood function and the
learning rate should be scheduled during learning to achieve good modeling
performance [1].
3
Delaunay Tessellation based Modeling
In the proposed framework, the operating regions are simplices defined by
characteristic points (also called nodes) pj = [pj,1 , pj,2 , . . . , pj,n ]T , with j =
1, . . . , np , where np is the number of nodes. To get a unique and efficient
partition, the input space is partitioned by Delaunay tessellation of the characteristic points. In the following, some definitions are given as they are
necessary for understanding this method. An n-polytope is the smallest convex set containing a finite set of points in Rn . An n-simplex is a polytope
containing n+1 linearly independent points in Rn . The Delaunay tessellation
of a set of points is a set of n-simplices such that the circumsphere of each
simplex does not contain other points in its interior. For example, in two
dimensions, three points form a simplex obtained by Delaunay triangulation
if and only if the circle that is determined by these points does not contain
any other point of that set. Consequently, the bounding spheres of the simplices are as small as possible, and the obtained triangles are as equilateral
as possible. It has been proven that among all mappings with a given bound
on their second derivative, the piece-wise linear approximation based on the
Delaunay triangulation (also called Delaunay tessellation) has the smaller
worst case error of all triangulations [7]. The simplices are represented by
the connectivity matrix V = [vs,j ], whose sth row contains the indices of the
points that form simplex Ts : Ts = conv(pvs,1 , . . . , pvs,n+1 ). For z ∈ Ts , the
membership function Ai (z) is defined through the barycentric coordinates
bs = [bs,1 , . . . , bs,n+1 ] of z: bs = Zs−1 z 0 , where z 0 is the extended regression
vector, z 0 = [z1 , . . . , zn , 1]T , and
pvs,1 · · · pvs,n+1
Zs =
.
1 ···
1
Because of the piecewise linear nature of the model, the model response can be
computed by linear interpolation of the d = [d1 , . . . , di . . . , dns ]T parameters
n+1
P
bs,j dvs ,j .
assigned to each characteristic point, pi , y =
j=1
The barycentric coordinates can also be used to determine in which simplex the observation is. If z ∈ Ts all the barycentric coordinates of z are
positive: bs,k > 0, ∀k = 1, . . . , n + 1. The main idea of the paper is to quantize the available x = [z T y]T input-output data to get a set of nr prototypes
M={m1 , . . . , mnr } and use the obtained codebooks as model parameters,
pi = [mi,1 , . . . , mi,n ]T , and di = mi,n+1 .
4
Application to a pH process
The prediction of the pH (the concentration of hydrogen ions) in a continuous stirred tank reactor (CSTR) is a well-known benchmark problem with
nonlinear dynamics. The dynamic model for the pH in the tank is given in
[8]. As the process can be modeled as a first-order dynamic system [8], the
model has the following form: pH(k + 1) = f (pH(k), FN aOH) . The SOM of
the process is shown in Fig. 1 where a 6 × 6 map has been used to map
the typical operating points of the process. The neurons of SOM containing
Fig. 1. SOM of the pH process
steady state operation data are marked by black hexagons with proportional
size to the number of steady state data in the operating regions of the neurons. It can be seen in Fig. 1 that the s-shaped titration curve, also shown in
Fig. 2, has also an s-shape rotated in the two-dimensional grid of SOM. This
fact nicely illustrates the distance preserving mapping property of SOM. The
resulting Delaunay tessellation based input partitions are shown in Fig. 2
along with the steady state behaviour of the process and the obtained model.
As the model is piece-wise linear, every simplex defines a local linear dynamic model. Hence it is possible to extract knowledge about the dynamic
properties of these local models. As an example of knowledge extraction, the
stability of the local models is evaluated [9,2] and depicted in Fig. 2. Unstable simplices are denoted by dashed-dotted lines (− · −). One can see that
the model correctly represents the dynamic properties of the system as it
contains stable local models in the steady state region of the process and
the steady state behaviour of the local models are near to the steady state
(titration) curve of the process.
11
10.5
10
9.5
pH(k)
9
8.5
8
7.5
7
6.5
512
514
516
518
F
(k)
520
522
524
526
NaOH
Fig. 2. Delaunay input partition of the model, stable simplex (—),unstable simplex
(− · −), steady-state of the model (− −), steady-state of the process (—).
Recurrent simulation with a separate validation data set was used to
evaluate the differences in the modeling performance of the models. The
performance was measured by the mean squared prediction error (MSE).
The results show that the models are much more accurate than the linear
model with M SE = 0.777. When the operating regions are constructed by
Voronoi diagram of the codebook, the parameters of the local are identified
by standard least squares (LS) method (M SE = 0.245). In case of Delaunay
tessellation, there is no need to re-estimate the parameters of the local models
as the d vector is determined directly from M (M SE = 0.140). However, it
is possible to use SOM only for the partitioning the input space. In this case
d is re-estimated based on the baricentric coordinates of z k by standard LS
method. It is interesting to see that because of the large number of neurons
the SOM initialized Delaunay Network model can be overparameterized and
the LS estimated model could have bad generalization properties (M SE =
0.205).
5
Conclusions
A new piecewise linear modeling framework has been developed which employs non-overlapping operating regimes obtained by the Delaunay triangulation of the codebook of Self-Organizing Maps. For sake of illustration, in
this paper the proposed approach has been applied to an easily visualizable
first-order system (pH reactor). However, the presented tools can be applied
for higher-order multi-input multi-output processes in a straightforward manner. Our further research is to show that as obtained the model can be easily
incorporated into model based control algorithms.
Acknowledgement
The financial support of the Hungarian Ministry of Culture and Education (FKFP-0073/2001) and the Hungarian Science Foundation (OTKA T
0373600) are greatly acknowledged. Janos Abonyi is grateful for the financial
support of the Janos Bolyai Research Fellowship of the Hungarian Academy
of Science.
References
1. T. Kohonen, The self-organizing map, Proceedings of the IEEE 78(9) (1990)
1464–1480.
2. J. Abonyi, R. Babuska, F. Szeifert, Fuzzy modeling with multivariate membership functions: Gray-box identification and control design, IEEE Trans. on
Systems, Man, and Cybernetics, Part B. 31(5) (2001) 755–767.
3. J. Principe, L. Wang, M. Motter, Local dynamic modleing with Self-Organizing
Maps and applications to nonlinear system identification and control, Proceedings of the IEEE 86(11) (1998) 2241–2258.
4. D. Cubanski, D. Cyganski, Multivariable classification through adaptive
Delaunay-based c0 spline approximation, IEEE Transactions on Pattern Analysis and Machine Intelligence 17 (1995) 403–417.
5. T. Ullrich, H. Tolle, Delaunay-based local model networks for nonlinear system identification, in: IASTED International Conference Applied Modelling and
Simulation, Banff, Canada, 1997.
6. M. Brown, T. Ullrich, Comparison of node insertion algorithms for delaunay networks, in: Proceedings of IMACS 2nd MATHMOD Conference, Wienna, 1997,
pp. 445–780.
7. S. M. Omohundro, The Delaunay triangulation and function learning, in: Technical Report 90-001, International Computer Science Institute Berkeley, California,
1989.
8. N. Bhat, T. McAvoy, Determining model structure for neural models by network
stripping, Computers and Chemical Engineering 16 (1992) 271–281.
9. J. Abonyi, R. Babuška, H. Verbruggen, F. Szeifert, Incorporating prior knowledge in fuzzy model identification, International Journal of Systems Science
31 (5) (2000) 657–667.