DAMTP /1993/NA6
Some algorithms for thin plate spline interpolation
to functions of two variables1
M.J.D. Powell
Abstract: A disadvantage of thin plate spline interpolation to n values of a
function of two variables is that all but a few elements of the interpolation matrix
are nonzero. Nevertheless, the calculation enjoys some structure that usually
allows the required coefficients to be computed in much less than O(n3) operations
for general positions of the interpolation points. In particular, linear combinations
of several thin plate spline terms can be replaced by a single truncated Laurent
expansion, when values of the linear combination are required at points that
are sufficiently far from the centres of the thin plate spline terms. Further, the
interpolation equations can be reduced to an (n - 3) x (n - 3) positive definite
system, in order to employ a pre-conditioned conjugate gradient method. These
techniques are described briefly and are applied to some numerical examples. vVe
find that our implementations of the ~!.~re1}1and .£2gJ.]g!~.<l£~~ent.~~Lh,,<?~
provide substantial reductions in the amount of computation when n 2:: 1000 and
n;:::: 100, respectively.
Department of Applied Mathematics and Theoretical Physics,
University of Cambridge,
Silver Street,
Cambridge CB3 9EvV,
England.
March, 1993.
1Presented at the International
(New Delhi, January, 1993)
Conference
on Advances
in Computational
Mathematics
3
Thin plate spline interpolation
1. Introduction
to thin plate spline interpolation
The general interpolation problem in two
dimensions is to pick a function s from
~~~--~_~;~':""~:"O7'~::::""
n2 to n that satisfies the equations
--------,,---'"'C'~d"
•. "-'·~·'·~
.'
- .•
S(~i)
i = 1, 2, ...
= Ii,
i
, n,
(1.1)
where the interpolation points L~i: = 1,2, ... , n} and the right hand sides
{Ii : = 1,2, ... , n} are given. All the interpolation points have to be different,
and we assume also that they do not lie on a straight line, which implies n ~ 3,
but we impose no other restrictions on their positions.
The survey papers of Sabin (1980) and Franke (1982) show that several choices
of s have been proposed. The method that seems to be used most in practice is
piecewise linear interpolation. Here the convex hull of the interpolation points in
R} is divided into triangles, such that {.f.i : = 1, 2, ... , n} is the set of vertices of
the triangles and no vertex is also the interior point of an edge of a triangle. Then
s is defined on each triangle by linear interpolation to the data at the vertices. It
follows that s is continuous across the edges, but we expect discontinuities to occur
in the first derivatives of s. The method is excellent when these discontinuities do
not matter. Indeed, it is usual in such cases for n to be large, and then it is highly
valuable that the amount of computation need only be of magnitude n log n.
On the other hand, when there are fewer than 100 interpolation points and the
curvature of the underlying function is important, then usually piecewise linear
interpolation is too crude. Further, for much larger values of n, the domain of
the data may be so extensive that curvature is important on each typical subdomain that contains about 100 interpolation points. Therefore there is a need
for interpolation algorithms that provide good smoothness when n = 104, say,
without an excessive amount of computation. Consequently, an algorithm that
requires O( n2) operations could be useful, but O( n3) complexity would probably
be unacceptable.
This demand can be met by interpolating functions s that are composed of
quadratic or cubic polynomial pieces. On the other hand, when n is such that
O( n3) operations are tolerable, then the use of thin plate splines is highly successful in several applications (see Barrodale, Kuwahara, Poeckert and Skea, 1992,
for example). Further, these interpolants have two strong advantages over piecewise polynomials. Specifically, there are no artificial divisions of the data into
subregions for the separate pieces of s, and thin plate spline interpolants provide
the solution to a very pertinent variational problem. Therefore, we are going to
__oL~~i~_l?!a.~~sp!ineinterpola,nts that may allow their
consider some ~~~§
coefficients to be calculated in f~:ver than,<2(1"!c~)operations.
In the variational problem, -;--1s required to have square integrable second
derivatives, and the freedom in s after satisfying the interpolation equations (1.1)
i
i
M.J.D. Powell
4
is taken up by minimizing the expression
V( s )=~
1<.2
where we reserve x and
that s has the form
Y
+ 2(82S(~))2
8X 8Y
8X 2
(82S(~))2
+
8Y 2~'
(82S(~))2d
( 1.~
'J)
for the components of ±. It is proved by Duchon (1977)
n
s(±)
=
L
Aj
II±-±j 112log II±-±j II + Co+CIX+C2Y,
(1.3)
j=l
the vector norm being Euclidean. Further, the coefficients
are constrained by the conditions
n
L
n
Aj
j=l
{Aj
:
j = 1,2, ... , n}
n
= 2: AjXj = 2: AjYj = 0,
j=l
(1.4)
j=l
where Xj and Yj are the components of ~j' in order that the integral (1.2) is finite.
We see that the equations (1.1) and (1.4) provide a square linear system in the
unknowns {Aj : = 1, 2, ... , n} and {Co, Cl, C2}'
We write this system in the form
j
(1.5)
j
Here <I>is n x n and has the elements {1I±i - ±j 112 log lI±i - ±j i, = 1,2, ... , n},
while P is n x 3, its i-th row being the vector (1 Xi yJ Further,~, ~ and fare
the vectors that have the components {Aj: j=1,2, ... ,n}, {Co,Cl,C2} and-{fi:
i = 1, 2, ... , n}, respectively. One can deduce from the above variational problem
that the (n + 3) x (n + 3) matrix of the system (1.5) is nQJ1~ipg~19:!'Therefore
the data define the function (1.3) .£,niqJMilir.This choice of s is the "thin plate
spline" interpolant, so our aim is to solve the equations (1.5) in fewer than O(n3)
operations.
Our work will depend on a property of <I>that we establish by integration by
parts. In order to begin this task, we note that the function
II
:
(1.6)
has the derivatives
"9 4qS(;~.)
2qS(;~:)
where
"92
=
= 487r(18(.f)
+ log lI.flD
.f E
1?..2
,
(1. 7)
},
is the Laplacian operator
(1.8)
V-
5
Thin plate spline interpolation
and where 8 is the delta function in
in Rn that satisfies the equations
and we introduce
R}.
n
n
Lej
j=l
= LejXj
j=l
Further, we let ft be any nonzero vector
n
= LejYj =
j=l
(1.9)
0,
the function
n
s(~)
= LOjll~-~jI12log
j=l
II~-~jll,
(1.10)
~ER2.
The conditions (1.9) imply that V(S) is finite, and two applications
by parts to the definition (1.2) give the identity
V( s)
= k2 s(~)
\7 4 s(~) d~
n
n
- ~n2 {L
L Oj8(.if.-.if.j)}
i=l Oill~-~iIl21og 11.;r.-3:.ill} j=l
{811"
n
811"
of integration
d~
n
L
L Oill.f.i-.f.j 112log lI~i-iJ
i=l j=l
OJ
=
811"
Ii? if> ft,
(1.11)
which depends on equations (1.6)-(1.10), the conditions (1.9) being important
because they imply that the boundary terms of the integrations by parts are zero.
Further ft =I 0 and the definition (1.2) imply that V( s) is positive. Thus we reach
the conclusion
Q=lO
and pTQ=O
:::;.
~T<I>Q>O.
(1.12)
Another consequence of formula (1.7) will also be useful later. It depends on
the remark that \7E.2rjJ(~-JlJ and \7y2rjJ(.;r._Jl) are identical, where \7 and \7y2 denote
the Laplacian operator applied to- functions of ~ and y, respectively. Specifically,
we will make use of the property
-
i
(1.13)
2. Advantages of conjugate gradient methods
Conjugate gradient methods are highly useful for solving systems of linear equations whose matrices are symmetric and positive definite, but the matrix of the
system (1.5) has some negative eigenvalues, because all its diagonal elements are
zero. On the other hand, expression (1.12) shows a conditional positive definite
property of if>. Fortunately, this property allows the calculation of ~ to be reduced
to the solution of an (n - 3) x (n - 3) positive definite system of linear equations.
Details of this construction are given by .-=._----_
Sibson .."and
Stone (1991), and they are
,.~-,.._...
,-_._-~._'
as follows.
.~._~<~_.,,-
M.J.D. Powell
6
The second half of the system (1.5) states that ~ is in the null space of P T,
which is an (n - 3)-dimensional subspace of nn. Therefore, if we let Q be any
nx(n-3) matrix whose columns span this subspace, then we can express ~ in the
form
(2.1)
for some vector f-l in nn-3. Further, we pre-multiply the first half of the system
(1.5), which is the equation <I>~+P.f= f, by QT. Thus, substituting for ~ and
noting that QTp is zero, we deduce the-;elation
(2.2)
The positive definiteness of Q T <I> Q is an elementary consequence of the property
(1.12). Therefore we recommend that f-l be calculated by applying a conjugate
gradient method to the system (2.2). Then formula (2.1) provides the vector .6..
Further, the three components of f can be derived from the identity Ps;.= f - <I>~j
which requires much less work than the calculation of ~ for n ~ 10.
There is a convenient way of £hoosi~4£. implicitly using l!£..1.!.§l.~b~der
tr.CLll~formati<2-!ls.Specifically, we define PI = P, and then, for j = 1,2,3, we calculate
:!l.j E nn such that the matrix
Pj+!
=
II~:II~Pj
( I - 2uou:f)
has zeros below the diagonal in its first
where n is the orthogonal matrix
j columns.
(2.3)
Thus n Tp is upper triangular,
(2.4)
+~
It follows that the last (n-3) columns of n span the null space of pT, so we let
Q have these columns. Consequently, QT<I>Q is the bottom right (n-3) x (n-3)
submatrix of n T <I> n. It can be calculated in O( n2) operations from expression
(2.4) after the components of :!l.1, :!l.2 and lli have been found, which requires only
O(n) operations. Further, the numbers on the right hand side of the system (2.2)
are just the last (n-3) components of nTL which can alsobe computed in O(n)
operations. It is important to efficiency that Q is not formed explicitly. Therefore
we write equation (2.1) as the product ~=np, where p is the vector in nn whose
components are three zeros followed by the c;-mponent-; of l!:.: Thus the calculation
of .6. from 1:: also has O( n) complexity. All of these details can be useful when n
is at most a few hundred, because then it may be best to generate the matrix
Q T <I> Q explicitly, in order to solve the system (2.2) by a direct method such as
Choleili- factorization.
'When n is large, however, then there are two ways in which a conjugate gradient procedure can improve substantially on the O(n3) work of a direct method.
7
Thin plate spline interpolation
Firstly, the number of operations of each iteration may be less than O(n2), and
may be
secondly the total number of iterations to achieve enough accuracy in
much less than n. The remainder of this section considers the first possibility,
while the number of iterations is the subject of Section 3.
The only task of an iteration of the conjugate gradient method that may require
more than O( n) operations when solving the system (2.2) is the multiplication
of a vector by the matrix QT<J? Q (see Golub and Van Loan, 1983, for instance).
As in the paragraph that includes equation (2.4), our choices of Q will be such
that the work of multiplying a vector by Q or by QT is only O(n). Therefore we
address the calculation of <J?~, where ~ is a general vector in nn. Fortunately, if
one can accept an approximation to this product whose accuracy is specified in
advance, then the following technique of Beatson and Newsam (1992) provides
the approximation in about O( n log n) operations.
The required components of <J?~ are the values of the function
f-L
n
s(~)
= L'\jll~-~jIl2log
j=l
II~-~jll,
(2.5)
i
at the interpolation points {~i : = 1,2, ... , n}. Therefore we seek a sufficiently
s say, to s such that the work of generating s is of
accurate approximation,
magnitude n log n, and such that the calculation of S(~i) for a typical ~i requires
only O(1og n) operations. The technique takes advantage of the smoothness of the
-;£jll : ;£ E n2} when;£ is bounded away from ;£j' Indeed,
term {1I;£-;£jIl2log 11;£
if .:T is a subset of the integers {I, 2, ... , n} such that the convex hull of the set
{.f.j :
E .:T} contains no other interpolation
points, then it can be helpful to
employ the approximation
j
.€J(~)
~
L "\jll~-;£i112log
jeJ
II~-~jll,
(2.6)
where .f:J is a truncated Laurent expansion of the right hand side about a suitable
point of the convex hull, ~ say. The error of this approximation is of magnitude
1I.f.~JII-T-l when ~ is sufficiently far from.f.J' where r is the order of the truncated
expanSIOn.
The method generates the coefficients of .€J for several sets.:T. In the ~Lrp,..gL~~,t
case, the dis2:~L~~tion <:>fdata points iSE~~:l~~Ei~~E1E:<?J:l~~~UIlit square, and a
k into four equafsquares
at level
recursive procedure divlcIes-afCsqu-aresatlevel
k+l, where ,k is the index of the recursion. The procedure begins by letting the
unit square be the only square at level k = 1, and it ends when each square at
Each set .:T contains the indices
level k+ 1 encloses ab~intYmQJ~t!()I!l?g!nts.
of all the interpolation points in a square at some level, and we include every set
.:T that can be constructed
in this way. Thus the number of different sets is about
n115, each interpolation point is in about log4(n/5) different sets, and the work
to calculate all the coefficients of all the Laurent expansions is bounded above by
a multiple of n log n if the order r is a constant.
!vI.J.D. Powell
8
n
€=0.30
10-6
27.30
0.68
4.43
3.21
55.4
625.00
1.00
7
1.97
0.40
0.99
25.00
7.83
0.25
6.25
9.73
18.76
100.00
Direct
E=
10-10
2000
1000
400
10000
4000
200
Table 1:
Timings in seconds for a Laurent approximation
to
<I>l
When an estimate of the function value SC~i) is required, we ask whether the
approximation
(2.6) is sufficiently accurate when .:J is the set at level one that
contains all the indices {I, 2, ... , n}. Almost certainly the answer is negative,
but this is a convenient way to begin a recursive procedure that provides s(;r;).
Indeed, for each set .:J that occurs in the recursion, we either employ £.:T(~i) if it
is accurate enough, or we introduce the four descendants of .:J at the next level
into the procedure, except that the final level may have been reached. Then all
terms of the sum (2.5) that have not yet been covered by Laurent expansions are
calculated explicitly. One can deduce that, if the required accuracy is a given
positive constant, then usually the number of sets that occur is of magnitude
log n. Further, the number of terms that have to be treated explicitly tends to
be independent of n for large n. Thus the complexity of calculating S(;ri) by this
method is only about O(1ogn).
A more sophisticated procedure is needed to achieve the required complexity
when the distribution of the interpolation points is highly irregular. This problem
is addressed by Powell (1992), and he gives details of a truncated Laurent expanto prescribed accuracy for general positions of
sion algorithm that calculates
the interpolation points. He includes execution times of a Fortran implementation
of the algorithm, when the points {~i : = 1, 2, ... , n} are very close to a square
grid and all components of.l are one. Some of his timings are shown in the second
and third columns of Table 1, where € is the prescribed tolerance on the error of
the approximation s ~ s. Further, the last column of the table gives the times that
are required to compute all of the numbers {sC~J : = 1,2, ... ,n} directly. The
calculations were run on a Sparc 2 workstation in double precision arithmetic. We
see that the Laurent expansion technique can reduce the work of each iteration
Any savings
of the conjugate gradient algorithm from O(n2) to about O(nlogn).
are rather slight, however, until n exceeds 1000.
~.l
i
i
9
Thin plate spline interpolation
3. Numerical experiments with pre-conditioners
The conjugate gradient algorithm would solve the system (2.2) in one iteration if
Q were a multiple of the unit matrix. Thus
the matrix Q were such that Q T <I>
the actual choice of Q is very important to the efficiency of the solution of the
interpolation equations. Therefore the purpose of this section is to investigate
some choices numerically. vVe employ two distributions of interpolation points
and three different right hand side vectors L these quantities being defined in a
way that allows an infinite sequence of values of n.
Specifically, n is always the square of an integer, in order that the first distribution is a regular square grid that covers the unit square [0,1] x [0,1]. In the
other distribution, the points {;fi : = 1,2, ... , n} are given the coordinates
i
Xi
=
sin(i1l"jn)
and
Yi=1[1+sin(2i1l"jn)],
i=1,2,
... ,n.
(3.1)
Vve call these distributions R (regular) and C (curve), respectively. We see that
the points (3.1) lie on a closed curve that is enclosed by and touches the four sides
of the unit square. Both of these kinds of distribution are of interest in practice.
We let the right hand sides have the components
Ii
(xy):
=
XiYi
X·<l
1_ 2
(01)
Ii
={
1, Xi>l
0,
i= 1, 2, ...
2
, n,
(3.2)
(sine)
the entries on the left hand side being the names that we use to distinguish these
choices. Thus the cases (xy), (01) and (sine) provide interpolation to an easy
smooth function, a discontinuous function, and an oscillatory smooth function
respecti vely.
We apply a straightforward implementation of the conjugate gradient algorithm to the system (2.2). In order to describe it briefly, we let !:!:..k be the value
of !!:.. at the beginning of the k-th iteration, while '£..1. denotes the corresponding
residual vector
'£..1.
= QT
t -Q
T
<PQ !:!:..k'
k
= 1, 2, 3,
....
(3.3)
The calculation ends if the condition
II s. ACC is satisfied, where ACC is a given
positive number. Otherwise, the k-th iteration chooses a search direction 511. E
11'£..1.
and it sets !:!:..k+1 to the vector !:!:..k+CikSlk, where the value of CYk is calculated
so that the new residual vector '£..k+1 is orthogonal to 11.' Specifically, the initial
search direction is 11 = '£..1' and, on all other iterations, !1k is a linear combination
of '£..k and 11.-1 that satisfies the conjugacy condition
nn-3,
(3.4)
M.J.D. Powell
10
42
137
401
161
153
132
260
51
75
86
44
72 37
190
36
181
116
1209
247
400
445
261
196
55
79
83
400
45
49
21
92
206
3
7100
n=49
270
72
175
108
71
33
519
715
150
8319
38
709
25
9
39
30
6
Problem
ACC=
10-6
ACC= 10-10
Table 2:
Iteration counts for Householder Q (no diagonal scaling)
It is well known that, in exact arithmetic, the residual vectors {1:k : k = 1,2,3, ...}
are mutually orthogonal, so termination occurs after at most n - 3 iterations.
Further, for k ~ 2, each !!:.k is the value of l!:. that minimizes the strictly convex
quadratic function
subject to the restriction that
is in the linear space that is spanned by the
previous residuals {1:1' 1:2' ... ,1:k:=-l}' Thus one can deduce the property
f.L
(3.6)
In practice, however, it is not unusual for the algorithm to run for more than
n-3 iterations, due to the effects of computer rounding errors. Nevertheless, the
inequality (3.6) is achieved on every iteration unless the matrix Q T ~ Q is very
ill-conditioned. Therefore we expect termination to occur eventually for any value
of ACC.
Table 2 presents the numbers of iterations that are required by our implementation when Q is defined implicitly by the Householder transformations of the
third paragraph of Section 2. We see that four values of n and two values of ACC
were tried for each of our six test problems. We see also that rounding errors are
causing severe inefficiencies in many of the ACC= 10-10 experiments.
Therefore we wish to promote some other ways of choosing the matrix Q T <I>
Q.
Ian Barrodale (private communication) recommends a simple technique that is
high~y useful for the system (1.5). It is to apply a constant scaling factor throughout n2 so that the greatest of the distances {11~i-~j
II: i, j = 1, 2, ... , n} is one, in
order that the corresponding element of is zero. We see that this scaling makes
<l>
11
Thin plate spline interpolation
152
170
30849
77
78
142
224
149
1441
28961
46089
221
346
74
437
187
60
95
466
n=49
196
100 54
109
115
132
46674
223
284
554
898
117
23039
38457
308
400
739
39
38
57
50
33
ACC=
10-6
ACC = 10-10
Problem
Table 3:
Iteration counts for Householder Q (with diagonal scaling)
the change
-'-J
(p..
Ilx,-x'll
~J= Ilx,-x'11210g
-~ -J
a2(P ~J
..
+ Ilx,-x'11210ga
-~-J
(3.7)
to each element of <1.>,
where a is the scaling factor. Now the three n x n matrices
whose (i,j)-th elements are IliJ2, -2;rT;rj and II;rjIl2 are annihilated by postmultiplication by Q, post-multiplication by Q (or pre-multiplication by QT) and
pre-multiplication by Q T, respectively. Therefore the change (3.7) only causes
Q T (p Q to be replaced by a2 Q T (p Q, so it does not provide a suitable solution to
our task.
Another technique that is sometimes helpful to the conjugate gradient method
is scaling the rows and columns of the matrix of the system, so that symmetry
is preserved and so that the matrix of the new system has ones on its diagonal.
Therefore we applied this scaling to the Q T <l? Q matrices that were generated in
the calculations of Table 2. The new numbers of iterations are reported in Table
3. It seems, unfortunately, that nearly all of the inefficiencies have become worse,
and not all of the loss is due to the fact that the termination condition becomes
harder to satisfy when the scaling factors exceed one.
Therefore we give attention to the suggestion at the beginning of this section,
that we should try to choose Q so that Q Tip Q is close to a multiple of the unit
matrix. Some guidance towards this objective is given by the identity (1.13).
Indeed, following Dyn and Levin (1983) and Dyn, Levin and Rippa (1986), we
can pick the elements in the j-th column of Q to provide the approximation
n
~ Qtj g(bJ ~ 'J2g(;rj)
(=1
(3.8)
M.J.D. Powell
12
when {g(~) : ~ E R2} is any smooth function. If both the i-th and j-th columns
of Q are chosen in this way, then we have the estimate
n
=
(QT<pQ)ij
n
I:I:QkiQlj<P(~k-b)
k=ll=l
n
~I:
k=l Qki ['V!2<p(~k-lL) ]y=x
- -) ~
['V~2'V!2<p(~-lL)
L=x,
- -"- y=x'
-)
(3.9)
where
is still the function (1.6). It follows from equation (1.13) that such
columns of Q may cause the resulting part of Q T <pQ to resemble a multiple of
the corresponding part of the (n-3) x (n-3) identity matrix. In particular, we
expect I( Q T <pQ )ij I to be very small when ~j is far from ~i'
We applied this idea to our regular grid test problems. The well-known approximation
<P
gives a column of Q for every interpolation point that is i,nterior to the grid, where
and f:.y are the coordinate vectors in R2 and h is the grid size, but we scaled
these columns by h2 in order that the nonzero elements of Q are of magnitude
one. Further, if ~j is on the edge of the grid without being a corner point, then
we let the j-th column of Q have the property
f:.x
n
L Qij 9(b)
i=1
= g(~j-h§.t)
- 2g(;£j)
+ g(:fj+hf:.t),
(3.11)
where §.t is the coordinate direction that is parallel to the edge. Assuming that
the interpolation points are ordered so that the corners of the grid are last, we
define the final column of Q by the equation
n
I: Qln-3
i=l
g(~)
= g(O) + g(.f.x+§.y)
- g(§.:c) - g(§.y).
(3.12)
Thus each of the columns of Q has only three, four or five nonzero elements, and
it can be proved that they span the null space of P T in nn.
The number of iterations of the conjugate gradient method for this choice of
Q is given in Table 4. Further, even better results are obtained if we scale the
columns of Q so that the diagonal elements of Q T <pQ are equal to one, as shown
in Table 5. It is particularly promising that, in every case, the growth in the
number of iterations as n increases is much slower than a multiple of n.
For the curve data, we try letting each column of Q have just four nonzero
elements, which are consecutive. Therefore we require the functionals
i+3
2:: Qij 9(b),
i=j
j = 1, 2, ...
, n-3,
(3.13)
Thin plate spline interpolation
13
ACC=10-6
ACC= 10-10
12
196
2
228
44
37
33
25
43
33
27
15
20
1
400
28
5
100
36
29
60
48
19
6 20
n=49
22
6
Problem
Table 4:
Iteration
counts when Q is derived from \7 2 (no diagonal scaling)
n=49
400
29
196
21
12
13
13
9
1720
20
23
26
37
31
100
6 14
21
28
18
15
19
16
6
ACC=1O-6 ACC=10-1O
n=49
Table 5:
Iteration
counts when Q is derived from V2 (with diagonal scaling)
to vanish when 9 is any linear polynomial, because this property is equivalent to
the requirement pTQ = O. Further, we scale the columns of Q so that again the
diagonal elements of Q Tit> Q are all one. Since the points (3.1) lie on a strictly
convex curve, it follows that, apart from the signs of the columns which are
unimportant,
these conditions define Q uniquely. Indeed, it is straightforward to
show that the nonzero part of the j-th column of Q is a multiple of the vector
sin ..J!..
2n [sin
U+l)1I"
n
_ sin 311"
2n [sin
sin 311"
2n [sin
+ sin
U+2)11"
n
(j+3)1I"
n
_ sin ..J!..
2n [sin j1r
n
U+2)1I"
n
+ sin
+ sin
+ sin
+ sin
U+3)11"
n
n
j1l"
U+l)1I"
n
U+3)1I"
n
+ sin
+ sin
n
j1l"
U+1)1I"
n
+ sin
U+2)1I"
n
+ sin
+ sin
+ sin
(3j+6)1I"
n
]
(3j+5)11"
]
n
(3.14)
(3j+4)1I"
n
+ sin
(3j+3)1I"
n
]
]
We see that, when n is large, the ratios of these components are approximately
1 : -3 : 3 : -1 for most values of j. Therefore expression (3.13) can be regarded
as a multiple of a third derivative of 9 along the curve, which we write in the form
(3.15)
M.J.D. Powell
14
122
137
368
178
n=49
196
727
100
213
1146
1726
1792
604
784
6048
180
385
557
58
1437
484
337
400
51
47
46
Problem
ACC=
10-6
ACC=
10-10
Table 6:
Iteration counts when Q is derived from curve points (no tri-diagonal scaling)
where c denotes the direction of the curve. Thus the "curve" analogue of the
"regular grid" expression (3.9) is the approximation
(3.16)
Hence, as before, the smoothness of away from its singularity gives the property
that I(QT<J> Q)iil is tiny when n is large and {fi is not close to ;!;..j'
The results in the "curve" case are given in Table 6, and we find that they
are bad. Indeed, the third derivatives of expression (3.16) seem to be far less
efficacious than the second derivative approximations that were employed earlier.
Therefore we are going to develop a successful pre-conditioner from the view that
expression (3.16) is analogous to the approximation
</>
i
where 6i and 6j are forward difference operators that act on the indices and j
respectively. Now, if the second derivative contributions to this approximation are
suitably large when = j, then the difference operators cause the large elements
of Q T <P Q to occur not only on the diagonal but also adjacent to the diagonal.
Therefore we apply the following technique, because it reverses the action of the
difference operators.
We let Q be the nx(n-3) matrix such that, for = 1, 2, ... , n-3, the elements
{Qej : .e=j,j+1,j+2,j+3}
are the components of the vector (3.14), while all
other elements of Cd are set to zero. vVedefine 9.1 to be the multiple of ~1 that
satisfies the condition 9.f<P!il = 1, where !Ij and ~j denote the j-th columns of Q
and Q respectively. Further, for j = 2,3, ... , n-3, we define -)
q. to be the multiple
i
j
15
Thin plate spline interpolation
76
51
24
22
102
101
77
40
33
50
32
65
62
68
38
45
29
58
n=49
400
100 19
88
196
17
20
ACC= 10-6ACC= 10-10
n=49
Table 7:
Iteration counts when Q is derived from curve points (with tri-diagonal scaling)
of the vector 1j-(!1f-l <l>
1)!1j-l that has the normalization!1f <l>!1j
= 1. The reason
Q has ones on its diagonal and
for these definitions is that the new matrix Q T <l>
zeros adjacent to its diagonal, but Q is not formed explicitly because most of its
elements are nonzero.
Cd is calculated, taking advantage of the structure of
Instead the matrix Cd T <l>
Q. Then, for = 1, 2, ... , n-3, we let Uj be a very sparse (n-3)x(n-3) matrix such
that the first columns of the product Cd U1 U2 ••• Uj are the same as the first
columns of Q, and, for < n-3, the (j+ 1)-th column of the product is a multiple of
~i+l' Fortunately, it follows from the previous paragraph that (Uj )jj and (Uj)j HI
are the only elements of Uj that may have to be different from elements of the
(n-3) x (n-3) identity matrix. Further, it is easy to obtain the values of these
elements. Thus, by applying the formula
j
j
j
j
(3.18)
we generate Q T Q in only O( n2) computer operations. Similarly, the right hand
side of the system (2.2) is constructed by pre-multiplying Cd Tf by the product
U!-3U!-2'" U{.
Table 7 shows the numbers of iterations of the conjugate gradient algorithm
when it is applied to this new system of equations. We see that the tri-diagonal
pre-conditioning provides an excellent improvement over the results of Table 6.
Indeed, in every test problem of Table 7 the number of iterations is substantially
less than n.
<I>
4. Conclusions
The numerical results of Tables 1, 5 and 7 suggest that the thin plate spline
interpolation equations can be solved in much less than O( n3) operations for
M.J.D. Powell
16
large n. There is a fundamental reason for this conclusion, which is eXplained
below because of its relevance to future research.
vVewrite the interpolant in the cardinal form
n
s(~)
= Lfjfj(~),
(4.1)
j=l
i
where each £j is independent of the function values {fi : = 1,2, ... , n}. Therefore
£j satisfies the conditions
(4.2)
and all the remaining freedom in £j is taken up by minimizing the functional V(£j),
where V is defined by equation (1.2). Thus the second derivatives of £j are no
larger than necessary to achieve the cqnstraints (4.2). Further, these constraints
help l£j(;f) I to be small when ~ is far from !fj' Thus the contribution from 1;
to s(~) in expression (4.1) is relatively small when ~ is in the convex hull of
all the interpolation points but is outside some neighbQurhood of ~j' A typical
neighbourhood of ~j might contain about twenty of the points {~i : = 1,2, ... , n}.
Therefore, if the value of s(~) is required to moderate accuracy, then often, at least
in theory, one can ignore data at points that are remote from~. Further, we expect
the number of data that are important to s(~) to be practically independent of
n when n is very large. In other words, there is much implicit sparsity in the
estimation of s(~) that may be accessible through the cardinal functions {fj :
= 1,2, ... , n}. Thus the variational problem has some structure, which suggests
that the solution of the interpolation equations need not be very expensive.
We ask whether the findings of Sections 2 and 3 are of much value in practice.
Tables 5 and 7 suggest that a careful choice of Q can provide substantial gains
in efficiency when n ~ 100, but Table 1 indicates that the complications of the
Laurent expansions are worthwhile only for n 2: 1000, approximately. Therefore
automatic methods for pre-conditioning would probably be much more valuable
than the development of computer software that applies the Laurent technique.
On the ot,her hand, many values of s(~) may be required after the coefficients
{Aj : = 1, 2, ... , n} and {Co, Cl, C2} of the thin plate spline function (1.3) have been
determined, and it is shown by Beatson and Newsam (1992) that this need can be
answered by extending the Laurent approximations to Taylor series expansions.
Alternatively, a single bicubic spline approximation to s may be more convenient
than several Taylor series expansions. In this case it would probably be important
to choose enough knots to preserve the shape of s between interpolation points,
because the shape seems to be of particular merit in many practical applications.
Two techniques are used to construct pre-conditioners in Section 3. Firstly,
some columns of Q are derived from the approximation (3.8). Dyn et al (1986)
show that this technique is also useful when the interpolation points are in general
i
j
j
17
Thin plate spline interpolation
position. Secondly, expressions (3.11)-(3.14) are less ambitious, because their
purpose is merely to generate columns of Q with few nonzero elements that are
in the null space of P T, this condition being the equation
n
L. Qlj
l=1
g(~)
=
0,
(4.3)
gE III,
where III is the 3-dimensional space of constant and linear polynomials from n2
to n. Sibson and Stone (1991) recommend a convenient way of satisfying equation
(4.3) for general positions of the interpolation points. Specifically, they form the
Dirichlet tessellation of {;£i : = 1,2, ... , n}, in order to express each interior
interpolation point as a convex combination of its neighbours. Thus they obtain
relations of the form
n
i
:£.j
=
L. (}
lj ~
(4.4 )
i=l
li-j
for several values of j, where the coefficients {(}lj : £ E {1,2"H,n}
\ {j}} are
nonnegative and sum to one, and where (}lj is zero if ~ is not a neighbour of
:£.j in the tessellation. Then they achieve the condition (4.3) by setting Qjj = 1
and Qlj = -(}lj, £ f- j. Further, they present a special procedure for the case
when :£.j is on the boundary of the convex hull of the interpolation points. Thus
they construct suitable matrices Q whose columns each have about seven nonzero
elements. Methods that require triangulations or tessellations are unattractive,
however, when the interpolation points lie on a curve.
We recall from Table 6 that our first shot at choosing Q for the curve distribution (3.1) was unsuccessful. Further, if the curve had not been strictly convex,
then the columns of Q might not have been linearly independent. For example,
suppose that :£'2, !f.3 and ;£4 lie on a straight line and that :£.1 and :£.5 are not on
this line. Then, if only {Qel : R.= 1,2,3,4} and {Qe2 : R.= 2,3,4, 5} are allowed
to be nonzero in the first and second columns of Q, respectively, it follows that
both Qn and Q52 are zero and that 9.2 is a multiple of 9.1' where 9.j is still the j-th
column of Q. It might be very worthwhile, therefore, to give more attention to
the problem of choosing Q automatically.
The second paragraph of this section suggests another way of solving the interpolation equations efficiently when n is large. It is to form approximations,
{Ij : j = 1,2, ... ,n} say, to the cardinal functions {£j : j = 1,2, ... , n} of expression (4.1), in order to calculate s by an iterative refinement procedure that
corrects the errors of the approximations. It should be adequate to define each Ij
by minimizing V(lj) subject to the equations {Ij(:£.i) = Oij : E Jj}, where Jj is
a carefully chosen subset of {I, 2, ... , n}. Typically, Jj might include the indices
of about ten interpolation points from a neighbourhood of :£.j and about four distant interpolation points. Thus the amount of work to generate the coefficients
of each Ij would be independent of n, and perhaps the number of iterations of
i
lVI.J.D. Powell
18
the refinement procedure would also be independent of n for most distributions of
interpolation points. The author is investigating these ideas with Rick Beatson.
Our numerical results and remarks indicate that it may be possible to enjoy
the advantages of thin plate splines in applications that require interpolation to
thousands of values of a function of two variables. The author intends to provide
some Fortran software for this task.
References
1. Barrodale, R. Kuwahara, R. Poeckert and D. Skea (1992), "Processing sidescan sonar images using thin plate splines", presented at the NATO
Advanced Research Workshop on Algorithms for Approximation (Oxford,
July, 1992).
R.K. Beatson and G.N. Newsam (1992), "Fast evaluation
tions: I", Comput. Math. Applic., Vol. 24, pp. 7-19.
of radial basis func-
J. Duchon (1977), "Splines minimizing rotation-invariant
seminorms in Sobolev
spaces", in Constructive Theory of Functions of Several Variables) Lecture
Notes in Mathematics 571, eds. W. Schempp and K. Zeller, Springer-Verlag
(Berlin), pp. 85-100.
N. Dyn and D. Levin (1983), "Iterative solution of systems originating from integral equations and surface interpolation", SIAM J. Numer. Anal., Vol. 20,
pp. 377-390.
N. Dyn, D. Levin and S. Rippa (1986), "Numerical procedures for surface fitting
of scattered data by radial functions", SIAM J. Sci. Statist. Comput., Vol. 7,
pp. 639-659.
V
R. Franke (1982), "Scattered data interpolation:
Comp., Vol. 38, pp. 181-200.
tests of some methods",
G.H. Golub and C.F. Van Loan (1983), Matrix Computations,
University Press (Baltimore).
Math.
The Johns Hopkins
M.J.D. Powell (1992), "Truncated Laurent expansions for the fast evaluation of
University of Cambridge.
thin plate splines", Report DAMTP 1992jNAI0,
M.A. Sabin (1980) "Contouring - a review of methods for scattered data", in
~Mathematical Methods in Computer Graphics and Design, ed. K.vV. Brodlie,
Academic Press (London), pp. 63-86.
R. Sibson and G. Stone (1991) "Computation of thin plates splines", SIAM J.
Sci. Statist. Comput., Vol. 12, pp. 1304-1313.