NUMERICAL LINEAR ALGEBRA WITH APPLICATIONS
Numer. Linear Algebra Appl. 2003; 10:335–355 (DOI: 10.1002/nla.306)
Application of the Lanczos algorithm for solving the linear
systems that occur in continuation problems
C.-S. Chien1; ∗; † and S.-L. Chang2
1 Department
2 Center
of Applied Mathematics; National Chung-Hsing University; Taichung 42; Taiwan
for General Education; Southern Taiwan University of Technology; Tainan 7; Taiwan
SUMMARY
We study the Lanczos method for solving symmetric linear systems with multiple right-hand sides.
First, we propose a numerical method of implementing the Lanczos method, which can provide all
approximations to the solution vectors of the remaining linear systems. We also seek possible application
of this algorithm for solving the linear systems that occur in continuation problems. Sample numerical
results are reported. Copyright ? 2002 John Wiley & Sons, Ltd.
KEY WORDS:
Lanczos method; symmetric linear systems; multiple right-hand sides;
parameter-dependent problems
1. INTRODUCTION
In this paper, we are concerned with solving linear systems of the form
AX = B
(1)
where A ∈ RN ×N is symmetric positive denite, X = [x(1) ; : : : ; x(k) ] ∈ RN ×k is the matrix of unknowns, and B = [b(1) ; : : : ; b(k) ] ∈ RN ×k is a number of right-hand sides to be solved. Equation
(1) arises, for instance, when the domain decomposition method [1, Chapter 13] combined
with nite dierences or nite elements is used to solve elliptic partial dierential equations.
If all the right-hand sides are available simultaneously, then the block-Lanczos or blockconjugate gradient algorithms can be successfully applied to solve (1), see e.g. Reference [2,
Chapters 9, 17, 19]. Alternatively, one may also solve (1) on a parallel computer.
In Reference [3] Parlett proposed the Lanczos–Galerkin projection method for solving (1).
In this method one begins by assuming that the Lanczos method has been used to solve the rst
linear system. Then one tries to use the information obtained by solving the rst linear system
to provide a good approximate solution for use in the second linear system. An analysis of the
Lanczos–Galerkin process can be found in Reference [4]. However, our numerical experiments
showed that the approximate solution provided by the Lanczos–Galerkin projection method
∗
†
Correspondence to: C.-S. Chien, Department of Applied Mathematics, National Chung-Hsing University, Taichung
42, Taiwan.
E-mail: [email protected]
Published online 30 October 2002
Copyright ? 2002 John Wiley & Sons, Ltd.
Received 11 April 2000
Revised 16 November 2000
336
C.-S. CHIEN AND S.-L. CHANG
for the second linear system in general is not accurate enough. Therefore, a restarted Lanczos–
Galerkin projection method is necessary for solving the second linear system.
Later, Smith et al. [5] developed a specic conjugate gradient (CG) algorithm for solving
(1). This algorithm generates a Krylov subspace from a set of direction vectors obtained by
solving the seed system, say Ax(1) = b(1) , by the CG method. Then the residuals of the remaining systems are projected orthogonally onto the generated Krylov subspace to get the approximate solutions. The whole process is repeated with, say, Ax(2) = b(2) as the seed system until
all the systems are solved. The method of Smith et al. was analysed by Chan and Wan [6].
Papadrakakis and Smerou [7] proposed numerical techniques of implementing the Lanczos
method, which can provide all approximations to the solution vectors of (1) simultaneously
without the necessity of keeping the tridiagonal matrix and the orthonormal basis in fast or
secondary storage. However, the accuracy of the approximate solutions thus obtained for the
remaining linear systems, say the jth linear system, depends on the distance of the right-hand
sides b(1) − b(j) . One has to implement the preconditioned Lanczos or CG algorithm on each
of the remaining linear systems to improve the accuracy of the approximate solutions.
Recently, Erhel and Guyomarc’h [8] presented two CG-type algorithms derived from the
Lanczos algorithm described in Reference [4] to solve consecutive symmetric positive denite
linear systems, while Simoncini and Gallopoulos [9, 10] proposed an iterative method using
the idea of the single seed algorithm together with a hybrid method to solve non-symmetric
linear systems.
In this paper, we use the preconditioned Lanczos method to solve the rst linear system
of (1), where the incomplete LLT factorization of the coecient matrix A is used as the
preconditioner. In particular, we also allow a limited number of ll-in elements to take place
so that the coecient matrix A is more closely approximated by the incomplete factorization,
say, the ILLT (k), where k denotes the number of subdiagonals with ll-in elements. We refer to
[1, Chapter 10] for details. Next, we use the ILLT (k) to solve the remaining linear systems and
perform the iterative improvement [2, pp. 126–128], until the approximations to the solution
vectors are accurate enough. Finally, all we need to do is to implement the preconditioned
Lanczos method again to improve the accuracy of the approximate solution vectors, where
only a few or a small number of the Lanczos iterations are required. The numerical examples
in Section 5 show that our numerical techniques are superior to the well-known Lanczos-type
methods described above.
This paper is organized as follows. In Section 2 we briey review some well-known algorithms for solving (1). We also give an error bound for the approximate solution of the
second linear system, where the Lanczos–Galerkin process is used to solve the rst two linear systems. In Section 3, we propose to use the preconditioned Lanczos algorithm together
with the iterative renement technique for solving symmetric linear systems with multiple
right-hand sides. We also seek the possible application of the proposed numerical method to
continuation problems. This is described in Section 4. Our numerical results are reported in
Section 5.
2. A BRIEF REVIEW OF WELL-KNOWN ALGORITHMS
In this section, we briey review some well-known CG-type algorithms and the Lanczos
algorithms for solving (1).
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
APPLICATION OF THE LANCZOS ALGORITHM
337
2.1. The Lanczos algorithms
We consider a linear system of the form
Ax = b
(2)
where A ∈ RN ×N is symmetric positive denite and b ∈ RN . Let x0 ∈ RN be the initial guess
to the solution of (2), and r0 = b − Ax0 be the corresponding residual. Let v1 = r0 =1 with
1 = r0 . Here and in what follows the norm · denotes the Euclidean norm. The Lanczos
algorithm generates a sequence of orthonormal vectors v1 ; : : : ; vj , called Lanczos vectors, for
the Krylov subspace K(A; v1 ; j) such that
span{v1 ; : : : ; vj } = span{v1 ; Av1 ; : : : ; Aj−1 v1 } ≡ K(A; v1 ; j) ≡ Kj
The Lanczos algorithm for solving (2) can be described as follows.
Algorithm 2.1. The Lanczos algorithm for solving linear systems
1. Start: Set r0 := b − Ax0 and v1 := r0 =1 with 1 := r0 .
2. Generate the Lanczos vectors:
For j = 1; 2; : : : ; do
(v0 ≡ 0)
rj := Avj − j vj−1 ;
j := (rj ; vj )
rj := rj − j vj
j+1 := rj If j+1 ¡ then set m := j and go to 3;
else, compute
vj+1 := rj =j+1
3. Form the approximate solution:
xm := Vm Tm−1 (1 e1 )
where Vm is the N × m matrix, Vm = [v1 ; v2 ; : : : ; vm ] and Tm is the tridiagonal
matrix


1 2


 2 2 3





.
.
.
.
.
.


.
.
.
Tm =



.. ..


.
.

m
m m
(3)
Under the assumptions given above, we have
AVj = Vj Tj + rj ejT
(4)
where Tj is dened as in (3), see, e.g. Reference [2, p. 474]. In step 3 of Algorithm 2.1, we let
Tm−1 (1 e1 ) = ym = [ym(1) ; : : : ; ym(m) ]T . Parlett [11] showed that the residual vector rˆm corresponding
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
338
C.-S. CHIEN AND S.-L. CHANG
to the approximate solution xm can be expressed as
rˆm = −rm ym(m)
(5)
That is, rˆm is a multiple of the vector rm in step 2, and by (4), is orthogonal to K(A; v1 ; m).
Moreover,
rˆm = m+1 |ym(m) |
therefore, for each j = 1 : m, the norm of the jth residual may be computed without forming
either xj or rˆj .
We recall the following results in [2, Chapter 9].
Theorem 2.1
Let A be an N × N symmetric matrix with eigenvalues 1 ¿ · · · ¿ N and corresponding
orthonormal eigenvectors u1 ; : : : ; uN . If 1 ¿ · · · ¿m are the eigenvalues of the matrix Tm
obtained after m steps of the Lanczos iteration, then
1 ¿1 ¿1 −
(1 − N )(tan 1 )2
[cm−1 (1 + 21 )]2
where cos 1 = |v1T · u1 |, 1 = (1 − 2 )=(2 − N ) and cm−1 (x) is Chebychev polynomial of
degree m − 1.
Corollary 2.2
Using the same notation as Theorem 2.1, we have
N 6m 6 N +
(1 − N ) tan2 N
2
cm−1
(1 + 2N )
where N = ( N −1 − N )=(1 − N −1 ) and cos N = |vNT · uN |.
In order to accelerate the rate of convergence of the Lanczos method, we need to impose
preconditioning techniques on the linear system (2). More precisely, we transform (2) into
Ax = b
(6)
where A = M −1 AM −T , x = M T x, b = M −1 b. Note that (6) is equivalent to
M −1 Ax = M −1 b
(7)
Let R = MM T , then A and R−1 A have the same eigenvalues. It is clear that the more the transformation matrix R resembles A, the more the rate of convergence will increase. In particular,
if A is symmetric positive denite, then we can choose R as the incomplete Choleski factorization of A. That is, we drop all ll-in elements that are generated during the standard Choleski
factorization process. This is referred to as ILLT (0). In order to obtain a better approximation
to the coecient matrix A, we also allow a limited amount of ll-in elements to appear along
certain subdiagonals to get incomplete factorization. If the incomplete factorization contains
k subdiagonals with ll-in elements, we refer to this as ILLT (k). See Reference [1, Chapter
10] for a similar discussion. A preconditioned Lanczos algorithm is given and described in
Reference [7] as follows:
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
APPLICATION OF THE LANCZOS ALGORITHM
339
Algorithm 2.2. The preconditioned Lanczos algorithm
1. Start: Choose x0 and let ”¿0 be the given tolerance.
q0 = c0 = d0 := 0, 1 := 0, 0 = 1
compute r0 := M −1 r0 , set 1 := r0 = (r0T R−1 r0 )1=2 , q1 := r0 = 1 .
2. Generate the Lanczos vectors:
For j = 1; 2; : : : ; do
uj := R−1 qj
j := ujT Auj
rj := Auj − j qj − j qj−1
j+1 := (rjT R−1 rj )1=2
qj+1 := rj = j+1
3. Form the approximate solution:
2
dj := j − j dj−1
j+1 := j+1 = dj
j := −j dj−1 j−1 = dj (with 1 = −1 = d1 )
cTj := qjT R−1 − j cTj−1
xj := xj−1 + j cj
4. If |j |·rj = r0 ¡”, then stop;
else, go to step 2.
For completeness the preconditioned Lanczos method proposed by Papadrakakis and Smerou
[7] for solving (1) is described as follows:
Algorithm 2.3. The preconditioned Lanczos method for solving (1)
1. Choose an initial guess x0(1) for the rst equation, set
r1(1) = b(1) − Ax0(1)
x0(j) := R−1 (b(j) − r0(1) ); j = 2 : k
1 := (r1(1) R−1 r1(1) )1=2
q1 = r1(1) = 1
q0 = c0 = d0 = 0; 1 = 0
For j = 1; 2; : : : ; Do
2. Perform Step 2 of Algorithm 2.2.
3. Perform Step 3 of Algorithm 2.2.
4. For i = 1 : k, Do
ji = (qjT R−1 r i − j dj−1 j−1;i )= dj
1
cTj is obtained in Step 3,
i
xji := xj−1
+ ji cj
(1)
(1)
5. |ji |·rj+1 = r1 ¡”
6. Solve the remaining systems by the preconditioned CG/Lanczos algorithm.
The initial vectors are those obtained from Step 4.
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
340
C.-S. CHIEN AND S.-L. CHANG
However, there is an unavoidable possibility that the Lanczos vectors could lose orthogonality among themselves because of rounding errors. To overcome this diculty, one can
incorporate the Householder transformations into the Lanczos process. See e.g. Reference
[2, pp. 482–483] and the references cited therein for details. By doing this, we can obtain Lanczos vectors that are orthogonal to machine precision. This is an example of a
complete reorthogonalization Lanczos procedure [12]. In practice, one can implement the
so-called selective reorthogonalization process, see e.g. Reference [11, Chapters 13, 24] for
details.
Finally, we recall the single seed algorithm described in Reference [6], where J denotes
the number of right-hand sides.
Algorithm 2.4. Single seed method for solving (1)
for l = 1; 2; : : : ; J
% we choose x01; l = 0
r01; l = bl − Ax01; l
end for
for k = 1; 2; 3; : : : until the systems are solved
Select the kth system as seed
% CG iteration
for i = 0; 1; 2; : : : ; mk
for j = k; k + 1; k + 2; : : : ; J
% each remaining unsolved RHS
if j = k then perform usual CG steps
if i¿0
k; k T k; k
) ri−1
k;i k = (rik; k )T rik; k =(ri−1
k; k
k; k
k; k k; k
pi = ri + i pi−1
else
pik; k = rik; k
end if
ik; k = (rik; k )T rik; k =(pik; k )T Apik; k
k; k
xi+1
= xik; k + ik; k pik; k
k; k
ri+1
= rik; k − ik; k Apik; k
else perform Galerkin projection
k;i j = (pik; k )T rik; j =(pik; k )T Apik; k
k; j
xi+1
= xik; j + k;i j pik; k
k; j
ri+1
= rik; j − k;i j Apik; k
end if
end for
end for
for s = k + 1; k + 2; : : : ; J
k; s
x0k+1; s = xi+1
k; s
r0k+1; s = ri+1
end for
end for
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
APPLICATION OF THE LANCZOS ALGORITHM
341
2.2. The restarted Lanczos–Galerkin method
We consider the following linear systems
Ax(i) = b(i) ;
i = 1; 2; : : : ; J
(8)
where A ∈ RN ×N is symmetric positive denite, and b(i) ∈ RN . Assume that m steps of the
Lanczos algorithm have been performed to solve the rst system in a rst pass, and xL(1)
is the accepted approximate solution. Let Pm be the orthogonal projection onto the Krylov
subspace Km . Assume that the Lanczos vectors {v1 ; : : : ; vm } as well as the tridiagonalization
Tm have been saved, possibly in some secondary storage. We wish to use the information
obtained during the solution of the rst linear system to provide a good approximation to the
second linear system
Ax(2) = b(2)
(9)
x0(2)
r0(2)
be the initial guess to the solution of (9), and let
be the corresponding residual
Let
(2)
(2)
(2)
vector, i.e., r0 = b − Ax0 . One may improve the approximation x0(2) by means of a Galerkin
projection onto the Krylov subspace Km generated for the solution of the rst linear system.
Such an approximation is obtained by solving the m-dimensional problem
Pm (b(2) − Az) = 0
(10)
for some z in the ane subspace x0(2) + Km , i.e., z = x0(2) + y with y ∈Km . It is obvious that
condition (10) can be translated into the Galerkin problem
Pm (r0(2) − Ay) = 0
(11)
or equivalently,
VmT (r0(2) − Ay) = 0
whose solution is y = Vm Tm−1 VmT r0(2) . Then the desired approximation to (9) can be expressed
as
z = x0(2) + y = x0(2) + Vm Tm−1 VmT r0(2)
(12)
The procedure described above is called Lanczos–Galerkin process, which requires solving
a tridiagonal linear system of size m and forming a linear combination of m vectors of
length N .
However, the accuracy of the approximation y obtained above may simply not be accurate
enough. Thus, we need to start a fresh Lanczos process from the current approximation y.
Let
and {v1 ; : : : ; vi } be the new Lanczos
r be the corresponding residual vector, i.e., r = b − Ay,
r.
vectors which are performed to solve the second linear system with v1 = r=
Suppose that the right-hand sides of the rst two linear systems are close enough, say,
b(1) − b(2) = d with d¡”∗ for some ”∗ ¿0. It is evident that we may choose the approximate
solution xL(1) to the rst linear system as an initial guess to the solution of (9). We are ready
to state the main result of this section.
Theorem 2.3
Assume that A is symmetric positive denite with eigenvalues 1 ¿ · · · ¿ N ¿0 and corresponding orthonormal eigenvectors u1 ; : : : ; uN . Let 1 ¿ · · · ¿m be the eigenvalues of the
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
342
C.-S. CHIEN AND S.-L. CHANG
matrix Tm obtained after m steps of the Lanczos iteration. If we choose x0(2) = xL(1) , then
z − x(2) 6
(N + 1)”∗
+ I − Vm Tm−1 VmT A·xL(1) − x(1) N
Proof
Since A is symmetric positive denite, we have A = U U T with
U = [u1 ; : : : ; uN ]
and
= diag(1 ; : : : ; N )
Note that z − x(2) can be decomposed as
z − x(2) = (z − xL(1) + xL(1) − x(1) ) + (x(1) − x(2) )
which implies that
z − x(2) 6z − xL(1) + xL(1) − x(1) + x(1) − x(2) (13)
We also have
A(x(1) − x(2) ) = b(1) − b(2) = d
or equivalently,
x(1) − x(2) = A−1 d = (U U T )−1 d =
N uT d
i
i=1
i
ui
which implies that
x(1) − x(2) 6
N |uT d|
i
i=1
i
6N
”∗
N
(14)
On the other hand, we have
z − xL(1) + xL(1) − x(1) = Vm Tm−1 VmT r0(2) + xL(1) − x(1) = Vm Tm−1 VmT (b(2) − AxL(1) ) + xL(1) − x(1) = Vm Tm−1 VmT (b(2) − b(1) ) + (I − Vm Tm−1 VmT A)(xL(1) − x(1) )
6
”∗
+ I − Vm Tm−1 VmT A xL(1) − x(1) m
6
”∗
+ I − Vm Tm−1 VmT A xL(1) − x(1) N
(15)
where the last inequality follows from Corollary 2.2. The result follows by substituting (14)
and (15) into (13).
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
APPLICATION OF THE LANCZOS ALGORITHM
343
3. PRECONDITIONED LANCZOS ALGORITHM AND ITERATIVE REFINEMENT
In this section, we reconsider Equation (1), namely,
Ax(i) = b(i) ;
i = 1; : : : ; J
(16)
where A is a real symmetric positive denite matrix of order N . We assume that m steps of
the Lanczos method have been performed to solve the rst preconditioned linear system
M −1 Ax(1) = M −1 b(1)
(17)
where the preconditioner M is dened as in Section 2. We will use the incomplete factorization
of A together with the iterative renement technique [2, pp. 126–127] to obtain approximate
solution vectors for the remaining linear systems of (16).
T
To be precise, let A ≈ L̃L̃ , where L̃ is obtained from the incomplete factorization ILLT (k).
We use this factorization to solve the remaining linear systems, say, Ax(2) = b(2) , where the
following iterative improvement algorithm is executed. A similar idea has been discussed, e.g.
References [13, 14]. Our aim here is to obtain an approximate solution for the second linear
system which will be used as an initial guess for the preconditioned Lanczos algorithm.
Algorithm 3.1. Iterative improvement
1.
2.
3.
4.
5.
6.
Set k := 0, given ”¿0, k max and x̃(2) , r̃ (2) := b(2) − Ax̃(2) .
Solve L̃y = r̃ (2) .
T
Solve L̃ z = y.
(2)
+ z.
Set x̃(2)
new = x̃
(2)
(2)
Set r̃ = b − Ax̃(2)
new .
If r̃ (2) ¡”, then stop;
else if k¡k max , k := k + 1, go to step 2.
else, stop.
To analyse the linear system sensitivity of the above iterative renement technique, we
consider the following linear system
Ax = b
(18)
and its associated parametrized system (see e.g. Reference [2, Section 2.7])
(A + ”F)x(”) = b + ”f;
x(0) = x
(19)
where A; F ∈ RN ×N with A symmetric positive denite, and b; f ∈ RN , and ” ∈ R is the parameter. Let 1 ¿
2 ¿ · · · ¿
N ¿0 be the singular values of A. To ensure uniqueness of the
solution of (19), the matrix
A + ”F = A(I + ”A−1 F)
must be non-singular, which implies that
”A−1 F 2 6|”| A−1 2 F 2 =
Copyright ? 2002 John Wiley & Sons, Ltd.
|”|
N
F 2 ¡1
Numer. Linear Algebra Appl. 2003; 10:335–355
344
C.-S. CHIEN AND S.-L. CHANG
or F 2 ¡
N = |”|. In practice, we may choose ” = 1 so that F 2 ¡
N . Note that whether the
iterative renement will converge or not depends on the condition number of the coecient
matrix A. Actually, we have
x(”) − x2
6(A)(A + b ) + O(”2 )
x 2
where (A) = 1 =
N is the two-norm condition number of A; A = |”| F 2 = A2 , and b =
|”|f2 = b2 . We refer to [2, Section 2.7] for details. Thus, if A is too ill-conditioned, the
iterative renement technique may fail to converge.
If the approximate solution obtained from Algorithm 3.1 is acceptable, say, r̃ (2) ¡10−1 ,
we use it as an initial guess and perform the preconditioned Lanczos algorithm again to
solve the second linear system, where the incomplete factorization ILLT (k) is used as the
preconditioner. Our numerical results show that only a few Lanczos iterations are required
to obtain an accurate approximating solution for the second linear system. The other linear
systems can be solved in a similar way.
4. APPLICATION TO CONTINUATION PROBLEMS
We consider parameter-dependent problems of the form
H (x; ) = 0
(20)
where H : RN × R → RN is a smooth mapping with x ∈ RN , ∈ R. We denote the Jacobian of
H by DH = [Dx H; D H ] and the solution curve c of (20) by
c = {y(s) = (x(s); (s)) | H (y(s)) = 0; s ∈ I }
Here I is any interval in R. Assume that a parametrization via arc length is available on c.
A complete discussion concerning the singularity of Dx H (y∗ ) for some point y∗ ∈ c can be
found, e.g. Reference [15, Chapters 6 and 7].
During the past decade, several well-known curve-tracking algorithms have been developed
to solve (20). See e.g., the HOMPACK 90 of Watson et al. [16]. In particular, various conjugate gradient-type methods have been implemented to solve linear systems in this context.
See e.g. References [17–20]. Suppose that the predictor–corrector continuation algorithm described in References [21, 22, Chapter 2] is implemented to trace a solution curve of (20). In
this case, we have to solve linear systems of the following form:
B p x
f
=
(21)
T
g
q
where p; q; f ∈ RN and ; g ∈ R, or equivalently,
Ay = b
(22)
In continuation problems the matrix B has a special structure (banded, symmetric, sparse,
and so on). This is probably why the block elimination (BE) algorithm [22, pp. 77–78]
is one of the most popular algorithms for solving the bordered linear system (21) in case
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
345
APPLICATION OF THE LANCZOS ALGORITHM
where both the matrices B and A are well-conditioned. However, BE may break down if B
is nearly singular, for instance, if we are near a turning point. Various numerical methods
have been proposed to solve bordered singular systems. Specically, Govaerts [13] proposed
a stable solver for (21), namely, the mixed block elimination algorithm (BEM). BEM is a
combination of BE using Crout and Doolittle factorization, respectively, plus one iterative
renement, where a solver for B is given as a ‘black box’.
We wish to exploit the Lanczos algorithm described in the previous sections to solve (22).
For zero-nding problems and certain large systems of non-linear equations, the matrix B
is symmetric and sparse. In general, the matrix A is non-symmetric. Desa et al. [23] have
proposed to choose p = q and a suitable so that A is symmetric and non-singular. For certain
semilinear elliptic eigenvalue problems, if one exploits symmetries of the domains, then the
original problems can be solved on some reduced subdomains. By doing this, a large amount
of computational cost can be saved, see, e.g. References [24, 18]. Probably the only price
to pay is that the discretization matrix B corresponding to the partial dierential operator is
quasi-symmetric [25, p. 106]. That is, one can nd some similarity transformation U such
that UBU −1 is symmetric. In this case, we can exploit BEM to solve (21), where the Lanczos
algorithm is used as a linear solver for B. We have the following result.
Theorem 4.1
Let the coecient matrix A of the linear system (22) be dened as in (21). If there exists a
non-singular symmetric transformation U such that UBU −1 is symmetric and q = U 2 p, then
A is also quasi-symmetric. Moreover, if rank[B p] = N , then the scalar can be chosen so
that A is non-singular.
Proof
Choosing V =
0
, we can readily verify that
1
B
p U −1
U 0
−1
A := VAV = T
0 1 (U 2 p)T 0T
U
0T
0
1
=
UBU −1
Up
(Up)T
which is symmetric, since B := UBU −1 is symmetric. Thus A is quasi-symmetric.
To show that A is non-singular, we consider the following two cases with rank[B p] = N .
The proof is a slight modication of the one given in Reference [23, p. 34].
Case 1. Suppose rank B = N − 1. Then rank B = N − 1. Since [B Up] = U [B p]V −1 , we
have
rank[B
Up] = rank[B
p] = N
Thus,
row rank
B
(Up)t
=N
and [(Up)t ]t is not a linear combination of the rst N columns of A for any choice of .
Thus, the column rank of A is N + 1. i.e., A is non-singular. Hence, A is non-singular.
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
346
C.-S. CHIEN AND S.-L. CHANG
Case 2. Suppose rank B = N . Then
B
=N
rank
(Up)t
61, hence if
Since dim[Ker(A)]
and Up is a unique linear combination of the columns of B.
= 0, then y = y for some ∈ R, where y t = (ŷt ; yN ) and (Up)t ŷ + yN = 0. Choosing any
Ay
= 0 and solving (Up)t ŷ + yN = for , we have rank A = N + 1. Thus A is non-singular.
However, if Theorem 4.1 is implemented directly in the context of the predictor–corrector
continuation method, then one may get an opposite tangent direction near a turning point.
This diculty can be overcome by performing a similarity transformation on the coecient
matrix in (21), which also can be viewed as an alternative to implementing Theorem 4.1. To
be precise, we consider
B
Up
−1
˜
A := VAV = T −1
q U
where V and B are dened as in Theorem 4.1. Then (21) can be expressed as
x
f
˜
=V
AV
g
or equivalently,
x
f
=
(23)
g
x
x
Ux
f
Uf
f
where
=V
=
, and analogously,
=V
=
. Similar to the technique
g
g
g
described in [23, Section 3.2], we split A˜ into the sum of A and a low rank modication L:
A˜
A˜ = A + L
where A is dened as in Theorem 4.1, and L = eN +1 vT with vT = [qT U −1 − (Up)T 0]T . If
−1
vT A eN +1 = −1, by using the Sherman–Morrison formula, the solution to (23) can be obtained
from
−1
x
A eN +1 vT
−1 f
= I−
(24)
A
−1
g
1 + vT A eN +1
Equation (24) requires the solution of two linear systems involving the same symmetric
The numerical method described in the previous sections can be used
invertible matrix A.
to solve these two linear systems. In practice, the discretization matrices corresponding to
the Laplacian and the biharmonic operators on the reduced subdomains are diagonally quasisymmetric. That is, the similarity transformation U is a diagonal matrix. Thus, the solution
to the original linear system (21) can be easily obtained.
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
APPLICATION OF THE LANCZOS ALGORITHM
347
Ω3
Ω1
Ω2
Figure 1. An L-shaped domain subdivided into three subdomains.
5. NUMERICAL RESULTS
Example 1
We consider the following semilinear elliptic eigenvalue problem dened on an L-shaped
region with homogeneous Dirichlet boundary conditions:
u + sinh u=0
u=0
in on @
(25)
where is obtained by cutting away a quarter of the unit square. Suppose that is divided
into three subregions as shown in Figure 1. We discretize (25) along the trivial solution
branch u ≡ 0 by the centred dierences with uniform meshsize h = 0:02 on the x- and y-axis,
respectively. Assume that edge-based partitioning is used on the domain . We label the
nodes by subdomain as shown in [1, p. 387], where the interface nodes are labelled last. The
linear system corresponding to Dx H (0) is of the following form:
 (1)
   
A
0
0
E1
r1
f1
 0




(2)
0
E2   r2   f2 
A


(26)

  = 
(3)
 0




r
f
0
A
E3
3
3
s
g
F1
F2
F3 C
where each ri represents the subvector of unknowns that are interior to subdomain i , and s
represents the vector of all interface unknowns. Moreover, E ≡ [E1 ; E2 ; E3 ]T represents the subdomain to interface coupling seen from the subdomains, while F ≡ [F1 ; F2 ; F3 ] = E T represents
the interface to subdomain coupling seen from the interface nodes. For = 0 we have
 (1)

A
0
0


(27)
A = 0
A(2)
0 ∈ R1875×1875
0
Copyright ? 2002 John Wiley & Sons, Ltd.
0
A(3)
Numer. Linear Algebra Appl. 2003; 10:335–355
348
C.-S. CHIEN AND S.-L. CHANG
where

A25
 −I25




A(i) =






−I25
A25
−I25
−I25
A25
..
.
..
.
..
.
−I25
−I25
A25
−I25





∈ R625×625 ;




−I25 
i=1:3
A25
(i)
with
A25 ∈ R25×25
obtained from A by replacing A25 and I25 by 4 and 1, respectively, and
A
0
∈ R50×50 and E ∈ R1875×50 . The non-zero entries of E can be expressed as:
C = 25
0 A25
For K = 1 : 25
For I = 0 : 1
E(601 × I + 25 × K; K) = −1
E(650 × I + 626 − K; 25 + K) = −1
Note that A is symmetric positive denite. Let 1 be the rst discrete bifurcation point of
the discrete problem associated with (25). If we choose ¿1 , then the coecient matrix
corresponding to the discrete problem becomes symmetric indenite.
The numerical methods described in the previous sections were implemented to solve three
symmetric linear systems Ax(j) = b(j) , j = 1; 2; 3. Here the coecient matrix A is dened as
in (27), b(1) = [1; : : : ; 1]T , b(2) = [1:1; : : : ; 1:1]T , and b(3) is a random vector chosen by the
computer such that b(3) − b(1) 2 ¿103 . First, we solved the rst linear system by using the
preconditioned Lanczos method with various preconditioners ILLT (k), k = 0; 1; 2; 3. In order to
supply the other two linear systems with starting approximate solution vectors, we performed:
(a) the Lanczos–Galerkin projection method; (b) the method of Papadrakakis and Smerou;
(c) the iterative renement. For completeness, let zi(j) be the approximate solution for the jth
linear system, j = 2; 3, which is obtained by implementing (a), (b), and (c) with i iterative
renements. Here and in the sequel the subscript i is required only when (c) is used. We
also let := r (2) 2 , := r (3) ∞ , where ri(j) = b(j) − Azi(j) . To improve the accuracy of the
approximate solutions provided by the methods mentioned above, we implemented the preconditioned Lanczos method with various preconditioners again for the second and the third
linear systems. Table I lists our numerical results, where the total number of oating point
operations (in million) and the total execution time (in seconds) of each method are also
included. The numerical continuation of the rst three non-trivial solution branches of (25)
can be found in Reference [26].
From Table I we see that (b) can provide an approximate solution for the second linear
system which is accurate up to 10−5 if b(2) − b(1) is small enough. Moreover, for any
random right-hand side vectors b(3) satisfying b(3) − b(1) 2 ¿103 , the method (b) did supply
an approximate solution with a smaller residual norm compared to the counterpart of (a).
However, it takes almost the same number of iterations for both methods to improve the
accuracy of the approximate solutions. In other words, both (a) and (b) are not as eective
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
349
APPLICATION OF THE LANCZOS ALGORITHM
Table I. Implementing the preconditioned Lanczos method with various preconditioners.
(a)
(b)
(c)
ILLT (0)
NI(1)
NI(2)
b(3) − b(1) 2
NI(3)
Total ops
Total execution time
32
3:6576 × 10−1
20
2:4966 × 103
1:9635 × 102
36
236:1495
5652
32
3:1543 × 10−5
16
2:4966 × 103
2:1427 × 101
35
220:1163
5416
32
30 = 3:9927 × 101
12
2:4966 × 103
30 = 7:7749
15
164:0906
4028
ILLT (1)
NI(1)
NI(2)
b(3) − b(1) 2
NI(3)
Total ops
Total execution time
21
3:9447 × 10−1
13
2:4804 × 103
1:9497 × 102
24
155:5151
3815
21
2:3416 × 10−6
7
2:4804 × 103
1:1563 × 101
23
138:8706
3486
21
22 = 9:4928 × 10−1
7
2:4804 × 103
25 = 9:3768 × 10−1
7
106:7372
2682
ILLT (2)
NI(1)
NI(2)
b(3) − b(1) 2
NI(3)
Total ops
Total execution time
17
3:9837 × 10−1
11
2:4785 × 103
2:0084 × 102
19
122:8131
3042
17
5:2194 × 10−6
7
2:4785 × 103
6.1994
18
114:9997
2814
17
15 = 7:8248 × 10−1
7
2:5123 × 103
16 = 8:9301 × 10−1
8
88:3496
2146
ILLT (3)
NI(1)
NI(2)
b(3) − b(1) 2
NI(3)
Total ops
Total execution time
16
4:0887 × 10−1
10
2:4625 × 103
1:4690 × 102
18
118:7478
2941
16
1:0158 × 10−5
6
2:4669 × 103
6.0067
16
105:4434
2566
16
12 = 7:3050 × 10−1
7
2:4427 × 103
12 = 9:8810 × 10−1
8
80:9197
1959
Note: ILLT (j), j = 0 : 3, for solving Ax(j) = b(j) , j = 1 : 3, where the methods (a), (b), and (c) are dened as
above, stopping criterion = 10−10 , NI(j) := number of iterations required for solving the jth linear system.
as (c) for solving the third linear system. Figures 2–4 show the convergence behaviour of
the preconditioned Lanczos method with various preconditioners ILLT (j), j = 0 : 3, and the
iterative renement, respectively.
Finally, the preconditioned single seed algorithm was implemented to solve the three linear systems given above, where we used the same preconditioners and stopping criterion as
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
350
C.-S. CHIEN AND S.-L. CHANG
2
: ILLT(0)
0
__ILL T(1)
log of residual norm
_ ILLT(2)
_2
_. ILLT(3)
_4
_6
_8
_10
_12
0
5
10
15
20
iterations
25
30
35
Figure 2. Convergence behaviour of the preconditioned Lanczos method with various preconditioners
ILLT (j), j = 0 : 3, for solving Ax(1) = b(1) .
2
log of residual norm
1.5
ILLT(0)
1
0.5
0
ILLT(3)
ILLT(2)
T
ILL (1)
_0.5
0
5
10
15
iterations
20
25
30
Figure 3. Convergence behaviour of the iterative renement for solving Ax(3) = b(3) .
Table I. Table II lists our numerical results. Because of its specic design, we see that the
(preconditioned) single seed algorithm, which is a CG-type algorithm, is cheaper than the
Lanczos-type algorithms we have discussed.
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
351
APPLICATION OF THE LANCZOS ALGORITHM
0
T
: ILL (0)
log of residual norm
_2
__ILL T(1)
_ ILLT(2)
_4
_. ILLT(3)
_6
_8
_10
_12
0
5
10
15
iterations
Figure 4. Convergence behaviour of the preconditioned Lanczos method with various preconditioners
ILLT (j), j = 0 : 3, for solving Ax(3) = b(3) .
Table II. Implementing the single seed algorithm for Example 1.
ILLT (0)
ILLT (1)
ILLT (2)
ILLT (3)
NI(1)
NI(2)
NI(3)
Total ops
Execution time
29
19
15
14
16
11
9
8
30
19
16
14
62.5719
41.3210
33.9447
30.6880
1609
1432
1140
1027
Example 2
Exploiting symmetry of the domain. To trace the rst solution branch of the following nonlinear eigenvalue problem:
u + sinh u=0
u=0
in = [0; 1]2
(28)
on @
we follow the rst solution branch of its reduced problem
u + sinh u=0
in = [0; 12 ]2
u=0 on x = 0
@u=@n=0
Copyright ? 2002 John Wiley & Sons, Ltd.
on x =
and y = 0
1
2
and y =
(29)
1
2
Numer. Linear Algebra Appl. 2003; 10:335–355
352
C.-S. CHIEN AND S.-L. CHANG
Equation (29) was discretized by the ve-point centred dierence approximations with uniform
meshsize h = 0:025 on the x- and y-axis, respectively. The associated discrete non-linear
system can be expressed as
H (U; ) = BU − h2 sinh U = 0
(30)
where B ∈ RK ×K is the coecient matrix associated to the Laplacian in (29) and U = [U1 ; U2 ;
: : : ; UK 2 ]T with K = 20. The Jacobian matrix of H is denoted by DH = [B p] with
2
2
B := DU H = B − h2 diag(cosh U1 ; : : : ; cosh UK 2 )
and
p := D H = −h2 [sinh U1 ; : : : ; sinh UK 2 ]T
Since B is diagonally quasi-symmetric, we can nd a diagonal matrix


D = diag 1 ; : : : ; K ; : : : ; 1 ; : : : ; K ; 1 ; : : : ; K 
1st
K−1th
Kth
with
1
i := √
for i = 1; : : : ; K − 1
2=2
for i = K
√
and
i :=
2=2 for i = 1; : : : ; K − 1
1=2
for i = K
such that B̃ = DBD−1 is symmetric, where

B̃K
 −I
 K


B̃ =





−IK
B̃K
..
.
−IK
..
.
..
.
..
.
..
.
√
− 2IK
√
− 2IK




 − h2 diag(cosh U1 ; : : : ; cosh UK 2 )




B̃K
with h = 1=2K, and with B̃K ∈ RK×K obtained from B̃ + h2 diag(cosh U1 ; : : : ; cosh UK 2 ) by replacing B̃K and IK by 4 and 1, respectively.
We traced the rst discrete solution branch of (30), where Theorem 4.1 combined with
the Lanczos method was applied to solve the associated linear systems. Table III lists our
sample numerical result, where the rst bifurcation point is detected at ≈ 19:7274. Figures
5 and 6 show the solution curve and its contour at ≈ 19:6555, respectively. Next, we used
the restarted Lanczos–Galerkin method to solve the two linear systems that appeared in (24).
The latter is more ecient than the former, since the average number of iterations required
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
353
APPLICATION OF THE LANCZOS ALGORITHM
Table III. Sample result for Example 2, h = 0:025, ” = 5:0 × 10−7 , tol = 10−10 , = 19:7274, using the
Lanczos method.
NCS
MAXNORM
2
Itr
NCI
Aitr
1
5
15
25
35
50
60
75
90
105
19.7274
19.6741
19.6006
19.4950
19.3581
19.0969
18.8873
18.3368
17.4118
11.5520
0.02993
0.17019
0.26319
0.35691
0.45092
0.53222
0.68654
0.89373
1.17739
3.84965
0.4827E+09
0.1517E+08
0.6279E+07
0.3411E+07
0.2139E+07
0.1244E+07
0.9286E+06
0.5519E+06
0.3218E+06
0.3079E+05
92
89
89
88
88
88
87
87
87
86
4
4
2
2
2
2
2
2
2
2
92
89
89
88
88
88
87
87
87
86
6
5
4
3
2
1
0
2
4
6
8
10
12
14
16
18
20
Figure 5. The solution curve u of Equation (29).
in the corrector process is almost reduced by half. The other results are similar to those of
Table III and are omitted here. The following notation is used in Table III.
NCS
”
2
tol
NCI
MAXNORM
ordering of the continuation steps
accuracy tolerance in Newton corrector
the two-norm condition number of Tm
stopping criterion for the Lanczos method
numbers of corrector iterations required at each continuation step
maximum norm of the approximating solution u
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
354
C.-S. CHIEN AND S.-L. CHANG
0.5
0.45
* _.1777
0.4
0.35
* _.1359
0.3
0.25
*_ .1130
0.2
*_.0601
*_.0151
0.15
0.1
0.05
0
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
Figure 6. Contour of the solution branch at ≈ 19:6555.
itr
aitr
iteration numbers required by using the Lanczos method to solve
linear systems in the predictor step
average iteration numbers required by using the Lanczos method
to solve linear systems in the corrector step
ACKNOWLEDGEMENTS
The authors thank two anonymous referees for their valuable suggestions.
REFERENCES
1. Saad Y. Iterative Methods for Sparse Linear Systems. PWS Publishing Co.: Boston, 1996.
2. Golub GH, Van Loan CF. Matrix Computations (3rd edn). The Johns Hopkins University Press: Baltimore,
MD, 1996.
3. Parlett BN. A new look at the Lanczos algorithm for solving symmetric systems of linear equations. Linear
Algebra and Its Applications 1980; 29:323 –346.
4. Saad Y. On the Lanczos method for solving symmetric linear systems with several right-hand sides. Mathematics
of Computation 1987; 48:651– 662.
5. Smith CF, Peterson AF, Mittra R. A conjugate gradient algorithm for the treatment of multiple incident
electromagnetic elds. IEEE Transactions Antennas and Propagation 1989; 37:1490 –1493.
6. Chan TF, Wan WL. Analysis of projection methods for solving linear systems with multiple-right-hand sides.
SIAM Journal on Scientic Computing 1997; 18:1698 –1721.
7. Papadrakakis M, Smerou S. A new implementation of the Lanczos method in linear problems. International
Journal for Numerical Methods in Engineering 1990; 29:141–159.
8. Erhel J, Guyomarc’h F. An augmented conjugate gradient method for solving consecutive symmetric positive
denite linear systems. SIAM Journal on Matrix Analysis and Applications 2000; 21:1279 –1299.
9. Simoncini V, Gallopoulos E. An iterative method for non-symmetric systems with multiple right-hand sides.
SIAM Journal on Scientic Computing 1995; 16(4):917– 933.
10. Simoncini V, Gallopoulos E. Convergence properties of block GMRES and matrix polynomials. Linear Algebra
and Its Applications 1996; 247:97–119.
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355
APPLICATION OF THE LANCZOS ALGORITHM
355
11. Parlett BN. The Symmetric Eigenvalue Problems. Prentice-Hall: Englewood Clis, NJ, 1980.
12. Simon HD. Analysis of the symmetric Lanczos algorithm with reorthogonalization methods. Linear Algebra
and Its Applications 1984; 61:101–131.
13. Govaerts WJF, Pryce JD. Block elimination with one iterative renement solves bordered linear systems
accurately. BIT 1990; 30:490 – 507.
14. Govaerts WJF. Stable solvers and block elimination for bordered systems. SIAM Journal on Matrix Analysis
and Applications 1991; 12:469 – 483.
15. Govaerts WJF. Numerical Methods for Bifurcations of Dynamical Equilibria. SIAM: Philadelphia, 2000.
16. Watson LT, Sosonkina M, Melville RC, Morgan AP, Walker HF. Algorithm 777: HOMPACK 90: a suite of
Fortran 90 codes for globally convergent homotopy algorithm. ACM Transactions on Mathematical Software
1997; 23:514 – 549.
17. Chien C-S, Chang S-L, Mei Z. Tracing the buckling of a rectangular plate with block GMRES method. Journal
of Computational and Applied Mathematics 2001; 136:199 – 218.
18. Chien C-S, Lin W-W, Mei Z. Conjugate gradient type methods for semilinear elliptic problems with symmetry.
Computers and Mathematics with Applications 1999; 37:3 –22.
19. Chien C-S, Weng Z-L, Shen C-L. Lanczos-type methods for continuation problems. Numerical Linear Algebra
with Applications 1997; 4:23 – 41.
20. Irani KM, Kamat MP, Ribbens CJ, Walker HF, Watson LT. Experiments with conjugate gradient algorithms
for homotopy curve tracking. SIAM Journal on Optimization 1991; 1:222–251.
21. Allgower EL, Georg K. Numerical path following. In Handbook of Numerical Analysis, Ciarlet PG, Lions JL
(eds), vol. 5. North-Holland: Amsterdam, 1996.
22. Keller HB. Lectures on Numerical Methods in Bifurcation Problems. Springer-Verlag: Berlin, 1987.
23. Desa C, Irani KM, Ribbens CJ, Watson LT, Walker HF. Preconditioned iterative methods for homotopy curve
tracking. SIAM Journal on Scientic and Statistical Computing 1992; 13:30 – 46.
24. Chien C-S, Kuo YJ, Mei Z. Symmetry and scaling properties of the von Karman equations. Journal of Applied
Mathematics and Physics (ZAMP) 1998; 49:710 –729.
25. Axelsson O. Iterative Solution Methods. Cambridge University Press: Cambridge, 1996.
26. Chien C-S, Chow H-S, Jeng B-W. A continuation-domain decomposition algorithm for bifurcation problems.
Numerical Algorithms 1999; 22:367–383.
27. O’Leary D. The block conjugate gradient algorithm and related methods. Linear Algebra and Its Applications
1980; 29:293 –322.
28. Simon HD. The Lanczos algorithm with partial reorthogonalization. Mathematics of Computation 1984; 42:
115 –142.
Copyright ? 2002 John Wiley & Sons, Ltd.
Numer. Linear Algebra Appl. 2003; 10:335–355