A Note on the Complexity of the PCG Algorithm for Solving
Toeplitz Systems with a Fisher-Hartwig Singularity
Seak-Weng VONG1∗ , Wei WANG2 , Xiao-Qing JIN3
1 Department
of Mathematics, University of Macau, Macao, China.
E-mail: [email protected]
2 Department
of Mathematics, University of Macau, Macao, China.
3 Department
of Mathematics, University of Macau, Macao, China, (The research of this
author is supported by the Grant 050/2005/A from FDCT.)
E-mail: [email protected]
Abstract Recently, Y. Lu and C. Hurvich showed that the complexity of T. Chan’s preconditioned
¡
¢ conjugate gradient algorithm for solving the Toeplitz system Tn (f )x = b is
O n log3 n where the generating function f is given by
f (ω) = |1 − e−iω |−2d h(ω)
¡
¢
with d ∈ − 12 , 12 \{0} and h(ω) being positive continuous on [−π, π] and differentiable on
[−π, π]\{0}. Although their results are interesting, there exist some improper expressions
in their proofs needed to be corrected. In this paper, we try to improve those improper
expressions and demonstrate these important results by some numerical tests.
Key Words: PCG algorithm, Toeplitz matrix, circulant matrix, Fisher-Hartwig singularity,
T. Chan’s circulant preconditioner.
AMS subject classifications: 65F10, 65F15, 65L05, 65N22.
1
Introduction
In this paper, we consider to use the preconditioned conjugate gradient (PCG) algorithm to
solve some special Toeplitz systems. The PCG algorithm has been a popular and effective
iterative method for solving Toeplitz systems Tn x = b since 1986 ([11]). It is well known that
Toeplitz systems arise in a variety of applications in science and engineering [2, 6, 7, 8, 10].
The Toeplitz matrix is defined as follows:

t0
t1
..
.
t−1
t0



t1
Tn = 


 tn−2 · · ·
tn−1 tn−2
· · · t2−n t1−n
t−1 · · · t2−n
..
..
.
t0
.
..
..
.
.
t−1
···
t1
t0








(1)
where the entries are given by
Z π
√
1
tk =
f (ω)e−ikω dω,
i = −1,
2π −π
for k = 0, ±1, ±2, · · ·. The function f is called the generating function of Tn . In the following,
we assume that Tn is symmetric, i.e., tk = t−k . The complexity of the PCG algorithm
with some suitable preconditioners, for instance, T. Chan’s circulant preconditioner and
superoptimal circulant preconditioner, is known to be O (n log n) for Toeplitz systems where
the generating function f (ω) is positive continuous on [−π, π], see [2, 6, 10].
Recently Y. Lu and C. Hurvich considered ([9]) the CG algorithm with T. Chan’s preconditioner for solving Tn x = b with the generating function f (ω) satisfying the following
assumption:
Assumption 1 ([4, 9]) The generating function f is given by
f (ω) = |1 − e−iω |−2d h(ω),
ω ∈ [−π, π],
¡ 1 1¢
where d ∈ − 2 , 2 and h(ω) is even positive continuous on [−π, π] and differentiable on
[−π, π]\{0}. Also log h is Riemann integrable on [−π, π], i.e.,
¯
Z π¯ 0
¯ h (ω) ¯
¯
¯
¯ h(ω) ¯ dω < ∞,
−π
and there exists c ∈ (0, ∞) such that
|h0 (ω)| ≤ c|ω|−1 ,
ω ∈ [−π, π]\{0}.
Moreover, h(ω) is assumed to be in B11
1
h(ω) .
T
L∞ and all conditions for h(ω) are also applied to
We recall that the Besov space B11 includes all functions g such that g ∈ L1 and
Z π
Z
1 π
|g(ω + t) − 2g(ω) + g(ω − t)|dωdt < ∞.
2
−π t
−π
Now let us give the definition of T. Chan’s circulant preconditioner proposed in [3]:
Definition 1 ([2, 3]) For Tn given as in (1), the diagonals of T. Chan’s circulant preconditioner cF (Tn ) are given by

(n − k)tk + ktk−n


, if
n
ck =


cn+k ,
if
0 ≤ k < n,
0 < −k < n.
Under Assumption 1, Tn has a Fisher-Hartwig singularity caused by the term |1−e−iω |−2d
in the generating function f and the condition number of Tn approaches ∞ as n increases,
see [4]. Y. Lu and C. Hurvich showed in ([9]) that the complexity of the CG algorithm
for solving Tn x = b without any preconditioning grows asymptotically as n1+|d| log n. With
T. Chan’s
optimal
circulant preconditioner cF (Tn ), the complexity of the PCG algorithm
¡
¢
is O n log3 n . But there exist some improper expressions in their proofs needed to be
corrected or revised. In this paper, we try to improve their proofs and demonstrate these
important results by some numerical examples.
2
Spectral analysis
We first introduce the following lemma and the proof can be found in [9].
Lemma 1 ([9]) If f satisfies Assumption 1, we then have
h
i
−1/2
−1/2
λmax cF (Tn (f )) Tn (f )cF (Tn (f )) = O(log2 n),
¡
¢¤
£
−1
λmin c−1
Tn (f −1 ) ≥ C1 ,
F (Tn (f )) cF
h
¢i
¢
1/2 ¡
1/2 ¡
λmin cF Tn (f −1 ) Tn−1 (f −1 )cF Tn (f −1 ) ≥ C2 · log−2 n,
where Ci , i = 1, 2, represent positive constants independent of n, λmin (M ) and λmax (M )
denote the smallest and the largest eigenvalues of M respectively.
Lemma 2 ([9]) If f satisfies Assumption 1, then Tn (f ) − Tn−1 (f −1 ) is positive semidefinite.
Proof: Let
³
´T
1
un (ω) = √ f −1/2 (ω) 1, eiω , · · · , ei(n−1)ω ,
2π
and
³
´T
1
vn (ω) = √ f 1/2 (ω) 1, eiω , · · · , ei(n−1)ω .
2π
Then,
Z π
un (ω)u∗n (ω)dω = Tn (f −1 ),
Z
−π
and
Z π
−π
π
−π
vn (ω)vn∗ (ω)dω = Tn (f )
un (ω)vn∗ (ω)dω = In .
Note that for any α(ω) = (α1 (ω), α2 (ω), · · · , αn (ω))T , the matrix
Z
B = α(ω)α∗ (ω)dω
is positive semidefinite since
Z
∗
x Bx = |x∗ α|2 dω ≥ 0,
for any x ∈ Cn . Therefore, for any n × n matrix A, we have
Z π
B=
[Aun (ω) + vn (ω)][Aun (ω) + vn (ω)]∗ dω = ATn (f −1 )A∗ + A + A∗ + T (f ).
(2)
−π
In particular, if we take A = −T −1 (f −1 ) in (2), we can conclude that Tn (f ) − Tn−1 (f −1 ) is
positive semidefinite.
Theorem 1 ([9]) If f satisfies Assumption 1, then we have
h
i
−1/2
−1/2
λmin cF (Tn (f )) Tn (f )cF (Tn (f )) ≥ C · log−2 n,
where C is a positive constant independent of n.
−1/2
Proof: We can factorize cF
−1/2
cF
−1/2
(Tn (f )) Tn (f )cF
−1/2
(Tn (f )) = B1 B2 B3 + B4 ,
−1/2
(Tn (f −1 )),
(Tn (f )) Tn (f )cF
(Tn (f )) as
where
−1/2
B1 = cF
(Tn (f ))cF
1/2
1/2
B2 = cF (Tn (f −1 ))Tn−1 (f −1 )cF (Tn (f −1 )),
−1/2
(Tn (f −1 ))cF
−1/2
(Tn (f ))[Tn (f ) − Tn−1 (f −1 )]cF
B3 = cF
B4 = cF
−1/2
(Tn (f )),
−1/2
(Tn (f )).
(3)
By using Lemmas 1 and 2, we notice that B1 , B2 , B3 are all Hermitian positive definite and
B4 is Hermitian positive semidefinite. Therefore, we have
n
o
−1/2
−1/2
λmin cF (Tn (f ))Tn (f )cF (Tn (f ))
≥ λmin (B1 B2 B3 ) + λmin (B4 )
≥ λmin (B1 )λmin (B2 )λmin (B3 ) + λmin (B4 )
≥ C · log−2 n + λmin (B4 )
≥ C · log−2 n.
Thus, the condition number of T. Chan’s preconditioned matrix satisfies
¢
¡
κ c−1
F (Tn (f ))Tn (f )
h
i
−1/2
−1/2
= κ cF (Tn (f ))Tn (f )cF (Tn (f ))
µ
= O
log2 n
C · log−2 n
¶
= O(log4 n).
Therefore, the number of iterations in the PCG algorithm is bounded by O(log2 n), see [5, 7].
Since each iteration needs O(n log n) operations using the FFT, the total complexity of the
PCG algorithm is still O(n log3 n).
3
Numerical tests
In this section, we verify Theorem 1 by the following problem.
Problem 1 ([1]) The generating function is given by
³ ω ´¯−2d
σ 2 ¯¯
¯
f (ω) = s ¯2 sin
−π ≤ ω ≤ π.
¯ ,
2π
2
Then we obtain
tk =
σs2 Γ(1 − 2d)Γ(k + d)
,
Γ(d)Γ(1 − d)Γ(k − d + 1)
k = 0, 1, 2, · · · .
Also, we choose b = (t1 , t2 , · · · , tn )T as the right-hand side of Tn x = b.
250
30
d=0.37
d=0.1
25
Number of iterations: y=f(x)
Number of iterations: y=f(x)
200
T. Chan Preconditioner
150
No Preconditioner
100
50
0
T. Chan Preconditioner
20
No Preconditioner
15
10
5
0
5
10
0
15
0
x
5
10
15
The number of matrix size (n=2x)
The number of matrix size (n=2 )
Figure 1: Two examples of iteration numbers.
All the experiments were performed in MATLAB. We used the MATLAB-provided M-file
“pcg” to solve the system. In our tests, the zero vector is the initial guess and the stopping
criterion is krj k2 /kr0 k2 < 10−10 where rj is the residual after the j-th iteration. We consider
the following two cases in Problem 1:
Case 1: Choose d = 0.37 and σs2 = 0.27.
Case 2: Choose d = 0.1 and σs2 = 0.27.
In Table 1, we give the number of iterations for convergence. In the table, n is the matrix
size. We compare the preconditioned system by using T. Chan’s preconditoner with the
system with no preconditioner. We note that when n increases, the number of iterations of
the preconditioned system is much less than that of the system with no preconditioner.
n
Case1
No preconditioner
Case1
T. Chan’s preconditioner
Case2
No preconditioner
Case2
T. Chan’s preconditioner
25
26
27
28
29
210
211
212
213
214
215
18
25
32
40
52
65
81
103
130
168
214
7
7
8
8
8
9
9
10
10
10
12
11
13
14
16
17
19
20
22
23
25
28
6
6
7
7
7
7
7
8
8
8
9
Table 1: Number of iterations for PCG algorithm.
Acknowledgments. The authors would like to thank Prof. Clifford M. Hurvich for his
useful suggestions on this topic.
References
[1] P. Brockwell and R. Davis, Time Series: Theory and Methods, Springer-Verlag, New
York, 1991.
[2] R. Chan and M. Ng, Conjugate Gradient Methods for Toeplitz Systems, SIAM Review,
Vol. 38 (1996), pp. 427–482.
[3] T. Chan, An Optimal Circulant Preconditioner for Toeplitz Systems, SIAM J. Sci.
Statist. Comput., Vol. 9 (1988), pp. 766–771.
[4] M. Fisher and R. Hartwig, Toeplitz Determinants–Some Applications, Theorems, and
Conjectures, Adv. Chem. Phys., Vol. 32 (1968), pp. 190–225.
[5] G. Golub and C. Loan, Matrix Computations, 3rd edition, Johns Hopkins University
Press, Baltimore, 1996.
[6] X. Jin, Developments and Applications of Block Toeplitz Iterative Solvers, Kluwer Academic Publishers, Dordrecht; and Science Press, Beijing, 2002.
[7] X. Jin and Y. Wei, Numerical Linear Algebra and Its Applications, Science Press, Beijing, 2004.
[8] R. King, M. Ahmadi, R. Gorgui-Naguib, A. Kwabwe and M. Azimi-Sadjadi, Digital
Filtering in One and Two Dimensions: Design and Applications, Plenum, New York,
1989.
[9] Y. Lu and C. Hurvich, On the complexity of the Preconditioned Conjugate Gradient
Algorithm for Solving Toeplitz Systems with a Fisher–Hartwig Singularity, SIAM J.
Matrix Anal. Appl., Vol. 27 (2005), pp. 638–653.
[10] M. Ng, Iterative Methods for Toeplitz Systems, Oxford University Press, Oxford, 2004.
[11] G. Strang, A Proposal for Toeplitz Matrix Calculations, Stud. Appl. Math., Vol. 74
(1986), pp. 171–176.