Optimal trigonometric preconditioners for nonsymmetric
Toeplitz systems
Daniel Potts
Medizinische Universitat zu Lubeck
Institut fur Mathematik
Wallstr. 40
D{23560 Lubeck
[email protected]
and
Gabriele Steidl
Universitat Mannheim
Fakultat fur Mathematik und Informatik
D{68131 Mannheim
[email protected]
Abstract. This paper is concerned with the solution of systems of linear equations
T N xN = bN , where fT N gN 2IN denotes a sequence of nonsingular nonsymmetricToeplitz
matrices arising from a generating function of the Wiener class. We present a technique
for the fast construction of optimal trigonometric preconditioners M N = M N (T 0N T N )
of the corresponding normal equation which can be extended to Toeplitz least squares
problems in a straightforward way. Moreover, we prove that the spectrum of the preconditioned matrix M ?N1T 0N T N is clustered at 1 such that the PCG-method applied to
the normal equation converges superlinearly. Numerical tests conrm the theoretical
expectations.
1991 Mathematics Subject Classication. 65F10, 65F15, 65T10.
Key words and phrases. Toeplitz matrix, Krylov space methods, CG-method, preconditioners, normal equation, clusters of eigenvalues.
1
1 Introduction
Consider the system of linear equations
T N xN = bN ;
(1.1)
(M ?N1T N )(M ?N1T N )xN = (M ?N1T N )M ?N1bN :
(1.2)
where T N 2 IRN;N denotes a nonsingular Toeplitz matrix. Toeplitz systems arise in
a variety of applications in mathematics and engineering (see [10] and the references
therein). While there exist fast and stable direct Toeplitz solvers for Hermitian positive
denite Toeplitz matrices T N , the non-Hermitian case requires additional eort (see
e.g. [22]). Iterative methods like GMRES and CG often provide a fast solution of (1.1)
if they are applied in connection with preconditioning techniques [10]. In particular,
these methods prot from the fact that the vector multiplication with the Toeplitz
matrix T N in each iteration step can be computed with O(N log N ) arithmetical operations by using the fast Fourier transform (FFT). Clearly, the multiplication with the
preconditioned matrix should have the same or smaller arithmetic complexity. Two
types of preconditioners are mainly exploited for linear Toeplitz systems, namely optimal (Cesaro) circulant preconditioners M N = C N (T N ) [14] and more simple so-called
\Strang" circulant preconditioners M N = SN (T N ) [12]. One reason for the choice of
circulant preconditioners is the fact that circulant matrices can be diagonalized by the
N ?1
Fourier matrix F N := p1N e?2ijk=N j;k=0 , where the multiplication of a vector with F N
takes only O(N log N ) arithmetical operations. Moreover, under certain assumptions
on the generating function of T N (see [13], [34]), it can be proved that the singular values of M ?N1 T N are clustered at 1. For non-Hermitian T N , this results in a superlinear
convergence of the CG-method applied to the system
To our knowledge, up to now, for non-Hermitian Toeplitz systems and Toeplitz least
squares problems only circulant preconditioners were constructed. See [13, 8] for circulant preconditioners with respect to some kind of normal equation (1.2), [9] for so-called
displacement preconditioners and [15]. Note that another approach to the solution of
(1.1) uses relations between Toeplitz{like matrices and Cauchy{like matrices [16, 25].
When we nished the paper, we became aware of new results of E.E. Tyrtyshnikov et
al. concerning the convergence behaviour of the preconditioned GMRES-method [35]
which avoids the transition of (1.1) to the normal equation. However, the preconditioners are again (improved) circulants, which were constructed with respect to T N .
In this paper, we restrict our attention to nonsymmetric real Toeplitz matrices T N .
Here, it seems to be natural, to replace the circulant matrices by matrices which are
diagonalizable by some real trigonometric matrices. In this way, arithmetic with complex numbers can be completely avoided. Of course, the commonly used trigonometric
transforms are closely related to the Fourier transform. Indeed, for symmetric Toeplitz
matrices T N with positive continuous 2{periodic generating functions, trigonometric preconditioning signicantly accelates the convergence of the CG-method (see
[3, 5, 4, 7, 11, 19, 24]).
2
In this paper, we suggest the solution of (1.1) by applying the CG-method to the preconditioned normal equation
M ?N1T 0N T N xN = M ?N1T 0N bN ;
where in contrast to (1.2), M N = M N (T 0N T N ) denotes the optimal preconditioner
with respect to T 0N T N . We demonstrate that the construction of such optimal preconditioners can be realized with only O(N log N ) arithmetical operations despite the fact
that T 0N T N is no longer a Toeplitz matrix. We prove that under certain assumptions
on T N the eigenvalues of M ?N1T 0N T N are clustered at 1. Although our approach works
in exactly the same way for dierent trigonometric transforms, we prefer to investigate the DCT{II preconditioner in detail and add only few facts concerning the other
trigonometric preconditioners. We hope that our notation makes the approach for other
trigonometric preconditioners immediately clear. Numerical tests were performed for
the dierent trigonometric preconditioners. In all examples our preconditioning was
superior over the method (1.2) with an optimal trigonometric preconditioner M N (T N )
of T N .
This paper is organized as follows: Section 2 contains the basic matrix notation. In
Section 3, we study the relations between trigonometric transforms and Toeplitz matrices. In particular, we introduce a method for the fast vector multiplication with real
nonsymmetric Toeplitz matrices based on real trigonometric transforms. In Section 4,
we introduce optimal trigonometric preconditioners. Section 5 is concerned with the
proof that the eigenvalues of the preconditioned matrix M ?N1T 0N T N are clustered at 1.
In Section 6, we present the fast construction of the optimal preconditioner. Finally,
Section 7 conrms the theoretical expectations by numerical tests.
2 Notation
For the sake of clarity, we collect the matrix notation in this preliminary section.
Let aN := (a0; : : :; aN ?1)0, bN := (b0; : : :; bN ?1)0 and let oN be the vector consisting of
N zeros. Here A0 is the transpose of A. By I N we denote the (N,N)-identity matrix
and by ek 2 IRN the k-th identity vektor. To describe Toeplitz and Hankel matrices,
we use the following
0 notation:
1
a
a
:
:
:
a
a
0
1
N
?
2
N
?
1
BB b1 a0 : : : aN ?3 aN ?2 CC
B .
CC (with a0 = b0) ;
... C
... . . . ...
toeplitz(a0; b0) := B
BB ..
B@ bN ?2 bN ?3 : : : a0 a1 CCA
bN ?1 bN ?2 : : : b1 a0
0
stoeplitz a : symmetric Toeplitz matrix with rst row a0,
atoeplitz a0: antisymmetric Toeplitz matrix with rst row a0, where a0 = 0,
3
:::
1
0
a
a
:
:
:
a
a
0
1
N
?
2
N
?
1
B
CC
B
a1 a2 : : : aN ?1 bN ?2 C
B
B
.
.
.
.
..
.. C
..
..
CC (with aN ?1 = bN ?1) ;
hankel(a0; b0) := B
B
B
@ aN ?2 aN ?1 : : : b2 b1 CA
aN ?1 bN ?2 : : : b1 b0
0
shankel a : persymmetric Hankel matrix with rst row a0,
ahankel a0: antipersymmetric Hankel matrix with rst row a0, where aN ?1 = 0.
Further, we introduce the matrices
0
1
0
1
0
1
0
1
:
:
:
0
0
Z 0N;1 := BB@ ... . . . ... CCA 2 IRN;N +1 ; Z 0N;2 = BB@ ... . . . ... ... CCA 2 IRN;N +1 ;
0 0 ::: 1
0 ::: 1 0
and
1
0
0
1
0
0
R0N := BB@ ... . . . ... ... CCA 2 IRN ?1;N +1 :
0 0 ::: 1 0
Let diag a be the diagonal matrix with diagonal a and let (A) := diag(ak;k )kN=0?1,
where ak;k is the (k; k){th entry of A. By
NX
?1
tr A := ak;k
k=0
we denote the trace of A. Moreover, we need the following matrix norms:
Spectral norm:
jjAjj2 := (maximum of the singular values of A)1=2,
Frobenius norm:
N ?1
jjAjjF := (j;kP=0 a2jk )1=2,
1{norm:
N ?1
jjAjj1 := maxf jP=0 jaj;kj : k = 0; : : : ; N ? 1g.
If it does not make confusion, we use the same notation for the norm of absolute
summable sequences a = fak gk2ZZ 2 l1, i.e.
X
jjajj1 := jak j :
k2ZZ
3 Trigonometric transforms and Toeplitz matrices
We introduce four discrete sine transforms (DST) and four discrete cosine transforms
(DCT) as classied by Wang [36]:
!N
2 1=2
jk
"Nj "Nk cos N
2 IRN +1;N +1;
DCT{I : C IN +1 := N
j;k=0
4
DCT{II :
DCT{III :
DCT{IV :
and
DST{I :
DST{II :
DST{III :
DST{IV :
p
!N ?1
2 1=2
j
(2
k
+
1)
N
"j cos 2N
2 IRN;N ;
C := N
j;k=0
II 0
III
N;N
C N := C N 2 IR ;
!N ?1
2 1=2
(2
j
+
1)(2
k
+
1)
C IVN := N
cos
2 IRN;N ;
4N
j;k=0
II
N
!N ?2
2 1=2
(
j
+
1)(
k
+
1)
sin
2 IRN ?1;N ?1;
S := N
N
j;k=0
!N ?1
2 1=2
(
j
+
1)(2
k
+
1)
II
N
N;N ;
SN := N
"j+1 sin
2
IR
2N
j;k=0
0
III
II
N;N
SN := SN 2 IR ;
!N ?1
2 1=2
(2
j
+
1)(2
k
+
1)
IV
sin
2 IRN;N ;
SN := N
4N
j;k=0
I
N ?1
where "Nk := 1= 2 (k = 0; N ) and "Nk := 1 otherwise. We refer to the corresponding
transforms as trigonometric transforms. It is well-known that the above matrices are
orthogonal and that the vector multiplication with any of these matrices takes only
O(N log N ) arithmetical operations. Fortunately, there exist implementations of algorithms for the vector multiplication with the above sine and cosine matrices, for example
a C{implementation based on [2] and [31].
Moreover, we use the slightly modied DCT{I and DST{I matrices
!N
!N ?1
jk
jk
I
N
2
I
C~ N +1 := ("k ) cos N
; S~N ?1 := sin N
j;k=0
j;k=1
and the slightly modied DCT{III and DST{III matrices
!N ?1
!N ?1
(2
j
+
1)
k
(2
j
+
1)(
k
+
1)
III
N
2
III
N
2
~
~
C N := ("k ) cos 2N
; SN := ("k+1) sin
:
2N
j;k=0
Then
j;k=0
C~ IN +1 C~ IN +1 = N2 I N +1 :
(3.1)
Theorem 3.1. There exist the following relations between trigonometric transforms
and Toeplitz matrices:
i) DCT{I and DST{I:
R0N C IN +1 D C IN +1RN = 12 stoeplitz(a0; : : :; aN ?2) + 12 shankel(a2; : : :; aN ?2; 0; 0) ;
5
S IN ?1 R0N D RN SIN ?1 = 12 stoeplitz(a0; : : : ; aN ?2) ? 21 shankel(a2; : : : ; aN ?2; 0; 0) ;
R0N C IN +1 D~ RN SIN ?1 = 12 atoeplitz(0; a1; : : :; aN ?2) + 21 ahankel(a2; : : : ; aN ?1; 0) ;
S IN ?1 R0N D~ C IN +1RN = ? 21 atoeplitz(0; a1; : : :; aN ?2) + 21 ahankel(a2; : : :; aN ?1; 0)
with
D := diag(d0; : : :; dN )0 ; D~ := diag(0; d~1; : : : ; d~N ?1; 0)0 ;
(d0; : : :; dN )0 := C~ IN +1 (a0; : : : ; aN ?2; 0; 0)0 ;
(d~1; : : :; d~N ?1)0 := S~IN ?1 (a1; : : : ; aN ?1)0 :
ii) DCT{II and DST{II:
II 0 0
II
C N Z N;2 D Z N;2C N
II 0 0
SN Z N;1 D Z N;1SIIN
II 0 0
C N Z N;2 D~ Z N;1SIIN
II 0 0 ~
SN Z N;1 D Z N;2C IIN
with
= 12 stoeplitz(a0; : : :; aN ?1) + 12 shankel(a1; : : : ; aN ?1; 0) ;
= 21 stoeplitz(a0; : : :; aN ?1) ? 21 shankel(a1; : : :; aN ?1; 0) ;
= 21 atoeplitz(0; a1; : : : ; aN ?1) + 21 ahankel(a1; : : : ; aN ?1; 0) ;
= ? 21 atoeplitz(0; a1; : : :; aN ?1) + 21 ahankel(a1; : : : ; aN ?1; 0)
D := diag(d0; : : :; dN )0 ; D~ := diag(0; d~1; : : : ; d~N ?1; 0)0 ;
(d0; : : :; dN )0 := C~ IN +1 (a0; : : : ; aN ?1; 0)0 ;
(d~1; : : :; d~N ?1)0 := S~IN ?1 (a1; : : : ; aN ?1)0 :
iii) DCT{IV and DST{IV:
C IVN D C IVN = 12 stoeplitz(a0; : : :; aN ?1) + 21 ahankel(a1; : : :; aN ?1; 0) ;
SIVN D SIVN = 21 stoeplitz(a0; : : : ; aN ?1) ? 12 ahankel(a1; : : :; aN ?1; 0) ;
C IVN D~ SIVN = 12 atoeplitz(0; a1; : : :; aN ?1) + 21 shankel(a1; : : :; aN ?1; 0) ;
SIVN D~ C IVN = ? 21 atoeplitz(0; a1; : : :; aN ?1) + 21 shankel(a1; : : : ; aN ?1; 0)
with
D := diag(d0; : : : ; dN ?1)0 ; D~ := diag(d~0; : : :; d~N ?1)0
0
(d0; : : :; dN ?1)0 := C III
N (a0; : : :; aN ?1) ;
0
(d~0; : : :; d~N ?1)0 := S~III
N (a1; : : :; aN ?1; 0) :
6
Note that for xed diagonal matrices D; D~ , the above decompositions into a Toeplitz
and a Hankel matrix are not unique.
Proof: We restrict the proofIIto the DCT{II. To simplify the notation, we drop the
index N and set C := Z N;2C N and D := diag(d0; : : :; dN )0. Then by
cos cos = 21 cos( ? ) + 21 cos( + ) ;
the (u; v){entry of the matrix C 0DC is
NX
?1
NX
?1
("Nk )2 dk cos (u ?Nv)k + 12 N2
("Nk )2 dk cos (u + vN+ 1)k ;
(C 0DC )u;v = 12 N2
k=0
k=0
or equivalently, since ?(?1)u?v dN = (?1)u+v+1 dN for arbitrary dN 2 IR,
N
N
X
X
(C 0DC )u;v = 21 N2 ("Nk )2 dk cos (u ?Nv)k + 12 N2 ("Nk )2 dk cos (u + vN+ 1)k :
k=0
k=0
Choosing dN 2 IR such that
N
X
("Nk )2 dk (?1)k = 0 ;
k=0
we get by symmetry properties of cosine function that
C 0D C = 12 stoeplitz(a0; : : :; aN ?1) + 12 shankel(a1; : : :; aN ?1; 0) ;
where
(a0; : : : ; aN ?1; 0)0 = N2 C~ IN +1 (d0; : : : ; dN )0 ;
i.e. by (3.1)
(d0; : : : ; dN )0 = C~ IN +1 (a0; : : :; aN ?1; 0)0 :
The other decomposition relations follow in a similar way by application of sin sin =
1 cos( ? ) ? 1 cos( + ) and sin cos = 1 sin( ? ) + 1 sin( + ).
2
2
2
2
Theorem 3.1 provides a new method for the fast multiplication of a real vector with
a real nonsymmetric Toeplitz matrix that avoids the complex arithmetic which comes
into the play if we exploit the usual FFT{based method for the fast vector { Toeplitz
matrix multiplication.
Corollary 3.2. (Fast vector multiplication with nonsymmetric Toeplitz matrices)
Let
T = T N := (tj?k )j;kN ?=01 = toeplitz((t0; t?1; : : : ; t?(N ?1)); (t0; t1; : : : ; tN ?1))
7
be given and let C := Z N;2C IIN , S := Z N;1SIIN . Then
~ ?
T = 12 (T + T 0) + 21 (T ? T 0) = C 0DC + S0DS + C 0DS
where
D := diag(d0; : : : ; dN )0 ; D~ := diag(0; d~1; : : :; d~N ?1; 0)0 ;
~ ;
S0DC
t +t
(d0; : : : ; dN )0 := C~ IN +1 (t0; t1 +2 t?1 ; : : : ; N ?1 2 ?(N ?1) ; 0)0 ;
(d~1; : : : ; d~N ?1)0 := S~IN ?1 ( t?1 2? t1 ; : : : ; t?(N ?1)2? tN ?1 )0 :
The vector multiplication with T requires except of O(N ) additions
- one DCT{I and one DST{I to build D and D~ in a precomputation step,
- one DCT{II and one DST{II,
- four multiplications of vectors with diagonal matrices,
~ and DSx ? DCx
~ ,
- one DCT{III and one DST{III of the vectors DCx + DSx
respectively, and takes therefore only O(N log N ) arithmetical operations.
Clearly, by Theorem 3.1, we can formulate similar algorithms for the fast multiplication
of vectors with Toeplitz or Hankel matrices with respect to the other trigonometric
transforms. Typewriting this paper, we got a ps-le of a paper of G. Heinig and K.
Rost [18], which contains results in a similar direction as presented in this section.
Following the hint of one of the referees, we realized that representations of symmetric
Toeplitz matrices based on DCT{I and DST{I were also given in [20, 21, 23].
4 Optimal trigonometric preconditioners
We are concerned with the solution of the system of linear equations
T N xN = bN
with a nonsingular nonsymmetric Toeplitz matrix T N 2 IRN;N . We intend to solve the
normal equation
T 0N T N xN = T 0N bN
(4.1)
by the CG-method. In Section 7, we will see that with a good preconditioner at hand,
this can be realized in a fast way. There are several requirements on a preconditioner
M N of (4.1) resulting from the construction and the convergence behaviour of the CGmethod as well as from the fact that the vector multiplication with T N requires only
O(N log N ) arithmetical operations. Therefore, we are looking for a preconditioner
with the following properties:
(P1) M N is symmetric and positive denite such that the bilinear form
(xN ; yN )M := x0N M N yN
N
arising in the left preconditioned CG-method is symmetric and positive denite, too.
8
(P2) The spectrum of M ?N1T 0N T N is clustered at 1.
(P3) The vector multiplication with M N can be computed with O(N log N ) arithmetical operations.
(P4) The construction of M N takes only O(N log N ) arithmetical operations.
Having property (P3) in mind, a straightforward idea consists in choosing M N from
an algebra
AO := fO0N (diag d) ON : d 2 IRN g
(4.2)
of matrices which are diagonalizable by some orthogonal matrix ON , where ON has the
additional property that its vector multiplication requires only O(N log N ) arithmetical
operations. As orthogonal matrices, we will use the trigonometric matrices of the
previous section which are closely related to the Fourier matrix F N , but have the
advantage of purely real entries. Moreover, if we choose M N 2 AO as so-called
optimal preconditioner of T 0N T N , then we will see that under certain assumptions on
T N , the properties
(P1), (P2) and (P4) are also fullled.
N;N
For AN 2 IR , the matrix M N (AN ) is called an optimal preconditioner of AN in
AO [14] if
jjM N (AN ) ? AN jjF = minfjjBN ? AN jjF : BN 2 AO g :
(4.3)
If ON is one of the orthogonal matrices which correspond to the DST{I, DST{II, DCT{
II, DST{IV or DCT{IV, respectively, then M N is said to be an optimal trigonometric
preconditioner of AN . The choice of the Frobenius norm in denition (4.3) results from
the fact that the Frobenius norm is induced by an inner product of IRN;N
NX
?1
hAN ; BN i := tr(A0N B N ) = aj;kbj;k :
(4.4)
N
N
N
N
j;k=0
In particular, we have
jjON AN O0N jj2F = tr(ON A0N O0N ON AN O0N ) = tr(A0N AN ) = jjAN jj2F :
(4.5)
The following lemma describes the optimal preconditioner of T 0N T N in two dierent
ways.
Lemma 4.1. Let AN 2 IRN;N and let AO be dened by (4.2) with respect to some
orthogonal matrix ON . Then the optimal preconditioner of AN is given by
M N (AN ) = O0N (ON AN O0N ) ON :
(4.6)
If fB0; : : :; B N ?1g denotes a basis of AO , then an alternative description of M N (AN )
reads as
N ?1
(4.7)
M N (AN ) := X k B k ;
N
N
k=0
:= (0; : : :; N ?1)0
N ?1 ;
:= (h k ; j i)j;k
=0
where the coecient vector G = ; G
B B
9
is determined by
:= (hAN ; B j i)jN=0?1 :
.
Proof: 1. By (4.5), it follows for M N := O0N (diag d) ON 2 AO that
jjM N ? AN jjF = jjdiag d ? ON AN O0N jjF
N
which implies (4.6) by the denition of the optimal preconditioner.
2. The computation of the optimal preconditioner of AN in AO is equivalent with the
computation of the element of best approximation of AN in the linear subspace AO
of the Hilbert space IRN;N equipped with the inner product (4.4). This can be done by
the Galerkin approach (4.7).
N
N
Now it is easy to verify that an optimal preconditioner of T 0N T N satises property (P1).
Corollary 4.2. Let AN 2 RN;N be a symmetric positive denite matrix and let AO
be dened by (4.2) with respect to some orthogonal matrix ON . Then the optimal
preconditioner M N = M N (AN ) is also symmetric and positive denite.
N
For the proof of Corollary 4.2 and in connection with (4.6) see for example [32, 20].
5 Clusters of eigenvalues
Let C2 denote the Banach space of 2-periodic complex-valued functions equipped
with the usual norm jj jj1. In this section, we are only interested in functions f =
fR + ifI 2 C2 with real Fourier coecients
Z
1
tk := 2 f (x)e?ikx dx (k 2 ZZ) ;
?
where we suppose that the real and the imaginary part of f
1
X
fR = t0 + (tk + t?k ) cos kx ;
k=1
1
X
fI = (tk ? t?k ) sin kx
k=1
do not vanish, respectively. Moreover, we assume that ftk gk2ZZ 2 l1, such that f belongs
to the Wiener class. Consider the N -th Toeplitz matrix corresponding to the generating
function f
T N := toeplitz((t0; t?1; : : : ; t?(N ?1)); (t0; t1; : : : ; tN ?1)) :
It is well-known that the singular values of TN are distributed as jf j [27]. Note that
the above result was extended to functions f 2 L22 C2 in [34].
The following denition is due to E.E. Tyrtyshnikov [34]. Let fkN gNk=1 be a sequence
of real numbers and let N (") denote the number of those among kN (k = 1; : : : ; N )
which are outside the "{ball centered at p. If N (") < K ("), where K (") is independent
10
of N , then p is called a proper cluster. In this sense, we say that the values kN are
clustered at p.
In the following, we restrict our attention to preconditioners of AC . By Theorem
3.1, the approach for the preconditioners with respect to the DST{I, the DST{II, the
DST{IV and the DCT{IV follows the same lines. Let C = C N := Z N;2C IIN and
S = SN := Z N;1SIIN . Regarding Theorem 3.1 ii), we associate with the sequence
fT N gN 2IN of Toeplitz matrices a sequence fH N gN 2IN of Hankel matrices
II
N
H N := hankel((t?1; : : : ; t?(N ?1); 0); (t1; : : : ; tN ?1; 0)) :
Let M N denote the optimal preconditioner of T 0N T N with respect to the DCT{II, i.e.
M N := (C IIN )0 (C IIN T 0N T N (C IIN )0) C IIN :
(5.1)
In this section, we prove that under certain assumptions on T N , the eigenvalues of the
preconditioned matrix M ?N1T 0N T N are clustered at 1. We follow the lines of R.H. Chan.
First, we show that for all " > 0 and N suciently large, the matrix T 0N T N ? M N splits
into a matrix of low rank independent of N and a matrix with spectral norm smaller
than ". Then we apply Cauchy's interlace theorem to verify that the eigenvalues of
M ?N1(T 0N T N ? M N ) = M N?1T 0N T N ? I N
are clustered at 0. Again, we drop the index N , if the dimension of the matrices follows
from the context.
In preparation of Theorem 5.3, we provide the following two lemmata.
Lemma 5.1. Let a = fakg1k=0 2 l1 and b = fbk g1k=0 2 l1. Then, for all " > 0, there
exists m = m(") such that for all N 2m the Hankel matrix
H := hankel((a0; a1; : : :; aN ?1 + bN ?1); (b0; b1; : : : ; aN ?1 + bN ?1))
splits as H = V H + W H , where
W H := hankel((a0; : : :; am?1; oN ?m); (b0; : : :; bm?1; oN ?m))
is a matrix of rank 2m and where V H := H ? W H satises jjV H jj2 < ".
Proof: Since a; b 2 l1, there exists for all " > 0 an integer m = m(") such that
1
X
k=m
jak j < "=2 ;
1
X
k=m
jbk j < "=2 :
Now the assertion follows from jjV H jj2 jjV H jj1 < "=2 + "=2.
11
Lemma 5.2. Let t = ftk gk2ZZ 2 l1 with jjtjj1 = be given. Further let T = T N
and H = H N be the corresponding N -th Toeplitz matrix and N -th Hankel matrix,
respectively. Then, for all " > 0, there exists m = m(") such that for all N 4m
HT + T 0H + H 2 = V + W ;
(5.2)
W := (wj;k )j;kN ?=01 with wj;k = 0 for 2m j + k 2N ? 2 ? 2m:
(5.3)
where jjV jj2 < " and
Proof: By construction, we have jjT jj2 , jjH jj2 . Since t 2 l1, there exists
m = m(") such that
1
X
jkj=m+1
Then
jtkj < 6" :
T = T " + T B ; H = H" + HB
with
T"
TB
H"
HB
:=
:=
:=
:=
(5.4)
(5.5)
toeplitz((0m+1; t?(m+1); : : : ; t?(N ?1)); (om+1; tm+1; : : :; tN ?1)) ;
toeplitz((t0; : : : ; t?m; oN ?m?1); (t0; : : : ; tm; oN ?m?1)) ;
hankel((om; t?(m+1); : : : ; t?(N ?2); 0); (om; tm+1; : : :; tN ?2; 0)) ;
hankel((t?1; : : :; t?m; oN ?m ); (t1; : : :; tm; oN ?m)) ;
where we obtain by (5.4) that jjT " jj2 6" , jjH "jj2 6" . Substituting (5.5) in (5.2),
we obtain the desired decomposition
HT + T 0H + H 2 = (H " + H B )(T " + T B ) + (T 0" + T 0B )(H " + H B ) + (H " + H B )2
= (HT " + T 0" H + H "T B + T 0B H " + H "H + H B H " )
+(T 0B H B + H B T B + H 2B ) :
Theorem 5.3. Let t = ftk gk2ZZ 2 l1 with jjtjj1 = be given and let T = T N be the
corresponding N {th Toeplitz matrix. Then, for all " > 0, there exists m = m(") such
that for all N 2m
T 0T = C 0DC + V + W ;
where D denotes some diagonal matrix,
W := (wj;k )j;kN ?=01 with wj;k = 0 for m j + k 2N ? 2 ? m;
and where jjV jj2 < ".
12
Proof: Let H = H N denote the N {Hankel matrix associated with t. Then it follows
by Theorem 3.1 ii) that
T 0T = (T 0 + H ? H )(T + H ? H )
= (C 0DaC + S 0D~ bC ? H )(C 0DaC + C 0D~ bS ? H ) ;
where
a = aN +1 := (2t0; t?1 + t1; : : : ; t?(N ?1) + tN ?1; 0)0 2 IRN +1 ;
b = bN ?1 := (t?1 ? t1; : : :; t?(N ?1) ? tN ?1)0 2 IRN ?1
and
Da = diag(d0; : : :; dN ?1; dN )0 ; (d0; : : : ; dN )0 := C~ IN +1a ;
D~ b = diag(0; d~1; : : : ; d~N ?1; 0)0 ; (d~0; : : : ; dN ?1)0 := S~IN ?1b :
By C IIN (C IIN )0 = I N , we further obtain that
T 0T = C 0D2aC + S0D~ 2b S + C 0DaD~ bS + S0D~ b DaC ? H (T + H ) ? (T 0 + H )H + H 2
and by Theorem 3.1 ii) that
T 0T = C 0D2aC + C 0D~ 2b C ? H D~ 2 + H~ D~ D ? (HT + T 0H + H 2) (5.6)
with
H D~ 2 := shankel(u1; : : : ; uN ) ;
(u0; : : :; uN )0 := N2 C~ IN +1(0; d~21; : : : ; d~2N ?1; 0)0 ;
H~ D~ D := ahankel(v1; : : : ; vN ?1; 0) ;
(v1; : : :; vN ?1)0 := N2 S~IN ?1(d1d~1; : : :; dN ?1d~N ?1)0 :
Set
H^ := H D~ 2 ? H~ D~ D = hankel((w1; : : :; wN ); (w?1; : : :; w?N ))
(5.7)
with wN = w?N = uN , wk = uk ? vk , w?k = uk + vk (k = 1; : : :; N ? 1). Then we have
for all N 2 IN that
NX
?1
wN = N2 (?1)k d~2k ;
k=1
2
(5.8)
jwN j N jjS~IN ?1bjj22 jjbjj22 jjbjj21 2 :
Moreover, we get by Theorem 3.1 i) for the rst row w := (w1; : : : ; wN ?1)0 of H^ that
w = N2 R0N C~ IN +1 diag(0; d~1; : : : ; d~N ?1; 0)0 RN S~IN ?1b
? N2 S~IN ?1R0N diag(d0; : : :; dN )0 RN S~IN ?1b
= toeplitz (?t0; ?t1; : : :; ?tN ?2); (?t0; ?t?1; : : :; ?t?(N ?2)) b
+hankel (t?2; : : :; t?(N ?2); bN ?1; 0); (t2; : : :; tN ?2; ?bN ?1; 0) b :
b
b
b
b
a
b
a
13
b
a
Thus, we obtain for all N 2 IN that
jjwjj1 2 2 :
Similarly, we conclude that
jj(w?1; : : :; w?(N ?1))0jj1 2 :
Together with (5.8), we see that H^ satises the assertion of Lemma 5.1. Hence, for
xed " > 0, there exists m^ = m^ (") > 0 such that for all N 2m^
H^ = V^ + W^
(5.9)
^ = hankel((w1; : : :; wm; oN ?m); (w?1; : : : ; w?m; oN ?m)) :
with jjV^ jj2 "=2, W
Furthermore, by Lemma 5.2, there exists m~ = m~ (") > 0 such that for N suciently
large
~
(HT + T 0H + H 2) = V~ + W
(5.10)
~ of the form (5.3). Applying (5.9) and
with jjV~ jj2 "=2 and with a low rank matrix W
(5.10) in (5.6), we obtain the assertion
T 0T = C 0(D2a + D~ 2b )C + (V~ ? V^ ) + W~ ? W^
with m := maxfm;
^ 2m~ g.
Lemma 5.4. For m > 0 and N > 2m, let V 2 IRN;N with jjV jj2 < "=2 and
W := (wj;k )j;kN ?=01 with wj;k = 0 for m j + k 2N ? 2 ? m;
N ?1
be given. Set ! := P jwj;k j. Then, for N > 4!=",
j;k=0
jj(CIIN (V + W ) (C IIN )0)jj2 < " :
Note that for xed m, the value ! does not depend on N .
Proof: On the one hand, we obtain that
jj(CIIN V (C IIN )0)jj2 jjCIIN V (C IIN )0jj2 = jjV jj2 < "=2 ;
and on the other hand that
NX
?1
2 ("N )2
n(2j + 1) cos n(2k + 1) j
jj(CIIN W (C IIN )0)jj2 n=0max
j
w
cos
j;k
n
;:::;N ?1 N
2N
2N
j;k=0
2!=N < "=2 (N > 4!=") :
14
Now summation implies the assertion.
Theorem 5.5. Let t = ftk gk2ZZ 2 l1 with jjtjj1 = be given and let T N be the cor-
responding N {th Toeplitz matrix. Moreover, assume that the singular values of T N
are larger than > 0 for all N 2 IN. Let M N = M N (T 0N T N ) denote the optimal
preconditioner of T 0N T N in AC . Then the eigenvalues of M ?N1T 0N T N are clustered at
1.
II
N
Proof: By Corollary 4.2, we see that jjM ?N1 jj2 < 1= for all N 2 IN. Let " > 0 be
xed. Then we obtain by (5.1), Theorem 5.3 and Lemma 5.4 that for N suciently
large, there exists M 2 IN independent of N such that
T 0N T N ? M N = V N + W N ? (C IIN )0 (C IIN (V N + W N ) (C IIN )0) C IIN = U N + W N ;
where W N is of low rank M and where jjU N jj2 < " . Now
M ?N1=2 T 0N T N M ?N1=2 ? I N = M ?N1=2 U N M ?N1=2 + M ?N1=2 W~ N M ?N1=2
~N;
= V~ N + W
~ N is at most M . Since V~ N and W~ N are
where jjV~ N jj2 " and where the rank of W
symmetric matrices, we can apply Cauchy's interlace theorem [37], which implies that
~ N have absolute value greater
for N suciently large, at most M eigenvalues
of V~ N + W
?
1=2 0
than ". Now the assertion follows since M N T N T N M ?N1=2 and M ?N1 T 0N T N possess
the same eigenvalues.
Remark.
Under the above assumptions on T N it was proved that the eigenvalues of
?1
?1
(M N T N )(M N T N ), where M N = C N (T N ) denotes the optimal circulant preconditioner of T N , are clustered at 1 [13, 34]. In general, the eigenvalues of
(M ?N1T N )0(M ?N1T N ) are not clustered at 1, if M N is the optimal trigonometric preN ?1 with t = 1 and t = ?t (k = 1; : : : ; N ? 1),
conditioner of T N . If T N = (tj?k )j;k
0
k
?k
=0
then the optimal trigonometric preconditioners of T N are M N = I N , i.e. we have no
preconditioning. If, for example, t?1 = ?t1 = 2 and tk = 0 (jkj > 1), then the matrices
T 20 N +1 (N 2 IN) satisfy the assumption of Theorem 5.5. However, the eigenvalues of
T 2N +1T 2N +1 are given by 9 ? 8 cos(j=(N + 1)) (j = 0; : : : ; N ).
6 Construction of optimal preconditioners of T 0T
In this section, we explain how optimal trigonometric preconditoners of T 0N T N can be
constructed with O(N log N ) arithmetical operations. In contrast to the construction
of optimal trigonometric preconditioners of T N , we are confronted with the fact that
T 0N T N is not a Toeplitz matrix. Again, we consider only C IIN {preconditioners. The
15
approach for the DST{I, DST{II, DST{IV and DCT{IV follows the same lines.
For the construction of the optimal preconditioner M N = M N (T 0N T N ) we use the
representation (4.7) of M N with the basis fBIIk : k = 0; : : : ; N ? 1g of AC [6], [17]:
II
N
BIIk := (C IIN )0 diag(Uk (cl))lN=0?1 C IIN ;
l and where U denotes the k{th Chebyshev polynomial of second kind
where cl := cos N
k
Uk (x) := sin((k + 1) arccos x)= sin(arccos x) (x 2 (?1; 1)) :
Moreover, we apply that fBIk : k = 0; : : : ; N ? 1g with
BIk := (C IIN )0 diag(Tk (cl))lN=0?1 C IIN
= stoeplitz e0k + shankel e0k?1 ; (k = 1; : : : ; N ? 1)
e?1 := oN , and with the k-th Chebyshev polynomial of rst kind
Tk (x) := cos(k arccos x) (x 2 [?1; 1]) ;
is another basis of AC . Both bases are related by
(6.1)
II
N
B I0 = B II0 = I ; B II1 = 2B I1 ;
B IIj = B IIj?2 + 2B Ij (j = 2; : : : ; N ? 1) ;
(6.2)
where the last equation follows by Uj = Uj?2 + 2Tj . Now we have by (4.7) that
N ?1
M N = X k B IIk
k=0
with
= G?1 II ; G := hB IIj ; BIIk i j;kN ?=01 ; II := hT 0N T N ; BIIj i jN=0?1 :
Clearly, we are not interested in M N itself, but in the diagonal matrix diag d with
M N = (C IIN )0 (diag d) C IIN :
(6.3)
If is known, then we obtain diag d by
diag d =
NX
?1
k=0
k C IIN BIIk (C IIN )0 =
NX
?1
k=0
k diag (Uk (cl))lN=0?1 ;
i.e. by denition of Uk by
NX
?1
(k + 1) k ; dk = ^kk (k = 1; : : : ; N ? 1) ;
d0 =
sin N
k=0
(^k )kN=1?1 = S~IN ?1 (k?1)kN=1?1 :
16
(6.4)
Thus the construction of d from given requires O(N log N ) arithmetical operations.
It remains to nd an ecient construction of the coecient vector . Therefore we use
the following lemmata.
N ?1
Lemma 6.1. The inverse matrix of G := hB IIj ; BIIk i j;k
, is given by
=0
0 3 0 ?1
1
BB 0 2 0 ?1
CC
BB ?1 0 2 0 ?1
CC
B
CC
... ... ... ...
CC ;
G?1 = 21N BBB
BB
?1 0 2 0 ?1 CCC
B@
?1 0 3 ?2 A
?1 ?2 3NN?2
such that the vector multiplication with G?1 can be computed with O(N ) arithmetical
operations.
Proof: We show that G?1 G = I . By denition, we have for the (k; l){th entry of G
that
gk;l = hB IIk ; B IIl i = tr (BIIl )0 BIIk
NX
?1
Ul(cj ) Uk (cj ) :
=
j =0
Then we get for l = 2; : : :; N ? 3 that
NX
?1
Uk (cj ) (2Ul (cj ) ? Ul?2(cj ) ? Ul+2(cj ))
2gk;l ? gk;l?2 ? gk;l+2 =
j =0
and further since
2Ul (x) ? Ul?2(x) ? Ul+2(x) = 4(1 ? x2) Ul(x)
and by denition of Ul that
2gk;l ? gk;l?2 ? gk;l+2 = 4
NX
?1
j =1
sin (k +N1)j sin (l +N1)j = 2N k;l
(k = 0; : : : ; N ? 1; l = 2; : : : ; N ? 3). Straightforward computation for l = 0; 1; N ?
2; N ? 1 completes the proof.
Set
I := hT 0N T N ; BIj i jN=0?1 :
17
Then we obtain by the recurrence relation (6.2) that
0II = 0I ; 1II = 21I ; kII = kII?2 + kI (k = 2; : : : ; N ? 1) :
(6.5)
Thus we can compute II from I with O(N ) additions. The following construction of
I is based on an idea of E.E. Tyrtyshnikov [33]. We split T = T N = (tj?k )j;kN ?=01 into
a lower and an upper triangular Toeplitz matrix
T = TL + TR
with diagonal entries t0=2. Then we obtain that
Ik = hT 0LT L; B Ik i + hT 0RT R; BIk i + hT 0LT R + T 0RT L; BIk i (k = 0; : : : ; N ? 1) : (6.6)
We consider the summands on the right-hand side. The matrix T 0LT R + T 0RT L is a
symmetric Toeplitz matrix. The matrices T 0LT L and T 0RT R have lost their Toeplitz
structure. For A 2 IRN;N , we introduce the vectors s(A) = (sk (A))kN=0?1, h(A) =
(hk (A))kN=0?1 and h~ (A) = (~hk (A))kN=0?1 by
sk (A) := hA; stoeplitz(("Nk )?2ek )i (k = 0; : : : ; N ? 1) ;
hk (A) := hA; hankel(ek ; 0)i (k = 0; : : : ; N ? 2) ;
h~ k (A) := hA; hankel(0; ek )i (k = 0; : : : ; N ? 2) :
Set h?1 := 0 and h~ ?1 := 0. Then it follows by (6.1) that
hA; B Ik i = ("Nk )2sk (A) + hk?1(A) + ~hk?1(A) (k = 0; : : : ; N ? 1) : (6.7)
Lemma 6.2. Let A := T 0LT R + T 0RT L and let r := T 0R(t0; t1; : : :; tN ?1)0. Then
sk (A) = 2(N ? k)rk (k = 0; : : : ; N ? 1) ;
h0(A) = r0 ; h1(A) = 2 r1 ;
hk (A) = 2 rk + hk?2 (A) (k = 2; : : : ; N ? 2) ;
h~ k (A) = hk (A) (k = 0; : : :; N ? 2) :
Proof: Since A = stoeplitz r, the assertion follows by denition of s; h and h~ .
Lemma 6.3. Let A := T 0RT R and let r := ("Nk )2t?k kN=0?1. Then
s(A) = 2 hankel(Nr0; (N ? 1)r1; : : : ; rN ?1); (0; : : : ; 0; rN ?1)) r ;
h0(A)
hk (A)
h~ 0(A)
h~ k (A)
=
=
=
=
x0 ; h1(A) = x1 ;
hk?2 (A) + xk (k = 2; : : : ; N ? 2) ;
y0=2 ; h~ 1(A) = y1 ;
h~ k?2 (A) + yk (k = 2; : : : ; N ? 2) ;
18
(6.8)
(6.9)
(6.10)
where
x := T 0Rr ;
y := hankel(2(r0; r1; : : : ; rN ?1); (?r2; ?r3; : : :; ?rN ?1; 0; 2rN ?1)) r :
Proof: Since T R is an upper triangular Toeplitz matrix, we have that
aj;k = aj?1;k?1 + rj rk (j; k = 1; : : :; N ? 1) :
(6.11)
Consequently, we obtain that
NX
?1?k
N?
Xk?1
(N ? j ? k)rj+k rj
aj+k;j =
sk = 2
j =0
j =0
which yields (6.8). The recursions (6.9) and (6.10) follow by straightforward calculation
from (6.11).
Lemma 6.4. Let B := T 0RT R and let J := shankel eN ?1 denote the N -th counteri-
dentity. Then
sk (B) = sk (J B J ) (k = 0; : : :; N ? 1) ;
hk (B) = h~ k (J B J ) (k = 0; : : : ; N ? 2) ;
h~ k (B) = hk (J B J ) (k = 0; : : : ; N ? 2) ;
such that s(B); h(B ) and h~ (B ) can be computed by (6.8) { (6.10).
Proof: The relations for s(B); h(B) and h~ (B ) follow by denition of J . By
J T 0L T L J = (J T L J )0 (J T L J )
and since
J T L J = toeplitz((t0; : : :; tN ?1); (t0; 0; : : : ; 0))
is an upper triangular Toeplitz matrix,
s(B); h(B ) and h(~B) by (6.8)
N ?calculate
Nwe2 can
1
{ (6.10) with A = J B J and r = ("k ) tk k=0 .
N ?1 . Then the optimal preconditioner M 2 A
Theorem 6.5. Let T N := (tj?k )j;k
N
C
=0
of
T 0N T N
can be constructed with O(N log N ) arithmetical operations.
19
II
N
Proof: We compute I by (6.6), (6.7) and by the Lemmata 6.2 { 6.4. Taking into
account that the multiplication of a vector with a Toeplitz matrix or a Hankel matrix
requires O(N log N ) operations, the whole construction of I takes O(N log N ) arithmetical operations. From I we compute II by (6.5) with O(N ) additions. Using
Lemma 6.1, we get := G?1II at the cost of O(N ) arithmetical operations. Finally,
the DST{I in (6.4) to obtain d from needs O(N log N ) arithmetical operations and
we are done with an arithmetical complexity of O(N log N ).
Remark. We can use similar ideas forIV the Iconstruction
of the optimal trigonometric
II
preconditioners with respect to the C N , SN ?1, S N and S IV
N . For the corresponding
bases of AO
N
we obtain
?1
BIk := O0dim diag(Tk (cl))ldim
=0 Odim
= stoeplitz e0k + hankel(u0k ; v0k ) ;
?1
B IIk := O0dim diag(Uk (cl))ldim
=0 O dim
0 3
B
0
B
B
B
?1
B
1
B
?
1
G = K BB
B
B
B
@
0
2
0
...
?1
0
2
...
?1
where
Odim
C IIN
C IVN
S IN ?1
SIIN
SIVN
cl
dim
cos Nl
N
cos (2l2+1)
N
N
N ?1
cos (l+1)
N
cos (l+1)
N
N
cos (2l2+1)
N
N
uk
ek?1
ek?1
?ek?2
?ek?1
?ek?1
?1
0
...
0
?1
1
CC
CC
?1
CC
...
CC ;
2 0 ?1 C
CC
0 g 1 g2 A
?1 g2 g3
vk
K
ek?1 2N
?ek?1 2N
?ek?2 2N + 2
?ek?1 2N
ek?1 2N
g1 g2
3 ?2
1 0
2 0
3 2
1 0
g3
3N ?2
N
1
3
.
3N ?2
N
1
Note that the construction of the optimal preconditioner with respect to the C IV
N or
the S IV
is
especially
simple.
N
7 Numerical Results
Finally, we present examples of nonsymmetric Toeplitz systems (1.1) for which the
preconditioning of the normal equation by an optimal trigonometric preconditioner
20
M N = M N (T 0N T N ) of T 0N T N signicantly accerlerates the convergence of the CG{
method. The algorithms were realized for the optimal preconditioners with respect
to the DCT{II, the DST{II, the DCT{IV and the DST{IV, respectively. Clearly, we
can also use the DST{I preconditioner. Note that for symmetric Toeplitz matrices, the
DST{I preconditioned CG{method for the solution of (1.1) shows a similar convergence
behaviour as the DST{II preconditioned CG{method.
The fast computation of the preconditioners in the initial step and the computation
of the preconditioned CG{method (PCG{method) were implemented in Matlab and
tested on a Sun SPARCstation 20. The fast trigonometric transforms appearing both
in the initialization and in the PCG{steps were taken from the C{implementation based
on [2, 31] by using the cmex{programm.
As transform length we choose N = 2n . The right{hand side bN of (1.1) is the vector consisting of N ones. The PCG{method starts with the zero vector and stops if
kr(j)k2=kr(0)k2 < 10?7 , where r(j) denotes the residual vector after
j iterations. Our
N
?
1
test matrices are the following four Toeplitz matrices T N = (tj?k )j;k=0 :
i) (see [26])
8
>
< 1= log(2 ? n) n ?1 ;
tn = > 1=(log(2) + 1) n = 0 ;
: 1=(1 + n)
n1
ii) (see [26])
8
>
2
n=0;
<
tn = > ?0:7 tn+1 n ?1 ;
: 0:9 tn?1 n 1 :
iii) Here we use the Toeplitz matrices T N arising from the generating function
f (x) = x2eix :
iv)
8 ?1:5 n = ?1 ;
>
>
<
tn = > 0:25 nn == 01 ;;
>
: 0
else:
As expected, also for large transform lenghts N , the initialisation and each PCG{
step can be computed in a very fast way which reects the arithmetic complexity of
O(N log N ) for these computations. We compare three dierent methods. The second
columns of the following tables show the number of iterations of the PCG{method
applied to the normal equation without preconditioning. The columns 3 and 4 contain
the numbers of iterations required by the CG{method applied to
(M ?N1T N )0(M ?N1T N )xN = (M ?N1T N )0M ?N1bN ;
(7.1)
where M N denotes the optimal trigonometric preconditioner of T N with respect to the
DCT{II and the DST{II, respectively.
21
The columns 5 { 8 contain the numbers of iterations required by the PCG{method
applied to
M ?N1T 0N T N xN = M ?N1T 0N bN ;
(7.2)
0
where M N denotes the optimal trigonometric preconditioner of T N T N with respect to
the DCT{II, DST{II, DCT{IV and the DST{VI, respectively.
We compare our results with the GMRES{method applied to the original problem
M ?N1T N xN = M ?N1bN :
(7.3)
Again, M N denotes the optimal trigonometric or circulant preconditioner of T N . Here
we avoid the transition of (1.1) to the normal equation. The ninth columns contain
the number of iteration steps of the GMRES{method without preconditioning. Since
the optimal trigonometric preconditioners are symmetric, we expect no good convergence behaviour of the corresponding PCG{method. The tenth columns conrm these
expectations. Finally, the last columns contain the number of iterations of the GMRES{
method with the optimal circulant preconditioner (see [35]). The number of iteration
steps in the last columns of the Tables 1, 2 and 4 are compatible with the number of iteration steps required by our new preconditioning method. Our preconditioning uses the
normal equation of (1.1) but avoids complex arithmetic. Note that very ill{conditioned
Toeplitz matrices will be handled in a forthcoming paper.
n
7
8
9
10
11
12
13
IN
24
32
43
57
86
121
176
CG applied to (7.1)
C
II
N
11
12
14
18
20
25
30
S
II
N
10
11
13
16
19
22
28
PCG applied to (7.2)
GMRES applied to (7.3)
8
8
8
9
9
9
9
22
28
34
43
54
> 70
> 70
C
II
N
S
II
N
15
17
19
20
20
22
22
C
IV
N
14
15
17
19
20
22
22
S
IV
N
11
11
11
11
12
12
12
IN
C IIN
15
18
21
25
30
36
40
FN
7
8
8
8
8
8
8
Table 1: Number of iterations for example i)
Although not all matrices in our examples full the assumptions of Theorem 5.5, the
preconditioning with an optimal trigonometric preconditioner of T 0N T N accerelates the
convergence of the CG{method signicantly.
Further extensive numerical tests (see [28]) with matrices from [8, 9] show that at least
one of our trigonometric preconditioners works better than the circulant preconditioners.
Except for the second example, where the condition numbers of T N are bounded for
N ! 1, the number of iterations diers, if we apply the preconditioners with respect
to the DCT{II and the DST{II. Heuristically, this can be explained by the dierent
structures of AC and AS and how ,,good" our example matrices t into this structure.
II
N
II
N
22
n
7
8
9
10
11
12
13
IN
CG applied to (7.1)
34
43
53
59
50
58
56
C IIN
SIIN
44
47
50
50
50
49
48
44
50
52
53
53
53
54
PCG applied to (7.2)
GMRES applied to (7.3)
9
8
7
7
6
6
6
>
>
>
>
>
>
>
C IIN SIIN C IVN S IVN
12
11
10
9
9
8
8
9
8
8
7
7
7
7
IN
14
13
12
11
10
10
9
70
70
70
70
70
70
70
C IIN
>
>
>
>
>
>
>
70
70
70
70
70
70
70
FN
8
8
8
8
8
8
8
Table 2: Number of iterations for example ii)
n IN
5
84
6 311
7 1226
8 5220
9 > 1000
CG applied to (7.1)
C IIN
72
124
264
626
1741
SIIN
68
176
412
980
3341
PCG applied to (7.2)
C IIN SIIN C IVN SIVN
29
52
116
256
664
21 47
26 84
33 173
40 405
74 1031
GMRES applied to (7.3)
IN
S IIN
24
32
32
39
64
53
63 > 70 > 70
136 > 70 > 70
310 > 70 > 70
FN
14
16
18
21
27
Table 3: Number of iterations for example iii)
A general criterion for the choice of the optimal trigonometric preconditioner would
be interesting. In this direction, it is remarkable, that the optimal preconditioner
M N (T N ) 2 AC of the auto-covariance matrix T N := (jj?kj)j;kN ?=01 is \asymptotically
equivalent" to T N if N ! 1, ! 1, while the optimal preconditioner M N (T N ) 2 AS
of T N is \asymptotically equivalent" to T N if N ! 1, ! 0 [29].
Now let T := (tj?k )Mj=0?;k1;N=0?1 denote a real (M N ) { Toeplitz matrix of rank N
(M N ). We are interested in the solution of least squares problems
II
N
II
N
min kbN ? TxN k2
(7.4)
by the PCG{method applied to the corresponding normal equation. Circulant preconditioning of the normal equation of (7.4) was considered in [8, 9].
Following the lines of Section 6, the computation of the optimal trigonometric preconditioner is straightforward. We consider the following example
v): (see [9])
( ?0:1(k?1)2
k = ?N + 1; : : :; 0 ;
tk := ee?0:1k2
k = 1; : : :; M ? 1 :
23
n
6
7
8
9
IN
88
201
435
929
CG applied to (7.1)
C IIN
S IIN
21
31
45
66
37
67
125
294
PCG applied to (7.2)
GMRES applied to (7.3)
21
27
36
47
64
36
> 70 53
> 70 > 70
> 70 > 70
C IIN S IIN C IVN SIVN
9
8
8
9
25
31
39
72
16
19
24
32
IN
SIIN
FN
11
11
12
12
Table 4: Number of iterations for example iv)
The number of iteration steps for the dierent optimal trigonometric preconditioners
are contained in Table 5.
n
4
5
6
7
8
9
M
32
64
128
256
512
1024
PCG applied to (7.2)
I N C IIN SIIN C IVN SIVN
22
46
75
125
167
193
9
10
9
8
8
8
20
29
28
23
18
16
19
28
25
19
16
15
18
24
20
17
15
13
Table 5: Number of iterations for example v)
Acknowledgement. The authors wish to thank M. Tasche for his useful hints concern-
ing the relations between trigonometric transforms and Toeplitz{plus{Hankel matrices.
Many thanks to the referees for their valuable comments, in particular for pointing out
various references to us.
References
[1] O. Axelsson. Iterative Solution Methods. Cambridge University Press, Cambridge,
1996.
[2] G. Baszenski and M. Tasche. Fast polynomial multiplication and convolution related to the discrete cosine transform. Linear Algebra Appl., 252:1 { 25, 1997.
[3] D. Bini and F. D. Benedetto. A new preconditioner for the parallel solution of positive denite Toeplitz systems. In Proc. Second ACM Symp. on Parallel Algorithms
and Architectures, pages 220 { 223, Crete, 1990.
24
[4] D. Bini and P. Favati. On a matrix algebra related to discrete Hartley transform.
SIAM J. Matrix Anal. Appl., 14:500 { 507, 1993.
[5] E. Boman and I. Koltracht. Fast transform based preconditioners for Toeplitz
equations. SIAM J. Matrix Anal. Appl., 16:628 { 645, 1995.
[6] E. Bozzo and C. D. Fiore. On the use of certain matrix algebras associated with
discrete trigonometric transforms in matrix displacement decomposition. SIAM J.
Matrix Anal. Appl., 16:312 { 326, 1995.
[7] R. H. Chan, T. F. Chan, and C. K. Wong. Cosine transform based preconditioners
for total variation minimization problems in image processing. Technical report
95{8, Chinese University of Hong Kong, 1995.
[8] R. H. Chan, J. Nagy, and R. Plemmons. Circulant preconditioned Toeplitz least
square problems. SIAM J. Matrix Anal. Appl., 15:80 { 97, 1994.
[9] R. H. Chan, J. Nagy, and R. Plemmons. Displacement preconditioners for Toeplitz
least squares iterations. Elec. Trans. Numer. Anal., 2:44 { 56, 1994.
[10] R. H. Chan and M. K. Ng. Conjugate gradient methods of Toeplitz systems. SIAM
Review, 38:427 { 482, 1996.
[11] R. H. Chan, M. K. Ng, and C. K. Wong. Sine transform based preconditioners for
symmetric Toeplitz systems. Linear Algebra Appl., 232:237 { 259, 1996.
[12] R. H. Chan and G. Strang. Toeplitz systems by conjugate gradients with circulant
preconditioner. SIAM J. Sci. Statist. Comput., 10:104 { 119, 1989.
[13] R. H. Chan and M.-C. Yeung. Circulant preconditioners constructed from kernels.
SIAM J. Numer. Anal., 29:1093 { 1103, 1992.
[14] T. F. Chan. An optimal circulant preconditioner for Toeplitz systems. SIAM J.
Sci. Statist. Comput., 9:766 { 771, 1988.
[15] R. Freund and T. Huckle. Iterative solution of linear systems with low displacement
rank by preconditionined conjugate gradient{type algorithms. In XII Householder
Symp., Lake Arrowhead.
[16] I. Gohberg, T. Kailath, and V. Olshevsky. Fast gaussian elimination with partial
pivoting for matrices with displacement structure. Math. Comp., 64:1557 { 1576,
1995.
[17] G. Heinig and A. Bojanczyk. Transformation techniques for Toeplitz and Toeplitz{
plus{Hankel matrices. Linear Algebra Appl., 254:193 { 226, 1997.
[18] G. Heinig and K. Rost. Representations of Toeplitz{plus{Hankel matrices using
trigonometric transforms with application to fast matrix{vector multiplication.
Linear Algebra Appl. in print.
25
[19] S. Holmgren and K. Otto. A framework for polynomial preconditioners based on
fast transforms I: Theory. Preprint Uppsala University, 1997.
[20] T. Huckle. Circulant/skewcirculant matrices as preconditioners for hermitian
Toeplitz systems. In IMACS Conference on Iterative methods in Linear Algebra,
Brussel, 1991.
[21] T. Huckle. Fast transforms for tridiagonal linear equations. Bit, 34:99 { 112, 1994.
[22] T. Huckle. A look{ahead algorithm for solving nonsymmetric linear Toeplitz equations. In Proccedings of the 5th SIAM conference, Snowbird:455 { 459, 1994.
[23] T. Huckle. Cauchy matrices and iterative methods for Toeplitz matrices. In SPIE{
95, 2563, San Diego:281 { 292, 1995.
[24] T. Kailath and V. Olshevsky. Displacement structure approach to discrete{
trigonometric{transform based preconditioners of G. Strang type and of T. Chan
type. Calcolo, 33:191 { 208, 1996.
[25] T. Kailath and V. Olshevsky. Diagonal pivoting for partially reconstructible
Cauchy{like Matries, with applications to Toeplitz{like linear equations and to
boundary rational matrix interpolation problems. Linear Algebra Appl, 254:251 {
302, 1997.
[26] T. Ku and C. Kuo. Spectral properties of preconditioned rational Toeplitz matrices.
SIAM J. Matrix Anal. Appl., 14:146 { 165, 1993.
[27] S. V. Parter. On the distribution of singular values of toeplitz matrices. Linear
Algebra Appl., 80:115 { 130, 1986.
[28] D. Potts. Schnelle Polynomtransformation und Vorkonditionierer fur Toeplitz{
Matrizen. PhD thesis, Universitat Rostock, 1998. submitted.
[29] K. Rao and P. Yip. Discrete Cosine Transforms. Academic Press, Boston, 1990.
[30] Y. Saad. Iterative Methods for Sparse Linear Systems. PWS Publ., Boston, 1996.
[31] G. Steidl and M. Tasche. A polynomial approach to fast algorithms for discrete
Fourier{cosine and Fourier{sine transforms. Math. Comp., 56:281 { 296, 1991.
[32] V. Strela and E. E. Tyrtyshnikov. Some generalizations of circulant preconditioner.
Matrix Methods and Algorithms, IVM RAN Moscow, 1990. 66 { 73.
[33] E. E. Tyrtyshnikov. Optimal and superoptimal circulant preconditioners. SIAM
J. Matrix Anal. Appl., 13:459 { 473, 1992.
[34] E. E. Tyrtyshnikov. A unifying approach to some old and new theorems on distribution and clustering. Linear Algebra Appl, 232:1 { 43, 1996.
[35] E. E. Tyrtyshnikov, A. Yeremin, and N. Zamarashkin. Clusters { preconditioners
{ convergence. Linear Algebra Appl, 263:25 { 48, 1997.
26
[36] Z. Wang. Fast algorithms for the discrete W transform and for the discrete Fourier
transform. IEEE Trans. Acoust. Speech Signal Process, 32:803 { 816, 1984.
[37] J. Wilkinson. The Algebraic Eigenvalue Problem. Clarendon Press, Oxford, 1965.
27