Department of Computer Science,
University of Illinois at Urbana-Champaign
Probing for Schur Complements and
Preconditioning Generalized
Saddle-Point Problems
Eric de Sturler,
[email protected],
http://www-faculty.cs.uiuc.edu/~sturler
Optimization Technology Center,
Northwestern University, October 6, 2004
Collaborators
„
„
„
„
„
Jörg Liesen, TU Berlin
Chris Siefert, Mike Parks, UIUC
Armand Beaudoin, UIUC: metal deformation
Lihua Zhu, UIUC: metal deformation
Alla Sheffer, UBC: surface parameterization
Overview
„
„
„
Crash course on Krylov subspace methods
Saddle-point problems
Preconditioners
…
„
„
„
„
Preconditioning Matrices with Nonzero (2,2) Block
Approximating the Schur complement
Probing
Numerical results
…
…
„
Numerical Example: Surface Parameterization
Oseen Problem (lid driven cavity)
Metal Deformation
Conclusions
Krylov Methods Crash Course
Consider Ax = b . Given x 0 and r0 = b − Ax 0 , compute optimal
update from K m (A, r0 ) ≡ span {r0 , Ar0 , …, Am −1r0 }:
min
z ∈K m (A,r0 )
b − A (x 0 + z ) 2
⇔
min
z ∈K m (A,r0 )
r0 − Az
2
Let K m = ⎡⎢r0 Ar0 A2r0 Am −1r0 ⎤⎥ , then z = K m ζ ,
⎣
⎦
and we must solve the following least squares problem
AK m ζ ≈ r0
⇔
⎡Ar A2r
0
⎢⎣ 0
Am r0 ⎤⎥ ζ ≈ r0
⎦
Do this accurately and efficiently every iteration for increasing m .
Arnoldi recurrence:
H
AVm = Vm +1 H m , where v1 = r0 / r0 2 , Vm +1Vm +1 = I m +1 , and
range (Vm +1 ) = range (K m +1 )
r0 − AVm ym
2
= Vm +1e1 r0
2
−Vm +1 H m ym
2
= e1 r0
2
− H m ym
2
Krylov Methods Crash Course
Consider Ax = b , and relate convergence to polynomials.
x m = x 0 + z m where z m ∈ span{r0 , Ar0 , A2r0 , …, Am −1r0 }
rm = r0 − Az m ∈ span{r0 , Ar0 , …, Am r0 }
Assume A = U ΛU
min
z ∈K m (A,r0 )
−1
(diagonalizable), then residual at step m
r0 − Az = min pm (A) r0 ≤ r0 U U −1 min max pm (λ )
pm (0)=1
pm (0)=1 λ ∈Λ(A)
For normal matrix this bound is sharp. For highly nonnormal
matrix this bound may not be useful.
κ (U ) small: convergence determined by minimal polynomial
Clustered eigenvalues yield fast convergence: preconditioning
Eigenvalues surrounding origin yields very poor convergence.
Krylov Methods Crash Course
1. A = diag (1,2, 3, …,100)
2. A = diag (−1, −100,1,2, …, 49, 52,
,100)
3. A = diag (−99, −97, …, −1,1, 3, …, 97, 99)
GMRES/MINRES
2
0
-2
log10|r|-4
-6
1
-8
3
2
-10
-12
-14
0
20
40
60
#iterations
80
100
120
Generalized Saddle-point Problems
We consider systems of the type
⎛A BT ⎞ ⎛x ⎞ ⎛ f ⎞
⎟⎟ ⎜ ⎟ ⎜ ⎟
⎜⎜
= ⎜ ⎟⎟
⎟
⎜⎜C 0 ⎟⎜
⎟
y
⎟⎟ ⎜⎝ ⎠ ⎜⎝g ⎠⎟
⎝
⎠
and
⎛A BT ⎞ ⎛x ⎞ ⎛ f ⎞
⎟⎟ ⎜ ⎟ ⎜ ⎟
⎜⎜
= ⎜ ⎟⎟
⎟
⎜⎜C D ⎟⎜
⎟
y
⎟⎟ ⎜⎝ ⎠ ⎜⎝g ⎠⎟
⎝
⎠
Systems of this type arise in a variety of problems:
¾ Constrained optimization problems
o FETI (type) methods (Mike Parks)
o Surface parameterization
¾ Systems of PDEs with continuity constraints
o
o
o
o
Navier-Stokes
Potential flow in porous media
Polycrystal plasticity – metal deformation
Electrostatics / electromagnetics
Preconditioners
„
Various preconditioners have been proposed
… Block
diagonal preconditioner
… Constraint
… Block
preconditioner
upper triangular
„
‘Ideal version’ leads to small number of nonzero
eigenvalues, and hence fast convergence for Krylov
methods
„
In general we use an approximation to these ideal
versions (otherwise too expensive)
„
Typically involve Schur complement type matrix
Preconditioners
Significant body of work by Elman, Golub, Wathen, Benzi,
Silvester, Gould, Nocedal, Hribar, Simoncini, Perugia, BP, …
Use splitting A = F − E (dS&L):
⎛F −1
⎞ ⎛F − E
⎟⎟ ⎜
⎜⎜
⎟⎟ ⎜⎜
⎜⎜
−
1
(CF −1BT ) ⎠⎟ ⎜⎝ C
⎜⎝
M = (CF B
BT ⎞⎟ ⎛⎜I − S
⎟⎟ = ⎜
0 ⎠⎟⎟ ⎜⎝⎜ M
N ⎞⎟
⎟⎟
0 ⎠⎟
T −1
−1 T
,
C
N
=
F
B , MN = I , (NM ) = NM
)
−1
⎛0 ⎞
Oblique projection:NM = U 1 U 2 ⎜⎜ I ⎟⎟ U 1 U 2
⎝
⎠
−1
2
(
)
(
)
Principal angles ( ωi = cos ϕi ) between null (NM ) and
range (NM ) play important role in eigenvalue bounds.
Preconditioners
N ⎞⎟ ⎛x ⎞ ⎛⎜ fˆ⎞⎟
⎟⎟ ⎜⎜ ⎟⎟ = ⎜⎜ ⎟⎟
0 ⎠⎟ ⎝⎜y ⎠⎟ ⎜⎜⎝gˆ⎠⎟⎟
1
⎛1 + ωmax ⎞ 2
⎟⎟ S , λ ∈ {1, (1 ± 5 ) / 2}
Eigenvalues: λS − λ ≤ 1.5 ⎜⎜
⎝⎜ 1 − ωmax ⎠⎟⎟
⎛I − S
Preconditioned system: ⎜⎜⎜
⎜⎝ M
Further splitting gives fixed point iteration that depends
only on x : x k +1 = (I − NM ) Sx k + f
Solves related system (I − (I − NM ) S ) x = f
Eigenvalues 1 − λR ≤ (1 − ω
−1 2
2
max
)
S
Preconditioners
This approach is applicable to general problems; for details
and analysis see dS&L 2002.
¾Eigenvalue bounds for general problems
¾Regarding related system
o Smaller system to solve
o Good bounds on eigenvalues, single cluster
o Shows bounds always better than for block diag. prec.
o Satisfies constraints
o Extends constraint preconditioners to general
matrices and is more efficient (CPU and memory).
Problem is that (this type of) iteration involves (CF B
−1
which may be expensive to compute and iterate with.
In that case, we must look for alternatives.
T −1
)
,
Invariance Property (also nonzero (2,2))
⎛ A
ˆ = ⎜⎜
A
⎜⎜R C
⎝ 1
B R2 ⎞⎟
⎟⎟,
0 ⎠⎟⎟
T
⎛F −1
⎜⎜
Preconditioner: ⎜
⎜⎜ 0
⎝
⎞
⎟⎟
⎟
−1 ⎟
−1 T
(R1CF B R2 ) ⎠⎟⎟
0
⎛I − S N ⎞
⎟⎟
⎜⎜
−1
− 1 T −1
−1 T
⎟⎟, where N = F B R2 and M = R2 (CF B ) C
⎜⎜
0 ⎠⎟⎟
⎜⎝ M
MN = R2−1MNR2 = I ,NM = NM (oblique) projection.
This matrix has exactly the same eigenvalues (and bounds)
as the original matrix with block diagonal preconditioner.
Also leads to the same fixed point iteration/related system
(I − (I − NM )S ) x = (I − (I − NM )S ) x = f
Surface Parameterization
Example: Texture Mapping
Angle-based Flattening
Solve mesh flattening as constrained optimization problem.
Minimize relative deformation of angles,
2
2
α
−
φ
/
φ
∑ ( i, j i, j ) i, j ,
i, j
Subject to constraints on valid 2D mesh
1. Angles between 0 and π (orientation), hardly ever a
problem (dealt with algorithmically)
2. Angles in triangle sum to π ,
3. Angles around interior node sum to 2π ,
4. Triangles at an interior node need to agree on edge
lengths: nonlinear constraint.
T
min ∑ (αi, j − φi , j ) / φ subject to ⎡⎢g 2 (α) g 3 (α) g 4 (α)⎤⎥ = 0
⎣
⎦
i, j
2
i, j
Critical point of Lagrangian L (α, λ ) = F (α ) + λT g (α)
Nonlinear System
Critical point of L(, ) = F( ) + Tg( ): ∫ , L(, ) = 0
Newton iteration:
∫ 2, L(, )
= −∫ , L(, )
∫ 2 [F( ) + T g( )] [∫ g( )] T
Jacobian:
=
(
)
∫ g 0
A + G k B T C Tk
B
0 0 (symmetric and indefinite)
Ck 0 0
j
A = diag 2w i and G k depends only on g (4 ) ( )
B depends only on g (2 ) ( ) and hence is constant (zero,one)
C k depends on g (3 ) and g (4 ) and is partially constant (g (3 ) )
Preconditioning
Consider the following choice of blocks
⎡ A BT ⎤
⎡F − Ek C kT ⎤
⎡Gk
⎢
⎥ with F = ⎢
⎥ and E = ⎢
k
⎢ C
⎥
⎢
⎥
⎢0
B
0
0
k
⎢⎣
⎥⎦
⎢⎣
⎥⎦
⎢⎣
0⎤
⎥.
0⎥⎥
⎦
Explicit inverse of F known (very good splitting).
⎡I − Sk
⎢
⎢ M
k
⎢⎣
N k ⎤ ⎡x ⎤ ⎡ f ⎤
⎥⎢ ⎥ = ⎢ ⎥
0 ⎥⎥ ⎢⎣y ⎥⎦ ⎢⎣g ⎥⎦
⎦
and
1
(I − (I − N k M k )Sk ) x = f
⎛1 + ωmax ⎞ 2
−1 2
2
⎟
S
λS − λ ≤ 1.5 ⎜⎜
⎟⎟ S and 1 − λR ≤ (1 − ωmax )
⎝⎜ 1 − ωmax ⎠⎟
Scaling Constraints
Scale constraints g 3 (α) and g 4 (α) by ε and precondition:
⎡I − εSk
Block diagonal preconditioner: ⎢⎢ −1
⎢⎣ ε M k
εN k ⎤ ⎡x ⎤ ⎡ f ⎤
⎥
⎥⎢ ⎥ = ⎢
⎢
⎥
0 ⎥⎥ ⎢⎣y ⎥⎦ ⎢ ε−1g ⎥
⎦
⎣
⎦
Related system (after splitting): (I − ε (I − N k M k ) Sk ) x = f
Eigenvalue bounds for the scaled system:
1
⎛1 + ωmax ⎞ 2
Block diagonal preconditioner: λS − λ ≤ 1.5ε ⎜⎜
⎟⎟⎟ S
⎜⎝ 1 − ωmax ⎠⎟
Related 1 − λR ≤ ε (1 − ω
−1 2
2
max
)
S
Very fast convergence even for modest ε , (but Newton …)
Convergence “Three Balls”
GMRES convergence (no scaling)
10
10
10
10
10
10
0
no prec
right prec.
B(S)
(I-F)
-2
-4
-6
-8
-10
20
40
60
80
100
Convergence “Three Balls”
10
Computed ||r n||/||r 0 ||
10
10
10
10
0
GMRES convergence with scaling
no prec
right prec.
B(S)
(I-F)
-2
-4
-6
=1
-8
= 0.1 = 1
10
= 0.1
-10
10
20
30
40
50
Eigenvalues of B(S)
=1
0
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
3
3.5
0
0.5
1
1.5
2
2.5
3
3.5
0
0.5
1
1.5
2
2.5
3
3.5
= 0.1
0
-1.5
-1
-0.5
= 0.01
0
-1.5
-1
-0.5
Eigenvalues of Related System
=1
0
0.7
0.8
0.9
1
1.1
1.2
1.3
1.4
0.9
1
1.1
1.2
1.3
1.4
0.9
1
1.1
1.2
1.3
1.4
= 0.1
0
0.7
0.8
= 0.01
0
0.7
0.8
Preconditioners – nonzero (2,2)
First, we consider saddle-point problems with nonzero (2,2)
block, such as stabilized finite element discretizations.
Block-diagonal preconditioner from splitting (1,1) block
⎞
0
⎛F − E BT ⎞ ⎜⎛F −1
⎟
⎟⎟ → ⎜
⎜⎜
−
1
⎟
D ⎠⎟ ⎜⎜ 0 − (D − CF −1BT ) ⎟⎟
⎜⎝ C
⎝
⎠
⎛ fˆ⎞
⎛I − S N ⎞ x
⎟⎟ y = ⎜⎜ ⎟⎟,
Preconditioned system: ⎜⎜ M
Q ⎠⎟
⎜⎝gˆ⎠⎟
⎝
2
where MN = I + Q and (NM ) = NM + NQM
()
Further splitting leads to fixed point iteration and a smaller
related system: (I − (I − NM ) S ) x = f
Preconditioners – nonzero (2,2)
⎛I − S N ⎞
⎟⎟?
Eigenvalues of B (S ) = ⎜⎜ M
Q
⎝
⎠⎟
⎧
2⎫
⎛ I N⎞
⎪
⎪
1
1
⎪
⎟
⎜
B (0) = ⎜M Q ⎟⎟ :
λ ∈ ⎨1, (1 + δj ) ± 1 + (1 + δj ) ⎪
⎬
⎝
⎠
⎪
⎪
4
⎪ 2
⎪
⎩
⎭
where δj is an eigenvalue of Q and typically small.
Let null (NM ) → U 1 and other eigenvectors NM → U 2 = U 2Θ
Eigenvalues of B (S ) from perturbation of B (0):
1
⎛1 + ωmax ⎞ 2
⎟⎟ S
λS − λ ≤ 2γκΘ ⎜⎜
⎝⎜ 1 − ωmax ⎠⎟⎟
(typically γκΘ modest)
(I − (I − NM )S ): 1 − λR ≤ (1 + ω1δ + ω1δκΘ )(1 − ω
−1 2
2
max
)
S
Approximating the Schur complement
Replace Schur complement type matrix by approximation:
⎛F −1 0 ⎞
⎛F −1 0 ⎞
⎟⎟ by ⎜
⎟⎟, where S −1 is approximated directly
⎜⎜
⎜⎜
−1 ⎟
F
⎜⎜⎝ 0 S F−1 ⎠⎟⎟
0
S
⎟
⎜⎝
F ⎠
⎛I − S
Preconditioned system ⎜⎜ M
⎜⎝ 2
N ⎞⎟
0 ⎠⎟⎟
⎛I − S N ⎞
⎟⎟
or ⎜⎜ M
⎜⎝ 2 Q2 ⎠⎟
where N = F −1BT , M 2 = S F−1C , M 2N = S F−1S F ≡ I + E , and
2
(NM 2 ) = N (I + E ) M 2 = NM 2 + N EM 2
⎛I − S N ⎞ ⎛ I
⎟⎟ = ⎜
Bounds: ⎜⎜ M
⎜⎝ 2 Q2 ⎠⎟ ⎝⎜⎜M
N ⎞⎟ ⎛ S
⎜⎜
−
⎟
Q ⎠⎟ ⎝⎜EM
0 ⎞⎟
EQ ⎠⎟⎟
Approximating the Schur complement
⎛I − S N ⎞ ⎛ I
⎟⎟ = ⎜
B (S , E ) = ⎜⎜ M
⎜⎝ 2 Q2 ⎠⎟ ⎝⎜⎜M
N ⎞⎟ ⎛ S
⎜⎜
−
⎟
Q ⎠⎟ ⎝⎜−EM
0 ⎞⎟
−EQ ⎠⎟⎟
Eigenvalues of B (S , E ) from perturbation of B (0, 0):
1
λS ,E
⎛1 + ωmax ⎞ 2
ˆV E
− λ ≤ γκΘ ⎜⎜
⎟⎟⎟ S + γκ
⎜⎝ 1 − ωmax ⎠⎟
ˆ V modest)
(typically γκΘ and γκ
When D = 0 (and hence Q,Q2 = 0 )
1
λS ,E
⎛1 + ωmax ⎞ 2
2
⎟
⎜
− λ ≤ 2⎜
E
⎟⎟ S +
⎜⎝ 1 − ωmax ⎠⎟
5
Approximating the Schur complement
Splitting as before leads to exactly the same related system.
⎛I − S N ⎞ ⎛ I
⎟⎟ = ⎜
New splitting: ⎜⎜ M
⎜⎝ 2 Q2 ⎠⎟ ⎝⎜⎜M 2
⎞ ⎛S 0 ⎞
N
⎟⎟ − ⎜
⎟⎟
M 2N − I ⎠⎟ ⎝⎜⎜ 0 E ⎠⎟
⎛f ⎞
−N E ⎞⎟ x
⎜⎜ ⎟⎟
=
⎟
I + E ⎠⎟⎟ y
⎜⎝g ⎠⎟⎟
⎛I − (I − NM 2 ) S
Linear system: ⎜⎜
−M 2S
⎜⎝
()
Eigenvalues cluster around 1.
1 − λR ≤ 1 + N
2
1 + M2
2
max ( S , E )
We lose advantages of smaller system and satisfying the
constraints.
Probing for the Schur Complement
Approximate − (D − CF B
−1
T −1
)
without having (D − CF −1BT )
Probing (Chan&Mathew’92): reconstruct matrix with small
bandwidth from products with selected vectors and invert.
Based on rapid decay from diagonal in matrix (1D interface).
We use decay of F −1 on mesh with structure of C , BT , D to
select pattern of large coefficients in matrix to probe for
Construct set of vectors to minimize lumping other coeff.s
Cheap to use multiple orderings and combine inverses of
matrices with small bandwidth (convergence not great)
Better to approximate matrix structure for multidimensional
problem directly and use incomplete decomposition
Oseen problem, driven cavity
„
„
Software by Elman, Silvester, Wathen, Ramage
System (A nonsymmetric):
⎡ A BT ⎤ ⎡x ⎤ ⎡ f ⎤
⎢
⎥⎢ ⎥ = ⎢ ⎥
⎢B −βC ⎥ ⎢y ⎥ ⎢g ⎥
⎣⎢
⎦⎥ ⎣ ⎦ ⎣ ⎦
„
viscosity ν=1/10 and stabilization β=1/4
Basic Splittings of (1,1) Block
0.6
Unpreconditioned
Diagonal Prec.
Tridiagonal Prec.
0.4
0.2
Imag.
0
−0.2
−0.4
−0.6
−0.8
−0.5
0
0.5
1
Real
1.5
2
Oseen problem
„
Not constrained to ‘structured’ splittings A=F-E
„
Multigrid cycle is a (complicated) splitting
„
Use number of V-cycles for F-1.
„
Useful because we can study convergence and
eigenvalue (bounds) for sequence of problems with
decreasing ||S||.
„
Likewise, we can analyze the effect of
approximations to the Schur complement (not as
easy)
1 V-cycle, Related System: Eigenvalues
0.1
0.05
0
−0.05
−0.1
−0.15
−0.2
0.7
0.75
0.8
0.85
0.9
0.95
1
1.05
2 V-cycles, Related System: Eigenvalues
0.1
0.05
0
−0.05
−0.1
−0.15
−0.2
0.7
0.75
0.8
0.85
0.9
0.95
1
1.05
4 V-cycles, Related System: Eigenvalues
0.1
0.05
0
−0.05
−0.1
−0.15
−0.2
0.7
0.75
0.8
0.85
0.9
0.95
1
1.05
6 V-cycles, Related System: Eigenvalues
0.1
0.05
0
−0.05
−0.1
−0.15
−0.2
0.7
0.75
0.8
0.85
0.9
0.95
1
1.05
Exact Schur Complement
Convergence for block-diagonal preconditioner for
various numbers of V-cycles for the splitting
0
10
1 V−Cycle
2 V−Cycles
3 V−Cycles
4 V−Cycles
5 V−Cycles
6 V−Cycles
−2
10
−4
k
0
||r ||/||r ||
10
−6
10
−8
10
−10
10
5
10
15
20
25
Iteration
30
35
40
45
Inexact Schur Complement - ILU(1.e-4)
Convergence for block-diagonal preconditioner for
various numbers of V-cycles for the splitting
0
10
1 V−Cycle
2 V−Cycles
3 V−Cycles
4 V−Cycles
5 V−Cycles
6 V−Cycles
−2
10
−4
||rk||/||r0||
10
−6
10
−8
10
−10
10
5
10
15
20
25
30
Iteration
35
40
45
50
Exact Schur Complement
Convergence for related system using various
numbers of V-cycles for the splitting
0
10
1 V−Cycle
2 V−Cycles
3 V−Cycles
4 V−Cycles
5 V−Cycles
6 V−Cycles
−2
10
−4
||rk||/||r0||
10
−6
10
−8
10
−10
10
2
4
6
8
Iteration
10
12
14
Inexact Schur Complement - ILU(1.e-4)
Convergence for related system using various
numbers of V-cycles for the splitting
0
10
1 V−Cycle
2 V−Cycles
3 V−Cycles
4 V−Cycles
5 V−Cycles
6 V−Cycles
−2
10
−4
k
0
||r ||/||r ||
10
−6
10
−8
10
−10
10
2
4
6
8
Iteration
10
12
14
Exact Schur Complement
Eigenvalue perturbations and bounds for block-diagonal
preconditioner using various numbers of V-Cycles for splitting
Exact Schur Complement
Eigenvalue perturbations and bounds for related system using
various numbers of V-Cycles for splitting
without κ ( Θ )
Inexact Schur Complement – ILU
Convergence for block-diagonal preconditioner using
various accuracies for Schur Complement ILU
0
10
ILU(1e−3)
ILU(1e−4)
ILU(1e−5)
ILU(1e−6)
−2
10
−4
||rk||/||r0||
10
−6
10
−8
10
−10
10
5
10
15
Iteration
20
25
30
Inexact Schur Complement - ILU
Convergence for related system using various
accuracies for Schur Complement ILU
0
10
ILU(1e−3)
ILU(1e−4)
ILU(1e−5)
ILU(1e−6)
−2
10
−4
k
0
||r ||/||r ||
10
−6
10
−8
10
−10
10
1
2
3
4
Iteration
5
6
7