Consider boundary value problem, example: Poisson equation
∆Φ(r) = ρ(r)
solution methods:
I
diagonalize in Fourier space, −k 2 Φ̂ = ρ̂
I
discretizise equation on grid, e.g. with finite differences. This results in a
linear system LΦ = ρ
How to solve the discrete linear system?
I
Gauss-elimination (direct solver). Too expensive, numerical problems for
large systems
I
Iterative solvers: constuct a sequenze of Φni which converges for n → ∞
to Φi .
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Physical interpretation: extend boundary value problem to a time dependent
diffusion problem Φ(r) → Ψ(r, t) (with “suitable” Dirichlet or von Neumann
boundary conditions),
∂Ψ(r, t)
= ∆Ψ(r, t) − ρ(r)
∂t
I
exact solution of the boundary value problem is stationary solution of the
diffusion problem
I
every stationary solution of the diffusion problem is a solution of the
boundary value problem
Fourier methods
LU decomposition
But: does an arbitrary initial conditions Ψ(r, 0) converge to a stationary
solution?
Laplace-Operator:
I
physically: yes, since “entropy” grows!
I
mathematically: yes, since all eigenvalues (for k 6= 0) of the
Laplace-operator are negativ.
I
numerically: yes, if discretized operator is negative definite.
Iterative solvers,
Multigrid
F (r, t) := Φ(r, t) − Ψ(r)
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
d(r, t) := ρ(r) − ∆Ψ(r, t)
Iterative solvers:
Conjugate Gradients
Define error
and defect or residuum
Literature
Fourier methods
We notice: error solves the homogenous diffusion equation
∂F (r, t)
= ∆F (r, t)
∂t
2
Fourier modes decay as e −k t .
Remark: there is no need to reduce the error to values lower than the
discretization error!
LU decomposition
The defect equation is equivalent to the boundary value problem
∆F (r, t) = d(r, t)
Simplest case: solve diffusion problem with forward Euler,
Ψn+1 = Ψn + ∆tLΨn − ∆tρ
Iterative solvers,
Multigrid
1D and standard discretization of Laplace operators, grid spacing h:
Ψn+1
= Ψni + ∆t
i
Ψni+1 − 2Ψni + Ψni−1
h2
− ∆tρi
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
1 2
h
2
largest possible timestep for stable iteration ∆t =
(CFL condition) results
in Jacobi-Iteration:
n
n
2
Ψi+1 + Ψi−1
h
−
ρi
Ψn+1
=
i
2
2
I
the “physical time” is eliminated.
I
Ψi is set to the value, which solves the difference equations with the old
values of the neighboring sites.
weakness of the Jacobi iteration: grid mode with Nyquist wavenumber
k = π/h will not be damped due to amplification factor −1.
Literature
Fourier methods
LU decomposition
Modification: ω-Jacobi:
Ψn+1
= (1 − ω)Ψni + ω
i
„
Ψni+1 + Ψni−1
2
−
h2
ρi
2
«
with 0 < ω ≤ 1.
I
former Jacobi iteration for ω = 1
I
optimal value in most cases: ω = 2/3
Iterative solvers,
Multigrid
more popular scheme: Gauss-Seidel iteration with
Ψn+1
=
i
Ψni+1 + Ψn+1
i−1
2
−
h2
ρi
2
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
I
similar to Jacobi, but use updated values as soon as they are available
Literature
Fourier methods
I
easy to programme: old values need not to be stored.
I
can define ω-Gauss-Seidel analog to ω-Jacobi
I
Relaxation properties not easy to calculate
Matlab programme:
iterate.m: smooth.m
LU decomposition
All schemes share
+ simple to programme like an Euler step!
+ can easily be extended to multi dimensions
+ quite universal
Iterative solvers,
Multigrid
– slow convergence on large grids
Why is convergence so bad?
I
error modes “diffuse” in “time” like e −k
2
t
2
e −k ∆t
I
reduction from n → n + 1 by a factor
I
longest wavelength at N grid points kmin = π/L = π/(Nh)
I
damping of longest wavelength

ff

ff
π 2 ∆t
π2
γmax = exp − 2 2 = exp −
N h
2N 2
I
N large ⇒ γmax ≈ 1, very weak damping
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
LU decomposition
That means that these interation schemes are good smoothers: short
wavelength are damped fast but long wavelength are slowly damped
Thus we have: many iterations (expensive)
Iterative solvers,
Multigrid
How can we cure this?
Idea: relax short wavelength on fine und long wavelengths on coarse grid!
Use not just one grid but several with distinct grid spacings h ⇒ multigrid.
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
first step: coarse grid correction
Iterative solvers:
Conjugate Gradients
equation to solve
Fourier methods
Literature
LΦ = ρ
ist aquivalent to
LF = d
with error F := Φ − Ψ and defect d := ρ − LΨ
LU decomposition
We need
I
original “fine” grid Ωh with values Ψni , ρi und grid spacing h
I
a smoother (see above), like Gauss-Seidel
I
a coarse grid Ω2h with grid spacing 2h
I
coarsening or restriction: transfer data from Ωh → Ω2h :
I
interpolation or prolongation: transfer data from Ω2h → Ωh
h
f2i−2
h
f2i−1
h
f2i
h
f2i+1
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
h
f2i+2
Iterative solvers:
Conjugate Gradients
Ωh
x
xh2i−3
xh2i−2
xh2i−1
xh2i
xh2i+1
xh2i+2
xh2i+3
Literature
Fourier methods
LU decomposition
2h
fi−1
fi2h
2h
fi+1
x2h
i−1
x2h
i
x2h
i+1
Ω2h
x
standard transfer operations for x2ih = xi2h :
restriction Ωh → Ω2h (fine to coarse):
I
injection fi 2h = f2ih (“Subsampling”).
I
h
h
full-weighting fi 2h = 14 f2i−1
(“floating average”).
+ 12 f2ih + 41 f2i+1
h
f2i−2
h
f2i−1
h
f2i
h
f2i+1
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
h
f2i+2
Ωh
x
fi2h
Fourier methods
1
4
1
4
2h
fi−1
Literature
1
2
1
Iterative solvers:
Conjugate Gradients
LU decomposition
2h
fi+1
Ω2h
x
restriction Ωh → Ω2h : left: injection, right “full weighting”
prolongation Ω2h → Ωh (coarse to fine)
I
h
2h )
linear interpolation: f2ih = fi 2h and f2i+1
= 21 (fi 2h + fi+1
Iterative solvers,
Multigrid
h
f2i−2
h
f2i
h
f2i−1
h
f2i+1
h
f2i+2
Ωh
x
1
1
1
2
2h
fi−1
Iterative solvers:
Conjugate Gradients
1
1
2
1
2
1
2
1
2
fi2h
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Literature
2h
fi+1
Fourier methods
Ω2h
LU decomposition
x
prolongation Ω2h → Ωh
How do we proceed?
1. Perform a few smoothing steps on Ωh .
⇒ short wavelengths will be damped pre-smoothing.
2. Calculate defect d = ρ − LΨ on Ωh . Defect contains wavelengths longer
than π/h.
3. Transfer defect to coarse grid Ω2h . Restriction of the defect.
4. Solve equation for defect LF = d on coarse grid. “Solution” is the error
F and contains the longer wavelengths of the error.
5. Transfer error to fine grid prolongation of the error.
6. Correct Ψ on fine grid: Ψ → Ψ + F . coarse-grid-correction.
7. Again smoothing on Ωh (1-2 steps). post-smoothing.
8. Repeat the whole interation until defekt d is below a given accuracy.
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Pre-Smooth
Post-Smooth
Pre-Smooth
Literature
Fourier methods
Ωh
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
Restrict
dh = ρh − Lh Ψh
Ω2h
11
00
00
11
00
11
11
00
00
11
00
11
LU decomposition
Prolong and correct
Ψh → Ψh − Fh
00
11
11
00
00
11
00
11
11
00
00
11
Solve L2h F2h = −d2h
00
11
11
00
00
11
But one problem remains:
How to solve the equation for the defect LF = d on Ω2h ?
The equation for the defect LF = d is now a new poisson equation on Ω2h .
Iterative solvers,
Multigrid
Thus: solve this with the same trick with a new coarse-grid-correction
Ω2h → Ω4h → Ω2h
For this we have to solve an equation of a new defect on Ω4h . This we solve
again with a coarse-grid-correction Ω4h → Ω8h → Ω4h , etc.
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
⇒ recursion to the coarsest grid and then
I
direct solver
I
a smoothing step on one gridpoint (= “direct solver”), or
I
several smoothing steps on a few grid points
LU decomposition
V-cyclus (complete up and down in the grid hierachy):
Ωh
Ω2h
Ω4h
11
00
00
11
00
11
11
00
00
11
00
11
00
00 11
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
omega_8h
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
11
00
00
11
00
11
Smooth
Fourier methods
Restrict defect
LU decomposition
Prolong correction
Solve coarsest
multigrid V-cyclus
Alternatives are W-cyclus, “full”-multigrid.
Ωh
Ω2h
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
00
11
11
00
00
11
Ω4h
00
11
11
00
00
11
00
11
11
00
00
11
omega_8h
00
11
11
00
00
11
11
00
00
11
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
00
11
11
00
00
11
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
11
00
00
11
00
11
Fourier methods
Smooth
LU decomposition
Restrict defect
Prolong correction
multigrid W-cyclus
Matlab programme
mg.m: solveOne.m smooth.m defect.m restrict.m prolong.m grid.m
Iterative solvers,
Multigrid
It holds: convergence rate per multigrid-cyclus is independent of the grid
resolution of the finest grid, complexity is O(N).
Typical convergence rates per multigrid-cyclus: reduction of the defekts by a
factor of 10 − 20.
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
LU decomposition
Multigrid methods belong to the fastest methods solving large systems again
because of the complexity O(N)!
Multigrid is art in its own! Many problem oriented special treatments.
Nonlinear problems, adaptive MG: full-approximation scheme
What do we have?
Consider problem: A[u] = f
1. smooth on Ωh and obtain guess v
=⇒ Ah [u h ] − Ah [v h ] = f h − Ah [v h ]
=⇒ Ah [e h ] = d h
with e h = u h − v h , d h = f h − Ah [v h ]
2. restrict and solve for defect on Ω2h
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
A2h [e 2h ] = d 2h = Ih2h d h
3. prolongate and correct on Ωh
Literature
Fourier methods
h e 2h
v h ← v h + I2h
LU decomposition
This does not work for nonlinear problems since
A[u] − A[v ] 6= A[e]
But use A[u] − A[v ] = d !!!
1. smooth on Ωh and obtain guess v
=⇒ Ah [u h ] − Ah [v h ] = f h − Ah [v h ]
=⇒ Ah [v h + e h ] = Ah [v h ] + d h
with e h = u h − v h , d h = f h − Ah [v h ]
right hand side is known !
2. restrict:
v 2h = Ih2h v h , d 2h = Ih2h d h
3. solve coarse grid equation on Ω2h
A2h [u 2h ] = A2h [v 2h ] + d 2h
4. extract error on Ω2h
e 2h = u 2h − v 2h
5. prolongate and correct on Ωh
h e 2h
v h ← v h + I2h
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
LU decomposition
Remarks:
I
If A is linear then FAS reduces to our former correction scheme:
A2h [v 2h + e 2h ] = A2h [v 2h ] + d 2h =⇒ A2h [e 2h ] = d 2h
I
An exact solution to the fine-grid problem is a fixed point of FAS
A2h [v 2h + e 2h ] = A2h [v 2h ] + Ih2h (f h − Ah [v h ]) =⇒ A2h [v 2h ] = A2h [v 2h ]
I
A slightly different view
A2h [u 2h ] = A2h [Ih2h v h ] + Ih2h (f h − Ah [v h ])
can be written as
with the tau correction term
τh2h
=
A2h [Ih2h v h ]
Iterative solvers:
Conjugate Gradients
Literature
−
LU decomposition
Ih2h Ah [v h ].
In general τh2h 6= 0 and alters the coarse grid equation to enhance
approximation properties.
usefull for adaptive MG
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Fourier methods
A2h [u 2h ] = Ih2h f h + τh2h
I
Iterative solvers,
Multigrid
Literature
I
Briggs, Henson, and McCormick, A Multigrid Tutorial, 2nd Edition
(SIAM 2000).
I
https:
//computation.llnl.gov/casc/people/henson/mgtut/welcome.html
I
hypre: https:
//computation.llnl.gov/casc/linear solvers/sls hypre.html
I
http://www.mgnet.org
I
Trottenberg, Oosterlee, and Schüller, Multigrid: Basics, Parallelism and
Adaptivity (Academic Press 2001).
Fourier methods
Wesseling, An Introduction to Multigrid Methods (Edwards 2004).
LU decomposition
I
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Conjugate Gradients
We search solution x to linear system
Ax = b
Iterative solvers,
Multigrid
Here, A is real, symmetric, positive definite
(this follows from many physical applications, especially variational principles)
Equivalent problem: minimize function
f (x) :=
1
1
xAx − x · b = xi Aij xj − xi bi
2
2
since: ∇x f (x) = Ax − b
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
General iteration scheme:
x0
and direction
LU decomposition
p0
I
Start with
I
Determine minimum of f in this direction ⇒ x1
I
Choose new direction p0 starting from x1
I
continue until ∇f = 0
Calculation of the minimum xn+1 with given xn and pn is easy, since f is a
quadratic form:
xn+1 = xn + αn pn
We want
0
=
d
f (xn + αn pn )
dαn
pn Axn + αn pn Apn − pn · b
=
−pn · rn + αn pn Apn
!
=
thus:
αn =
pn · r n
pn Apn
mit Residuum
rn := b − Axn = −∇f (xn )
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
LU decomposition
So we have
xn+1 = xn + αn pn
with the new residuum
rn+1 = rn + A(xn − xn+1 ) = rn − αn Apn
and pn · rn+1 = 0 (new residuum ⊥ old search direction)
Question: How to choose the search directions?
I
Simplest choice: successive choice of canonical directions e1 , e2 , e3 , ...
in general bad, since no information of f bzw. A is been used
I
classical choice: “steepest descent” against the gradient of f :
pi = −∇f (xi ) = ri .
problem: does produce narrow zick-zack paths in the x-space, since
always pn+1 ⊥ pn
I
best choice: conjugate gradients, determine xn+1 together with pn+1 .
CG-Iteration starting with x0 , r0 = b − Ax0 and p0 = r0 :
αn
:=
(rn )2
pn · r n
= n n
n
n
p Ap
p Ap
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
n+1
:=
x +α p
Fourier methods
rn+1
:=
rn − αn Apn
LU decomposition
βn
:=
(rn+1 )2
(rn )2
pn+1
:=
r n + β n pn
x
n
n n
search directions pn are not orthogonal, but A-orthogonal or conjugated:
pn Apm = 0
for n < m
In addition, it holds
pn · r m
=
0
for n < m
rn · rm
=
0
for n < m
(by complete induction).
The last equation states:
Scalarproduct of the new residuum with all former residua vanishes!
In
RN
it holds: after N iterations
rN
= 0 and
xN
is exact solution!
Iterative solvers,
Multigrid
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
I
We build up a subspace in which f is mininzed.
I
With every iteration we enlarge the dimension of this subspace until f is
minimized in RN after N steps.
I
Robust method mostly used with preconditioning
I
Scheme contains only vector-matrix multiply
⇒ efficient for sparse matrices
LU decomposition
Bi-conjugierte gradients:
I
Generalization of CG, A doesn’t need to be symmetric and positive
definite.
I
Two pairs of residua and search directions, ri ,r̄i and pi , p̄i are iterated.
Iterative solvers,
Multigrid
standard library for CG-like schemes: “GMRES”.
Literature
I
Yousef Saad, Iterative Methods for Sparse Linear Systems (Siam 2003)
see also http://www-users.cs.umn.edu/∼saad/books.html
I
J. R. Shewchuk, An Introduction to the Conjugate Gradient Method
Without the Agonizing Pain, http://www.cs.cmu.edu/∼quake-papers/
painless-conjugate-gradient.pdf
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
I
PETSc: http://www-unix.mcs.anl.gov/petsc/petsc-as
I
hypre: https:
//computation.llnl.gov/casc/linear solvers/sls hypre.html
Fourier methods
LU decomposition
Fourier methods
Iterative solvers,
Multigrid
I
FFTW: http://www.fftw.org
I
p3dfft: http://www.sdsc.edu/us/resources/p3dfft.php
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
LU decomposition
LU decomposition
Iterative solvers,
Multigrid
I
superLU: http://crd.lbl.gov/∼xiaoye/SuperLU
Jacobi-Iteration,
Gauss-Seidel
Multigrid
Nonlinear problems,
adaptive MG:
full-approximation
scheme
Literature
Iterative solvers:
Conjugate Gradients
Literature
Fourier methods
LU decomposition