Section 10
LU Factorization
If we want to solve a system
Ax = b,
Gaussian elimination is probably the best method. There may be some
situations where we want other methods, though. For example, suppose
we have several systems
Ax = b1
Ax = b2
..
.
Ax = bn
with the same matrix of coefficients. In this case, it might make more
sense to find A−1 and solve the systems by x = A=1 b. The flaw in this
otherwise brilliant plan is that A may not be invertible. It may be useful
to develop other methods.
Recall that a square matrix in row echelon form is upper triangular.
Like row echelon matrices, if the matrix of coefficients of a system of
equations is upper triangular then the system can be solved with backsubstitution. Similarly, if the matrix of coefficients of a system of equations
is lower triangular, then the system can be solved with front-substitution;
namely, you use the first equation to solve for the first variable, then move
to the next equation to solve for the next variable, etc.
1
Example. Let
L=
2 0
,
4 3
x
4
x=
and b =
.
y
5
Then
Lx = b
corresponds to
2x
=4
4x + 3y = 5.
The first equation 2x = 4 gives us x = 2, the second equation 4x + 3y = 5
gives us y = (5 − 4x )/3 = (5 − 4 · 2)/3 = −1.
2
So if the matrix of coefficients of a system is upper or lower triangular,
the system can be solved fairly easily. As we shall see, if the matrix of
coefficients is a product of triangular matrices then the system can still be
solved fairly easily.
Suppose we want to solve the system
Ax = b,
where A = LU. Then the system can be rewritten
LUx = b
or
Ly = b where Ux = y.
The system can then be solved in two relatively simple steps.
Example. Given that the matrix


3 1 2
A = 3 3 4
3 2 6
can be written A = LU, where


1 0 0
L = 1 2 0
1 1 3
2
and


3 1 2
U = 0 1 1 ,
0 0 1
solve the system
3x + y + 2z = 3
3x + 3y + 4z = 5
3x + 2y + 6z = 1.
The system is
Ax = b,
where
 
 
x
3



x = y and b = 5 .
z
1
Since this can be written
LUx = b,
it is enough to solve
Ly = b
and
Ux = y,
where
 
u
y = v .
w
Now Ly = b represents the system
u
=3
u + 2v
=5
u + v + 3w = 1,
which has solution
u=3
v = (5 − u)/2 = (5 − 3)/2 = 1
w = (1 − u − v)/3 = (1 − 3 − 1)/3 = −1.
3
Next, Ux = y represents
3x + y + 2z = 3
y + z =1
z = −1,
which has solution
z = −1
y = 1 − z = 1 − (−1) = 2
x = (3 − y − 2z)/3 = (3 − 2 − 2(−1))/3 = 1.
So the solution is x = 1, y = 2, z = −1.
2
Definition. If a square matrix A can be written as the product of a lower
triangular matrix L and an upper triangular matrix U,
A = LU,
then LU is an LU decomposition of A.
2
Even if a matrix has an LU decomposition, it isn’t necessarily unique.
Example. For
both
A=
2 2
,
6 4
1 0
3 1
2 2
0 −2
2 0
6 2
1 1
0 −1
A=
and
A=
are LU decompositions of A.
2
Given a matrix, we still have to find an LU factorization for it, if such
a factorization is possible. The key observation is that when performing
Gaussian elimination to turn a matrix into an upper triangular matrix, the
elementary row operations that are used are:
4
1. multiply a row by a non-zero number
2. add a multiple of one row to a later row
3. switch rows.
(The row operations which aren’t used are adding a multiply of a row to
an earlier row.) The row operations listed above, except row switching,
correspond to lower triangular elementary matrices. We start with the
following proposition.
Proposition. If E = E1 E2 . . . En is a product of elementary matrices, where no
elementary matrix corresponds to switching rows or adding a multiply of a row to
a previous row, then both E and E−1 = En−1 En−−1 1 . . . E1−1 are lower triangular.2
Recall that when finding the inverse of a product of matrices, the order
of the factors is reversed.
Given a matrix A, suppose we can use Gaussian elimination (without
row switching) to put A in upper triangular form; we can then reduce
( A| B)
to
(U | E ) ,
and EA = U. We then get A = E−1 U, or
A = LU,
where L = E−1 is lower triangular.
In Gaussian elimination, multiplying a row by a non-zero number is
used to get a leading 1. If we do not care if we have leading 1s or not (and
if all we are trying to do is get A in upper triangular form, then we don’t
need leading 1s), then we don’t need to multiply a row by a non-zero
number.
Example. Let


2 1 2
A = 4 4 7  .
0 6 13
5
Try to find an LU decomposition for A.
We will try to put A into upper triangular form without switching rows,
and without multiplying a row by a non-zero number. In other words,
only by subtracting multiples of one row from a lower row.
In this case, we will begin by subtracting 2 times row 1 from row 2. We
get


2 1 2
E1 A = 0 2 3  ,
0 6 13
where


1 0 0
E1 = −2 1 0 .
0 0 1
Next, subtracting 3 times row 2 from row 3, we get


2 1 2
E2 E1 A = 0 2 3 ,
0 0 4
where
So


1 0 0
E2 = 0 1 0 .
0 −3 1


2 1 2
U = 0 2 3
0 0 4
and
L = ( E2 E1 )−1
= E1−1 E2−1



1 0 0
1 0 0
= 2 1 0 2 1 0
0 0 1
0 3 1


1 0 0
= 2 1 0 .
0 3 1
6
Notice that L has 1s on the diagonal, and below the diagonal there is a
2 in the second row, first column (which comes from the row operation
of subtracting 2 times row 1 from row 2) and a 3 in the third row, second
column (which comes from the row operation of subtracting 3 times row
2 from row 3). We can use this observation to simplify LU factorization,
in the case where no rows need to be switched.
2
Example. Find an LU factorization of


1 −3 0
2
6 .
A = 0
2 −10 2
Starting with


1 −3 0
2
6 .
A = 0
2 −10 2
we can begin by subtracting 2 times row 1 from row 3 to get a 0 in the
third row, first column, giving us


1 −3 0
0 2 6 .
0 −4 2
To keep track of this, we will put the 2 (as in, subtracting 2 times row 1
from row 3) in the 0 that we get.


1 −3 0
 0
2 6

.
2
−4 2
The circle represents the 0, and the number inside represents the multiplier that we used to get it.
Next, we’ll subtract −2 times row 2 to from row 3. (This is the same as
adding 2 times row 2 to row 3, but we want the multiplier to be −2, we
we subtract.) This gives us


1
−3 0
 0
2
6
.



−2
2
14
7
We now have our upper triangular matrix;


1 −3 0
6 .
U = 0 2
0 0 14
Our lower triangular matrix will have 1s on the diagonal, and the multipliers (the numbers in the circles) below the diagonal:


1 0 0
L = 0 1 0 .
2 −2 1
2
So if A can be put in upper triangular form using elementary row
operations (but no row switches), then we know how to find an LU
factorization. If A cannot be put in upper triangular form without row
switches, then A has no LU factorization.
To put a square matrix in LU form, it is sometimes necessary to rearrange the rows beforehand. These row switches can be kept track of in
a permutation matrix, which is simply the identity matrix with the rows
rearranged. Given a square matrix A, there will be a permutation matrix
P such that PA has an LU decomposition. To find P, L and U, do Gaussian
elimination as above, but also start with a copy of the identity matrix.
Every time you need to switch rows while doing the Gaussian elimination,
also switch the corresponding rows in your copy of the identity matrix.
At the end, the former identity matrix will be a permutation matrix P, and
you will have an LU decomposition for PA.
Example. Let


1 3 3
A = 1 3 2 .
2 2 7
Find a permutation matrix P, a lower triangular matrix L and an upper
triangular matrix U such that PA = LU.
Start with


1 3 3
A = 1 3 2
2 2 7
8
and a copy of the identity matrix


1 0 0
P0 = 0 1 0 .
0 0 1
We can begin by subtracting 1 times the first row of A from the second
row of A, giving us


1 3 3
 1


0 −1 .
2 2 7
and keeping track of this step by putting the 1 in the 0 that we were trying
to get. Next, we can subtract twice row 1 from row 3, giving us

1
3



1
0
2
−4
3


−1
.
1
In order to put this in upper triangular form, we will need to switch rows
2 and 3, giving us


1
3
3

 2
.

−
4
1


1
0
−1
To keep track of this row switch, we will switch the corresponding rows
of P0 , giving us


1 0 0
P1 = 0 0 1 .
0 1 0
We now have our upper triangular matrix;


1 3
3
U = 0 −4 1  .
0 0 −1
9
Our lower triangular matrix will have 1s on the diagonal, and the multipliers (the numbers in the circles) below the diagonal:


1 0 0
L = 2 1 0 .
1 0 1
Our permuation matrix will be


1 0 0
P = 0 0 1 .
0 1 0
We then have PA = LU.
2
10