5.27 Normal Equations in Matrix Form
The normal equations are written in matrix form as
AT Ax = AT y
which, since AT A is now a square matrix can easily be solved.
Existence of a solution is not guaranteed. In that case a Pseudo-inverse is
usually employed to calculate the solution.
If however A has full column rank then the solution exists and is unique!
Note: AT A is usually ill-conditioned.
C. Führer/ A. Sopasakis: FMN050/FMNF01-2013
107
5.28 Existence
Theorem. There exist a solution for the least squares problem
min kAx − bk.
x∈Rn
A vector x solves the least squares problem above if and only if that same
vector solves the normal equations AT Ax = Ab.
The computational cost of AT A is approximately mn2 while that of Ab is
mn. After that we must also add the cost of solving the resulting system...
C. Führer/ A. Sopasakis: FMN050/FMNF01-2013
108
5.29 Example-Quadratic Fitting
Fit the curve c1 + c2t + c3t2 = y to the data (−1, 1), (0, 0), (1, 0), (2, −2).
Applying the data to this curve gives the following system Ax = b




1
1 −1 1
c
1


 1 0 0 

  c2  =  0 
 0 
 1 1 1 
c3
−2
1 2 4


The normal equations are AT Ax = AT b, which solving via LU factorization
gives the solution as c1 = .45, c2 = −.65, c3 = −.25.
Therefore the curve is y = .45 − .65t − .25t2
The residual error is R2 = .152 + (−.45)2 + (−.45)2 + (−.15)2 = .45.
C. Führer/ A. Sopasakis: FMN050/FMNF01-2013
109
5.30 - Gram-Schmidt Orthogonalization Procedure
Given vectors u1, u2, . . . produce orthonormal vectors q1, q2, . . . .
Let v1 = u1.
u2 ,v1
Let v2 = u2 − projv1 u2 = u2 − kv
2 v1 .
1k
u3 ,v1
u3 ,v2
Let v3 = u3 − projv1 u3 − projv2 u3 = u3 − kv
v
−
v .
1
2
kv2 k2 2
1k
etc. . .
Finally normalize all vectors qi = vi/kvik2 for i = 1, . . . , n.
C. Führer/ A. Sopasakis: FMN050/FMNF01-2013
110
5.31 Example - Orthogonalization
Find an orthonormal basis for the space spanned by the vectors u1 = (1, 2, 2)
and u2 = (−4, 3, 2).
Clearly v1 = u1 and q1 = (1/3, 2/3, 2/3).
u2 ,v1
v = (−4, 3, 2) − 2/3(1, 2, 2) = (−14/3, 5/3, 2/3).
Now v2 = u2 − kv
k2 1
1
And finally q2 = (−14/15, 5/15, 2/15).
CHECK: q1 · q2 = 0, q1 · q1 = 1 and q2 · q2 = 0.
C. Führer/ A. Sopasakis: FMN050/FMNF01-2013
111
5.32 Solving Ax = b with QR factorization
• Factor A into QR form where Q is an orthogonal matrix and R is upper
triangular
• Rewrite the system Ax = b as QRx = b and invert matrix Q.
• Since Q−1 = QT obtain Rx = QT b.
• Solve the resulting upper triangular system.
Note: to construct Q use Gram-schmid on the columns of A
Note: matrix A does not need to be square!
Note: to construct the upper triangular R = QT A.
C. Führer/ A. Sopasakis: FMN050/FMNF01-2013
112
5.33 Example




1 −4
−3
x
1
Use QR factorization to solve  2 3 
=  15 
x2
2 2
9
From our previous example above in 5.31 we have already used GramSchmidt on the columns of A and therefore the matrix Q is given below
together with the calculation of the matrix R,


1/3 −14/15
3 2
T


2/3
5/15
, R=Q A=
Q=
0 5
2/3
2/15


−3
x1
We solve R
= QT  15  has the solution x1 = 3.8, x2 = 1.8.
x2
9
C. Führer/ A. Sopasakis: FMN050/FMNF01-2013
113