Matrix Computation
Chapter 5
ORTHOGONALIZATION AND
LEAST SQUARES
-Mohammed
CONTENTS
1.
2.
3.
4.
5.
6.
Householder and Givens Transformations
The QR Factorization
The Full-Rank Least Squares Problem
Other Orthogonal Factorizations
The Rank Deficit Least Square Problem
Square and Undetermined System
1. HOUSEHOLDER & GIVENS TRANSFORMATIONS
HOUSEHOLD REFLECTIONS
Let v be nonzero. An m-by-m matrix P of the form
𝑃 = 𝐼 − 𝛽𝑣𝑣 𝑇 ,
𝛽=
2
𝑣𝑇𝑣
is a Household reflection. The vector v is the Household vector. If a
vector x is multiplied by P, then it is reflected in the hyperplane 𝑣 𝑇
• Household matrices are symmetric and orthogonal.
• Household reflections are similar to Gauss transformations.
• They are rank-1 modifications of the identity and can be used to zero
selected components of a vector.
In particular suppose we have 0≠ x Є Rм and want
to be a multiple of 𝑒1 = 𝐼𝑚 : , 1 . From this we conclude that 𝑣 ∈ 𝑠𝑝𝑎𝑛 𝑥, 𝑒1 .
Setting
Computing The Householder Vector
• Setting 𝑣1 = 𝑥1 − 𝑥 2 leads to the nice property that 𝑃𝑥 is a positive
multiple of 𝑒 1 .
• The recipe is dangerous if x is close to a positive multiple of 𝑒 1 because
severe cancellations would occur.
• However the following formula does not suffer from this defect in the
x1 >0 case.
• In practice, it is handy to normalize the Household vector so that
𝑣1 = 1.
• This permits the storage of v(2:m) where the zeroes have been
introduced in x.
• We refer to v(2:m) as the essential part of the household vector.
•
•
Here, length(.) returns the dimension of a vector. This algorithm involves about 3,
flops.
The computed Household matrix that is orthogonal to machine precision.
(discussed further)
Round Off Properties
• The roundoff properties associated with Household matrices are
very favorable.
• House produces a Household vector ṽ that is very close to the
exact v.
The Factored Form Representation
• Many Householder based factorization algorithms are available that
compute products of Household matrices.
• It is not necessary to compute Q explicitly. For example, if C Є 𝑹𝑚×𝑝 and
we wish to compute QTC, then we merely execute the loop
for j = 1:n
C = 𝑄𝑗 𝐶
end
• The storage of the Household vector 𝑣 1 , … , 𝑣 𝑛 and the corresponding
𝛽𝑗 amounts to a factored form representation of Q.
THE WY Representation
Givens Rotations
• Householder reflections are exceedingly useful for introducing zeroes
on a grand scale e.g. annihilation of all but the first component of a
vector.
• However, in calculations where it is necessary to zero elements more
selectively, Givens rotations are the transformation of choice.
• These are rank-2 corrections to the identity of the form
where c=cos(θ) , s=sin(θ) for some θ. Givens rotations are Orthogonal.
Applying Givens Rotations
• Suppose 𝐴 𝜖 𝑅𝑚×𝑛 , 𝑐 = cos 𝜃 𝑎𝑛𝑑 𝑠 = sin 𝜃
Products of Givens Rotations
• Suppose 𝑄 = 𝐺1 , . . . , 𝐺𝑡 is a product of Givens Rotations.
• As with household reflections, it is more economical to keep
Q in the factored form rather than to compute the product of
rotations explicitly.
• The idea used by Stewart is to associate a single floating point
number ρ with each rotation. Specifically if
Complex Case
Complex Case (cont)
QR Factorization
• A rectangular matrix 𝐴𝑚×𝑛 can be factored into a
product of an orthogonal matrix 𝑄𝑚×𝑚 and an upper
triangular matrix 𝑅𝑚×𝑛
A = QR
• This factorization is referred to as QR factorization.
• It plays a vital role in solving linear least square
problems.
• QR factorization is related to well known GramSchimdt process.
Existence & Properties of QR
Householder QR
• Suppose m=6, n=5, and assume that Householder matrices 𝐻1
and 𝐻2 have been computed so that
• Concentrating on the highlighted entries, we determine a
Household matrix Ĥ3 (4×4) such that
Householder QR
Givens QR method
• Givens rotations can also be used to compute the QR
factorization and the 4 by 3 case illustrates the general idea:
• The two vectors that define Givens rotations are highlighted.
• If 𝐺𝑗 denotes the 𝑗𝑡ℎ Givens rotation in the reduction then
𝑄𝑇 𝐴 = 𝑅 is upper triangular, where 𝑄 = 𝐺1 … 𝐺𝑡 and t is the
total number of rotations.
Hessenberg QR via Givens
• Example depicting how Givens rotations can be employed to
compute the QR factorization of an upper Hessenberg matrix.
• Suppose n=6 and after two steps we have computed:
Classical Gram Schimdt algorithm
• If rank(A) = n, then equation
can be solved for qk
• This leads to the classical Gram-Schimdt Algorithm for
computing A=Q1R1
Modified Gram-Schimdt Algorithm
• Unfortunately the CGS method has very poor numerical properties wherein
there is a severe loss of orthogonality among the computed qi.
• Modified GS leads to a more reliable procedure.
• In the kth step, the kth column of Q (qk) and the kth row of R (rkᵗ) are
determined.
• A matrix A(k) is defined as: