COT4501 Spring 2012
Homework II Solutions
This assignment has eight problems and they are equally weighted. The assignment
is due in class on Tuesday, Tuesday 14, 2012. There are six regular problems and two
computer problems (using MATLAB). For the computer problems, turn in your results
(e.g., graphs, plots, simple analysis and so on) and also a printout of your (MATLAB)
code.
Problem 1 Provide short answers to the following questions:
• True or false: At the solution to a linear least squares problem Ax ' b, the residual
vector r = b − Ax is orthogonal to span(A).
Solution: True. It is an important fact that the residual vector is orthogonal to
span(A).
• True or false: In solving a linear least squares problem Ax ' b, if the vector b lies
in span(A), then the residual is 0.
Solution: True. This means that there is a vector x, such that Ax = b and r =
Ax − b = 0.
• In a linear least squares problem Ax ' b, where A is an m×n matrix, if rank(A) <
n, then which of the following situation are possible?
1. There is no solution.
2. There is a unique solution.
3. There is a solution, but it is not unique.
Solution: There is a solution but it is not unique. The non-uniqueness comes from
the fact that columns of A are linearly dependent.
• in Solving an overdetermined least squares problem Ax ' b,which would be more
serious difficulty: that the rows of A are linearly dependent, or that the columns
of A are linear dependent? Explain.
Solution: If columns of A are (nearly) linear dependent, the matrix A> A becomes
(nearly) singular with a large condition number. This implies that the problem
Ax ' b could be unstable with respect to small perturbation of b. Dependent rows
of A typically do not cause serious problems.
Problem 2 Provide short answers to the following questions:
• Which of the following properties of an m × n matrix A, with m > n, indicate
that the minimum residual solution of the least squares problem Ax ' b is not
unique?
1
1. The columns of A are linearly dependent.
2. The rows of A are linear dependent.
3. The matrix A> A is singular.
Solution: 1 and 3. Note that conditions 1 and 3 are equivalent.
• Which of the following properties does an n × n orthogonal matrix necessarily
have?
1. It is nonsingular.
2. It preserves the Euclidean vector norm when multiplied times a vector.
3. Its transpose is its inverse.
4. Its columns are orthonormal.
5. It is symmetric.
6. It is diagonal.
7. Its Euclidean matrix norm is 1.
8. Its Euclidean condition number is 1.
Solution: All except 5 and 6.
• What condition must a nonzero n-vector w satisfy to ensure that the matrix H =
I − 2ww> is orthogonal?
Solution: We know that H is symmetric. Therefore, if it is orthogonal, H > H =
HH = I. Or
I − 4ww> + 4ww> ww> = I − (4 − w> w)w> = I − 2ww>
√
√
This implies that√w> w = 2, or kwk2 = w> w = 2, i.e., w must be a vector
with magnitude 2.
Problem 3 In fitting a straight line y = x0 + x1 t to the three data points (ti , yi ) =
(0, 0), (1, 0), (1, 1), is the least squares solution unique? Why?
Solution: Put the problem into the standard form of Ax ' b, we have


 
1 0
0
x0



A= 1 1 , x=
, b = 0 .
x1
1 1
1
It is clear that columns of A are linearly independent and hence the solution must be
unique.
Problem 4
2
• What is the Euclidean norm of
linear least squares problem?

1
 0
0
the minimum residual vector for the following

 
1 2
x1
1 
' 1 
x2
0
1
• What is the solution vector x for this problem?
Solution: Notice that the span(A) is the subspace with vectors of the form
 
a

v = b .
0
This immediately implies that the residual vector is [0 0 1]> with Euclidean norm of 1,
and the solution is given by the linear equation
2
1 1
x1
=
,
x2
1
0 1
whose solution is clearly [1 1]> .
Problem 5 Determine the Householder transformation that annihilates all but the first
entry of the vector [1 1 1 1]> . Specifically, if
   
1
α
> 


vv
1   0 

=
(I − 2 > ) 
v v  1   0 
1
0
what are the values of α and v?
Solution: Let a = [1 1 1 1]> and we know that α = ±kak2 = 2. As we have gone over
in class, the formula for v is v = a−αe1 with the sign of α chosen to avoid substraction.
This gives v = a + 2e1 = [3 1 1 1]> .
Problem 6 Suppose that you are computing the QR factorization of the matrix


1 1 1
 1 2 4 

A=
 1 3 9 
1 4 16
by Householder transformations.
3
1. How many Householder transformations are required?
Solution: You need three Householder transformations because there are three
columns.
2. What does the first column of A becomes as a result of applying the first Householder transformation?
Solution: The first column after applying the first Householder transformation is
computed in the previous problem and it is [−2 0 0 0]> .
3. What does the first column then become as a result of applying the second Householder transformation?
Solution: It is still [−2 0 0 0]> because the second Householder transformation
has no effect on the first column.
Computer Problem 1 For n = 0, 1, ..., 5, fit a polynomial of degree n by least squares
to the following data:
t:
y:
0.0
1.0
1.0
2.7
2.0
5.8
3.0
6.6
4.0
7.5
5.0
9.9
Make a plot of the original data points along with each resulting polynomial curve (you
may make separate graphs for each curve or a single graph containing all of the curves).
Which polynomial would you say captures the general trend of the data better? Obviously, this is a subjective questions, and its answer depends on both the nature of the
given data (e.g., the uncertainty of the data values) and the purpose of the fit. Explain
your assumptions in answering.
Solution:
1
2
3
4
5
6
7
8
9
10
11
12
13
function cp03_01 %% polynomial fitting
t = [0; 1; 2; 3; 4; 5];
y = [1; 2.7; 5.8; 6.6; 7.5; 9.9]; plot(t,y,'ko');
hold on; m = size(t,1); A = ones(m,1); p = 51; ts = ...
linspace(0,5,p)';
for n = 1:m
x = A(:,1:n)\y; ys(1:p,n) = x(n);
for k = 2:n
ys(:,n) = ys(:,n).*ts+x(n-k+1);
end
A(:,n+1) = A(:,n).*t;
end;
plot(ts, ys); legend('data points','deg = 0','deg = 1',...
'deg = 2', 'deg = 3', 'deg = 4', 'deg = 5',4)
Computer Problem 2 A common problem in surveying is to determine the altitudes of
a series of points with respect to some reference point. The measurements are subject
4
to error, so more observations are taken than are strictly necessary to determine the
altitudes, and the resulting overdetermined system is solved in the least squares sense to
smooth out errors. Suppose that there are four points whose altitudes x1 , x − 2, x3 , x4
are to be determined. In addition to direct measurements of each xi with respect to
the reference point measurements are also taken of each point with respect to all of the
others. The resulting measurements are
x1 = 2.95,
x3 = −1.45,
x1 − x2 = 1.23,
x1 − x4 = 1.61,
x2 − x4 = 0.45,
x2 = 1.74,
x4 = 1.32,
x1 − x3 = 4.45,
x2 − x3 = 3.21,
x3 − x4 = −2.75.
Set up the corresponding least squares system Ax ' b and use a library routine, or one of
your own design, to solve it for the best values of the altitudes. How do your computed
values compare with the direct measurements? Solution:
1
2
3
4
5
6
function cp03_02 % least squares fit to surveying data
b = [2.95; 1.74; -1.45; 1.32; 1.23; 4.45; 1.61; 3.21; 0.45; -2.75];
A = [1 0 0 0; 0 1 0 0; 0 0 1 0; 0 0 0 1; ...
1 -1 0 0; 1 0 -1 0; 1 0 0 -1; ...
0 1 -1 0; 0 1 0 -1; 0 0 1 -1];
x = A\b
5