Unit 3: Solution of Overdetermined Linear System
of Equations
(Curve Fitting)
In this section, we consider linear systems of the form
Ax = b with A ∈ R
m×n
with m >> n.
Führer: FMA421 2010/11
38
Curve Fitting Example
We would like to fit the data
i
ti
yi
0
-1.0
1.0
1
-0.5
0.5
2
0.0
0.0
3
0.5
0.5
4
1.0
2.0
by a quadratic polynomial of the form
2
p(t) = a2t + a1t + a0.
Führer: FMA421 2010/11
39
Curve Fitting Example (Cont.)
We set up interpolation conditions p(ti) = yi:

1
1


1

1
1
t0
t1
t2
t3
t4

t20
 
t21
 a0

t22 a1

t23 a2
t24


y0
y 
 1
 
= y2 .
 
y3
y4
This leads to a nonsquare linear system.
Führer: FMA421 2010/11
40
Least Squares Solution
What do we mean when we look for a solution of a linear system without a solution?
Find an x̂ with
kAx̂ − bk2 = min kAx − bk2 = min kr(x)k2
x
x
with the residual vectors r(x) := b − Ax.
Führer: FMA421 2010/11
41
Length of a vector
By kvk2 we mean the (Euclidean) length of a vector:
T
kvk2 = (v v)
1/2
=
n
X
2 1/2
.
vi
i=1
(called also the 2-norm of v , see help norm )
Führer: FMA421 2010/11
42
Is there a unique solution?
Problem:
Find an x̂ with
kAx̂ − bk2 = min kAx − bk2 = min kr(x)k2
x
x
A necessary condition:
d
2
kr(x)k2
dx
x=x̂
= 0.
Führer: FMA421 2010/11
43
Is there a unique solution? (Cont.)
From
2
T
T
T
T
T
T
T
kr(x)k2 = r r = (b − Ax) (b − Ax) = b b − 2x A b + x A Ax
we take the first derivative w.r.t. x.
Flerdim-course:
T
T
A Ax̂ − A b = 0
Führer: FMA421 2010/11
44
Normal Equations
T
T
A Ax̂ − A b = 0
We can see them as a linear system
Ax = b̄
where b̄ is the orthogonal projection of b onto the range space of A (bildrummet,
kolonnrummet).
Führer: FMA421 2010/11
45
Normal Equations (Cont.)
Consider the range space R(A). It is spanned by the columns of A.
T
T
A (b − Ax) = A r = 0
Thus r⊥A. This justifies the name ”normal” equations.
r
b
A x
Im(A)
Führer: FMA421 2010/11
46
In MATLAB
Alternative 1: x=(A0 ∗ A)\ A0*b
Alternative 2: x=A\ b
there are even more stable methods.
Führer: FMA421 2010/11
47
Oat Porridge Problem
Führer: FMA421 2010/11
48
Oat Porridge Problem (Cont.)
We pose the following questions:
• How much water is needed for three portions?
• How much water is needed for 300 portions?
To answer these questions, we first have to set up a correct mathematical model and then
make a least squares fit to determine the unknown parameters in the model.
Führer: FMA421 2010/11
49
Oat Porridge Problem (Cont.)
The discussion of the different models will take place in the lecture - not here.
Führer: FMA421 2010/11
50