Jim Lambers
COS 702
Spring Semeseter 2010-11
Lecture 1 Notes
Discrete Least Squares Approximations
One of the most fundamental problems in science and engineering is data fitting–constructing a
function that, in some sense, conforms to given data points. Two such data-fitting techniques are
polynomial interpolation and piecewise polynomial interpolation. Interpolation techniques, of any
kind, construct functions that agree exactly with the data. That is, given points (𝑥1 , 𝑦1 ), (𝑥2 , 𝑦2 ),
. . ., (𝑥𝑚 , 𝑦𝑚 ), interpolation yields a function 𝑓 (𝑥) such that 𝑓 (𝑥𝑖 ) = 𝑦𝑖 for 𝑖 = 1, 2, . . . , 𝑚.
However, fitting the data exactly may not be the best approach to describing the data with a
function. We have seen that high-degree polynomial interpolation can yield oscillatory functions
that behave very differently than a smooth function from which the data is obtained. Also, it
may be pointless to try to fit data exactly, for if it is obtained by previous measurements or other
computations, it may be erroneous. Therefore, we consider revising our notion of what constitutes
a “best fit” of given data by a function.
One alternative approach to data fitting is to solve the minimax problem, which is the problem
of finding a function 𝑓 (𝑥) of a given form for which
max ∣𝑓 (𝑥𝑖 ) − 𝑦𝑖 ∣
1≤𝑖≤𝑛
is minimized. However, this is a very difficult problem to solve.
Another approach is to minimize the total absolute deviation of 𝑓 (𝑥) from the data. That is,
we seek a function 𝑓 (𝑥) of a given form for which
𝑚
∑
∣𝑓 (𝑥𝑖 ) − 𝑦𝑖 ∣
𝑖=1
is minimized. However, we cannot apply standard minimization techniques to this function, because, like the absolute value function that it employs, it is not differentiable.
This defect is overcome by considering the problem of finding 𝑓 (𝑥) of a given form for which
𝑚
∑
[𝑓 (𝑥𝑖 ) − 𝑦𝑖 ]2
𝑖=1
is minimized. This is known as the least squares problem. We will first show how this problem is
solved for the case where 𝑓 (𝑥) is a linear function of the form 𝑓 (𝑥) = 𝑎1 𝑥 + 𝑎0 , and then generalize
this solution to other types of functions.
1
When 𝑓 (𝑥) is linear, the least squares problem is the problem of finding constants 𝑎0 and 𝑎1
such that the function
𝑚
∑
𝐸(𝑎0 , 𝑎1 ) =
(𝑎1 𝑥𝑖 + 𝑎0 − 𝑦𝑖 )2
𝑖=1
is minimized. In order to minimize this function of 𝑎0 and 𝑎1 , we must compute its partial derivatives
with respect to 𝑎0 and 𝑎1 . This yields
𝑚
𝑚
∑
∂𝐸
=
2(𝑎1 𝑥𝑖 + 𝑎0 − 𝑦𝑖 ),
∂𝑎0
∑
∂𝐸
=
2(𝑎1 𝑥𝑖 + 𝑎0 − 𝑦𝑖 )𝑥𝑖 .
∂𝑎1
𝑖=1
𝑖=1
At a minimum, both of these partial derivatives must be equal to zero. This yields the system of
linear equations
(𝑚 )
𝑚
∑
∑
𝑥𝑖 𝑎1 =
𝑦𝑖 ,
𝑚𝑎0 +
𝑖=1
(
𝑚
∑
)
𝑥𝑖
𝑎0 +
𝑖=1
(𝑚
∑
)
𝑥2𝑖
𝑖=1
𝑎1 =
𝑖=1
𝑚
∑
𝑥𝑖 𝑦𝑖 .
𝑖=1
These equations are called the normal equations.
Using the formula for the inverse of a 2 × 2 matrix,
]
[
]−1
[
1
𝑑 −𝑏
𝑎 𝑏
,
=
𝑎
𝑐 𝑑
𝑎𝑑 − 𝑏𝑐 −𝑐
we obtain the solutions
) ∑𝑚
∑𝑚
∑
( 𝑖=1 𝑦𝑖 ) − ( 𝑚
𝑖=1 𝑥𝑖 ) ( 𝑖=1 𝑥𝑖 𝑦𝑖 )
,
∑
∑𝑚
2
2
𝑚 𝑚
𝑖=1 𝑥𝑖 − ( 𝑖=1 𝑥𝑖 )
∑𝑚
∑𝑚
∑
𝑚 𝑚
𝑖=1 𝑥𝑖 𝑦𝑖 − ( 𝑖=1 𝑥𝑖 ) ( 𝑖=1 𝑦𝑖 )
.
∑𝑚
∑
2
2
𝑚 𝑚
𝑖=1 𝑥𝑖 − ( 𝑖=1 𝑥𝑖 )
(∑𝑚
𝑎0 =
𝑎1 =
2
𝑖=1 𝑥𝑖
Example We wish to find the linear function 𝑦 = 𝑎1 𝑥 + 𝑎0 that best approximates the data shown
in Table 1, in the least-squares sense. Using the summations
𝑚
∑
𝑥𝑖 = 56.2933,
𝑖=1
𝑚
∑
𝑚
∑
𝑥2𝑖 = 380.5426,
𝑖=1
𝑖=1
𝑦𝑖 = 73.8373,
𝑚
∑
𝑥𝑖 𝑦𝑖 = 485.9487,
𝑖=1
we obtain
𝑎0 =
𝑎1 =
380.5426 ⋅ 73.8373 − 56.2933 ⋅ 485.9487
742.5703
=
= 1.1667,
2
10 ⋅ 380.5426 − 56.2933
636.4906
10 ⋅ 485.9487 − 56.2933 ⋅ 73.8373
702.9438
=
= 1.1044.
2
10 ⋅ 380.5426 − 56.2933
636.4906
2
𝑖
1
2
3
4
5
6
7
8
9
10
𝑥𝑖
2.0774
2.3049
3.0125
4.7092
5.5016
5.8704
6.2248
8.4431
8.7594
9.3900
𝑦𝑖
3.3123
3.8982
4.6500
6.5576
7.5173
7.0415
7.7497
11.0451
9.8179
12.2477
Table 1: Data points (𝑥𝑖 , 𝑦𝑖 ), for 𝑖 = 1, 2, . . . , 10, to be fit by a linear function
We conclude that the linear function that best fits this data in the least-squares sense is
𝑦 = 1.1044𝑥 + 1.1667.
The data, and this function, are shown in Figure 1. □
It is interesting to note that if we define the 𝑚 × 2 matrix 𝐴, the 2-vector a, and the 𝑚-vector
y by
⎡
⎤
⎡
⎤
1 𝑥1
𝑦1
[
]
⎢ 1 𝑥2 ⎥
⎢ 𝑦2 ⎥
𝑎0
⎢
⎥
⎢
⎥
𝐴=⎢ .
, a=
, y = ⎢ . ⎥,
⎥
.
.
.
.
𝑎
⎣ .
⎣ . ⎦
1
. ⎦
1 𝑥𝑚
𝑦𝑚
then a is the solution to the system of equations
𝐴𝑇 𝐴a = 𝐴𝑇 y.
These equations are the normal equations defined earlier, written in matrix-vector form. They arise
from the problem of finding the vector a such that
∣𝐴a − y∣
is minimized, where, for any vector u, ∣u∣ is the magnitude, or length, of u.
In this case, this expression is equivalent to the square root of the expression we originally
intended to minimize,
𝑚
∑
(𝑎1 𝑥𝑖 + 𝑎0 − 𝑦𝑖 )2 ,
𝑖=1
3
Figure 1: Data points (𝑥𝑖 , 𝑦𝑖 ) (circles) and least-squares line (solid line)
but the normal equations also characterize the solution a, an 𝑛-vector, to the more general linear
least squares problem of minimizing ∣𝐴a − y∣ for any matrix 𝐴 that is 𝑚 × 𝑛, where 𝑚 ≥ 𝑛, whose
columns are linearly independent.
We now consider the problem of finding a polynomial of degree 𝑛 that gives the best leastsquares fit. As before, let (𝑥1 , 𝑦1 ), (𝑥2 , 𝑦2 ), . . ., (𝑥𝑚 , 𝑦𝑚 ) be given data points that need to be
approximated by a polynomial of degree 𝑛. We assume that 𝑛 < 𝑚 − 1, for otherwise, we can use
polynomial interpolation to fit the points exactly.
Let the least-squares polynomial have the form
𝑝𝑛 (𝑥) =
𝑛
∑
𝑎𝑗 𝑥𝑗 .
𝑗=0
Our goal is to minimize the sum of squares of the deviations in 𝑝𝑛 (𝑥) from each 𝑦-value,
⎡
⎤2
𝑚
𝑚
𝑛
∑
∑
∑
⎣
𝐸(a) =
[𝑝𝑛 (𝑥𝑖 ) − 𝑦𝑖 ]2 =
𝑎𝑗 𝑥𝑗𝑖 − 𝑦𝑖 ⎦ ,
𝑖=1
𝑖=1
4
𝑗=0
where a is a column vector of the unknown coefficients of 𝑝𝑛 (𝑥),
⎤
⎡
𝑎0
⎢ 𝑎1 ⎥
⎥
⎢
a = ⎢ . ⎥.
⎣ .. ⎦
𝑎𝑛
Differentiating this function with respect to each 𝑎𝑘 yields
⎡
⎤
𝑚
𝑛
∑
∑
∂𝐸
=
2⎣
𝑎𝑗 𝑥𝑗𝑖 − 𝑦𝑖 ⎦ 𝑥𝑘𝑖 , 𝑘 = 0, 1, . . . , 𝑛.
∂𝑎𝑘
𝑖=1
𝑗=0
Setting each of these partial derivatives equal to zero yields the system of equations
(𝑚
)
𝑛
𝑚
∑
∑ 𝑗+𝑘
∑
𝑥𝑖
𝑎𝑗 =
𝑥𝑘𝑖 𝑦𝑖 , 𝑘 = 0, 1, . . . , 𝑛.
𝑗=0
𝑖=1
𝑖=1
These are the normal equations. They are a generalization of the normal equations previously
defined for the linear case, where 𝑛 = 1. Solving this system yields the coefficients {𝑎𝑗 }𝑛𝑗=0 of the
least-squares polynomial 𝑝𝑛 (𝑥).
As in the linear case, the normal equations can be written in matrix-vector form
𝐴𝑇 𝐴a = 𝐴𝑇 y,
where
⎡
⎢
⎢
⎢
𝐴=⎢
⎢
⎣
1
1
1
..
.
𝑥0
𝑥1
𝑥2
..
.
𝑥20
𝑥21
𝑥22
..
.
⋅⋅⋅
⋅⋅⋅
⋅⋅⋅
..
.
𝑥𝑛0
𝑥𝑛1
𝑥𝑛2
..
.
1 𝑥𝑚 𝑥2𝑚 ⋅ ⋅ ⋅ 𝑥𝑛𝑚
⎤
⎥
⎥
⎥
⎥,
⎥
⎦
⎡
⎢
⎢
a=⎢
⎣
𝑎0
𝑎1
..
.
𝑎𝑛
⎤
⎥
⎥
⎥,
⎦
⎡
⎢
⎢
y=⎢
⎣
𝑦1
𝑦2
..
.
⎤
⎥
⎥
⎥.
⎦
𝑦𝑛
The normal equations equations can be used to compute the coefficients of any linear combination
of functions {𝜙𝑗 (𝑥)}𝑛𝑗=0 that best fits data in the least-squares sense, provided that these functions
are linearly independent. In this general case, the entries of the matrix 𝐴 are given by 𝑎𝑖𝑗 = 𝜙𝑖 (𝑥𝑗 ),
for 𝑖 = 1, 2, . . . , 𝑚 and 𝑗 = 0, 1, . . . , 𝑛.
Example We wish to find the quadratic function 𝑦 = 𝑎2 𝑥2 + 𝑎1 𝑥 + 𝑎0 that best approximates the
data shown in Table 2, in the least-squares sense. By defining
⎡
⎤
⎡
⎤
1 𝑥1 𝑥21
𝑦1
⎡
⎤
𝑎0
⎢ 1 𝑥2 𝑥2 ⎥
⎢ 𝑦2 ⎥
2 ⎥
⎢
⎢
⎥
⎣
⎦
𝑎
𝐴=⎢ .
,
a
=
,
y
=
⎢ .. ⎥ ,
⎥
..
..
1
⎣ ..
⎦
⎣
. ⎦
.
.
𝑎2
2
1 𝑥10 𝑥10
𝑦10
5
𝑖
1
2
3
4
5
6
7
8
9
10
𝑥𝑖
2.0774
2.3049
3.0125
4.7092
5.5016
5.8704
6.2248
8.4431
8.7594
9.3900
𝑦𝑖
2.7212
3.7798
4.8774
6.6596
10.5966
9.8786
10.5232
23.3574
24.0510
27.4827
Table 2: Data points (𝑥𝑖 , 𝑦𝑖 ), for 𝑖 = 1, 2, . . . , 10, to be fit by a quadratic function
and solving the normal equations
𝐴𝑇 𝐴a = 𝐴𝑇 y,
we obtain the coefficients
𝑐0 = 4.7681,
𝑐1 = −1.5193,
𝑐2 = 0.4251,
and conclude that the quadratic function that best fits this data in the least-squares sense is
𝑦 = 0.4251𝑥2 − 1.5193𝑥 + 4.7681.
The data, and this function, are shown in Figure 2. □
Least-squares fitting can also be used to fit data with functions that are not linear combinations
of functions such as polynomials. Suppose we believe that given data points can best be matched
to an exponential function of the form 𝑦 = 𝑏𝑒𝑎𝑥 , where the constants 𝑎 and 𝑏 are unknown. Taking
the natural logarithm of both sides of this equation yields
ln 𝑦 = ln 𝑏 + 𝑎𝑥.
If we define 𝑧 = ln 𝑦 and 𝑐 = ln 𝑏, then the problem of fitting the original data points {(𝑥𝑖 , 𝑦𝑖 )}𝑚
𝑖=1
with an exponential function is transformed into the problem of fitting the data points {(𝑥𝑖 , 𝑧𝑖 )}𝑚
𝑖=1
with a linear function of the form 𝑐 + 𝑎𝑥, for unknown constants 𝑎 and 𝑐.
Similarly, suppose the given data is believed to approximately conform to a function of the form
𝑦 = 𝑏𝑥𝑎 , where the constants 𝑎 and 𝑏 are unknown. Taking the natural logarithm of both sides of
this equation yields
ln 𝑦 = ln 𝑏 + 𝑎 ln 𝑥.
6
Figure 2: Data points (𝑥𝑖 , 𝑦𝑖 ) (circles) and quadratic least-squares fit (solid curve)
If we define 𝑧 = ln 𝑦, 𝑐 = ln 𝑏 and 𝑤 = ln 𝑥, then the problem of fitting the original data points
{(𝑥𝑖 , 𝑦𝑖 )}𝑚
𝑖=1 with a constant times a power of 𝑥 is transformed into the problem of fitting the data
points {(𝑤𝑖 , 𝑧𝑖 )}𝑚
𝑖=1 with a linear function of the form 𝑐 + 𝑎𝑤, for unknown constants 𝑎 and 𝑐.
Example We wish to find the exponential function 𝑦 = 𝑏𝑒𝑎𝑥 that best approximates the data
shown in Table 3, in the least-squares sense. By defining
⎡
⎤
⎡
⎤
1 𝑥1
𝑧1
[ ]
⎢ 1 𝑥2
⎥
⎢ 𝑧2 ⎥
𝑐
⎢
⎥
⎢
⎥
𝐴 = ⎢ . . . ⎥, c =
, z = ⎢ . ⎥,
𝑎
⎣ .. .. .. ⎦
⎣ .. ⎦
1 𝑥5
𝑧5
where 𝑐 = ln 𝑏 and 𝑧𝑖 = ln 𝑦𝑖 for 𝑖 = 1, 2, . . . , 5, and solving the normal equations
𝐴𝑇 𝐴c = 𝐴𝑇 z,
we obtain the coefficients
𝑎 = 0.4040,
𝑏 = 𝑒𝑐 = 𝑒−0.2652 = 0.7670,
7
𝑖
1
2
3
4
5
𝑥𝑖
2.0774
2.3049
3.0125
4.7092
5.5016
𝑦𝑖
1.4509
2.8462
2.1536
4.7438
7.7260
Table 3: Data points (𝑥𝑖 , 𝑦𝑖 ), for 𝑖 = 1, 2, . . . , 5, to be fit by an exponential function
and conclude that the exponential function that best fits this data in the least-squares sense is
𝑦 = 0.7670𝑒0.4040𝑥 .
□
8