Jim Lambers
MAT 772
Fall Semester 2010-11
Lecture 12 Notes
These notes correspond to Sections 9.2 and 9.3 in the text.
Best Approximation in the 2-norm
Suppose that we wish to obtain a function 𝑓𝑛 (𝑥) that is a linear combination of given functions
{𝜙𝑗 (𝑥)}𝑛𝑗=0 , and best fits a function 𝑓 (𝑥) at a discrete set of data points {(𝑥𝑖 , 𝑓 (𝑥𝑖 ))}𝑚
𝑖=1 in a
least-squares sense. That is, we wish to find constants {𝑐𝑗 }𝑛𝑗=0 such that
⎡
⎤2
𝑚
𝑚
𝑛
∑
∑
∑
⎣
[𝑓𝑛 (𝑥𝑖 ) − 𝑓 (𝑥𝑖 )]2 =
𝑐𝑗 𝜙𝑗 (𝑥𝑖 ) − 𝑓 (𝑥𝑖 )⎦
𝑖=1
𝑖=1
𝑗=0
is minimized. This can be accomplished by solving a system of 𝑛 + 1 linear equations for the {𝑐𝑗 },
known as the normal equations.
Now, suppose we have a continuous set of data. That is, we have a function 𝑓 (𝑥) defined on
an interval [𝑎, 𝑏], and we wish to approximate it as closely as possible, in some sense, by a function
𝑓𝑛 (𝑥) that is a linear combination of given functions {𝜙𝑗 (𝑥)}𝑛𝑗=0 . If we choose 𝑚 equally spaced
points {𝑥𝑖 }𝑚
𝑖=1 in [𝑎, 𝑏], and let 𝑚 → ∞, we obtain the continuous least-squares problem of finding
the function
𝑛
∑
𝑓𝑛 (𝑥) =
𝑐𝑗 𝜙𝑗 (𝑥)
𝑗=0
that minimizes
∫
𝐸(𝑐0 , 𝑐1 , . . . , 𝑐𝑛 ) =
𝑏
[𝑓𝑛 (𝑥) − 𝑓 (𝑥)]2 𝑑𝑥 =
𝑎
∫
𝑎
𝑏
⎡
⎤2
𝑛
∑
⎣
𝑐𝑗 𝜙𝑗 (𝑥) − 𝑓 (𝑥)⎦ 𝑑𝑥.
𝑗=0
To obtain the coefficients {𝑐𝑗 }𝑛𝑗=0 , we can proceed as in the discrete case. We compute the
partial derivatives of 𝐸(𝑐0 , 𝑐1 , . . . , 𝑐𝑛 ) with respect to each 𝑐𝑘 and obtain
⎡
⎤
∫ 𝑏
𝑛
∑
∂𝐸
=
𝜙𝑘 (𝑥) ⎣
𝑐𝑗 𝜙𝑗 (𝑥) − 𝑓 (𝑥)⎦ 𝑑𝑥,
∂𝑐𝑘
𝑎
𝑗=0
and requiring that each partial derivative be equal to zero yields the normal equations
]
∫ 𝑏
𝑛 [∫ 𝑏
∑
𝜙𝑘 (𝑥)𝜙𝑗 (𝑥) 𝑑𝑥 𝑐𝑗 =
𝜙𝑘 (𝑥)𝑓 (𝑥) 𝑑𝑥, 𝑘 = 0, 1, . . . , 𝑛.
𝑗=0
𝑎
𝑎
1
We can then solve this system of equations to obtain the coefficients {𝑐𝑗 }𝑛𝑗=0 . This system can
be solved as long as the functions {𝜙𝑗 (𝑥)}𝑛𝑗=0 are linearly independent. That is, the condition
𝑛
∑
𝑐𝑗 𝜙𝑗 (𝑥) ≡ 0,
𝑥 ∈ [𝑎, 𝑏],
𝑗=0
is only true if 𝑐0 = 𝑐1 = ⋅ ⋅ ⋅ = 𝑐𝑛 = 0. In particular, this is the case if, for 𝑗 = 0, 1, . . . , 𝑛, 𝜙𝑗 (𝑥) is a
polynomial of degree 𝑗. This can be proved using a simple inductive argument.
Example We approximate 𝑓 (𝑥) = 𝑒𝑥 on the interval [0, 5] by a fourth-degree polynomial
𝑓4 (𝑥) = 𝑐0 + 𝑐1 𝑥 + 𝑐2 𝑥2 + 𝑐3 𝑥3 + 𝑐4 𝑥4 .
The normal equations have the form
𝑛
∑
𝑎𝑖𝑗 𝑐𝑗 = 𝑏𝑖 ,
𝑖 = 0, 1, . . . , 4,
𝑗=0
or, in matrix-vector form, 𝐴c = b, where
∫ 5
∫ 5
5𝑖+𝑗+1
𝑖 𝑗
, 𝑖, 𝑗 = 0, 1, . . . , 4,
𝑎𝑖𝑗 =
𝑥 𝑥 𝑑𝑥 =
𝑥𝑖+𝑗 𝑑𝑥 =
𝑖+𝑗+1
0
0
∫ 5
𝑥𝑖 𝑒𝑥 𝑑𝑥, 𝑖 = 0, 1, . . . , 4.
𝑏𝑖 =
0
Integration by parts yields the relation
𝑏𝑖 = 5𝑖 𝑒5 − 𝑖𝑏𝑖−1 ,
𝑏0 = 𝑒5 − 1.
Solving this system of equations yields the polynomial
𝑓4 (𝑥) = 2.3002 − 6.226𝑥 + 9.5487𝑥2 − 3.86𝑥3 + 0.6704𝑥4 .
As Figure 1 shows, this polynomial is barely distinguishable from 𝑒𝑥 on [0, 5].
However, it should be noted that the matrix 𝐴 is closely related to the 𝑛 × 𝑛 Hilbert matrix 𝐻𝑛 ,
which has entries
1
[𝐻𝑛 ]𝑖𝑗 =
, 1 ≤ 𝑖, 𝑗 ≤ 𝑛.
𝑖+𝑗−1
This matrix is famous for being highly ill-conditioned, meaning that solutions to systems of linear
equations involving this matrix that are computed using floating-point arithmetic are highly sensitive to roundoff error. In fact, the matrix 𝐴 in this example has a condition number of 1.56 × 107 ,
which means that a change of size 𝜖 in the right-hand side vector b, with entries 𝑏𝑖 , can cause a
change of size 1.56𝜖 × 107 in the solution c. □
2
Figure 1: Graphs of 𝑓 (𝑥) = 𝑒𝑥 (red dashed curve) and 4th-degree continuous least-squares polynomial approximation 𝑓4 (𝑥) on [0, 5] (blue solid curve)
Inner Product Spaces
As the preceding example shows, it is important to choose the functions {𝜙𝑗 (𝑥)}𝑛𝑗=0 wisely, so that
the resulting system of normal equations is not unduly sensitive to round-off errors. An even better
choice is one for which this system can be solved analytically, with relatively few computations. An
ideal choice of functions is one for which the task of computing 𝑓𝑛+1 (𝑥) can reuse the computations
needed to compute 𝑓𝑛 (𝑥).
To that end, recall that two 𝑚-vectors u = ⟨𝑢1 , 𝑢2 , . . . , 𝑢𝑚 ⟩ and v = ⟨𝑣1 , 𝑣2 , . . . , 𝑣𝑚 ⟩ are orthogonal if
𝑚
∑
u⋅v =
𝑢𝑖 𝑣𝑖 = 0,
𝑖=1
where u ⋅ v is the dot product, or inner product, of u and v.
By viewing functions defined on an interval [𝑎, 𝑏] as infinitely long vectors, we can generalize
the inner product, and the concept of orthogonality, to functions. To that end, we define the inner
product of two real-valued functions 𝑓 (𝑥) and 𝑔(𝑥) defined on the interval [𝑎, 𝑏] by
∫
⟨𝑓, 𝑔⟩ =
𝑏
𝑓 (𝑥)𝑔(𝑥) 𝑑𝑥.
𝑎
3
Then, we say 𝑓 and 𝑔 are orthogonal with respect to this inner product if ⟨𝑓, 𝑔⟩ = 0.
In general, an inner product on a vector space 𝒱 over ℝ, be it continuous or discrete, has the
following properties:
1. ⟨𝑓 + 𝑔, ℎ⟩ = ⟨𝑓, ℎ⟩ + ⟨𝑔, ℎ⟩ for all 𝑓, 𝑔, ℎ ∈ 𝒱
2. ⟨𝑐𝑓, 𝑔⟩ = 𝑐⟨𝑓, 𝑔⟩ for all 𝑐 ∈ ℝ and all 𝑓 ∈ 𝒱
3. ⟨𝑓, 𝑔⟩ = ⟨𝑔, 𝑓 ⟩ for all 𝑓, 𝑔 ∈ 𝒱
4. ⟨𝑓, 𝑓 ⟩ ≥ 0 for all 𝑓 ∈ 𝒱, and ⟨𝑓, 𝑓 ⟩ = 0 if and only if 𝑓 = 0.
This inner product can be used to define the norm of a function, which generalizes the concept
of the magnitude of a vector to functions, and therefore provides a measure of the “magnitude” of
a function. Recall that the magnitude of a vector v, denoted by ∥v∥, can be defined by
∥v∥ = (v ⋅ v)1/2 .
Along similar lines, we define the 2-norm of a function 𝑓 (𝑥) defined on [𝑎, 𝑏] by
∥𝑓 ∥2 = (⟨𝑓, 𝑓 ⟩)
1/2
(∫
=
𝑏
2
)1/2
[𝑓 (𝑥)] 𝑑𝑥
.
𝑎
As we will see, it can be verified that this function does in fact satisfy the properties required of a
norm. The continuous least-squares problem can then be described as the problem of finding
𝑓𝑛 (𝑥) =
𝑛
∑
𝑐𝑗 𝜙𝑗 (𝑥)
𝑗=0
such that
(∫
∥𝑓𝑛 − 𝑓 ∥2 =
𝑏
)1/2
[𝑓𝑛 (𝑥) − 𝑓 (𝑥)] 𝑑𝑥
2
𝑎
is minimized. This minimization can be performed over 𝐶[𝑎, 𝑏], the space of functions that are
continuous on [𝑎, 𝑏], but it is not necessary for a function 𝑓 (𝑥) to be continuous for ∥𝑓 ∥2 to be
defined. Rather, we consider the space 𝐿2 (𝑎, 𝑏), the space of real-valued functions such that ∣𝑓 (𝑥)∣2
is integrable over (𝑎, 𝑏).
One very important property that ∥ ⋅ ∥2 has is that it satisfies the Cauchy-Schwarz inequality
∣⟨𝑓, 𝑔⟩∣ ≤ ∥𝑓 ∥2 ∥𝑔∥2 ,
𝑓, 𝑔 ∈ 𝒱.
This can be proven by noting that for any scalar 𝑐 ∈ ℝ,
𝑐2 ∥𝑓 ∥22 + 2𝑐⟨𝑓, 𝑔⟩ + ∥𝑔∥22 = ∥𝑐𝑓 + 𝑔∥22 ≥ 0.
4
The left side is a quadratic polynomial in 𝑐. In order for this polynomial to not have any negative
values, it must either have complex roots or a double real root. This is the case if the discrimant
satisfies
4⟨𝑓, 𝑔⟩2 − 4∥𝑓 ∥22 ∥𝑔∥22 ≤ 0,
from which the Cauchy-Schwarz inequality immediately follows. By setting 𝑐 = 1 and applying this
inequality, we immediately obtain the triangle-inequality property of norms.
Suppose that we can construct a set of functions {𝜙𝑗 (𝑥)}𝑛𝑗=0 that is orthogonal with respect to
the inner product of functions on [𝑎, 𝑏]. That is,
∫
⟨𝜙𝑘 , 𝜙𝑗 ⟩ =
𝑏
{
𝜙𝑘 (𝑥)𝜙𝑗 (𝑥) 𝑑𝑥 =
𝑎
0
𝑘 ∕= 𝑗
.
𝛼𝑘 > 0 𝑘 = 𝑗
Then, the normal equations simplify to a trivial system
[∫ 𝑏
]
∫ 𝑏
2
[𝜙𝑘 (𝑥)] 𝑑𝑥 𝑐𝑘 =
𝜙𝑘 (𝑥)𝑓 (𝑥) 𝑑𝑥,
𝑎
𝑘 = 0, 1, . . . , 𝑛,
𝑎
or, in terms of norms and inner products,
∥𝜙𝑘 ∥22 𝑐𝑘 = ⟨𝜙𝑘 , 𝑓 ⟩,
𝑘 = 0, 1, . . . , 𝑛.
It follows that the coefficients {𝑐𝑗 }𝑛𝑗=0 of the least-squares approximation 𝑓𝑛 (𝑥) are simply
𝑐𝑘 =
⟨𝜙𝑘 , 𝑓 ⟩
,
∥𝜙𝑘 ∥22
𝑘 = 0, 1, . . . , 𝑛.
If the constants {𝛼𝑘 }𝑛𝑘=0 above satisfy 𝛼𝑘 = 1 for 𝑘 = 0, 1, . . . , 𝑛, then we say that the orthogonal
set of functions {𝜙𝑗 (𝑥)}𝑛𝑗=0 is orthonormal. In that case, the solution to the continuous least-squares
problem is simply given by
𝑐𝑘 = ⟨𝜙𝑘 , 𝑓 ⟩, 𝑘 = 0, 1, . . . , 𝑛.
Next, we will learn how sets of orthogonal polynomials can be computed.
5