Jim Lambers
MAT 460/560
Fall Semeseter 2009-10
Lecture 37 Notes
These notes correspond to Section 8.2 in the text.
Orthogonal Polynomials and Least Squares Approximations, cont’d
Previously, we learned that the problem of finding the polynomial 𝑓𝑛 (𝑥), of degree 𝑛, that best
approximates a function 𝑓 (𝑥) on an interval [𝑎, 𝑏] in the least squares sense, i.e., that minimizes
(∫
∥𝑓𝑛 − 𝑓 ∥ =
)1/2
𝑏
2
[𝑓𝑛 (𝑥) − 𝑓 (𝑥)] 𝑑𝑥
,
𝑎
is easy to solve if we represent 𝑓𝑛 (𝑥) as a linear combination of orthogonal polynomials,
𝑓𝑛 (𝑥) =
𝑛
∑
𝑐𝑗 𝑝𝑗 (𝑥).
𝑗=0
Each polynomial 𝑝𝑗 (𝑥) is of degree 𝑗, and the set of polynomials 𝑝0 (𝑥), 𝑝1 (𝑥), . . . , 𝑝𝑛 (𝑥) are orthogonal with respect to the inner product
∫ 𝑏
⟨𝑓, 𝑔⟩ =
𝑓 (𝑥)𝑔(𝑥) 𝑑𝑥.
𝑎
That is,
∫
⟨𝑝𝑘 , 𝑝𝑗 ⟩ =
𝑏
𝑝𝑘 (𝑥)𝑝𝑗 (𝑥) 𝑑𝑥 = 0,
𝑘 ∕= 𝑗.
𝑎
Given this sequence of orthogonal polynomials, the coefficients 𝑐𝑗 in the linear combination used
to compute 𝑓𝑛 (𝑥) are given by
𝑐𝑗 =
⟨𝑝𝑗 , 𝑓 ⟩
,
⟨𝑝𝑗 , 𝑝𝑗 ⟩
𝑐𝑗 = 0, 1, . . . , 𝑛.
Now, we focus on the task of finding such a sequence of orthogonal polynomials.
Recall the process known as Gram-Schmidt orthogonalization for obtaining a set of orthogonal
vectors p1 , p2 , . . . , p𝑛 from a set of linearly independent vectors a1 , a2 , . . . , a𝑛 :
p1 = a 1
p2 = a 2 −
p1 ⋅ a2
p1
p1 ⋅ p1
1
..
.
p𝑛 = a 𝑛 −
𝑛−1
∑
𝑗=0
p𝑗 ⋅ a 𝑛
p𝑗 .
p𝑗 ⋅ p𝑗
By normalizing each vector p𝑗 , we obtain a unit vector
q𝑗 =
1
p𝑗 ,
∣p𝑗 ∣
and a set of orthonormal vectors {q𝑗 }𝑛𝑗=1 , in that they are orthogonal (q𝑘 ⋅ q𝑗 = 0 for 𝑘 ∕= 𝑗), and
unit vectors (q𝑗 ⋅ q𝑗 = 1).
We can use a similar process to compute a set of orthogonal polynomials. For simplicitly, we will
require that all polynomials in the set be monic; that is, their leading (highest-degree) coefficient
must be equal 1. We then define 𝑝0 (𝑥) = 1. Then, because 𝑝1 (𝑥) is supposed to be of degree 1, it
must have the form 𝑝1 (𝑥) = 𝑥 − 𝛼1 for some constant 𝛼1 . To ensure that 𝑝1 (𝑥) is orthogonal to
𝑝0 (𝑥), we compute their inner product, and obtain
0 = ⟨𝑝0 , 𝑝1 ⟩ = ⟨1, 𝑥 − 𝛼1 ⟩,
so we must have
𝛼1 =
⟨1, 𝑥⟩
.
⟨1, 1⟩
For 𝑗 > 1, we start by setting 𝑝𝑗 (𝑥) = 𝑥𝑝𝑗−1 (𝑥), since 𝑝𝑗 should be of degree one greater
than that of 𝑝𝑗−1 , and this satisfies the requirement that 𝑝𝑗 be monic. Then, we need to subtract
polynomials of lower degree to ensure that 𝑝𝑗 is orthogonal to 𝑝𝑖 , for 𝑖 < 𝑗. To that end, we apply
Gram-Schmidt orthogonalization and obtain
𝑝𝑗 (𝑥) = 𝑥𝑝𝑗−1 (𝑥) −
𝑗−1
∑
⟨𝑝𝑖 , 𝑥𝑝𝑗−1 ⟩
𝑖=0
⟨𝑝𝑖 , 𝑝𝑖 ⟩
𝑝𝑖 (𝑥).
However, by the definition of the inner product, ⟨𝑝𝑖 , 𝑥𝑝𝑗−1 ⟩ = ⟨𝑥𝑝𝑖 , 𝑝𝑗−1 ⟩. Furthermore, because
𝑥𝑝𝑖 is of degree 𝑖 + 1, and 𝑝𝑗−1 is orthogonal to all polynomials of degree less than 𝑗, it follows that
⟨𝑝𝑖 , 𝑥𝑝𝑗−1 ⟩ = 0 whenever 𝑖 < 𝑗 − 1.
We have shown that sequences of orthogonal polynomials satisfy a three-term recurrence relation
2
𝑝𝑗 (𝑥) = (𝑥 − 𝛼𝑗 )𝑝𝑗−1 (𝑥) − 𝛽𝑗−1
𝑝𝑗−2 (𝑥),
2
where the recursion coefficients 𝛼𝑗 and 𝛽𝑗−1
are defined to be
𝛼𝑗 =
⟨𝑝𝑗−1 , 𝑥𝑝𝑗−1 ⟩
,
⟨𝑝𝑗−1 , 𝑝𝑗−1 ⟩
2
𝑗 > 1,
𝑗 > 1,
𝛽𝑗2 =
⟨𝑝𝑗−1 , 𝑥𝑝𝑗 ⟩
⟨𝑥𝑝𝑗−1 , 𝑝𝑗 ⟩
⟨𝑝𝑗 , 𝑝𝑗 ⟩
∥𝑝𝑗 ∥2
,
=
=
=
⟨𝑝𝑗−1 , 𝑝𝑗−1 ⟩
⟨𝑝𝑗−1 , 𝑝𝑗−1 ⟩
⟨𝑝𝑗−1 , 𝑝𝑗−1 ⟩
∥𝑝𝑗−1 ∥2
𝑗 ≥ 1.
Note that ⟨𝑥𝑝𝑗−1 , 𝑝𝑗 ⟩ = ⟨𝑝𝑗 , 𝑝𝑗 ⟩ because 𝑥𝑝𝑗−1 differs from 𝑝𝑗 by a polynomial of degree at most
𝑗 − 1, which is orthogonal to 𝑝𝑗 . The recurrence relation is also valid for 𝑗 = 1, provided that we
define 𝑝𝑗−1 (𝑥) ≡ 0, and 𝛼1 is defined as above. That is,
𝑝1 (𝑥) = (𝑥 − 𝛼1 )𝑝0 (𝑥),
𝛼1 =
⟨𝑝0 , 𝑥𝑝0 ⟩
.
⟨𝑝0 , 𝑝0 ⟩
If we also define the recursion coefficient 𝛽0 by
𝛽02 = ⟨𝑝0 , 𝑝0 ⟩,
and then define
𝑞𝑗 (𝑥) =
𝑝𝑗 (𝑥)
,
𝛽0 𝛽1 ⋅ ⋅ ⋅ 𝛽𝑗
then the polynomials 𝑞0 , 𝑞1 , . . . , 𝑞𝑛 are also orthogonal, and
⟨𝑞𝑗 , 𝑞𝑗 ⟩ =
⟨𝑝𝑗 , 𝑝𝑗 ⟩
2
𝛽0 𝛽12 ⋅ ⋅ ⋅ 𝛽𝑗2
= ⟨𝑝𝑗 , 𝑝𝑗 ⟩
⟨𝑝𝑗−1 , 𝑝𝑗−1 ⟩
⟨𝑝0 , 𝑝0 ⟩
1
⋅⋅⋅
= 1.
⟨𝑝𝑗 , 𝑝𝑗 ⟩
⟨𝑝1 , 𝑝1 ⟩ ⟨𝑝0 , 𝑝0 ⟩
That is, these polynomials are orthonormal.
If we consider the inner product
∫
1
⟨𝑓, 𝑔⟩ =
𝑓 (𝑥)𝑔(𝑥) 𝑑𝑥,
−1
then a sequence of orthogonal polynomials, with respect to this inner product, can be defined as
follows:
𝐿0 (𝑥) = 1,
𝐿1 (𝑥) = 𝑥,
2𝑗 + 1
𝑗
𝐿𝑗+1 (𝑥) =
𝑥𝐿𝑗 (𝑥) −
𝐿𝑗−1 (𝑥),
𝑗+1
𝑗+1
𝑗 = 1, 2, . . .
These are known as the Legendre polynomials. One of their most important applications is in the
construction of Gaussian quadrature rules. Specifically, the roots of 𝐿𝑛 (𝑥), for 𝑛 ≥ 1, are the nodes
of a Gaussian quadrature rule for the interval [−1, 1]. However, they can also be used to easily
compute continuous least-squares polynomial approximations, as the following example shows.
Example We will use Legendre polynomials to approximate 𝑓 (𝑥) = cos 𝑥 on [−𝜋/2, 𝜋/2] by a
quadratic polynomial. First, we note that the first three Legendre polynomials, which are the ones
of degree 0, 1 and 2, are
𝐿0 (𝑥) = 1,
𝐿1 (𝑥) = 𝑥,
3
1
𝐿2 (𝑥) = (3𝑥2 − 1).
2
However, it is not practical to use these polynomials directly to approximate 𝑓 (𝑥), because they
are orthogonal with respect to the inner product defined on the interval [−1, 1], and we wish to
approximate 𝑓 (𝑥) on [−𝜋/2, 𝜋/2].
To obtain orthogonal polynomials on [−𝜋/2, 𝜋/2], we replace 𝑥 by 2𝑡/𝜋, where 𝑡 belongs to
[−𝜋/2, 𝜋/2], in the Legendre polynomials, which yields
(
)
2𝑡
1 12 2
˜
˜
˜
𝐿0 (𝑡) = 1, 𝐿1 (𝑡) = , 𝐿2 (𝑡) =
𝑡 −1 .
𝜋
2 𝜋2
Then, we can express our quadratic approximation 𝑓2 (𝑥) of 𝑓 (𝑥) by the linear combination
˜ 0 (𝑥) + 𝑐1 𝐿
˜ 1 (𝑥) + 𝑐2 𝐿
˜ 2 (𝑥),
𝑓2 (𝑥) = 𝑐0 𝐿
where
𝑐𝑗 =
˜𝑗 ⟩
⟨𝑓, 𝐿
,
˜𝑗 , 𝐿
˜𝑗 ⟩
⟨𝐿
𝑗 = 0, 1, 2.
Computing these inner products yields
˜ 0⟩ =
⟨𝑓, 𝐿
∫
𝜋/2
cos 𝑡 𝑑𝑡
−𝜋/2
= 2,
∫ 𝜋/2
2𝑡
˜
⟨𝑓, 𝐿1 ⟩ =
cos 𝑡 𝑑𝑡
−𝜋/2 𝜋
= 0,
)
∫ 𝜋/2 (
1 12 2
˜
⟨𝑓, 𝐿2 ⟩ =
𝑡 − 1 cos 𝑡 𝑑𝑡
𝜋2
−𝜋/2 2
2 2
=
(𝜋 − 12),
𝜋2
∫ 𝜋/2
˜ 0, 𝐿
˜ 0⟩ =
⟨𝐿
1 𝑑𝑡
−𝜋/2
= 𝜋,
∫ 𝜋/2 ( )2
2𝑡
˜
˜
⟨𝐿1 , 𝐿1 ⟩ =
𝑑𝑡
𝜋
−𝜋/2
8𝜋
=
,
3
)]2
∫ 𝜋/2 [ (
1 12 2
˜
˜
⟨𝐿2 , 𝐿2 ⟩ =
𝑡 −1
𝑑𝑡
𝜋2
−𝜋/2 2
𝜋
=
.
5
4
It follows that
and therefore
2
,
𝜋
2 5 2
10
(𝜋 − 12) = 3 (𝜋 2 − 12),
2
𝜋 𝜋
𝜋
(
)
2
12 2
5 2
𝑓2 (𝑥) = + 3 (𝜋 − 12)
𝑥 − 1 ≈ 0.98016 − 0.4177𝑥2 .
𝜋 𝜋
𝜋2
𝑐0 =
𝑐1 = 0,
𝑐2 =
This approximation is shown in Figure 1. □
Figure 1: Graph of cos 𝑥 (solid blue curve) and its continuous least-squares quadratic approximation
(red dashed curve) on [−𝜋/2, 𝜋/2]
It is possible to compute sequences of orthogonal polynomials with respect to other inner products. A generalization of the inner product that we have been using is defined by
∫ 𝑏
⟨𝑓, 𝑔⟩ =
𝑓 (𝑥)𝑔(𝑥)𝑤(𝑥) 𝑑𝑥,
𝑎
where 𝑤(𝑥) is a weight function. To be a weight function, it is required that 𝑤(𝑥) ≥ 0 on (𝑎, 𝑏), and
that 𝑤(𝑥) ∕= 0 on any subinterval of (𝑎, 𝑏). So far, we have only considered the case of 𝑤(𝑥) ≡ 1.
5
Another weight function of interest is
𝑤(𝑥) = √
1
,
1 − 𝑥2
−1 < 𝑥 < 1.
A sequence of polynomials that is orthogonal with respect to this weight function, and the associated
inner product
∫ 1
1
𝑑𝑥
⟨𝑓, 𝑔⟩ =
𝑓 (𝑥)𝑔(𝑥) √
1 − 𝑥2
−1
is the sequence of Chebyshev polynomials
𝐶0 (𝑥) = 1,
𝐶1 (𝑥) = 𝑥,
𝐶𝑗+1 (𝑥) = 2𝑥𝐶𝑗 (𝑥) − 𝐶𝑗−1 (𝑥),
𝑗 = 1, 2, . . .
which can also be defined by
𝐶𝑗 (𝑥) = cos(𝑗 cos−1 𝑥),
−1 ≤ 𝑥 ≤ 1.
It is interesting to note that if we let 𝑥 = cos 𝜃, then
∫ 1
1
𝑑𝑥
⟨𝑓, 𝐶𝑗 ⟩ =
𝑓 (𝑥) cos(𝑗 cos−1 𝑥) √
1 − 𝑥2
−1
∫ 𝜋
𝑓 (cos 𝜃) cos 𝑗𝜃 𝑑𝜃.
=
0
In later lectures, we will investigate continuous and discrete least-squares approximation of functions
by linear combinations of trigonometric polynomials such as cos 𝑗𝜃 or sin 𝑗𝜃, which will reveal one
of the most useful applications of Chebyshev polynomials.
6