Orthogonality and the Least Squares Approximation
N
Goal: To find the best approximation of f ( x) on [a, b] by SN ( x)   cnn ( x) for a set of
n 1
fixed functions n ( x); i.e., to find the cn ’s such that SN ( x) approximates f ( x) in the least
squares sense.
Definitions:

b
Mean Square Deviation:  [ f ( x)  S N ( x)]2  ( x) dx, with weight function  ( x)  0.
a

b
Converges in the mean:  [ f ( x)  S N ( x)]2  ( x) dx  0 as N  .
a

b
Inner product: ( , )    ( x) ( x)  ( x) dx.
a

b
Orthogonal functions: ( , )    ( x) ( x)  ( x) dx  0.
a

Mutually Orthogonal set {n ( x)}n 1 : (n ,m )  0, m  n.
Minimization in Least Squares Sense
b
b
N
a
n 1
2
2
 [ f ( x)  S N ( x)]  ( x) dx   [ f ( x)   cnn ( x)]  ( x) dx
a
b
b
N
b N
N
a n 1
m 1
  f 2 ( x)  ( x) dx  2  f ( x ) cnn ( x ) ( x ) dx    cnn ( x ) cmm ( x ) ( x ) dx
a
a
N
n 1
N
N
 ( f , f )  2 cn ( f , n )   cn cm (n , m )
n 1
n 1 m 1
N
N
n 1
n 1
 ( f , f )  2 cn ( f , n )   cn2 (n , n ).
Aiming to find coefficients, so complete the square in cn . Focusing on the last two terms,
N
c
n 1
2
n
N
N
n 1
n 1
(n , n )  2 cn ( f , n )   (n , n )cn2  2( f , n )cn
N

2( f , n ) 
  (n , n ) cn2 
cn 
(n , n ) 
n 1

N

( f , n )   ( f , n ) 
  (n , n )  cn 
 

(n , n )   (n , n ) 

n 1
2
The mean square deviation is minimized by choosing
Coefficients!

.

( f , n )
cn 
.
(n , n )
2
These are the Fourier
Bessel’s Inequality
Inserting in the mean square deviation yields
b
0   [ f ( x)  S N ( x)]2  ( x) dx
a
N
N
n 1
n 1
 ( f , f )  2 cn ( f , n )   cn2 (n , n )
N
 ( f , f )   cn2 (n , n ).
n 1
N
Thus, we obtain Bessel’s Inequality: ( f , f )   cn2 (n , n ).
n 1
Parseval’s Equality
Let N  . Then
N
c
n 1
2
n
b
(n , n ) converges if ( f , f )   f 2 ( x)  ( x) dx  . The space of all such f
a
is denoted L (a, b), the space of square integrable functions on (a, b) with weight  ( x).
Thus, from Calculus II we know that  an converges implies that an  0 as n  .
Therefore, in this problem the terms cn2 (n , n )  0 as n  . This is only possible if
2
N
cn  0 as n  . Thus, if  cnn converges in the mean to f , then
n 1
b
N
 [ f ( x)   c  ]
2
a
n 1
n n
 ( x) dx approaches zero as N  . This implies from the above derivation
N
of Bessel’s inequality that ( f , f )   cn2 (n , n )  0. This leads to Parseval’s equality:
n 1

( f , f )   cn2 (n , n ).
n 1
b
N
a
n 1
( f ( x)   cnn ( x)) 2  ( x) dx  0. If this is true for
Parseval’s equality holds if and only if Nlim
 
every square integrable function in L (a, b), then the set of functions {n ( x)}n 1 is said to be
complete. One can view these functions as an infinite dimensional basis for the space of
square integrable functions on (a, b) with weight  ( x)  0.
2
One can extend the above limit cn  0 as n  , by assuming that
b
n ( x )
is uniformly
n
bounded and that  | f ( x) |  ( x) dx  . This is the Riemann-Lebesque Lemma, but will
a
not be proven now.