ELG5377 Adaptive Signal
Processing
Lecture 13: Method of Least
Squares
Introduction
• Given a sequence of observations x(1), x(2), …, x(N) which
occur at time t1, t2, …, tN.
– The requirement is to construct a curve that is used to fit
these points in some optimum fashion.
– Let us denote this curve as f(ti).
– The objective is to minimize the squares of the differences
between f(ti) and x(i). J = Σ(f(ti)-x(i))2.
• The method of least squares can be viewed as an alternative to
Wiener filters
– Wiener filters are based on ensemble averages
– MLS is deterministic in approach and is based on time
averages.
Statement of the Linear Least-Squares
Estimation Problem
• Consider a physical phenomena that is characterized by
two sets of variables
– d(i) and x(i).
• The variable d(i) is observed at time ti in response to the
subset of variables x(i), x(i-1), … x(i-M+1).
• The response d(i) is modelled by
M 1
*
d (i )   wok
x(i  k )  eo (i )
(1)
k 0
– where wok are unknown parameters of the model and eo(i)
represents a measurement error.
– The measurement error is an unobservable random variable
that is introduced to the model to account for its inaccuracy.
Statement of the Linear Least-Squares
Estimation Problem 2
• It is customary to assume that the measurement error is white
with 0 mean and variance s2.
• The implication of this assumption is that
M 1
*
E[d (i )]   wok
x(i  k )
k 0
– where the values of x(i), x(i-1), …, x(i-M+1) are all known.
– Hence the mean of the response d(i) is uniquely determined by the
model.
Statement of the Linear Least-Squares
Estimation Problem 3
• The problem we have to solve is to estimate the unknown
parameters wok of the multiple linear regression model of (1)
given the two observable sets of variables x(i) and d(i) for i= 1,
2, …, N.
• To do this we use the linear transversal filter shown below as
the model of interest, whose output is y(i) and we use d(i) as the
desired response.
x(i-1)
x(i)
w0
*
…
*
w1
x(i-M+1)
wM-1
e(i)
*
y(i)
+
-
+
d(i)
Statement of the Linear Least-Squares
Estimation Problem 4
e(i )  d (i )  y (i )
M 1
y (i )   wk* x(i  k )
k 0
M 1
e(i )  d (i )   wk* x(i  k )
k 0
i2
J   e(i )
i i1
2
Must chose tap weights so as to
minimize this cost function
Data Windowing
•
•
•
•
Typically, we are given data for i= 1 to N, where N > M.
It is only at time M, where d(i) is a function of known data.
Also, for i > N, d(i) has unknown data in its equation.
Covariance method
– No assumptions on unknown data, therefore i1 = M and i2 =
N.
• Autocorrelation method
– i1 = 1 and i2 = N+M-1. We assume that x(i) = 0 for i<M and
i>N.
• Prewindowing and postwindowing.
• We will use covariance method.
Orthogonality Principle Revisited
N
N
J   e(i )   e(i )e* (i )
2
iM
iM
M 1
e(i )  d (i )   (ak  jbk ) x(i  k )
k 0
where wk  ak  jbk
N
 e(i ) *

J
J
e* (i )
e(i ) *
e* (i )
k J 
j
  
e (i ) 
e(i )  j
e (i )  j
e(i ) 
ak
bk i  M  ak
ak
bk
bk

e(i )
e* (i )
  x(i  k ),
  x* (i  k )
ak
ak
e(i )
e* (i )
 jx(i  k ),
  jx* (i  k )
bk
bk
N


 k J  2  x(i  k )e* (i )  0 for all k
iM
(2)
Orthogonality Principle Revisited 2
• Let e(i) = emin(i) if we select w0, w1, …, wM-1 such that J is
minimized.
N
*
• Then from (2)  x(i  k )emin  0
iM
– The minimum-error time series is orthogonal to the time
series x(i-k) applied to tap k of a transversal filter of
length M for k = 1, 2, … ,M-1 when the filter is operating
in its least-squares condition.
• Let ymin(i) be the output of the filter when it is operating in its
least squares condition.
• d(i) = ymin(i)+emin(i).
Energy of Desired
N
N
Ed   d (i )   ymin (i )  emin (i )
2
iM
2
iM
N
N
N
Ed   ymin (i )   emin (i )   ymin (i )e
2
iM
2
iM
M 1
iM
N
ymin (i )   w x(i  k ),  ymin (i )e
k 0
*
k
*
min
iM
N
*
min
N
N
*
(i )   ymin
(i )emin (i )
iM
N M 1
(i )    w x(i  k )e
i  M k 0
*
k
Therefore Ed   ymin (i )   emin (i )  Eest  Emin
iM
2
iM
2
*
min
M 1
(i )   w
k 0
*
k
N
 x(i  k )e
iM
*
min
(i )
Normal Equations and Linear Least-Squares
Filters
• Let the filter described by the tap weights, w0, w1, …, wM-1
be operating in its least-squares condition.
– Therefore
M 1
emin (i )  d (i )   wt* x(t  k )
t 0
e
iM
N M 1
N
N
(i ) x (i  k )   d (i ) x (i  k )    wt* x(t  k )x* (i  k )
*
*
min
i  M t 0
iM
N M 1
N
0   d (i ) x (i  k )    wt* x(t  k )x* (i  k )
*
i  M t 0
iM
M 1
N
N
 w  x(t  k )x (i  k )   d (i) x (i  k )
t 0
*
t
M 1
*
*
iM
iM
N
N
 w  x(i  k )x (t  k )   d
*
t 0
t
iM
iM
*
(i ) x(i  k )
k = 0, 1, …, M-1
Normal Equations and Linear Least-Squares
Filters 2
N
Let  (t , k )   x(i  k ) x * (i  t ), 0  (t , k )  M  1
iM
This is the time averaged autocorrel ation function
N
Let z ( k )   x(i  k )d * (i )
iM
This is the time average cross - corrrelati on
Normal Equations and Linear Least-Squares
Filters: Matrix formulation
 (1,0)
  (0,0)
  (0,1)
 (1,1)

φ




 (0, M  1)  (1, M  1)
z  z (0), z (1),..., z (1  M )
T
ˆ  [ w0 , w1 ,..., wM 1 ]T
w
ˆ  z, w
ˆ  φ 1z
φw


 ( M  1,0) 
 ( M  1,1) 


  ( M  1, M  1)


Example
•
•
•
x(1)=2, x(2) = 1, x(3) = -0.5, x(4)
= 1.2
d(2) = 0.5, d(3) = 1, d(4) = 0
Find the two-tap LS filter.