'
$
The Geometric Tools of Hilbert Spaces
Erlendur Karlsson
February 2009
c
2009
by Erlendur Karlsson
All rights reserved. No part of this manuscript is to be
reproduced without the writen consent of the author.
&
%
Contents
1 Orientation
1
2 Definition of Inner Product Spaces
2
3 Examples of Inner Product Spaces
3
4 Basic Objects and Their
4.1 Vectors and Tuples . .
4.2 Sequences . . . . . . .
4.3 Subspaces . . . . . . .
5
5
7
8
Properties
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Inner Product and Norm Properties
9
6 The Projection Operator
11
7 Orthogonalizations of Subspace Sequences
y
7.1 Different Orthogonalizations of St,p
. . . . . . . . . . .
y
7.1.1 The Backward Orthogonalization of St,p
. . . .
y
7.1.2 The Forward Orthogonalization of St,p
. . . . .
y
7.1.3 The Eigenorthogonalization of St,p
. . . . . . .
Z . . . . . . . . . . .
7.2 Different Orthogonalizations of St,p
Z
7.2.1 The Backward Block Orthogonalization of St,p
Z
7.2.2 The Forward Block Orthogonalization of St,p
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
16
16
16
16
17
18
18
19
y
Z
8 Procedures for Orthogonalizing St,p
and St,p
19
8.1 The Gram–Schmidt Orthogonalization Procedure . . . . . . . . . . . . . . 19
8.2 The Schur Orthogonalization Procedure . . . . . . . . . . . . . . . . . . . 20
9 The Dynamic Levinson Orthogonalization Procedure
9.1 The Basic Levinson Orthogonalization Procedure . . . . . . . . . . . .
y
9.1.1 The Basic Levinson Orthogonalization Procedure for St,p
. . .
Z
9.1.2 The Basic Block Levinson Orthogonalization Procedure for St,p
9.2 Relation Between The Projection Coefficients of Different Basis . . . .
y
9.2.1 Relation Between Direct Form and Lattice Realizations on St,i
Z
9.2.2 Relation Between Direct Form and Lattice Realizations on St,i
9.3 Evaluation of Inner Products . . . . . . . . . . . . . . . . . . . . . . .
y
9.3.1 Lattice Inner Products for St,p
. . . . . . . . . . . . . . . . . .
Z
9.3.2 Lattice Inner Products for St,p
. . . . . . . . . . . . . . . . . .
9.4 The Complete Levinson Orthogonalization Procedure . . . . . . . . .
i
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
22
22
23
25
27
27
29
30
30
30
31
1
1
Orientation
Our goal in this course is to study and develop model based methods for adaptive systems. Although it is possible to do this without explicit use of the geometric tools
of Hilbert spaces, there are great advantages to be gained from a geometric formulation. These are largely derived from our familiarity with Euclidean geometry and in
particular with the concepts of orthogonality and orthogonal projections. The intuition
gained from Euclidean geometry will serve as our guide to constructive development of
complicated algorithms and will often help us to understand and interpret complicated
algebraic results in a geometrically obvious way. These notes are devoted to a study of
those aspects of Hilbert space theory which are needed for the constructive geometric
development. For the reader who wishes to go deeper into the theory of Hilbert spaces
we can recommend the books by Kreyszig [1] and Máté [2].
In two– and three–dimensional Euclidean spaces we worked with geometric objects such
as vectors and subspaces. We studied various geometric properties of these objects such
as the norm of a vector, the angle between two vectors and the orthogonality of one
vector with respect to another. We also learned about geometrical operations on these
objects such as projecting a vector onto a subspace and orthogonalizing a subspace.
These powerful geometric concepts and tools were also intuitive, natural and easy to
grasp as we could actually see them or visualize them in two– and three–dimensional
drawings.
These geometric concepts and tools were easily extended to the n–dimensional Euclidean
space (n > 3), through the scalar product. Even here simple two– and three–dimensional
drawings can be useful in illustrating or visualizing geometric phenomenon.
One proof of how essetial the geometric machinery is in mathematics is that it can also be
extended to infinite–dimensional vector spaces, this time using an extesion of the scalar
product, namely the inner product. Going to an infinite–dimensional vector space makes
things more complicated. The lack of a finite basis is one such complication that creates difficulties in applications. Fortunately, many infinite–dimensional normed spaces
contain an infinite sequence of vectors called a fundamental sequence, such that every
vector in the space can be approximated to any accuracy with a linear combination of
vectors belonging to the fundamental sequence. Such infinite dimensional spaces occur
frequently in engineering and one can work with those in pretty much the same way
one works with finite dimensional spaces. Another complication is that of completeness.
When an inner product space is not complete, it has holes, meaning that there exist
elements of finite norm that are not in the inner product space but can be approximated
to any desired accuracy by an element of the inner product space. The good news is that
all incomplete inner product spaces can be constructively extended to complete inner
product spaces. A Hilbert space is a complete inner product space.
2
2
DEFINITION OF INNER PRODUCT SPACES
In these notes we go through the following moments:
2
3
4
5
6
7
8
9
2
Definition of Inner Product Spaces
Examples of Inner Product Spaces
Basic Objects and Their Properties
Inner Product and Norm Properties
The Projection Operator
y
Z
Orthogonalizations of the Subspaces St,p
and St,p
y
Z
Procedures for Orthogonalizing St,p and St,p
The Dynamic Levinson Orthogonalization Procedure
Definition of Inner Product Spaces
Definition 2-1 An Inner Product Space
An inner product space {H, h·, ·i} is a vector space H equipped with an inner product
h·, ·i, which is a mapping from H × H onto the scalar field of H (real or complex).
For all vectors x, y and z in H and scalar α, the inner product satisfies the four axioms
IP1 :
hx + y, zi = hx, zi + hy, zi
IP2 :
hαx, yi = α hx, yi
IP3 :
hx, yi = hy, xi∗
IP4 :
hx, xi ≥ 0
hx, xi = 0 ⇔ x = 0.
An inner product on H defines a norm on H given by
||x|| =
q
hx, xi
and a metric on H given by
d(x, y) = ||x − y|| =
q
hx − y, x − yi.
Remark 2-1 Think Euclidean
The inner product is a natural generalization of the scalar product of two vectors in an
n–dimensional Euclidean space. Since many of the properties of the Euclidean space
carry over to inner product spaces, it will be helpful to keep the Euclidean space in mind
in all that follows.
3
3
Examples of Inner Product Spaces
Example 3-1 The Euclidean space {C n , hx, yi =
Pn
∗
i=1 xi yi }
The n–dimensional vector space over the complex field, C n , with the inner product
between two vectors




y1
x1




and
y =  ... 
x =  ... 
yn
xn
defined by
hx, yi =
n
X
xi yi∗
i=1
where
norm
yi∗
is the complex conjugation of yi , is an inner product space. This gives the
v
u n
uX
||x|| = hx, xi1/2 = t |xi |2
i=1
and the Euclidean metric
v
u n
uX
1/2
d(x, y) = ||x − y|| = hx − y, x − yi = t |xi − yi |2
i=1
Example 3-2 The space of square summable sequences
P
∗
{l2 , hx, yi = ∞
i=1 xi yi }
The vector space of square summable sequences with the inner product between the two
sequemces x = {xi } and y = {yi } defined by
hx, yi =
∞
X
xi yi∗
i=1
where yi∗ is the complex conjugation of yi , is an inner product space. This gives the
norm
v
||x|| = hx, xi
1/2
u∞
uX
= t xi x∗i
i=1
and the metric
d(x, y) = ||x − y|| = hx − y, x − yi
1/2
v
u∞
uX
= t |xi − yi |2
i=1
Example 3-3 The space of
square integrable functions
R ∗
2
{L , hf, gi = fg }
The vector space of square integrable functions defined on the interval [a, b], with the
inner product defined by
hf, gi =
Z b
a
f (τ )g ∗ (τ )dτ
4
3
EXAMPLES OF INNER PRODUCT SPACES
where g ∗ (τ ) is the complex conjugation of g(τ ), is an inner product space. This gives
the norm
||f|| = hf, fi
1/2
s
Z b
=
f (τ )f ∗ (τ )dτ
a
and the metric
1/2
d(f, g) = ||f − g|| = hf − g, f − gi
s
Z b
=
|f (τ ) − g(τ )|2 dτ
a
Example 3-4 The continuous functions over [0, 1], {C([0, 1]), hf, gi =
R
fg∗ }
The vector space of continuous functions defined on the interval [0, 1], with the inner
product defined by
hf, gi =
Z 1
f (τ )g ∗ (τ )dτ
0
where g ∗ (τ ) is the complex conjugation of g(τ ), is an inner product space. This gives
the norm
s
||f|| = hf, fi1/2 =
Z 1
f (τ )f ∗ (τ )dτ
0
and the metric
s
d(f, g) = ||f − g|| = hf − g, f − gi
1/2
Z 1
=
|f (τ ) − g(τ )|2 dτ
0
Example 3-5 Random variables with finiteR second order moments,
{L2 (Ω, B, P ), hx, yi = E(xy∗ ) = Ω x(ω)y ∗ (ω)dP (w)}
The space of random variables with finite 2nd moments, L2 (Ω, B, P ), with the inner
product defined by
hx, yi = E(xy∗ ) =
Z
x(ω)y ∗ (ω)dP (w)
Ω
where E is the expectation operator, is an inner product space. Here a random variable
denotes the equivalence class of random variables on (Ω, B, P ), where y is equivalent to
x if E|x − y|2 = 0.
This gives the norm
||x|| = hx, xi1/2 =
q
E|x|2
and the metric
d(x, y) = ||x − y|| = hx − y, x − yi1/2 =
q
E|x − y|2 .
5
4
Basic Objects and Their Properties
Here we will get acquainted with the main geometrical objects of the space and their
properties. We begin by looking at the most basic objects vectors and tuples. We then
proceed with sequences and subspaces.
4.1
Vectors and Tuples
Definition 4.1-1 Vector
The basic element of a vector space is called a vector. In the case of the n–dimensional
Euclidean vector space this is the standard n–dimensional vector, in the space of square
summable sequences this is a square summable sequence and in the space of square integrable functions this is a square integrable function.
Definition 4.1-2 Tuple
An n–tuple is an ordered set of n elements. An n–tuple made up of the n vectors
x1 , x2 , . . . , xn is denoted by {x1 , x2 , . . . , xn }. We will mostly be using n–tuples of vectors and parameter coefficients (scalars).
Definition 4.1-3 Norm of a Vector
The norm of a vector x, denoted by kxk, is a measure of its size. More precisely it is a
mapping from the vector space onto the positive reals, which for all vectors x and y in
H and scalar α satisfies the following axioms:
N2 :
kxk ≥ 0
kxk = 0 ⇔ x = 0
kx + yk ≤ kxk + kyk
N3 :
kαxk = |α|kxk
N1 :
In an inner product space the norm is always derived from the inner product as follows:
||x|| = hx, xi1/2
It is an easy exercise to show that the above definition satisfies the norm axioms.
Definition 4.1-4 Orthogonality
Two vectors x and y of an inner product space H are said to be orthogonal if
hx, yi = 0,
and we write x ⊥ y.
An n–tuple of vectors, {x1 , x2 , . . . xn }, is said to be orthogonal if
hxn , xm i = 0
for all n 6= m.
Definition 4.1-5 Angle Between Two Vectors
In a real inner product space the angle θ between the two vectors x and y is defined by
the formula
hx, yi
cos(θ) =
||x||||y||
.
6
4
BASIC OBJECTS AND THEIR PROPERTIES
Definition 4.1-6 Linear Combination of Vectors and Tuples of Vectors
A linear combination of vectors x1 , x2 , . . . , xn is an expression of the form
α(1)x1 + α(2)x2 + . . . + α(n)xn =
n
X
α(i)xi
i=1
where the coefficients α(1), α(2), . . . α(n) are any scalars. Sometimes we will also want
to express this sum in the form of a matrix multiplication
α(1)
 .. 
α(i)xi = {x1 , x2 , . . . , xn }  .  = {x1 , x2 , . . . , xn }α,
i=1
α(n)

n
X

|
{z
}
α
where we consider the n–tuple {x1 , x2 , . . . , xn } as being a form of a row vector multiplying the column vector α.
Similarly a linear combination of m–tuples X1 , X2 , . . . , Xn where Xk is an m–tuple of
vectors {xk,1 , xk,2 , . . . , xk,m } is actually a linear combination of the vectors x1,1 , . . . , x1,m ,
x2,1 , . . . , x2,m , . . . , xn,1 , . . . , xn,m given by
m
n X
X
α(i, j)xi,j
i=1 j=1
which can be expressed in matrix notation as
m
n X
X
i=1 j=1
α(i, j)xi,j
α(i, 1)


..
{xi,1 , . . . , xi,m } 
=

.
i=1
α(i, m)

n
X

|
=
n
X
{z
αi
}
Xi αi
i=1
α1
 .. 
= {X1 , . . . , Xn }  .  .
αn


Still another linear combination is when an m-tuple Z = {z1 , z2 , . . . , zm } is formed from
linear combinations of the n-tuple X = {x1 , x2 , . . . , xn }, where
αi (1)


zi =
αi (j)xj = {x1 , x2 , . . . , xn }  ...  = {x1 , x2 , . . . , xn }αi .
j=1
αi (n)

n
X
|

{z
αi
}
We can then write
Z = {z1 , z2 , . . . , zm }
= {x1 , x2 , . . . , xn } (α1 , . . . , αm )
|
= Xα
{z
α
}
4.2
Sequences
7
Definition 4.1-7 Linear Independence and Dependence
The set of vectors x1 , x2 , . . . xn is said to be linearly independent if the only n–tuple of
scalars α1 , α2 , . . . αn that makes
α1 x1 + α2 x2 + . . . + αn xn = 0
(4.1)
is the all zero n–tuple with α1 = α2 = . . . = αn = 0. The set of vectors is said to be
linearly dependent if it is not linearly independent. In that case there exists an n–tuple
of scalars different from the all zero n–tuple, that satisfies equation (4.1).
Definition 4.1-8 Inner Product Between Tuples of Vectors
Given the p–tuple of vectors X = {x1 , . . . , xp } and the q–tuple of vectors Y = {y1 , . . . , yq },
the inner product between the two is defined as the matrix
hx1 , y1 i . . . hxp , y1 i


..
..
..
hX, Yi = 

.
.
.
hx1 , yq i . . . hxp , yq i


This inner product between tuples of vectors is often used to simplify notaion.
For the vector–tuples X, Y and Z and the coefficient matrices α and β the following
properties hold:
IP1 :
hX + Y, Zi = hX, Zi + hY, Zi
IP2 :
hXα, Yi = hX, Yi α
hX, Yβi = β H hX, Yi
IP3 :
IP4 :
4.2
hX, Yi = hY, XiH
hX, Xi ≥ 0
hX, Xi = 0 ⇔ X = 0.
Sequences
Definition 4.2-1 Sequence
A sequence is an infinite ordered set of elements. We will denote sequences of vectors by
{xn } or {xn : n = 1, 2, . . .}.
Definition 4.2-2 Convergence of Sequences
A sequence {xn } in a normed space X is said to converge to x if x ∈ X and
lim ||xn − x|| = 0
n→∞
Definition 4.2-3 Cauchy Sequence
A sequence {xn } in a normed space X is said to be Cauchy if for every > 0 there is a
positive integer N such that
||xn − xm || < for all m, n > N.
8
4.3
4
BASIC OBJECTS AND THEIR PROPERTIES
Subspaces
Definition 4.3-1 Linear Subspace
A subset S of H is called a linear subspace if for all x, y ∈ S and all scalars α, β in the
scalar field of H, it holds that αx + βy ∈ S. S is therefore a linear space over the same
field of scalars. S is said to be a proper subspace of H if S =
6 H.
Definition 4.3-2 Finite and Infinite Dimensional Vector Spaces
A vector space H is said to be finite dimensional if there is a positive integer n such
that H contains a linearly independent set of n vectors whereas any set of n + 1 or more
vectors is linearly dependent. n is called the dimension of H, written dim H = n. By
definition, H = {0} is finite dimensional and dim H = 0. If H is not finite dimensional,
it is said to be infinite dimensional.
Definition 4.3-3 Span and Set of Spanning Vectors
For any nonempty subset M ⊂ H the set S of all linear combinations of vectors in M is
called the span of M, written S = Sp{M}. Obviously S is a subspace of H and we say
that S is spanned or generated by M. We also say that M is the set of spanning vectors
of S.
Definition 4.3-4 Basis for Vector Spaces
For an n–dimensional vector space S, an n–tuple of linearly independent vectors in S
is called a basis for S. If {x1 , x2 , . . . xn } is a basis for S, every x ∈ S has a unique
representation as a linear combination of the basis vectors
x = α1 x1 + α2 x2 + . . . + αn xn .
This idea of a basis can be extended to infinite–dimensional vector spaces with the so
called Schauder basis. Not all infinite–dimensional vector spaces have a Schauder basis,
however, the ones of interest for engineers do.
For an infinite–dimensional vector space H, a sequence of vectors in H, {xn }, is a
Schauder basis for H, if for every x ∈ H there is a unique sequence of scalars {αn } such
that
n
lim ||x −
n→∞
The sum
P∞
k=1 αk xk
X
αk xk || = 0
k=1
is then called the expansion of x with respect to {xn }, and we write
x=
∞
X
α k xk
k=1
Proposition 4.3-1 Tuples of Vectors Spanning the Same Subspace
Given an n–dimensional subspace S = Sp{X}, where X is an n–tuple of the linearly
independent vectors x1 , . . . , xn , then X is a basis for S and each vector in S is uniquely
written as a linear combination of the vectors in X. If we now pick the n–vectors
y1 , . . . , yn in S, where
yk =
n
X
i=1
αk (i)xi
9
αk (1)


..
= {x1 , . . . , xn } 

.
αk (n)

|

{z
αk
}
= Xαk ,
then Y = {y1 , . . . , yn } is also a basis for S if the parameter vectors αk for k = 1, 2, . . . , n
are linearly independent. A convenient way of expressing this linear mapping from X to
Y is to write
Y = Xα,
where α is the matrix of parameter vectors
α1 (1) . . . αn (1)


..
..
..
α=
 = (α1 , . . . , αn ) .
.
.
.
α1 (n) . . . αn (n)


Here it helps to look at Xα as the matrix multiplication between the n–tuple X (conP
sidered as a row vector) and the matrix α. Then yk can be seen as Xαk = ni=1 αk (i)xi .
When the parameter vectors in α are linearly independent, α has full rank and is invertible. Then it is also possible to write X as a linear mapping from Y
X = Yα−1 ,
where α−1 is the inverse matrix of α.
Definition 4.3-5
Orthogonality
A vector y is said to be orthogonal to a finite subspace Sn = Sp{x1 , . . . , xn } if y is
orthogonal to all the spanning vectors of Sn , x1 . . . , xn , and we write y ⊥ Sn .
Definition 4.3-6
Completeness
In some inner product spaces there exist sequences that converge to a vector that is not
a member of the inner product space. It is for example easy to construct a sequence
of continous functions that converges to a discontinous function. These types of inner
product spaces have ”holes” and are said to be incomplete. Inner product spaces where
every Cauchy sequence converges to a member of the inner product space are said to be
complete. A Hilbert space is a complete inner product space.
5
Inner Product and Norm Properties
Theorem 5-1 The Cauchy–Schwartz inequality
In an inner product space H there holds:
| hx, yi | ≤ kxk kyk
for all x, y ∈ H.
(5.1)
Equality holds in (5.1) if and only if x and y are linearly dependent or either of the two
is zero.
10
6
THE PROJECTION OPERATOR
Proof: The equality obviously holds if x = 0 or y = 0. If x 6= 0 and y 6= 0 then for any
scalar λ we have
0 ≤ hx − λy, x − λyi
= hx, xi + |λ|2 hy, yi − hλy, xi − hx, λyi
(5.2)
∗
= hx, xi + |λ|2 hy, yi − hλy, xi − hλy, xi
= hx, xi + |λ|2 hy, yi − 2Re(λ hy, xi).
For λ =
hx,yi
hy,yi
we have
0 ≤ hx, xi −
| hx, yi |2
hy, yi
and the Cauchy–Schwartz inequality is obtained by suitable rearrangement. From the
properties of the inner product it follows at once that if x = λy, then the equality holds.
Conversely, if the equality holds, then either x = 0, y = 0, or else it follows from (5.2)
that there exists a λ such that
x − λy = 0,
i.e. x and y are linearly dependent.
Proposition 5-1 The triangle inequality
In an inner product space H there holds:
kx + yk2 ≤ (kxk + kyk)2
for all x, y ∈ H.
(5.3)
Equality holds in (5.3) if and only if x and y are linearly dependent or either of the two
is zero.
Proof: This follows directly from the Cauchy–Schwartz inequality
kx + yk2 = hx + y, x + yi
= hx, xi + 2Re(hx, yi) + hy, yi
≤ kxk2 + 2| hx, yi | + kyk2
≤ kxk2 + 2kxk kyk + kyk2
= (kxk + kyk)2 .
Proposition 5-2 The parallelogram equality
In an inner product space H there holds:
kx + yk2 + kx − yk2 = 2(kxk2 + kyk2 )
for all x, y ∈ H.
Proof: There holds:
kx ± yk2 = kxk2 ± 2Re(hx, yi) + kyk2
By adding we get the parallelogram equality.
(5.4)
11
H
6
y 1
6
y − PS (y) ⊥ S
s = PS (y)
-
-
S
Figure 1: The Projection Operator
6
The Projection Operator
The projection operator is an operator that maps a given vector y to a vector s in a
given subspace S in a way that minimizes the norm between the two vectors, ky − sk
and what characterizes the projection is that the projection error vector is orthogonal to
the subspace S. This is easily visualized in the simple two dimensional plot in Figure 1.
Now for a formal statement and proof of this.
Theorem 6-1
The Orthogonal Projection Theorem
Let S be a proper subspace of a Hilbert space H. Then for every y ∈ H there is a unique
vector PS (y) ∈ S that minimizes the metric ky − sk over all s ∈ S, i.e. there is a unique
vector PS (y) ∈ S that satisfies
ky − PS (y)k = inf ky − sk = d(y, S).
s∈S
(6.1)
PS (y) is called the projection of y onto the subspace S.
A unique property of the projection PS (y) is that the projection error y − PS (y) is
orthogonal to S, i.e. hy − PS (y), xi = 0 for all x ∈ S.
Proof: The theorem has two main components. First there is the existence and uniqueness of the projection, then there is the unique orthogonality property of the projection
error. Let us begin by proving the existence and uniqueness.
Existence and uniqueness: We begin by noting that inf s∈S ky − sk must be positive and
hence there exists a sequence {sn } ∈ S such that
ky − sn k → d = inf ky − sk as n → ∞.
s∈S
Since S is a linear subspace 21 (sn + sm ) ∈ S and thus k 12 (sn + sm ) − yk ≥ d. From the
parallelogram equality it then follows that
ksn − sm k2 = k(sn − y) − (sm − y)k2
sn + sm
− y)k2
2
≤ 2ksn − yk2 + 2ksm − y)k2 − 4d2 .
= 2ksn − yk2 + 2ksm − y)k2 − 4k
(6.2)
12
6
THE PROJECTION OPERATOR
The right-hand side of (6.2) tends to zero as m, n → ∞ and thus {sn } is a Cauchy
sequence. Since H is complete and S is closed
sn → PS (y) ∈ S and ky − PS (y)k = d.
If we substitute in the expression (6.2) sn = z and sm = PS (y) we obtain z = PS (y),
from which the uniqueness follows.
Unique orthogonality property of the projection error: What we need to show is that
z = PS (y) if and only if y − z is orthogonal to every s ∈ S, or in other words if and only
if y − z ∈ S ⊥ . We begin by showing that e = y − PS (y) ⊥ S. Let w ∈ S with kwk = 1.
Then for any scalar λ, PS (y) + λw ∈ S. The minimal property of PS (y) then implies
that for any scalar λ
kek2 = ky − PS (y)k2 ≤ ky − PS (y) − λwk2
= ke − λwk2 = he − λw, e − λwi .
(6.3)
That is
0 ≤ −λ hw, ei − λ∗ he, wi + |λ|2
for all scalars λ.
Choosing λ = he, wi then gives 0 ≤ −| he, wi |2 . Consequently he, wi = 0 for all w ∈ S
of unit norm, and it follows this orthogonal relation holds for all w ∈ S.
Next we show that if hy − z, wi = 0 for some z ∈ S and all w ∈ S, then z = PS (y).
Let z ∈ S and assume that hy − z, wi = 0 for all w ∈ S. Then for z ∈ S we have that
y − z ∈ S and that x − y ⊥ y − z. Consequently we have that
kx − zk2 = k(x − y) + (y − z)k2 = k(x − y)k2 + k(y − z)k2 .
As a result,
kx − zk2 ≥ k(x − y)k2
for all z ∈ S, with equality only if y − z = 0, as required.
Corollary 6-1
Projecting a Vector onto a Finite Subspace
Given a finite subspace spanned by the linearly independent vectors x1 , . . . , xn , S =
Sp{x1 , . . . , xn }, the projection of a vector y onto S is given by
PS (y) =
n
X
α(i)xi
i=1
α(1)
 .. 
= {x1 , . . . , xn }  . 
α(n)


{z
|
}
α
where the projection coefficient vector α satisfies the normal equation
hx1 , x1 i . . . hxn , x1 i
α(1)
hy, x1 i

  ..  

..
..
..
.
.

 .  = 
.
.
.
.
.
hx1 , xn i . . . hxn , xn i
α(n)
hy, xn i





Defining the n–tuple of vectors X = {x1 , . . . , xn } and using the definition of an inner
product between tuples of vectors the normal equation can be written in the more
compact form
hX, Xi α = hy, Xi
13
For a 1-dim subspace S = Sp{x}, the projection of y onto S simply becomes
PS (y) = αx,
where
α=
hy, xi
hy, xi
=
.
hx, xi
||x||2
Proof: The normal equation comes directly from the orthogonality property of the
projection error, namely
y − PSn (y) ⊥ x1 , . . . , xn
⇓
hy − PSn (y), xq i = 0
for q = 1, . . . , n
⇓
hy, xq i = hPSn (y), xq i
for q = 1, . . . , n
⇓
n
X
α(i) hxi , xq i = hy, xq i
for q = 1, . . . , n
i=1
Writing this up in matrix format gives the above normal equation.
Corollary 6-2
Projecting a Tuple of Vectors onto a Finite Subspace
Given a finite subspace spanned by the linearly independent n–tuple X = {x1 , . . . , xn },
S = Sp{X} = Sp{x1 , . . . , xn }, we have seen in Corollary 6-1 that the projection of a
vector yk onto S is given by
PS (yk ) =
n
X
αk (i)xi
i=1
αk (1)


..
= {x1 , . . . , xn } 

.
αk (n)


|
{z
}
αk
where the projection coefficient vector αk satisfies the normal equation
hx1 , x1 i . . . hxn , x1 i
αk (1)
hyk , x1 i






..
.
.
..
.
..
..
..


=
.
.
.
hx1 , xn i . . . hxn , xn i
αk (n)
hyk , xn i





It is then easily seen that the projection of the m-tuple Y = {y1 , . . . , ym } onto S is
given by
PS (Y) = {PS (y1 ), . . . , PS (ym )}
= {
n
X
α1 (i)xi , . . . ,
i=1
n
X
αm (i)xi }
i=1
α1 (1) . . . αm (1)


..
..
..
= {x1 , . . . , xn } 

.
.
.
α1 (n) . . . αm (n)

|

{z
α
}
14
6
THE PROJECTION OPERATOR
where the projection coefficient matrix α satisfies the normal equation
α1 (1) . . . αm (1)
hy1 , x1 i . . . hym , x1 i
hx1 , x1 i . . . hxn , x1 i






.
.
..
..
..
.
.
..
.
..
..
..
..
..

=
,

.
.
.
.
α1 (n) . . . αm (n)
hy1 , xn i . . . hym , xn i
hx1 , xn i . . . hxn , xn i





which is more compactly writen as
hX, Xi α = hY, Xi .
Corollary 6-3
Projecting an m–tuple Y onto the span of the m–tuples
X1 , . . . , Xn
Given a finite subspace S spanned by the linearly independent set of m–tuples X1 , . . . , Xn
with Xi = {xi,1 , . . . , xi,m } we have that
S = Sp{X1 , X2 , . . . , Xn }
= Sp{x1,1 , . . . , x1,m , x2,1 , . . . , x2,m , . . . , xn,1 , . . . , xn,m }
Then from Corollary 6-1 we have that the projection of a vector yk onto S is given by
PS (yk ) =
m
n X
X
αk (i, j)xi,j
i=1 j=1












= {x1,1 , . . . , x1,m , x2,1 , . . . , x2,m , . . . , xn,1 , . . . , xn,m } 











αk (1, 1) 

..
αk (1)
.


αk (1, m)

αk (2, 1) 

..
αk (2)
.

αk (n, m)

























αk (n) 



{z
}
αk (2, m)
..
.
αk (n, 1)
..
.
|
αk
where the projection coefficient vector αk satisfies the normal equation



hyk , x1,1 i
hx1,1 , x1,1 i . . . hx1,m , x1,1 i
hxn,1 , x1,1 i . . . hxn,m , x1,1 i  αk (1, 1) 
..
..
..
..
..
..



..
..
 



.
.
.
.
 
 hx ,.x i . . . hx ,. x i . . . hx ,.x i . . . hx ,. x i  


αk (1, m) 
 1,1 1,m
 hyk , x1,m i 
1,m
1,m
n,1
1,m
n,m
1,m  






 



 
..
..
.
..
..
,


..
=
.



.
.
.





 




hxn,1 , xn,1 i . . . hxn,m , xn,1 i   αk (n, 1)   hyk , xn,1 i 
 hx1,1 , xn,1 i . . . hx1,m , xn,1 i






.
.
.
.
.
.
..
..




..
..
..
..
..
..
.
.
...
αk (n, m)
hx1,1 , xn,m i . . . hx1,m , xn,m i
hxn,1 , xn,m i . . . hxn,m , xn,m i
hyk , xn,m i

which again is written in more compact for as
αk (1)
hyk , X1 i
hX1 , X1 i . . . hXn , X1 i


 

..
..
..
..
.
.


=
.
.
.
.
.
.
hX1 , Xn i . . . hXn , Xn i
αk (n)
hyk , Xn i





15
It is then easily seen that the projection of the m-tuple Y = {y1 , . . . , ym } onto S is
given by
PS (Y) = {PS (y1 ), . . . , PS (ym )}
n X
m
X
= {
α1 (i, j)xi,j , . . . ,
i=1 j=1
n
X
n
X
i=1
i=1
= {
n X
n
X
αm (i, j)xi,j }
i=1 i=1
Xi α1 (i), . . . ,
Xi αm (i)}
α1 (1) . . . αm (1)


..
..
..
= {X1 , . . . , Xn } 

.
.
.
α1 (n) . . . αm (n)


{z
|
}
α
where the projection coefficient matrix α satisfies the normal equation
hy1 , X1 i . . . hym , X1 i
α1 (1) . . . αm (1)
hX1 , X1 i . . . hXn , X1 i






..
..
.
.
..
.
..
.
.
..
..
..
..
..
,
=


.
.
.
.
hy1 , Xn i . . . hym , Xn i
α1 (n) . . . αm (n)
hX1 , Xn i . . . hXn , Xn i





which can be written slightly more compactly as
hY, X1 i
α1 (1) . . . αm (1)
hX1 , X1 i . . . hXn , X1 i






..
.
.
..
.
.
.
..
..
..
..
..
.
=


.
.
hY, Xn i
α1 (n) . . . αm (n)
hX1 , Xn i . . . hXn , Xn i





Remark 6-1 Subspace Based Orthogonal Decomposition of H
From the above we also see that given a subspace S ∈ H, H can be decomposed into the
two orthogonal subspaces S and S ⊥ , where S ⊥ is the subspace of all vectors in H that
are orthogonal to S, and all vectors in H can be uniquely written as the sum of a vector
in S and a vector in S ⊥ ,
D
6S ⊥
6P
E
⊥ (y)
Sn
H
PSn (y), PSn⊥ (y) = 0
n
1
y
PSn (y)
-
-
Sn
Corollary 6-4 Projections onto an Orthogonally Decomposed Subspace
A projection of a vector onto a finite subspace that is orthogonally decomposed into
other sub-subspaces equals the sum of the projections of the vector onto each of the
orthogonal sub-subspaces.
16
7
7
ORTHOGONALIZATIONS OF SUBSPACE SEQUENCES
Orthogonalizations of Subspace Sequences
One of the most important subspaces that shows up in adaptive systems is the subspace
y
St,p
= Sp{yt−1 , yt−2 , . . . , yt−p }
(7.1)
where {yt }∞
t=0 is a sequence of vectors yt in some Hilbert space H, which can be either
deterministic or stochastic. This kind of subspace comes up when modelling single input
single output (SISO) moving average (MA) and autoregressive (AR) systems.
Another important subspace is
Z
St,p
= Sp {Zt−1 , Zt−2 , . . . , Zt−p }
(7.2)
where {Zt }∞
t=0 is a sequence of q-tuples of vectors, Zt = {zt,1 , . . . , zt,q }, in some Hilbert
space H. This subspaces can be seen as a block version of the former and comes up
when modelling single input single output ARX systems, certain types of time–varying
systems and multi input multi output (MIMO) systems.
The adaptive algorithms we will be studying in this course are based on orthogonalizations of these two subspaces. Let us therefore take a closer look at the main orthogonalizations.
7.1
y
Different Orthogonalizations of St,p
y
There exist infinitely many orthogonalizations of St,p
, but there are three commonly
used orthogonalizations that are of interest in adaptive systems. These are the backward
orthogonalization, the forward orthogonalization and the eigenorthogonalization.
7.1.1
y
The Backward Orthogonalization of St,p
y
The most common orthogonalization of St,p
= Sp{yt−1 , yt−2 , . . . , yt−p } is the backward
orthogonalization where the latest spanning vector yt−1 is choosen as the first basis
vector and one then moves backwards in time to orthogonalize the rest of the vectors
with respect to the vectors in their future domain as shown in Figure 2 below.
yt−1 , . . . , yt−i ← yt−(i+1) yt−(i+2) , . . . , yt−p
|
{z
y
St,i
}
| {z }
bt,i = yt−(i+1) − Pt,i (yt−(i+1) )
y
Figure 2: Backward Orthogonalization of St,p
= Sp{yt−1 , yt−2 , . . . , yt−p }
The vectors bt,i are called the backward projection errors and with this orthogonalization
y
we can write St,p
as
y
St,p
= Sp{bt,0 , bt,1 , . . . , bt,p−1 }.
7.1.2
y
The Forward Orthogonalization of St,p
y
The forward orthogonalization of St,p
is a dual orthogonalization to the backward orthogonalization. There the oldest spanning vector yt−p is chosen as the first basis vector
and one then moves forward in time to orthogonalize the rest of the vectors with respect
to the vectors in their past domain. The orthogonalization will then be the one shown
below.
7.1
y
Different Orthogonalizations of St,p
17
yt−1 , . . . , yt−(p−i−1) yt−(p−i) → yt−(p−i+1) , . . . , yt−p
| {z }
|
{z
y
St−(p−i),i
yt−(p−i) − Pt−(p−i),i (yt−(p−i) ) = et−(p−i),i
}
y
Figure 3: Forward Orthogonalization of St,p
= Sp{yt−1 , yt−2 , . . . , yt−p }.
The vectors et−(p−i),i are called the forward projection errors and with this orthogonaly
ization we can write St,p
as
y
= Sp{et−1,p−1 , . . . , et−(p−1),1 , et−p,0 }.
St,p
7.1.3
y
The Eigenorthogonalization of St,p
The eigenorthogonalization is the most decoupled orthogonalization there is, in that it
y
orthogonalizes St,p
in such a way that the projection coefficient vectors describing the
orthogonal basis vectors in terms of the given spanning vectors also form an orthogonal
set of vectors. The name eigenorthogonalization comes from the close coupling of this
orthogonalization to the eigendecomposition of the related Gram–matrix. This is probably too abstract to follow, so let us go through this more slowly.
As we mentioned in the beginning of this section, there are infinitely many ways of
orthogonalizing the subspace
y
St,p
= Sp{yt−1 , yt−2 , yt−3 , . . . , yt−p }.
y
An interesting question is, therefore, if there exists an orthogonalization of St,p
where
the orthogonal basis vectors are given by
xt,k =
p
X
αk (i)yt−i for k = 1, 2, . . . , p
i=1
and the parameter vectors describing the basis vectors in terms of the given spanning
vectors


αk (1)


 αk (2) 

αk = 
 ..

 .

αk (p)
also form an orthogonal basis in C p ?
The answer turnes out to be yes. To obtain a unique solution the parameter vector
basis α1 , . . . , αp is made orthonormal (orthogonal and unit norm). We call the orthogonalization the eigenorthogonalization because the parameter vectors αk satisfy the
eigenequation
hYt , Yt i αk = ||xt,k ||2 αk ,
(7.3)
where Yt is the p–tuple {yt−1 , yt−2 , . . . , yt−p }.
To see this we first need the relation between hXt , Xt i and hYt , Yt i. The result then
follows from looking at the inner product hxt,k , Yt i. Now as Xt = Yt (α1 , . . . , αp ) =
Yt A we have that

kxt,1 k2
0
..
hXt , Xt i = 

0



.
kxt,p k2
18
7
ORTHOGONALIZATIONS OF SUBSPACE SEQUENCES
= hYt A, Yt Ai
= AH hYt , Yt i A
and
hYt , Yt i = A−H hXt , Xt i A−1 .
Then as the parameter vector basis {α1 , . . . , αp } is orthonormal we have that AH A = I,
i.e. A−1 = AH and A−H = A, which again gives
hYt , Yt i = A hXt , Xt i AH .
With these two relations between hXt , Xt i and hYt , Yt i we can move onto the inner
product hxt,k , Yt i :
hxt,k , Yt i = hYt αk , Yt i
= hYt , Yt i αk
= A hXt , Xt i AH αk
| {z }
ek
2
= Akxt,k k ek
= kxt,k k2 αk
which gives us the eigenequation (7.3).
Assuming that the spanning vectors yt−1 , . . . , yt−p are linearly independent, hYt , Yt i
is a positive definite matrix, which will have real and positive eigenvalues and orthogonal eigenvectors. We see here that the eigenvalues will be the squared norms of the
eigenorthogonal basis vectors and the parameter vectors describing those basis vectors
in terms of the given spanning vectors will be the eigenvectors. From this we see that
the eigenorthogonalization actually exists and that it can be evaluated with the use of
the eigendecomposition of the Gram-matrix hYt , Yt i or the singular decomposition of
Yt .
7.2
Z
Different Orthogonalizations of St,p
Z and S y is that S Z is the span of the n q–tuples Z
The difference between St,p
t−1 , . . . , Zt−p ,
t,p
t,p
y
while St,p is the span of the n vectors yt−1 , . . . , yt−p . Instead of fully orthogonalizing
Z into nq one dimensional subspaces, we can orthogonalize it into n q–dimensional
St,p
y
subspaces in pretty much the same way we orthogonalized St,p
, working with q–tuples
instead of vectors. Let us look at the backward block orthogonalization and the forward
Z .
block orthogonalization of St,p
7.2.1
Z
The Backward Block Orthogonalization of St,p
Z = Sp{Z
The most common orthogonalization of St,p
t−1 , Zt−2 , . . . , Zt−p } is the backward
block orthogonalization where the latest spanning tuple Zt−1 is choosen as the first basis
tuple and one then moves backwards in time to orthogonalize the rest of the vector
tuples with respect to the vector tuples in their future domain as shown in Figure 4.
The vector tuples Bt,i are called the backward projection errors and with this orthogoZ as
nalization we can write St,p
Z
St,p
= Sp{Bt,0 , Bt,1 , . . . , Bt,p−1 }.
19
Zt−1 , . . . , Zt−i ← Zt−(i+1) Zt−(i+2) , . . . , Zt−p
|
{z
Z
St,i
}
|
{z
}
Bt,i = Zt−(i+1) − Pt,i (Zt−(i+1) )
Z = Sp{Z
Figure 4: Backward Block Orthogonalization of St,p
t−1 , Zt−2 , . . . , Zt−p }
7.2.2
Z
The Forward Block Orthogonalization of St,p
Z is a dual orthogonalization to the backward
The forward block orthogonalization of St,p
orthogonalization. There the oldest spanning vector-tuple Zt−p is chosen as the first
basis tuple and one then moves forward in time to orthogonalize the rest of the tuples
with respect to the ones in their past domain. The orthogonalization will then be the
one shown in Figure 5.
Zt−1 , . . . , Zt−(p−i−1) Zt−(p−i) → Zt−(p−i+1) , . . . , Zt−p
|
{z
}
Zt−(p−i) − Pt−(p−i),i (Zt−(p−i) ) = Et−(p−i),i
|
{z
}
Z
St−(p−i),i
Z = Sp{Z
Figure 5: Forward Block Orthogonalization of St,p
t−1 , Zt−2 , . . . , Zt−p }.
The vector tuples Et−(p−i),i are called the forward projection errors and with this orZ as
thogonalization we can write St,p
Z
St,p
= Sp{Et−1,p−1 , . . . , Et−(p−1),1 , Et−p,0 }.
8
y
Z
Procedures for Orthogonalizing St,p
and St,p
We have often been confronted with the problem of orthogonalizing a given subspace,
and are well familiar with the Gram-Schmidt orthogonalization procedure. A familiar
example is the subspace of n-th order polynomials defined on the interval R[−1, 1], where
1
the inner product between the functions f and g is defined by hf , gi = −1
f (t)g(t)dt.
k
This space is spanned by the functions f0 , f1 , . . . , fn , where fk (t) = t for t ∈ [−1, 1] and
k = 0, 1, . . . , n. Orthogonalizing this subspace with the Gram-Schmidt orthogonalization
procedure results in the well known Legendre polynomial basis functions. In this section
we will study that and the related Schur–orthogonalization procedure for orthogonalizing
y
Z .
the subspaces St,p
and St,p
8.1
The Gram–Schmidt Orthogonalization Procedure
The Gram–Schmidt orthogonalization procedure is an efficent orthogonalization procedure for orthogonalizations such as the backward and forward orthogonalizations. The
main idea behind the procedure is that the subspace is orthogonalized in an order recursiva manner so that the subspaces beeing projected onto are always orthogonal or block
orthogonal. This allows each projection to be evaluated in terms of orthogonal or block
orthogonal basis vectors, thus making the procedure relatively efficient.
y
The orthogonalization precedure for the backward orthogonalization of St,p
is shown
y
below, where Pt,k (x) denotes the projection of the vector x onto St,k :
20
8
Y
Z
PROCEDURES FOR ORTHOGONALIZING ST,P
AND ST,P
1-st orth. basis vector:
bt,0
= yt−1
2-nd orth. basis vector:
bt,1
= yt−2 − Pt,1 (yt−2 )
= yt−2 −
3-rd orth. basis vector:
bt,2
hy2 , bt,0 i
bt,0
kbt,0 k2
= yt−3 − Pt,2 (yt−3 )
= yt−3 −
1
X
hyt−3 , bt,i i
i=0
..
.
p-th orth. basis vector:
kbt,i k2
bt,i
..
.
bt,p−1
= yt−p − Pt,p−1 (yt−p )
= yt−p −
p−2
X
hyt−p , bt,i i
i=0
kbt,i k2
bt,i
Counting the operations in the evaluation procedure we see that it involves 1 + 2 + 3 +
. . . + (p − 1) = p(p − 1)/2 projections on to one dimensional subspaces, which is relatively
efficient.
Z is very
The orthogonalization precedure for the backward block orthogonalization of St,p
similar and shown below, where Pt,k (X) denotes the projection of the vector tuple X
Z :
onto St,k
1-st orth. basis tuple:
Bt,0
= Zt−1
2-nd orth. basis tuple:
Bt,1
= Zt−2 − Pt,1 (Zt−2 )
= Zt−2 − Bt,0 hBt,0 , Bt,0 i
3-rd orth. basis tuple:
Bt,2
−1
hZt−2 , Bt,0 i
= Zt−3 − Pt,2 (Zt−3 )
= Zt−3 −
1
X
−1
hZt−3 , Bt,i i
−1
hZt−p , Bt,i i
Bt,i hBt,i , Bt,i i
i=0
..
.
..
.
p-th orth. basis tuple:
Bt,p−1
= Zt−p −
p−1
X
Bt,i hBt,i , Bt,i i
i=0
Counting the operations in the evaluation procedure we see that it involves 1 + 2 + 3 +
. . . + (p − 1) = p(p − 1)/2 projections on to q-dimensional subspaces, which is relatively
efficient.
8.2
The Schur Orthogonalization Procedure
The Schur orthogonalization procedure is another efficient orthogonalization procedure
for orthogonalizations such as the backward and forward orthogonalizations. It too, like
the Gram-Schmidt orthogonalization procedure, accomplishes the job with 1 + 2 + 3 +
. . .+(p−1) = p(p−1)/2 projections onto one or q–dimensional subspaces. The approach
is, however, somewhat differrent. It is best explained with a concrete example, so let us
begin by looking at the Schur algorithm for evaluating the backward orthogonalization
y
for St,p
. First we define the following projection errors:
xjt,i = yt−j − Pt,i (yt−j )
for i = 0, . . . , p − 1 and j = i + 1, . . . , p
8.2
The Schur Orthogonalization Procedure
21
y
where Pt,i (yt−j ) denotes the projection of yt−j onto St,i
.
As in the Gram-Schmidt procedure, the first orthogonal basis vector in the backward
orthogoalization, bt,0 , is chosen as bt,0 = yt−1 . To obtain the second orthogonal basis
y
vector, bt,1 , the rest of the spanning vectors in St,p
= Sp{yt−1 , yt−2 , . . . , yt−p } are
orthogonalized with respect to yt−1 . This gives bt,1 as the first vector in the resulting
y
subspace, which has dimension one less than that of St,p
. This operation is shown below.
I−P
Sp{yt−1 } ←− Sp{yt−2 , yt−3 , . . . , yt−p }
↓
x
Sp{x2t,1 , x3t,1 , . . . , xpt,1 } = St,1
where xjt,1 is the projection error from projecting yt−j onto the span of yt−1 and bt,1 =
x2t,1 . The third orthogonal basis vector, bt,2 , is similarly obtained by orthogonalizing
x with respect to x2 . This gives b
the rest of the spanning vectors in St,1
t,2 as the first
t,1
x . This
vector in the resulting subspace, which has dimension one less than that of St,1
operation is shown below.
I−P
Sp{x2t,1 } ←− Sp{x3t,1 , x4t,1 , . . . , xpt,1 }
↓
x
Sp{x3t,2 , x4t,2 , . . . , xpt,2 } = St,2
where xjt,2 is the projection error from projecting xjt,1 onto the span of x2t,1 , and bt,2 =
x3t,2 . Iterating this operation until there is only a one dimensional subspace left will
then give the complete orthogonal basis of the backward orthogonalization. Defining the
y
x = Sp{x1 , x2 , . . . , xp }, where xj = y
original subspace St,p
as St,0
t−j for k = 1, . . . , p,
t,0
t,0
t,0
t,0
the complete orthogonalization procedure can be summarized as shown below:
1-st orth. basis vector:
Sp{x1t,0 , x2t,0 , . . . , xpt,0 }
=
x
St,0
bt,0 = x1t,0
2-nd orth. basis vector:
Sp{x2t,0 , x3t,0 , . . . , xpt,0 }
↓ (I − P) onto the span of x1t,0
Sp{x2t,1 , x3t,1 , . . . , xpt,1 }
=
x
St,1
bt,1 = x2t,1
3-rd orth. basis vector:
Sp{x3t,1 , x4t,1 , . . . , xpt,1 }
↓ (I − P) onto the span of x2t,1
Sp{x3t,2 , x4t,2 , . . . , xpt,2 }
=
x
St,2
bt,2 = x3t,2
..
.
p-th orth. basis vector:
Sp{xpt,p−2 }
↓ (I − P) onto the span of xp−1
t,p−2
Sp{xpt,p−1 }
=
x
St,p−1
.
bt,p−1 = xpt,p−1
Counting the operations in the evaluation procedure we see that it involves (p − 1) +
(p − 2) + . . . + 2 + 1 = p(p − 1)/2 projections on to one dimensional subspaces, which is
22
9
THE DYNAMIC LEVINSON ORTHOGONALIZATION PROCEDURE
exactly the same number of operations as in the Gram-Schmidt procedure.
Z is
The corresponding Schur algorithm for the backward block orthogonalization of St,p
X = S Z and X0 = Y
summarized below, where St,0
t−1−k .
t,p
t,k
1-st orth. basis tuple:
Sp{X1t,0 , X2t,0 , . . . , Xpt,0 }
X
= St,0
Bt,0 = X1t,0
2-nd orth. basis tuple:
Sp{X2t,0 , X3t,0 , . . . , Xpt,0 }
↓ (I − P) onto the span of X1t,0
Sp{X2t,1 , X3t,1 , . . . , Xpt,1 }
X
= St,1
Bt,1 = X2t,1
3-rd orth. basis tuple:
Sp{X3t,1 , X4t,1 , . . . , Xpt,1 }
↓ (I − P) onto the span of X2t,1
Sp{X3t,2 , X4t,2 , . . . , Xpt,2 }
X
= St,2
Bt,2 = X3t,2
..
.
p-th orth. basis tuple:
Sp{Xpt,p−2 }
p−1
↓ (I − P) onto the span of Xt,p−2
Sp{Xpt,p−1 }
X
= St,p−1
.
Bt,p−1 = Xpt,p−1
Again, counting the operations in the evaluation procedure we see that it involves
(p − 1) + (p − 2) + . . . + 2 + 1 = p(p − 1)/2 projections on to q–dimensional subspaces,
which is exactly the same number of operations as in the Gram-Schmidt procedure.
9
The Dynamic Levinson Orthogonalization Procedure
In the previous two sections we have studied orthogonalizations and orthogonalization
y
Z . When dealing with adaptive systems,
procedures for the two subspaces St,p
and St,p
y
Z , but to
however, the problem is not to orthogonalize the single subspaces St,p
and St,p
y ∞
Z
∞
orthogonalize the sequences of subspaces {St,p }t=0 , and {St,p }t=0 in a time recursive
manner. We can of course always use the methods we have just studied to orthogonalize
these sequences of subspaces, but it is not unlikely that we will be able to do this more
efficiently if we use the special structure of these sequences of subspaces. A method that
accomplishes this is the Levinson orthogonalization procedure. What characterizes the
y ∞
y
Z }∞ ) is that S y
Z
Z
subspace sequence {St,p
}t=0 ({St,p
t=0
t+1,p (St+1,p ) is obtained from St,p (St,p )
by removing the span of the vector yt−p (vector–tuple Zt−p ) and adding the span of yt
(Zt ). The Levinson procedure uses this structure to efficiently evaluate the backward
orthogonal basis vectors at time t + 1 from the ones at time t using only one projection
onto the span of a single vector (vector–tuple).
9.1
The Basic Levinson Orthogonalization Procedure
We begin by studying the basic Levinson orthogonalization procedure for orthogonalizing
y
St,p
following with the corresponding Levinson block orthogonalization procedure for
Z .
St,p
9.1
The Basic Levinson Orthogonalization Procedure
9.1.1
23
y
The Basic Levinson Orthogonalization Procedure for St,p
We begin by looking at the backward orthogonal basis vectors at time t+1. The (i+1)-th
orthogonal basis vector is given by
bt+1,i+1 = yt+1−(i+2) − Pt+1,i+1 (yt+1−(i+2) )
(9.1)
= yt−(i+1) − Pt+1,i+1 (yt−(i+1) )
y
which is the projection error from projecting yt−(i+1) onto St+1,i+1
as shown below
P
y
St+1,i+1
= Sp{yt , yt−1 , . . . , yt−i } ← yt−(i+1) : Pt+1,i+1 (yt−(i+1) )
The orthogonal basis vector at time t involving the same yt−(i+1) vector is bt,i =
yt−(i+1) − Pt,i (yt−(i+1) ), which is the projection error from projecting yt−(i+1) onto
y
St,i
as shown below
P
y
St,i
= Sp{yt−1 , . . . , yt−i } ← yt−(i+1) : Pt,i (yt−(i+1) )
Comparing the two projections we see that at time t + 1 we are projecting onto the
y
y
, but has
which contains the subspace we projected onto at time t, St,i
subspace St+1,i+1
one additional dimension, namely the span of yt . We can easily orthogonalize this one
y
dimensional subspace with respect to St,i
and obtain
y
= Sp{yt } + Sp{yt−1 , . . . , yt−i }
St+1,i+1
= Sp{et,i } ⊕ Sp{yt−1 , . . . , yt−i }
= Sp{et,i } ⊕
(9.2)
y
St,i
y
where et,i is the projection error from projecting yt onto St,i
, et,i = yt − Pt,i (yt ), and
y
now
the ⊕ denotes that we have a sum of two orthogonal subspaces. With St+1,i+1
y
is
decomposed into two orthogonal subspaces the projection of yt−(i+1) onto St+1,i+1
simply obtained as the sum of the projections of yt−(i+1) onto each of the two orthogonal
subspaces, namely
Pt+1,i+1 (yt−(i+1) ) = Pt,i (yt−(i+1) ) + Pet,i (yt−(i+1) )
(9.3)
where Pet,i (yt−(i+1) ) denotes the projection of yt−(i+1) onto the span of et,i . With this
relation between the projections at time t + 1 and t we obtain the following relation
between the orthogonal projection errors,
bt+1,i+1 = yt−(i+1) − Pt+1,i+1 (yt−(i+1) )
= yt−(i+1) − Pt,i (yt−(i+1) ) + Pet,i (yt−(i+1) )
= bt,i − Pet,i (yt−(i+1) ).
(9.4)
y
Now as we can write yt−(i+1) as the sum of its projection onto St,i
and the resulting
projection error, yt−(i+1) = Pt,i (yt−(i+1) ) + bt,i , and the projection is orthogonal to et,i
we have that
Pet,i (yt−(i+1) ) = Pet,i (bt,i )
which then gives us the relation
bt+1,i+1 = bt,i − Pet,i (bt,i )
hbt,i , et,i i
= bt,i −
et,i
ket,i k2
(9.5)
24
9
THE DYNAMIC LEVINSON ORTHOGONALIZATION PROCEDURE
From this we see that we have a very efficent way of updating the orthogonal basis vectors
at time t+1 from the ones at time t, if we can efficently genearte the projection errors et,i .
In the signal processing litterature, the projection errors et,i and bt,i are respectively
called the forward and the backward projection errors. This is related to the fact that both
y
involve projections onto the same subspace St,i
, where the forward projection involves
y
the projection of a vector that is timewise in front of St,i
, while the backward projection
involves the projection of a vector that is timewise in back of it,
P
P
Pt,i (yt ) : yt → Sp{yt−1 , . . . , yt−i } ← yt−(i+1) : Pt,i (yt−(i+1) ).
|
{z
}
y
St,i
y
Due to the structure of St,i
the forward projections can be evaluated in pretty much the
same way as the backward projections. The (i + 1)-th forward projection is a projection
y
of yt onto St,i+1
, which is easily orthogonally decomposed as follows:
y
St,i+1
= Sp{yt−1 , . . . , yt−i } + Sp{yt−(i+1) }
= Sp{yt−1 , . . . , yt−i } ⊕ Sp{bt,i }
=
y
St,i
(9.6)
⊕ Sp{bt,i }
y
This orthogonal decomposition allows the projection of yt onto St,i+1
to be evaluated as
the sum of the projections of yt onto each orthogonal subspace, namely
Pt,i+1 (yt ) = Pt,i (yt ) + Pbt,i (yt )
(9.7)
where Pbt,i (yt ) denotes the projection of yt onto the span of bt,i . Then following the
exact same line of reasoning that was done on the backward projections we obtain the
recursion
et,i+1 = et,i − Pbt,i (et,i )
het,i , bt,i i
= et,i −
bt,i ,
kbt,i k2
(9.8)
which is a very efficient way of generating the forward projection error vectors given the
availability of the backward projection error vectors.
Combining the two projection error recursions (equations (9.5) and (9.8)) we obtain the
following block diagram of the projection operations:
et,i
@
- m
bt+1,i
-D
-
-
het,i , bt,i i
bt,i
kbt,i k2
|
@
ktb (i)@
kte (i)
et,i+1 = et,i −
{z
kte (i)
}
A A
A
hbt,i , et,i i
AAU
bt+1,i+1 = bt,i −
et,i
- m ket,i k2
| {z }
This block takes two input vectors, et,i
projecting one vector onto the other.
ktb (i)
and bt,i , and outputs the projection errors from
To evaluate the basis vectors bt+1,0 , bt+1,1 , . . . , bt+1,p−1 all we need to do is to cascade
p − 1 of these blocks together and we have the complete orthogonalization procedure:
9.1
The Basic Levinson Orthogonalization Procedure
yt
et,0
et,1
- k -
@
ktb (0)@
bt+1,0
25
et,p−2
-
et,p−1
- k -
@
ktb (p − 2)@ A
A
···
A
A
e
e
kt (0)
kt (p − 2) A
A
AU bt+1,1 bt+1,p−2
AU bt+1,p−1
-D - k -D - k Evaluating the bt,i ‘s in this manner only requires the evaluation of 2(p − 1) projections
onto one dimensional subspaces, which is much more efficient than the p2 (p−1) operations
needed by the Gram–Schmidt and Schur procedures.
9.1.2
Z
The Basic Block Levinson Orthogonalization Procedure for St,p
We begin by looking at the backward block orthogonal basis vectors at time t + 1. The
(i+1)-th block orthogonal basis tuple is given by
Bt+1,i+1 = Zt+1−(i+2) − Pt+1,i+1 (Zt+1−(i+2) )
(9.9)
= Zt−(i+1) − Pt+1,i+1 (Zt−(i+1) )
Z
which is the projection error from projecting Zt−(i+1) onto St+1,i+1
as shown below
P
Z
St+1,i+1
= Sp{Zt , Zt−1 , . . . , Zt−i } ← Zt−(i+1) : Pt+1,i+1 (Zt−(i+1) )
The orthogonal basis tuple at time t involving the same Zt−(i+1) tuple is Bt,i = Zt−(i+1) −
Z as shown
Pt,i (Zt−(i+1) ), which is the projection error from projecting Zt−(i+1) onto St,i
below
P
Z = Sp{Z
St,i
t−1 , . . . , Zt−i } ← Zt−(i+1) : Pt,i (Zt−(i+1) )
Comparing the two projections we see that at time t + 1 we are projecting onto the
Z
Z , but has
subspace St+1,i+1
which contains the subspace we projected onto at time t, St,i
the additional span of Zt . We can easily orthogonalize the additional subspace with
Z and obtain
respect to St,i
Z
St+1,i+1
= Sp{Zt } + Sp{Zt−1 , . . . , Zt−i }
= Sp{Et,i } ⊕ Sp{Zt−1 , . . . , Zt−i }
= Sp{Et,i } ⊕
(9.10)
Z
St,i
Z, E
where Et,i is the projection error from projecting Yt onto St,i
t,i = Zt − Pt,i (Zt ),
Z
and the ⊕ denotes that we have a sum of two orthogonal subspaces. With St+1,i+1
now
Z
decomposed into two orthogonal subspaces the projection of Zt−(i+1) onto St+1,i+1
is
simply obtained as the sum of the projections of Zt−(i+1) onto each of the two orthogonal
subspaces, namely
Pt+1,i+1 (Zt−(i+1) ) = Pt,i (Zt−(i+1) ) + PEt,i (Zt−(i+1) )
(9.11)
where PEt,i (Zt−(i+1) ) denotes the projection of Zt−(i+1) onto the span of Et,i . With this
relation between the projections at time t + 1 and t we obtain the following relation
between the orthogonal projection errors,
Bt+1,i+1 = Zt−(i+1) − Pt+1,i+1 (Zt−(i+1) )
= Zt−(i+1) − Pt,i (Zt−(i+1) ) + PEt,i (Zt−(i+1) )
= Bt,i − PEt,i (Zt−(i+1) ).
(9.12)
26
9
THE DYNAMIC LEVINSON ORTHOGONALIZATION PROCEDURE
Z and the resulting
Now as we can write Zt−(i+1) as the sum of its projection onto St,i
projection error, Zt−(i+1) = Pt,i (Zt−(i+1) ) + Bt,i , and the projection is orthogonal to Et,i
we have that
PEt,i (Zt−(i+1) ) = PEt,i (Bt,i )
which then gives us the relation
Bt+1,i+1 = Bt,i − PEt,i (Bt,i )
= Bt,i − Et,i hEt,i , Et,i i−1 hBt,i , Et,i i
(9.13)
From this we see that we have a very efficent way of updating the orthogonal basis tuplets at time t + 1 from the ones at time t, if we can efficently genearte the projection
error tuplets Et,i .
Z the forward projections can be evaluated in pretty much the
Due to the structure of St,i
same way as the backward projections. The (i + 1)-th forward projection is a projection
Z
of Zt onto St,i+1
, which is easily orthogonally decomposed as follows:
Z
St,i+1
= Sp{Zt−1 , . . . , Zt−i } + Sp{Zt−(i+1) }
= Sp{Zt−1 , . . . , Zt−i } ⊕ Sp{Bt,i }
=
Z
St,i
(9.14)
⊕ Sp{Bt,i }
Z
This orthogonal decomposition allows the projection of Zt onto St,i+1
to be evaluated
as the sum of the projections of Zt onto each orthogonal subspace, namely
Pt,i+1 (Zt ) = Pt,i (Zt ) + PBt,i (Yt )
(9.15)
where PBt,i (Zt ) denotes the projection of Zt onto the span of Bt,i . Then following the
exact same line of reasoning that was done on the backward projections we obtain the
recursion
Et,i+1 = Et,i − PBt,i (Et,i )
= Et,i − Bt,i hBt,i , Bt,i i−1 hEt,i , Bt,i i ,
(9.16)
which is a very efficient way of generating the forward projection error vectors given the
availability of the backward projection error vectors.
Combining the two projection error recursions (equations (9.13) and (9.16)) we obtain
the following block diagram of the projection operations:
Et,i
@
- m
@
@
kB
t (i)
kE
t (i)
Bt+1,i
-D
-
Et,i+1 = Et,i − Bt,i hBt,i , Bt,i i−1 hEt,i , Bt,i i
-
|
{z
kE
t (i)
}
A A
A
AA
U
Bt+1,i+1 = Bt,i − Et,i hEt,i , Et,i i−1 hBt,i , Et,i i
- m |
{z
}
kB
t (i)
This block takes two input tuples, Et,i and Bt,i , and outputs the projection error tuples
from projecting one tuple onto the other.
To evaluate the basis tuples Bt+1,0 , Bt+1,1 , . . . , Bt+1,p−1 all we need to do is to cascade
p − 1 of these blocks together and we have the complete orthogonalization procedure:
9.2
Relation Between The Projection Coefficients of Different Basis
Zt
Et,0
Et,1
- k -
@
@
kB
t (0)
Et,p−2
-
27
Et,p−1
- k -
@
@ kB
t (p − 2)
A
A
···
A
A
E
E
kt (0)
kt (p − 2) A
A
Bt+1,0
AU Bt+1,1 Bt+1,p−2
AU Bt+1,p−1
-D - k -D - k Evaluating the Bt,i ‘s in this manner only requires the evaluation of 2(p − 1) projections
onto q–dimensional subspaces, which is much more efficient than the p2 (p − 1) operations
needed by the Gram–Schmidt and Schur procedures.
9.2
Relation Between The Projection Coefficients of Different Basis
We have seen that the projection of a vector onto a given subspace is parameterized as
a linear combination of the basis vectors of the subspace. Different sets of basis vectors
will, therefore, lead to different parameterizations. For us it is important to be able to
map one set of projection coefficients to another.
A case of special interest is the parameterization of the forward and backward projecy
Z , which we have studied in some detail in this section.
and St,i
tions onto the spaces St,i
These projections can be realized in terms of the given spanning vectors, yt−1 , . . . , yt−i
y
Z , which we will call the direct form realization. Another
for St,i
and Zt−1 , . . . , Zt−i for St,i
important realization is the one in terms of the backward orthogonalized basis vectors,
y
Z , which we call the lattice realization.
bt,0 , . . . , bt,i−1 for St,i
and Bt,0 , . . . , Bt,i for St,i
The relationship between the two sets of projection coefficients is embedded in the projection order recursions that have been the key components in the development of the
Levinson orthogonalization procedure in sections 9.1.1 and 9.1.2. The derivation of this
relationship is thus simply a matter of
i) Writing down the projection order recursions.
ii) Writing the individual projections out in terms of the direct form realization
vectors.
iii) Matching terms.
y
Z , as it
We begin by looking at the relationship for St,i
and then summarize it for St,i
follows in exactly the same manner.
9.2.1
y
Relation Between Direct Form and Lattice Realizations on St,i
Please recall the sequence of steps taken in developing the projection error recursions
in Section 9.1.1. It began with the forward and backward orthogonal decompositions of
y
the space St,i
= Sp{yt−1 , . . . , yt−i }
y
y
St,i+1
= St,i
⊕ Sp{bt,i }
y
y
St+1,i+1
= Sp{et,i } ⊕ St,i
where the backward projection error bt,i = yt−(i+1) − Pt,i (yt−(i+1) ) is the i-th backward
orthogonal spanning vector at time t and et,i is the forward projection error et,i =
28
9
THE DYNAMIC LEVINSON ORTHOGONALIZATION PROCEDURE
yt − Pt,i (yt ). These orthogonal decompositions then led to the two projection order
recursions
Pt,i+1 (yt ) = Pt,i (yt ) + kte (i)bt,i
= Pt,i (yt ) + kte (i) yt−(i+1) − Pt,i (yt−(i+1) )
Pt+1,i+1 (yt−(i+1) ) = Pt,i (yt−(i+1) ) + ktb (i)et,i
= Pt,i (yt−(i+1) ) + ktb (i) (yt − Pt,i (yt ))
and the projection error order recursions were achieved by subtracting these projection
order recursions from yt . The block diagrams in Section 9.1 illustrating the Levinson
orthogonalization procedure give a clear picture of the projection error order recursions,
and as the projection order recursions differ from them, only by an additive vector, they
too are easily read from the lattice block diagrams. Then, once the projection order
recursions are written down and the projections expressed in terms of the direct form
realization vectors, the direct form projection coefficient order recursions follow naturally from a direct matching of terms. This last step is straight forward, but involves
some bookkeeping.
By writing the two projections, Pt,i (yt ) and Pt,i (yt−(i+1) ) in terms of the direct form
spanning vectors
Pt,i (yt ) =
i
X
αt,i (j)yt−j
j=1
Pt,i (yt−(i+1) ) =
i
X
βt,i (j)yt−j ,
j=1
and expressing the direct form projection coefficients in vector form
αt,i (1)


=  ... 

αt,i
βt,i (1)
 .. 
= . 


β t,i
αt,i (i)

βt,i (i)
the direct form filter coefficient order recursions can be read directly from the projection
order recursions for the two projections above. This gives
Pt,i+1 (yt ) = Pt,i (yt ) + kte (i) yt−(i+1) − Pt,i (yt−(i+1) )
Pt+1,i+1 (yt−(i+1) ) = Pt,i (yt−(i+1) ) + ktb (i) (yt − Pt,i (yt ))
=⇒
αt,i+1 =
β t+1,i+1 =
αt,i
0
0
β t,i
−
kte (i)
−
ktb (i)
β t,p
−1
−1
αt,i
Note that the recursion for αt,i is strictly order recursive while the recursion for β t,i
is both time and order recursive. The exact algorithm that evaluates the direct form
projection coefficients αt,p is the following
9.2
Relation Between The Projection Coefficients of Different Basis
αt,1 = kte (0) ; β t+1,1
for i = 1 upto p − 1 do
αt,i+1
β t+1,i+1
αt,i
0
0
β t,i
=
ktb (0)
=
=
− kte (i)
−
29
ktb (i)
β t,i
−1
−1
αt,i
The algorithm assumes that the direct form projection coefficients are evaluated at each
time t.
9.2.2
Z
Relation Between Direct Form and Lattice Realizations on St,i
y
The development here is the exact same as the one for St,i
where we work with vector
tuples instead of vectors, so we just summarize the main results.
Writing the two projections, Pt,i (Zt ) and Pt,i (Zt−(i+1) ) in terms of the direct form
spanning vector tuples we have that
αt,i (1)


..
Zt−j αt,i (j) = {Zt−1 , . . . , Zt−i } 
Pt,i (Zt ) =

.
j=1
αt,i (i)

i
X
|

{z
αt,i
}


β t,i (1)


..
.
Pt,i (Zt−(i+1) ) =
Zt−j β t,i (j) = {Zt−1 , . . . , Zt−i } 
.


j=1
β t,i (i)
i
X
|
{z
β t,i
}
The direct form filter coefficient order recursions can then be read directly from the
projection order recursions for the two projections, giving
Pt,i+1 (Zt ) = Pt,i (Zt ) + Zt−(i+1) − Pt,i (Zt−(i+1) ) kE
t (i)
Pt+1,i+1 (Zt−(i+1) ) = Pt,i (Zt−(i+1) ) + (Zt − Pt,i (Zt )) kB
t (i)
=⇒
αt,i+1 =
β t+1,i+1 =
αt,i
0
0
β t,i
β t,p
−I
kE
t (i)
−I
αt,i
kB
t (i)
−
−
Note that the recursion for αt,i is strictly order recursive while the recursion for β t,i
is both time and order recursive. The exact algorithm that evaluates the direct form
projection coefficients αt,p is the following
30
9
THE DYNAMIC LEVINSON ORTHOGONALIZATION PROCEDURE
αt,1 = kE
; β t+1,1
t (0)
for i = 1 upto p − 1 do
αt,i+1
=
β t+1,i+1
=
kB
t (0)
=
αt,i
0
0
β t,i
β t,p
−I
kE
t (i)
−I
αt,i
kB
t (i)
−
−
The algorithm assumes that the direct form projection coefficients are evaluated at each
time t.
9.3
9.3.1
Evaluation of Inner Products
y
Lattice Inner Products for St,p
y
To fully implement the Levinson orthogonalization procedure on St,p
the three inner
products het,i , et,i i, het,i , bt,i i and hbt,i , bt,i i need somehow to be evaluated. Given the
inner products hyt , yt−i i for i = 0, . . . , p there is a simple way to evaluate the three
lattice inner products that is only based on geometric properties.
From the order recursion of et,i we have that
ket,i+1 k2 = het,i+1 , et,i+1 i
= het,i+1 , yt i
= het,i − kte (i)bt,i , yt i
= het,i , yt i − kte (i) hbt,i , yt i
= het,i , et,i i − kte (i) hbt,i , et,i i
= (1 − kte (i)ktb (i))ket,i k2 .
Similarly for kbt+1,i+1 k2 we have
kbt+1,i+1 k2 = (1 − kte (i)ktb (i))kbt,i k2 ,
while the het,i+1 , bt,i+1 i inner product can be evaluated directly from the hyt , yt−j i inner
products in the following manner
het,i+1 , bt,i+1 i =
D
et,i+1 , yt−(i+2)
*
yt −
=
i+1
X
E
+
αt,i+1 (j)yt−j , yt−(i+2)
j=1
=
D
E
yt , yt−(i+2) −
i+1
X
D
E
αt,i+1 (j) yt−j , yt−(i+2) .
j=1
There does also exist another way of evaluating the inner product het,i+1 , bt,i+1 i, that
is directly coupled to the Schur orthogonalization procedure. This is the topic of one of
the problems in Practice Problem Set # 2.
9.3.2
Z
Lattice Inner Products for St,p
y
In the same manner as for St,p
we have that
hEt,i+1 , Et,i+1 i = hEt,i+1 , Yt i
=
D
Et,i − Bt,i kE
t (i), Yt
E
9.4
The Complete Levinson Orthogonalization Procedure
31
= hEt,i , Yt i − hBt,i , Yt i kE
t (i)
= hEt,i , Et,i i − hBt,i , Et,i i kE
t (i)
E
= hEt,i , Et,i i I − kB
t (i)kt (i) ,
and
B
hBt+1,i+1 , Bt+1,i+1 i = hBt,i , Bt,i i I − kE
t (i)kt (i) ,
as well as
hEt,i+1 , Bt,i+1 i =
D
Et,i+1 , Yt−(i+2)
*
Yt −
=
i+1
X
E
+
Yt−j αt,i+1 (j), Yt−(i+2)
j=1
=
D
E
Yt , Yt−(i+2) −
i+1 D
X
E
Yt−j , Yt−(i+2) αt,i+1 (j).
j=1
9.4
The Complete Levinson Orthogonalization Procedure
Starting with the basic Levinson orthogonalization procedure and complementing it with
the order recursions relating the direct form parameters with the lattice parameters and
the inner product recursions we have the complete Levinson orthogonalization procedure
that evaluates both the lattice and direct form parameters given the autocorrelation sequence for {yt } or {Zt }
y
Tables 1 and 2 summarize the algorithm for St,p
, while the algorithm summaries for
Z
St,p are given in tables 3 and 4. Tables 1 and 3 give the time recursive structure of
the algorithm while tables 2 and 4 show how the algorithm is first started up in an
order buildup mode before it can be run in full order mode. In order buildup mode
the algorithm starts by evaluating a first order projection block. It then proceeds to
increment the order at each time step untill a full order is obtained, after which it runs
in the standard full order mode.
32
9
THE DYNAMIC LEVINSON ORTHOGONALIZATION PROCEDURE
for t = t0 , t0 + 1, t0 + 2, . . .
2
ket,0 k
kbt+1,0 k2
0-th lattice block
= kyt k2
= kyt k2
het,0 , bt,0 i = hyt , yt−1 i
kte (0)
=
ktb (0)
=
αt,1
=
het,0 , bt,0 i
kbt,0 k2
∗
het,0 , bt,0 i
ket,0 k2
β t+1,1
(kte (0))
= ktb (0)
ket,1 k2
=
kbt+1,1 k2
=
1 − kte (0)ktb (0) ket,0 k2
1 − kte (0)ktb (0) kbt,0 k2
for i = 1 : p − 1
i-th lattice block
i
X
∗
βt,i
(j) hyt , yt−j i
het,i , bt,i i = yt , yt−(i+1) −
j=1
kte (i)
=
ktb (i)
=
αt,i+1
=
β t+1,i+1
=
ket,i+1 k2
=
kbt+1,i+1 k2
=
het,i , bt,i i
kbt,i k2
∗
het,i , bt,i i
ket,i k2
αt,i
β t,i
e
− kt (i)
0
−1
−1
0
b
− kt (i)
αt,i
β t,i
e
b
1 − kt (i)kt (i) ket,i k2
1 − kte (i)ktb (i) kbt,i k2
y
Table 1: The Complete Levinson Orthogonalization Procedure for St,p
.
9.4
The Complete Levinson Orthogonalization Procedure
33
kbt0 ,0 k2 = kyt0 −1 k2
for t = t0 : t0 + p − 2
(Order buildup)
0-th lattice block
for i = 1 : t − t0
i-th lattice block
for t = t0 + p − 1, t0 + p, . . .
(Full order p)
0-th lattice block
for i = 1 : p − 1
i-th lattice block
y
Table 2: The Complete Levinson Orthogonalization Procedure for St,p
with Order
Buildup.
34
9
THE DYNAMIC LEVINSON ORTHOGONALIZATION PROCEDURE
for t = t0 , t0 + 1, t0 + 2, . . .
0-th lattice block
hEt,0 , Et,0 i = hYt , Yt i
hBt+1,0 , Bt+1,0 i =
hEt,0 , Bt,0 i =
hYt , Yt i
hYt , Yt−1 i
−1
kE
t (0)
= hBt,0 , Bt,0 i
kB
t (0)
= hEt,0 , Et,0 i
= kE
t (0)
= kB
t (0)
αt,1
β t+1,1
−1
hEt,1 , Et,1 i =
hBt+1,1 , Bt+1,1 i =
hEt,0 , Bt,0 i
H
hEt,0 , Bt,0 i
E
hEt,0 , Et,0 i I − kB
t (0)kt (0)
B
hBt,0 , Bt,0 i I − kE
t (0)kt (0)
for i = 1 : p − 1
i-th lattice block
hEt,i , Bt,i i =
i
X
βH
Yt , Yt−(i+1) −
t,i (j) hYt , Yt−j i
j=1
−1
kE
t (i)
=
hBt,i , Bt,i i
kB
t (i)
=
αt,i+1
=
β t+1,i+1
=
hEt,1 , Et,1 i hEt,i , Bt,i i
αt,i
β t,i
−
kE
t (i)
0
−I
0
−I
−
kB
t (i)
β t,i
αt,i
hEt,i+1 , Et,i+1 i =
hBt+1,i+1 , Bt+1,i+1 i =
−1
hEt,i , Bt,i i
H
E
hEt,i , Et,i i I − kB
t (i)kt (i)
B
hBt,i , Bt,i i I − kE
t (i)kt (i)
Z .
Table 3: The Complete Levinson Orthogonalization Procedure for St,p
9.4
The Complete Levinson Orthogonalization Procedure
35
hBt0 ,0 , Bt0 ,0 i = hYt0 −1 , Yt0 −1 i
for t = t0 : t0 + p − 2
(Order buildup)
0-th lattice block
for i = 1 : t − t0
i-th lattice block
for t = t0 + p − 1, t0 + p, . . .
(Full order p)
0-th lattice block
for i = 1 : p − 1
i-th lattice block
Z with Order
Table 4: The Complete Levinson Orthogonalization Procedure for St,p
Buildup.
36
REFERENCES
References
[1] Erwin Kreyszig: Introductory functional analysis with applications, John Wiley &
Sons, 1998.
[2] Lásloó Máté: Hilbert space methods in science and engineering, Adam Hilger, 1989.
[3] Peter Brockwell and Richard A. Davis: Time series: Theory and methods, Springer–
Verlag, 1987.
[4] G. W. Stewert: Introduction to Matrix Computations, Academic Press, 1973.
[5] G. H. Golub and C. F. Van Loan: Matrix Computations, John Hopkins University
Press, 1989.
[6] N. Levinson, The Wiener RMS Error Criterion in Filter Design and Prediction, J.
Math. Physics, vol. 25, pp. 261–278, 1947.
[7] E. Karlsson and M. H. Hayes, Least Squares Modeling of Linear Time–Varying
Systems: Lattice Filter Structures and Fast RLS Algorithms, IEEE Trans Acoustic,
Speech and Signal Processing, vol. ASSP-35, No. 7, pp. 994–1014, July 1987.
[8] S. Haykin: Adaptive Filter Theory, Third Ed., Prentice Hall, 1996.