Orthogonal
Transforms
Fourier
Review
• Introduce the concepts of base functions:
– For Reed-Muller, FPRM
– For Walsh
•
•
•
•
•
Linearly independent matrix
Non-Singular matrix
Examples
Butterflies, Kronecker Products, Matrices
Using matrices to calculate the vector of spectral
coefficients from the data vector
Our goal is to discuss the best
approximation of a function using
orthogonal functions
Orthogonal
Functions
Orthogonal Functions
Note that these are arbitrary functions,
we do not assume sinusoids
Illustrate it for Walsh and RM
Mean
Square
Error
Mean Square Error
Important result
• We want to minimize this kinds of
errors.
• Other error measures are also used.
Unitary
Transforms
Unitary Transforms
• Unitary Transformation for 1-Dim. Sequence
– Series representation of
{u (n),0  n  N  1}
N 1
v  Au  v(k )   a(k , n)u (n), 0  k  N  1
n 0
N 1
u  A* v  u (n)   a * (k , n)v(k ), 0  n  N  1
n 0
where A1  A*T (unitary matrix)
– Basis vectors :
a*k  {a* (k , n), 0  n  N  1}T
– Energy conservation :
v  Au  || v ||2 || u ||2
N 1
Here is the
proof
N 1
( || v ||   | v(k ) |  u A Au  u u   | u (n) |2 || u ||2 )
2
2
k 0
*T
*T
*T
n 0
• Unitary Transformation for 2-Dim. Sequence
– Definition :
N 1 N 1
v(k , l )   u (m, n)ak ,l (m, n), 0  k , l  N  1
m 0 n 0
N 1 N 1
u (m, n)   v(k , l )ak*,l (m, n), 0  m, n  N  1
k 0 l 0
*
{
a
– Basis images :
k ,l (m, n)}
– Orthonormality and completeness properties
• Orthonormality :
• Completeness :
N 1 N 1
*
'
'
a
(
m
,
n
)
a
(
m
,
n
)


(
k

k
,
l

l
)
 k ,l
k ' ,l '
m 0 n 0
N 1 N 1
*
' '
'
'
a
(
m
,
n
)
a
(
m
,
n
)


(
m

m
,
n

n
)
 k ,l
k ,l
k 0 l 0
• Unitary Transformation for 2-Dim. Sequence
– Separable Unitary Transforms
ak ,l (m, n)  ak (m)bl (n)
N 1 N 1
v(k , l )   ak (m)u (m, n)al (n)  V  AUAT
m 0 n 0
N 1 N 1
u (m, n)   ak* (m)v(k , l )al* (n)  U  A*T VA*
k 0 l 0
• separable transform reduces the number of multiplications and
additions from O( N 4 ) to O( N 3 )
– Energy conservation
N 1 N 1
N 1 N 1
m 0 n 0
k 0 l 0
2
2
|
u
(
m
,
n
)
|

|
v
(
k
,
l
)
|


Properties of Unitary Transform
transform
Consider v  Au, then R v  AR u (A * )t
Covariance
matrix
A 2  1 zero mean vecto r u is trasforme d
1 3
v 
2 1
1 
u
3
where
1 ρ 
Ru  

ρ
1


ρ
1  3 ρ

2
2

then R v  
 ρ
1 3 ρ 
2
2

 σ 2u (0)  1  σ 2u (1) ; total energy is equally distribute d
but σ 2v (0)  1  3 ρ
 σ 2v (1)  1  3 ρ
2
if ρ  0.95 , then 91% energy in v(0)
2
and the correlatio n between v(0) and v(1) is
Ε[v(0)v(1) ]
ρ v (0,1) 
 0.83
σ v (0) σ v (1)
 energy compaction and decorrelat ion
Example of arbitrary
basis functions being
rectangular waves
This determining first
function determines next
functions
0
1
Small error with
just 3 coefficients
This slide shows four
base functions
multiplied by their
respective coefficients
This slide shows that
using only four base
functions the
approximation is quite
good
End of example
Orthogonality
and
separability
Orthogonal and separable Image
Transforms
Extending general transforms to 2dimensions
Forward
transform
inverse
transform
separable
Fourier
Transforms in
new notations
1. We emphasize generality
2. Matrices
Fourier Transform
separable
Extension of Fourier Transform to
two dimensions
Discrete Fourier Transform (DFT)
The DFT of u( n ) ,
N 1
v(k)   u (n) wN
n  0,1,2, ,N-1
is
k  0,1, , N  1
kn
n 0
where
wN  e
New
notation
 j 2N
 not scaled properly t o be unitary tr ansform
1
v(k ) 
N
1
u ( n) 
N
N 1
 u (n)wN ,
k  0,1, ,N  1
kn
n 0
N 1
 v(k )wN
k 0
 kn
,
n  0,1, ,N  1
For N  4
1 1
 v ( 0) 
1 w1
 v(1) 
4

1
v(2) 2 1 w42



3
1
w
v
(
3
)


4

1
w42
w40
w42
1  u (0) 
w43   u (1) 
 v  Fu
2
w4  u (2)

1 
w4  u (3) 
 1 kn   1  j 2N kn 
F
wN   
e

 N
  N

 easy to handle mathematic ally
Fast Algorithms for Fourier
Transform
1. F 1  F *
(F * )t  F *  F 1 , since F(F * )t  I
2. Fast algorithm
Task for students:
Ο( N 2 )  O( N log 2 N ), where N  2 n
2
v(0)  1 w 0 0  1 0 w40 0  u (0) 

 v ( 2)  

0 
u
(
1
)
1
w
0
0
0
1
0
w
4 




 v(1)  0 0 1 w14  1 0 w42 0  u (2)

 

3 
2 
v
(
3
)
u
(
3
)

 0 0 1 w4  0 1 0 w4  

direct :16 multiplica tions , 12 additions
0
4
0
4
factorizat ion : 4 multiplica tions , 8 additions
Draw the butterfly for
these matrices,
similarly as we have
done it for Walsh and
Reed-Muller
Transforms
Pay attention to
regularity of kernels
and order of columns
corresponding to
factorized matrices
Fast Factorization Algorithms are
general and there is many of them
In general, if A  A1 A 2  A p : factorizat ion
where A i , i  1,2, , p( N ) are matrices with
just a few nonzero elements, say r  N ,
then v  A x requires rpN operations .
 Fourier, Cosine, Sine transform
• 1-dim. DFT (cont.)
– Calculation of DFT : Fast Fourier Transform Algorithm
(FFT)
• Decimation-in-time algorithm
N 1
G ( k )   g ( n )e
Derivation of
decimation in time
kn
 j 2
N
n 0
N

2 1
 g ( 2 n )e
 j 2
k 2 n 
N
n 0
N

2 1
 g ( 2 n )e
n 0
N
2 1
  g (2n  1)e
 j 2
k  2 n 1
N
n 0
 j 2
kn
N
2
k
 j 2   j 2 N

N
2
   g (2n  1)e
e
n 0 

N
2 1
kn
kn
kn
k N 2 1
 j 2
 j 2
 N 2 1

j
2

N
N
N
2
2
e
  g (2n  1)e
, 0  k  N 1
  g ( 2 n )e
2
 n 0
n 0
  N 1
kn
kn
k  N 2  N 2 1
 j 2
 j 2
2

j
2

N
N
 g ( 2 n )e
N
2
2
e
  g (2n  1)e
,N kN

 n 0
2
n 0
Decimation in
Time versus
Decismation in
Frequency
• 1-dim. DFT (cont.)
– FFT (cont.)
• Decimation-in-time algorithm (cont.)
Butterfly for
Derivation of
decimation in time
Please
note
recursion
• 1-dim. DFT (cont.)
– FFT (cont.)
• Decimation-in-frequency algorithm (cont.)
N 1
G ( k )   g ( n )e
 j 2
kn
N
n 0
N

2 1
 g ( 2 n )e
 j 2
kn
N

n 0
N 1
 g ( n )e
 j 2
n N 2
k N
 j 2


N 2
N
   g ( n )  g ( n  2 )e
n 0 
N

2 1
 g (n)  g (n 
N
2 1
n 0
kn
N
Derivation of
Decimation-infrequency
algorithm
N
2

) 1 e
k
  j 2 kn
N
e

 j 2
kn
N
k n
 j 2
 N 2 1
N
2
, k  2k 
  g ( n)  g ( n  N 2 )e

  Nn 01
k n
n
 j 2
2
 j 2
N

N
2

g ( n)  g ( n  N 2 )e
e
, k  2k   1


 n 0
• 1-dim. DFT (cont.)
– FFT (cont.)
Decimation in frequency
butterfly shows recursion
• Decimation-in-frequency algorithm (cont.)
Conjugate Symmetry of
DFT
– For a real sequence, the DFT is conjugate symmetry
v * ( N  k )  v( k )
N  
N 

 v(0) Rev(k ) , k  1,  ,  1 , Imv(k ) , k  1,  ,  1
2  
2 

and v( N2 ) completely defines the DFT
 do not increase the dimensiona lity
Use of Fourier
Transforms
for fast
convolution
Calculations for circular matrix
Let us consider a circular matrix
 c 0 c1
c
 N 1 c 0
C 

 
 c1 
  c N 1 

c1  c N  2 

 


 
  c 0 
Let w *N  exp( j 2N )
  is one of the N distinct
then w
w 
* kN
N
* k
N
 1 , k  0,1,, N  1
roots of unity.
By
multiplying
 k   c0  c1 w

    c
w   k   c w   c w     c
 c  c w     c
w   k   c  c w     c
* k
N
* k
N
* k
N
0
N 1
* 2k
N
N 2
* 2k
N
 c2 w
* 2k
N
1
* k
N
0
N 1
* k
N
w 
w 

w 

w 
 N 1k
N 1
*
N
N 1
* Nk
N
N 2
*
N
N 3
*
N
N 1k
N 1k


w 
*
N
 N 1k
 k   c1  c2 w

* k
N
 
   c0 w
*
N
 N 1k
In a matrix form,
1


 w* k   c0 c1
N
c


c
N

1
0
* 2k

 k  wN

  




c1 c2
 N 1k

*
 wN

 
 
 




1


c N 1  

* k
w
N

c N  2  
* 2k
 wN

 



c0  *  N 1k 
 wN

 
 
 
  (k )w k   Cw k   x  Ax 
*
*
  (k ) is an eigen valu e and w k , is an eigen vect or of C
*
there are N distinct eigen vect ors w k , thus
*
w *  Cw *
In matrix form next slide
W * =
Cw*
1 1
1 w1

1 w2

where w *  1
1


1 

1
w
2
w4
 0  0
 0
 1

Λ 
0


 
 0
0
 
C  w  w
*
 
or   w
* 1
w = Cw
*
*
1  1
 


1

 ; inverse Fourier tr ansform matrix( F )





 
0 
0 
0 

  ; diagonal matrix


 
 0   N  1
1
* 1
 F 1 AF
Cw *
 circular matrix C is diagonaliz ed by DFT
linear convolutio n :
n 1
Here is the formula for linear
convolution, we already discussed for
1D and 2D data, images
y n    x j hn  j 
j 0
assume :
x0 , x1, x2 , x3, x4  N  5
h0 , h1, h2 , h3
L4
M  N  L 1  8
y 0   x0 h0 
y 1  x0 h1  x1h0 
y 2   x0 h2   x1h1  x2 h0 

Linear convolution can be presented in
matrix form as follows:
 y 0  h0 
 y 1   h1

 
 y 2  h2 

 


y
3

   h3
 y 4   0

 
 y 5  0
 y 6   0

 
 y 7   0
8 1
0
0
h0  0
h1 h0 
0
0
0
h2  h1 h0 
h3 h2  h1
0
h3 h2
0
0
0
0
8 5
h3
0
0 
0 
 x0 
0 



x
1


0 
 x2  ; y  H x
h0  

  x3
h1 
 x4 

h2 

h3
5 1
Circular convolutio n
N 1
y n    x j hn  j N , n  0,1,  N  1
j 0
where hn N  hnmod N 
For N  4,
 y 0   h0 
 y 1   h1


 y 2  h2 

 


y
3

  h3
h3 h2  h1   x0 
h0  h3 h2   x1 
; y  Hc x
h1 h0  h3  x2 


h2  h1 h0   x3
In general
h N  1
 h0
 h1
h0

Hc 
 


h N  1 h N  2
 h1 
 h2
  

 h0
As we see, circular
convolution can be also
represented in matrix form
remember
 
λk   c0  c1 w
* k
N
 
* 2k
N
 c2 w
 
   c N 1 w
*
N
 N 1k
Thus the first row of H c is h j , since h j  is a periodic sequence
N 1
H k    h j  exp  2N kj
j 0

  N 1
 h j  exp 
j 0
 2
N
kj ; sum from 0 to   N  1 is equal to 0 to  N-1
N 1
  h j  exp  j 2N kj
j 0
 H k 
 eigen valu e of H c is the DFT of hi 
Important
result
now,
y  Hc x
and
F 1 ΛF  H c
y  F 1 ΛF x
where
0

0
 H 0 

 0



H
1


Λ
 

0

0


0
 H  N  1
 0
1F x ; forward DFT of x
2 Λ F x  ; H k  X k 
3F 1 ΛF x  ; inverse DFT of H k  X k 
 fast algorithm for DFT
Inverse DFT of
convolution
•Thus we derived a fast
algorithm for linear
convolution which we
illustrated earlier and
discussed its
importance.
linear convolutio n
 xi , i  0,1,  , N  1

 hi , i  0,1,  L  1
y i   xi   hi , i  0,1,  ,  N  L  2 
let M  N  L  2  2 m
then
 xi ,
~
x i   
 0 ,
hi ,
~
h i   
 0 ,
i  0,1,  , N  1
i  N ,  , M  1 ; zero padding 
i  0,1,  , L  1
i  L,  , M  1
; zero padding 
then perform ~
y  Hc ~
x
 the first  N  L  2  components of ~
y are exactly
equal to the  N  L  2  components of y
 fast algorithm for linear convolutio n filtering

•This result is very
fundamental since it
allows to use DFT with
inverse DFT to do all
kinds of image
processing based on
convolution, such as
edge detection,
thinning, filtering, etc.
2-D DFT
2-D DFT
2  D unitary DFT is
1 N 1 N 1

km ln
v
(
k
,
l
)

u
(
m
,
n
)
w
0  k, l  N 1

N wN ,

N m 0 n 0


N 1 N 1
1
 km ln
u (m, n) 
v
(
k
,
l
)
w
0  m, n  N  1

N wN ,

N k 0 l 0
1.V  FUF since F  F t
U  F * VF *
vector  space notation
v  Fu and
where
u  F* v
F F  F
2.separable  2 N 1 - D DFT 2 NO Nlog 2 N 
3.conjugat e symmetry, if u(m,n) is real
vk , l   v*  N  k , N  l , 0  k , l  N  1
4. 2  D circular convolutio n theorem
y m, n   hm, n  u m, n , M  N  L  1
M M
L L
N N
Circular
convolution
works for 2D
images
hm, n , 0  m, n  N  1
~
h m, n   
 0 , N  m, n  M  1
u m, n , 0  m, n  L  1
~
u m, n   
 0 , N  m, n  M  1
~
~
~
Y k , l   H k , l U k , l 
1 ~
~
y m, n   DFT Y k , l  , 0  m, n  M  1 N+L-1

N+L-1

M
M
Circular convolution works for 2D images
So we can do all kinds of edge-detection, filtering etc
very efficiently
• 2-Dim. DFT (cont.)
– example
(a) Original Image
(b) Magnitude
2  dim DFT of a 512  512 Lena image
(c) Phase
• 2-Dim. DFT (cont.)
– Properties of 2D DFT
• Separability
F (k , l ) 
N 1
W

N
1
m0
f (m, n) 
km
N
 1

 N
N 1

n 0
 km  1
WN 
N k 0
 N
1
N 1


f (m, n)W ,

N 1
 F (k , l )W
l 0
k , l  0,, N  1
ln
N
 ln
N

,

m,n  0,, N  1
• 2-Dim. DFT (cont.)
– Properties of 2D DFT (cont.)
• Rotation
m  r cos , n  r sin  ,
k   cos , l   sin 
f (r,   0 )  F ( ,    0 )
(a) a sample image
(b) its spectrum
(c) rotated image
(d) resulting spectrum
• 2-Dim. DFT (cont.)
– Properties of 2D DFT
• Circular convolution and DFT
f (m, n)  g (m, n) 
 f ( p, q) g
p
C
(m  p, n  q)
q
f (m, n) * g (m, n)  F (k , l )G(k , l )
f (m, n) g (m, n)  F (k , l ) * G(k , l )
• Correlation
R fg (m, n)  f (m, n)  g (m, n) 

p
f * ( p, q) g C (m  p, n  q)
q
f (m, n)  g (m, n)  F * (k , l )G (k , l )
f * (m, n) g (m, n)  F (k , l )  G (k , l )
• 2-Dim. DFT (cont.)
Three ways of
calculating 2-D
DFT
– Calculation of 2-dim. DFT
• Direct calculation
N1 1 N 2 1
F (k , l )    f (m, n)e
 2 
 2 
 j
 km  j 
 ln
 N 
 N 
e
m 0 n 0
– Complex multiplications & additions :
• Using separability
,
0  k  N1  1,
0  l  N2 1
 ( N12 N 22 )
 2 
 2 

 j
 ln   j 
 km
0  k  N1  1,
 N  
 N 

F (k , l )    f (m, n)e
e
,


0  l  N2 1
m 0  n 0

N1 1 N 2 1
– Complex multiplications & additions :
• Using 1-dim FFT
 ( N1N2 ( N1  N2 ))
– Complex multiplications & additions : ???
Questions to Students
1.
2.
You do not have to remember derivations but you have to understand the
main concepts.
Much software for all discussed transforms and their uses is available on
internet and also in Matlab, OpenCV, and similar packages.
1. How to create an algorithm for edge detection based
on FFT?
2. How to create a thinning algorithm based on DCT?
3. How to use DST for convolution – show example.
4. Low pass filter based on Hadamard.
5. Texture recognition based on Walsh