Annealed Neural Dynamics for
Independent Component Analysis
Jiann-Ming Wu
Department of Applied Mathematics
National Donghwa University
Intelligent Numerical Computation
1
Outlines
1.
2.
3.
4.
5.
6.
Blind source separation
Independent component analysis
Previous works, JadeICA, fastICA
ICA using Potts models, PottsICA
A comparison on the three methods
Conclusions and future works
Intelligent Numerical Computation
2
Blind Source Separation (BSS)
2
2
0
0
-2
0
2
20
40
60
80
100
120
140
160
180
200
0
-2
0
2
20
40
60
80
100
120
140
160
180
200
20
40
60
80
100
120
140
160
180
200
20
40
60
80
100
120
140
160
180
200
20
40
60
80
100
120
140
160
180
200
20
40
60
80
100
120
140
160
180
200
0
-2
0
2
20
40
60
80
100
120
140
160
180
200
0
-2
0
2
0
-2
0
2
20
40
60
80
100
120
140
160
180
200
0
-2
0
2
0
-2
0
2
20
40
60
80
100
120
140
160
180
200
0
-2
0
2
0
-2
-2
0
20
40
60
80
100
120
140
Sources
160
180
200
0
Unknown
Mixing
Structure
Intelligent Numerical Computation
Observations
3
BSS by PottsICA
IEEE Trans. On Neural Networks(2001)
PottsICA
Observations
2
0
-2
0
2
20
40
60
80
100
120
140
160
180
200
20
40
60
80
100
120
140
160
180
200
0
20
40
60
80
100
120
140
160
180
200
-2
0
2
20
40
60
80
100
120
140
160
180
200
20
40
60
80
100
120
140
160
180
200
0
-2
0
2
0
-2
Recovered
sources
2
0
0
-2
0
Intelligent Numerical Computation
4
Blind source separation:
voice-music
BSS for voice-music separation
Intelligent Numerical Computation
5
BSS by ICA
ICA
Intelligent Numerical Computation
6
Blind Source separation : Fetal
EKG
ICA
Intelligent Numerical Computation
7
Previous works
FastICA:Helsinki University of Technology
JadeICA:by JF Cardoso
InformaxICA: Salk Institute
The others…
Intelligent Numerical Computation
8
The ICA problem
Unknown mixing structure:
Unkown statistical independent sources: S=
Observations:
Intelligent Numerical Computation
9
The goal of ICA
Recover independent sources by
such that
The joint distribution is as close as possible to the product of
the marginal distributions
Intelligent Numerical Computation
10
Intelligent Numerical Computation
11
Intelligent Numerical Computation
12
The unmixing problem
or
Intelligent Numerical Computation
13
Intelligent Numerical Computation
14
Then
Intelligent Numerical Computation
15
Partition the range of each output component
…
…
Intelligent Numerical Computation
16
Non-overlapping projection
the active state is
where
Intelligent Numerical Computation
17
Internal representations
Intelligent Numerical Computation
18
Output component i
Bin or state number
…
1
2
3
4
k… K-1 K
Membership vector
=[0
0 0 0 …
1…0 0]
the kth bit
Intelligent Numerical Computation
19
A mathematical programming
for non-overlapping projection
Intelligent Numerical Computation
20
Statistical mechanism
Intelligent Numerical Computation
21
Normalized histogram
Intelligent Numerical Computation
22
Discrete marginal entropy
Intelligent Numerical Computation
23
A mathematical programming for ICA
Intelligent Numerical Computation
24
Energy function for ICA
To minimize L is to solve a mixed integer and linear programming
Intelligent Numerical Computation
25
Annealed neural dynamics
Boltzmann distribution
Use mean field equations to find the mean configuration at each
Intelligent Numerical Computation
26
Derivation of mean field equations
Free energy by
Intelligent Numerical Computation
27
Mean field equations
Intelligent Numerical Computation
28
A hybrid of mean field annealing and
gradient descent method
MFE
Intelligent Numerical Computation
29
A hybrid of mean field annealing and
gradient descent method
Gradient descent method
Intelligent Numerical Computation
30
The PottsICA algorithm
Intelligent Numerical Computation
31
Numerical simulations
Artificial problem I
Independent
sources
Intelligent Numerical Computation
32
Numerical simulations
The normalized histogram of the five sources
Intelligent Numerical Computation
33
Numerical simulations
The mixed signals of the five sources using
a randomly generated mixing matrix
Intelligent Numerical Computation
34
Performance evaluations by Amari
Intelligent Numerical Computation
35
Performance evaluations
Intelligent Numerical Computation
36
Recovered signals by PottsICA
Intelligent Numerical Computation
37
Numerical simulations
The normalized histograms of the eight sources
Gaussian
Sub-Gaussian
Super-Gaussian
Intelligent Numerical Computation
38
Performance evaluations
Intelligent Numerical Computation
39
Performance evaluations
Intelligent Numerical Computation
40
Conclusions and Future works
1. Apply Potts encoding to ICA
2. PottsICA is better than JadeICA and FastICA
3. PottsICA is of collective decisions, PDP
4. PottsICA is potential for real world applications
Intelligent Numerical Computation
41