ICA
Alphan Altinok
Outline
 PCA
 ICA





Foundation
Ambiguities
Algorithms
Examples
Papers
PCA & ICA
 PCA
 Projects d-dimensional data onto a lower dimensional subspace
in a way that is optimal in Σ|x0 – x|2.
 ICA
 Seek directions in feature space such that resulting signals show
independence.
PCA
 Compute d-dimensional μ (mean).
 Compute d x d covariance matrix.
 Compute eigenvectors and eigenvalues.
 Choose k largest eigenvalues.
 k is the inherent dimensionality of the subspace governing the
signal and (d – k) dimensions generally contain noise.
 Form a d x k matrix A with k columns of eigenvalues.
 The representation of data by principal components
consists of projecting data into k-dimensional subspace by
x = At (x – μ).
PCA
 A simple 3-layer neural network can form such a
representation when trained.
ICA
 While PCA seeks directions that represents data best in a
Σ|x0 – x|2 sense, ICA seeks such directions that are most
independent from each other.
 Used primarily for separating unknown source signals
from their observed linear mixtures.
 Typically used in Blind Source Separation problems.
ICA is also used in feature extraction.
ICA – Foundation
 q source signals s1(k), s2(k), …, sq(k)




with 0 means
k is the discrete time index or pixels in images
scalar valued
mutually independent for each value of k
 h measured mixture signals x1(k), x2(k), …, xh(k)
 Statistical independence for source signals
 p[s1(k), s2(k), …, sq(k)] = П p[si(k)]
ICA – Foundation
 The measured signals will be given by
 xj(k) = Σsi(k)aij + nj(k)
 For j = 1, 2, …, h, the elements aij are unknown.
 Define vectors x(k) and s(k), and matrix A
 Observed:
x(k) = [x1(k), x2(k), …, xh(k)]
 Source:
s(k) = [s1(k), s2(k), …, sq(k)]
 Mixing matrix: A = [a1, a2, …, aq]
 The equation above can be stated in vector-matrix form
 x(k) = As(k) + n(k) = Σsi(k)ai + n(k)
Ambiguities with ICA
 The ICA expansion
 x(k) = As(k) + n(k) = Σsi(k)ai + n(k)
 Amplitudes of separated signals cannot be determined.
 There is a sign ambiguity associated with separated
signals.
 The order of separated signals cannot be determined.
ICA – Using NNs
 Prewhitening – transform input vectors x(k) by
 v(k) = V x(k)
 Whitening matrix V can be obtained by NN or PCA
 Separation (NN or contrast approximation)
 Estimation of ICA basis vectors (NN or batch approach)
ICA – Fast Fixed Point Algorithm
 FFPA converges rapidly to the most accurate solution
allowed by the data structure.
ICA – Example
ICA – Example
ICA – Example
ICA – Example
ICA – Example
ICA – Example
 BSS of recorded speech and music signals.
speech / music
speech / speech
speech / speech in
difficult
environment
mic1
mic1
mic1
mic2
mic2
mic2
separated1
separated1
separated1
separated2
separated2
separated2
http://www.cnl.salk.edu/~tewon/ica_cnl.html
ICA – Example
 Source images
 separation demo
http://www.open.brain.riken.go.jp/demos/researchBSRed.html
ICA – Papers
 Hinton – A New View of ICA
 Interprets ICA as a probability density model.
 Overcomplete, undercomplete, and multi-layer non-linear ICA
becomes simpler.
 Cardoso – Blind Signal Separation, Statistical Principles





Modelling identifiability.
Contrast functions.
Estimating functions.
Adaptive algorithms.
Performance issues.
ICA – Papers
 Hyvarinen – ICA Applied to Feature Extraction from
Color and Stereo Images
 Seeks to extend ICA by contrasting it to the processing done in
neural receptive fields.
 Hyvarinen – Survey on ICA
 Lawrence – Face Recognition, A Convolutional Neural
Network Approach
 Combines local image sampling, a SOM, and a convolutional
NN that provides partial invariance to translation, rotation,
scaling, and deformations.
ICA – Papers
 Sejnowski – Independent Component Representations for
Face Recognition
 Sejnowski – A Comparison of Local vs Global Image
Decompositions for Visual Speechreading
 Bartlett – Viewpoint Invariant Face Recognition Using
ICA and Attractor Networks
 Bartlett – Image Representations for Facial Expression
Coding
ICA – Links
 http://sig.enst.fr/~cardoso/
 http://www.cnl.salk.edu/~tewon/ica_cnl.html
 http://nucleus.hut.fi/~aapo/
 http://www.salk.edu/faculty/sejnowski.html