On the Uniqueness and Admissible
Solutions of Non-Negative Matrix
Factorization
David Brie
CRAN UMR 7039 - Université de Lorraine - CNRS
Winter school : Search for Latent Variables : ICA, Tensors and NMF
2 – 4 February 2015, Villard de Lans
1
Problem statement
Uniqueness and admissible solutions : basics
Uniqueness : geometric approach
MVES, Regularization, Multimodality
Conclusions
References
2
Problem statement

KEEP
CALM
AND
BE
POSITIVE
© 2015 KeepCalmStudio.com
3
Non-negative Matrix Factorization
– Data analysis
– Bioinformatics
– Dictionnary Learning
– Musical signal analysis
– Chemometrics (MCR)
– Hyperspectral imaging : remote sensing, microscopy, ...
4
Chemical mixture analysis by spectroscopy
– Each measured spectrum is a linear combination of pure component spectra
– Acquisition of multiple spectra in varying physico-chimical conditions
0.08
0.07
0.06
0.06
0.04
0.05
0.04
0.02
0
0
0.03
20
40
60
80
100
120
140
0.04
=
0.02
0
0
20
40
60
80
100
120
140
0.03
0.02
0.01

2/3
 1/2
1/3

1/3
1/2  ·
2/3
0.02
0.01
0
0
20
40
60
80
100
120
140
120
140
0.045
0.04
0.035
0.03
0.025
0.02
0.015
0.01
0
0
20
40
60
80
100
120
140
0.005
0
0
Instantaneous mixing model
X
ak (x)sk (λ)
X(x, λ) =
k
5
20
40
60
80
100
Hyperspectral Imaging
y
Unfolding
Instantaneous mixing model
X
ak (x, y)sk (λ)
X(x, y, λ) =
⇒
...
λ
k
x
λ
6
Hyperspectral Image Unmixing
s1(λ)
sp(λ)
s2(λ)
+
a1(x, y)
+
···
+
ap(x, y)
a2(x, y)
7
Mars Hyperspectral Image : OMEGA, IR, Image size
(300,128,174)
H2 O (Ice)
CO2 (Ice)
8
Dust
Instantaneous Mixing Model
X = A ST
with
X : (m × n) data matrix
A : (m × p) mixing matrix to estimate
S : (n × p) source matrix to estimate
Non-negativity constraint
A ≥ 0 et S ≥ 0
⇒ Non-Negative Matrix Factorization (NMF)
9
Algorithms
– Joint estimation : ALS, NMF, Regularized ALS, Regularized NMF, Bayesian
methods ...
– Geometrical methods : N-Findr, Minimum Volume Enclosing Simplex, ...
Considered Problems
– What are the conditions on S et A for having a unique NMF of X ?
– If the solution is not unique, what is the set of admissible solutions ?
– How to reduced the set of admissible solution ?
Assumptions
– Existence of the NMF
– The rank is known and rank+ = rank
– Noise-free situation
10
Uniqueness and admissible solutions : basics

KEEP
CALM
AND
BE
UNIQUE
© 2015 KeepCalmStudio.com
11
Indeterminacies
A and S are rank p matrices (full column rank)
T
T
A S = Ã S̃ ⇔ ∃ T(p × p) invertible such as
(
à = AT−1
S̃T = TST
What are the conditions on S, A and T under which à and S̃ are
non-negative ?
12
Scale indeterminacies
T = Diag(λ1 , ..., λp ) with λi > 0, ∀i = 1, ..., p, ⇒ S̃ ≥ 0 and à ≥ 0.
Normalization of A or S (for example Sum to One)
Order indeterminacies
T line permutation matrix ⇒ T−1 column permutation matrix
⇒ S̃ ≥ 0 and à ≥ 0.
The index is arbitrary
13
Admissible solution
A pair (A, S) is said to be an admissible solution if A et S are non-negative
matrices exactly yielding X.
Essential uniqueness
Let X be a matrix admitting a NMF given by X = A ST , the solution is said
essentially unique if the only indeterminacies are the scale and order
indeterminacies.
14
The 2 sources case
A = [a1 a2 ]
S = [s1 s2 ]
a1 , a2 : vectors of dimension (m, 1) ;
s1 , s2 : vectors of dimension (n, 1).
T(α, β) =
1−α
β
α
1−β
⇒ T−1 (α, β) =
1
1−α−β
α + β < 1 ⇒ no order indeterminacies and T invertible ;
p
P
tji = 1 ⇒ no scale indeterminacies.
i=1
15
1−β
−β
−α
1−α
.
Admissible solutions
(
S̃T = T(α, β) ST ;
à = A T−1 (α, β);
β ∈ [βmin , βmax ].
s2k
βmin = − min
k∈K2
s1k − s2k
aℓ1
βmax = min
ℓ
aℓ1 + aℓ2
α ∈ [αmin , αmax ]
s1k
αmin = − min
k∈K1
s2k − s1k
aℓ2
αmax = min
ℓ
aℓ1 + aℓ2
K1 = {k ∈ {1, ..., n}; s2k > s1k } .
K2 = {k ∈ {1, ..., n}; s1k > s2k }
16
Uniqueness
The NMF of X according to
X = A ST with A ≥ 0, S ≥ 0,
is unique if and only if ∃ (k1 , k2 , ℓ1 , ℓ2 ), with k1 6= k2 and ℓ1 6= ℓ2 , such as

s1k = 0, and s2k1 6= 0,


 1

s2k2 = 0, and s1k2 6= 0,

aℓ1 1 = 0, and aℓ1 2 6= 0,



aℓ2 2 = 0, and aℓ2 1 6= 0.
17
Example
Mimicking the spectroscopic analysis of chemical mixtures
Spectra of endmembers
Abundance
0.02
1
Component 2
Component 1
Component 1
Component 2
0.8
0.6
0.01
0.4
0.2
0
0
100
200
300
400
0
1
500
2
3
4
5
6
Measurements
Samples
Figure: Endmembers and abundance
X The uniqueness condition is not fulfilled !
18
7
8
9
10
Admissible Solutions
−0.0163 ≤ α ≤ 0.1898,
−0.0198 ≤ β ≤ 0.1924.
Abundance
Endmembers
0.02
1
0.8
0.6
0.01
0.4
0.2
0
0
100
200
300
400
0
1
500
2
3
Samples
4
5
6
Measures
Figure: Admissible Solutions
19
7
8
9
10
The case of p > 2 sources
A = [a1 · · · ap ] ,
S = [s1 · · · sp ]
Uniqueness Necessary Condition
If the NMF of X = A ST with A, S ≥ 0 is unique, then the following condition
are fulfilled :
(A1) ∀ (i, j), ∃ k such as : si (k) = 0 and sj (k) > 0.
(A2) ∀ (i, j), ∃ ℓ such as : ai (ℓ) = 0 and aj (ℓ) > 0.
20
Conditions (A1), (A2) are not sufficient


0.95
0
0.45 0.44
T
0
0.61 
S =  0.23 0.89
0
0.76 0.82 0.79



0.94
0.44
0.95
0
0.45
−1
1.00
0  ⇔ T =  −0.24
T =  0.23 0.89
0.22 −0.93
0
0.76 0.82


à = AT−1 ≥ 0, ∀A ≥ 0;








1 0 0 0.28





S̃T = TST =  0 1 0 0.61  ≥ 0.




0 0 1 0.39
21

−0.52
0.13 
1.09
A sufficient uniqueness condition
The NMF of X = A ST with A, S ≥ 0 is unique if the following conditions are
fulfilled :
(B1) ∃ a (p, p) monomial sub matrix
permutation, ST can be written as :

s11
0
s22
S= 0
0
0
of S i.e., after line and column
0
0
s33
(B2) ∃ a (p, p) monomial sub matrix of A
22
s14
s24
s34
s15
s25
s35

...
... 
...
A sufficient uniqueness condition on X
The NMF of X = A ST with A, S ≥ 0 is unique if the following conditions are
fulfilled :
(B1) ∃ a (p, p) monomial sub matrix of X
The uniqueness can be checked on the data directly ! but can be time
consuming on large data set.
23
Admissible Solutions
For p > 2, no analytical solution for the set of admissible solutions
Numerical approach
T̂ = arg min C(S̃, Ã)
T
T
T
where S̃ = TS , Ã = AT
C(S̃, Ã) =
−1
.
X X
j
2
min(s̃j,k , 0) +
X
i
k
min(ãi,j , 0)
2
!
Admissible solutions : set of matrices T resulting in a cost function = 0.
Starting from an initial solution, the admissible solution are determined by
Monte-Carlo simulations of the entries of T.
24
Example (p = 3)

1 − t1 − t2
T=
t3
t5
0.1
t1
1 − t3 − t4
t6

t2

t4
1 − t5 − t6
0.5
0.4
0.4
0.3
0.3
0.2
t6
t4
t2
0.2
0
0.1
0.1
0
0
−0.1
−0.1
0
0.1
0.2
t1
0.3
0.4
−0.1
−0.1
0
0.1
0.2
0.3
0.4
−0.1
−0.1
t3
Figure: Set of admissible solutions
25
0
t5
0.1
Uniqueness : a geometric approach

KEEP
CALM
AND
BE
GEOMETRIC
© 2015 KeepCalmStudio.com
26
Geometric approach
Convex cone : A subset C ⊆ Rn
+ is a convex cone if :
– v1 , v2 ∈ C ⇒ v1 + v2 ∈ C
– v ∈ C, λ ∈ R+ ⇒ λv ∈ C
The polyhedral cone generated by {v1 , . . . vm } is the set C such as :
X
xj vj , xj > 0}
C = {y|y =
j
Defining V = [v1 , . . . vm ] ∈ Rn×m
, we note :
+
C(V) = {y|y = Vx, x ∈ Rm
+}
A simplicial cone C(V) is a polyhedral cone having exactly p vertices where p
is the dimension of the column space of V
27
Remark : The previous polyhedral cone definition is a ”vertices based”
definition. There is an equivalent ”half-space intersection” definition. For
example, if V is invertible :
m
−1
C(V) = {y|y = Vx, x ∈ Rm
x ≥ 0}
+ } = {x ∈ R+ |V
Dual cone : The dual cone of C(V) is the polyhedral cone noted C(V)∗
defined by :
T
C(V)∗ = {x ∈ Rm
+ |V x ≥ 0}
Properties :
– if V is invertible, C(V)∗ = C(V−1 )
– if C1 ⊆ C2 , then C∗2 ⊆ C∗1
28
Geometric interpretation of the NMF
The NMF X = A ST , with A ≥ 0, S ≥ 0, indicates that each (column)
vectors of X belongs to C(A). In other words :
C(X) ⊆ C(A) ⊆ Rm
+
Dual point of view The NMF XT = S AT implies
C(XT ) ⊆ C(S) ⊆ Rn
+
C(S)
Rn
+
C(XT )
29
Reduced dimension working space Rn
+ ∩ col(S)
Rn+ ∩ col(S)
s2
s1
e1
e2
Rn+
Figure: Working space, natural base
n
T
Remark : as rank(XT ) = rank(S) = p, Rn
+ ∩ col(S) = R+ ∩ col(X )
30
T
Remark : Rn
+ ∩ col(X ) is not necessarily a simplicial cone of order p
R3+
R4+
col(XT )
col(XT )
b
b
b
b
b
b
Figure: The Working space in two different cases
This is connected to the notion of non-negative rank !
31
Sum to One constraint ⇒ affine projection
e3
1
1
1
e1
Figure: Projection affine
32
e2
Uniqueness of the NMF : Chen’s approach
The NMF X = A ST , with A ≥ 0, S ≥ 0, is unique iff the simplicial cone C of
order p such as : C(XT ) ⊆ C is unique.
e1
e1
s2
s1
b
b
s1
b
b
b
b
e3
b
b
b
b
b
b
b
b
b
Rn+ ∩ col(S)
Rn+ ∩ col(S)
b
b
C
b
b
b
b
b
b
b
b
b
C
b
b
b
b
b
b
b
b
b
s3
s2
e2
e2
Figure: Example of non-unique NMF
– This does not provide any numerical condition !
– This is the starting point of Donoho’s sufficient condition
33
b
s3
e3
Sufficient condition for a unique NMF
(e1 , · · · , ep ) : natural base of Rn
+ ∩ col(S).
s1 = e1
s1 = e1
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
s2 = e2
s3 = e3
b
b
s2 = e2
Moussaoui
s3 = e3
Donoho
Figure: Two sufficient uniqueness conditions
34
Uniqueness of the NMF : Thomas’ approach
If rank(X) = p, the NMF X = A ST , with A ≥ 0, S ≥ 0, is unique iff the only
simplicial cone C of order p that satisfies C(ST ) ⊆ C ⊆ C(AT )∗ is C = Rp+
Figure: A unique (left) and a non unique (right) NMF
– Starting point of Laurberg and Huang’s sufficient uniqueness conditions
– The main difference between Thomas and Chen’s approaches :
col(ST ) = Rp (Thomas) while col(S) ⊆ Rn (Chen)
– Laurberg’s condition ⇒ Donoho’s condition (equivalent if
rank+ (X) = rank(X) ?)
35
Uniqueness of the NMF and latent variable sparsity
All the sufficient uniqueness conditions implies that :
– the sources coincide with the vector of the natural base ⇒ each source
includes at least p − 1 null values ;
– Each mixing coefficient contains at least p − 1 null values.
⇒ latent variable sparsity
It is possible to find (pathological) counter-examples such as, even if the
previous sparsity conditions are fulfilled, the corresponding NMF is not unique.
⇒ Generic uniqueness ?
36
MVES, Regularization, Multimodality

KEEP
CALM
AND
BE
REGULARIZED
© 2015 KeepCalmStudio.com
37
Minimum Volume Enclosing Simplex
e1
ŝ1
b
b
b
b
b
b
b
b
ŝ2
b
b
b
b
b
b
b
b
b
b
b
b
ŝ3
e3
e2
Geometrical approach
Identifiability : when do the vertices of the MVES correspond to the sources ?
38
Pure Pixel Condition
s1
s1
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
b
s4
b
b
b
b
b
b
b
b
b
b
b
b
b
b
s2
b
b
s3
s2
b
s3
Remark : the pure pixel condition is a condition on the mixing coefficient.
There is a not often mentionned equivalent condition on the sources
(Simplisma).
39
Another sufficient identifiability condition
s1
s1
MVES
d = sup(di) < 1/3
b
b
b
b
b
b
b
b
b
s2
b
b
s2
s3
A formal proof only for the case p = 3 !
40
b
s3
Regularized NMF
Constrained optimization
min
A≥0,S≥0
kX − AST k2F
s. c.
C1 (A) ≤ ε1 , C2 (S) ≤ ε2
C1 (A) et C2 (S) are cost function enforcing additional properties
⇒ reducing the set of admissible solutions
Lagrangian formulation
C(A, S) = kX − AST k2F + λ1 C1 (A) + λ2 C2 (S)
(Â, Ŝ) = arg
min
A≥0,S≥0
41
C(A, S)
NMF with sparsity constraints
min
(ak ≥0,sk ≥0)
kX −
X
ak sTk k2F + λ1
X
kak k1 + λ2
ksk k1
(P1)
k
k
k
X
Problem (P1) is equivalent to (Papalexakis) :
X p
X
2 λ1 λ2 kak k1 ksk k1
ak sTk k2F +
min
kX −
(ak ≥0,sk ≥0)
k
k
This means that :
– it is impossible to individually controlled the sparsity of A and S
– uniqueness cannot be guaranteed by imposing the sparsity on both A and S !
42
NMF with sparsity constraints
min
(A≥0,S≥0)
kX − ASk2F + λ1 kAk1 + λ2 kSk1
min
(A≥0,S≥0)
kX − ASk2F + λkSk1
b
b
b
b
b
min
(A≥0,S≥0)
kX − ASk2F
b
b
b
min
(A≥0,S≥0)
b
Figure: Different sparse solutions
43
kX − ASk2F + λkAk1
Multimodality
Modalité 2
Modalité 1
Modalité 1 et 2
Figure: Reducing the set of admissible by multimodality
44
Example : polarized Raman spectroscopy
Figure: Data
Figure: Data acquisition
45
NMF Results
0.16
0.14
0.16
0.14
0.12
0.14
0.1
0.12
0.12
0.1
0.1
0.08
0.08
0.08
0.06
0.06
0.06
0.04
0.04
0.04
0.02
0.02
0
100
200
300
400
500
600
wavenumber (cm−1)
700
800
0
100
0.02
200
300
400
500
600
wavenumber (cm−1)
700
0
100
800
0.5
200
300
400
500
600
wavenumber (cm−1)
700
800
0.4
0.35
0.4
0.3
0.25
0.3
0.2
0.2
0.15
0.1
0.1
0.05
0
0
50
100
χ(degrees)
150
0
0
200
0.5
50
χ(degrees)
100
150
200
100
150
200
0.45
0.4
0.4
0.35
0.3
0.3
0.25
0.2
0.2
0.15
0.1
0.1
0.05
0
0
DX = AX S
50
100
χ(degrees)
150
DY = AY S
46
0
0
200
DX
DY
50
χ(degrees)
=
AX
AY
S
Some key points on NMF Uniqueness
In general
– Uniqueness is not guaranteed (hyperspectral imaging)
– Uniquenes requires the sparsity of the latent variables – Set of admissible
solutions can be obtained numerically
– Additional constraints, regularization
Geometric approach
– a very intuitive and effective interpretative framework
– a geometric interprétation of sparsity constraints
– Algorithm design
– Source positiveness is not easy to incorporate in the MVES approach
47
References

KEEP
CALM
AND
BE
BLIOGRAPHY
© 2015 KeepCalmStudio.com
48
J. C. Chen.
Nonnegative rank factorisation of nonnegative matrices.
Linear Algebra and its Applications, 62 :207–217, 1984.
J.E. Cohen and U.G. Rothblum.
Non-negative ranks decompositions and factorizations of non-negative
matrices.
Linear Algebra and its Applications, 190 :149–168, 1993.
D. Donoho and V. Stodden.
When does non-negative matrix factorization give a correct decomposition
into parts ?
In Advances in Neural Information Processing Systems 16, Cambridge,
United States, 2003. MIT Press.
P.J. Gemperline.
Computation of the range of feasible solutions in self modeling curve
resolution algorithms.
Analytical Chemistry, 71 :5398–5404, 1999.
49
N. Gillis and F. Glineur.
On the Geometric Interpretation of the Nonnegative Rank.
Linear Algebra and its Applications, 437 :2685–2712, 2012.
K. Huang, N. D. Sidiropoulos, A. Swami, A.
Non-Negative Matrix Factorization Revisited : Uniqueness and Algorithm
for Symmetric Decomposition.
IEEE Transactions on Signal Processing, 62 :211–224, 2014.
P.O. Hoyer.
Non-negative sparse coding.
In Proceedings of IEEE Workshop on Neural Networks for Signal
Processing (NNSP’2002), pages 557–565, 2002.
P.O. Hoyer.
Nonnegative matrix factorization with sparseness constraints.
Journal of Machine Learning Research, 5 :1457–1469, 2004.
50
H. Laurberg, M. G. Christensen, M. D. Plumbley, L. K. Hansen, S. H.
Jensen, S. H. .
Theorems on positive data : On the uniqueness of NMF.
Computational intelligence and neuroscience, 2008.
W. Lawton and E. Sylvestre.
Self-modeling curve resolution.
Technometrics, 13 :617–633, 1971.
T.-L. Markham.
Factorizations of non-negative matrices.
Proceedings of American Mathematical Society, 32 :45–47, 1972.
S. Miron , M. Dossot, C. Carteret , S. Margueron, D. Brie.
Joint processing of the parallel and crossed polarized Raman spectra and
uniqueness in blind nonnegative source separation.
Chemometrics and Intelligent Laboratory Systems, 105 : 7–18, 2011.
51
S. Moussaoui, D. Brie, and J. Idier.
Non-negative source separation : Range of admissible solutions and
conditions for the uniqueness of the solution.
In proceedings of IEEE International Conference on Acoustics, Speech, and
Signal Processing (ICASSP’2005), Philadelphia, USA, March 2005.
S. Moussaoui, D. Brie, C. Carteret and A. Mohammad-Djafari.
Bayesian analysis of spectral mixture data using Markov Chain Monte
Carlo Methods.
Chemometrics and Intelligent Laboratory Systems, 81 : 137–148 , 2006.
S. Moussaoui, H. Hauksdóttir, F. Schmidt, C. Jutten, J. Chanussot, D.
Brie, S. Douté, J. A. Benediksson.
On the Decomposition of Mars Hyperspectral Data by ICA and Bayesian
Positive Source Separation.
Neurocomputing, 71 : 2194–2208, 2008.
W.S.B. Ouedraogo, A. Souloumiac, M. Jaidane, C. Jutten.
Non-Negative Blind Source Separation Algorithm Based on Minimum
Aperture Simplicial Cone.
IEEE Transactions on Signal Processing, 62 :376–389, 2014.
52
E. E. Papalexakis, N. D. Sidiropoulos, R. Bro.
From K-means to higher-way co-clustering : multilinear decomposition
with sparse latent factors.
IEEE Transactions on Signal Processing, 61 :493-506, 2013.
P. Paatero and U. Tapper.
Positive matrix factorization : A non–negative factor model with optimal
utilization of error estimates of data values.
Environmetrics, 5 :111–126, 1994.
M.D. Plumbley.
Conditions for nonnegative independent component analysis.
IEEE Signal Processing Letters, 9(6) :177–180, 2002.
A.K. Smilde, H.C.J. Hoefsloot, H.A.L. Kiers, S. Bijlsma, and H.F.M.
Boelens.
Sufficient condition for unique solutions within a certain class of curve
resolution models.
Journal of Chemometrics, 15 :405–411, 2001.
53
R. Tauler.
Calculation of maximum and minimum band bounderies of feasible
solutions for species profiles obtained by multivariate curve resolution.
Journal of Chemometrics, 15 :627–646, 2000.
L.-B. Thomas.
Rank factorizations of nonnegative matrices.
SIAM Review, 16 :393–394, 1974.
J.M. Van den Hof and J.H. Van Schuppen.
Positive matrix factorization via extremal polyhedral cones.
Linear Algebra and its Applications, 293 :171–186, 1999.
54