A Fast-Nonegativity-Constrained
Least Squares Algorithm
R. Bro, S. D. Jong
J. Chemometrics,11,393-401, 1997
By : Maryam Khoshkam
1
Introduction
Algorithm
of classical non-negative least
square (nnls)
A
numerical example
Fast
nonegative least square (Fnnls)
Results
and discussion
2
introduction
Estimation of models subject to non-negativity constraints is
of practical importance in chemistry.
The time required for estimating true least squares nonnegativity constrained model is typicaly many times
longer than for estimating unconstrained model.
Approximations procedures
Unconstrained estimation+ Setting negative value to zero
What is the problem of force to zero algorithm????
3
1. There is no guarantee whatsoever for the quality of the
model.
2. When included in multiway algorithm, it can cause the
algorithm to diverge. (specially, in noisy data or difficult
models)
Non-negativity constrained linear least square
4
NNLS
NNLS will be stated using the following nomenclature:
x ( L1)  Z ( LM )d ( M 1)
f (d)  x( L1)  Z ( LM )d ( M 1)
dm  0
2
For all m
lsqnonneg command in MATLAB is based on
this algorithm
Lawson, C.L. and R.J. Hanson, Solving Least-Squares Problems, PrenticeHall, Chapter 23, p. 161, 1974.
5
Some important aspects of the algorithm
•This is an active set algorithm.
Thus the nnls first find the true passive and active
sets, then perform the least square on corresponding
columns of Z.
HOW…????
First of all, it suppose that all elements of d, are in active set.
R={1,2,….,M}
P={}
If the all elements of d are
active, estimate that how the
form of initial d….?????
6
•Initial solution vector d is feasible and set equal to
a M×1 zero vector
•Then it removes the passive sets elements of d,
one by one, and STEP by STEP
HOW…????
•Using the vector w
w ( M 1)  Z( LM ) (x( L1)  Z( LM )d( M 1) )
T
7
f (d)  x( L1)  Z( LM )d ( M 1)
2
f ' (d)  2ZT (Zd  x)
w=ZT ( x – Z d )
1
wm   f ' (d m )
2
One necessary condition for optimally of a solution , is
that the derivatives with respect to the parameters of the
passive set be zero. WHY ??
How is the f ’ (dm) if m is an active set ????
8
if
mP
f (d)  min  f ' (d m )  0  wm  0
if
mR
f (d)  0  f ' (d m )  0  wm  0
Thus at optimal solution, we expect that:
wm  0
m P
wm  0
m R
When we are being at optimal condition, wm >0. wht is the
meaning of positive value for wm?
9
positive wm shows that by increasing dm to a
more positive value, the change in residual is
negative (f ', which is the slope is negative).
It means that the residual becomes less, and
more close to zero, when dm goes toward
positive.
10
Algorithm nnls
1.
2.
3.
4.
R 
P=Ø
R={1,2,…,M}
d=0
w=ZT(x-Zd)
& max ( wn )  tol
loop A
No
Opt. sol. Of d
nR
yes
m  arg max ( wn )
nR
B
P={m}Loop
& R=R-{m}
p
Build s
SP=[(ZP)TZP]-1(ZP)Tx
d=s
w=ZT(x-Zd)
No
yes
min s P  0
Loop C
all sp>0
11
A simple numerical example
Z ( 43)
73 71 52 
87 74 46


72 2 7 


80
89
71


x ( 41)
49
67
 
68
 
20
p  {};
R  {1,2,3}
0 
d  0
0
 15902 
w   10353
 7526 
Max(wn)>1e-15 & R≠Ø
nR
Loop B
12
Loop B
p  {1};
R  {2,3};
s p  0.6495
min s P  0
No
0.6495
s   0 
 0 
ds
0

w  - 1913.5
- 1555.8
Max(wn)>1e-15 & R≠Ø
nR
Opt. sol. Of d
13
Graphical representation of nonlinear least square algorithm:
H3A ↔ H+ + H2AH2A- ↔ H+ + HA2HA2- ↔ H+ + A3-
pka1=2.6
pka2=4.0
pka3=6.3
spectral Profiles for H2A
Concentration Profiles
1.40.15
0.02
0.02
0.018
0.018
X
Z
0.016
0.016
1.2
0.1
0.014
0.014
0.012
0.012
absorbtion
Concentration
D
D̂
1
0.010.01
0.008
0.008
0.006
0.006
0.05
0.8
0.6
0
0.4
-0.05
0.004
0.004
0.2
0.002
0.002
0
0
0 300
2350
4
400
4506
pH
500 8
550
10 600
12650
0-0.1
300 0
35050
100
400
150
200
450
500
wavelength
D=X+.Z
(ILS)
250550 300 600 350
650400
14
x10x1=X(:,1)
D(4x100)
X(10x100)
=
Z(10x4)
d(4x1)
Step1
1) P=[];
2) R={1,2,3,4}
-13
-14
1.49
x 10
0.005 x D(:,1)
8
1.2
7
0 
0 
3) d   
0 
 
0 
1
6
0.85
0.64
w
3
0.4
2
0.2
1
4) w ( 41)  Z (x  Zd )
T
0
1
1.5
22
2.5
2.5
33
3.5
3.5
44
15
1.4
nnls
1.2
0.14
0.12 Force to zero
Variation of w during nonnegative
least square
1
0.1
0.8
0.08
0.6
0.06
0.4
0.04
0.2
0.02
0
0
50
100
150
200
250
300
350
400
0
wm  0
m P
wm  0
m R
0
50
100
150
200
250
300
350
400
16
-13
-13
14
x 10
3
Residual matrix
12
x 10
w
2.5
10
2
8
6
1.5
4
1
2
0
0.5
-2
0
-4
-6
300
350
400
450
500
550
600
650
-0.5
300
350
400
450
500
550
600
650
w=ZT(x-Zd)
wi  Z (i ,:)  r
=
w(4×1)
Z (10x4)T r(10x1)
wi in each λ, is the contribution of
ith species in in residual vector in
17
each λ.
Why it is necessary to modify the
NNLS algorithm?
18
PARAFAC-ALS
K
J
I
Least square
X( I  JK )  A(C  B)'E
Z ( JK F )  C  B
min X  A(C  B)'
A , B ,C
19
2
ˆ  XZ(Z' Z) 1
X( I  JK )  A ( I F ) ZT ( JK F )  E( I  JK )  A
If the size of X is 10x200x5 in a 3 component system,
the size of Z is:
500x3
And so on….
Computation of Z, can be computationally costly for
large arrayes
Excessive memory is required to calculate X(IxJK), X(JxIK)
and X(K,IJ)
20
1
ˆ
A  XZ (Z' Z)
XZ  [ X1 ... XK ][ D1B' ... D K B']'
 X1BD1  ...  X K BDK
Z ' Z  (C  B)' (C  B)  (C ' C ) * ( B' B)
K


ˆ
A    X k BDk (C' C) * (B' B)
 k 1

And in similar way for estimation of B and C
21
Nnls can not used for this simplified
version of PARAFAc, Why??
22
In estimating A, and using non-negativity constraint on it,
xi  Z ( JK F ) ai (1F )  ei
A(IxF) =
X(IxJK)
Z
SLOW!!!!!!!
A
modification…
23
Fast non-negativity least square
1. Accept the cross product (xZ and Z’Z) instead of the
row data
w=ZT ( x – Z d )  w=ZTx – (ZTZ)d
SP=[(ZP)TZP]-1(ZP)Tx SP=[(ZTZ)P]-1(ZTx)
2. Set the passive and active variables, before enter to loop B
d is not the zero vector in this case
24
Thanks
25
Ex.
73 71 52 
87 74 46

Z
72 2 7 


80
89
71


49
67 
x 
68
 
20
d=Z\x
d  1.123 0.917  2.068
Force to zero
d  1.123 0.917 0
RMS=103
nnlsq
d  0.650 0 0
RMS=20
26
In unconstrained solution of d, the mth negative element is
active set
d  1.123 0.917  2.068
3th element of d
In constrained solution of d, the mth zero element is active set
d  0.650 0 0
2th and 3th elements of d
Thus it is not possible to realize the true active set form
unconstraint least square solution.
What happen if the true active set is known?
27
For example, if we know that 2th and 3th elements of d
are true active sets,
NOTE!!!!!
1) d  0
if
dm  0
if
m
mP
md
1 R


d0
2) f(d) minimized by columns of Z
corresponding to passive
 0 sets. Thus
active sets which for them, d =0, is
m
not take a part in minimizing the
residuals
Then perform the simple unconstrained least square
with the 1st column of Z. (columns corresponding to
passive sets. selectivity constraint !!!!
28
Loop C
 dn 
1)    min 

nP d  s
 n n
2)
d  d   (s  d)
3)
Update R and P
4)
SP=[(ZP)TZP]-1(ZP)Tx
5)
SR=0
29