LAD Problem
Basic properties
New method
Aplication
A new method for searching an L1 solution of an
overdetermined system of linear equations and
applications
Goran Kušec1
1 Faculty
Ivana Kuzmanović2
Rudolf Scitovski2
Kristian Sabo2
of Agriculture, University of Osijek
2 Department
of Mathematics, University of Osijek
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
LAD Problem
A ∈ Rm×n , (m n), b ∈ Rm









A=








|
?
?
?
?
..
.
..
.
?
?
?
?
?
?
?
?
..
.
..
.
?
?
?
?
...
...
...
...
..
.
..
.
...
...
...
...
{z
n
Kušec, Kuzmanović, Sabo, Scitovski
?
?
?
?
..
.
..
.
?
?
?
?










,
















b=








?
?
?
?
..
.
..
.
?
?
?
?




















 m



















}
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
LAD Problem
A ∈ Rm×n , (m n), b ∈ Rm









A=








|
?
?
?
?
..
.
..
.
?
?
?
?
?
?
?
?
..
.
..
.
?
?
?
?
...
...
...
...
..
.
..
.
...
...
...
...
{z
n
Kušec, Kuzmanović, Sabo, Scitovski
?
?
?
?
..
.
..
.
?
?
?
?










,
















b=








?
?
?
?
..
.
..
.
?
?
?
?




















 m



















}
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
system of linear equations Ax = b
b ∈ R(A)
b
Kušec, Kuzmanović, Sabo, Scitovski
R(A)
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
overdetermined system of linear equations Ax ≈ b
b∈
/ R(A)
b
R(A)
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
kb − Axk → minn
x∈R
Euclidean l2 -norm (Least Square (LS) solution)
l1 -norm (Least absolute deviation (LAD) solution)
l∞ -norm (Tschebishev solution)
applications in various fields of applied research
J. A. Cadzow, Minimum l1 , l2 and l∞ Norm Approximate Solutions to an Overdetermined System
of Linear Equations, Digital Signal Processing 12(2002) 524–560
S. Dasgupta, S. K. Mishra, Least absolute deviation estimation of linear econometric models: A
literature review, Munich Personal RePEc Archive, 1-26, 2004
C. Gurwitz, Weighted median algorithms for L1 approximation, BIT 30(1990) 301–310
Dodge Y. (Ed.), Statistical Data Analysis Based on the L1 -norm and Related Methods, Proceedings
of The Thrid International Conference on Statistical Data Analysis Based on the L1 -norm and
Related Methods, Elsevier, Neuchâtel, 1997
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
kb − Axk → minn
x∈R
Euclidean l2 -norm (Least Square (LS) solution)
l1 -norm (Least absolute deviation (LAD) solution)
l∞ -norm (Tschebishev solution)
applications in various fields of applied research
J. A. Cadzow, Minimum l1 , l2 and l∞ Norm Approximate Solutions to an Overdetermined System
of Linear Equations, Digital Signal Processing 12(2002) 524–560
S. Dasgupta, S. K. Mishra, Least absolute deviation estimation of linear econometric models: A
literature review, Munich Personal RePEc Archive, 1-26, 2004
C. Gurwitz, Weighted median algorithms for L1 approximation, BIT 30(1990) 301–310
Dodge Y. (Ed.), Statistical Data Analysis Based on the L1 -norm and Related Methods, Proceedings
of The Thrid International Conference on Statistical Data Analysis Based on the L1 -norm and
Related Methods, Elsevier, Neuchâtel, 1997
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
kb − Axk → minn
x∈R
Euclidean l2 -norm (Least Square (LS) solution)
l1 -norm (Least absolute deviation (LAD) solution)
l∞ -norm (Tschebishev solution)
applications in various fields of applied research
J. A. Cadzow, Minimum l1 , l2 and l∞ Norm Approximate Solutions to an Overdetermined System
of Linear Equations, Digital Signal Processing 12(2002) 524–560
S. Dasgupta, S. K. Mishra, Least absolute deviation estimation of linear econometric models: A
literature review, Munich Personal RePEc Archive, 1-26, 2004
C. Gurwitz, Weighted median algorithms for L1 approximation, BIT 30(1990) 301–310
Dodge Y. (Ed.), Statistical Data Analysis Based on the L1 -norm and Related Methods, Proceedings
of The Thrid International Conference on Statistical Data Analysis Based on the L1 -norm and
Related Methods, Elsevier, Neuchâtel, 1997
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Vector x∗ ∈ Rn such that
min kb − Axk1 = kb − Ax∗ k1
x∈Rn
is called the best LAD-solution of overdetermined system Ax ≈ b.
operational research literature - a parameter estimation
problem of a hyperplane on the basis of a given set of points
N. M. Korneenko, H. Martini, Hyperplane approximation and related topics, in: New Trends in
Discrete and Computational Geometry, (J. Pach, Ed.), Springer-Verlag, Berlin, 1993.
A. Schöbel, Locating Lines and Hyperplanes: Theory and Algorithms, Springer Verlag, Berlin, 1999.
statistics literature - a parameter estimation problem for linear
regression
S. Dasgupta, S. K. Mishra, Least absolute deviation estimation of linear econometric models: A
literature review, Munich Personal RePEc Archive, 1-26, 2004
E. Z. Demidenko, Optimization and Regression, Nauka, Moscow, 1989, in Russian
Dodge Y. (Ed.) (1997): Statistical Data Analysis Based on the L1 -norm and Related Methods,
Proceedings of The Thrid International Conference on Statistical Data Analysis Based on the
L1 -norm and Related Methods, Elsevier, Neuchâtel
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Vector x∗ ∈ Rn such that
min kb − Axk1 = kb − Ax∗ k1
x∈Rn
is called the best LAD-solution of overdetermined system Ax ≈ b.
operational research literature - a parameter estimation
problem of a hyperplane on the basis of a given set of points
N. M. Korneenko, H. Martini, Hyperplane approximation and related topics, in: New Trends in
Discrete and Computational Geometry, (J. Pach, Ed.), Springer-Verlag, Berlin, 1993.
A. Schöbel, Locating Lines and Hyperplanes: Theory and Algorithms, Springer Verlag, Berlin, 1999.
statistics literature - a parameter estimation problem for linear
regression
S. Dasgupta, S. K. Mishra, Least absolute deviation estimation of linear econometric models: A
literature review, Munich Personal RePEc Archive, 1-26, 2004
E. Z. Demidenko, Optimization and Regression, Nauka, Moscow, 1989, in Russian
Dodge Y. (Ed.) (1997): Statistical Data Analysis Based on the L1 -norm and Related Methods,
Proceedings of The Thrid International Conference on Statistical Data Analysis Based on the
L1 -norm and Related Methods, Elsevier, Neuchâtel
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
principle is considered to have been proposed by the Croatian
mathematician J. R. Bošković in the mid-eighteenth century
D. Birkes, Y. Dodge, Alternative Methods of Regression, Wiley, New York, 1993
P. Bloomfield, W. Steiger, Least Absolute Deviations: Theory, Applications, and Algorithms,
Birkhauser, Boston, 1983
Dodge Y. (Ed.) (1997): Statistical Data Analysis Based on the L1 -norm and Related Methods,
Proceedings of The Thrid International Conference on Statistical Data Analysis Based on the
L1 -norm and Related Methods, Elsevier, Neuchâtel
Numerical methods
classical nondifferentiable minimization methods cannot be
applied directly-unreasonably long computing time
specialized algorithms
J. A. Cadzow, Minimum l1 , l2 and l∞ Norm Approximate Solutions to an Overdetermined System of Linear
Equations, Digital Signal Processing 12(2002) 524–560
E. Castillo, R. Mı́nguez, C. Castillo, A. S. Cofinño, Dealing with the multiplicity of solutions of the l1 and l∞
regression models, European J. Oper. Res. 188(2008), 460–484.
Y. Li, A maximum likelihood approach to least absolute deviation regression, EURASIP Journal on Applied
Signal Processing 12(2004), 1762–1769
N. T. Trendafilov, G. A. Watson, The l1 oblique procrustes problem, Statistics and Computing 14(2004)
39–51
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
principle is considered to have been proposed by the Croatian
mathematician J. R. Bošković in the mid-eighteenth century
D. Birkes, Y. Dodge, Alternative Methods of Regression, Wiley, New York, 1993
P. Bloomfield, W. Steiger, Least Absolute Deviations: Theory, Applications, and Algorithms,
Birkhauser, Boston, 1983
Dodge Y. (Ed.) (1997): Statistical Data Analysis Based on the L1 -norm and Related Methods,
Proceedings of The Thrid International Conference on Statistical Data Analysis Based on the
L1 -norm and Related Methods, Elsevier, Neuchâtel
Numerical methods
classical nondifferentiable minimization methods cannot be
applied directly-unreasonably long computing time
specialized algorithms
J. A. Cadzow, Minimum l1 , l2 and l∞ Norm Approximate Solutions to an Overdetermined System of Linear
Equations, Digital Signal Processing 12(2002) 524–560
E. Castillo, R. Mı́nguez, C. Castillo, A. S. Cofinño, Dealing with the multiplicity of solutions of the l1 and l∞
regression models, European J. Oper. Res. 188(2008), 460–484.
Y. Li, A maximum likelihood approach to least absolute deviation regression, EURASIP Journal on Applied
Signal Processing 12(2004), 1762–1769
N. T. Trendafilov, G. A. Watson, The l1 oblique procrustes problem, Statistics and Computing 14(2004)
39–51
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
principle is considered to have been proposed by the Croatian
mathematician J. R. Bošković in the mid-eighteenth century
D. Birkes, Y. Dodge, Alternative Methods of Regression, Wiley, New York, 1993
P. Bloomfield, W. Steiger, Least Absolute Deviations: Theory, Applications, and Algorithms,
Birkhauser, Boston, 1983
Dodge Y. (Ed.) (1997): Statistical Data Analysis Based on the L1 -norm and Related Methods,
Proceedings of The Thrid International Conference on Statistical Data Analysis Based on the
L1 -norm and Related Methods, Elsevier, Neuchâtel
Numerical methods
classical nondifferentiable minimization methods cannot be
applied directly-unreasonably long computing time
specialized algorithms
J. A. Cadzow, Minimum l1 , l2 and l∞ Norm Approximate Solutions to an Overdetermined System of Linear
Equations, Digital Signal Processing 12(2002) 524–560
E. Castillo, R. Mı́nguez, C. Castillo, A. S. Cofinño, Dealing with the multiplicity of solutions of the l1 and l∞
regression models, European J. Oper. Res. 188(2008), 460–484.
Y. Li, A maximum likelihood approach to least absolute deviation regression, EURASIP Journal on Applied
Signal Processing 12(2004), 1762–1769
N. T. Trendafilov, G. A. Watson, The l1 oblique procrustes problem, Statistics and Computing 14(2004)
39–51
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
LAD problem ⇔
On the basis of the given set of points
(i)
(i)
Λ = {Ti (x1 , . . . , xn−1 , z (i) ) ∈ Rn : i ∈ I }, I = {1, . . . , m}, vector
∗
∗
∗
∗
a = (a0 , a1 , . . . , an−1 ) ∈ Rn of optimal parameters of hyperplane
P
f (x; a) = a0 + n−1
j=1 aj xj should be determined such that:
G (a∗ ) = minn G (a),
a∈R
m n−1
X
X
(i)
(i) G (a) =
aj xj .
z − a0 −
i=1 j=1
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2
(i)
Data-points Λ = {(x1 , z (i) ), i = 1, . . . , m}
LAD line z = a0∗ + a1∗ x1
m
m
X
X
(i)
(i)
(i)
|z − a0 − a1 x1 | =
|z (i) − a0∗ − a1∗ x1 |
min
(a0 ,a1 )∈R2
i=1
Kušec, Kuzmanović, Sabo, Scitovski
i=1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2
(i)
Data-points Λ = {(x1 , z (i) ), i = 1, . . . , m}
LAD line z = a0∗ + a1∗ x1
m
m
X
X
(i)
(i)
(i)
|z − a0 − a1 x1 | =
|z (i) − a0∗ − a1∗ x1 |
min
(a0 ,a1 )∈R2
i=1
Kušec, Kuzmanović, Sabo, Scitovski
i=1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2
(i)
Data-points Λ = {(x1 , z (i) ), i = 1, . . . , m}
LAD line z = a0∗ + a1∗ x1
m
m
X
X
(i)
(i)
(i)
|z − a0 − a1 x1 | =
|z (i) − a0∗ − a1∗ x1 |
min
(a0 ,a1 )∈R2
i=1
Kušec, Kuzmanović, Sabo, Scitovski
i=1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2
(i)
Data-points Λ = {(x1 , z (i) ), i = 1, . . . , m}
LAD line z = a0∗ + a1∗ x
m
m
X
X
(i)
(i)
(i)
|z − a0 − a1 x1 | =
|z (i) − a0∗ − a1∗ x1 |
min
(a0 ,a1 )∈R2
i=1
Kušec, Kuzmanović, Sabo, Scitovski
i=1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
LAD solution always exists
N. M. Korneenko, H. Martini, Hyperplane approximation and related topics, in: New Trends in Discrete and
Computational Geometry, (J. Pach, Ed.), Springer-Verlag, Berlin, 1993.
A. Pinkus, On L1 -approximation, Cambridge University Press, New York, 1993.
G. A. Watson, Approximation Theory and Numerical Methods, John Wiley & Sons, Chichester, 1980.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
LAD solution is generally not unique
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
n=2, LAD solution is not unique
1
0.8
0.6
0.4
0.2
0.2
0.4
Kušec, Kuzmanović, Sabo, Scitovski
0.6
0.8
1
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Definition
Matrix A ∈ Rm×n (m ≥ n) is said to satisfy the Haar condition if
every n × n submatrix is nonsingular.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Theorem 1
Let A ∈ Rm×n , m > n, be a matrix of full column rank and
b = (b1 , . . . , bm )T ∈ Rm a given vector. Then there exists a permutation
matrix Π ∈ Rm×m such that
b1
A1
, A1 ∈ Rn×n , A2 ∈ R(m−n)×n , b1 ∈ Rn , b2 ∈ Rm−n ,
, Πb =
ΠA =
b2
A2
whereby A1 is a nonsingular matrix, and there exists a LAD-solution x∗ ∈ Rn
such that A1 x∗ = b1 .
Furthermore, if matrix [A b] satisfies the Haar condition, then the solution
x∗ ∈ Rn of the system A1 x = b1 is a solution of LAD problem if and only if
vector
v = (A1 T )−1 A2 T s, si := sign (b2 − A2 x∗ )i
satisfies kvk∞ ≤ 1.
Furthermore, x∗ is a unique solution of LAD problem if and only if kvk∞ < 1.
A. Pinkus, On L1 -approximation, Cambridge University Press, New York, 1993.
G. A. Watson, Approximation Theory and Numerical Methods, John Wiley & Sons, Chichester, 1980.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
parameter estimation problem of a hyperplane for a set of
points in a plane
n = 2 for the given set of points in a plane, with condition that
not all of them lie on some line parallel to the second axis,
there exists the best LAD-line passing through at least two
different data points
n = 3 for the given set of points in the space, with condition
that not all of them lie on some plane parallel to the third axis,
there exists the best LAD-plane passing through at least three
different data points
Y. Li, A maximum likelihood approach to least absolute deviation regression, EURASIP Journal on Applied
Signal Processing 12(2004), 1762–1769
R. Scitovski, K. Sabo, I. Kuzmanović, I. Vazler, R. Cupec, R. Grbić, Three points method for searching the
best least absolute deviations plane, 4th Croatian Mathematical Congress, Osijek, June 17 – 20, 2008
A. Schöbel, Locating Lines and Hyperplanes: Theory and Algorithms, Springer Verlag, Berlin, 1999.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
parameter estimation problem of a hyperplane for a set of
points in a plane
n = 2 for the given set of points in a plane, with condition that
not all of them lie on some line parallel to the second axis,
there exists the best LAD-line passing through at least two
different data points
n = 3 for the given set of points in the space, with condition
that not all of them lie on some plane parallel to the third axis,
there exists the best LAD-plane passing through at least three
different data points
Y. Li, A maximum likelihood approach to least absolute deviation regression, EURASIP Journal on Applied
Signal Processing 12(2004), 1762–1769
R. Scitovski, K. Sabo, I. Kuzmanović, I. Vazler, R. Cupec, R. Grbić, Three points method for searching the
best least absolute deviations plane, 4th Croatian Mathematical Congress, Osijek, June 17 – 20, 2008
A. Schöbel, Locating Lines and Hyperplanes: Theory and Algorithms, Springer Verlag, Berlin, 1999.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
New method
I. Kuzmanović, K. Sabo, R. Scitovski, Method for searching the best least absolute deviations solution of an
overdetermined system of linear equations, submitted
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
numerical methods for searching the best LAD-solution:
Hooke and Jeeves Method, Differential Evolution, Nelder
Mead, Random Search, Simulated Annealing
J. A. Nelder, R. Mead, A simplex method for function minimization, Comput. J. 7(1965)
308–313
C. T. Kelley, Iterative Methods for Optimization, SIAM, Philadelphia, 1999.
various specializations of the Gauss-Newton method
E. Z. Demidenko, Optimization and Regression, Nauka, Moscow, 1989, in Russia
M. R. Osborne, Finite Algorithms in Optimization and Data Analysis, Wiley, Chichester,
1985.
Linear Programming methods:
A. Barrodale, F. D. K. Roberts, An improved algorithm for discrete l1 linear approximation,
SIAM J. Numer. Anal. 10(1973) 839–848
N. M. Korneenko, H. Martini, Hyperplane approximation and related topics, in: New Trends
in Discrete and Computational Geometry, (J. Pach, Ed.), Springer-Verlag, Berlin, 1993.
some of specialized methods: Barrodale-Roberts,
Bartels-Conn-Sinclair, Bloomfield-Steiger
N. T. Trendafilov, G. A. Watson, The l1 oblique procrustes problem, Statistics and
Computing 14(2004) 39–51
P. Bloomfield, W. Steiger, Least Absolute Deviations: Theory, Applications, and Algorithms,
Birkhauser, Boston, 1983
Y. Dodge, Statistical Data Analysis Based on the L1 -norm and Related Methods,
of The
International
Conference
Kušec,Proceedings
Kuzmanović,
Sabo,Thrid
Scitovski
A new
methodon
forStatistical
searching Data
an L1Analysis
solutionBased
of an on
overdetermined
LAD Problem
Basic properties
New method
Aplication
Lemma 1
Let y1 P
≤ y2 ≤ . . . ≤ ym be real numbers with corresponding weights wi > 0,
W := m
i=1 wi , and I = {1, . . . , m}, m ≥ 2 a set of indices. Denote
P
P
max ν ∈ I : νi=1 wi ≤ W2 , if ν ∈ I : νi=1 wi ≤ W2 6= ∅
ν0 =
0,
otherwise
Furthermore,
P let F : R → R be a function defined by the formula
F (α) = m
i=1 wi |yi − α|. Then
Pν0
(i) if i=1 wi < W2 , then the minimum of the function F is attained at the
point α? = yν0 +1 ;
P 0
wi = W2 , then the minimum of the function F is attained at every
(ii) if νi=1
?
point α from the segment [yν0 , yν0 +1 ].
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Lemma 2
Let x0 be a unique solution of the system Ax = b, where A ∈ Rn×n is a square
nonsingular matrix and b ∈ Rn a given vector.
If the j-th equation of this system aTj x = bj is replaced with equation uT x = β,
whereby uT dj 6= 0, where dj ∈ Rn is the j-th column of matrix A−1 , then the
solution of a new system is given by
x = x0 + αdj ,
Kušec, Kuzmanović, Sabo, Scitovski
α=
β − uT x0
.
uT dj
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Definition
Vector x∗Jk ∈ Rn , Jk = {i1 , . . . , ik } ⊆ I , k ≤ n is called the best
Jk -LAD-solution of problem
min kb − Axk1 = kb − Ax∗ k1
x∈Rn
if on x∗Jk the minimum of functional x 7→ kr(x)k1 is attained with
condition
ri (x∗Jk ) := bi − (Ax)i = 0, i ∈ Jk .
the best Jn -LAD-solution is obtained by solving the system of
linear equations
ri (x) = 0,
i = i1 , . . . , in ,
the best Jn−1 -LAD-solution is obtained as a solution of one
weighted median problem.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Lemma 3
Let A ∈ Rm×n , (m > n) be a matrix of full column rank, b ∈ Rm a given vector and e = (1, . . . , m)T a vector
of indices. Furthermore, let Π ∈ Rm×m be a permutation matrix such that
"
ΠA =
A−
1
A+
2
#
=
A11
A21
a1k
a2k
A12
A22
"
,
Πb =
b−
1
b+
2
#
,
where A11 ∈ R(n−1)×(k−1) , A12 ∈ R(n−1)×(n−k) , A21 ∈ R(m−n+1)×(k−1) , A22 ∈ R(m−n+1)×(n−k) ,
n−1
m−n+1
a1k , b−
, a2k , b+
, whereby index k is such that A1k := [A11 A12 ] is a nonsingular matrix.
1 ∈ R
2 ∈ R
Let A2k := [A21 A22 ], I − = {(Πe)k : k = 1, . . . , n − 1} and I + = {(Πe)k : k = n, n + 1, . . . , m}.
Then there exists i0 ∈ I + and the best Jn -LAD-solution x∗ ∈ Rn , such that ri (x∗ ) = 0 for all i ∈ I − ∪ {i0 }.
The k-th component xk∗ of vector x∗ is thereby a solution of the weighted median problem
+
kA2k η − b2 − α(A2k ξ − a2k )k1 → min,
α
(1)
whereby ξ, i.e. η, is a unique solution of systems
A1k ξ = a1k ,
i.e.
−
A1k η = b1 ,
(2)
whereas other components of vector x∗ are obtained by solving the system
A1k (x1 , . . . , xk−1 , xk+1 , . . . , xn )
Kušec, Kuzmanović, Sabo, Scitovski
T
−
∗
= b1 − xk a1k .
(3)
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
matrices A and [A b] - satisfy the Haar condition.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Step 0 (Input) m, n, e = (1, . . . , m)T , A ∈ Rm×n , b ∈ Rm , Π0
Step 1 (Defining the first submatrix) Determine index k such that
nonsingular matrix, where
#
"
A11
a1k
A−
1
=
H0 := Π0 A =
+
A21
a2k
A
2
∈ Rm×m .
A1k := [A11 A12 ] ∈ R(n−1)×(n−1) is a
A12
A22
"
,
Π0 b =
b−
1
b+
2
#
.
Set I + = {(Π0 e)i : i = n, n + 1, . . . , m} and e := Π0 e.
Step 2 (Searching for a new equation) According to Lemma 3, determine i0 ∈ I + and solution x(0) . Calculate
(0)
G0 = G (x(0)
kb
#
# − Ax k1
"
" ) :=
A−
b−
1
1
∈ Rn .
∈ Rn×n , b1 =
Let A1 =
b
aT
i0
i
0
m−n
Furthermore, let A2 ∈ R(m−n)×n be a matrix obtained from A+
2 by dropping the i0 -th row, b2 ∈ R
a vector obtained from b+
by
dropping
the
i
-th
component
0
2
Solve system
T
T
A1 v = A2 s,
si := sign (b2 − A2 x)i , i = 1, . . . , n,
and denote the solution by v(0)
(0)
|
0
Determine j0 such that |vj
=
max
i=1,...,n
(0)
|vi
| =: vM
If vM ≤ 1, STOP; otherwise go to Step 3
Step 3 (Equations exchange) Let Π1 ∈ Rm×m be a permutation matrix which in matrix H0 replaces the j0 -th
row with the i0 -th row. Determine index k such that A1k := [A11 A12 ] ∈ R(n−1)×(n−1) is a nonsingular
matrix, where
"
#
"
#
A11
a1k
A12
A−
b−
1
1
H1 := Π1 H0 =
=
, Π1 b =
,
+
+
A21
a2k
A22
A
b
2
2
Set I + = {(Π1 e)i : i = n, n + 1, . . . , m} and e := Π1 e.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Step 4 (Searching for a new equation) According to Lemma 2, determine i0 ∈ I + and solution x(1) . Calculate
G1 = G (x(1) ) := kb − Ax(1) k1 ;
#
#
"
"
A−
b−
1
1
∈ Rn .
∈ Rn×n , b1 =
Step 5 (Preparation for a new loop) Let A1 =
b
aT
i0
i
0
m−n
Furthermore, let A2 ∈ R(m−n)×n be a matrix obtained from A+
2 by dropping the i0 -th row, b2 ∈ R
a vector obtained from b+
by
dropping
the
i
-th
component.
0
2
Solve system
T
T
si := sign (b2 − A2 x)i , i = 1, . . . , n,
A1 v = A2 s,
and denote the solution by v(1)
(1)
(1)
Determine j0 such that |vj | = max |vi
0
i=1,...,n
| =: vM
If vM ≤ 1, STOP; otherwise put H0 = H1 and go to Step 3
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Initialization
The initial matrix A−
1
matrix A satisfies the Haar condition
the first (n − 1) rows of matrix A
Π0 = I- identity matrix
A nonsingular submatrix A1k of matrix
− A1
A11 a1k
H0 := Π0 A =
=
A+
A21 a2k
2
A12
A22
in Step 1 - by applying QR factorization with column pivoting
A−
1 Π := [A11 a1k A12 ] Π = Q[R ρ],
Π ∈ Rn×n - permutation matrix
Q ∈ R(n−1)×(n−1) - an orthogonal matrix
R ∈ R(n−1)×(n−1) - an upper triangular nonsingular matrix
ρ ∈ R(n−1)
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Initialization
The initial matrix A−
1
matrix A satisfies the Haar condition
the first (n − 1) rows of matrix A
Π0 = I- identity matrix
A nonsingular submatrix A1k of matrix
− A1
A11 a1k
H0 := Π0 A =
=
A+
A21 a2k
2
A12
A22
in Step 1 - by applying QR factorization with column pivoting
A−
1 Π := [A11 a1k A12 ] Π = Q[R ρ],
Π ∈ Rn×n - permutation matrix
Q ∈ R(n−1)×(n−1) - an orthogonal matrix
R ∈ R(n−1)×(n−1) - an upper triangular nonsingular matrix
ρ ∈ R(n−1)
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
A1k := [A11 A12 ] = Q R ΠT
1,
a1k = Qρ,
Π1 is the matrix - from Π by dropping the k th row and the
nth column.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Theorem 2
Let x̄ be approximation of LAD-problem obtained in Step 2, i.e.
Step
4, of the
A−
1
Algorithm as a solution of the system A1 x = b1 , where A1 =
∈ Rn×n ,
aTi0
− b1
b1 =
∈ Rn . Furthermore, let A2 ∈ R(m−n)×n be a matrix obtained from
bi0
m−n
A+
a vector obtained from b+
2 by dropping the i0 -th row, b2 ∈ R
2 by
dropping the i0 -th component, and v ∈ Rn a solution of the system
AT1 v = AT2 s,
si := sign (b2 − A2 x)i ,
i = 1, . . . , n.
Denote
|vj0 | = max |vi |.
i=1,...,n
I. If |vj0 | ≤ 1, then x̄ is the best LAD-solution;
II. If |vj0 | > 1, then j0 6= n, and maximum decreasing of minimizing function
values is attained such that in Step 3 the j0 -th row of matrix A1 is
replaced.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Convergence Theorem 4
Let A ∈ Rm×n be a matrix and b ∈ Rm a given vector, m ≥ n,
such that matrices A and [A b] satisfy the Haar condition. Then
sequence (x(n) ) defined by the iterative procedure in the Algorithm
converges in finitely many steps to the best LAD-solution.
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Example 1
Table 1.
i
1
2
3
4
5
6
7
8
9
10
11
12
42
31
13
45
25
-37
-12
-42
19
-21
28
32
7
46
-11
-35
-34
-24
26
31
2
-26
26
27
A
-28
-31
-1
43
37
-9
-49
28
-29
-45
-38
-7
34
-12
-35
-9
-30
-35
-49
-5
-33
-44
-39
9
13
17
31
-36
-2
-35
-11
-45
0
5
-8
-43
-49
18
-33
-8
32
21
-22
47
-46
26
-16
-22
b
6
-42
48
-16
13
-16
-32
-28
35
47
-6
-38
G0 = 236.346
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Example 1
Table 1.
i
1
2
3
4
5
6
7
8
9
10
11
12
42
31
13
45
25
-37
-12
-42
19
-21
28
32
7
46
-11
-35
-34
-24
26
31
2
-26
26
27
A
-28
-31
-1
43
37
-9
-49
28
-29
-45
-38
-7
34
-12
-35
-9
-30
-35
-49
-5
-33
-44
-39
9
13
17
31
-36
-2
-35
-11
-45
0
5
-8
-43
-49
18
-33
-8
32
21
-22
47
-46
26
-16
-22
b
6
-42
48
-16
13
-16
-32
-28
35
47
-6
-38
G1 = 180.613
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Example 1
Table 1.
i
1
2
3
4
5
6
7
8
9
10
11
12
42
31
13
45
25
-37
-12
-42
19
-21
28
32
7
46
-11
-35
-34
-24
26
31
2
-26
26
27
A
-28
-31
-1
43
37
-9
-49
28
-29
-45
-38
-7
34
-12
-35
-9
-30
-35
-49
-5
-33
-44
-39
9
13
17
31
-36
-2
-35
-11
-45
0
5
-8
-43
-49
18
-33
-8
32
21
-22
47
-46
26
-16
-22
b
6
-42
48
-16
13
-16
-32
-28
35
47
-6
-38
G2 = 171.134
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Example 1
Table 1.
i
1
2
3
4
5
6
7
8
9
10
11
12
42
31
13
45
25
-37
-12
-42
19
-21
28
32
7
46
-11
-35
-34
-24
26
31
2
-26
26
27
A
-28
-31
-1
43
37
-9
-49
28
-29
-45
-38
-7
34
-12
-35
-9
-30
-35
-49
-5
-33
-44
-39
9
13
17
31
-36
-2
-35
-11
-45
0
5
-8
-43
-49
18
-33
-8
32
21
-22
47
-46
26
-16
-22
b
6
-42
48
-16
13
-16
-32
-28
35
47
-6
-38
G3 = 148.12
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Example 1
Table 1.
i
1
2
3
4
5
6
7
8
9
10
11
12
42
31
13
45
25
-37
-12
-42
19
-21
28
32
7
46
-11
-35
-34
-24
26
31
2
-26
26
27
A
-28
-31
-1
43
37
-9
-49
28
-29
-45
-38
-7
34
-12
-35
-9
-30
-35
-49
-5
-33
-44
-39
9
13
17
31
-36
-2
-35
-11
-45
0
5
-8
-43
-49
18
-33
-8
32
21
-22
47
-46
26
-16
-22
b
6
-42
48
-16
13
-16
-32
-28
35
47
-6
-38
G4 = 146.663
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Application
(i)
(i)
Data-points Ti = (x1 , x2 , z (i) ), i = 1, . . . , m
(i)
x1 -fat thickness
(i)
x2 -lumbar muscle thickness
z (i) - muscle percent of the i th pig
z = a0 + a1 x1 + a2 x2
outliers
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Application
(i)
(i)
Data-points Ti = (x1 , x2 , z (i) ), i = 1, . . . , m
(i)
x1 -fat thickness
(i)
x2 -lumbar muscle thickness
z (i) - muscle percent of the i th pig
z = a0 + a1 x1 + a2 x2
outliers
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Application
(i)
(i)
Data-points Ti = (x1 , x2 , z (i) ), i = 1, . . . , m
(i)
x1 -fat thickness
(i)
x2 -lumbar muscle thickness
z (i) - muscle percent of the i th pig
z = a0 + a1 x1 + a2 x2
outliers
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Application
(i)
(i)
Data-points Ti = (x1 , x2 , z (i) ), i = 1, . . . , m
(i)
x1 -fat thickness
(i)
x2 -lumbar muscle thickness
z (i) - muscle percent of the i th pig
z = a0 + a1 x1 + a2 x2
outliers
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Application
(i)
(i)
Data-points Ti = (x1 , x2 , z (i) ), i = 1, . . . , m
(i)
x1 -fat thickness
(i)
x2 -lumbar muscle thickness
z (i) - muscle percent of the i th pig
z = a0 + a1 x1 + a2 x2
outliers
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Root-Mean-Square Error of Prediction (RMSEP)
D. Causeur, G. Daumas, T. Dhorne, B. Engel, M. Font, I. Furnols, S. Hojsgaard, Statistical handbook for
assessing pig classification methods: Recommendations from the EUPIGCLASS project group
EU reference method
P. Walstra, G. S. M. Merkus, Procedure for assessment of the lean meat percentage as a consequence of the
new EU reference dissection method in pig carcass classification, DLO-Research Institute for Animal Science
and Health (ID - DLO). Research Branch Zeist, P.O. Box 501, 3700 AM Zeist, The Netherlands, 1995
Kušec, Kuzmanović, Sabo, Scitovski
A new method for searching an L1 solution of an overdetermined
LAD Problem
Basic properties
New method
Aplication
Example 2
m=145
EU reference method - RMSEP
a2∗ = 0.71259, a1∗ = 0.02312,
a0∗ = 67.77137
LAD method
a2∗ = 0.696471, a1∗ = 0.0176915,
Kušec, Kuzmanović, Sabo, Scitovski
a0∗ = 67.8906
A new method for searching an L1 solution of an overdetermined