Optimality conditions
for
constrained optimisation
Recall: Optimisation conditions for unconstrained
problem with one variable function
• One variable function: f : R1  R1
One variable function: f : R1  R1
Result: Let c  R1 such that
df  c 
f  x   f c 
f c 
 lim
 0.
x c
dx
xc
Then c is neither a local minimum nor a local maximum of f .
Furthermore,
i ) if f   c   0, There exists an interval  c   , c    with   0 such that
f  x   f c
 x  c   , c 
f c  f  x 
 x   c, c   
and f is increasing at c
ii ) if f   c   0, there exists an interval  c   , c    with   0 such that
f  x   f c
 x  c   , c 
f c  f  x 
 x   c, c   
and f is decreasing at c
Result : (Second derivative test)
Let c  R1 such that f   c   0.
Assume also that the second derivative f   x  exists x  B  c  .
i ) If f   c   0, then c is a local maximum of f .
ii ) If f   c   0, then c is a local minimum of f .
Remark: f   c  denotes the second derivative of f à c :
f  x   f c d 2 f
df 
f   c  

 c   lim
c .
x c
dx
xc
dx
Intuitive justification of ii ) :
If f   c   0, then f  is an increasing function at c,
and it follows that c is a local minimum.
Recall: Optimisation conditions for unconstrained
problem with several variables function
Several variables function:
f : R n  R1
To extend the preceding results obtained for one variable function:
f : R1  R1  f : R n  R1
 f  x

f  x   f  x   
,
  x1
  2 f  x

  x1 x1
f   x   2 f  x   
 2
  f  x
 x x
 n 1
 f  x 
,
 xn 
T
 2 f  x 

 x1 xn 


2
 f  x 
 xn xn 
To extend the preceding results obtained for one variable function:
f : R1  R1  f : R n  R1
  x   x Dx   x1 ,
T
 d11
, xn  
d
 n1
d1n   x 
 1 
 
d nn   xn 
 f  x
f   x   f  x   
,
  x1
  2 f  x

  x1 x1
2
f   x    f  x   
 2
  f  x
 x x
 n 1
 f  x 
,
 xn 
T
 2 f  x 

 x1 xn 


 2 f  x 
 xn xn 
Definition: The quadratic form associated with the a real valued
n  n matrix D is the fonction  : R n  R1 specified as follows
  x   x T Dx.
Définition: A real valued n  n matrix D is positive semi-definite
( positive definite) if   x   0 x  R n   x   0 x  R n , x  0  .
Results: A real valued n  n matrix D is positive semi-definite
( positive definite) if and only if all its eigenvalues are non negative
(positive).
Necessary conditions
Lemma: Let X  R n be an open set and f  C 2 / X be twice
continuously differentiable. If x  X is a local minimum of f
on X , then f  x  =0 and 2 f  x  is a positive semi-definite
matrix.
Sufficient conditions
Lemma: Let X  R n be an open set and f  C 2 / X be twice
continuously differentiable. If f  x * =0 and 2 f  x * is
a positive definite matrix, then x* is a local minimum of f
on X .
Conter-exemple: The conditions f  x  =0 and 2 f  x 
being a positive semi definite matrix are not sufficient to
garantee that x is a local minimum.
At the point x  y  0
f  x, y   x 3  y 3
f  x, y   0
f  x, y   3x ,3 y 
f  0,0   0, 0
 6x 0 
 0 0
2
2
 f  x, y   
 f  0,0   
pos. semi def.


 0 6y
 0 0
Then the conditions are satisfied at x  y  0.
  
But for an  >0 sufficiently small,   ,    B  0,0  and
 2 2
3
3
2 3
      
f  ,         
 0  f  0,0  ,
8
 2 2  2  2
and 0,0 is not a local minimum even if the conditions
are satisfied.
2
2
T
T
Conter-exemple: The conditions f  x  =0 and 2 f  x 
being a positive semi definite matrix are not sufficient to
garantee that x is a local minimum.
At the point x  y  0
f  x, y   x 3  y 3
f  x, y   0
f  x, y   3x ,3 y 
f  0,0   0, 0
 6x 0 
 0 0
2
2
 f  x, y   
 f  0,0   
pos. semi def.


 0 6y
 0 0
Then the conditions are satisfied at x  y  0.
  
But for an  >0 sufficiently small,   ,    B  0,0  and
 2 2
3
3
2 3
      
f  ,         
 0  f  0,0  ,
8
 2 2  2  2
and 0,0 is not a local minimum even if the conditions
are satisfied.
2
2
T
T
Conter-exemple: The conditions f  x  =0 and 2 f  x 
being a positive semi definite matrix are not sufficient to
garantee that x is a local minimum.
At the point x  y  0
f  x, y   x 3  y 3
f  x, y   0
f  x, y   3x ,3 y 
f  0,0   0, 0
 6x 0 
 0 0
2
2
 f  x, y   
 f  0,0   
pos. semi def.


 0 6y
 0 0
Then the conditions are satisfied at x  y  0.
  
But for an  >0 sufficiently small,   ,    B  0,0  and
 2 2
3
3
2 3
      
f  ,         
 0  f  0,0  ,
8
 2 2  2  2
and 0,0 is not a local minimum even if the conditions
are satisfied.
2
2
T
T
Conter-exemple: The conditions f  x  =0 and 2 f  x 
being a positive semi definite matrix are not sufficient to
garantee that x is a local minimum.
At the point x  y  0
f  x, y   x 3  y 3
f  x, y   0
f  x, y   3x ,3 y 
f  0,0   0, 0
 6x 0 
 0 0
2
2
 f  x, y   
 f  0,0   
pos. semi def.


 0 6y
 0 0
Then the conditions are satisfied at x  y  0.
  
But for an  >0 sufficiently small,   ,    B  0,0  and
 2 2
3
3
2 3
      
f  ,         
 0  f  0,0  ,
8
 2 2  2  2
and 0,0 is not a local minimum even if the conditions
are satisfied.
2
2
T
T
Conter-exemple: The conditions f  x  =0 and 2 f  x 
being a positive semi definite matrix are not sufficient to
garantee that x is a local minimum.
At the point x  y  0
f  x, y   x 3  y 3
f  x, y   0
f  x, y   3x ,3 y 
f  0,0   0, 0
 6x 0 
 0 0
2
2
 f  x, y   
 f  0,0   
pos. semi def.


 0 6y
 0 0
Then the conditions are satisfied at x  y  0.
  
But for an  >0 sufficiently small,   ,    B  0,0  and
 2 2
3
3
2 3
      
f  ,         
 0  f  0,0  ,
8
 2 2  2  2
and 0,0 is not a local minimum even if the conditions
are satisfied.
2
2
T
T
Lagrangean multipliers
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
where X  R n and the functions f : X  R1 , f i : X  R1 , i  1,
(1)
, m.
To obtain the lagrangean associated with (1), we associate a
lagrangean multiplicateur i with each constraint fonction f i :
L , x   f  x  
m
 f  x .
i i
i 1
Without any assumption on X or on the fonctions f et f i , we can derive
sufficient conditions for a point x * to be an optimal global solution
for problem (1).
Lagrangean multipliers
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
where X  R n and the functions f : X  R1 , f i : X  R1 , i  1,
(1)
, m.
To obtain the lagrangean function of (1), we associate a
lagrangean multiplicateur i with each constraint fonction f i :
L , x   f  x  
m
 f  x .
i i
i 1
Without any assumption on X or on the fonctions f et f i , we can derive
sufficient conditions for a point x * to be an optimal global solution
for problem (1).
Optimisation conditions for constrained problem
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
where X  R n and the functions f : X  R1 , f i : X  R1 , i  1,
(1)
, m.
To obtain the lagrangean function of (1), we associate a
lagrangean multiplicateur i with each constraint fonction f i :
L , x   f  x  
m
 f  x .
i i
i 1
Without any assumption on X or on the fonctions f and f i , we can derive
sufficient conditions for a point x * to be a global optimal solution
for problem (1).
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
(1)
Théorème 1: Assume that the lagrangean associated with (1)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x* on X when the multiplier vector
   * . If f i x* =0 for all i  1, , m, then x* is a gobal optimale
solution of (1).
 
Proof. For contradiction suppose that x*is not a global optimal solution
of (1). Then there exists another solution x such that f i  x  =0 for all
i  1,
 
, m, and f  x   f x* .
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
(1)
Theorem 1: Assume that the lagrangean function of (1)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x* on X when the multiplier vector
   * . If f i x*  0 for all i  1, , m, then x* is a gobal optimal
solution of (1).
 
Proof. For contradiction suppose that x*is not a global optimal solution
of (1). Then there exists another solution x such that f i  x  =0 for all
i  1,
 
, m, and f  x   f x* .
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
(1)
Theorem 1: Assume that the lagrangean function of (1)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x* on X when the multiplier vector
   * . If f i x*  0 for all i  1, , m, then x* is a gobal optimal
solution of (1).
 
Proof. For contradiction, suppose that x*is not a global optimal solution
of (1). Then there exists another solution x  X such that f i  x   0 for all
i  1,
 
, m, and f  x   f x* .
Theorem 1: Assume that the lagrangean function of (1)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x * on X when the multiplier vector
   * . If f i x*  0 for all i  1, , m, then x* is a gobal optimal
solution of (1).
 
Proof. For contradiction, suppose that x *is not a global optimal solution
of (1). Then there exists another solution x  X such that f i  x   0 for all
 
i  1, , m, and f  x   f x * .
Hence, for all 
m

i f i  x  
i 1
m

i 1
i f i  x *   0
and
f x
m

i 1
i f i  x   f  x*  
m

i 1
i f i  x * .
If    * , then the preceding relation contradicts the fact that
x * is a global minimum of the lagrangean on X when    * .
Theorem 1: Assume that the lagrangean function of (1)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x * on X when the multiplier vector
   * . If f i x*  0 for all i  1, , m, then x* is a gobal optimal
solution of (1).
 
Proof. For contradiction, suppose that x *is not a global optimal solution
of (1). Then there exists another solution x  X such that f i  x   0 for all
 
i  1, , m, and f  x   f x * .
Hence, for all 
m

i f i  x  
i 1
m

i 1
i f i  x *   0
and
f x
m

i 1
i f i  x   f  x*  
m

i 1
i f i  x * .
If    * , then the preceding relation contradicts the fact that
x * is a global minimum of the lagrangean on X when    * .
Theorem 1: Assume that the lagrangean function of (1)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x * on X when the multiplier vector
   * . If f i x*  0 for all i  1, , m, then x* is a gobal optimal
solution of (1).
 
Proof. For contradiction, suppose that x *is not a global optimal solution
of (1). Then there exists another solution x  X such that f i  x   0 for all
 
i  1, , m, and f  x   f x * .
Hence, for all 
m

i f i  x  
i 1
m

i 1
i f i  x *   0
and
f x
m

i 1
i f i  x   f  x*  
m

i 1
i f i  x * .
If    * , then the preceding relation contradicts the fact that
x * is a global minimum of the lagrangean on X when    * .
Theorem 1: Assume that the lagrangean function of (1)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x * on X when the multiplier vector
   * . If f i x*  0 for all i  1, , m, then x* is a gobal optimal
solution of (1).
 
Proof. For contradiction, suppose that x *is not a global optimal solution
of (1). Then there exists another solution x  X such that f i  x   0 for all
 
i  1, , m, and f  x   f x * .
Hence, for all 
m

i f i  x  
i 1
m

i 1
i f i  x *   0
and
f x
m

i 1
i f i  x   f  x*  
m

i 1
i f i  x * .
If    * , then the preceding relation contradicts the fact that
x * is a global minimum of the lagrangean on X when    * .
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
(2)
Theorem 2: Assume that the lagrangean associated with (2)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x* on X when the multiplier vector
   * . If f i x* =0, i , and i f i x* =0 for all i  1, , m,
 
 
then x* is a gobal optimale solution of (2).
Proof. For contradiction suppose that x*is not a global optimal solution
of (2). Then there exists another solution x such that f i  x   0 for all
i  1,
 
, m, and f  x   f x * .
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
(2)
Theorem 2: Assume that the lagrangean function of (2)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x* on X when the multiplier vector
   * . If f i x*  0, i*  0, and i* f i x *  0 for all i  1, , m,
 
 
then x* is a gobal optimal solution of (2).
Proof. For contradiction, suppose that x*is not a global optimal solution
of (2). Then there exists another solution x such that f i  x   0 for all
i  1,
 
, m, and f  x   f x * .
Consider the following mathematical programming problem
Min f  x 
s.t. f i  x   0 i  1, , m
x X
(2)
Theorem 2: Assume that the lagrangean function of (2)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x* on X when the multiplier vector
   * . If f i x*  0, i*  0, and i* f i x*  0 for all i  1, , m,
 
 
then x* is a gobal optimal solution of (2).
Proof. For contradiction, suppose that x*is not a global optimal solution
of (2). Then there exists another solution x  X such that f i  x   0 for all
i  1,
 
, m, and f  x   f x * .
Theorem 2: Assume that the lagrangean function of (2)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x * on X when the multiplier vector
   * . If f i x*  0, i*  0, and i* f i x*  0 for all i  1, , m,
 
 
then x * is a gobal optimal solution of (2).
Proof. For contradiction, suppose that x*is not a global optimal solution
of (2). Then there exists another solution x  X such that f i  x   0 for all
i  1,
 
, m, and f  x   f x * .
Hence, for  *  0
m

i f i  x   0
m

and
i 1
i 1
i* f i  x *   0
and
f x
m

i 1
i* f i  x   f  x *  
m

i 1
i* f i  x * .
The preceding relation contradicts the fact that x* is a global minimum
of the lagrangean on X when    *.
Theorem 2: Assume that the lagrangean function of (2)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x * on X when the multiplier vector
   * . If f i x*  0, i*  0, and i* f i x*  0 for all i  1, , m,
 
 
then x * is a gobal optimal solution of (2).
Proof. For contradiction, suppose that x*is not a global optimal solution
of (2). Then there exists another solution x  X such that f i  x   0 for all
i  1,
 
, m, and f  x   f x * .
Hence, for  *  0
m

i f i  x   0
m

and
i 1
i 1
i* f i  x *   0
and
f x
m

i 1
i* f i  x   f  x *  
m

i 1
i* f i  x * .
The preceding relation contradicts the fact that x* is a global minimum
of the lagrangean on X when    *.
Theorem 2: Assume that the lagrangean function of (2)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x * on X when the multiplier vector
   * . If f i x*  0, i*  0, and i* f i x*  0 for all i  1, , m,
 
 
then x * is a gobal optimal solution of (2).
Proof. For contradiction, suppose that x*is not a global optimal solution
of (2). Then there exists another solution x  X such that f i  x   0 for all
i  1,
 
, m, and f  x   f x * .
Hence, for  *  0
m

i f i  x   0
m

and
i 1
i 1
i* f i  x *   0
and
f x
m

i 1
i* f i  x   f  x *  
m

i 1
i* f i  x * .
The preceding relation contradicts the fact that x* is a global minimum
of the lagrangean on X when    *.
Theorem 2: Assume that the lagrangean function of (2)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x * on X when the multiplier vector
   * . If f i x*  0, i*  0, and i* f i x*  0 for all i  1, , m,
 
 
then x * is a gobal optimal solution of (2).
Proof. For contradiction, suppose that x*is not a global optimal solution
of (2). Then there exists another solution x  X such that f i  x   0 for all
i  1,
 
, m, and f  x   f x * .
Hence, for  *  0
m

i f i  x   0
m

and
i 1
i 1
i* f i  x *   0
and
f x
m

i 1
i* f i  x   f  x *  
m

i 1
i* f i  x * .
The preceding relation contradicts the fact that x* is a global minimum
of the lagrangean on X when    *.
First order Karush-Kuhn-Tucker (KKT)
optimality conditions
To have conditions easier to verify, we need additional
assumptions on X and on the fonctions f and f i .
If X is convex, and if f and f i are differentiable and convex in
problem (2)
Min f  x 
s.t. f i  x   0 i  1, , m
x X
and if i  0, i  1, , m,
L , x   f  x  
then the lagrangean
m
 f  x
i i
i 1
is also a convex fonction of x on X since
i  0 et f i  x  convex  i f i  x  convex
f  x 
m
  f  x  sum of convex fonctions.
i i
i 1
(2)
If f and f i are differentiable and convex, then the lagrangean
L , x   f  x  
m
  f  x  is also a differentiable and convex function in x,
i i
i 1
and hence x is a global minimum on X when    * if
*


    f  x   0.
 x L  * , x*  f x* 
Thus, theorem 2 can be written as:
Theorem 2: Assume that the
lagrangean function of problem (2)
L , x   f  x  
m
 f  x
i i
i 1
has a global minimum x* on X
when the multiplier vector    * .
If f i x*  0 , i*  0, and i* f i x*  0
for all i  1, , m,
then x* is a gobal optimal solution of (2).
 
 
m
i 1
*
i
*
i
If f and f i are differentiable and convex, then the lagrangean
L , x   f  x  
m
  f  x  is also a differentiable and convex function in x,
i i
i 1
and hence x is a global minimum on X when    * if
*


    f  x   0.
 x L  * , x*  f x* 
Thus, theorem 2 can be written as:
Theorem 2: Assume that the
lagrangean function of problem (2)
L , x   f  x  
m

i f i  x 
i 1
m
i 1
 
*
i
K-K-T
If there exists a vector  *such that for x*  X
 
 f x   0
f x   0
m
    f  x   0
 x L  * , x*  f x* 
has a global minimum x* on X
*
*
i
i
when the multiplier vector    * .
*
*
*
*
*
If f i x  0 , i  0, and i f i x  0 i*
i  0
for all i  1, , m,
then x* is a gobal optimal solution of (2).
 
*
i
i 1
*
i
i  1,
,m
i  1,
,m
i  1,
,m
*
i
Sufficiency of the K-K-T conditions
Referring to theorem 2, we can show that the K-K-T conditions are
sufficient when in addition X is convex and the fonctions f and f i
are convex on X .
Sufficiency of the K-K-T conditions
Referring to theorem 2, we can show that the K-K-T conditions are
sufficient when in addition X is convex and the fonctions f and f i
are convex on X .
We can also use the gradient inequality to show the same result.
Theorem 3: Assume that X is convex and that the fonctions f et f i are
differentiable and convex. If the K-K-T conditions are verified at x* ,
then x* is a global minimum of the problem  2  .
Proof. Since the lagrangean is convex (shown before),
then it follows from the gradient inequality that for all x  X
f  x 
m

i 1
 f  x  f  x  
*
i i
*
m

i 1

 f x  f x* 

*
i i
 
*
m
  
i 1

 f i x 

*
i
T
  x  x 
*
*
Sufficiency of the K-K-T conditions
Referring to theorem 2, we can show that the K-K-T conditions are
sufficient when in addition X is convex and the fonctions f et f i
are convex on X .
We can also use the gradient inequality to show the same result.
Theorem 3: Assume that X is convex and that the fonctions f and f i are
differentiable and convex. If the K-K-T conditions are verified at x* ,
then x* is a global minimum of the problem  2  .
Proof. Since the lagrangean is convex (shown before),
then it follows from the gradient inequality that for all x  X
f  x 
m

i 1
 f  x  f  x  
*
i i
*
m

i 1

 f x  f x* 

*
i i
 
*
m
  
i 1

 f i x 

*
i
T
  x  x 
*
*
Theorem 3: Assume that X is convex and that the fonctions f and f i are
differentiable and convex. If the K-K-T conditions are verified at x* ,
then x* is a global minimum of the problem  2  .
Proof. Since the lagrangean is convex (shown before),
then it follows from the gradient inequality that for all x  X
f  x 
m

i 1
 f  x  f  x  
*
i i
*
m

i 1

 f x  f x * 

*
i i
 
*
m
  
i 1

 f i x 

  x  x 
*
i
*
0
Alors, pour tout x  X
 
f x*  f  x  
et
 
T
m

i 1
f x*  f  x  .
0
i* f i  x   0
*
Theorem 3: Assume that X is convex and that the fonctions f and f i are
differentiable and convex. If the K-K-T conditions are verified at x* ,
then x* is a global minimum of the problem  2  .
Proof. Since the lagrangean is convex (shown before),
then it follows from the gradient inequality that for all x  X
f  x 
m

i 1
 f  x  f  x  
*
i i
*
m

i 1

 f x  f x * 

*
i i
 
*
m
  
i 1

 f i x 

  x  x 
*
i
*
0
Alors, pour tout x  X
K-K-T
 
 f x   0
f x   0
m
  
 x L  * , x*  f x* 
*
i i
*
*
i
i*  0
i 1
 
f x*  f  x  
i*f i  x*   0
i  1,
,n
i  1,
,n
i  1,
,n
et
 
T
m

i 1
f x*  f  x  .
0
i* f i  x   0
*
Theorem 3: Assume that X is convex and that the fonctions f and f i are
differentiable and convex. If the K-K-T conditions are verified at x* ,
then x* is a global minimum of the problem  2  .
Proof. Since the lagrangean is convex (shown before),
then it follows from the gradient inequality that for all x  X
f  x 
m

i 1
 f  x  f  x  
*
i i
*
m

i 1

 f x  f x * 

*
i i
0
K-K-T
 
 f x   0
f x   0
m
  
 x L  * , x*  f x* 
*
i i
*
*
i
i*  0
i 1
 
*
,n
i  1,
,n
i  1,
,n
  
i 1

 f i x 

T
  x  x 
*
i
*
0
Alors, pour tout x  X
 
f x*  f  x  
i*f i  x*   0
i  1,
m
et
 
m

i 1
f x*  f  x  .
i* f i  x   0
*
Theorem 3: Assume that X is convex and that the fonctions f and f i are
differentiable and convex. If the K-K-T conditions are verified at x* ,
then x* is a global minimum of the problem  2  .
Proof. Since the lagrangean is convex (shown before),
then it follows from the gradient inequality that for all x  X
f  x 
m

i 1
 f  x  f  x  
*
i i
*
m

i 1

 f x  f x * 

*
i i
0
K-K-T
 
 f x   0
f x   0
m
  
 x L  * , x*  f x* 
*
i i
*
*
i
i*  0
i 1
i*f i  x*   0
i  1,
,n
i  1,
,n
i  1,
,n
 
m
  
*
i 1

 f i x 

*
i
T
  x  x 
*
0
Then, for all x  X
 
f x*  f  x  
and
 
m

i 1
f x*  f  x  .
i* f i  x   0
*
Theorem 3: Assume that X is convex and that the fonctions f and f i are
differentiable and convex. If the K-K-T conditions are verified at x* ,
then x* is a global minimum of the problem  2  .
Proof. Since the lagrangean is convex (shown before),
then it follows from the gradient inequality that for all x  X
f  x 
m

i 1
 f  x  f  x  
*
i i
*
m

i 1

 f x  f x * 

*
i i
0
K-K-T
m
   f  x     f  x   0
 f x   0
i  1, , n
f x   0
i  1, , n
x L  , x
*
*
*
i 1
*
i i
*
i
*
*
i
i*  0
i  1,
,n
*
i
 
m
  
*
i 1

 f i x 

*
i
T
  x  x 
*
0
Then, for all x  X
 
f x  f x 
*
and
 
m

i 1
f x*  f  x  .
i* f i  x   0
*
Necessity of the K-K-T conditions
 
Assume that x*  X , f i x*  0, i  1,
problem  2  :
, m, and is a local minimum of
Min f  x 
s.t. f i  x   0 i  1,
x X.
Are the K-K-T satisfied; i.e.,
can we find a multiplier vector  *such that
 
 f x   0
f x   0
    f  x   0
 x L  * , x*  f x* 
*
i i
*
*
i
i*  0
m
,m
i 1
*
i
i  1,
,n
i  1,
,n
i  1,
,n ?
*
i
(2)
 
Assume that x*  X , f i x*  0, i  1,
problem  2  :
, m, and is a local minimum of
Min f  x 
s.t. f i  x   0 i  1,
x X.
Are the K-K-T satisfied; i.e.,
can we find a multiplier vector  *such that
 
 f x   0
f x   0
x L  , x
*
*
i i
*
*
i
i*  0
*
m
  
 f x 
*
i 1
i*f i  x*   0
i  1,
,n
i  1,
,n
i  1,
,m
,n ?
(2)
 
Assume that x*  X , f i x*  0, i  1,
problem  2  :
, m, and is a local minimum of
Min f  x 
s.t. f i  x   0 i  1,
x X.
Are the K-K-T satisfied; i.e.,
can we find a multiplier vector  *such that
 
 f x   0
f x   0
*
*
i
i*  0
(2)
m
    f  x   0
 x L  * , x*  f x* 
*
i i
,m
i 1
*
i
i  1,
,n
i  1,
,n
*
i
i  1, , n ?
It is more difficult to answer to this question than it was in studying the
suffidiency.
The K-K-T are satisfied when the feasible domain of problem (2)
(and hence the constraint functions f i , i  1, , n ) verifies some
conditions.
Different sets of such conditions exist.
To analyse the necessity of the K-K-T conditions, additional notions and
preliminary results related to theorems of alternatives are required
Definitions.
The hyperplan specified by a point a  R n and a scalar  is the following set in R n
H  a,    x  R n : a T x   .


The half spaces (closed) associated with the hyperplan H  a,   are the following
sets in R n :
H  a,    x  R n : a T x  
H   a,  
H

 a ,    x  R
n
: aT

x   .
Remark. It is easy to verify that these sets are convex.
H  a,  
H   a,  
To analyse the necessity of the K-K-T conditions, additional notions and
preliminary results related to theorems of alternatives are required
Definitions.
The hyperplan specified by a point a  R n and a scalar  is the following set in R n
H  a,    x  R n : a T x   .


The half spaces (closed) associated with the hyperplan H  a,   are the following
sets in R n :
H  a,    x  R n : a T x  
H   a,  
H

 a ,    x  R
n
: aT

x   .
Remark. It is easy to verify that these sets are convex.
H  a,  
H   a,  
To analyse the necessity of the K-K-T conditions, additional notions and
preliminary results related to theorems of alternatives are required
Definitions.
The hyperplan specified by a point a  R n and a scalar  is the following set in R n
H  a,    x  R n : a T x   .


The half spaces (closed) associated with the hyperplan H  a,   are the following
sets in R n :
H  a,    x  R n : a T x  
H   a,  
H

 a ,    x  R
n
: aT

x   .
Remark. It is easy to verify that these sets are convex.
Remark: It is easy to verify that these sets are convex.
H  a,  
H   a,  
Definition. Separating hyperplan .
The hyperplan H  a,   separates two non empty sets X and Y if


 i.e., Y  H a,   .
a T x   for all x  X i.e., X  H  a ,  

a T y   for all y  Y
The separation is strict if the inequalities in both preceding relations
are strict.

H a1 , 1

Y
  hyperplan de séparation
H  a ,   hyperplan de séparation stricte


H a1 , 1
2
2
X
H a2 , 2
Definition. Separating hyperplan .
The hyperplan H  a,   separates two non empty sets X and Y if


 i.e., Y  H a,   .
a T x   for all x  X i.e., X  H  a ,  

a T y   for all y  Y
The separation is strict if the inequalities in both preceding relations
are strict.

H a1 , 1

  separating hyperplan
H  a ,   strict separating hyperplan
H a1 , 1
2
Y
2
X

H a2 , 2

Theorem 4: Let the vectors x, y , a  R n . If a T y  a T x, then for all
   0,1 ,
a T y  a T  x  1    y   a T x.
Proof.
a T  x  1    y   a T  x  1    x   a T x.
a T  x  1    y   a T  y  1    y   a T y.
Theorem 4: Let the vectors x, y , a  R n . If a T y  a T x, then for all
   0,1 ,
a T y  a T  x  1    y   a T x.
Proof.
a T  x  1    y   a T  x  1    x   a T x.
a T  x  1    y   a T  y  1    y   a T y.
Théorème 5: (Separating theorem) If X  R n is a non empty convex
set and if y  X , then there exists an hyperplan separating (strictly)
X and y.
X
Proof. There exists a point x 0  X such that
X
0
x  y  min x  y

xX

y

z  z T z denotes the euclidean norm of z .
where
It is easy to verify that X is a convex set, and consequently the
line segment  x 0 , x  X for all x  X . Then for all   0,1

Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
0
0
0



x 0  y   x  1    x 0  y .
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
 x  x  x  x 
 y   x  1    x
0
0
T
0
0
T
0
T
0
0
0
0
T
0
2
0
T
0
Théorème 5: (Separating theorem) If X  R n is a non empty convex
set and if y  X , then there exists an hyperplan separating (strictly)
X and y.
0
0
x
Proof. There exists a point x  X such that

X
y
0
x  y  min x  y

xX

z  z T z denotes the euclidean norm of z .
where
It is easy to verify that X is a convex set, and consequently the
line segment  x 0 , x  X for all x  X . Then for all   0,1

Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
0
0
0



x 0  y   x  1    x 0  y .
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
 x  x  x  x 
 y   x  1    x
0
0
T
0
0
T
0
T
0
0
0
0
T
0
2
0
T
0
Théorème 5: (Separating theorem) If X  R n is a non empty convex
set and if y  X , then there exists an hyperplan separating (strictly)
X and y.
0
0
x
Proof. There exists a point x  X such that

X
y
0
x  y  min x  y

xX

z  z T z denotes the euclidean norm of z .
where
It is easy to verify that X is a convex set, and consequently the
line segment  x 0 , x  X for all x  X . Then for all   0,1

Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
0
0
0



x 0  y   x  1    x 0  y .
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
 x  x  x  x 
 y   x  1    x
0
0
T
0
0
T
0
T
0
0
0
0
T
0
2
0
T
0
Théorème 5: (Separating theorem) If X  R n is a non empty convex
set and if y  X , then there exists an hyperplan separating (strictly)
X and y.
0
0
x
Proof. There exists a point x  X such that

X
y
0
x  y  min x  y

xX

z  z T z denotes the euclidean norm of z .
where
It is easy to verify that X is a convex set, and consequently the
line segment  x 0 , x  X for all x  X . Then for all   0,1

Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
0
0
0



x 0  y   x  1    x 0  y .
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
 x  x  x  x 
 y   x  1    x
0
0
T
0
0
T
0
T
0
0
0
0
T
0
2
0
T
0
Théorème 5: (Separating theorem) If X  R n is a non empty convex
set and if y  X , then there exists an hyperplan separating (strictly)
X and y.
0
0
x
Proof. There exists a point x  X such that

X
y
0
x  y  min x  y

xX

z  z T z denotes the euclidean norm of z .
where
It is easy to verify that X is a convex set, and consequently the
line segment  x 0 , x  X for all x  X . Then for all   0,1

Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
0
0
0



x 0  y   x  1    x 0  y .
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
 x  x  x  x 
 y   x  1    x
0
0
T
0
0
T
0
T
0
0
0
0
T
0
2
0
T
0
Théorème 5: (Separating theorem) If X  R n is a non empty convex
set and if y  X , then there exists an hyperplan separating (strictly)
X and y.
0
0
x
Proof. There exists a point x  X such that

X
y
0
x  y  min x  y

xX

z  z T z denotes the euclidean norm of z .
where
It is easy to verify that X is a convex set, and consequently the
line segment  x 0 , x  X for all x  X . Then for all   0,1

Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
0
0
0



x 0  y   x  1    x 0  y .
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
 x  x  x  x 
 y   x  1    x
0
0
T
0
0
T
0
T
0
0
0
0
T
0
2
0
T
0
Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
  x  x   x  x .
0
 y   x  1    x
0
0
0
T
0
0
T
0
0
T
0
0
0
T
0
0
2
Consequently for all   0,1

2 x  x

xx
 x
T

 y 
0
2
T
 x  x   x  x   0.
T
0
0

  x  y   0. Indeed if
  x  y   0, then for  >0 small enough
2  x  x   x  y     x  x   x  x 
But this implies x  x
0
0
0
T
0
T
0
0
0
and then we would have

2 x  x
0
T
0
 x
T
0
2

 y 
2
0
T
0
 x  x   x  x   0.
0
T
0
0
Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
  x  x   x  x .
0
 y   x  1    x
0
0
0
T
0
0
T
0
0
T
0
0
0
T
0
0
2
Consequently for all   0,1

2 x  x

xx
 x
T

 y 
0
2
T
 x  x   x  x   0.
T
0
0

  x  y   0. Indeed if
  x  y   0, then for  >0 small enough
2  x  x   x  y     x  x   x  x 
But this implies x  x
0
0
0
T
0
T
0
0
0
and then we would have

2 x  x
0
T
0
 x
T
0
2

 y 
2
0
T
0
 x  x   x  x   0.
0
T
0
0
Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
  x  x   x  x .
0
 y   x  1    x
0
0
0
T
0
0
T
0
0
T
0
0
0
T
0
0
2
Consequently for all   0,1

2 x  x

xx
 x
T

 y 
0
2
T
 x  x   x  x   0.
T
0
0

  x  y   0. Indeed if
  x  y   0, then for  >0 small enough
2  x  x   x  y     x  x   x  x 
But this implies x  x
0
0
0
T
0
T
0
0
0
and then we would have

2 x  x
0
T
0
 x
T
0
2

 y 
2
0
T
0
 x  x   x  x   0.
0
T
0
0
Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
  x  x   x  x .
0
 y   x  1    x
0
0
0
T
0
0
T
0
0
T
0
0
0
T
0
0
2
Consequently for all   0,1

2 x  x

xx
 x
T

 y 
0
2
T
 x  x   x  x   0.
T
0
0

  x  y   0. Indeed if
  x  y   0, then for  >0 small enough
2  x  x   x  y     x  x   x  x 
But this implies x  x
0
0
0
T
0
T
0
0
0
and then we would have

2 x  x
0
T
0
 x
T
0
2

 y 
2
0
T
0
 x  x   x  x   0.
0
T
0
0
Thus for all   0,1
x
x
x
0
0
0
 x
 y x
 y x
y
T
T
T
 
  y    x  1    x   y 
 y    x  y     x  x   x  y     x  x 
 y    x  y   x  y   2  x  x   x  y  
  x  x   x  x .
0
 y   x  1    x
0
0
0
T
0
0
T
0
0
T
0
0
0
T
0
0
2
Consequently for all   0,1

2 x  x

xx
 x
T

 y 
0
2
T
 x  x   x  x   0.
T
0
0

  x  y   0. Indeed if
  x  y   0, then for  >0 small enough
2  x  x   x  y     x  x   x  x 
But this implies x  x
0
0
0
T
0
T
0
0
0
and then we would have

2 x  x
0
T
0
 x
T
0
2

 y 
2
0
T
0
 x  x   x  x   0.
0
T
0
0



x  x  y   x  x  y .
Since y  X , x  y   x  y   x  y   0, or
y  x  y   x  x  y .
Thus applying theorem 4 with the vectors  x  y  ,
We then have x  x
0T
0 T
x 0  y  0 or
0
T
2
0
T
T
0
0
 5.3
0
0T
0
 5.4 
0
0
1
x , y , using  = , and referring to  5.4 
2
T
1 0
T
0
y x y  x y
x 0  y  x 0T x 0  y .
2
Combining with relation  5.3 ,
0

y
T





1 0
x y  x y
2
0


T








x 0  y  x 0T x 0  y  x T x 0  y .



x  x  y   x  x  y .
Since y  X , x  y   x  y   x  y   0, or
y  x  y   x  x  y .
Thus applying theorem 4 with the vectors  x  y  ,
We then have x  x
0T
0 T
x 0  y  0 or
0
T
2
0
T
T
0
0
 5.3
0
0T
0
 5.4 
0
0
1
x , y , using  = , and referring to  5.4 
2
T
1 0
T
0
y x y  x y
x 0  y  x 0T x 0  y .
2
Combining with relation  5.3 ,
0

y
T





1 0
x y  x y
2
0


T








x 0  y  x 0T x 0  y  x T x 0  y .



x  x  y   x  x  y .
Since y  X , x  y   x  y   x  y   0, or
y  x  y   x  x  y .
Thus applying theorem 4 with the vectors  x  y  ,
We then have x  x
0T
0 T
x 0  y  0 or
0
T
2
0
T
T
0
0
 5.3
0
0T
0
 5.4 
0
0
1
x , y , using  = , and referring to  5.4 
2
T
1 0
T
0
y x y  x y
x 0  y  x 0T x 0  y .
2
Combining with relation  5.3 ,
0

y
T





1 0
x y  x y
2
0


T








x 0  y  x 0T x 0  y  x T x 0  y .



x  x  y   x  x  y .
Since y  X , x  y   x  y   x  y   0, or
y  x  y   x  x  y .
Thus applying theorem 4 with the vectors  x  y  ,
We then have x  x
0T
0 T
x 0  y  0 or
0
T
2
0
T
T
0
0
 5.3
0
0T
0
 5.4 
0
0
1
x , y , using  = , and referring to  5.4 
2
T
1 0
T
0
y x y  x y
x 0  y  x 0T x 0  y .
2
Combining with relation  5.3 ,
0

y
T





1 0
x y  x y
2
0


T








x 0  y  x 0T x 0  y  x T x 0  y .
Combining with relation  5.3 ,



1 0
y x y  x y
2
Thus
T
0



T









T
1 0
y x y  x y
x0  y  xT x0  y .
2
Since this relation holds for all x  X , it follows that
T
0


the hyperplan H  a,   where a  x 0  y and

1 0
=
x y
2

T


x 0  y  x 0T x 0  y  x T x 0  y .
x 0  y is separating (strictly) X and y.
Combining with relation  5.3 ,



1 0
y x y  x y
2
Thus
T
0



T









T
1 0
y x y  x y
x0  y  xT x0  y .
2
Since this relation holds for all x  X , it follows that
T
0


the hyperplan H  a,   where a  x 0  y and

1 0
=
x y
2

T


x 0  y  x 0T x 0  y  x T x 0  y .
x 0  y is separating (strictly) X and y.
Combining with relation  5.3 ,



1 0
y x y  x y
2
Thus
T
0



T









T
1 0
y x y  x y
x0  y  xT x0  y .
2
Since this relation holds for all x  X , it follows that
T
0


the hyperplan H  a,   where a  x 0  y and

1 0
=
x y
2

T


x 0  y  x 0T x 0  y  x T x 0  y .
x 0  y is separating (strictly) X and y.
X
X
y
y
X
X
y
y
In theorem 5, the convexity of X is a sufficient condition
insuring that an hyperplan exists to separate it
(strictly) from y  X .
But as illustrated in the following figure, the convexity of X
is not a necessary condition.
y
X
In theorem 5, the convexity of X is a sufficient condition
insuring that an hyperplan exists to separate it
(strictly) from y  X .
But as illustrated in the following figure, the convexity of X
is not a necessary condition.
y
X
In theorem 5, the convexity of X is a sufficient condition
insuring that an hyperplan exists to separate it
(strictly) from y  X .
But as illustrated in the following figure, the convexity of X
is not a necessary condition.
y
X
Theorem 6:(Farkas Lemma) Let the vectors
a1 ,
,a n and b  R m . A sufficient condition for b
to be a non negative linear combination of the a j
(i.e., a sufficient condition for the existence of non negative
scalars x1,
, xn such that b  a1 x1 
 a n xn ) is that each time
there exists y  R m such that y T a j  0 for all j  1,
, n, then
necessarely it follows that y T b  0.
Proof. We have to show that
for all y  R m ,
 T j
if y a  0 for all j  1, , n,

T
then
necessarily
y
b0


  x1 , , xn  0 such that 


1
n

  b  a x1   a xn

Theorem 6:(Farkas Lemma) Let the vectors
a1 ,
,a n and b  R m . A sufficient condition for b
to be a non negative linear combination of the a j
(i.e., a sufficient condition for the existence of non negative
scalars x1,
, xn such that b  a1 x1 
 a n xn ) is that each time
there exists y  R m such that y T a j  0 for all j  1,
, n, then
necessarely it follows that y T b  0.
Proof. We have to show that
for all y  R m ,
 T j
if y a  0 for all j  1, , n,

T
then
necessarily
y
b0


  x1 , , xn  0 such that 


1
n

  b  a x1   a xn

Proof. We have to show that
for all y  R m ,

 T j
  x1 , , xn  0 such that 
if y a  0 for all j  1, , n,   

1
n


  b  a x1   a xn
T
 then necessarily y b  0

We will rather show the contrapose of the implication:
for all y  R m ,
 T j
 if y a  0 for all j  1, , n,

T
then
necessarily
y
b0

i.e.,


 x1 , , xn  0 such that 
  

1
n
 b  a x1   a xn



 y  R m , such that
  x1 , , xn  0 such that   T j

   y a  0 for all j  1,
1
n
 b  a x1   a xn
 
T
and
y
b0



, n, 


Proof. We have to show that
for all y  R m ,

 T j
  x1 , , xn  0 such that 
if y a  0 for all j  1, , n,   

1
n


  b  a x1   a xn
T
 then necessarily y b  0

We will rather show the contrapose of the implication:
for all y  R m ,
 T j
 if y a  0 for all j  1, , n,

T
then
necessarily
y
b0

i.e.,


 x1 , , xn  0 such that 
  

1
n
 b  a x1   a xn



 y  R m , such that
  x1 , , xn  0 such that   T j

   y a  0 for all j  1,
1
n
 b  a x1   a xn
 
T
and
y
b0



, n, 


Proof. We have to show that
for all y  R m ,

 T j
  x1 , , xn  0 such that 
if y a  0 for all j  1, , n,   

1
n


  b  a x1   a xn
T
 then necessarily y b  0

We will rather show the contrapose of the implication:
for all y  R m ,
 T j
 if y a  0 for all j  1, , n,

T
then
necessarily
y
b0

i.e.,


 x1 , , xn  0 such that 
  

1
n
 b  a x1   a xn



 y  R m , such that
  x1 , , xn  0 such that   T j

   y a  0 for all j  1,
1
n
 b  a x1   a xn
 
T
and
y
b0



, n, 


To show that
 y  R m , such that
  x1 , , xn  0 such that   T j

   y a  0 for all j  1,
1
n
 b  a x1   a xn
 
T
and
y
b0

consider the following set

Z  z  R m : x1,
, xn  0 such that z  a1x1 


, n, 



 a n xn .
It is easy to show that Z is convex. It is also possible to show
that Z is a close set  i.e. Z  Z  .
z1  a1 x11 
 a n x1n
z 2  a1 x12 
 z1  1    z 2  a1  x11  1    x12  
and then  z1  1    z 2  Z
 a n x 2n
 a n  x1n  1    xn2 
To show that
 y  R m , such that
  x1 , , xn  0 such that   T j

   y a  0 for all j  1,
1
n
 b  a x1   a xn
 
T
and
y
b0

consider the following set

Z  z  R m : x1,
, xn  0 such that z  a1x1 


, n, 



 a n xn .
It is easy to show that Z is convex. It is also possible to show
that Z is a close set  i.e. Z  Z  .
z1  a1 x11 
 a n x1n
z 2  a1 x12 
 z1  1    z 2  a1  x11  1    x12  
and then  z1  1    z 2  Z
 a n x 2n
 a n  x1n  1    xn2 
To show that
 y  R m , such that
  x1 , , xn  0 such that   T j

   y a  0 for all j  1,
1
n
 b  a x1   a xn
 
T
and
y
b0

consider the following set

Z  z  R m : x1,
, xn  0 such that z  a1x1 


, n, 



 a n xn .
It is easy to show that Z is convex. It is also possible to show
that Z is a close set  i.e. Z  Z  .
z1  a1 x11 
 a n x1n
z 2  a1 x12 
 z1  1    z 2  a1  x11  1    x12  
and then  z1  1    z 2  Z
 a n x 2n
 a n  x1n  1    xn2 
To show that
 y  R m , such that
  x1 , , xn  0 such that   T j

   y a  0 for all j  1,
1
n
 b  a x1   a xn
 
T
and
y
b0

consider the following set

Z  z  R m : x1 ,
, xn  0 such that z  a1 x1 


, n, 



 a n xn .
It is easy to show that Z is convex. It is also possible to show
that Z is a close set  i.e. Z  Z  .
Since that an assumptio of the contrapose is that b  Z  Z , then
by theorem 5, there exists an hyperplan H ( p,  ) separating
strictly Z and b :
pTb    p T z
for all z  Z .
(5.5)
To show that
 y  R m , such that
  x1 , , xn  0 such that   T j

   y a  0 for all j  1,
1
n
 b  a x1   a xn
 
T
and
y
b0

consider the following set

Z  z  R m : x1 ,
, xn  0 such that z  a1 x1 


, n, 



 a n xn .
It is easy to show that Z is convex. It is also possible to show
that Z is a close set  i.e. Z  Z  .
Since that an assumption of the contrapose is that b  Z  Z , then
by theorem 5, there exists an hyperplan H ( p,  ) separating
strictly Z and b :
pTb    p T z
for all z  Z .
(5.5)

Z  z  R m : x1,

, xn  0 such that z  a1 x1 
 a n xn .
Since that an assumption of the contrapose is that b  Z  Z , then
by theorem 5, there exists an hyperplan H ( p,  ) separating
strictly Z and b :
pTb    pT z
for all z  Z .
(5.5)
But 0  Z ,implying that   0,and consequently
p T b  0.
Furthermore, p T z  0 for all z  Z . Indeed, if z  Z were existing
such that p T z  0, then since Z is a cone (i.e., if z  Z then
 z  Z for all   0), we would get that p T   z    
 
contradicting (5.5).

Z  z  R m : x1,

, xn  0 such that z  a1 x1 
 a n xn .
Since that an assumption of the contrapose is that b  Z  Z , then
by theorem 5, there exists an hyperplan H ( p,  ) separating
strictly Z and b :
pTb    pT z
for all z  Z .
(5.5)
But 0  Z ,implying that   0,and consequently
p T b  0.
Furthermore, p T z  0 for all z  Z . Indeed, if z  Z were existing
such that p T z  0, then since Z is a cone (i.e., if z  Z then
 z  Z for all   0), we would get that p T   z    
 
contradicting (5.5).

Z  z  R m : x1,

, xn  0 such that z  a1 x1 
 a n xn .
Since that an assumption of the contrapose is that b  Z  Z , then
by theorem 5, there exists an hyperplan H ( p,  ) separating
strictly Z and b :
pTb    pT z
for all z  Z .
(5.5)
But 0  Z ,implying that   0,and consequently
p T b  0.
Furthermore, p T z  0 for all z  Z . Indeed, if z  Z were existing
such that p T z  0, then since Z is a cone (i.e., if z  Z then
 z  Z for all   0), we would get that p T   z    
 
contradicting (5.5).

Z  z  R m : x1,

, xn  0 such that z  a1 x1 
 a n xn .
Since that an assumption of the contrapose is that b  Z  Z , then
by theorem 5, there exists an hyperplan H ( p,  ) separating
strictly Z and b :
pTb    pT z
for all z  Z .
(5.5)
But 0  Z ,implying that   0,and consequently
p T b  0.
Furthermore, p T z  0 for all z  Z . Indeed, if z  Z were existing
such that p T z  0, then since Z is a cone (i.e., if z  Z then
 z  Z for all   0), we would get that p T   z    
 
contradicting (5.5).
But 0  Z ,implying that   0,and consequently
p T b  0.
Furthermore, p T z  0 for all z  Z . Indeed, if z  Z were existing
such that p T z  0, then since Z is a cone (i.e., if z  Z then
 z  Z for all   0), we would get that p T   z    
 
contradicting (5.5).
Since it is easy to verify that a j  Z , j  1,

Z  z  R m : x1 ,
, n,
, xn  0 tel que z  a1 x1 
it follows that p T a j  0, j  1,
, n.
We have shown the contrapose since p  R m is
such that p T a j  0, j  1,
, n and p Tb  0.
 a n xn

But 0  Z ,implying that   0,and consequently
p T b  0.
Furthermore, p T z  0 for all z  Z . Indeed, if z  Z were existing
such that p T z  0, then since Z is a cone (i.e., if z  Z then
 z  Z for all   0), we would get that p T   z    
 
contradicting (5.5).
Since it is easy to verify that a j  Z , j  1,

Z  z  R m : x1 ,
, n,
, xn  0 such that z  a1 x1 
it follows that p T a j  0, j  1,
, n.
We have shown the contrapose since p  R m is
such that p T a j  0, j  1,
, n and p Tb  0.
 a n xn

But 0  Z ,implying that   0,and consequently
p T b  0.
Furthermore, p T z  0 for all z  Z . Indeed, if z  Z were existing
such that p T z  0, then since Z is a cone (i.e., if z  Z then
 z  Z for all   0), we would get that p T   z    
 
contradicting (5.5).
Since it is easy to verify that a j  Z , j  1,

Z  z  R m : x1 ,
, n,
, xn  0 such that z  a1 x1 
it follows that p T a j  0, j  1,
, n.
We have shown the contrapose since p  R m is
such that p T a j  0, j  1,
, n and p Tb  0.
 a n xn

Theorem 6:(Farkas Lemma) Let the vectors
a1 ,
,a n and b  R m . A sufficient condition for b
to be a non negative linear combination of the a j
(i.e., a sufficient condition for the existence of non negative
scalars x1,
, xn such that b  a1 x1 
 a n xn ) is that each time
there exists y  R m such that y T a j  0 for all j  1,
, n, then
necessarely it follows that y T b  0.
Proof. We have to show that
for all y  R m ,
 T j
if y a  0 for all j  1, , n,

T
then
necessarily
y
b0


  x1 , , xn  0 such that 


1
n

  b  a x1   a xn

Corollary 7: (Theorem of alternatives) Let A be a matrix
m  n. Exactly one of the two following alternatives holds:
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
The system a1 a2
an  x  b, x  0 has a solution x  R n
T
is easya toT verify
IIProof.
The Itsystem
y  0, that
j  1,the, two
n, balternatives
y  0 has a cannot
solutionhold
y  Rm.
I
j
simultaneously, since otherwise the following relation would be
verified:
0  bT y  x T AT y  0
a contradiction.
Corollary 7: (Theorem of alternatives) Let A be a matrix
m  n. Exactly one of the two following alternatives holds:
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
Proof. It is easy to verify that the two alternatives cannot hold
simultaneously, since otherwise the following relation would be
verified:
0  bT y  x T AT y  0
a contradiction.
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
I
The system a1 a2
an  x  b, x  0 has a solution x  R n
II The system a j T y  0, j  1,
Let a j , j  1,
, n, bT y  0 has a solution y  R m .
, n, be the j ième column of A.
Consider the alternative II. It is verified or not.
In the case where it is not holding, It follows that the system
AT y  0, bT y  0 has no solution y  R m ;
i.e., the system a j T y  0, j  1,
, n, bT y  0 has no
solution y  R m . Then for all y  R m ,
if a j T y  0, j  1,
, n, then necessarily bT y  0.
Then by theorem 6, there exists a vector x  R n , x  0 such
that Ax  a1 x1 
 an xn  b, and alternative I holds.
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
I
The system a1 a2
an  x  b, x  0 has a solution x  R n
II The system a j T y  0, j  1,
Let a j , j  1,
, n, bT y  0 has a solution y  R m .
, n, be the j ième column of A.
Consider the alternative II. It is verified or not.
In the case where it is not holding, It follows that the system
AT y  0, bT y  0 has no solution y  R m ;
i.e., the system a j T y  0, j  1,
, n, bT y  0 has no
solution y  R m . Then for all y  R m ,
if a j T y  0, j  1,
, n, then necessarily bT y  0.
Then by theorem 6, there exists a vector x  R n , x  0 such
that Ax  a1 x1 
 an xn  b, and alternative I holds.
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
I
The system a1 a2
an  x  b, x  0 has a solution x  R n
II The system a j T y  0, j  1,
Let a j , j  1,
, n, bT y  0 has a solution y  R m .
, n, be the j ième column of A.
Consider the alternative II. It is verified or not.
In the case where it is not holding, it follows that the system
AT y  0, bT y  0 has no solution y  R m ;
i.e., the system a j T y  0, j  1,
, n, bT y  0 has no
solution y  R m . Then for all y  R m ,
if a j T y  0, j  1,
, n, then necessarily bT y  0.
Then by theorem 6, there exists a vector x  R n , x  0 such
that Ax  a1 x1 
 an xn  b, and alternative I holds.
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
I
The system a1 a2
an  x  b, x  0 has a solution x  R n
II The system a j T y  0, j  1,
Let a j , j  1,
, n, bT y  0 has a solution y  R m .
, n, be the j ième column of A.
Consider the alternative II. It is verified or not.
In the case where it is not holding, it follows that the system
AT y  0, bT y  0 has no solution y  R m ;
i.e., the system a j T y  0, j  1,
, n, bT y  0 has no
solution y  R m . Then for all y  R m ,
if a j T y  0, j  1,
, n, then necessarily bT y  0.
Then by theorem 6, there exists a vector x  R n , x  0 such
that Ax  a1 x1 
 an xn  b, and alternative I holds.
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
I
The system a1 a2
an  x  b, x  0 has a solution x  R n
II The system a j T y  0, j  1,
Let a j , j  1,
, n, bT y  0 has a solution y  R m .
, n, be the j ième column of A.
Consider the alternative II. It is verified or not.
In the case where it is not holding, it follows that the system
AT y  0, bT y  0 has no solution y  R m ;
i.e., the system a j T y  0, j  1,
, n, bT y  0 has no
solution y  R m . Then for all y  R m ,
if a j T y  0, j  1,
, n, then necessarily bT y  0.
Then by theorem 6, there exists a vector x  R n , x  0 such
that Ax  a1 x1 
 an xn  b, and alternative I holds.
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
I
The system a1 a2
an  x  b, x  0 has a solution x  R n
II The system a j T y  0, j  1,
Let a j , j  1,
, n, bT y  0 has a solution y  R m .
, n, be the j ième column of A.
Consider the alternative II. It is verified or not.
In the case where it is not holding, it follows that the system
AT y  0, bT y  0 has no solution y  R m ;
i.e., the system a j T y  0, j  1,
, n, bT y  0 has no
solution y  R m . Then for all y  R m ,
if a j T y  0, j  1,
, n, then necessarily bT y  0.
Then by theorem 6, there exists a vector x  R n , x  0 such
that Ax  a1 x1 
 an xn  b, and alternative I holds.
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m
I
The system a1 a2
an  x  b, x  0 has a solution x  R n
II The system a j T y  0, j  1,
, n, bT y  0 has a solution y  R m .
The case where the system Ax  b, x  0 has a solution.
cT d  c d cos 
a1
a1x1
b y0
T
a2 x2
a1T y  0
b
a2
and
a2T y  0
The system AT y  0, bT y  0
has no solution
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m
I
The system a1 a2
an  x  b, x  0 has a solution x  R n
II The system a j T y  0, j  1,
, n, bT y  0 has a solution y  R m .
The case where the system Ax  b, x  0 has no solution solution.
b
a1
a1T y  0
b y0
T
bT y  0
b
a2
and
a2T y  0
The system AT y  0, bT y  0
has a solution
Now return to analyse the necessity of the K-K-T conditions using
Corollary 7.
 
Suppose that x*  X , f i x *  0, i  1,
problem 2.
, m, is a local optimal solution of
Notation: Denote the set of active constraints
  
  
A x*  i : f i x *  0  i1 ,
, ik   1,
, m.
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
f i x
*
*
T
d 0
T
d 0
 
i  A x*
 5.6 
Now return to analyse the necessity of the K-K-T conditions using
Corollary 7.
 
Suppose that x*  X , f i x *  0, i  1, , m, is a local optimal solution of
problem 2.
Min f  x 
Sujet à f i  x   0 i  1,
x X
,m
(2)
Notation: Denote the set of active constraints
  
  
A x*  i : f i x *  0  i1 ,
, ik   1,
, m.
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
f i x
*
*
T
d 0
T
d 0
 
i  A x*
 5.6 
Now return to analyse the necessity of the K-K-T conditions using
Corollary 7.
 
Suppose that x*  X , f i x *  0, i  1, , m, is a local optimal solution of
problem 2.
Min f  x 
Sujet à f i  x   0 i  1,
x X
,m
(2)
Notation: Denote the set of active constraints
  
  
A x*  i : f i x *  0  i1 ,
, ik   1,
, m.
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
f i x
*
*
T
d 0
T
d 0
 
i  A x*
 5.6 

x*
Now return to analyse the necessity of the K-K-T conditions using
Corollary 7.
 
Suppose that x*  X , f i x *  0, i  1, , m, is a local optimal solution of
problem 2.
Min f  x 
Sujet à f i  x   0 i  1,
x X
,m
(2)
Notation: Denote the set of active constraints
  
  
A x*  i : f i x *  0  i1 ,
, ik   1,
, m.
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
f i x
*
*
T
d 0
T
d 0
 
i  A x*
 5.6 
Assumption to be verified : Suppose that we can show that
T
there is no vector d  R n such that
*
 
f  x 
f i x*
*
T
T
 
d 0
i  A x*
d 0
 
fi1 x
 
fik x*
T
d 0
d 0
 
 f x *
 i1


*
fik x


d  0
T

T
 
Then the system
 
 
 
T
f i x* , , f i x *  d  0,
k
 1

has no solution.
Now apply Corollary 7:

 
 

 
 
T
f x
 
*
T
d 0

II :  f i1 x* , , f ik x *  d  0, f x * d  0 has no solution d  R n


implies that
I:  f i1 x* , , f ik x *   * =f x * ,  * = i1* , , i*k   0 has a solution


 
T

Assumption to be verified : Suppose that we can show that
T
there is no vector d  R n such that
*
 
f  x 
f i x*
*
T
T
 
d 0
i  A x*
d 0
 
fi1 x
 
fik x*
T
d 0
d 0
 
 f x *
 i1


*
fik x


d  0
T

T
 
Then the system
 
 
 
T
f i x* , , f i x *  d  0,
k
 1

has no solution.
Now apply Corollary 7:

 
 

 
 
T
f x
 
*
T
d 0

II :  f i1 x* , , f ik x *  d  0, f x * d  0 has no solution d  R n


implies that
I:  f i1 x* , , f ik x *   * =f x * ,  * = i1* , , i*k   0 has a solution


 
T

Assumption to be verified : Suppose that we can show that
T
there is no vector d  R n such that
*
 
f  x 
f i x*
*
T
T
 
d 0
i  A x*
d 0
 
fi1 x
 
fik x*
T
d 0
d 0
 
 f x *
 i1


*
fik x


d  0
T

T
 
Then the system
 
 
 
T
f i x* , , f i x *  d  0,
k
 1

has no solution.
Now apply Corollary 7:

 
 

 
 
T
f x
 
*
T
d 0

II :  f i1 x* , , f ik x *  d  0, f x * d  0 has no solution d  R n


implies that
I:  f i1 x* , , f ik x *   * =f x * ,  * = i1* , , i*k   0 has a solution


 
T

Assumption to be verified : Suppose that we can show that
T
there is no vector d  R n such that
f x * d  0
 
f  x 
f i x
*
*
T
T
 
d 0
i1
i  A x*
d 0
 
 
fik x*
T
d 0
 
 f x *
 i1


*
fik x


d  0
T

T
 
Then the system
 
 
 
T
f i x , , f i x  d  0,
k
 1

has no solution.
Now apply Corollary 7:
*
*

 
 

 
 
f x
 
T
*
T
d 0

II :  f i1 x , , f ik x  d  0, f x * d  0 has no solution d  R n


implies that
I:  f i1 x* , , f ik x *   * =f x * ,  * = i1* , , i*k   0 has a solution


*
*
T
 

Corollary 7: (Theorem of alternatives) Let A be a matrix
m  n. Exactly one of the two following alternatives holds:
I
The system Ax  b, x  0 has a solution x  R n
II The system AT y  0, bT y  0 has a solution y  R m .
Now apply Corollary 7:

 
 

 
 
 
T

II :  f i1 x , , f ik x  d  0, f x * d  0 has no solution d  R n


implies that
I:  f i1 x * , , f ik x *   * =f x * ,  * = i*1 , , i*k   0 has a solution .


*
*
 
 
 

 

 
This can be written as


i*f i x* =f x* ,  * = i*1 ,
 I: 

iA x* 

Let i*  0 for all i  A x * . Then

T


, ik*   0 has a solution 


i*f i  x*   0
i A x*
i* f i  x*   0 for all i  A  x *  .
Now apply Corollary 7:

 
 

 
 
 
T

II :  f i1 x , , f ik x  d  0, f x * d  0 has no solution d  R n


implies that
I:  f i1 x * , , f ik x *   * =f x * ,  * = i*1 , , i*k   0 has a solution .


*
*
 
 
 

 

 
This can be written as


i*f i x* =f x* ,  * = i*1 ,
 I: 

iA x* 

Let i*  0 for all i  A x * . Then

T


, ik*   0 has a solution 


i*f i  x*   0
i A x*
i* f i  x*   0 for all i  A  x *  .
Now apply Corollary 7:

 
 

 
 
 
T

II :  f i1 x * , , f ik x *  d  0, f x * d  0 has no solution d  R n


implies that
I:  f i1 x * , , f ik x *   * =f x * ,  * = i*1 , , i*k   0 has a solution .


 
 
 

 

 
This can be written as


*
*
*
*
*

I:



f
x
=

f
x
,

=


i
i
 i1 ,

iA x* 

Let i*  0 for all i  A x * . Then

T


, ik*   0 has a solution 


i*f i  x*   0
i A x*
i* f i  x*   0 for all i  A  x *  .
This can be written as


i*f i x* =f x* ,  * = i1* ,
 I: 

iA x* 

Let i*  0 for all i  A x* . Then

 
 
 

 


, ik*   0 has a solution 


i*f i  x*   0
iA x*
i* f i  x*   0 for all i  A  x*  .
Hence we obtain the K-K-T conditions
m
  
 f x   0
f x   0
f x* 
i 1
*
i i
*
*
i
i*  0
i*f i  x*   0
i  1,
,n
i  1,
,n
i  1,
,n
This can be written as


i*f i x* =f x* ,  * = i1* ,
 I: 

iA x* 

Let i*  0 for all i  A x* . Then

 
 
 

 


, ik*   0 has a solution 


i*f i  x*   0
iA x*
i* f i  x*   0 for all i  A  x*  .
Hence we obtain the K-K-T conditions
m
  
 f x   0
f x   0
f x* 
i 1
*
i i
*
*
i
i*  0
i*f i  x*   0
i  1,
,n
i  1,
,n
i  1,
,n
This can be written as


i*f i x* =f x* ,  * = i1* ,
 I: 

iA x* 

Let i*  0 for all i  A x* . Then

 
 
 

 


, ik*   0 has a solution 


i*f i  x*   0
iA x*
i* f i  x*   0 for all i  A  x*  .
Hence we obtain the K-K-T conditions
m
  
 f x   0
f x   0
f x* 
i 1
*
i i
*
*
i
i*  0
i*f i  x*   0
i  1,
,n
i  1,
,n
i  1,
,n
This can be written as


i*f i x* =f x* ,  * = i1* ,
 I: 

iA x* 

Let i*  0 for all i  A x* . Then

 
 
 

 


, ik*   0 has a solution 


i*f i  x*   0
iA x*
i* f i  x*   0 for all i  A  x*  .
Hence we obtain the K-K-T conditions
m
  
 f x   0
f x   0
f x* 
i 1
*
i i
*
*
i
i*  0
i*f i  x*   0
i  1,
,n
i  1,
,n
i  1,
,n
BUT
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
* T
d 0
* T
d 0
f i x
 
i  A x*
is not necessarily verified for all local solution x* for all
problems as illustrated in the following example.
 5.6 
Min f  x1, x2    x1
s.t. f1  x1, x2    x1  1  x2  0
3
f 2  x1 , x2    x1  0
f 3  x1 , x2    x2  0.
The feasible domain of this problem is illustrated in the
figure below under the curve f1  x1, x2  , above the x1 axis,
and on the right of the x2 axis.
x2
x1
Min f  x1, x2    x1
s.t. f1  x1, x2    x1  1  x2  0
3
f 2  x1 , x2    x1  0
f 3  x1 , x2    x2  0.
The feasible domain of this problem is illustrated in the
figure below under the curve f1  x1, x2  , above the x1 axis,
and on the right of the x2 axis.
x2
x1
Min f  x1, x2    x1
s.t. f1  x1, x2    x1  1  x2  0
3
f 2  x1 , x2    x1  0
f 3  x1 , x2    x2  0.
The feasible domain of this problem is illustrated in the
figure below under the curve f1  x1, x2  , above the x1 axis,
and on the right of the x2 axis.
It is easy to verify that
x2
x*  1,0 is a global optimal
T
solution of this problem.
 
Moreover, A x* = 1,3.
x1
Min f  x1, x2    x1
s.t. f1  x1, x2    x1  1  x2  0
3
f 2  x1 , x2    x1  0
f 3  x1 , x2    x2  0.
The feasible domain of this problem is illustrated in the
figure below under the curve f1  x1, x2  , above the x1 axis,
and on the right of the x2 axis.
It is easy to verify that
x2
x*  1,0 is a global optimal
T
solution of this problem.
 
Moreover, A x* = 1,3.

x*
x1
Min f  x1, x2    x1
s.t. f1  x1, x2    x1  1  x2  0
3
f 2  x1 , x2    x1  0
f 3  x1 , x2    x2  0.
 
f x*   1,0 ,
T
T
 
T
f1  x   3  x1  1 ,1 et f1 x*  0,1


2
x2
   0, 1
f 3 x
T
*

*
x
x1
 
f x*   1,0 ,
T
 
2
T
*


f1  x   3  x1  1 ,1 et f1 x  0,1


   0, 1
f 3 x
*
T
On aimerait que le système
Mais le système
  d  d  0
f  x  d  d  0
f  x  d   d  0
f x
*
T
1
*
T
1
x2
2
*
3
T
2
ne possède pas une solution d  1, 0 .

*
x
x1
 
f x*   1,0 ,
T
 
2
T
*


f1  x   3  x1  1 ,1 et f1 x  0,1


   0, 1
f 3 x
*
T
We would like the system
But the system
  d  d  0
f  x  d  d  0
f  x  d   d  0
f x
* T
1
* T
1
x2
2
* T
3
2
to have no solution d  1,0.

*
x
x1
 
f x*   1,0 ,
T
 
2
T
*


f1  x   3  x1  1 ,1 et f1 x  0,1


   0, 1
f 3 x
*
T
We would like the system
But the system
  d  d  0
f  x  d  d  0
f  x  d   d  0
f x
* T
1
* T
1
x2
2
* T
3
2
tohas
have no solution d  1,0.

*
x
x1
But the system
  d  d  0
f  x  d  d  0
f  x  d  d  0
f x
* T
1
* T
1
2
* T
3
2
has a solution d  1,0.
If we look at the la direction d  1,0 at the point x* ,
x2
then it points directly out of the feasible domain.
We will restrict the constraints
of the problems to eliminate

*
x
those where such a situation
d
x1
exists.
Kuhn-Tucker constraints qualification
Notation: R denotes the feasible domain of problem 2
 R  x  R : i,1  i  m, such that f i  x   0 .


Definition. Let the point x   R, f1 ,
, f m satisfy the constraints
qualification at the point x if for any vector dˆ where the system
T
f  x  dˆ  0, i  A  x  , is verified, there exists a differentiable function
i
 : 0,1  R such that   0   x and    0    dˆ ,   0.
Notation: R denotes the feasible domain of problem  5.2 
 R  x  R : i,1  i  m, such that f i  x   0.
Definition. Let the point x   R, f1 ,
, f m satisfy the constraints
qualification at the point x if for any vector dˆ where the system
T
f  x  dˆ  0, i  A  x  , is verified, there exists a differentiable function
i
 : 0,1  R such that   0   x and    0    dˆ ,   0.
In the preceding example, the constraints f1 , f 2 , f 3
are not satisfying the constraints qualification
at the point x*   R.
x2

*
x
d
x1
Notation: R denotes the feasible domain of problem  5.2 
 R  x  R : i,1  i  m, such that f i  x   0.
Definition. Let the point x   R, f1 ,
, f m satisfy the constraints
qualification at the point x if for any vector dˆ where the system
T
f  x  dˆ  0, i  A  x  , is verified, there exists a differentiable function
i
 : 0,1  R such that   0   x and    0    dˆ ,   0.
In the preceding example, the constraints f1 , f 2 , f 3
are not satisfying the constraints qualification
at the point x*   R.
x2

*
x
d
x1
Indeed, there is no differentiable
function  taking its values
in R and having its slope at 0 equal to
a positive multiple of d ,since d points
out directly outside R.
Definition. Let the point x   R, f1 ,
, f m satisfy the constraints
qualification at the point x if for any vector dˆ where the system
T
f  x  dˆ  0, i  A  x  , is verified, there exists a differentiable function
i
 :0,1  R such that   0   x and    0    dˆ ,   0.
Graphic interpretation.
x2
f1  x 

x
T
f1  x  dˆ  0
f 2  x 
f 2  x 
T
    x   dˆ
 0  x
dˆ  0
   0   dˆ
x  dˆ
f2  x   0
f1  x   0
x1

x *   d
x*
Theorem 8: (Necessity of K-K-T optimality conditions) Let x*  X be a
local optimal solution of problem 2, and let X be open. Furthermore,
if x*   R , then f1 , , f m satisfy the constraints qualification at point x* .
Then there exists a vector of multipliers  * = 1* , , m*   0 such that
Min f  x 
s.t. f i  x   0 i  1,
x X
m
    f  x   0
 f x   0
f x 
*
*
i
*
i
i 1
*
i i
*
i  1,
,m
, m.
 
then let   0 for all i  1, , m. Indeed, in this case , if f  x  would take
a value different from 0, then the direction d  f  x  would be a descent
Proof. If x* is a point inside the feasible domain R (i.e., f i x*  0 for all i ),
*
i
*
*
direction for f at x* . Hence, it would exist a scalar  >0 sufficiently small
to have x*   d  B x* R and f x*   d  f x * , a contradiction.


 


 
Theorem 8: (Necessity of K-K-T optimality conditions) Let x*  X be a
local optimal solution of problem 2, and let X be open. Furthermore,
if x*   R , then f1 , , f m satisfy the constraints qualification at point x* .
Then there exists a vector of multipliers  * = 1* , , m*   0 such that
Min f  x 
s.t. f i  x   0 i  1,
x X
m
    f  x   0
 f x   0
f x 
*
*
i
*
i
i 1
*
i i
*
i  1,
,m
, m.
 
then let   0 for all i  1, , m. Indeed, in this case , if f  x  would take
a value different from 0, then the direction d  f  x  would be a descent
Proof. If x* is a point inside the feasible domain R (i.e., f i x*  0 for all i ),
*
i
*
*
direction for f at x* . Hence, it would exist a scalar  >0 sufficiently small
to have x*   d  B x* R and f x*   d  f x * , a contradiction.


 


 
Theorem 8: (Necessity of K-K-T optimality conditions) Let x*  X be a
local optimal solution of problem 2, and let X be open. Furthermore,
if x*   R , then f1 , , f m satisfy the constraints qualification at point x* .
Then there exists a vector of multipliers  * = 1* , , m*   0 such that
Min f  x 
s.t. f i  x   0 i  1,
x X
m
    f  x   0
 f x   0
f x 
*
*
i
*
i
i 1
*
i i
*
i  1,
,m
, m.
 
then let   0 for all i  1, , m. Indeed, in this case , if f  x  would take
a value different from 0, then the direction d  f  x  would be a descent
Proof. If x* is a point inside the feasible domain R (i.e., f i x*  0 for all i ),
*
i
*
*
direction for f at x* . Hence, it would exist a scalar  >0 sufficiently small
to have x*   d  B x* R and f x*   d  f x * , a contradiction.


 


 
Recall
Lemma: Let X  R n , f  C 1 / X , and x  X . If d  R n is
T
a feasible direction at x and f  x  d  0, then there exists
a scalar   0 such that for all 0    
f  x   d   f  x .
(i.e., d is a descent direction at x.)
Proof : Since lim
 0
f x d   f x

 f  x  d  0, then
T
there exists a scalar  >0 such that for all   0,       ,
f  x d   f x
 0.

Then restrict  to be positive in order to have
f  x   d   f  x   0 ou f  x   d   f  x  .
Theorem 8: (Necessity of K-K-T optimality conditions) Let x*  X be a
local optimal solution of problem 2, and let X be open. Furthermore,
if x*   R , then f1 , , f m satisfy the constraints qualification at point x* .
Then there exists a vector of multipliers  * = 1* , , m*   0 such that
Min f  x 
s.t. f i  x   0 i  1,
x X
m
    f  x   0
 f x   0
f x 
*
*
i
*
i
i 1
*
i i
*
i  1,
,m
, m.
 
then let   0 for all i  1, , m. Indeed, in this case , if f  x  would take
a value different from 0, then the direction d  f  x  would be a descent
Proof. If x* is a point inside the feasible domain R (i.e., f i x*  0 for all i ),
*
i
*
*
direction for f at x* . Hence, it would exist a scalar  >0 sufficiently small
to have x*   d  B x* R and f x*   d  f x * , a contradiction.


 


 
Let x*   R . Let us show that
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
f i x
*
T
d 0
 
 5.6 
i  A x*
T
d 0
is verified under the assumptions of the theorem. Indeed, for contradiction
assume that such a vector dˆ would exist. Since f1 , , f m satisfy the constraints
qualification at point x* , then there exists a differentiable fontion  : 0,1  R
such that   0   x* and    0    dˆ ,   0. Thus,
*
lim
   f
f      f x*
x 
*
T
 
   0   f x *
T
dˆ  0,

implying the existence of ˆ  0,1 sufficiently small to have  ˆ  B  x * 
 0
  
 


such that f  ˆ  f x* , a contradiction since  ˆ  R.
The rest of the proof is completed as before when we were assuming that the
hypothesis was verified.
Let x*   R . Let us show that
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
f i x
*
T
d 0
 
 5.6 
i  A x*
T
d 0
is verified under the assumptions of the theorem. Indeed, for contradiction
assume that such a vector dˆ would exist. Since f1 , , f m satisfy the constraints
qualification at point x* , then there exists a differentiable fontion  : 0,1  R
such that   0   x* and    0    dˆ ,   0. Thus,
*
lim
   f
f      f x*
f  x  d   f  x
T
T  f  x  d
 *
 x     0   f  x 
*
T
lim
 0
dˆ  0,

implying the existence of ˆ  0,1 sufficiently small to have  ˆ  B  x * 
 0
  
 


such that f  ˆ  f x* , a contradiction since  ˆ  R.
The rest of the proof is completed as before when we were assuming that the
hypothesis was verified.
Let x*   R . Let us show that
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
f i x
*
T
d 0
 
 5.6 
i  A x*
T
d 0
is verified under the assumptions of the theorem. Indeed, for contradiction
assume that such a vector dˆ would exist. Since f1 , , f m satisfy the constraints
qualification at point x* , then there exists a differentiable fontion  : 0,1  R
f  x  d   f  x
such that   0   x* and    0    dˆ ,   0. Thus,
*
lim
   f
f      f x
*

 0
 x     0   f  x 
*
T
 f  x  d
T
lim
*
T
dˆ  0,

implying the existence of ˆ  0,1 sufficiently small to have  ˆ  B  x * 
 0
  
 


such that f  ˆ  f x* , a contradiction since  ˆ  R.
The rest of the proof is completed as before when we were assuming that the
hypothesis was verified.
Let x*   R . Let us show that
Assumption to be verified : Suppose that we can show that
there is no vector d  R n such that
 
f  x 
f i x
*
T
d 0
 
 5.6 
i  A x*
T
d 0
is verified under the assumptions of the theorem. Indeed, for contradiction
assume that such a vector dˆ would exist. Since f1 , , f m satisfy the constraints
qualification at point x* , then there exists a differentiable fontion  : 0,1  R
f  x  d   f  x
such that   0   x* and    0    dˆ ,   0. Thus,
*
lim
   f
f      f x
*

 0
 x     0   f  x 
*
T
 f  x  d
T
lim
*
T
dˆ  0,

implying the existence of ˆ  0,1 sufficiently small to have  ˆ  B  x * 
 0
  
 


such that f  ˆ  f x* , a contradiction since  ˆ  R.
The rest of the proof is completed as before when we were assuming that the
hypothesis was verified.
K-K-T condition for linear programming
K-K-T conditions  duality and complementary slackness results.
For linear programming,
 K-K-T conditions are sufficient since linear fonctions are
convex.
 K-K-T conditions are necessary since linear fonctions always satisfy
the constraints qualification.
Consider the following linear programming problem:
n
c x
Min
j
j
j 1
n
s.t.
a x
ij
j 1
j
 bi
xj  0
i  1,
j  1,
,m
, n.
K-K-T condition for linear programming
K-K-T conditions  duality and complementary slackness results.
For linear programming,
 K-K-T conditions are sufficient since linear fonctions are
convex.
 K-K-T conditions are necessary since linear fonctions always satisfy
the constraints qualification.
Consider the following linear programming problem:
n
c x
Min
j
j
j 1
n
s.t.
a x
ij
j 1
j
 bi
xj  0
i  1,
j  1,
,m
, n.
K-K-T condition for linear programming
K-K-T conditions  duality and complementary slackness results.
For linear programming,
 K-K-T conditions are sufficient since linear fonctions are
convex.
 K-K-T conditions are necessary since linear fonctions always satisfy
the constraints qualification.
Consider the following linear programming problem:
n
c x
Min
j
j
j 1
n
s.t.
a x
ij
j 1
j
 bi
xj  0
i  1,
j  1,
,m
, n.
K-K-T condition for linear programming
K-K-T conditions  duality and complementary slackness results.
For linear programming,
 K-K-T conditions are sufficient since linear fonctions are
convex.
 K-K-T conditions are necessary since linear fonctions always satisfy
the constraints qualification.
Consider the following linear programming problem:
n
c x
Min
j
j
j 1
n
s.t.
a x
ij
j 1
j
 bi
xj  0
i  1,
j  1,
,m
, n.
Consider the following linear programming problem:
n
c x
Min
j
j
j 1
n
s.t.
a x
ij
j
j 1
 bi
i  1,
xj  0
,m
j  1,
, n.
This problem is equivalent to
n
Min
c x
j
j
j 1
n
s.t. 
a x
ij
j 1
j
 bi  0
i  1,
,m
 xj  0
j  1, , n.
Associate a multiplier i with each of the m first constraints and a
multiplier m  j with each of the n last constraints.
i
m  j
Consider the following linear programming problem:
n
c x
Min
j
j
j 1
n
s.t.
a x
 bi
i  1,
,m
xj  0
j  1,
, n.
ij
j 1
j
This problem is equivalent to
n
Min
c x
j
j
j 1
n
s.t. 
a x
ij
j 1
j
 bi  0
i  1,
,m
 xj  0
j  1, , n.
Associate a multiplier i with each of the m first constraints and a
multiplier m  j with each of the n last constraints.
i
m  j
Consider the following linear programming problem:
n
c x
Min
j
j
j 1
n
s.t.
a x
ij
j
j 1
 bi
i  1,
xj  0
,m
j  1,
, n.
This problem is equivalent to
n
Min
c x
j
j
j 1
n
s.t. 
a x
ij
j 1
j
 bi  0
i  1,
,m
 xj  0
j  1, , n.
Associate a multiplier i with each of the m first constraints and a
multiplier m  j with each of the n last constraints.
i
m  j
n
Min
c x
j
j
j 1
n
s.t.

a x
ij
j
 bi  0
i  1,
i
,m
j 1
 xj  0
j  1, , n.
m  j
Associate a multiplier i with each of the m first constraints and a
multiplier m  j with each of the n last constraints.
The K-K-T conditions are the following:
f  x     f  x   0
m
f  x  m
f i  x  m  n f i  x 

i

i
 c j  i aij m  j  0 j  1, , n
x j
x j
x j
i 1
i  m 1
i 1
m
*


i 1


i  


*
i

aij x j  bi   0

j 1

m  j  x j  0
*
i
n


i  1,
,m

j  1,
,n
 bi  0
i  1,
,m
 xj  0
j  0
j  1, , n
j  1, , m  n.
n

a x
ij
j 1
j
n
Min
c x
j
j
j 1
n
s.t.

a x
ij
j
 bi  0
i  1,
i
,m
j 1
 xj  0
j  1, , n.
m  j
Associate a multiplier i with each of the m first constraints and a
multiplier m  j with each of the n last constraints.
The K-K-T conditions are the following:
f  x     f  x   0
m
f  x  m
f i  x  m  n f i  x 

i

i
 c j  i aij m  j  0 j  1, , n
x j
x j
x j
i 1
i  m 1
i 1
m
*


i 1


i  


*
i

aij x j  bi   0

j 1

m  j  x j  0
*
i
n


i  1,
,m

j  1,
,n
 bi  0
i  1,
,m
 xj  0
j  0
j  1, , n
j  1, , m  n.
n

a x
ij
j 1
j
Consider the dual problem:
n
c x
Primal Min
j
m
Dual
j
Max
j 1
a x
ij
j 1
cj 
 a
i ij
j
 bi i  1,
xj  0
,m
j  1,
m  j  0
,n
j  1,
,n
i  1,
,m

j  1,
,n
 bi  0
i  1,
,m
 xj  0
i  0
j  1, , n
i  1, , m  n

aij x j  bi   0

j 1

m  j  x j  0


n

a x
ij
j 1
j
s.t.
a y  c
ij
i 1
i 1
n

i  


i
m
K-K-T conditions
m
i
i 1
n
s.t.
b y
i
j
j  1,
yi  0
,n
i  1,
, m.
Consider the dual problem:
n
c x
Primal Min
j
m
Dual
j
Max
j 1
a x
ij
j 1
cj 
 a
i ij
j
 bi i  1,
xj  0
,m
j  1,
m  j  0
,n
j  1,
,n
i  1,
,m

j  1,
,n
 bi  0
i  1,
,m
 xj  0
i  0
j  1, , n
i  1, , m  n

aij x j  bi   0

j 1

m  j  x j  0


n

a x
ij
j 1
j
s.t.
a y  c
ij
i 1
i 1
n

i  


i
m
K-K-T conditions
m
i
i 1
n
s.t.
b y
i
j
j  1,
yi  0
,n
i  1,
, m.
Consider the dual problem:
n
Primal Min
c x
j
m
Dual
j
Max
j 1
a x
ij
j 1
cj 
 a
i ij
j
 bi i  1,
xj  0
,m
j  1,
m  j  0
j  1,
,n
i 1
n

i  



aij x j  bi   0

j 1

m  j  x j  0




j 1
aij x j  bi  0
 xj  0
i  0
s.t.
i  1,
j  1,
a y  c
ij
i 1
i
m
 a
m  j  0
 a
m  j  0
i ij
i 1
m
,m
cj 
,n
,m
j  1,
,n
i  1,
, m.
The vector 1 , , m  is a feasible
solution for the dual: for j  1, , n
i ij
i 1
m
i  1,
j
yi  0
,n
cj 
n

i
m
K-K-T conditions
m
i
i 1
n
s.t.
b y
 a
i ij
cj
i 1
Furthermore,
j  1, , n
i  0 i  1,
i  1, , m  n
, m.
Consider the dual problem:
n
c x
Primal Min
j
m
Dual
j
Max
j 1
a x
ij
j 1
cj 
 a
i ij
j
 bi i  1,
xj  0
,m
j  1,
m  j  0
,n
j  1,
,n
i  1,
,m

j  1,
,n
 bi  0
i  1,
,m
 xj  0
i  0
j  1, , n
i  1, , m  n

aij x j  bi   0

j 1

m  j  x j  0


n

a x
ij
j 1
j
s.t.
a y  c
ij
i 1
i 1
n

i  


i
m
K-K-T conditions
m
i
i 1
n
s.t.
b y
i
j
j  1,
yi  0
,n
i  1,
, m.
Consider the dual problem:
n
Primal Min
c x
j
m
Dual
j
b y
Max
i
j 1
i 1
n
a x
s.t.
ij
j 1
m
j
 bi i  1,
xj  0
,m
j  1,
cj 
 a
i ij
m  j  0
j  1,

aij x j  bi   0

j 1

m  j  x j  0



i  1,
,n

j 1
aij x j  bi  0
 xj  0
j  0
i
j
j  1,
yi  0
,m
m
cjxj 
j  1,
,n
i  1,
,m
j  1, , n
j  1, , m  n
,n
i  1,
The vector 1 , , m  is an optimal
solution for the dual: for j  1, , n
m


xj cj 
i aij m  j   0

i 1



n

ij
,n
i 1
n

i  


a y  c
s.t.
i 1
K-K-T conditions
m
i
 a x
i ij
j
m  j x j  0
i 1
n
n
m
 c x    a x
j
j 1
j
i ij
j 1 i 1
j
, m.
Consider the dual problem:
n
c x
Primal Min
j
m
Dual
j
b y
Max
i
j 1
i 1
n
a x
s.t.
ij
j 1
m
j
 bi i  1,
xj  0
,m
j  1,
cj 
 a
i ij
m  j  0
j  1,
,n
,n
i 1
n

i  



aij x j  bi   0

j 1

m  j  x j  0



a x
ij
j 1
j
 bi  0
,m
j  1,
,n
 xj  0
j  0
i
j  1,
j
solution for the dual:
n
n
,m
j  1, , n
j  1, , m  n
m
 c x    a x
j
i ij
j 1
i  1,
i  1,
ij
,n
yi  0 i  1, , m.
The vector 1 , , m  is an optimal
j
n

a y  c
s.t.
i 1
K-K-T conditions
m
i
j 1 i 1
For i  1, , m
 n

i   aij x j  bi   0
 j 1




i bi  i
n
a x
ij
j 1
m
m
j
n
  b    a x
i i
i 1
i ij
i 1
j 1
j
j
Consider the dual problem:
n
c x
Primal Min
j
m
Dual
j
b y
Max
i
j 1
i 1
n
a x
s.t.
ij
j 1
m
j
 bi i  1,
xj  0
,m
j  1,
K-K-T conditions
m
cj 
 a
i ij
m  j  0
j  1,
,n
i 1
n

i  



aij x j  bi   0

j 1

m  j  x j  0



i  1,
,m
a x
ij
j 1
j
 bi  0
 xj  0
j  0
a y  c
s.t.
ij
i
i 1
,n
i  1,
n
n
i  1,
m
 c x    a x
j
j 1
m
i ij
j
j 1 i 1
m
n
  b    a x
i ij
i 1
i 1
j
j 1
Consequently
n
,m
,n
, m.
The vector 1 , , m  is an optimal
solution for the dual:
i i
j  1,
j  1,
j
yi  0
,n
j
n

i
m
 c x   b
j
j 1
j
i
i 1
j  1, , n
and the result follows from the
j  1, , m  n weak duality theorem.
Consider the dual problem:
n
c x
Primal Min
j
m
Dual
j
Max
j 1
a x
ij
j 1
cj 
 a
i ij
j
 bi i  1,
xj  0
,m
j  1,
m  j  0
,n
j  1,
,n
i  1,
,m

j  1,
,n
 bi  0
i  1,
,m
 xj  0
i  0
j  1, , n
i  1, , m  n

aij x j  bi   0

j 1

m  j  x j  0


n

a x
ij
j 1
j
s.t.
a y  c
ij
i 1
i 1
n

i  


i
m
K-K-T conditions
m
i
i 1
n
s.t.
b y
i
j
j  1,
yi  0
,n
i  1,
, m.
Consider the dual problem:
n
c x
Primal Min
j
m
Dual
j
Max
j 1
a x
ij
j 1
cj 
 a
i ij
j
 bi i  1,
xj  0
,m
j  1,
m  j  0
j  1,

aij x j  bi   0

j 1

m  j  x j  0


a x
ij
j 1
j
a y  c
ij
i
For j  1,
m
,n
cj 
 a
i ij
j
j  1,
yi  0
,n
,n
i  1,
,n
m  j  0
i 1
i  1,
,m

j  1,
,n
 bi  0
i  1,
,m
 xj  0
j  0
j  1, , n
j  1, , m  n
n

s.t.
i 1
i 1
n

i  


i
m
K-K-T conditions
m
i
i 1
n
s.t.
b y
m


xj cj 
i aij m  j   0

i 1


m


xj cj 
i aij   x j m  j  0.

i 1


For i  1, , m
 n

i   aij x j  bi   0.
 j 1






, m.