Chapter 8
The Maximum Principle:
Discrete Time
For many purposes it is convenient to assume that time is represented by
a discrete variable, k = 0, 1, 2, ..., T, rather than by a continuous variable
t ∈ [0, T ]. This is particularly true when we wish to solve a large control
theory problem by means of a computer. It is also desirable, even when
solving small problems which have state or adjoint differential equations
whose solutions cannot be expressed in closed form, to formulate them as
discrete problems, and let the computer solve them in a stepwise manner.
We will see that the maximum principle, which is to be derived in this
chapter, is not valid for the discrete-time problem in as wide a sense as
for the continuous-time problem. In fact we will reduce it to a nonlinear
programming problem and state necessary conditions for its solution
by using the well-known Kuhn-Tucker theorem. In order to follow this
procedure, we have to make some simplifying assumptions and hence
will obtain only a restricted form of the discrete maximum principle. In
Section 8.2.5 we state without proof a more general form of the discrete
maximum principle.
8.1
Nonlinear Programming Problems
We begin by stating a general form of a nonlinear programming problem.
Let y be an n-component column vector, a be an r-component column
vector, and b an s-component column vector. Let h : E n → E 1 , g :
E n → E r , and w : E n → E s be given functions. We assume functions g
and w to be column vectors with r and s components, respectively. We
217
8.1. Nonlinear Programming Problems
219
r-component row vector λ such that
Ly = hy + λgy = 0,
(8.6)
Lλ = g(y) − a = 0.
(8.7)
Note that (8.7) states simply that y ∗ is feasible according to (8.2).
The system of n + r equations (8.6) and (8.7) has n + r unknowns.
Since some or all of the equations are nonlinear, the solution method
will, in general, involve nonlinear programming techniques, and may be
difficult. In other cases, e.g., when h is linear and g is quadratic, it may
only involve the solution of linear equations. Once a solution (y ∗ , λ) is
found satisfying the necessary conditions (8.6) and (8.7), the solution
must still be checked to see whether it satisfies sufficient conditions for
a global maximum. Such sufficient conditions will be stated in Section
8.1.3.
Suppose (y ∗ , λ) is in fact a solution to the equations (8.6) and (8.7).
Note that y ∗ depends on a and we can show this dependence by writing
y ∗ = y ∗ (a). Now h∗ = h∗ (a) = h(y ∗ (a)) is the optimum value of the
objective function. The Lagrange multipliers satisfy the relation which
has an important managerial interpretation
h∗a = −λ,
(8.8)
namely, λi is the negative of the imputed value or shadow price of having
one unit more of the resource ai . In Exercise 8.4 you are asked to provide
a proof of (8.8).
Example 8.1 Consider the two-dimensional problem:




max {h(x, y) = −x2 − y 2 }




subject to







2x + y = 10.
Solution. We form the Lagrangian
L(x, y, λ) = (−x2 − y 2 ) + λ(2x + y − 10).
The necessary conditions found by partial differentiation are
Lx = −2x + 2λ = 0,
Ly = −2y + λ = 0,
Lλ = 2x + y − 10 = 0.
220
8. The Maximum Principal: Discrete Time
From the first two equations we get
λ = x = 2y.
Solving this with the last equation yields the quantities
x∗ = 4, y ∗ = 2, λ = 4, h∗ = −20,
which can be seen to give a maximum value to h, since h is concave
and the constraint set is convex. The interpretation of the Lagrange
multiplier λ = 4 can be obtained to verify (8.8) by replacing the constant
10 by 10 + ǫ and expanding the objective function in a Taylor series; see
Exercise 8.5.
8.1.2
Inequality Constraints
Now suppose we want to solve the inequality-constrained problem defined
by (8.1) and (8.3), without (8.2). The latter constraints will be appended
to the problem in Section 8.1.3.
As before we define the Lagrangian
L(y, µ) = h(y) + µ[w(y) − b].
(8.9)
The Kuhn-Tucker necessary conditions for this problem cannot be as
easily derived as for the equality-constrained problem in the preceding
section. We will write them first, and then give interpretations to make
them plausible. The necessary condition for y ∗ to be a solution of (8.1)
and (8.3) is that there exists an s-dimensional row vector µ such that
Ly = hy + µwy = 0,
(8.10)
w ≥ b,
(8.11)
µ ≥ 0, µ(w − b) = 0.
(8.12)
Note that (8.10) is analogous to (8.6). Also (8.11) repeats the inequality constraint (8.3) in the same way that (8.7) repeated the equality constraint (8.2). However, the conditions in (8.12) are new and are
particular to the inequality-constrained problem. We will see that they
include some of the boundary points of the feasible set of points as well as
unconstrained maximum solution points, as candidates for the solution
to the maximum problem. This is best brought out by examples.
225
8.1. Nonlinear Programming Problems
The necessary conditions are
2
Lx = − λx−1/3 = 0,
3
Ly = 1 − λ + µ = 0,
2/3
λ ≥ 0, λ(−x
(8.30)
(8.31)
− y + 1) = 0,
µ ≥ 0, µy = 0,
(8.32)
(8.33)
together with (8.27) and (8.28). From (8.30) we get λ = 0, since x−1/3 is
never 0 in the range −1 ≤ x ≤ 1. But substitution of λ = 0 into (8.31)
gives µ = −1 < 0, which fails to satisfy (8.33).
You may think perhaps that the reason for the failure of the method
is due to the non-differentiability of constraint (8.27) when x = 0. That
is part of the reason, but the next example shows that something deeper
is involved.
Example 8.6 Consider the problem:
max {h(x, y) = y}
(8.34)
(1 − y)3 − x2 ≥ 0,
(8.35)
y ≥ 0.
(8.36)
subject to
The constraints are now differentiable and the set of feasible solutions is
exactly the same as for Example 8.5, and is shown in Figure 8.2. Hence,
the optimum solution is (x∗ , y ∗ ) = (0, 1) and h∗ = 1. But once again the
Kuhn-Tucker method fails, as we will see. The Lagrangian is
L = y + λ[(1 − y)3 − x2 ] + µy,
(8.37)
so that the necessary conditions are
Lx = −2xλ = 0,
(8.38)
2
Ly = 1 − 3λ(1 − y) + µ = 0,
3
2
(8.39)
λ ≥ 0, λ[(1 − y) − x ] = 0,
(8.40)
µ ≥ 0, µy = 0,
(8.41)
together with (8.35) and (8.36). From (8.41) we get, either y = 0 or
µ = 0. Since y = 0 minimizes the objective function, we choose µ = 0.
226
8. The Maximum Principal: Discrete Time
From (8.38) we get either λ = 0 or x = 0. Since substitution of λ = µ = 0
into (8.39) shows that it is not satisfied, we choose x = 0, λ 6= 0. But
then (8.40) gives (1 − y)3 = 0 or y = 1. However, µ = 0 and y = 1 means
once more that (8.39) is not satisfied.
The reason for failure of the method in Example 8.6 is that the constraints do not satisfy what is called the constraint qualification, which
will be discussed in the next section.
Example 8.6 shows the necessity of imposing some kind of condition
to rule out cusp points such as the (0, 1) point in Figure 8.2, since the
method shown will fail to find the solution when the answer occurs at
such points. A brief mathematical description of cusp points is given
shortly hereafter in this section. A complete study of the problem is
beyond the scope of this book, but we state here a version of the constraint qualification sufficient for our purposes. For further information,
see Mangasarian (1969).
In order to motivate the definition we illustrate two different situations in Figure 8.3. In Figure 8.3(a) we show two boundary curves
w1 (y) = b1 and w2 (y) = b2 intersecting the boundary point ȳ. The two
tangents to these curves are shown, and v is a vector lying between the
two tangents. Starting at ȳ, there is a differentiable curve c(τ ), 0 ≤ τ ≤ 1,
drawn so that it lies entirely within the feasible set Y , such that its initial slope is equal to v. Whenever such a curve can be drawn from every
boundary point ȳ in Y and every v contained between the tangent lines,
Figure 8.3: Kuhn-Tucker Constraint Qualification
8.1. Nonlinear Programming Problems
227
we say that the constraints defining Y satisfy the Kuhn-Tucker constraint
qualification at ȳ.
Figure 8.3(b) illustrates a case of a cusp at which the constraint
qualification does not hold. Here the two tangents to the graphs w1 (y) =
b1 and w2 (y) = b2 coincide, so that v1 and v2 are vectors lying between
these two tangents. Notice that for vector v1 it is possible to find the
differentiable curve c(τ ) satisfying the above conditions, but for vector
v2 no such curve exists. Hence, the constraint qualification does not hold
for the example in Figure 8.3(b).
8.1.3
Theorems from Nonlinear Programming
We now state without proof two nonlinear programming theorems which
we will use in deriving our version of the discrete maximum principle.
For proofs, see Mangasarian (1969).
We first state the constraint qualification symbolically. For the problem defined by (8.1), (8.2), and (8.3), let Y be the set of all feasible
vectors satisfying (8.2) and (8.3), i.e.,
Y = {y|g(y) = a, w(y) ≥ b} .
Let ȳ be any point of Y and let z(ȳ) = d be the vector of tight constraints at point ȳ, i.e., z includes all the g constraints in (8.2) and those
constraints in (8.3) which are satisfied as equalities.
Define the set
∂z(ȳ)
V = v
v≤0 .
(8.42)
∂y
Then, we shall say that the constraint set Y satisfies the Kuhn-Tucker
constraint qualification at ȳ ∈ Y if z is differentiable at ȳ and if, for every
v ∈ V , there exists a differentiable curve c(τ ) defined for 0 ≤ τ ≤ 1 such
that
(i)
c(0) = ȳ,
(ii)
c(τ ) ∈ Y for all t satisfying 0 ≤ τ ≤ 1,
dc(τ ) = kv for some constant k > 0.
dτ τ =0
(iii)
The interpretation of this condition was given for the two dimensional
case in the preceding section.
We now state two theorems for giving sufficient and necessary conditions, respectively, for the problem given by (8.1)-(8.3). The Lagrangian
228
8. The Maximum Principal: Discrete Time
function for this problem is
L(y, λ, µ) = h + λ(g(y) − a) + µ(w(y) − b).
(8.43)
The Kuhn-Tucker conditions at ȳ ∈ Y for this problem are
Ly (ȳ, λ, µ) = hy (ȳ) + λgy (ȳ) + µwy (ȳ) = 0,
(8.44)
g(ȳ) = a,
(8.45)
w(ȳ) ≥ b,
(8.46)
µ ≥ 0, µw(ȳ) = 0,
(8.47)
where λ and µ are row vectors of multipliers to be determined.
Theorem 8.1 (Sufficient Conditions). If h, g, and w are differentiable,
h is concave, g is affine, w is concave, and (ȳ, λ, µ) solve the conditions
(8.44)-(8.47), then ȳ is a solution to the maximization problem (8.1)(8.3).
Theorem 8.2 (Necessary Conditions). If h, g, and w are differentiable,
ȳ solves the maximization problem, and the constraint qualification holds
at ȳ, then there exist multipliers λ and µ such that (ȳ, λ, µ) satisfy conditions (8.44)-(8.47).
8.2
A Discrete Maximum Principle
We shall now use the nonlinear programming results of the previous
section to derive a special form of the discrete maximum principle. Two
references in this connection are Luenberger (1972) and Mangasarian
and Fromovitz (1967). A more general discrete maximum principle will
be stated in Section 8.3.
8.2.1
A Discrete-Time Optimal Control Problem
In order to state a discrete-time optimal control problem over the periods
0, 1, 2, ..., T , we define the following:
Θ =
the set {0, 1, 2, ..., T − 1},
k
=
an n-component column state vector; k = 0, 1, ..., T,
k
=
an m-component column control vector; k = 0, 1, 2, ..., T − 1,
k
=
an s-component column vector of constants; k = 0, 1, ..., T − 1.
x
u
b
236
8. The Maximum Principal: Discrete Time
in the system dynamics, i.e., when the state of the system in a period
depends not only on the state and the control in the previous period, but
also on the values of these variables in prior periods, it is easy to adapt
the discrete maximum principle to deal with such systems; see Burdet
and Sethi (1976). Exercise 8.18 presents an advertising model containing
lags in its sales-advertising dynamics.
Some concluding remarks on the applications of the discrete-time optimal control problems are appropriate. Examples of real-life problems
that can be modeled as such problems include the following: payments
of principal and interest on loans; harvesting of crops; production planning for monthly demands; etc. Such problems would require efficient
computational procedures for their solution. Some references dealing
with computational methods for discrete optimal control problems are
Murray and Yakowitz (1984), Dunn and Bertsekas (1989), Pantoja and
Mayne (1991), Wright (1993), and Dohrmann and Robinett (1999). Another reason which makes the discrete optimal control theory important
arises from the fact that digital computers are being used increasingly
in the control of dynamic systems.
Finally, Pepyne and Cassandras (1999) have recently explored an optimal control approach to treat discrete event dynamic systems (DEDS).
They also apply the approach to a transportation problem, modeled as
a polling system.
EXERCISES FOR CHAPTER 8
8.1 Determine the critical points of the following functions:
(a) h(y, z) = −5y 2 − z 2 + 10y + 6z + 27,
(b) h(y, z) = 5y 2 − yz + z 2 − 10y − 18z + 17.
8.2
Let h be twice differentiable with its Hessian matrix defined to
be H = hyy . Let ȳ be a critical point, i.e., a solution of hy = 0.
Let Hj be the jth principal minor, i.e., the j × j submatrix found
in the first j rows and the first j columns of H. Let |Hj | be the
determinant of Hj . Then, ȳ is a local maximum of h if
H1 < 0, |H2 | > 0, |H3 | < 0, ..., (−1)n |Hn | = (−1)n |H| > 0
evaluated at ȳ; and ȳ is a local minimum of h if
H1 > 0, |H2 | > 0, |H3 | > 0, ..., |Hn | = |H| > 0
evaluated at ȳ. Apply these conditions to Exercise 8.1 to identify
local minima and maxima of the functions in (a) and (b).
239
Exercises for Chapter 8
8.15 Current-Value Formulation. Obtain the current-value formulation
of the discrete maximum principle. Assume that r is the discount
rate, i.e., 1/(1 + r) is the discount factor.
8.16 Convert the Bolza form problem (8.48)-(8.50) to the equivalent
linear Mayer form; see Section 2.1.4 for a similar conversion in the
continuous-time case.
8.17 Convert the problem defined by (8.48) and (8.74) to its Lagrange
form. Then, obtain the assumptions on the salvage value function S(xT , T ) so that the results of Section 8.3 apply. Under these
assumptions, state the maximum principle for the Bolza form problem defined by (8.48) and (8.74).
8.18∗ An Advertising Model (Burdet and Sethi, 1976). Let xk denote
the sale and uk , k = 1, 2, ..., T −1, denote the amount of advertising
in period k. Formulate the sales-advertising dynamics as
△xk = −δxk + r
k
X
fkl (xl , ul ), x0 given,
l=0
where δ and r are decay and response constants, respectively, and
fkl (xl , ul ) is a nonnegative function that decreases with xl and increases with ul . In the special case when
fkl (xl , ul ) = γ lk ul , γ lk > 0,
obtain optimal advertising amounts to maximize the total discounted profit given by
TX
−1
(πxk − uk )(1 + ρ)−k ,
k=1
where, as in Section 7.2.1, π denotes per unit sales revenue and ρ
denotes the discount rate, and where 0 ≤ uk ≤ Qk represent the
restriction on the advertising amount uk . For a continuous-time
version of problems with lags, see Hartl and Sethi (1984b).
8.19 Constraint Qualification. Show that the feasible set in two dimensions, determined by the constraints (1 − x)3 − y ≥ 0, x ≥ 0, and
y ≥ 0, does not satisfy the Kuhn-Tucker constraint qualification at
the boundary point (1,0).