Lecture – 26
Classical Numerical Methods to Solve
Optimal Control Problems
Dr. Radhakant Padhi
Asst. Professor
Dept. of Aerospace Engineering
Indian Institute of Science - Bangalore
Necessary Conditions of
Optimality in Optimal Control
State Equation
∂H
X=
= f ( t , X ,U )
∂λ
z
Costate Equation
⎛ ∂H
λ = −⎜
⎝ ∂X
z
Optimal Control
Equation
⎛ ∂H
⎞
=
0
⎜
⎟ ⇒ U =ψ ( X , λ )
⎝ ∂U
⎠
z
z
Boundary Condition
λf =
∂ϕ
∂X f
⎞
⎟ = g ( t , X ,U )
⎠
X ( t0 ) = X0 :Fixed
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
2
Necessary Conditions of
Optimality: Salient Features
z
State and Costate equations are dynamic equations
z
State equation develops forward whereas Costate
equation develops backwards
z
Optimal control equation is a stationary equation
z
The formulation leads to Two-Point-Boundary-Value
Problems (TPBVPs), which demand
computationally-intensive iterative numerical
procedures to obtain the optimal control solution
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
3
Classical Methods to Solve
TPBVPs
z
Gradient Method
z
Shooting Method
z
Quasi-Linearization Method
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
4
Gradient Method
Dr. Radhakant Padhi
Asst. Professor
Dept. of Aerospace Engineering
Indian Institute of Science - Bangalore
Gradient Method
z
Assumptions:
• State equation satisfied
• Costate equation satisfied
• Boundary conditions satisfied
z
Strategy:
• Satisfy the optimal control equation
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
6
Gradient Method
⎡ ∂φ
⎤
− λf ⎥
⎢
⎢⎣ ∂X f
⎥⎦
δ J = (δ X f )
T
tf
+ ∫ (δ X )
T
t0
tf
+ ∫ (δ U )
T
t0
tf
+ ∫ (δλ )
T
t0
⎡ ∂H
⎤
⎢⎣ ∂X + λ ⎥⎦ dt
⎡ ∂H ⎤
⎢⎣ ∂U ⎥⎦ dt
⎡ ∂H
⎤
⎢⎣ ∂λ − X ⎥⎦ dt
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
7
Gradient Method
z
After satisfying the state & costate equations
and boundary conditions, we have
tf
δ J = ∫ (δ U )
T
t0
z
⎡ ∂H ⎤
δ U ( t ) = −τ ⎢ ⎥ , τ > 0
⎣ ∂U ⎦
Select
tf
z
⎡ ∂H ⎤
⎢⎣ ∂U ⎥⎦ dt
This leads to
⎡ ∂H ⎤ ⎡ ∂H ⎤
δ J = −τ ∫ ⎢ ⎥ ⎢ ⎥ dt
t0 ⎣ ∂U ⎦ ⎣ ∂U ⎦
T
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
8
Gradient Method
z
z
We select
This lead to
⎡ ∂H ⎤
δ U i ( t ) = ⎡⎣U i +1 ( t ) − U i ( t ) ⎤⎦ = −τ ⎢ ⎥
⎣ ∂U ⎦
⎡ ∂H ⎤
U i +1 ( t ) = U i ( t ) − τ ⎢
⎣ ∂U ⎥⎦
tf
z
z
i
i
⎡ ∂H ⎤ ⎡ ∂H ⎤
δ J = −τ ∫ ⎢ ⎥ ⎢ ⎥ dt ≤ 0
t0 ⎣ ∂U ⎦ ⎣ ∂U ⎦
Note:
Eventually,
δJ =0
T
⇒
∂H
=0
∂U
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
9
Gradient Method: Procedure
z
Assume a control history (not a trivial task)
z
Integrate the state equation forward
z
Integrate the costate equation backward
z
Update the control solution
•
z
This can either be done at each step while integrating
the costate equation backward or after the integration
of the costate equation is complete
Repeat the procedure until convergence
tf
⎡ ∂H ⎤ ⎡ ∂H ⎤
∫t ⎢⎣ ∂U ⎥⎦ ⎢⎣ ∂U ⎥⎦ dt ≤ γ
0
T
(a pre-selected constant)
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
10
Gradient Method: Selection of
z
z
τ
Select τ so that it leads to a certain
percentage reduction of J
Let the percentage be α
t
T
Then τ ∫ ⎡⎢ ∂H ⎤⎥ ⎡⎢ ∂H ⎤⎥ dt = α J
f
z
t0
z
⎣ ∂U ⎦ ⎣ ∂U ⎦
This leads to
100
α
J
100
T
f
⎡ ∂H ⎤ ⎡ ∂H ⎤
∫t ⎢⎣ ∂U ⎥⎦ ⎢⎣ ∂U ⎥⎦ dt
0
τ=t
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
11
A Real-Life Challenging Problem
Objective:
Air-to-air missiles are usually launched from an aircraft in the forward direction.
However, the missile should turn around and intercept a target “behind the aircraft”.
Aircraft
Missile
Target
To execute this task, the missile should turn around by -180o and lock onto its
target (after that it can be guided by its own homing guidance logic).
Note: Every other case can be considered as a subset of this extreme scenario!
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
12
A Real-Life Challenging Problem
MATHEMATICAL PERSPECTIVE:
• Minimum time optimization problem
• Fixed initial conditions and free final time problem
SYSTEM DYNAMICS:
Equations of motion for a missile in vertical plane. The non-dimensional equations
of motion (point mass) in a vertical plane are:
M ' = − S w M 2CD − sin(γ ) + Tw cos(α )
1
[ S w M 2CL + Tw sin(α ) − cos(γ )]
M
where prime denotes differentiation with respect to the non-dimensional time τ
γ '=
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
13
A Real-Life Challenging Problem
The non-dimensional parameters are defined as follows:
g
T
ρ a2S
V
τ = ; Tw =
; Sw =
; M=
at
mg
a
2mg
where M = flight Mach number
γ = flight path angle
T = thrust
m = mass of the missile
S = reference aerodynamic area
V = speed of the missile
CL = lift coefficient
CD = drag coefficient
g = the acceleration due to gravity
a = the local speed of sound
ρ = the atmospheric density
t = flight time after launch
NOTE: CL , CD are usually functions of α & M (tabulated data)
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
14
A Real-Life Challenging Problem
COST FUNCTION:
Mathematically the problem is possed as follows to find the
control minimizing cost function:
tf
J =
∫ dt
0
Constraints γ (0) = 0o ,
M (0) = initial Mach number
γ (t f ) = −180o , M (t f ) = 0.8
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
15
A Real-Life Challenging Problem
Choosing γ as the independent variable the equations are reformulated as follows:
2
dM ( − S w M CD − sin(γ ) + Tw cos(α ) ) M
=
dγ
S w M 2CL − cos(γ ) + Tw sin(α )
dt
aM
=
d γ g ( S w M 2CL − cos(γ ) + Tw sin(α ) )
and the transformed cost function is
tf
−π
0
0
J = ∫ dt =
∫
dt
dγ =
dγ
−π
∫
0
aM
dγ
2
g ( S w M CL − cos(γ ) + Tw sin(α ) )
⎛ A difficult minimum-time problem has been converted to a relatively ⎞
⎜
⎟
easier
fix
=
ed
final-time
problem
(with
hard
constraint:
M
(
γ
)
0.8)!
f
⎝
⎠
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
16
Task
Solve the problem using gradient method. Assume M (0) = 0.5 and
engagement height as 5 km. Next, generate the trajectories and
tabulate the values of M f for various q values.
Use the following system parameters
(typical for an air-to-air missile):
m = 240 kg
S = 0.0707 m2
T = 24,000 N
CD = 0.5
CL = 3.12
Use standard atmosphere chart for the atmospheric data.
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
17
Shooting Method
Dr. Radhakant Padhi
Asst. Professor
Dept. of Aerospace Engineering
Indian Institute of Science - Bangalore
Necessary Conditions of
Optimality (TPBVP): A Summary
State Equation
∂H
X=
= f ( t , X ,U )
∂λ
z
Costate Equation
⎛ ∂H
λ = −⎜
⎝ ∂X
z
Optimal Control
Equation
∂H
=0
∂U
z
z
Boundary Condition
λf =
∂ϕ
∂X f
⎞
⎟ = g ( t , X ,U , λ )
⎠
X ( t0 ) = X0 :Fixed
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
19
Shooting Method
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
20
Shooting Method
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
21
Shooting Method
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
22
Shooting Method
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
23
Shooting Method
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
24
Quasi-Linearization Method
Dr. Radhakant Padhi
Asst. Professor
Dept. of Aerospace Engineering
Indian Institute of Science - Bangalore
Quasi-Linearization Method
Problem:
Differential Equation: Z = F ( Z , t ) ,
Boundary condition:
Z
⎡⎣ X
T
λ ⎤⎦
T
T
C ( ti ) , Z ( ti ) = CiT Z i = bi
ti ∈ t , i ∈ {1,… , n}
Assumption:
This vector differential equation has a unique solution over t ∈ ⎡⎣t0 , t f ⎤⎦
Trick:
The nonlinear multi-point boundary value problem is transformed into
a sequence of linear non-stationary boundary value problems, the solution
of which is made to approximate the solution of the true problem.
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
26
Quasi-Linearization Method
(1) Guess an approximate solution Z N ( t )
( N = 1)
(it need not satisfy the B.C.)
For updating this solution, proceed with the following steps:
(2) Linearize the system dynamics about Z N ( t )
⎡ ∂F ⎤
ΔZ N = ⎢ ⎥ ΔZ N ,
⎣ ∂Z ⎦ Z N
where, ΔZ N ( t )
A( t )
Z N +1 ( t ) − Z N ( t )
To be found
ΔZ N = A ( t ) ΔZ N
(3) Enforce the boundary with respect to the updated solution Z N +1 ( t )
C ( ti ) , Z N +1 ( ti ) = C ( ti ) , Z N ( ti ) + ΔZ N ( ti ) = bi
C ( ti ) , ΔZ N ( ti ) = − C ( ti ) , Z N ( ti ) + bi
Philosophy: Solve this linear system
and update the solution!
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
27
Quasi-Linearization Method:
Solution by STM Approach
(1) From the linearized system dynamics, we can write
Z N +1 = Z N + A ( t ) ( Z N +1 − Z N )
= A ( t ) Z N +1 + ⎡⎣ F ( Z N , t ) − A ( t ) Z N ⎤⎦
Homogeneous
Forcing function
(2) The solution Z N +1 ( t ) to the above equation is given by
Z N +1 ( t ) =
Φ N +1 ( t , t0 )
Z N +1 ( t0 ) + p N +1 ( t )
State transition matrix (STM)
Particular solution
(3) The solution for STM Φ N +1 ( t , t0 ) can be obtained from the fact that
it satisfies the following differential equation and boundary conditions
∂
⎡⎣Φ N +1 ( t , t0 ) ⎤⎦ = A ( t ) Φ N +1 ( t , t0 )
∂t
Φ N +1 ( t0 , t0 ) = I
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
28
Quasi-Linearization Method:
Solution by STM Approach
(4) The particular solution p N +1 ( t ) can be obtained by observing that
it satisfies the the following differential equation and boundary condition
Substituting the complete solution Z N +1 ( t ) in the original equation
∂
⎡⎣Φ N +1 ( t , t0 ) Z N +1 ( t0 ) ⎤⎦ + p N +1 ( t ) = A ( t ) ⎡⎣Φ N +1 ( t , t0 ) Z N +1 ( t0 ) + p N +1 ( t ) ⎤⎦
∂t
+ ⎡⎣ F ( Z N , t ) − A ( t ) Z N ⎤⎦
p N +1 ( t ) = A ( t ) p N +1 ( t ) + ⎡⎣ F ( Z N , t ) − A ( t ) Z N ⎤⎦
(5) The boundary condition p N +1 ( t0 ) can be obtained by observing that
Z N +1 ( t0 ) = Φ N +1 ( t0 , t0 ) Z N +1 ( t0 ) + p N +1 ( t0 )
I
p N +1 ( t0 ) = 0
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
29
Quasi-Linearization Method:
Solution by STM Approach
(6) The boundary condition Z N +1 ( t0 ) can be obtained as follows
C ( ti ) , Z N +1 ( ti ) = bi
C ( ti ) , Φ N +1 ( ti , t0 ) Z N +1 ( t0 ) + p N +1 ( ti ) = bi
C ( ti ) , Φ N +1 ( ti , t0 ) Z N +1 ( t0 ) = − C ( ti ) , p N +1 ( ti ) + bi
Solve the above system to obtain Z N +1 ( t0 )
Once Z N +1 ( t0 ) is determined, the solution Z N +1 ( t ) is available from the
STM solution:
Z N +1 ( t ) = Φ N +1 ( t , t0 ) Z N +1 ( t0 ) + p N +1 ( t )
STM
Particular solution
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
30
Quasi-Linearization Method:
Convergence Property
Under the assumption that the problem admits a unique solution for t ∈ ⎡⎣t0 , t f ⎤⎦
it can be shown that the sequence of vectors {Z N +1 ( t )} converge to the true solution.
Morover, the process can be shown to have "quadratic convergence" in general
i.e., it can be shown that Z N +1 ( t ) − Z N ( t ) ≤ k Z N ( t ) − Z N −1 ( t ) , where k ≠ f ( N ) .
Further more, for a large class of systems, it can be shown to have "monotone
convergence" as well, i.e. there won't be any over-shooting in the convergence process.
Reference : R. Kabala, "On Nonlinear Differential Equations, The Maximum Operation
and Monotone Convergence", J. of Mathematics and Mechanics, Vol. 8,
1959, pp. 519-574.
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
31
A Demonstrative Example
1
1
Problem: Minimize J = ∫ ( x 2 + u 2 ) dt for the system x = − x 2 + u , x ( 0 ) = 10.
20
Solution:
1
Hamiltonian: H = ( x 2 + u 2 ) + λ ( − x 2 + u )
2
1) State Equation:
x = − x2 + u
2) Optimal Control Equation: u + λ = 0 ⇒ u = −λ
3) Costate Equation:
λ = − ( ∂H / ∂x ) = − x + 2λ x
4) Bounary Conditions:
x ( 0 ) = 10, λ (1) = ( ∂Φ / ∂ x ) = 0
Substituting the expression for u in the state equation, we can write
x = − x2 − λ,
x ( 0 ) = 10
λ = − x + 2λ x ,
λ (1) = 0
Task : Solve this problem using shooting and quasi-linearization methods.
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
32
References on Numerical Methods in
Optimal Control Design
z
D. E. Kirk, Optimal Control Theory: An Introduction,
Prentice Hall, 1970.
z
S. M. Roberts and J.S. Shipman, Two Point
Boundary Value Problems: Shooting Methods,
American Elsevier Publishing Company Inc., 1972.
z
A. E. Bryson and Y-C Ho, Applied Optimal Control,
Taylor and Francis, 1975.
z
A. P. Sage and C. C. White III, Optimum Systems
Control (2nd Ed.), Prentice Hall, 1977.
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
33
Survey of Classical Methods
z
M. Athans (1966), “The Status of Optimal
Control Theory and Applications for
Deterministic Systems”, IEEE Trans. on
Automatic Control, Vol. AC-11, July 1966,
pp.580-596.
z
H. J. Pesch (1994), “A Practical Guide to the
Solution of Real-Life Optimal Control
Problems”, Control and Cybernetics, Vol.23,
No.1/2, 1994, pp.7-60.
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
34
ADVANCED CONTROL SYSTEM DESIGN
Dr. Radhakant Padhi, AE Dept., IISc-Bangalore
35