Optimal control
T. F. Edgar
Spring 2012
Optimal Control
• Static optimization (finite dimensions)
• Calculus of variations (infinite dimensions)
• Maximum principle (Pontryagin) / minimum principle
Based on state space models
Min 𝑉 𝒙, 𝒖
S.t. 𝒙 = 𝒇 𝒙, 𝒖, 𝑡
𝒙 𝑡0 is given
𝑡𝑓
𝑉 𝒙, 𝑢 = Φ 𝒙 𝑡𝑓
+
𝐿 𝒙, 𝒖, 𝑡 𝑑𝑡
𝑡0
General nonlinear control problem
2
Special Case of 𝑽
• Minimum fuel:
𝑡𝑓
0
• Minimum time:
𝑡𝑓
1𝑑𝑡
0
• Max range :
𝒖 𝑑𝑡
𝑥 𝑡𝑓
• Quadratic loss:
𝑡𝑓
0
𝒙𝑇 𝑸𝒙 + 𝒖𝑇 𝑹𝒖 𝑑𝑡
Analytical solution if state equation is linear, i.e.,
𝒙 = 𝑨𝒙 + 𝑩𝒖
3
“Linear Quadratic” problem - LQP
𝑡𝑓 2
𝑥
𝑑𝑡
0
• Note 𝐼𝑆𝐸 =
is not solvable in a
realistic sense (𝑢 is unbounded), thus need
control weighting in 𝑉
• E.g., 𝑉 =
𝑡𝑓
0
𝑥 2 + 𝑟𝑢2 𝑑𝑡
• 𝑟 is a tuning parameter (affects overshoot)
4
• 𝑉 = 𝑃𝑟𝑜𝑓𝑖𝑡 ?
Ex. Maximize conversion in exit of tubular reactor
max 𝑥3 𝑡𝑓
𝑥3 : Concentration
𝑡: Residence time parameter
In other cases, when 𝑥 and 𝑢 are deviation variables,
𝑥2 +
5
• Initial conditions
(a) 𝑥 0 ≠ 0, 𝑥 𝑡𝑓 → 𝑥𝑑 = 0 or 𝑉 =
𝑡𝑓
0
𝑥 − 𝑥𝑑 2 𝑑𝑡
Set point change, 𝑥𝑑 is the desired 𝑥
(b) 𝑥 0 ≠ 0, impulse disturbance,
𝑥𝑑 = 0
(c) 𝑥 0 = 0, model includes disturbance term
𝑥𝑑 = 0
6
Other considerations:
“open loop” vs. “closed loop”
• “open loop”: optimal control is an explicit function of time,
depends on 𝑥 0 -- “programmed control”
• “closed loop”: feedback control, 𝑢 𝑡 depends on 𝑥 𝑡 , but
not on 𝑥 0 . e.g., 𝑢 𝑡 = −𝐾 𝑡 𝑥 𝑡
Feedback control is advantageous in presence of noise,
model errors.
Optimal feedback control arises from a specific optimal
control problems, the LQP.
7
Derivation of Minimum Principle
𝑡𝑓
min 𝑉 𝒙, 𝒖 = Φ 𝒙 𝑡𝑓
+
𝐿 𝒙 𝑡 , 𝒖 𝑡 , 𝑡 𝑑𝑡
0
𝒙 = 𝒇 𝒙, 𝒖, 𝑡
𝒙𝑛×1 , 𝒖𝑟×1
Φ, 𝐿, 𝑓 have continuous 1st partial w.r.t. 𝒙, 𝒖, 𝑡
Form Lagrangian
𝑡𝑓
𝑉 𝑢 =Φ+
𝐿 + 𝝀𝑇 𝒇 − 𝒙 𝑑𝑡
𝑡0
Multipliers: adjoint variables, costates
8
• Define 𝐻 = 𝐿 + 𝝀𝑇 𝒇 (Hamiltonian)
𝑡𝑓
𝑉 𝑢 =Φ+
𝐻 − 𝝀𝑇 𝒙 𝑑𝑡 = Φ 𝑥 − 𝝀𝑇 𝒙
𝑡0
( 𝝀𝑇 𝒙𝑑𝑡 = 𝝀𝑇 𝒙
𝑡𝑓
− 𝝀𝑇 𝒙
𝑡𝟎
+
𝑡𝑓
𝑡𝑓
+
𝐻 + 𝝀𝑇 𝒙 𝑑𝑡
𝑡0
𝝀𝑇 𝒙 𝑑𝑡)
• Since 𝑉 is Lagrangian, we treat as unconstrained problem with
variables: 𝒙 𝑡 , 𝝀 𝑡 , 𝒖 𝑡
• Use variations: 𝛿𝒙 𝑡 , 𝛿𝒖 𝑡 , 𝛿 𝑉 (for 𝛿𝝀 𝑡 => original constraint,
the state equation.)
𝛿𝑉 = 0
=
𝑑Φ
− 𝜆𝑇
𝑑𝑥
+ 𝜆𝑇 𝛿𝑥
𝑡𝑓
𝑡𝑓
𝑡0
+
𝐻𝑢 𝛿𝑢 + 𝐻𝑥 𝛿𝑥 + 𝜆𝑇 𝛿𝑥 𝑑𝑡
𝑡0
9
• Since 𝛿𝑥 𝑡 , 𝛿𝑢 𝑡 are arbitrary (≠ 0), then
𝜕𝐻
𝜕𝑥
+ 𝜆 = 0 𝜆 =
𝜕𝐻
𝜕𝑢
= 0, “optimality equation” for weak minimum
𝑡=
𝜕Φ
𝑡𝑓 ,
𝜕𝑥
𝜕𝐻
−
𝜕𝑥
(n equations. “adjoint equation”)
− 𝜆 = 0  𝜆 𝑡𝑓 =
𝜕Φ
−
𝜕𝑥 𝑡𝑓
(n boundary conditions)
If 𝑥 𝑡0 is specified, then 𝛿𝑥 𝑡0 = 0
Two point boundary value problem (“TPBVP”)
10
• Example:
𝑑𝑥1
𝑑𝑡
= 𝑢 − 𝑥1 (1st order transfer function)
min 𝑉 =
1 𝑡𝑓
2 0
𝑥12 + 𝑢2 𝑑𝑡
LQP
1 2
𝐻 = 𝑥1 + 𝑢2 + 𝜆1 𝑢 − 𝑥1
2
𝜆1 = −𝑥1 + 𝜆1 , 𝜆1 𝑡𝑓 = 0
𝐻𝑢 = 𝑢 + 𝜆1 = 0
𝑢𝑜𝑝𝑡 = −𝜆1
(but don’t know 𝜆1 𝑡 yet)
11
• Free canonical equations (eliminate 𝑢)
(1) 𝑥1 = 𝑢 − 𝑥1 = −𝜆1 − 𝑥1 (𝑥1 0 is known)
(2) 𝜆1 = −𝑥1 + 𝜆1 , 𝜆1 𝑡𝑓 = 0
Combine (1) and (2),
𝜆1 = 2𝜆1  𝜆1 = 𝑘1 𝑒
0 = 𝑘1 𝑒
2𝑡𝑓
+ 𝑘2 𝑒 −
2𝑡
+ 𝑘2 𝑒 −
2𝑡
2𝑡𝑓
𝑥1 = 𝜆1 − 𝜆1 = 𝑘1 1 − 2 𝑒
2𝑡
+ 𝑘2 1 + 2 𝑒 −
2𝑡
𝑥1 0 = 𝑘1 1 − 2 + 𝑘2 1 + 2
𝑢𝑜𝑝𝑡
𝑡 =
= 𝑐1 𝑒
2𝑡
𝑥 0
2−1 +
− 𝑐2 𝑒 −
2 + 1 𝑒2
2𝑡𝑓
𝑒
2𝑡
− 𝑒2
2𝑡𝑓 − 2𝑡
2𝑡
𝑢 < 0 ∀𝑡 for 𝑥 0 > 0, initially correct to reduce 𝑥 𝑡
12
• Another example:
𝑥1 = 𝑥2
𝑥2 = 𝑢 (double integrator)
1
𝑉=
2
∞
0
𝑥12 + 𝑥22 + 𝑢2 𝑑𝑡
1 2 1 2 1 2
𝐻 = 𝑥1 + 𝑥2 + 𝑢 + 𝜆1 𝑥2 + 𝜆2 𝑢
2
2
2
𝜕𝐻
𝜆1 = −
= −𝑥1
𝜕𝑥1
𝜕𝐻
𝜆2 = −
= −𝑥2 − 𝜆1
𝜕𝑥2
𝐻𝑢 = 0 = 𝑢 + 𝜆2  𝑢𝑜𝑝𝑡 = −𝜆2
13
• Free canonical equations
𝑥1 = 𝑥2
𝑥2 = −𝜆2
𝜆1 = −𝑥1
𝜆2 = −𝑥2 − 𝜆1 (𝒙, 𝝀 coupled)
𝜆2 − 𝜆2 + 𝜆2 = 0
Char. Equation: 𝑟 4 − 𝑟 2 + 1 = 0  𝑟′2 − 𝑟 ′ + 1 = 0
𝑟 ′ = 0.5 ± 0.707𝑗
𝑟 = ±0.85 ± 0.4𝑗 (4 roots, apply boundary condition)
14
• Can motivate feedback control via discrete time, one step
ahead
𝑥𝑘+1 = 𝑒𝑥𝑘 + 𝑓𝑢𝑘
Set 𝑘 = 0, 𝑥1 = 𝑒𝑥0 + 𝑓𝑥0 (𝑥0 fixed)
min 𝑉 = 𝑥12 + 𝑎𝑢02
𝑉 = 𝑒𝑥0 + 𝑓𝑢0
2
+ 𝑎𝑢02
𝜕𝑉
= 2𝑓 𝑒𝑥0 + 𝑓𝑢0 + 2𝑎𝑢0 = 0
𝜕𝑢0
𝑎
𝑓
0 = 𝑒𝑥0 + 𝑓𝑢0 + 𝑢0 𝑢0 =
−𝑒𝑥0
𝑎
𝑓+𝑓
Feedback control
15
Continuous Time LQP
𝒙 = 𝑨𝒙 + 𝑩𝒖
1 𝑇
1
𝑉 = 𝒙 𝑡𝑓 𝑺𝒙 𝑡𝑓 +
2
2
𝑡𝑓
𝒙𝑇 𝑸𝒙 + 𝒖𝑇 𝑹𝒖 𝑑𝑡
0
𝑺, 𝑸 ≥ 𝑶, 𝑹 ≥ 𝑶
𝐻=
𝝀𝑇
1 𝑇
1 𝑇
𝑨𝒙 + 𝑩𝒖 + 𝒙 𝑸𝒙 + 𝒖 𝑹𝒖
2
2
𝝀 = −𝑸𝒙 − 𝑨𝑇 𝝀, 𝝀 𝑡𝑓 = 𝑺𝒙 𝑡𝑓
𝑯𝒖 = 𝑶 = 𝑩𝑇 𝝀 + 𝑹𝒖
𝒖𝑜𝑝𝑡 = −𝑹−1 𝑩𝑇 𝝀 (𝑹 > 𝑶)
𝑯𝒖𝒖 = 𝑹 > 𝑶
16
• Free canonical equations
𝒙 = 𝑨𝒙 − 𝑩𝑹−1 𝑩𝑇 𝝀 (𝒙 0 given)
𝝀 = −𝑸𝒙 − 𝑨𝑇 𝝀 (𝝀 𝑡𝑓 given)
Let 𝝀 = 𝑷𝒙 (Riccati transformation)
𝒖𝑜𝑝𝑡 = −𝑹−1 𝑩𝑇 𝑷𝒙, let 𝑲 = 𝑹−1 𝑩𝑇 𝑷 (feedback control)
Then we have ODE in 𝑷
𝒙 = 𝑨𝒙 − 𝑩𝑹−1 𝑩𝑇 𝑷𝒙 (1)
𝝀 = −𝑸𝒙 − 𝑨𝑇 𝝀  𝑷𝒙 + 𝑷𝒙 = −𝑸𝒙 − 𝑨𝑇 𝑷𝒙 (2)
17
Substitute Eq. (1) into Eq. (2):
𝑷 + 𝑷𝑨 + 𝑨𝑇 𝑷 − 𝑷𝑩𝑹−1 𝑩𝑇 𝑷 + 𝑸 = 𝑶 (Riccati ODE)
𝑷 𝑡𝑓 = 𝑺
( backward time integration)
At steady state, 𝑷 → 𝑷𝑒 for 𝑡𝑓 → ∞, solve steady state
equation.
𝑷 is symmetric, 𝑷 = 𝑷𝑇
18
• Example
𝑸=
0 0
, 𝑡𝑓 → ∞
0 1
−1 0
1
,𝑩=
, 𝑅 = 0.1
1 0
0
Plug into Riccati Equation (Steady state)
𝑨=
2
5𝑃11
+ 𝑃11 − 𝑃12 = 0
𝑃11 = 0.1706
2
𝑃22 = 0.8556

10𝑃12
−1=0
𝑃12 = 𝑃21 = 0.3162
1 + 10𝑃11 𝑃12 − 𝑃22 = 0
Feedback Matrix:
𝑲 = 𝑹−1 𝑩𝑇 𝑷 = −1.706 −3.162
19
• Generally 3 ways to solve steady state Riccati
Equation:
(1) integration of ode’s  steady state;
(2) Newton-Raphson (non linear equation
solver);
(3) transition matrix (analytical solution).
20
• Transition matrix approach
𝒙
𝑨 −𝑩𝑹−1 𝑩𝑇
=𝜸=
𝑇 𝜸
−𝑸
−𝑨
𝝀
Reverse time integration (Boundary Condition: at 𝑡 = 𝑡𝑓 ):
Let 𝜏 = 𝑡𝑓 − 𝑡
When 𝑡 = 𝑡𝑓 , 𝜏 = 0
𝑑𝜸
−𝑨 𝑩𝑹−1 𝑩𝑇
=𝜸=
𝜸
𝑸
𝑨𝑇
𝑑𝜏
𝜸 = 𝑒𝒛 𝜏 𝜸 𝜏 = 0
Partition exponential
𝜃11
𝒙
=𝜸=
𝝀
𝜃21
𝜃12
𝜸 𝜏=0
𝜃22
21
𝒙 𝜏 = 𝜃11 𝒙 𝑡𝑓 + 𝜃12 𝝀 𝑡𝑓 = 𝜃11 𝒙 𝑡𝑓 + 𝜃12 𝑷 𝑡𝑓 𝒙 𝑡𝑓 (1)
𝝀 𝜏 = 𝜃21 𝒙 𝑡𝑓 + 𝜃22 𝝀 𝑡𝑓
𝑷 𝜏 𝒙 𝜏 = 𝜃21 𝒙 𝑡𝑓 + 𝜃22 𝑷 𝑡𝑓 𝒙 𝑡𝑓 (2)
Combine (1) and (2), factor out 𝒙 𝑡𝑓
𝑷 𝜏 𝜃11 + 𝜃12 𝑷 𝑡𝑓
= 𝜃21 + 𝜃22 𝑷 𝑡𝑓
Fix integration ∆𝑡, 𝜃𝑖𝑗 Δ𝑡 is fixed
𝑷 𝑡 − ∆𝑡 𝜃11 + 𝜃12 𝑷 𝑡
= 𝜃21 + 𝜃22 𝑷 𝑡
Boundary condition: 𝑷 𝑡𝑓 = 𝑺
Backward time integration of 𝑃, then forward time integration
𝒙 = 𝑨𝒙 + 𝑩𝒖
𝒖 = −𝑹−1 𝑩𝑇 𝑷𝒙
22
Integral Action (eliminate offset)
• Add terms 𝒖𝑇 𝑹𝒖 or 𝒙1𝑇 𝑸𝒙1 to objective function
Example: 𝑥1 = 𝑎𝑥1 + 𝑏𝑢
1
𝑉=
2
𝑑𝑢
2
2
𝑞𝑥1 + 𝑟𝑢 + 𝑞
𝑑𝑡
2
𝑑𝑡
Augment state equation
𝑥1 = 𝑎𝑥1 + 𝑏𝑢 (new state variable)
𝑑𝑢
𝑑𝑡
= 𝑤 (new control variable)
Calculate feedback control
𝑤 𝑜𝑝𝑡 = −𝑘1 𝑥1 − 𝑘2 𝑢
Integrate: 𝑢 = 𝑘′1
𝑑𝑢
1
= −𝑘1 𝑥1 − 𝑘2 𝑥1 − 𝑎𝑥1
𝑑𝑡
𝑏
𝑥1 𝑑𝑡 + 𝑘′2 𝑥1
23
• Second method:
𝑥0 =
𝑥1 𝑑𝑡; 𝑥0 = 𝑥1
1
𝑉=
2
𝑞𝑥12 + 𝑟𝑢2 + 𝑞 𝑥0
2
𝑑𝑡
𝑥0 = 𝑥1
𝑥1 = 𝑎𝑥1 + 𝑏𝑢
Optimal control:
𝑢 = −𝑘1 𝑥1 − 𝑘0 𝑥0 = −𝑘1 𝑥1 − 𝑘0
𝑥1 𝑑𝑡
With more state variables,  PID controller
24