Dynamic programming – Maximisation problem
Problem 2: Find the route through the network
from A to H with maximum total weight.
B
5
3
4
2
A
C
F
4
2
3
1
E
H
5
3
1
D
2
5
3
6
G
Dynamic programming – Maximisation problem
The first step is to label all the nodes with stage and state variables.
The stage variable for a node is the maximum number of transitions required to get from
this node to the end node, H. All nodes with the same stage variable are then given a
different state variable, starting with 1 at the top of the page and working downwards.
H has stage variable 0, and is given the state variable 1.
B
5
3
4
2
A
C
F
4
2
3
1
E
H
(0, 1)
5
3
1
D
2
5
3
6
G
Dynamic programming – Maximisation problem
The first step is to label all the nodes with stage and state variables.
The stage variable for a node is the maximum number of transitions required to get from
this node to the end node, H. All nodes with the same stage variable are then given a
different state variable, starting with 1 at the top of the page and working downwards.
F and G have stage variables of 1. F is given state variable 1, and G is given state variable 2.
B
5
3
4
2
A
C
F
4
2
(1, 1)
3
1
E
H
(0, 1)
5
3
1
D
2
5
3
6
G
(1, 2)
Dynamic programming – Maximisation problem
The first step is to label all the nodes with stage and state variables.
The stage variable for a node is the maximum number of transitions required to get from
this node to the end node, H. All nodes with the same stage variable are then given a
different state variable, starting with 1 at the top of the page and working downwards.
E is the only node with stage variable 2. E is given state variable 1.
B
5
3
4
2
A
C
F
4
2
(1, 1)
3
1
E
You could go via
G instead which
also gives two
transitions
(2, 1)
H
(0, 1)
5
3
1
D
2
5
3
6
G
(1, 2)
Dynamic programming – Maximisation problem
The first step is to label all the nodes with stage and state variables.
The stage variable for a node is the maximum number of transitions required to get from
this node to the end node, H. All nodes with the same stage variable are then given a
different state variable, starting with 1 at the top of the page and working downwards.
C is the only node with stage variable 3. C is given state variable 1.
B
5
3
4
2
A
F
4
2
C
3
1
E
(2, 1)
(3, 1)
3
1
D
(1, 1)
H
(0, 1)
5
2
5
3
6
G
(1, 2)
Dynamic programming – Maximisation problem
The first step is to label all the nodes with stage and state variables.
The stage variable for a node is the maximum number of transitions required to get from
this node to the end node, H. All nodes with the same stage variable are then given a
different state variable, starting with 1 at the top of the page and working downwards.
B and D have stage variables of 4. B is given state variable 1, and D is given state variable 2.
B
(4, 1)
5
3
4
2
A
F
4
2
C
3
1
E
(2, 1)
(3, 1)
1
D
(1, 1)
H
(0, 1)
5
3
2
(4, 2)
5
3
6
G
(1, 2)
Dynamic programming – Maximisation problem
The first step is to label all the nodes with stage and state variables.
The stage variable for a node is the maximum number of transitions required to get from
this node to the end node, H. All nodes with the same stage variable are then given a
different state variable, starting with 1 at the top of the page and working downwards.
A has stage variable 5. A is given state variable 1.
B
(4, 1)
5
3
4
2
A
(5, 1)
F
4
2
C
3
1
E
(2, 1)
(3, 1)
1
D
(1, 1)
H
(0, 1)
5
3
2
(4, 2)
5
3
6
G
(1, 2)
Dynamic programming – Maximisation problem
The next step is to set up a table to show your working.
B
(4, 1)
5
3
4
2
A
(5, 1)
F
4
2
C
3
1
E
(2, 1)
(3, 1)
1
D
(1, 1)
H
(0, 1)
5
3
2
(4, 2)
5
3
6
G
(1, 2)
Dynamic programming – Maximisation problem
Stage
1
State Action
F (1, 1)
1
Value
3
Current
maximum
3
5
B
(4, 1)
2
A
(5, 1)
4
3
4
2
C
(3, 1)
3
1
D
2
(4, 2)
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 1, consider nodes with stage
variable 1. These are F and G.
There is only one possible action from
F, which is FH.
This has value 3.
The maximum value of a route starting
from F is therefore 3 .
Dynamic programming – Maximisation problem
Stage
1
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
Current
maximum
3
5
B
(4, 1)
6
2
A
(5, 1)
4
3
4
2
C
(3, 1)
3
1
D
2
(4, 2)
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 1, consider nodes with stage
variable 1. These are F and G.
There is only one possible action from
G, which is GH.
This has value 6.
The maximum value of a route starting
from G is therefore 6 .
Dynamic programming – Maximisation problem
Stage
1
2
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
E (2, 1)
1
2
3
1+3=4
Current
maximum
3
5
B
(4, 1)
6
2
A
(5, 1)
4
3
4
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 2, consider nodes with stage
variable 2. This is E only.
There are three possible actions from E.
Action 1 is EF.
The value of EF is 1. From stage 1, the
maximum value of a route from F is 3.
So the value of this route is 4.
Dynamic programming – Maximisation problem
Stage
1
2
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
E (2, 1)
1
2
3
1+3=4
5
Current
maximum
3
5
B
(4, 1)
6
2
A
(5, 1)
4
3
4
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 2, consider nodes with stage
variable 2. This is E only.
There are three possible actions from E.
Action 2 is EH.
The value of EH is 5.
Dynamic programming – Maximisation problem
Stage
1
2
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
E (2, 1)
1
2
3
1+3=4
5
3+6=9
Current
maximum
3
5
B
(4, 1)
6
2
A
(5, 1)
4
3
4
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 2, consider nodes with stage
variable 2. This is E only.
There are three possible actions from E.
Action 3 is EG.
The value of EG is 3. From stage 1, the
maximum value of a route from G is 6.
So the value of this route is 9.
Dynamic programming – Maximisation problem
Stage
1
2
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
E (2, 1)
1
2
3
1+3=4
5
3+6=9
Current
maximum
3
2
A
(5, 1)
4
3
4
6
9
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 2, consider nodes with stage
variable 2. This is E only.
The maximum value of a route starting
from E is therefore 9.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
E (2, 1)
C (3, 1)
1
2
3
1+3=4
5
3+6=9
1
2
4+3=7
Current
maximum
3
2
A
(5, 1)
4
3
4
6
9
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 3, consider nodes with stage
variable 3. This is C only.
There are two possible actions from C.
Action 1 is CF.
The value of CF is 4. From stage 1, the
maximum value of a route from F is 3.
So the value of this route is 7.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
E (2, 1)
C (3, 1)
1
2
3
1+3=4
5
3+6=9
1
2
4+3=7
2+9=11
Current
maximum
3
2
A
(5, 1)
4
3
4
6
9
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 3, consider nodes with stage
variable 3. This is C only.
There are two possible actions from C.
Action 2 is CE.
The value of CE is 2. From stage 2, the
maximum value of a route from E is 9.
So the value of this route is 11.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
E (2, 1)
C (3, 1)
1
2
3
1+3=4
5
3+6=9
1
2
4+3=7
2+9=11
Current
maximum
3
2
A
(5, 1)
11
4
3
4
6
9
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 3, consider nodes with stage
variable 3. This is C only.
The maximum value of a route starting
from C is therefore 11.
Dynamic programming – Maximisation problem
Stage
1
2
3
4
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
1
2
3
1+3=4
5
3+6=9
C (3, 1)
1
2
4+3=7
2+9=11
B (4, 1)
1
2
5+3=8
E (2, 1)
Current
maximum
3
2
A
(5, 1)
11
4
3
4
6
9
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 4, consider nodes with stage
variable 4. These are B and D.
There are two possible actions from B.
Action 1 is BF.
The value of BF is 5. From stage 1, the
maximum value of a route from F is 3.
So the value of this route is 8.
Dynamic programming – Maximisation problem
Stage
1
2
3
4
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
1
2
3
1+3=4
5
3+6=9
C (3, 1)
1
2
4+3=7
2+9=11
B (4, 1)
1
2
5+3=8
3+11=14
E (2, 1)
Current
maximum
3
2
A
(5, 1)
11
4
3
4
6
9
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 4, consider nodes with stage
variable 4. These are B and D.
There are two possible actions from B.
Action 2 is BC.
The value of BC is 3. From stage 3, the
maximum value of a route from C is 11.
So the value of this route is 14.
Dynamic programming – Maximisation problem
Stage
1
2
3
4
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
Current
maximum
3
1+3=4
5
3+6=9
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
E (2, 1)
2
A
(5, 1)
9
4
3
4
6
1
2
3
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 4, consider nodes with stage
variable 4. These are B and D.
The maximum value of a route starting
from B is therefore 14.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
Current
maximum
3
1+3=4
5
3+6=9
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
3+11=14
D (4, 2)
1
2
3
E (2, 1)
4
2
A
(5, 1)
9
4
3
4
6
1
2
3
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 4, consider nodes with stage
variable 4. These are B and D.
There are three possible actions from D.
Action 1 is DC.
The value of DC is 3. From stage 3, the
maximum value of a route from C is 11.
So the value of this route is 14.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
Current
maximum
3
1+3=4
5
3+6=9
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
D (4, 2)
1
2
3
3+11=14
2+9=11
E (2, 1)
4
2
A
(5, 1)
9
4
3
4
6
1
2
3
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 4, consider nodes with stage
variable 4. These are B and D.
There are three possible actions from D.
Action 2 is DE.
The value of DE is 2. From stage 2, the
maximum value of a route from E is 9.
So the value of this route is 11.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
Current
maximum
3
1+3=4
5
3+6=9
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
D (4, 2)
1
2
3
3+11=14
2+9=11
5+6=11
E (2, 1)
4
2
A
(5, 1)
9
4
3
4
6
1
2
3
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 4, consider nodes with stage
variable 4. These are B and D.
There are three possible actions from D.
Action 3 is DG.
The value of DG is 5. From stage 1, the
maximum value of a route from G is 6.
So the value of this route is 11.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
Value
F (1, 1)
1
3
G (1, 2)
1
6
Current
maximum
3
1+3=4
5
3+6=9
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
D (4, 2)
1
2
3
3+11=14
2+9=11
5+6=11
E (2, 1)
4
2
A
(5, 1)
9
14
4
3
4
6
1
2
3
5
B
(4, 1)
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 4, consider nodes with stage
variable 4. These are B and D.
The maximum value of a route starting
from D is therefore 14.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
F (1, 1)
1
3
G (1, 2)
1
6
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
D (4, 2)
1
2
3
3+11=14
2+9=11
5+6=11
4+14=18
A (5, 1)
1
2
3
E (2, 1)
9
14
4
3
4
2
A
(5, 1)
1+3=4
5
3+6=9
5
B
(4, 1)
6
1
2
3
4
5
Value
Current
maximum
3
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 5, consider nodes with stage
variable 5. This is A only.
There are three possible actions from A.
Action 1 is AB.
The value of AB is 4. From stage 4, the
maximum value of a route from B is 14.
So the value of this route is 18.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
F (1, 1)
1
3
G (1, 2)
1
6
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
D (4, 2)
1
2
3
3+11=14
2+9=11
5+6=11
A (5, 1)
1
2
3
4+14=18
2+11=13
E (2, 1)
9
14
4
3
4
2
A
(5, 1)
1+3=4
5
3+6=9
5
B
(4, 1)
6
1
2
3
4
5
Value
Current
maximum
3
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 5, consider nodes with stage
variable 5. This is A only.
There are three possible actions from A.
Action 2 is AC.
The value of AC is 2. From stage 3, the
maximum value of a route from C is 11.
So the value of this route is 13.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
F (1, 1)
1
3
G (1, 2)
1
6
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
D (4, 2)
1
2
3
3+11=14
2+9=11
5+6=11
A (5, 1)
1
2
3
4+14=18
2+11=13
1+14=15
E (2, 1)
9
14
4
3
4
2
A
(5, 1)
1+3=4
5
3+6=9
5
B
(4, 1)
6
1
2
3
4
5
Value
Current
maximum
3
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 5, consider nodes with stage
variable 5. This is A only.
There are three possible actions from A.
Action 3 is AD.
The value of AD is 1. From stage 4, the
maximum value of a route from D is 14.
So the value of this route is 15.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
F (1, 1)
1
3
G (1, 2)
1
6
C (3, 1)
1
2
4+3=7
2+9=11
11
B (4, 1)
1
2
5+3=8
3+11=14
14
3+11=14
2+9=11
5+6=11
14
D (4, 2)
1
2
3
4+14=18
2+11=13
1+14=15
18
A (5, 1)
1
2
3
E (2, 1)
9
4
3
4
2
A
(5, 1)
1+3=4
5
3+6=9
5
B
(4, 1)
6
1
2
3
4
5
Value
Current
maximum
3
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
In Stage 5, consider nodes with stage
variable 5. This is A only.
The maximum value of a route starting
from A is therefore 18.
Dynamic programming – Maximisation problem
Stage
1
2
3
State Action
F (1, 1)
1
3
G (1, 2)
1
6
C (3, 1)
1
2
4+3=7
2+9=11
B (4, 1)
1
2
5+3=8
3+11=14
3+11=14
2+9=11
5+6=11
14
D (4, 2)
1
2
3
4+14=18
2+11=13
1+14=15
18
A (5, 1)
1
2
3
E (2, 1)
9
11
14
4
3
4
2
A
(5, 1)
1+3=4
5
3+6=9
5
B
(4, 1)
6
1
2
3
4
5
Value
Current
maximum
3
2
C
(3, 1)
3
1
D
(4, 2)
2
F(1, 1)
3
1
E (2, 1)
5
3
5
H
(0, 1)
6
G
(1, 2)
From the completed table, the maximum
weight for a route from A to H is 18.
The table shows that you need to take
the first action from A, which is AB,
then the second action out of B, which is BC,
then the second action out of C, which is CE,
then the third action out of E, which is EG,
and finally the action GH.
So the maximum weight route is ABCEGH.