EMGT 6412/MATH 6665
Mathematical Programming
Spring 2016
Integer Programming
Dincer Konur
Engineering Management and Systems
Engineering
1
Outline
• Integer Programming
– Formulations
– Linear Relaxation and Integrality
– Branch and Bound
• Scheme, Fathoming, Branching, Node Selection
– Cutting Plane Algorithm
– Branch and Cut
• Mixed Integer Programming
– Formulations
– Bender’s Decomposition
– Branch and price
2
Outline
• Integer Programming
– Formulations
– Linear Relaxation and Integrality
– Branch and Bound
• Scheme, Fathoming, Branching, Node Selection
– Cutting Plane Algorithm
– Branch and Cut
• Mixed Integer Programming
– Formulations
– Bender’s Decomposition
– Branch and price
3
Integer Programming
• Linear programming assumes that the decision
variables can take fractional values
• In practice, decision variables might be restricted
to be integers or binary
– Or we might need to define binary variables for
modeling conditional constraints
4
Integer Programming: Example
• A product-mix problem
– A company decides to produce two new products
• A door with aluminum framing
– Unit profit for doors is $300
• A wood-framed glass window
– Unit profit for windows is $500
– The company has 3 plants
– What should be the product mix to maximize profits?
• How many of each item to produce weekly?
Production Time Used for Each Unit Produced
Plant
Doors
Windows
Availability/we
ek
1
1 hour
0
4 hours
2
0
2 hours
12 hours
3
3 hours
2 hours
18 hours
5
Integer Programming: Example
• Suppose, we will produce • Suppose, we will produce
for one week only
continuously
– D: number of doors/week,
W: number of windows/week
Maximize
subject to
300D+500W
D≤ 4
2W ≤ 12
3D+2W ≤ 18
D≥0, W ≥0
– D: number of doors,
W: number of windows
Maximize 300D+500W
subject to D≤ 4
2W ≤ 12
3D+2W ≤ 18
D≥0, W ≥0
D integer, W integer
6
Integer Programming: Example
Production rate for windows
W
10
3 D + 2 W = 18
8
D= 4
6
2 W =12
4
Feasible
2
0
region
2
4
Production rate for doors
6
8
Decision Variables are continuous
D
Decision Variables are integers
7
Binary Programming
• A special class of integer programming problems
• Decision variables are binary
– A binary decision variable can be 0 or 1
– Generally used for Yes and No decisions
• Suppose that your decision is to buy or not to buy a house
– Let X be 0 if you decide not to buy the house
– Let X be 1 if you decide to buy the house
X=
0 𝑖𝑓 𝑦𝑜𝑢 𝑑𝑜 𝑛𝑜𝑡 𝑏𝑢𝑦
1 𝑖𝑓 𝑦𝑜𝑢 𝑑𝑜 𝑏𝑢𝑦
8
Binary Programming: Example
• Southwestern Airways Crew
Scheduling
– Southwestern Airways needs to assign
crews to cover all its upcoming flights.
– We will focus on assigning 3 crews
based in San Francisco (SFO) to 11
flights.
Seat tl e
(SEA)
San Francis co
(SFO)
Chi cago
ORD)
Denver
(DEN)
Los Angel es
(LAX)
• Question: How should the 3 crews
be assigned 3 sequences of flights
so that every one of the 11 flights is
covered?
9
Binary Programming: Example
• We are given the following data
Feasible Sequence of Flights
Flights
1
1. SFO–LAX
1
2. SFO–DEN
2
3
5
6
1
1
3. SFO–SEA
1
2
3
3
3
4
9. DEN–ORD
3
4
4
4
6
7
3
5
5
3
3
4
5
7
2
4
2
3
2
2
2
11. SEA–LAX
1
Flight 2
then 9
then 6
then 8
5
2
10. SEA–SFO
12
4
7. ORD–SEA
2
11
1
3
3
10
1
1
2
2
9
1
2
8. DEN–SFO
8
1
1
6. ORD–DEN
Cost, $1,000s
7
1
4. LAX–ORD
5. LAX–SFO
4
8
5
2
4
4
2
9
9
8
9
10
Binary Programming: Example
• Mathematical formulation
– Let xj = 1 if flight sequence j is assigned to a crew; 0
otherwise. (j = 1, 2, … , 12).
– We are minimizing total cost
Minimize Cost = 2x1 + 3x2 + 4x3 + 6x4 + 7x5 + 5x6 +
7x7 + 8x8 + 9x9 + 9x10 + 8x11 + 9x12
– We have at most three crews to assign
x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 + x12 ≤ 3
11
Binary Programming: Example
• Mathematical formulation continues…
– Each flight has to be covered
• Sequences 1, 4, 7, 10 cover flight 1
• So at least one of these sequences should be selected
x1 + x4 + x7 + x10 ≥ 1
Flight 1 covered
• Sequences 2, 5, 8, 11 cover flight 2
• So at least one of these sequences should be selected
x2 + x5 + x8 + x11 ≥ 1
Flight 2 is covered
12
Binary Programming: Example
Minimize Cost = 2x1 + 3x2 + 4x3 + 6x4 + 7x5 + 5x6 + 7x7 + 8x8
+ 9x9 + 9x10 + 8x11 + 9x12
subject to
Flight 1 covered:
x1 + x4 + x7 + x10 ≥ 1
Flight 2 covered:
x2 + x5 + x8 + x11 ≥ 1
:
:
Flight 11 covered: x6 + x9 + x10 + x11 + x12 ≥ 1
Three Crews:
x1 + x2 + x3 + x4 + x5 + x6 + x7 +
x8 + x9 + x10 + x11 + x12 ≤ 3
and
xj are binary (j = 1, 2, … , 12).
13
Binary Programming: Example
• Conditional constraints:
– If sequences 1, 2, 3 are all selected, then 4 cannot be selected
• x1 + x2 + x3 + x4 ≤ 3
– If sequence 5 is selected, 6 cannot be selected
• x5 + x6 ≤ 1
– If 10 is selected, 11 has to be selected
• x11 ≥ x10
– If 5 is selected, then at most 1 other can be selected
• x1+x2+x3+x4+x6+x7+x8+x9+x10+x11+x12 ≤ 11-10x5
• More examples in Supplement 4
14
Outline
• Integer Programming
– Formulations
– Linear Relaxation and Integrality
– Branch and Bound
• Scheme, Fathoming, Branching, Node Selection
– Cutting Plane Algorithm
– Branch and Cut
• Mixed Integer Programming
– Formulations
– Bender’s Decomposition
– Branch and price
15
Linear Relaxation
• Linear relaxation of an integer programming
model is achieved by removing the integer
constraints
IP:
LP:
– Let x* be solution of IP and x** be solution of LP
16
Linear Relaxation
• Linear relaxation of an binary programming
model is achieved by removing the binary
constraints
BIP:
LP:
– Let x* be solution of BIP and x** be solution of LP
17
Linear Relaxation
• Consider the following example:
Z=10
Z=11
Rounding the solution of
the relaxation will not
necessarily work
18
Linear Relaxation
• Linear relaxation solution provides a bound for
the integer model
– If the solution of the linear relaxation is integer, it is the
solution of the integer programming model as well
– How can we be sure if the linear relaxation solution will
be integral?
• If all extreme points of the linear relaxation are integral!
19
Integral Polyhedron
• A polyhedron is a set of inequalities (x is in Rn)
• A convex hull of a set S is the set of all convex
combinations of the points in the set, conv(S)
– Integer hull of P is
– Note that PI is also a polyhedron
• P is integral if and only if
20
Integral Polyhedron
• Let
and P is integral, then
• So if we can show that P is integral, then we do
not need to solve the integer model
– We can solve the linear relaxation
21
Integral Polyhedron
• A face of P is
where
is a valid inequality for P
– So a face is the intersection of the polyhedron and a
valid inequality
• A polyhedron is integral if and only if
– Every face of P has an integer vector
22
Integral Polyhedron
• A face of P is minimal if it does not contain any
other face
– A polyhedron is integral if and only if
• Every minimal face of P has an integer vector
• A face of P is a facet if and only if dim(F)=dim(P)-1
– A polyhedron is integral if and only if
• Every facet of P has an integer vector
23
Integral Polyhedron
• Let
be non-empty.
• P is integral if and only if for all integral vectors c
j
is integral
• Integrality and total unimodularity
– A matrix A is totally unimodular if every square
submatrix of A has determinant -1, 0, or 1
• A consists of -1, 0, 1
• If A is TU, then AT is TU
24
Integral Polyhedron
• Proof question:
• If A is totally unimodular and b is integral, then
every extreme point of the following linear
programming model is integral
• If A is totally unimodular, then
is integral for all integral b
25
Outline
• Integer Programming
– Formulations
– Linear Relaxation and Integrality
– Branch and Bound
• Scheme, Fathoming, Branching, Node Selection
– Cutting Plane Algorithm
– Branch and Cut
• Mixed Integer Programming
– Formulations
– Bender’s Decomposition
– Branch and price
26
Branch and Bound
• This is an exact method to solve integer
programming model
– Any kind of integer programming model:
• Pure-integer
• Binary
• Linear and non-linear integer models
– The main idea is to use the linear relaxation of an
integer programming model
IP:
LP:
27
Branch and Bound: Scheme
• Consider the following integer programming
model:
– You sequentially solve linear relaxation of subproblems
– A sub-problem is the original IP plus additional
constraints achieved through branching
• These constraints are referred to as cuts
– You stop branching (i.e., creating sub-problems) out of
a given subproblem in three cases
28
Branch and Bound: Scheme
• The first problem is the original model (node 0)
– Initially, let say x is integral and feasible to IP
– So, the best integer value so far has z=cx
Solve its LP
relaxation
(LRP)
Let x0 be the
opt. solution of
LRP0
If x0 is integral, it is optimum to node 0,
hence, stop
Else,
• If cx0>=z, you cannot improve node 0,
so stop
• Else, let xi0
be fractional
• Either
29
• Or
Branch and Bound: Fathoming
• Stop branching on a node (fathoming a node)
(assuming minimization)
– If the node is infeasible
– If the optimum solution to its LRP is integral
• When the optimum solution is integral for a node, compare it
with the best integer solution and update the best integer
solution if you have a better integer solution at this node
– If the optimum solution to its LRP has an objective
function value >= (<= for maximization problem) the
objective function value of the current best integer
solution
30
– Else, continue branching
Branch and Bound: Branching
• Branching rules:
–
–
–
–
Select the fractional value closest to the its rounded value
Select the fractional value closest to 0.5 ?
Randomly select one
Adding multiple of them at the same time will not work!!
• Every time you add a cut, you divide a node into 2 new subproblems,
this can be done by adding the same function with >= and <=, (for
instance, xi>=5, xi<=4)
– If you find other constraints that will divide a subproblem into two
subproblems that use function of the variables, it is also possible!!
• For instance, 2x1+3x3<=10 and 2x1+3x3>=11… note that since the
coefficients are integers, 2x1+3x3 should be integer for integer x1
and x3 values…
• But this is referred to as cutting method, we will see more!
31
Branch and Bound: Node-Selection
• At an intermediate point, you might have couple
of nodes that you can explore
– Node selection rule defines which node to explore first
– Two common approaches
• Depth-first… always explore the new subproblems
• Breadth-first… always explore the subproblems in the same
level
• Different methods can have different efficiencies for different
problem characteristics/classes
32
Branch and Bound: Node-Selection
Depth-First
Breadth-First
1
1
2
3
7
5
4
8
8
4
6
3
2
9
5
6
7
9
33
Branch and Bound: Pseudo-Code
• A Pseudo-code:
•
– Let L be the list of nodes: L={0} initially
– And z be the best integer solution’s obj. function value
While L is not empty
– Pick a node from L based on node selection rule and remove it from L
– If node is infeasible, do not branch (fathom due to infeasibility)
– Else if the solution is integral, do not branch (fathom due to integrality)
• Compare it to the best solution
• Update the best integer solution if this is better
– If solution is not integral
• If the objective function value of LRP is worse, do not branch (fathom due to bound)
• Else, branch and create two nodes |L|+1 and |L|+2
– Update L such that L=L U {|L|+1, |L|+2}
• End
• Return the best integer solution
34
Branch and Bound
• Consider the following binary model:
– See Supplement 4 for numerical solution with branchand-bound
35
Outline
• Integer Programming
– Formulations
– Linear Relaxation and Integrality
– Branch and Bound
• Scheme, Fathoming, Branching, Node Selection
– Cutting Plane Algorithm
– Branch and Cut
• Mixed Integer Programming
– Formulations
– Bender’s Decomposition
– Branch and price
36
Cutting Plane Algorithm
• The idea of the cutting plane algorithm is to
iteratively add feasible cuts to the linear
relaxation until the solution of the linear relaxation
of an IP is integer
– Cut: It is a constraint that will make some of the
fractional points infeasible while keeping all of the
original integer points feasible
– There are different types of cuts that can be used for
different problems
• Cut generation is another problem itself
37
Cutting Plane Algorithm
• Convex Hull: The convex hull or convex envelope of a set X of
points in the Euclidean plane or Euclidean space is the smallest
convex set that contains X (wikipedia)
For instance, the pink area
is the convex hull of the
black points
• When we have a linear integer programming model
– The convex hull of the feasible integer points will be a polyhedral
– So, if we can find the convex hull of the set of feasible points of a linear IP, we can
solve its linear relaxation over this convex hull and guarantee that the solution of
the linear relaxation within the convex hull will be optimum to our IP model
38
Cutting Plane Algorithm
• An illustration:
–
–
–
–
Let’s say we have the original constraints defined by blue-lines
The red dots are the feasible integer points
The red lines define the convex hull
The green line is a feasible cut
Original IP model:
Max cx
s.t. blue lines
x integer
After adding cuts
to LRP model:
Max cx
s.t. red lines
Its LRP model:
Max cx
s.t. blue lines
Not guaranteed
that we have an
integer optimum
solution
Still a linear model and
it is guaranteed that its
solution is integer and
optimum to the original
IP model
39
Cutting Plane Algorithm
•
However, we do not need to generate all of the cuts defining the convex hull
– Note that when the polyhedral is equal to the convex hull of the integer points in it,
then that polyhedral is integral!! (i.e., when blue lines are red lines also)
– After adding a cut, if the solution of the linear relaxation is integer, we do not need
to add further cuts, we have the optimum solution to the IP model
• The iterations:
1: Solve the linear relaxation
2:
If the solution is integer, stop, it is the optimum solution to the IP
3:
If the solution is not integer
» Generate a feasible cut
» Add it to the model
» Go to 1
40
Outline
• Integer Programming
– Formulations
– Linear Relaxation and Integrality
– Branch and Bound
• Scheme, Fathoming, Branching, Node Selection
– Cutting Plane Algorithm
– Branch and Cut
• Mixed Integer Programming
– Formulations
– Bender’s Decomposition
– Branch and price
41
Branch and Cut
• Branch and Cut has the same idea with branch and
bound
• The only difference is in adding cuts
– The purpose of the cuts (cutting planes) is to cut off the optimum
solutions of the linear relaxation problems that are non-integral
– For instance, at a node of the branch and bound, you solved the
linear relaxation, which is not infeasible
• If the solution is integral, we stop branching
• If the solution is not integral but worse than best integer solution, we
stop branching
• Else, we would continue with branching and creating new nodes
42
Branch and Cut
• Branch and Cut has the same idea with branch and
bound
• In the branch and cut method
– For instance, at a node of the branch and cut, you solved the
linear relaxation, which is not infeasible
• If the solution is integral, we stop branching
• If the solution is not integral but worse than best integer solution, we
stop branching
• Else, we add a cut to the linear relaxation and solve it again, then
check, if now we can fathom the node due to bound
– Note that the cut will be feasible
43
Branch and Cut
• Consider a minimization problem with objective function:
2x1+3x2
• Suppose that the best integer solution at an intermediate
step is x1=1 and x2=5 with objective function value 17…
– Consider a node
• If infeasible, stop branching
• If the solution is integral, stop branching
• Let’s say solution is not integral
– Let x1=1.2 and x2=6.1 objective function value is 20.7
– Then, we know that 2x1+3x2<=20… (as x1 and x2 has to be integers)
Cutting plane
• See Supplement 4 for a numerical example
44
Outline
• Integer Programming
– Formulations
– Linear Relaxation and Integrality
– Branch and Bound
• Scheme, Fathoming, Branching, Node Selection
– Cutting Plane Algorithm
– Branch and Cut
• Mixed Integer Programming
– Formulations
– Bender’s Decomposition
– Branch and price
45
Mixed Integer Programming
• When the model has both integer/binary and continuous
variables, it is a mixed integer programming model
• When we need:
– We might have constraints on the continuous variables that
depend on our yes/no decisions
• If you locate a factory, you can produce there
• If you choose to manufacture a product, you have to produce at least
a specific number
• If you setup a machine, you can use it
• You need to select some of the resources to be used
• Fixed and variable costs for the resources:
– a fixed truck cost and unit transportation cost for each item shipped
46
Formulation
• GOOD PRODUCTS COMPANY has developed three possible new
products. How much of each to produce to maximize profit?
• We can formulate this problem easily!
LP model
47
Formulation
• But we have the following restrictions:
– Restriction 1: From the three possible new products, at most two
should be chosen to be produced.
• The number of strictly positive decision variables (x1, x2, x3) must be <=2
– Restriction 2: Just one of the two plants should be chosen to be
the sole producer of the new products.
•
• Let’s focus on Restriction 1:
– Now, it is our decision to select which products to produce: we
either produce a product or not
– Yes/No type of decision, so we need to define binary variables
– Define:
48
Formulation
• Restriction 1:
– Now, we need to associate yj with xj
– That is, when yj=0, xj=0 and when yj=1, xj>=0
– To do so, we use big M constraints
• Suppose M is a large number
•
• So, based on the above constraints
– If yj=0, then xj<=0, i.e., xj=0
– If yj=1, xj<=M, where M is a very large number, i.e., we can produce as
much as we want (the other constraints will take care of the resource
limitations anyway)
– Finally, we need to assure that we produce at most two types of
products, so we have:
•
49
Formulation
• Let’s focus on Restriction 2:
– Just one of the two plants should be chosen to be the sole producer of
the new products. So, it is our decision to choose one of the plants
– We choose plant 1 or plant 2, so let use define a binary variable:
– Now, we need to associate the resource constraints with our choice
variable. To do so, we will use the big M again
If y4=1, plant 2 is chosen, so the plant
1 constraint is redundant and plant 2
constraint is valid
If y4=0, plant 1 is chosen, so the plant
1 constraint is valid and plant 2
constraint is redundant
50
Formulation
• Combining everything:
Alternatively, one can define
• p1=1 if plant 1 is chosen, p1=0 otherwise
• p2=1 if plant 2 chose, p2=0 otherwise
Then, we can add:
• 3x1+4x2+2x3<=30+Mp2
• 4x1+6x2+2x3<=40+Mp1
• p1+p2=1
• p1, p2 binary
– More formulation examples on Supplement 4
51
Bender’s Decomposition
• Consider the following linear model:
– x and y are the decision variable vectors
– Suppose y variables are complicating
52
Bender’s Decomposition
• Problem (1) can be written as follows:
where q(y) is the optimum solution of
53
Bender’s Decomposition
• Note that Problem (3) is a linear model
– If (3) is unbounded for a given y, then Problem (2) is
unbounded, which means Problem (1) is unbounded
– Assuming (3) is bounded, we can solve its dual
• Note that feasible region of the dual in (4) does not depend on y
54
Bender’s Decomposition
• Consider the dual of (3) given in (4)
– If feasible region of (4) is empty
• Primal can be unbounded, i.e., (3) is unbounded, hence, (2) is
unbounded, so (1) is unbounded
• Primal can be infeasible, which means (3) is infeasible for all y,
which means (1) is infeasible
55
Bender’s Decomposition
• (4) can be formulated in terms of its extreme
points and rays…
– Extreme points
– Extreme rays
–
are #’s of extreme points and rays
– Given y, (4) can be solved by
• Checking whether
for extreme rays
– If there is such a ray, dual in (4) is unbounded hence (3) is
infeasible so (1) is infeasible
– So, feasibility of (1) implies
• Finding extreme point with maximum
– In this case, both (4) and (3) have finite optimum
56
Bender’s Decomposition
• Then (4) is equal to:
– Insert this to (2)
57
Bender’s Decomposition
• Then (2) is equal to:
Master
Problem
– But I and J are generally exponential…
– So, it is not practical to generate all…
58
Bender’s Decomposition
• Bender’s decomposition starts with a subset of
the extreme rays and extreme directions
– This is called as relaxed master problem (as we have
less constraints than the master problem)
Relaxed
Master
Problem
– Let
be the optimum solution of RMP
59
Bender’s Decomposition
• With the given
, solve the dual given in (4)
– If the dual in (4) is unbounded,
for
some extreme ray, so we will add a constraint in the
following form
to RMP and extend it
• This is called “Bender’s feasibility cut” as it enforces feasibility
of model (3)
– If the dual in (4) has an optimum solution, which will be
one of the extreme points, say extreme point p
• If
in the following form
, we will add a constraint
to RMP and extend it
– This is called “Bender’s optimality cut”
• If
, then stop, we have found optimum y and x
60
Branch and Price
• Suppose that you have a mixed integer model
with too many continuous variables
– You can use branch and bound
– However, solving the linear relaxations at the nodes of
the branch and bound tree will be challenging
• And we might have many nodes
– Recall that we can use column generation to solve
large scale linear programming models
61
Branch and Price
• Branch and price uses column generation to solve the
linear relaxations at the nodes of a branch and bound tree
• Need to be careful:
– Each node will have different set of constraints
– So, each node will have different columns being feasible
• Finding initial set of feasible columns (and good ones) is important to
speed up the column generation
– Fewer columns generated, faster solution for a node
• Pricing problem should be easy to solve
62
Next time
• If there is a next time
– Overall of heuristics
•
•
•
•
•
•
A simple greedy search for 0-1 knapsack
Genetic algorithms
Tabu search
Simulated annealing
Ant colony optimization
Many more….
63