Integer Programming
Prof. Iiro Harjunkoski
Hillier & Lieberman: Introduction to Operations Research
Main points to take-away
• What are integer programming problems?
• Why do we need integer variables?
• How to model integer problems?
• Most common techniques to solve them
• Principle of Branch & Bound algorithm
• Some practical training and insights
• Lots of fun! J
Outline
• Introduction
• Linear and Discrete Programming Basics
• Linear programming with examples
• Motivation for discrete programming
• Mixed Integer Linear Programming Introduction
• Mixed Integer Linear Programming: Branch & bound
Solution of Mathematical Problems
Real problem?
Model
Main focus here!
Apply results
Mathematical model
Solver
Results
Mathematical Problems
• A mathematical problem formulation consists of four main elements
• Parameters (known)
• Storage amount, maximum capacity, current temperature, ...
• Constraints (limits the decision space)
• Produce maximally 3 different products, do not exceed capacity, ...
• Variables (the unknown, represent the decisions)
• Start time, line speed, product sequence, ...
• Objective function (measurement of „goodness“ using variables)
• Profit, production makespan, total costs, ...
• Optimization is the activity that finds the variable values that satisfies
the constraints and maximizes or minimizes a selected objective
• In many practical problems, the decision variables actually make sense
only if they have integer values (0, 1, 2, 3, …).
• The mathematical formulation for IP is the LP model with one additional
restriction that the variables must have integer values.
Examples of Linear Optimization Problems
• Stock portfolio
• Which shares to buy and sell?
• Maximize the value of investment (simplified)
• Capacity optimization
• Prioritization of limited resources
• Production optimization
• A number of raw-material exist
• Which customer products to make?
• Coordination of resources
• Who should make them?
• When are the resources available?
Simple Example
• A company „Woodpecker“ makes chairs and tables
• The company has 11 units of wood
• Making a chair consumes 2 units
• Making a table consumes 3 units
• How many chairs and tables should
„Woodpecker“ make?
• Sales prices
• Chairs: 60€
• Tables: 100€
Linear and Discrete
Programming Basics
Linear programming (LP)
• Linear constraints (define the decision space)
• Continuous variables (decisions)
• Convex feasible region
Nonlinear programming (NLP)
• Linear + nonlinear constraints
• Continuous variables
• Convex/nonconvex feasible region
Optimization: Find Optimal point - min f(x,y)
OR?
Optimization in Linear Programming (LP)
• Linear programming characteristics
• Main principle: One (or more) of the corner points is always the optimal one
(Simplex points)
• Simplex algorithm efficiently solves an LP-problem
• Created by americal Mathematician George Danzig 1947
• Still the main algorithm for solving LP problems
• Other methods also exist, e.g. Interior point (Karmarkar)
What if one variable is discrete?
x: discrete
y: discrete
y
y
x
• Linear constraints
• Non-continuous variables → MILP
• Non-convex feasible region!
feasible region
x
Both variables discrete
• Constraints still linear
• Non-convex and non-continuous feasible region
• Only 8 feasible points
• All Simplex points infeasible!
non-convex MINLP problem
y
x
Problem!
• Discrete variables include binary {0,1}, integer {0,1,2,3...} and
fractional {1/2, 3/5, 7/11,...} cases
• Most common: binary or integer!
• Simplex method cannot be expected to find the optimal solution for
discrete linear problems
• Since the problem is non-continuous and the feasible space is thus
also non-convex, even the most clever NLP methods may fail
• We will need another type of methods
Abbreviations
• Linear Programming: LP
• Integer Programming: IP
• Mixed Integer Programming: MIP
• Mixed Integer Linear Programming: MILP
• Mixed Integer Non-Linear Programming: MINLP
• Sometimes: Binary Programming (BIP or 0-1 progr.)
Linear constr
Linear + nonlinear constr
Cont vars
Int vars
Cont + Int vars
LP
ILP
MILP
NLP
INLP
MINLP
Simple Example
• A company „Woodpecker“ makes chairs and tables
• The company has 11 units of wood
• Making a chair consumes 2 units
• Making a table consumes 3 units
• How many chairs and tables should
„Woodpecker“ make?
• Sales prices
• Chairs: 60€
• Tables: 100€
Simple Example - Solution
?
• Variables
• x = amount of chairs produced
• y = amount of tables produced
• Constraints
y ≤ 11/3
• Maximal production amounts
• x ≤ 11/2 = 5,5 chairs
• y ≤ 11/3 = 3,67 tables
• 2 x + 3 y ≤ 11
x ≤ 11/2
• Total wood consumption
3
2
• Objective function: Maximise revenues 1
Feasible region
• Maximize f = 60 x + 100 y
1
2
3
4
5
Simple Example - Solution
?
• Solution: produce 3,67 tables (367€)!
• BUT
• No-one will buy 0,67 table!
• Correct constraints!
y≤3
• Constraints
• Maximal production amounts
2
• Corrected solution: 360 €
• Make 3 tables and 1 chair
• All wood was consumed J
x≤5
• x ≤ 11/2 ≈ 5 chairs
• y ≤ 11/3 ≈ 3 tables
3
1
Feasible region
1
2
3
4
5
Discrete Variables
• The „Woodpecker“ example revealed one important question...
• What if the solution had still been fractional?
• Solution candidate on the right = 5 chairs + 0,33 tables
• Simplex method does not consider discrete variables!
• In production planning problems integer/binary variables are often
necessary!
• How can we deal with them?
Method 1: Complete Enumeration
• "Brute-force" method. Go through all
possible combinations: Fix integer
variables and solve the corresponding
continuous problem
• Number of combinations: 2n, where n
is number of binary (0-1) variables
• 3 binaries: 23 = 8 combinations
• … our example had in practice 16 alternatives
(of total 24)
• What if we had more wood (say 1000 units)?
• 167334 alternatives to check!
1:
2:
3:
4:
5:
6:
7:
8:
000
001
010
011
100
101
110
111
Method 2: Rounding
• Simple solution: forget about the discrete variables and
solve the LP/NLP problem and use some round-off
strategy
Rounding Method: Example
x2
f =4
f =5
f =6
max f ( x1 , x2 ) = x1 + 0.64 x2
f =7
5
50 x1 + 31x2 £ 250
4
- 3 x1 + 2 x2 £ 4
x1 , x2 Î Z
3
LP-solution: (376/193, 950/193)
where f = 5.098
2
1
0
IP-solution: (5,0)
where f = 5.0
1
2
3
4
5
x1
Rounding Method: Result
• LP-solution completely different
than IP solution
• No way to round (1.95, 4.92) → (5,0)
• It is not straightforward to deal with integers
• Why then need discrete variables in the first place?
Rounding example from Hillier & Lieberman
Maximize x2
Binary / Integer Variables
• Needed to represent non-continuous or logical decisions
•
•
•
•
yes/no
1/0
on/off
before/after
• and numbers that must be discrete
• people (1, 2, 3, ...)
• cars, buses, airplanes
• processing units (e.g. reactors, distillation columns)
Example integer programming problems
• Airline Crew Scheduling
• Assignment Problem
• Production Planning
• Electricity Generation Planning
• Ground Holding of Aircraft
• Cutting Problems
• Process Synthesis Problem
• Molecular Biology
• etc...
Airline crew scheduling
• Weekly working hours, linking flights, limited
total flying time, minimum rests, return to
starting point, salary, aircraft type, ...
• Minimize costs!
?
Design of a Processing Plant
• Superstructure (all possible alternatives)
• Complex chemical & physical constraints
• Equipment not identical
• Choose optimal configuration!
Mixed Integer Linear
Programming
Introduction
Mixed Integer Programming
• Systematic framework for the modeling and solution of optimization
problems involving discrete and continuous variables
• Problem types: ILP, MILP, INLP, MINLP
• Main challenges:
• Combinatory
• Nonlinearities
Problem
• The feasible region of an integer problem is a subset of the
continuous one
• MILP or IP = Initial LP + integer variable constraints
• Important: A solution can never improve if a constraint is added to
the problem!
→ The initial IP-solution is never
better as the original LP
Integer Gap
• Integer gap: measurement of complexity of a MILP = distance
between the “relaxed” LP solution and optimal integer solution
• Zero gap: LP solution = MILP solution (not always!)
• "Good relaxation" ≈ small integer gap
integer gap
zero gap (min y)
Convex Hull
• An IP can in principle be formulated in several ways
• Convex Hull joins all the feasible extreme points into a convex
envelope
• Convex Hull ideal = tightest possible IP formulation
Mathematical Problem Formulation
• Mixed Integer Linear Programming (MILP) problem
min f = c1 x + c2 y
subject to
Ax + By £ d
x, y ³ 0, y Î Z
positive
integer
Objective function with
two cost-coefficients
Linear constraints
Variable definitions
Most Common: Binary Variables
• It is typical to only use 0-1 variables for the problem formulation
• Easier to represent logics and manipulate constraints with binary variables
• All integers (z) can be represented by binaries (y), e.g.
z = j× yj
J
z = å 2 j -1 × y j
j =1
y j Î {0,1}
z = 3:
(2
0
J
OR
åy
j =0
j
=1
y j Î {0,1}
)
+ 21 , y1 = y2 = 1
( y3 = 1)
Modeling with 0-1 variables
• 0-1 allows to model logic constraints and discontinuous functions
• At least one item
åy
• Exactly one item
åy
jÎJ
jÎJ
• At most one item
j
³1
Every product must go through
minimum one production step
j
=1
One driver per bus needed
å yj £1
jÎJ
Select at most one of the parallel
equipment
ì1, if item j is selected (true)ü
yj = í
ý
0
,
otherwise
(false)
î
þ
Modeling with 0-1 variables
• AND is defined with multiple constraints:
(1 or 2) AND (3 or 4)
y1 + y2 ³ 1
y3 + y4 ³ 1
At least one of the pumps 1 or 2 and
container 3 or 4 must be selected
• Implication (if a then b)
ya - yb £ 0
ya = 1 Þ
ya = 0 Þ
yb = 1
yb = {0,1}
If we perform a maintenance (a) then
a mechanic (b) must be sent
Assignment problem
• Minimize assignment costs
• Binary variable: job i assigned to machine j :
(can be solved as an LP)
n
m
min åå cij yij
i =1 j =1
m
åy
j =1
ij
= 1, i = 1,2,..., n
ij
= 1,
n
åy
i =1
yij = 1
yij Î {0,1}
j = 1,2,..., m
job i
machine j
Knapsack problem
• Maximise value with limited weight (W)
• Select object i :
yi = 1
n
max å ci yi
i =1
n
åw y
i =1
i
i
£W
yi Î {0,1}
knapsack
item i
Logic in 0-1
• Let pj represent a selection/action (yj = 1)
Expression Logic
Algebraic
NOT
Øp j
1- y j
OR
pi Ú p j
yi + y j ³ 1
XOR
pi Ú p j
yi + y j = 1
AND
pi Ù p j
yi = 1, y j = 1
IMPL
Øpi Ú p j
y j ³ yi
• Often better to start with logical model (general) before deriving the
mathematical model (CNF)
Exercise on Logical Variables
Suppose that you are interested in choosing a set of
investments {1,...,7} using 0-1 variables. Model the
following constraints:
•
•
•
•
•
•
You cannot invest in all of them
You must choose at least one of them
Investment 1 cannot be chosen if investment 3 is chosen
Investment 4 can be chosen only if investment 2 is also chosen
You must choose either both investments 1 and 5 or neither
You must choose either at least one of investments 1,2,3 or at least two
investments from 4,5,6
Exercise - Answers
• You cannot invest in all of them
åy
i
£6
åy
i
³1
i
• You must choose at least one of them
• Investment 1 cannot be chosen if investment 3 is
chosen
• Investment 4 can be chosen only if investment 2 is
also chosen
• You must choose either both investments 1 and 5
or neither
i
y1 + y3 £ 1
y4 £ y2
y1 = y5
• You must choose either at least one of investments
1,2,3 or at least two investments from 4,5,6
2 y1 + 2 y2 + 2 y3 + y4 + y5 + y6 ³ 2
Modeling of Disjunctions (OR)
Ú [Ai x £ bi ]
iÎD
• Big-M formulation (e.g. M=1000)
OR
1
2
Ai x £ b + M (1 - yi ) "i Î D
åy
iÎD
i
=1
• Non-trivial to choose best M-value (as small as possible). Can have a big
impact on LP-relaxation
if yi = 1 ® Ai x £ b
if yi = 0 ® Ai x £ b + M
Example
x2
[x1 - x2 £ -2]Ú [- x1 + x2 £ -2]
0 £ x1 , x2 £ 4
Big-M (M=6)
x1 - x2 £ -2 + M (1 - y1 )
- x1 + x2 £ -2 + M (1 - y2 )
y1 + y2 = 1
0 £ x1 , x2 £ 4
y1 , y2 Î {0,1}
4
3
2
1
0
0
1
2
3
4
x1
Mixed Integer Linear
Programming:
Branch-and-Bound
MILP Techniques
• Complete enumeration, "Brute Force"
• Branch and Bound (Land and Doig, 1960; Balas, 1963; Dakin, 1965)
• Cutting Plane Method (Gomory, 1958)
Main principle: „Divide and Conquer“
• Constraint Programming (Hentenryck, 1989)
Solve
subsequent LP-problems
• Heuristic & evolutionary
methods
in a tree structure and add bounds
• Monte Carlo Simulation
• Simulated Annealingto the „children“ based on the
• Genetic Algorithms solution values, trying to restrict the
search space...
Proposition 1
• Let
S = S1 È S 2 È K È S K
• be a decomposition of S into smaller sets, and let
z k = max{cx : x Î S k } k = 1,K , K
• Then
z = max k z k
obj
2
1
• "Divide and Conquer"
4
3
5
Binary Enumeration Tree (three 0-1 variables)
S
x1 = 0
x1 = 1
S0
S1
x2 = 0
x2 = 1
S00
S01
S10
S11
x3 = 0
S000
S001
S010
S011
S100
8 solution candidates
S101
S110
S111
Proposition 2
• Let
S = S1 È S 2 È K È S K
• be a decomposition of S into smaller sets, and let
z k = max{cx : x Î S k } k = 1, K, K
z k = upper bound for z k
„Relaxed (no integers) solution“
z = lower bound for z k
„Best solution found“
k
• Then
z = max k z k
z = max k z
k
Example of Bounds
• Many methods relax first the problem, e.g. treat the integer variables as
continuous.
• Relaxed solution
• Dual bound
• lower bound (minimizing)
• upper bound (maximizing)
• Best integer solution found
• Primal bound
• upper bound (minimizing)
• lower bound (maximizing)
• (Sub)problem solved: UB = LB or UB-LB ≤ ε