IBM Advanced Analytics Summit – April 2011
Efficient Modeling with CPLEX
Studio
© 2011 IBM Corporation
Outline
 Using sets and structured types (tuples)
– Sorted and ordered sets
 Data initialization
 Database transformations
– Generic array transformations
 Sparse indexing
– Assignment model
 Output
 Conditional constraints
 Logical and nonlinear expressions
 Slicing and performance
 Handling infeasibility
2
© 2011 IBM Corporation
Sets
 The fundamental types in OPL are
–float
–int
–string
 You can declare data consisting of sets of any of these
types using one of two syntaxes
–setof(string) Products = ...;
–{int} ProductIDs = ...;
 OPL has set operators
–setof(string) B = A union C;
–setof(int) F = D inter E;
–setof(string) G = A diff B;
3
© 2011 IBM Corporation
Using sets: Order within Sets

Sets are ordered (default) or sorted

sorted sets keep the elements in their natural order
Use ord(SetName, Elem) to get position within any set
setof(string) S1 = {
sorted setof(string)
tuple pair {
int
v;
string s;
};
setof(pair) P1 = { <
setof(pair) P2 = { <

ord(S1,s), s > | s in S1 };
ord(S2,s), s > | s in S2 };
Results
P1 = {
<v:0,s:“C">,
<v:1,s:“B">,
<v:2,s:“A">
}
4
"C", "B", "A" };
// Ordered set
S2 = { "C", "B", "A" };
// Sorted set
P2 = {
<v:0,s:“A">,
<v:1,s:"B">,
<v:2,s:“C">
}
© 2011 IBM Corporation
Ordered and Sorted Simple Sets
 Use sorted sets when the creation order is not relevant or the
sort order is important
Example:
{int} s1 = {3,5,1}; // Ordered set
{int} s2 = {4,2};
// Ordered set
{int} orderedS = s1 union s2;
sorted {int} sortedS = s1 union s2;
execute {
writeln("ordered union = ", orderedS);
writeln("sorted union = ", sortedS);
}
The output is:
ordered union = {3 5 1 4 2}
sorted union = {1 2 3 4 5}
5
© 2011 IBM Corporation
Structured types (Tuples)
 OPL allows you to create structured data types, called tuples
tuple pair {
int
v;
string s;
};
 Data for a set of these types
setof(pair) S = { <5, "A">, <6, "B">, <5, "C"> };
 Create variables and arrays over these sets
dvar int x[S] in 0..1;
6
© 2011 IBM Corporation
Sorted and Ordered Tuple Sets
 When a tuple set uses no keys, the entire tuple, is taken into account for sorting.
 For tuple sets with keys, sorting takes place on all keys in their order of declaration.
7
© 2011 IBM Corporation
Example of Sorted and Ordered Tuples
tuple person {
string firstname;
The
string lastname;
string nickname;
}
tuple personKeys {
key string firstname;
key string lastname;
string nickname;
}
{person} devTeam = {
<"David", “Atkinson", “Dave">,
<"David", "Doe", "Skinner">,
<"Gregory", "Simons", “Greg">,
<"David", "Smith", "Lewis">,
<"Kevin", "Morgan", "Gil">,
<"Gregory", "McNamara ", "Mac">
};
sorted {personKeys} sortedDevTeam =
{
<i,j,k> | <i,j,k> in devTeam
};
8
sorted devTeam is
<"David", “Atkinson", “Dave">
<"David", “Doe", “Skinner">
<"David", "Smith", “Lewis">
<"Greg", "McNamara", "Mac">
<"Gregory", "Simons", “Greg">
<"Kevin", "Morgan", "Gil">
© 2011 IBM Corporation
Converting between tuples and sets
 Origin/destination pairs can be represented as
tuple Pair { string o; string d; };
 A set of pairs would be represented as
setof(Pair) odpairs =
{<"MIA","EWR">,<"MIA","SFO">,
<"SFO","BOS">,<"EWR","SFO"> };
 The origins and destinations can be computed as
setof(string) origins = { o | <o,d> in odpairs};
setof(string) destinations =
{d | <o,d> in odpairs};
 The set of all cities can be computed as
setof(string) Cities = origins union
destinations;
9
© 2011 IBM Corporation
Data initialization: declarative syntax
 Simple values
float c = i / sqrt(j);
 Sets
setof(int) indices = { i | i in range :
x[i] > 0 };
 Arrays
float coef[i in
float flow[i in
(i == src)
(i == dst)
1..5] = 1.0/i;
Nodes] =
? –total :
? total : 0.0;
© 2011 IBM Corporation
OPL data
 The power of OPL relies on high-level data
– Sparse data make OPL able to solve large models
– OPL can read data and write solutions directly to/from spreadsheets and databases
– OPL data can help summarize solution results
© 2011 IBM Corporation
Database transformations
 Data records in a database
– Keys uniquely identify a record
– Other data is associated with the keys
 OPL reads data from databases as sets of records
– A record is a tuple
– Each record contains the keys and associated data
 OPL can manage data as arrays or tuples
© 2011 IBM Corporation
Database example
© 2011 IBM Corporation
Read data
Model File
Data File
setof(string) People = ...;
setof(string) Jobs = ...;
DBConnection dbassign
(“access", “dbassign.mdb");
tuple Triples {
string
person;
string
job;
int
benefit;
};
People from DBRead(dbassign,
setof(Triples) tripset = ...;
"select Person from Dense");
Jobs from DBRead (dbassign,
"select Job from Dense");
tripset from DBRead (dbassign,
"select Person, Job, Benefit
from Dense");
© 2011 IBM Corporation
Transform data
 The set tripset contains
<"Bob"
"Plumber"
<"Bob"
"Electrician" 20>
<"Bob"
"Carpenter"
<"John" "Plumber"
10>
15>
18>
<"John" "Electrician" 14>
<"John" "Carpenter"
12>
<"Sam"
"Plumber"
11>
<"Sam"
"Electrician" 14>
<"Sam"
"Carpenter"
17>
 Transform to 2-D Array:
int value[People][Jobs] =
[ i : [ j : v ] | <i,j,v> in tripset ];
© 2011 IBM Corporation
Simple Assignment Model
dvar int x[People][Jobs] in 0..1;
maximize sum (p in People, j in Jobs) value[p][j]*x[p][j];
subject to {
forall (p in People)
sum (j in Jobs) x[p][j] == 1;
forall (j in Jobs)
sum (p in People) x[p][j] == 1;
};
execute {
for (i in People) for (j in Jobs) {
if (x[i][j] > 0) writeln(i, " assigned to ", j);
}
};
 Solution
Optimal solution found with objective: 55
Bob assigned to Electrician
John assigned to Plumber
Sam assigned to Carpenter
© 2011 IBM Corporation
Problem with this model
It will not scale
–1000 people, 1000 jobs
yields 1,000,000 variables!
Typically, data is sparse
–With 1000 people, each
person can probably only do
some of the jobs
Take advantage of
sparse data with OPL
© 2011 IBM Corporation
Read sparse data and arrays
Model File
tuple Pairs {
string
person;
string
job;
};
setof(Pairs) assign = ...;
int value[assign] = ...;
Data File
DBConnection dbassign
(“access", “dbassign.mdb");
assign, value from
DBRead(dbassign, "select
Person, Job, Benefit from
Dense");
© 2011 IBM Corporation
Sparse Assignment Model
dvar int x[assign] in 0..1;
maximize
sum (pj in assign) value[pj] * x[pj];
subject to {
forall (i in People)
sum (<i,j> in assign) x[<i,j>] == 1;
forall (j in Jobs)
sum (<i,j> in assign) x[<i,j>] == 1;
};
execute {
Slicing efficiently matches
for (k in assign) {
jobs and people
if (x[k] > 0)
writeln(k.person, " assigned to ", k.job);
}
};
© 2011 IBM Corporation
Modeling with Dense and Sparse Data
Dense Version
Sparse Version
 Objective Function
 Objective Function
sum (p in People, j in Jobs)
value[p][j]*x[p][j]
sum (pj in assign)
value[pj] * x[pj]
 Constraint
 Constraint
forall (p in People)
sum (j in Jobs)
x[p][j] == 1;
forall (i in People)
sum (<i,j> in assign)
x[<i,j>] == 1;
 Output
 Output
for (i in People)
for (j in Jobs) {
if (x[i][j] > 0)
writeln(i,
" assigned to ",
j);
}
for (k in assign) {
if (x[k] > 0)
writeln(k.person,
" assigned to ",
k.job);
}
© 2011 IBM Corporation
Read data as sparse tuple set
Model File
tuple Triples {
key string
key string
int
person;
job;
benefit;
};
setof(Triples) tripset = ...;
Data File
DBconnection dbassign ("access",
"dbassign.mdb“);
tripset from DBread(dbassign,
"select Person, Job, Benefit from
Dense");
setof(string) People =
{i | <i,j,b> in tripset};
setof(string) Jobs =
{j | <i,j,b> in tripset};
© 2011 IBM Corporation
Sparse Assignment Model with tuple set
dvar int x[tripset] in 0..1;
Only tuple key
elements are
needed to index
into array
maximize
sum (k in tripset) k.benefit * x[k];
subject to {
forall (i in People)
sum (<i,j,b> in tripset) x[<i,j>] == 1;
forall (j in Jobs)
sum (<i,j,b> in tripset) x[<i,j>] == 1;
};
execute {
for (k in tripset) {
if (x[k] > 0)
writeln(k.person, " assigned to ", k.job);
}
};
© 2011 IBM Corporation
Solution output
 Create set of results
setof(Pairs) result =
{ <i,j> | <i,j> in assign : x[<i,j>] == 1 };
 Write to a database (in data file!)
DBExecute(dbassign,
"create table Assign (Person string, Job string)");
result to DBupdate(dbassign,
"insert into Assign (Person,Job) values (?,?)");
 Output to a file (in model file)
execute {
var fp = new IloOplOutputFile("c:\\temp\\sol.out");
fp.writeln(result);
fp.close();
};
© 2011 IBM Corporation
Formatting output
 Output in a model has no format control
– Previous code creates output (9 pairs per line!)
{<"P000519" "J000802"> <"P000779" "J000209"> <"P000999" "J000996">...
<"P000149" "J000804"> <"P000448" "J000545"> <"P000725" "J000873">...
...
 Must iterate over individual items to get formatting control
execute {
var fp = new IloOplOutputFile("c:\\temp\\sol2.out");
for (p in result) {
fp.writeln(p.person, " ", p.job);
}
fp.close();
};
– Creates output
P000519 J000802
P000779 J000209
P000999 J000996
...
© 2011 IBM Corporation
Conditional loops
 Standard Loop
forall (i in 1..10) {
 Conditional Loop
forall (i in 1..10 : mydata[i] <= 10) {
 Using Sets
– When same loop needed more than once
setof(int) loopdata =
{ i | i in 1..10 : mydata[i] <= 10 };
forall (i in loopdata) {
© 2011 IBM Corporation
Data driven relaxations
 Use data to conditionally state a collection of constraints
– Data declarations
int includeBundleConstraint = ...;
– Conditional constraints
if (includeBundleConstraint == 1) {
forall (i in S1) {
sum (j in S2) x[j] <= b[i];
};
}
else { …
}
© 2011 IBM Corporation
Logical and nonlinear expressions
 OPL linearizes key nonlinear expressions
– min/minl, max/maxl: min/max over a list
– abs: absolute value
– Piecewise linear expressions
 OPL supports basic logical expressions
– AND (&&), OR (||), NOT (!)
– Imply (=>), Equivalent (==), Different (!=)
– These can combine multiple constraints
© 2011 IBM Corporation
Examples of logical and nonlinear expressions
 Semi-continuous order quantity:
(x == 0) || (x >= 100);
 Logical indicators
(x == 0) => (y == 0);
Zeros == sum (i in Index) (x[i] == 0);
 Maximum deviation from a target value
deviation ==
max (i in Index) abs(x[i]-target);
Benefits
–Model is easier to develop and maintain
–Can eliminate some big-M bounds
© 2011 IBM Corporation
Pattern Matching via Slicing
setof(string) products = {"bands", "coils"};
setof(string) steelTypes =
{"lowCarbon", "mediumCarbon", "highCarbon"};
range steelPieceID = 1..100000;
tuple steelPiece {
int id;
// steelPieceID
string p;
// Product
string t;
// Type
};
tuple band {
int id;
// steelPieceID
float w;
// width
float z;
// weight
};
setof(steelPiece) steelPieces = ...;
setof(band) bands = ...;
© 2011 IBM Corporation
Pattern Matching via Slicing (2)
 OPL Data
steelPieces = {
tuple steelPiece {
int id;
string p;
string t;
};
<1 bands lowCarbon>,
<2 coils mediumCarbon>,
<4 coils mediumCarbon>,
<5 bands lowCarbon>,
<6 bands lowCarbon>,
… };
bands = {
tuple band {
int id;
float w;
float z;
};
<2 82 178>,
<3 55 49>,
… };
© 2011 IBM Corporation
Slicing and efficiency
 Arrays of Sets
tuple steelPiece {
int id;
string p;
string t;
};
–This can be inefficient
setof(int) piecesPerProduct[p in products] =
{ i | sp in steelPieces, i in steelPieceID :
sp.p == p & sp.id == i };
–Using pattern matching is more elegant
setof(int) piecesPerProduct[p in products] =
{ id | <id,p,t> in steelPieces };
 Use slicing rather than explicit equality conditions
© 2011 IBM Corporation
Slicing in constraints
 When using forall and sum statements
– Inelegant inner loop
forall (<id,p,t> in steelPieces)
forall (<idB,wi,we> in bands :
idB == id && wi <= 50)
– More elegant by value
forall (<id,p,t> in steelPieces)
forall (<id,wi,we> in bands : wi <= 50)
 OPL internally optimizes the first loop to the second form
© 2011 IBM Corporation
Summary
 Create structured types
– Take advantage of data sparsity
 Use Slicing
 Read Efficient Modeling in OPL
– ILOG white paper provides more details on how to compress models and data to
improve runtime performance and memory consumption
– http://www-01.ibm.com/software/sw-library/en_US/detail/U838284X67897H19.html
© 2011 IBM Corporation