ST 810, Advanced computing
Eric B. Laber & Hua Zhou
Department of Statistics, North Carolina State University
February 25, 2013
“One thing a computer can do that most humans can’t is be
sealed up in a cardboard box and sit in a warehouse.”
—The epic of Gilgamesh
Roadmap
We will parallel the LP lecture:
I
Quadratic programming
I
Canonical example
I
quadprog and LowRankQP in R
I
Hands on examples
I
Quick note on QCQPs and SOCPs
Quadratic programming
I
An optimization problem with a quadratic objective and linear
constraint functions is called a quadratic program (QP)
minx
I
I
1 |
2 x Px
+ c |x
Ax = b
Gx h
p
P ∈ S+
, A ∈ Rm × Rp , G ∈ Rq × Rp
QPs are convex optimization problems (Why?)
QP in standard form and positive definiteness
I
I
Some software (e.g., solveQP in R) requires the QP be
entered in the ‘standard’ form:
1 |
2 x Px
minx
− c |x
Gx h
minx
1 |
2 x Px
How can we convert
+ c |x
Ax = b
Gx h
into standard form?
I
Except in special circumstances conversion to standard format
will ensure that P is not strictly positive definite.
PROBLEM: some solvers in R (e.g. solveQP and ipop)
require P be strictly positive definite.
Least squares regression
I
A quadratic program we’re all familiar with
n
X
min
(yi − xi| β)2
β
I
i=1
Let Y , (y1 , . . . , yn ) and X = (xij ) then we can rewrite OLS
as
min β | XX | β − 2Y | X β + Y | Y
β
I
How can we write this in standard form?
Least squares regression cont’d
I
Constrained coefficient regression
minβ β | XX | β − 2YX | β
lj ≤ βj ≤ uj j = 1, . . . , p
I
Constraints often driven by scientific or practical
considerations
I
How can we write this in the form:
minx
1 |
2 x Px
+ c |x
Ax = b
Gx h
quadprog in R
I
The quadprog package in R contains a QP solver that
requires problems be entered in standard form
I
I
I
I
Widely recommended on R forums
Uses Goldfarb and Idnani (1983) algorithm which requires P
be strictly positive definite
Algorithm fast and stable but possibly of limited use
Canonical unmotivated example
min x12 + 2x22 + 4x32 − x1 − x2 + 5x3
x1 + x3 ≤ 1
x1 ≥ 5
x2 ≤ 0
quadprog documentation
2
solve.QP
solve.QP
Solve a Quadratic Programming Problem
Description
This routine implements the dual method of Goldfarb and Idnani (1982, 1983) for solving quadratic
programming problems of the form min( dT b + 1/2bT Db) with the constraints AT b >= b0 .
Usage
solve.QP(Dmat, dvec, Amat, bvec, meq= , factorized=FALSE)
Arguments
Dmat
matrix appearing in the quadratic function to be minimized.
dvec
vector appearing in the quadratic function to be minimized.
Amat
matrix defining the constraints under which we want to minimize the quadratic
function.
bvec
vector holding the values of b0 (defaults to zero).
meq
the first meq constraints are treated as equality constraints, all further as inequality constraints (defaults to 0).
factorized
logical flag: if TRUE, then we are passing R
matrix D in the argument Dmat.
1
(where D = RT R) instead of the
Value
a list with the following components:
solution
vector containing the solution of the quadratic programming problem.
value
scalar, the value of the quadratic function at the solution
unconstrained.solution
vector containing the unconstrained minimizer of the quadratic function.
iterations
vector of length 2, the first component contains the number of iterations the
algorithm needed, the second indicates how often constraints became inactive
after becoming active first.
Lagrangian
vector with the Lagragian at the solution.
iact
vector with the indices of the active constraints at the solution.
References
D. Goldfarb and A. Idnani (1982). Dual and Primal-Dual Methods for Solving Strictly Convex
Quadratic Programs. In J. P. Hennart (ed.), Numerical Analysis, Springer-Verlag, Berlin, pages
226–239.
D. Goldfarb and A. Idnani (1983). A numerically stable dual method for solving strictly convex
quadratic programs. Mathematical Programming, 27, 1–33.
quadprog in R cont’d
library(’quadprog’);
##
## min x1^2 +2x2^2 + 4x3^2 - x1 - x2 + 5x3
##
x1 + x3 <= 1
##
x1 >= 5
##
x2 <= 0
##
P = 2*diag (c (1, 2, 4));
d = c (-1, -1, 5);
A = matrix (0, nrow=3, ncol=3);
A[1,] = c(-1, 0, -1);
A[2,] = c( 1, 0, 0);
A[3,] = c( 0, -1, 0);
b = c(-1, 5, 0);
xHat = solve.QP (P, -d, t(A), b)$solution;
Stat ex: LASSO
I
I
I
Observe iid sample {(xi , yi )}ni=1 where x ∈ Rp and y ∈ R
Simultaneously select and estimate a model:
P
minβ Pni=1 (yi − xiT β)2
p
j=1 |βj | ≤ s
Challenge: how can we write this in a form that can be used
by solveQP?
Stat ex: LASSO cont’d
I
To use solveQP we have to employ 2p inequality constraints!
I
Ex. |β1 | + |β2 | ≤ s is equivalent to
β1 + β2
−β1 + β2
β1 − β2
−β1 − β2
≤ s
≤ s
≤ s
≤ s
lassoEnum = function (x, y, s){
require ("quadprog");
n = dim(x)[1];
p = dim(x)[2];
P = 2*t(x)%*%x;
d = 2*t(y)%*%x;
A = as.matrix (expand.grid (rep (list (c(-1,1)), p)));
b = rep (-s, 2^p);
bHat = solve.QP (P, d, -t(A), b)$solution;
return (round(bHat, digits=5));
}
Rewriting the LASSO more compactly
I
We can rewrite the LASSO optimization problem as
minβ + ,β −
(β + − β −1 )| XX | (β + − β − ) − 2YX | (β + − β − )
P
p
+
−
j=1 (βj + βj ) ≤ s
β + 0, β − 0
I
This has 3p constraints compared with 2p
I
If we try to write this in standard form P will not be strictly
positive definite. Define Q , (I , −I ) ∈ Rp × R2p , and
R , (I , I ) ∈ Rp × R2p . In std form:
minβ̃ β̃ | Q | XX | Q β̃ − 2YX | Q β̃
−R β̃ −s
β̃ 0
LowRankQP
I
LowRankQP uses an interior point method and can handle
positive semidefinite P
I
LowRankQP requires problems to be of the form:
minx
1 |
|
2 x VV x
+ c |x
Ax = b
l x u
I
How can we rewrite the LASSO in this form?
LowRankQP
lassoQP = function (x, y, s){
require ("LowRankQP");
n = dim(x)[1];
p = dim(x)[2];
Vn = x%*% cbind (cbind (diag(p), -diag(p)), 0);
yx = apply(sweep (x, MARGIN=1, -y, ’*’), 2, sum);
Zn = c (2*yx, -2*yx, 0);
bOls = lm.fit (x, y)$coefficients;
u = c(abs(bOls), abs(bOls), sum(abs(bOls)));
A = matrix (c(rep (1, 2*p), 1), nrow=1);
b = c(min(s, sum(abs(bOls))));
soln = LowRankQP (sqrt(2)*t(Vn), Zn, A, b, u, method="LU");
return (round(soln$alpha[1:p] - soln$alpha[(p+1):(2*p)], digits=5));
}
Classic ex: Markowitz portfolio optimization
I
Suppose there are p assets held over a fixed period and let βj
denote the amount invested in th jth asset.
I
Let R be a p-variate random variable denoting the relative
price change of each asset. We assume R has mean µR and
covariance ΣR .
I
Markowitz (1952) proposed choosing the asset allocation
which solves
minβ β | ΣR β
β | µR ≥ `
1| β = 1
β0
Classic ex: Markowitz portfolio optimization: data
sixMonthData = function (oldDate=as.Date("2012-08-01")){
require ("quantmod");
symlst = read.table ("top100Symb.txt", header=F)[,1];
env = new.env ();
prices = NULL;
colNames = NULL;
for (sym in symlst){
tc = tryCatch (getSymbols (sym, env=env, src=’yahoo’,
from=oldDate), error=identity);
if (inherits (tc, "error")){
next;
}
opVals = as.numeric (Op (with (env, eval (parse (text=sym)))));
if (length(prices) == 0){
prices = cbind (prices, opVals);
colNames = c(colNames, sym);
}
else if (dim(prices)[1] == length(opVals)){
prices = cbind (prices, opVals);
colNames = c(colNames, sym);
}
}
colnames(prices) = colNames;
return(prices);
}
Classic ex: Markowitz portfolio optimization
makeFortune = function (prices, l){
require ("quadprog");
p = dim(prices)[2];
strt = prices[1,];
prices = sweep (prices, MARGIN=2, strt, ’-’);
prices = sweep (prices, MARGIN=2, strt, ’/’);
D = cov(prices);
mu = apply (prices, 2, mean);
d = rep (0, p);
A = rbind (rep (1, p), mu);
A = rbind (A, diag(p));
b = c(1, min (l, max(mu)-.05), rep(0, p));
soln = solve.QP (2*D, d, t(A), b, meq=1)$solution;
names (soln) = colnames(prices);
return (round(soln, 3));
}
Markowitz usage
> source ("markowitz.R")
> prices = sixMonthData ()
> makeFortune (prices, -0.15)
ATVI ADBE AKAM ALXN ALTR
0.000 0.000 0.000 0.000 0.000
BIIB BRCM CHRW CELG CERN
0.000 0.000 0.085 0.000 0.000
DLTR EBAY EXPE EXPD ESRX
0.074 0.000 0.000 0.000 0.000
GOOG GMCR HSIC ILMN INFY
0.000 0.000 0.000 0.012 0.000
MXIM MCHP MSFT NTAP NFLX
0.000 0.000 0.008 0.000 0.000
QGEN QCOM ROST SNDK SHLD
0.000 0.000 0.000 0.000 0.032
VRSN VRTX VMED WCRX WYNN
0.039 0.000 0.000 0.000 0.000
>
AMZN
0.000
CHKP
0.000
FFIV
0.000
INTC
0.095
NWSA
0.000
SIAL
0.000
XLNX
0.000
AMGN
0.221
CSCO
0.000
FAST
0.000
INTU
0.000
NIHD
0.000
SIRI
0.000
YHOO
0.025
APOL
0.000
CTXS
0.000
FSLR
0.000
ISRG
0.000
NVDA
0.000
SPLS
0.026
AAPL
0.000
CTSH
0.000
FISV
0.000
KLAC
0.000
ORLY
0.109
SBUX
0.000
AMAT
0.000
COST
0.000
FLEX
0.000
LRCX
0.026
ORCL
0.000
SRCL
0.119
ADSK
0.000
CTRP
0.000
FLIR
0.000
LIFE
0.000
PCAR
0.000
SYMC
0.000
BIDU
0.000
DELL
0.000
GRMN
0.000
LLTC
0.000
PAYX
0.000
TEVA
0.060
BBBY
0.034
XRAY
0.000
GILD
0.036
MRVL
0.000
PCLN
0.000
URBN
0.000
QCQP
I
An optimization problem with both a quadratic objective and
quadratic constraint functions is called a Quadratically
Constrained Quadratic Program (QCQP)
minx
1 |
|
2 x Px + c x
|
1 |
2 x Pi x + c i ≤
hi i = 1, . . . , m
Ax = b
I
p
P, P1 , . . . , Pm ∈ S+
I
This is a convex optimization problem
I
R does not a QCQP solver, but packages exist to call third
party libraries, e.g., CPLEX and MOSEK
SOCP
I
A second-order cone program (SOCP) is an optimization
problem of the form
minx
f |x
||Ai x + bi ||2 ≤ ci| x + di , i = 1, . . . , m
Fx = g
I
Constraint requires (Ax + b, c | x + d) belongs to the
second-order cone in Rk+1
I
This is a convex optimization problem
I
Again this requires third party software, e.g. CPLEX or
MOSEK
CPLEX
I
Optimization library
I
I
I
I
I
I
Solves LPs, QPs, QCQPs, SOCPs
Allows integer variables, specially ordered sets, logical
constraints, and much more!
Capable of handling large problems (thousands of variables and
constraints)
Stable, extremely flexible, lots of customization possible
Automatic parallel execution
Interfaces with
I
I
C/C++, Java, python, .NET
Runs on linux, mac (sort of), solaris, and windows
I
Third part wrapper written for R, not fully functional
I
Free under academic license! (IBM academic initiative.)
CPLEX cont’d
I
More than a callable library
I
I
I
I
I
Interactive optimizer good for exploration and debugging
CPLEX concert technology is a collection of classes that makes
model building easy and intuitive
Based on meta-programming and operator-overloading
Ex. add the constraint βj ≤ 4 in C++ using concert
technology model.add(beta[j] <= 4.0)
Ex. Declare a binary variable that is one if X1 ≥ 0 using
concert technology z = (x[1] >= 0), this is also called a
logical variable
Discrete optimization
I
A mixed integer optimization problem has the form
minx,z
x ∈ Rp , z ∈ Zq
I
f (x, z)
A(x, z) = b
G (x, z) h
Why are these so hard?
I
I
I
Discreteness destroys smoothness
Have to examine or ‘rule out’ all feasible values (there are
usually a lot of them!)
A standard strategy is to solve a series of relaxed (continuous)
problems and prove the difference between the relaxed solution
and the integer solution is negligible.
Warm-up: solving an LP
I
To start we’re going to solve the following LP
minx
x1 − 2x2 − x3
x1 + x3 ≤ 1
x1 + x2 ≤ 3
0 ≤ x1 ≤ 3
−5 ≤ x2
I
Login to the teaching machine
I
Type cplex and hit enter
I
WAIT!
Interactive solver
CPLEX> enter
Enter name for problem: st810
Enter new problem [’end’ on a separate line terminates]:
minimize x1 - 2 x2 - x3
subject to x1 + x3 <= 1.0
x1 + x2 <= 3
bounds
0 <= x1 <= 3
-5 <= x2
end
CPLEX> optimize
Tried aggregator 1 time.
LP Presolve eliminated 2 rows and 3 columns.
All rows and columns eliminated.
Presolve time = 0.00 sec. (0.00 ticks)
Dual simplex - Optimal: Objective = -7.0000000000e+00
Solution time =
0.00 sec. Iterations = 0 (0)
Deterministic time = 0.00 ticks (2.51 ticks/sec)
CPLEX> display solution variables Variable Name
Solution Value
x2
3.000000
x3
1.000000
All other variables in the range 1-3 are 0.
Mystery function
IloCplex cplex (model);
if (!cplex.solve ()){
env.end ();
return false;
}
IloNumArray cpxBeta (env);
cplex.getValues (cpxBeta, b);
beta.resize (p);
for (int j = 0; j < p; ++j){
beta (j) = cpxBeta[j];
}
env.end ();
#ifndef _MYSTERY_HPP
#define _MYSTERY_HPP
#include
#include
#include
#include
#include
#include
#include
#include
#include
#include
<ilcplex/ilocplex.h>
<boost/numeric/ublas/vector.hpp>
<boost/numeric/ublas/vector_proxy.hpp>
<boost/numeric/ublas/matrix.hpp>
<boost/numeric/ublas/triangular.hpp>
<boost/numeric/ublas/lu.hpp>
<boost/numeric/ublas/io.hpp>
<boost/random.hpp>
<boost/random/normal_distribution.hpp>
<iostream>
namespace misteree{
ILOSTLBEGIN
namespace ublas = boost::numeric::ublas;
bool mysteryFn (const ublas::vector<double>& y,
const ublas::matrix<double>& x,
ublas::vector<double>& beta){
IloEnv env;
IloModel model (env);
try{
int n = x.size1 ();
int p = x.size2 ();
IloNumVarArray b (env);
for (int j = 0; j < p; ++j){
b.add (IloNumVar (env, -IloInfinity, +IloInfinity));
}
IloExpr objExpr (env);
for (int i = 0; i < n; ++i){
IloExpr xtb (env);
for (int j = 0; j < p; ++j){
xtb += x (i,j)*b[j];
}
objExpr += IloAbs (y(i) - xtb);
xtb.end ();
}
IloObjective obj (env, objExpr, IloObjective::Minimize);
model.add (obj);
objExpr.end ();
}
catch (IloException& e){
cerr << e << endl;
env.end ();
return false;
}
catch (...){
cerr << "Unknown exception caught,\n";
env.end ();
return false;
}
return true;
}
}
#endif // _MYSTERY_HPP
Anatomy of a CPLEX problem
1. Create cplex environment, this performs memory management
and its where all subsequent cplex objects ‘live’
IloEnv env;
2. Create a cplex model object, this will store the objective
function, variables, and constraints. Constructor takes the
environment as an argument.
IloModel model (env);
3. Create an (empty) array to store variables
IloNumVarArray b (env);
4. Populate the array with variables
for (int j = 0; j < p; ++j){
b.add (IloNumVar (env, -IloInfinity, +IloInfinity));
}
Anatomy of a CPLEX problem, cont’d
5. Create an cplex expressions store functions of variables and
constants, these are used to build an objective function and
add constraints.
IloExpr objExpr (env);
6. Lets create
objective function with the sum of absolute
Pan
n
residuals i=1 |yi − xi| β|
for (int i = 0; i < n; ++i){
IloExpr xtb (env);
for (int j = 0; j < p; ++j){
xtb += x (i, j)*b[j];
}
objExpr += IloAbs (y(i) - xtb);
xtb.end ();
}
IloObjective obj (env, objExpr, IloObjective::Minimize)
model.add (obj);
objExpr.end ();
Anatomy of CPLEX problem, cont’d
7. Add model constraints, let’s add the constraint that ||β||2 ≤ λ
IloExpr bNorm (env);
for (int j = 0; j < p; ++j){
bNorm += b[j]*b[j];
}
model.add (bNorm <= lambda);
bNorm.end ();
8. Create a cplex objective
IloCplex cplex (model);
9. Optimize!
cplex.solve ();
10. Extract solution
IloNumArray bCpx (env);
cplex.getValues (cpxBeta, b);
Our first MIP: L0 regression
I
I
We observe {(yi , xi )}ni=1 with x ∈ Rp and y ∈ R
For any s ∈ {0, 1, . . . , p} define
P
βbns , arg minβ∈Rp ni=1 (yi − xi| β)2
||β||0 ≤ s,
I
||β||0 ,
Pp
j=1 1βj 6=0 .
This is a discrete (combinatorial) optimization problem (why?)
Our first MIP: L0 regression, cont’d
I
The L0 norm can be modeled using logical constraints in
CPLEX:
I
I
I
I
I
&& Logical AND
myExp += ((b[0] >= 0) && (b[1] <= -3))
adds 1β0 ≥0, β1 ≤3 to the expression objExpr
|| Logical OR
myExpr += ((b[0] <= 0) || (b[1] <= 0))
adds 1min(β0 ,β1 )≤0 to the expression
== equivalence
z = (b[0] == 0)
sets the variable z = 1β0 =0
! = different from
z = (b[0] != 0)
sets the variable z = 1β0 6=0
=> Logical implies
((b[0] >= 7) => (b[1] <= 10))
if β0 ≥ 7 then β1 ≤ 10
Components of the L0 regression
1. Setup an environment and model object
IloEnv env;
IloModel model (env);
2. Create and populate coefficient vector
IloNumVarArray b (env);
for (int j = 0; j < p; ++j){
b.add (IloNumVar (env, -IloInfinity, +IloInfinity));
}
Components of the L0 regression cont’d
3. Create expression to store the objective
IloExpr objExpr (env);
for (int i = 0; i < n; ++i){
IloExpr xtb (env);
for (int j = 0; j < p; ++j){
xtb += x(i, j)*b[j];
}
xtb.end ();
}
IloObjective obj (env, objExpr, IloObjective::Minimize)
objExpr.end ();
Components of the L0 regression cont’d
4. Add the constraint ||β||0 ≤ s, do not include intercept
IloExpr l0Norm (env);
for (int j = 1; j < p; ++j){
l0Norm += (b[j != 0);
}
model.add (l0Norm <= s);
l0Norm.end ();
5. Extract model and set tuning parameters for the optimization
IloCplex cplex (model);
cplex.setParam (IloCplex::MIPDisplay, 2);
cplex.setParam (IloCplex::Threads, 8);
Putting it all together
#ifndef _L0_HPP
#define _L0_HPP
/**
* @file l0.hpp
* @brief Computes L0 regression.
* @author Eric B. Laber
* @note Example for st810
*/
#include <ilcplex/ilocplex.h>
#include <boost/numeric/ublas/vector.hpp>
#include <boost/numeric/ublas/vector_proxy.hpp>
#include <boost/numeric/ublas/matrix.hpp>
#include <boost/numeric/ublas/triangular.hpp>
#include <boost/numeric/ublas/lu.hpp>
#include <boost/numeric/ublas/io.hpp>
#include <boost/random.hpp>
#include <boost/random/normal_distribution.hpp>
#include <iostream>
namespace l0{
ILOSTLBEGIN
namespace ublas = boost::numeric::ublas;
/**
* @fn l0Reg (const ublas::vector<double>& y,
*
const ublas::matrix<double>& x,
*
const int s,
*
ublas::vector<double>& beta)
* @brief Computes \arg\min_{beta} \sum (yi - xi^{\T}\beta)^2
*
s.t. ||\beta||_0 \le s
* @param y response vector
* @param x design matrix, first column should be ones
* @param s number of nonzero components
* @param beta return value
*/
bool l0Reg (const ublas::vector<double>& y,
const ublas::matrix<double>& x,
const int s,
ublas::vector<double>& beta){
if (s <= 0){ // s = 0 corresponds to all zero soln
beta.resize (x.size2 ());
beta.assign (ublas::zero_vector<double> (x.size2 ()));
// Set intercept
for (int i = 0; i < x.size1 (); ++i){
beta(0) += y (i);
}
beta(0) /= static_cast<double> (x.size1());
return true;
}
IloEnv env; // Create cplex environment to store vars
IloModel model (env);
try{
int n = x.size1 ();
int p = x.size2 ();
IloNumVarArray b(env); // Regression coefficients
for (int j = 0; j < p; ++j){
b.add (IloNumVar (env, -IloInfinity, +IloInfinity));
}
// Create squared objective function
IloExpr objExpr (env);
for (int i = 0; i < n; ++i){
IloExpr xtb (env);
for (int j = 0; j < p; ++j){
xtb += x (i, j)*b[j];
}
objExpr += (y(i) - xtb)*(y(i) - xtb);
xtb.end ();
}
IloObjective obj (env, objExpr, IloObjective::Minimize);
model.add (obj);
objExpr.end ();
// Add constraint on L0-norm of b, use logical constraints
IloExpr l0Norm (env);
for (int j = 1; j < p; ++j){ // Don't pen intercept
l0Norm += (b[j] != 0); // 1_{beta_j \ne 0}
}
model.add (l0Norm <= s);
l0Norm.end ();
IloCplex cplex (model);
// // Display MIP progress
cplex.setParam (IloCplex::MIPDisplay, 2);
cplex.setParam (IloCplex::Threads, 8); // parallize
// Attempt to solve
if (!cplex.solve ()){
env.end ();
std::cout << cplex.getStatus () << std::endl;
return false;
Putting all together cont’d
}
// Extract solution
IloNumArray cpxBeta (env);
cplex.getValues (cpxBeta, b);
beta.resize (p);
for (int j = 0; j < p; ++j){
beta (j) = cpxBeta[j];
}
env.end ();
}
catch (IloException& e){
cerr << e << endl;
env.end ();
return false;
}
catch (...){
cerr << "Unknown exception caught,\n";
env.end ();
return false;
}
return true;
}
} // end namespace
#endif // _L0_HPP
I
Including comments and
blank lines this code
took less than 115 lines.
I
Wrapper to call from R is
in l0Wrap.cpp
Calling our L0 regression function
> dyn.load ("l0Wrap.so") # load shared library
> x = matrix (rnorm (10000), nrow=100) # is 100 x 100
> beta = c( rep(1, 5), rep (0, 95)) # true beta has s = 5
> y = x%*%beta + rnorm (100)
> .Call ("l0Regression", y, x, c(4))
Tried aggregator 2 times.
MIQP Presolve eliminated 0 rows and 1 columns.
Aggregator did 99 substitutions.
Number of nonzeros in lower triangle of Q = 4950
Using Approximate Minimum Degree ordering
Total time for automatic ordering = 0.00 sec. (0.10 ticks)
Summary statistics for factor of Q:
Rows in Factor
= 100
Integer space required
= 100
Total non-zeros in factor = 5050
Total FP ops to factor
= 338350
Reduced MIQP has 199 rows, 496 columns, and 594 nonzeros.
Reduced MIQP has 297 binaries, 0 generals, 0 SOSs, and 297 indicators.
Reduced MIQP objective Q matrix has 10000 nonzeros.
Presolve time = 0.01 sec. (1.80 ticks)
Probing time = 0.00 sec. (0.11 ticks)
Clique table members: 99.
MIP emphasis: balance optimality and feasibility.
Calling our L0 regression function
MIP search method: dynamic search.
Parallel mode: deterministic, using up to 8 threads.
Root relaxation solution time = 0.02 sec. (6.81 ticks)
Node
Nodes
Left
Objective
IInf
Best Integer
Cuts/
Best Bound
ItCnt
Gap
0
0
0.0000
99
0.0000
3
*
0+
0
500.1878
0.0000
3 100.00%
0
2
0.0000
99
500.1878
0.0000
3 100.00%
Elapsed time = 0.10 sec. (58.21 ticks, tree = 0.01 MB, solutions = 1)
*
12+
12
492.7080
0.0000
22 100.00%
14
16
11.3554
96
492.7080
0.0000
25 100.00%
*
16
16
integral
0
484.6651
0.0000
124 100.00%
*
17
15
integral
0
370.2290
0.0000
220 100.00%
*
35
25
integral
0
368.9205
3.8354
486
98.96%
*
59
33
integral
0
342.8561
4.9004
955
98.57%
*
60
32
integral
0
257.8631
4.9004
1048
98.10%
*
77
31
integral
0
254.4350
5.6297
1536
97.79%
223
31
60.5994
82
254.4350
5.6297
3238
97.79%
383
31
135.4747
73
254.4350
5.6297
4468
97.79%
*
824+
38
252.6468
5.8167
7620
97.70%
1330
38
40.2703
58
252.6468
5.8167
11252
97.70%
Elapsed time = 3.29 sec. (1926.53 ticks, tree = 0.01 MB, solutions = 13)
* 1761
51
integral
0
250.8679
5.8167
13295
97.68%
* 1768
53
integral
0
249.7385
6.0932
13417
97.56%
* 1771
51
integral
0
248.9982
6.0932
13478
97.55%
* 1773
50
integral
0
167.0022
6.0932
13574
96.35%
* 1777
50
integral
0
166.9860
6.0932
13736
96.35%
Calling our L0 regression function
*
*
*
1780
2427+
2791
10747
50
59
62
142
integral
0
integral
85.9991
0
78
166.9269
161.4303
91.6108
91.6108
6.0932
6.6431
7.8559
47.2667
Root node processing (before b&c):
Real time
=
0.08 sec. (54.28 ticks)
Parallel b&c, 8 threads:
Real time
=
5.17 sec. (3376.36 ticks)
Sync time (average)
=
0.04 sec.
Wait time (average)
=
0.00 sec.
-----------Total (root+branch&cut) =
5.25 sec. (3430.64 ticks)
$beta
[1] 0.7748981 1.0455677 1.0476337 1.0650337 0.8995803
[8] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[15] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[22] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[29] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[36] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[43] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[50] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[57] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[64] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[71] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[78] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[85] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[92] 0.0000000 0.0000000 0.0000000 0.0000000 0.0000000
[99] 0.0000000 0.0000000
13831
18610
20735
35892
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
96.35%
95.88%
91.42%
48.40%
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
0.0000000
Enumeration
I
I
I
In some problems enumerating all feasible integer values is of
interest (e.g., see Laber et. al. 2013; Ryon and Kosorok 2013)
CPLEX 11.0 and has solution-pool generation capabilities,
this will generate all feasible integer values that meet
user-supplied criteria
As an example, let’s find feasible solutions to the L0 regression
problem that are within 10% of the optimal solution
1. Tell CPLEX we want to generate all feasible solutions
cplex.setParam (IloCplex::SolnPoolIntensity, 4);
2. Set 10% tolerance
cplex.setParam (IloCplex::SolnPoolGap, 0.1);
3. Tell CPLEX to generate 200 solutions at a time
cplex.setParam (IloCplex::PopulateLim, 200);
4. Repeatedly call the populate function, this is called in the
same way that the solve function is called
cplex.populate ();
Enumeration code
#ifndef _L0_ENUM_HPP
#define _L0__ENUM_HPP
/**
* @file l0Enum.hpp
* @brief Enumerates all a% optimal feasible solns to
*
to L0 penalized regression.
* @author Eric B. Laber
* @note Example for st810
*/
#include <ilcplex/ilocplex.h>
#include <boost/numeric/ublas/vector.hpp>
#include <boost/numeric/ublas/vector_proxy.hpp>
#include <boost/numeric/ublas/matrix.hpp>
#include <boost/numeric/ublas/triangular.hpp>
#include <boost/numeric/ublas/lu.hpp>
#include <boost/numeric/ublas/io.hpp>
#include <boost/random.hpp>
#include <boost/random/normal_distribution.hpp>
#include <iostream>
namespace l0EN{
ILOSTLBEGIN
namespace ublas = boost::numeric::ublas;
/**
* @fn l0Enum (const ublas::vector<double>& y,
*
const ublas::matrix<double>& x,
*
const int s,
*
const double a,
*
ublas::matrix<double>& models)
* @brief Computes \arg\min_{beta} \sum (yi - xi^{\T}\beta)^2
*
s.t. ||\beta||_0 \le s
* @param y response vector
* @param x design matrix, first column should be ones
* @param s number of nonzero components, s >= 1
* @param a in nonnegative real indicating the acceptable
*
objective fn reduction, e.g, a value a=0.05 says
*
to return all solutions that are within 5% of the
*
optimum
* @param models return value with p columns, each row corresponds
*
to a feasible model that is within a of being optimal
*/
bool l0Enum (const ublas::vector<double>& y,
const ublas::matrix<double>& x,
const int s,
const double a,
ublas::matrix<double>& models){
IloEnv env; // Create cplex environment to store vars
IloModel model (env);
try{
int n = x.size1 ();
int p = x.size2 ();
IloNumVarArray b(env); // Regression coefficients
for (int j = 0; j < p; ++j){
b.add (IloNumVar (env, -IloInfinity, +IloInfinity));
}
// Create squared objective function
IloExpr objExpr (env);
for (int i = 0; i < n; ++i){
IloExpr xtb (env);
for (int j = 0; j < p; ++j){
xtb += x (i, j)*b[j];
}
objExpr += (y(i) - xtb)*(y(i) - xtb);
xtb.end ();
}
IloObjective obj (env, objExpr, IloObjective::Minimize);
model.add (obj);
objExpr.end ();
// Add constraint on L0-norm of b, use logical constraints
IloExpr l0Norm (env);
for (int j = 1; j < p; ++j){ // Don't pen intercept
l0Norm += (b[j] != 0); // 1_{beta_j \ne 0}
}
model.add (l0Norm <= s);
l0Norm.end ();
IloCplex cplex (model);
// // Display MIP progress
cplex.setParam (IloCplex::MIPDisplay, 2);
cplex.setParam (IloCplex::Threads, 8); // parallize
cplex.setParam (IloCplex::SolnPoolIntensity, 4);
cplex.setParam (IloCplex::SolnPoolGap, 0.1);
cplex.setParam (IloCplex::PopulateLim, 200);
// Attempt to solve
if (!cplex.populate ()){
env.end ();
std::cout << cplex.getStatus () << std::endl;
Enumeration full code cont’d
return false;
}
int curSolns;
do {
curSolns = cplex.getSolnPoolNsolns ();
cplex.populate ();
} while (curSolns != cplex.getSolnPoolNsolns());
// Extract models
models.resize (curSolns, p);
for (int i = 0; i < curSolns; ++i){
IloNumArray cpxBeta (env);
cplex.getValues (cpxBeta, b, i);
for (int j = 0; j < p; ++j){
models(i, j) = cpxBeta[j];
}
}
env.end ();
}
catch (IloException& e){
cerr << e << endl;
env.end ();
return false;
}
catch (...){
cerr << "Unknown exception caught,\n";
env.end ();
return false;
}
return true;
}
} // end namespace
#endif // _L0_HPP
I
Example output
> .Call ("l0Regression", y, x, 5,
$models
[,1]
[,2]
[,3]
[,4]
[1,] 0.9920 1.0642 0.9590 1.0308
[2,] 0.9720 1.0483 0.9614 1.0648
[3,] 0.9718 1.0478 0.9609 1.0653
[4,] 0.9701 1.0478 0.9590 1.0660
[5,] 0.9721 1.0479 0.9610 1.0651
[6,] 0.9746 1.0473 0.9610 1.0676
[7,] 0.9719 1.0479 0.9609 1.0651
[8,] 0.9720 1.0479 0.9609 1.0651
[9,] 0.9708 1.0482 0.9621 1.0645
[10,] 0.9720 1.0479 0.9608 1.0652
[11,] 0.9719 1.0478 0.9608 1.0652
[,8]
[,9]
[,10]
[1,] 0.0000 -0.1586 0.0000
[2,] 0.0000 0.0000 0.0000
[3,] 0.0000 0.0010 0.0000
[4,] 0.0000 0.0000 -0.0152
[5,] 0.0000 0.0000 0.0010
[6,] 0.0000 0.0000 0.0000
[7,] 0.0000 0.0000 0.0000
[8,] 0.0000 0.0000 0.0000
[9,] -0.0214 0.0000 0.0000
[10,] 0.0010 0.0000 0.0000
[11,] 0.0000 0.0000 0.0000
.1)
[,5]
[,6]
[,7]
1.0285 0.0000 0.0000
1.0324 0.0000 0.0045
1.0321 0.0000 0.0000
1.0323 0.0000 0.0000
1.0321 0.0000 0.0000
1.0297 -0.0369 0.0000
1.0322 0.0010 0.0000
1.0321 0.0000 0.0000
1.0384 0.0000 0.0000
1.0318 0.0000 0.0000
1.0320 0.0000 -0.0010
Another enumeration example
I
How can we use CPLEX to find all hyperplane generated
labelings? (AKA HW2 Problem 1.)
I
I
I
Let x1 , . . . , xn ∈ Rp be fixed, we say that ` ∈ {0, 1}n is feasible
if ∃ γ ∈ Rp s.t. `i = 1x | γ≥0
In the homework p = 4 and x has an intercept term
In class.
Well, it’s that time.
I
We’ve seen
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Reproducible research (R packages, Sweave, etc.)
Programming style
Calling and inlining C++ from R
Cloud computing
Some convex analysis
General nonlinear optimization
Linear programming
Quadtratic programming
Integer and mixed integer programming
Enumeration
Well, it’s that time.
I
We’ve seen
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Reproducible research (R packages, Sweave, etc.)
Programming style
Calling and inlining C++ from R
Cloud computing
Some convex analysis
General nonlinear optimization
Linear programming
Quadtratic programming
Integer and mixed integer programming
Enumeration
And...a bunch of quotes
“Once my student, always my student.”
–Eric Laber