Integer quadratic programming in the plane
Alberto Del Pia
∗
Abstract
We show that the problem of minimizing a quadratic polynomial with integer coefficients over the integer points in a
general two-dimensional rational polyhedron is solvable in
time bounded by a polynomial in the input size.
1 Introduction
Among the most important theorems in the theory of
fixed-dimension optimization is Lenstra’s result [6]. It
states that it is possible to minimize a linear function
with integer coefficients over the integer points in a
rational polyhedron in time bounded by a polynomial
in the size of the input, provided that the number
of integer variables is a constant. Note that the
polyhedron can be given either by an external or
internal description, since in fixed dimension one can
pass from one to the other in polynomial time. This
polynomiality result does not extend if one replaces
a linear objective function with a general polynomial
as shown in [2], not even when the dimension is
equal to two. In fact, Manders and Adleman [8]
proved that the following decision problem is NPcomplete. (See problem AN1 on page 249 of [3].)
AN1 Given positive integers a, b, and c, is there a
positive integer x < c such that x2 is congruent
to a modulo b?
It is an easy exercise to reformulate problem AN1 as
a minimization problem with a degree four polynomial
with integer coefficients as an objective function, and
a feasible domain consisting of all integer points in a
two-dimensional box. This shows that minimizing a
polynomial of degree four with integer coefficients over
the integer points in a rational polyhedron is NP-hard
already in dimension two. This naturally leads to the
question about the complexity status of minimizing
a polynomial of degree two or three over the integer
points in a rational polyhedron in dimension two.
Let us formally introduce the problem
(1.1)
min{f k (z) : z ∈ P ∩ Zn },
∗ Goldstine Fellow, Business Analytics and Mathematical Sciences department, IBM Watson Research Center, United States
† Institute for Operations Research, Department of Mathematics, ETH Zürich, Switzerland
Robert Weismantel
†
where f k is a polynomial function of degree at most k
with integer coefficients, and P is a rational polyhedron
in Rn . We recall that a rational polyhedron is the set of
points that satisfy a system of linear inequalities with
rational data. Problem (1.1) is bounded if there exists
γ ∈ Z such that f k (z) ≥ γ for every z ∈ P ∩ Zn , and
it is unbounded otherwise. We say that we can solve
problem (1.1) if we can either show that P ∩ Zn = ∅,
or show that problem (1.1) is unbounded, or find an
integer point in P that minimizes f . The complexity
of problem (1.1) in fixed dimension is summarized in
Table 1.
k=1
k=2
k=3
k=4
k general
n=1
P
P
P
P
P
n=2
P
?
?
NPH
NPH
n fixed
P
?
?
NPH
NPH
Table 1: Complexity of problem (1.1) in fixed dimension. “P” stands for polynomial time solvable, and
“NPH” stands for NP-hard.
For n = 1 problem (1.1) is easily solvable in
polynomial time as follows. It is easy to check that
problem (1.1) is unbounded if and only if there exists
an integer vector v in the recession cone of P such that
av k ≤ −1, where a is the leading coefficient of f . If
problem (1.1) is bounded, then compute all the roots
of the derivative f 0 of f (this can be done by finding
the eigenvalues of the companion matrix of f 0 ). Then
evaluate f only on the integer boundary of P , and on
the integer points at distance at most one from such
zeroes of f 0 . Therefore the first non-trivial case is when
n = k = 2.
Our main result is the following.
Theorem 1.1. If n = k = 2, problem (1.1) can be
solved in polynomial time.
Note that Theorem 1.1 is best possible in the sense
that if we add only one quadratic constraint to problem (1.1) with n = k = 2, then the problem becomes
NP-hard. To see this, we use again the the reduction
from the NP-complete problem AN1. Problem AN1
is equivalent to asking whether the minimum of the
quadratic polynomial function x2 − a − by over the integer points (x, y) in the bounded set defined by in2
≤ y ≤ (c−1)b −a , and
equalities 1 ≤ x ≤ c − 1, 1−a
b
x2 − a − by ≥ 0, is zero or not.
where f : R2 → R is a quadratic polynomial with
integer coefficients
Let us finally mention that Theorem 1.1 is not only
of relevance from the optimization point of view. Rather
there are number theory type questions that can be
transformed into the problem of minimizing a degree
two polynomial over the integer points in a rational
polyhedron in dimension two. As one such example let
us mention the Pell equation.
The Pell equation is the equation
(2.4)
x2 = dy 2 + 1,
to be solved in positive integers x, y for a given positive
and non-square integer d.
We refer to [7] for a treatment of history and
solution methods for this equation. The Pell equation
always has a solution, but all its solutions can be of
size exponential in the size of d. In turn, Theorem 1.1
cannot be directly applicable to find one. Theorem 1.1
can be applied, however, to a bounded version of the
Pell equation:
(2.3)
f (x, y) = ax2 + bxy + cy 2 + dx + ey,
and where P is a rational polyhedron in R2 .
The graph of the quadratic equation
f (x, y) = γ
is always a conic section, though it may be degenerate.
The conic sections described by equation (2.4) can be
classified with the discriminant
∆ = b2 − 4ac.
If the conic section is non-degenerate, then if ∆ < 0 the
equation represents an ellipse, if ∆ = 0 the equation
represents a parabola, and if ∆ > 0 the equation
represents a hyperbola. (See Figure 1.)
To distinguish the degenerate cases from the nondegenerate cases, let
Γ = ∆γ + bde − ae2 − cd2 .
Then the conic section is non-degenerate if and only if
Γ 6= 0. If Γ = 0 we have a point in the case of an ellipse,
two parallel lines (possibly coinciding with each other)
in the case of a parabola, or two intersecting lines in the
case of a hyperbola.
given a positive and non-square integer d, and a
positive integer u, do positive integers x, y exist
The following Lemma 2.1 gives an important necwith x ≤ u and x2 = dy 2 + 1?
essary condition of the boundedness of problem (2.2).
Later, in Section 4, we will see that such condition is
In order to solve this question one can proceed as also sufficient, and we will see how to use it in order to
follows.
First compute a rational approximation d˜ of understand if problem (2.2) is bounded or not in poly√
d such that
nomial time. We denote by “rec” the recession cone (see
for example [10]).
2
˜ x ≤ u} =
{(x, y) ∈ Z+ : x ≥ dy,
= {(x, y) ∈ Z2+ : x2 ≥ dy 2 , x ≤ u}.
This can be accomplished in polynomial time, for
example by means of continued fractions. Consider now
the problem
˜ 1 ≤ x ≤ u, (x, y) ∈ Z2 }
min{x2 − dy 2 : x ≥ dy,
+
that we can solve in polynomial time by means of
Theorem 1.1. The minimum of such problem is 1 if
and only if there exists a solution of the Pell equation
with the first coordinate smaller than u.
2
Ingredients for the proof of Theorem 1.1
From now on we will study the problem
(2.2)
min{f (x, y) : (x, y) ∈ P ∩ Z2 },
Lemma 2.1. Assume that P contains an integer point.
If problem (2.2) is bounded, then for every nonzero
v in Z2 ∩ rec P , either avx2 + bvx vy + cvy2 ≥ 1, or
avx2 + bvx vy + cvy2 = 0 and for every (x, y) ∈ Z2 ∩ P ,
(2ax + by + d)vx + (bx + 2cy + e)vy ≥ 0.
Proof. Let z̄ = (x̄, ȳ) ∈ P ∩ Z2 , and let v be a nonzero
vector in Z2 ∩ rec P . Then
f (z̄ + λv) − f (z̄) = λ2 (avx2 + bvx vy + cvy2 )+
+ λ((2ax̄ + bȳ + d)vx + (bx̄ + 2cȳ + e)vy ).
If avx2 + bvx vy + cvy2 < 0, then limλ→∞ f (z̄ + λv) = −∞,
contradicting that problem (2.2) is bounded. Thus
avx2 + bvx vy + cvy2 ≥ 0. If avx2 + bvx vy + cvy2 > 0, then by
the integrality of a, b, c, v it follows that avx2 + bvx vy +
cvy2 ≥ 1 and we are done.
Given a polyhedron Q, we will denote by QI the
integer hull of Q, i.e. QI = conv(Q ∩ Z2 ). Meyer [9]
showed that if Q is a rational polyhedron, then QI is
again a (rational) polyhedron. Given a convex set S,
we say that f : S → R is quasi-convex if each set
{(x, y) ∈ S : f (x, y) ≤ γ} is convex. f is quasi-concave
if −f is quasi-convex. It is well-known that a convex
function is quasi-convex, and that a concave function is
quasi-concave.
(a) An ellipse
(b) A parabola
Lemma 2.2. If P is rational and f is quasi-concave on
PI , then problem (2.2) can be solved in polynomial time.
Proof. As P is rational, it follows by Section 4.2 in
Hartmann [4] (see also remark (2.4) in Cook, Hartmann,
Kannan, and McDiarmid [1]) that we can describe the
integer hull PI of P in polynomial time. It is wellknown that a quasi-concave function takes on only
higher values on convex combinations than it takes on
the defining point with the smallest function value. As
any integer point in P can be written as a convex
combination of integer points in the boundary of PI ,
it follows that the minimum is achieved at an integer
point in the boundary of PI . Since it is possible to
optimize in polynomial time the polynomial f over the
integer points of every (at most) one-dimensional face
of PI , the statement follows.
We are now prepared to give the proof of Theorem 1.1 under the assumption that ∆ ≤ 0.
3
Proof of Theorem 1.1 for ∆ ≤ 0
First assume a = c = 0. As b2 ≤ 4ac, it follows that
b = 0. Then f is linear, and by Lenstra [6] we can solve
Figure 1: Classification of conic sections
problem (2.2) in polynomial time.
We now assume that a and c are not both zero, wlog
a 6= 0. Then (2.3) can be rewritten as follows:
Otherwise avx2 + bvx vy + cvy2 = 0. Then
b 2 −∆ 2
f (z̄ +λv)−f (z̄) = λ((2ax̄+bȳ +d)vx +(bx̄+2cȳ +e)vy ). (3.5)
y + dx + ey.
f (x, y) = a x + y +
2a
4a
If (2ax̄ + bȳ + d)vx + (bx̄ + 2cȳ + e)vy < 0, then
limλ→∞ f (z̄ + λv) = −∞, again contradicting that If a > 0, it follows that −∆/4a ≥ 0. From the
problem (2.2) is bounded. Hence (2ax̄ + bȳ + d)vx + representation of f by means of equation (3.5) it follows
(bx̄ + 2cȳ + e)vy ≥ 0.
that f is a convex function. Thus by Khachiyan and
Porkolab [5] we can solve problem (2.2) in polynomial
Important ingredients of the proof of Theorem 1.1 time. Otherwise, a < 0. Then −∆/4a ≤ 0 and so the
are the special cases where f is linear, convex or representation of f by means of equation (3.5) proves
concave. In the case where f is a convex function, that f is a concave function. Hence by Lemma 2.2 we
then problem (2.2) can be solved in polynomial time, for can solve problem (2.2) in polynomial time.
(c) A hyperbola
example by Theorem 1.2 in Khachiyan and Porkolab [5].
Note that their result is actually much more general,
and later on we will make use of such generality. In the
case where f is a concave function, then the next lemma
provides us with an algorithm to solve the optimization
problem (2.2).
4
Proof of Theorem 1.1 for ∆ > 0
In the remainder of the paper we assume ∆ > 0. As
a first step we establish a canonic representation of the
quadratic function f based on its degenerate section.
From this representation several properties follow.
A canonic representation of f . We define the
value γ̄ as follows:
(
if a = c = 0,
− de
(4.6)
γ̄ = ae2b+cd2 −bde
otherwise.
∆
The following three lemmas give structural properties that will be main tools to solve a quadratic integer
optimization problem in polynomial time. We illustrate
Lemma 4.1 and Lemma 4.2 in Figure 2.
First assume a = c = 0. As b2 > 4ac, it follows that
b 6= 0. In this case (2.3) can be rewritten in the form:
e d
(4.7)
b x+
y+
+ γ̄.
b
b
Otherwise, a and c are not both zero, wlog a 6= 0.
In this case (2.3) can be rewritten in the form:
bd − 2ae 2
b
d 2 ∆ y+
(4.8) a x + y +
−
+ γ̄.
2a
2a
4a
∆
The important property of the number γ̄ is that
the conic section f (x, y) = γ̄ is degenerate. This allows
us to decompose R2 into four subregions each of which
will be studied separately. The cases a > 0 and a < 0
are analogous, thus we will from now on assume a ≥ 0
in order to streamline the presentation. The analogy
of the two cases a > 0 and a < 0 follows from the
representation of f by means of (4.8) where either a > 0
∆
and − 4a
< 0, or conversely.
Next we define linear functions `1 , `2
follows.
(
by + d
1
√
` (x, y) =
√
2ax + (b + ∆)y + d + bd−2ae
∆
(
x + e/b
√
`2 (x, y) =
√
2ax + (b − ∆)y + d − bd−2ae
∆
: R2 → R as
(b) Lines L1 and L2 with some level sets
if a = c = 0
if a > 0,
if a = c = 0
if a > 0.
Consider the two distinct lines
L1 = {(x, y) ∈ R2 : `1 (x, y) = 0},
2
2
(a) Lines L1 and L2
2
L = {(x, y) ∈ R : ` (x, y) = 0},
and the four translated cones
C 1 = {(x, y) ∈ R2 : `1 (x, y) ≥ 0, `2 (x, y) ≤ 0},
Figure 2: Illustration of Lemma 4.1 and Lemma 4.2
Lemma 4.1. f (x, y) = γ̄ for the points in L1 ∪ L2 ,
f (x, y) < γ̄ for the points in (int C 1 ) ∪ (int C 2 ), and
f (x, y) > γ̄ for the points in (int C 3 ) ∪ (int C 4 ).
Proof. For a = c = 0, this follows easily from (4.7),
thus we now assume a > 0. It follows from (4.8) that,
for ◦ ∈ {=, <, >}, the points satisfying f (x, y)◦γ̄ are the
points that satisfy the following equivalent conditions
b
d 2 ∆ bd − 2ae 2
a
x
+
y
+
◦
y
+
,
2a
2a
4a
∆
C 2 = {(x, y) ∈ R2 : `1 (x, y) ≤ 0, `2 (x, y) ≥ 0},
r
√ C 3 = {(x, y) ∈ R2 : `1 (x, y) ≥ 0, `2 (x, y) ≥ 0},
b
d ∆ bd − 2ae y+
a x+ y+
◦
,
4
2
1
2
2a
2a
4a
∆
C = {(x, y) ∈ R : ` (x, y) ≤ 0, ` (x, y) ≤ 0}.
√ bd − 2ae Note that, if a = 0 all such regions are rational,
2ax + by + d ◦ ∆ y +
.
∆
while if a > 0 they might be not rational. However, if
1
2
a > 0, then the following rational inequalities will be Thus f (x, y) = γ̄ for the points in L ∪ L , f (x, y) < γ̄
1
2
1
2
for the points in C ∪ C \ (L ∪ L ), and f (x, y) > γ̄
useful later.
for the points in C 3 ∪ C 4 \ (L1 ∪ L2 ).
Remark 4.1. If a > 0, then ∆y + bd − 2ae ≥ 0 is valid
for C 1 , ∆y+bd−2ae ≤ 0 is valid for C 2 , 2ax+by+d ≥ 0 Lemma 4.2. For every γ ≤ γ̄, the set of points that
is valid for C 3 , and 2ax + by + d ≤ 0 is valid for C 4 .
satisfy f (x, y) ≤ γ is the union of two convex sets, one
contained in C 1 , and the other contained in C 2 . For Lemma 4.3. For γ ∈ Q with γ ≤ γ̄, and for i = 1, 2,
every γ ≥ γ̄, the set of points that satisfy f (x, y) ≥ γ is the set {(x, y) ∈ C i : f (x, y) ≤ γ}, can be described by
the union of two convex sets, one contained in C 3 , and means of polynomial rational inequalities.
the other contained in C 4 .
Proof. We prove the statement for i = 1, the case i = 2
1
Proof. For γ = γ̄ this follows from Lemma 4.1, thus we being symmetric. Let γ ≤ γ̄, and F = {(x, y) ∈ C :
1
now assume γ 6= γ̄. Let γ > γ̄. By Lemma 4.1, the f (x, y) ≤ γ}. If a = 0, then the polyhedron C is
set of points that satisfy f (x, y) ≥ γ is contained in described by rational linear inequalities, therefore the
set F is defined by polynomial rational inequalities.
(int C 3 ) ∪ (int C 4 ). We show that the set of points
If a > 0, then the difficulty arises that the polyheW = {(x, y) ∈ int C 3 : f (x, y) ≥ γ}
dron C 1 might be not rational. In this case we make
use of Remark 4.1. The inequality ∆y + bd − 2ae ≥ 0 is
is convex. The other cases follow in a similar way.
valid for C 1 , while inequality ∆y + bd − 2ae ≤ 0 is valid
3
First assume a = c = 0. Since int C = {(x, y) ∈ for C 2 . It follows that
2
R : by + d > 0, x + e/b > 0}, by (4.7) we have
F = {(x, y) ∈ R2 : ∆y + bd − 2ae ≥ 0, f (x, y) ≤ γ}.
o
n
e
by + d ≥ γ − γ̄
W = (x, y) ∈ int C 3 : x +
This trick allows us to again describe the convex set F
b
n
o
by means of polynomial rational inequalities.
3
= (x, y) ∈ int C : g(x) − h(y) ≥ 0 ,
4.1 Bounded case We are now prepared to complete the proof of Theorem 1.1 under the assumption
that problem (2.2) is bounded. Later in Section 4.2
∂2g
1
e −3/2
we will show how to check in polynomial time if prob(x) = − x +
< 0,
2
lem (2.2) is bounded or not.
∂x
4
b
2
−5/2
Note that by Lenstra [6] we can check in polynomial
√
∂ h
3
(y) = b2 γ − γ̄ by + d
≥ 0,
time
if P contains integer points, thus from now on we
2
∂y
4
will assume that P contains an integer point, i.e. PI 6= ∅.
the function g(x) is concave, and the function h(y) is
Let
convex, thus the function g(x) − h(y) is concave, which
P i = P ∩ C i , for i = 1, . . . , 4.
shows that W is convex.
We begin investigating the two polyhedra P 1 and P 2 .
Now assume a > 0. By (4.8), W is the set of points
More precisely, for i = 1, 2, we will solve the problem
(x, y) in C 3 that satisfy
min{f (x, y) : (x, y) ∈ P i ∩ Z2 }, i.e. we will either show
that P i ∩ Z2 = ∅, or we will find an integer point in P i
b
d 2
∆
bd − 2ae 2
a x+ y+
≥
y+
+ γ − γ̄.
that minimizes f . The following observation is crucial.
2a
2a
4a
∆
If there is at least an integer point in P 1 ∪ P 2 , then
b
d
Since by Remark 4.1 the inequality x + 2a y + 2a ≥ 0 is we do not need to consider integer points in P 3 ∪ P 4 ,
since by Lemma 4.1 they have higher objective value. It
valid for C 3 , we have
will remain then to analyze the situation if there are no
n
o
√ b
d
1
2
i
3
W = (x, y) ∈ C : a x + y +
− g(y) ≥ 0 , integer points in P ∪ P . Then for each polyhedron P ,
i
2a
2a
i = 3, 4, we will first construct a relaxation of P in form
r of a rational polyhedron using Remark 4.1. This will
2
∆
bd−2ae
allow us to solve the problems min{f (x, y) : (x, y) ∈
where g(y) = 4a y + ∆
+ γ − γ̄. Since
P i ∩ Z2 }, for i = 3, 4.
Polyhedra P 1 and P 2 . Let γ ∈ Q with γ ≤ γ̄.
γ − γ̄
∂2g
(y) = q By Lemma 4.2, the set {(x, y) ∈ P 1 : f (x, y) ≤ γ} is
2
3/2 > 0,
∂y 2
4a(γ−γ̄)
∆
bd−2ae
convex. Since P 1 = P ∩ C 1 and P is rational, it follows
y+ ∆
+
4a
∆
by Lemma 4.3 that the set {(x, y) ∈ P 1 : f (x, y) ≤
√
the function g(y) is convex, thus the function a(x + γ} can be described by means of polynomial rational
d
b
inequalities. Hence, by Khachiyan and Porkolab [5],
2a y + 2a ) − g(y) is concave, which shows that the set
W is convex.
we can solve in polynomial time the integer feasibility
problem for the set {(x, y) ∈ P 1 : f (x, y) ≤ γ}, i.e. we
Note that Lemma 4.2 implies that f is quasi-convex on can test in polynomial time whether such set contains
C 1 and on C 2 , and is quasi-concave on C 3 and on C 4 . or not integer points.
where g(x) =
p
x + eb , and h(y) =
q
γ−γ̄
by+d .
Since
Using the algorithm of Khachiyan and Porkolab as
described above in combination with a binary search
allows us then to solve the problem min{f (x, y) :
(x, y) ∈ P 1 ∩ Z2 }. For this to work we derive an
upper bound and a lower bound on the range of values
{f (x, y) : (x, y) ∈ P 1 ∩ Z2 }. The upper bound follows
from Lemma 4.1, since for every point (x, y) ∈ P 1 ,
f (x, y) ≤ γ̄. We now explain how to calculate a lower
bound.
First we show that if the polyhedron P 1 is not
bounded, then f (x, y) ≥ γ̄ for every (x, y) ∈ P 1 ∩ Z2 .
Since P 1 is not bounded, rec P 1 contains a nonzero
vector. rec P 1 is the intersection of rational cone
rec P with full-dimensional cone rec C 1 , therefore rec P 1
contains a nonzero integer vector. Since P contains at
least an integer point and problem (2.2) is bounded,
Lemma 2.1 implies that for every nonzero integer vector
v in rec P , either avx2 + bvx vy + cvy2 ≥ 1, or avx2 +
bvx vy + cvy2 = 0 and for every (x, y) ∈ Z2 ∩ P ,
(2ax + by + d)vx + (bx + 2cy + e)vy ≥ 0. By Lemma 4.1
applied to g(x, y) = ax2 + bxy + cy 2 , this means that
for every nonzero integer v in rec P 1 , either v ∈ rec C 3
and inequality `2 (x, y) ≥ 0 is valid for P ∩ Z2 , or
v ∈ rec C 4 and inequality `1 (x, y) ≤ 0 is valid for P ∩Z2 .
Since rec P 1 contains a nonzero integer vector, such
vector can be either in rec C 3 or in rec C 4 . Assume
that it is in rec C 3 (if it is in rec C 4 the argument is
symmetric). Then `2 (x, y) ≥ 0 is valid for P ∩ Z2 ,
therefore f (x, y) ≥ γ̄ for every (x, y) ∈ P 1 ∩ Z2 .
We now explain how to get a lower bound on
{f (x, y) : (x, y) ∈ P 1 ∩ Z2 } in the case where P 1
is bounded. Let lx = min{x : (x, y) ∈ P 1 ∩ Z2 },
ux = max{x : (x, y) ∈ P 1 ∩ Z2 }, ly = min{y :
(x, y) ∈ P 1 ∩ Z2 }, uy = max{y : (x, y) ∈ P 1 ∩ Z2 }.
All these optimization problems have a finite optimum
since P 1 is bounded. Moreover they can be solved
in polynomial time using the algorithm by Khachiyan
and Porkolab since we are optimizing linear functions
over the integer points in the set P 1 , which is convex
and can be described by means of polynomial rational
inequalities by Lemma 4.3. In order to get a lower
bound on the objective function value, then one can
bound separately the monomials in (2.3): for every
(x, y) ∈ P 1 ∩ Z2 , ax2 ≥ min{0, alx2 , au2x }, bxy ≥
min{blx ly , blx uy , bux ly , bux uy }, cy 2 ≥ min{0, cly2 , cu2y },
dx ≥ min{dlx , dux }, ey ≥ min{ely , euy }.
Therefore, by using the polynomial time algorithm
by Khachiyan and Porkolab in a binary search setting,
we can solve the problem min{f (x, y) : (x, y) ∈ P 1 ∩
Z2 }. The problem min{f (x, y) : (x, y) ∈ P 2 ∩ Z2 } is
symmetric, thus it can be solved in the same way.
If there is an integer point in P 1 ∪ P 2 , then by
Lemma 4.1 we have solved problem (2.2), and we are
done. Thus we now assume that there is no integer
point in P 1 ∪ P 2 .
Polyhedra P 3 and P 4 . We show how to solve the
problem min{f (x, y) : (x, y) ∈ P 3 ∩ Z2 } in polynomial
time. By Lemma 4.2, f is quasi-concave on C 3 , and
so also on P 3 and PI3 . If a = 0, the polyhedron P 3
is rational, thus by applying Lemma 2.2 to P 3 we can
solve the problem min{f (x, y) : (x, y) ∈ P 3 ∩ Z2 } in
polynomial time.
If a > 0, then P 3 might be not rational. In this
case we construct a rational polyhedron P̃ 3 such that
P̃I3 = PI3 . This can be achieved with the help of
Remark 4.1. Set P̃ 3 = {(x, y) ∈ P : 2ax + by + d ≥ 0}.
Since 2ax + by + d ≤ 0 is valid for P 4 and the polyhedra
P 1 and P 2 contain no integer points, we have the
property that P̃I3 = PI3 . Since the function f is quasiconcave on P̃I3 , and since P̃ 3 is a rational polyhedron,
by applying Lemma 2.2 to P̃ 3 we can solve the problem
min{f (x, y) : (x, y) ∈ P 3 ∩ Z2 } in polynomial time.
The same argument applies to solve the problem
min{f (x, y) : (x, y) ∈ P 4 ∩ Z2 } in polynomial time.
This completes the proof of Theorem 1.1 under the
assumption that problem (2.2) is bounded.
4.2 Unboundedness We now drop the assumption
that problem (2.2) is bounded, and we show how to
check in polynomial time if it is bounded or not.
Note that by Lenstra we can solve the integer
feasibility problem for P , thus we now assume that
P contains an integer point. The following lemma
characterizes when problem (2.2) is unbounded.
Lemma 4.4. Assume that P contains an integer point
and that ∆ > 0. Then problem (2.2) is unbounded if
and only if there exists a nonzero vector v̄ in Z2 ∩ rec P
such that either av̄x2 + bv̄x v̄y + cv̄y2 ≤ −1, or av̄x2 +
bv̄x v̄y + cv̄y2 = 0 and there exists (x̄, ȳ) ∈ Z2 ∩ P such
that (2ax̄ + bȳ + d)v̄x + (bx̄ + 2cȳ + e)v̄y ≤ −1.
Proof. The sufficiency of the condition follows by
Lemma 2.1. To show the necessity of the condition,
assume that problem (2.2) is unbounded. Therefore
there exist infinitely many distinct integer points in P
with objective value strictly smaller than γ̄. Since by
Lemma 4.1 all the integer points with objective value
strictly smaller than γ̄ are contained either in int P 1 or
in int P 2 , then at least one among int P 1 and int P 2 contains infinitely many integer points. Wlog int P 1 contains infinitely many integer points. Therefore PI1 and
P 1 are unbounded polyhedra.
Since P 1 is unbounded, rec P 1 6= {0}. rec P 1
is the intersection of rational cone rec P with fulldimensional cone rec C 1 , therefore rec P 1 contains a
nonzero integer vector v̄. Since v̄ ∈ rec P 1 , it follows by
Lemma 4.1 (applied to g(x, y) = ax2 + bxy + cy 2 ) that
av̄x2 + bv̄x v̄y + cv̄y2 ≤ 0. If av̄x2 + bv̄x v̄y + cv̄y2 < 0, then the
integrality of a, b, c, v̄ implies av̄x2 + bv̄x v̄y + cv̄y2 ≤ −1
and we are done, thus assume av̄x2 + bv̄x v̄y + cv̄y2 = 0.
Again by Lemma 4.1 this implies that v̄ lies on the
boundary of rec P 1 , therefore it lies in either rec L1
or in rec L2 , wlog in rec L1 . Since there exists an
integer point (x̄, ȳ) in int P 1 ⊆ int C 1 , we have that
(2ax̄ + bȳ + d)v̄x + (bx̄ + 2cȳ + e)v̄y < 0, and by the
integrality of a, b, c, d, e, v̄, x̄, ȳ, we have (2ax̄+bȳ+d)v̄x +
(bx̄ + 2cȳ + e)v̄y ≤ −1.
It can be shown that Lemma 4.4 is valid also if the
assumption ∆ > 0 is dropped. However we do not need
such statement since we have already fully solved the
case ∆ ≤ 0 with other methods in Section 3.
We now show how to use Lemma 4.4 to deduce
in polynomial time if problem (2.2) is bounded or
unbounded.
Fist we need to check in polynomial time if there
exists a nonzero vector v̄ in Z2 ∩ rec P such that
av̄x2 + bv̄x v̄y + cv̄y2 ≤ −1. By Lemma 4.2 applied
to g(x, y) = ax2 + bxy + cy 2 , this amounts to solve
the integer feasibility problem for the two convex sets
(rec P )∩{(x, y) ∈ rec C i : ax2 +bxy+cy 2 ≤ −1}, for i =
1, 2. Since rec P is rational, it follows by Lemma 4.3 that
each set (rec P ) ∩ {(x, y) ∈ rec C i : ax2 + bxy + cy 2 ≤
−1} can be described by means of polynomial rational
inequalities. Hence, by Khachiyan and Porkolab, we can
solve the integer feasibility problem for each set.
Now we show how to find all nonzero vectors v
in Z2 ∩ rec P that satisfy avx2 + bvx vy + cvy2 = 0.
By Lemma 4.1 applied to g(x, y) = ax2 + bxy + cy 2 ,
g(x, y) = 0 for the points in (rec L1 ) ∪ (rec L2 ). The
lines rec L1 and rec L2 contain an integer nonzero vector
if and only if they are rational, and this happens if
a = c = 0, or if a > 0 and ∆ is a square. (We again
omit the case a < 0 since it is analogous to the case
a > 0.) If the lines rec L1 and rec L2 are not rational,
then there are no nonzero integer vectors that satisfy
g(x, y) = 0. Otherwise, if rec L1 and rec L2 are rational,
then there are only four nonzero integer vectors (up to
multiplication by a scalar) that satisfy g(x, y) = 0 and
we can find them in polynomial time.
Let (v̄x , v̄y ) be a vector such that av̄x2 +bv̄x v̄y +cv̄y2 =
0. By Lenstra [6], we can now check in polynomial time
if there exists an integer point in Z2 ∩P that satisfies the
linear inequality (2ax+by+d)v̄x +(bx+2cy+e)v̄y ≤ −1.
References
[1] W. Cook, M. Hartman, R. Kannan, and C. McDiarmid. On integer points in polyhedra. Combinatorica,
12(1):27–37, 1992.
[2] J.A. De Loera, R. Hemmecke, M. Köppe, and R. Weismantel. Integer polynomial optimization in fixed
dimension.
Mathematics Of Operations Research,
31(1):147–153, 2006.
[3] M.R. Garey and D.S. Johnson.
Computers and
Intractability:
A Guide to the Theory of NPCompleteness. Freeman, San Francisco, 1979.
[4] M. Hartmann. Cutting planes and the complexity
of the integer hull. Technical Report 819, School
of Operations Research and Industrial Engineering,
Cornell University, 1989.
[5] L. Khachiyan and L. Porkolab. Integer optimization on
convex semialgebraic sets. Discrete & Computational
Geometry, 23(2):207–224, 2000.
[6] H.W. Jr. Lenstra. Integer programming with a fixed
number of variables. Mathematics of Operations Research, 8(4):538–548, 1983.
[7] H.W. Jr. Lenstra. Solving the Pell equation. Notices
of the AMS, 49(2):182–192, 2002.
[8] K. Manders and L. Adleman. NP-complete decision
problems for binary quadratics. Journal of Computer
and System Sciences, 16:168–184, 1978.
[9] R.R. Meyer. On the existence of optimal solutions
to integer and mixed-integer programming problems.
Mathematical Programming, 7(1):223–235, 1974.
[10] A. Schrijver. Theory of Linear and Integer Programming. Wiley, Chichester, 1986.