Homoclinic solutions of Hamiltonian systems with
symmetry
Gianni Arioli
Andrzej Szulkin∗
Dipartimento di Scienze e T.A., via Cavour 84
15100 Alessandria, Italy
Department of Mathematics, Stockholm University
106 91 Stockholm, Sweden
Abstract
We prove the existence of infinitely many homoclinic solutions for a first order Hamiltonian
system, symmetric with respect to an action of a compact Lie group, by means of variational
methods. We make no convexity assumption on the Hamiltonian.
1
Introduction
Let H : R2N × R → R be a continuously differentiable function and consider the Hamiltonian
system
(1)
ż = JHz (z, t),
where z = (p, q) ∈ RN × RN = R2N and
J=
0 −I
I
0
!
is the standard symplectic matrix. Recall that a solution z of (1) is said to be homoclinic (to 0)
if z 6≡ 0 and z(t) → 0 as |t| → ∞. Suppose H(z, t) = 12 Az · z + F (z, t), where A is a symmetric
2N × 2N –matrix with constant entries, σ(JA) ∩ iR = ∅ (σ denotes the spectrum) and F is periodic
in t and superquadratic at z = 0 and |z| = ∞. It has been shown by Coti Zelati, Ekeland and
Séré [8] that if F is convex in z, then (under some additional assumptions) (1) has at least two
homoclinic solutions. Later Séré [18, 19] showed that (1) has in fact infinitely many homoclinics.
The convexity assumption has been removed by Hofer and Wysocki [14] and Tanaka [21] who
showed that (1) has at least one homoclinic for such more general F .
d
The conditions on A imply that if L := −J dt
− A, then in an appropriate function space L
is invertible, i.e. σ(L) ∩ (−α, α) = ∅ for some α > 0. As a function space one can choose e.g.
H 1/2 (R, R2N ). In a recent work Ding and Willem [13] relaxed the above conditions. They allowed
A to be t–dependent, periodic and such that σ(L) ∩ (0, α) = ∅ for some α > 0, and showed that
(1) still has a homoclinic solution. Subsequently Ding and Girardi [12] showed that if in addition
H is even in z, then (1) has infinitely many homoclinics.
In the present paper we assume that A is independent of t, σ(JA) ∩ iR = ∅ and F is invariant
with respect to an action of a compact Lie group. We show that if F is superquadratic at 0
and at infinity, then (1) has infinitely many geometrically distinct homoclinics. Our result includes
∗
Supported in part by the Swedish Natural Science Research Council
1
Hamiltonian systems with even H (which corresponds to the antipodal action of Z/2) as a particular
case. However, if a larger group of symmetries is present, then sometimes the existence of one
homoclinic solution suffices to imply the existence of infinitely many ones which are distinct in the
Z/2–sense. In such situation our result gives more information, see Remark 2.2 below. As we have
already mentioned, it was shown in [18, 19] (and in [9] for second order Hamiltonian systems) that if
F is convex, then (1) has infinitely many homoclinics even without any symmetry assumption. This
result is not applicable here because our function F need not be convex. It would be interesting to
know if the result of [18, 19] remains valid for such more general F .
Our proof is by variational arguments and we use a combination of ideas which may be found in
[1] and [15]. In Section 2, after recalling some definitions and facts from representation theory and
equivariant topology, we state the main result. In Section 3 we set up a variational framework and
study geometric properties of the functional and behavior of the Palais–Smale sequences. Section
4 is concerned with a Borsuk–Ulam type theorem and index theories which are suitable for the
problem. In Section 5 we prove a deformation lemma and finally in Section 6 we give a proof of
the main result.
We would like to point out that our approach can be modified so that it includes systems (1)
with the more general linear term considered in [12, 13]. Since a functional analytic framework for
such systems has already been established in [12, 13], in order to minimize technicalities we prefer
to restrict our attention to systems having invertible linear part.
2
Preliminaries and statement of the main result
We start by summarizing some definitions and facts from representation theory. More information
may be found e.g. in [3, 7, 11].
Let G be a compact Lie group. G is solvable if there exists a sequence G 0 ⊂ G1 ⊂ . . . ⊂ Gr = G
of subgroups of G such that G0 is a torus (possibly trivial, G0 = {e}), Gi−1 is normal in Gi and
Gi /Gi−1 ∼
= Z/pi for 1 ≤ i ≤ r, where the pi ’s are prime numbers. Let us remark that usually
solvable (algebraic) groups are defined in terms of commutators and one shows that G is solvable
if and only if there exists a sequence of groups {e} = H 0 ⊂ H1 ⊂ . . . ⊂ Hq = G such that Hi−1 is
normal in Hi and Hi /Hi−1 is abelian for 1 ≤ i ≤ q, see e.g. [6, I.6.4]. Since a compact Lie group
G is solvable if and only if there exists a torus T ⊂ G such that G/T is finite solvable [11, p. 263],
it is easily seen that for compact Lie groups our definition of solvability is equivalent to the usual
one. Note also that abelian groups are necessarily solvable.
Let now G be a compact Lie group, E, Ẽ two Hilbert spaces and ρ, ρ̃ representations of G in
respectively E and Ẽ. A subset A of E is said to be invariant if ρ(g)A ⊂ A for all g ∈ G. We shall
sometimes call A G–invariant or ρ–invariant if we want to distinguish between different groups or
representations. A functional Φ : E → R is invariant if Φ(ρ(g)z) = Φ(z) for all g, z, and a function
f : E → Ẽ is equivariant if f (ρ(g)z) = ρ̃(g)f (z) for all g, z. Usually ρ, ρ̃ will be omitted from
notation, so in particular we write gz for ρ(g)z and gf (z) for ρ̃(g)f (z). The space
E G := {z ∈ E : gz = z ∀g ∈ G}
will be called the fixed point space of (the representation of) G, and the orbit of z is defined by
OG (z) := {gz : g ∈ G}. Similarly, OG (A) := {gz : g ∈ G, z ∈ A}. Sometimes we shall omit the
subscript G from notation.
Let V be a finite-dimensional representation space of G. V is called admissible if for each open,
bounded and invariant neighborhood U of 0 in V k (k ≥ 1) and each equivariant map f : U → V k−1 ,
f −1 (0) ∩ ∂U 6= ∅. The corresponding representation ρ will also be called admissible. It is known [3,
2
Theorem 3.7] that V (and ρ) is admissible if and only if there exist subgroups K ⊂ H of G such
that K is normal in H, H/K is solvable, V K 6= 0 and V H = 0. Moreover, if G is solvable, then
any finite-dimensional representation space V with V G = 0 is admissible.
Suppose that ρ : G → GL(2N, R) is a representation of G in R 2N . Then ρ is said to be
symplectic if ρ(g)t Jρ(g) = J for all g ∈ G.
Next we state the assumptions on the Hamiltonian H, where as previously, H(z, t) = 12 Az · z +
F (z, t).
(H1) A is a constant symmetric 2N × 2N –matrix and σ(JA) ∩ iR = ∅.
(H2) F and Fz are 1–periodic in t and continuous.
(H3) Fz (z, t)/|z| → 0 uniformly in t as z → 0.
(H4) There exists γ > 2 such that 0 < γF (z, t) ≤ z · F z (z, t) for all z 6= 0.
(H5) There exist c, r > 0 such that |Fz (z, t)|2 ≤ cz · Fz (z, t) for all |z| ≤ r.
(H6) There exist c, R > 0 and q ∈ (1, 2) such that |F z (z, t)|q ≤ cz · Fz (z, t) for all |z| ≥ R.
(H7) There exist c̄, ε0 > 0 and p > 2 such that |Fz (z + w, t) − Fz (z, t)| ≤ c̄|w|(1 + |z|p−1 ) for all t
and all w, z with |w| ≤ ε0 .
We note that (H6) implies
|Fz (z, t)| ≤ c̃(1 + |z|p−1 ),
(2)
where p = q/(q − 1). Moreover, since this inequality and the one in (H7) remain valid if p is
replaced by any p̃ > p, we may assume that p ≥ q/(q − 1) > 2 and the p’s in (H7) and (2) are the
same. Assuming (H3) and (2) it is easy to see that (H5) and (H6) hold if the angle between z and
Fz is acute and bounded away from the right angle. A simple example of nonconvex F satisfying
(H1)–(H7) is F (z, t) = h(t)(|z| − sin |z|)|z| p−1 , where p > 3 and h is 1–periodic and positive (take
γ = p − 1 in (H4)).
Let
Φ(z) :=
1
2
Z
R
(−J ż − Az) · z dt −
Z
F (z, t) dt.
R
Assume for the moment that Φ ∈ C 1 (E, R), where E := H 1/2 (R, R2N ), and the critical points of
Φ are homoclinic solutions of (1) (this will be shown in the next section). In what follows we shall
use the following notation for the functional Φ:
K(Φ) = {z ∈ E : Φ0 (z) = 0}
and
Φb = {z ∈ E : Φ(z) ≤ b},
Φa = {z ∈ E : Φ(z) ≥ a},
Φba = Φb ∩ Φa .
For each k ∈ Z, let (k ∗ z)(t) := z(t + k). This defines a representation of Z in E, and it follows
from (H2) that Φ is Z–invariant. Let ρ be a symplectic representation of a compact Lie group
G in R2N and suppose that the Hamiltonian H is invariant with respect to ρ. Then ρ induces a
representation of G in E by means of the formula (gz)(t) := g(z(t)) and it is easy to see that Φ is
G–invariant. Moreover, Φ is also invariant with respect to the representation of Z × G in E given
by
(3)
((k, g)z)(t) := g(z(t + k)).
3
Let now O(z) = OZ×G (z) ≡ {k ∗ gz : k ∈ Z, g ∈ G} be the orbit of z ∈ E. If z is a critical point
of Φ, then O(z) will be called the critical orbit of z, and two homoclinic solutions of (1) are said
to be geometrically distinct if they are not in the same critical orbit. In other words, z 1 and z2 are
geometrically distinct if there are no k ∈ Z and g ∈ G such that z 2 (t) = gz1 (t + k) for all t ∈ R.
Now we are ready to state the main result of this paper.
Theorem 2.1 Suppose that H satisfies (H1)–(H7), ρ is an admissible symplectic representation of
a compact Lie group G in R2N and H is invariant with respect to ρ. Then (1) has infinitely many
geometrically distinct homoclinic solutions.
Remark 2.2 If H is even in z, then it follows from Theorem 2.1 that (1) has infinitely many pairs
of homoclinic solutions ±zk . As we have already mentioned in the introduction, this result under
somewhat weaker assumptions on the linear part has been obtained in [12]. Suppose now that
G is connected and infinite, the representation ρ of G in R 2N is symplectic and the Hamiltonian
H is invariant with respect to ρ. If {−1, 1} =: Z/2 ⊂ G and (−1)z = −z, then H is even. Let
z̄ be a homoclinic solution of (1). Then O Z×G (z̄), which is a single orbit with respect to the
action of Z × G, contains infinitely many orbits O Z×Z/2 (.). Hence (1) already has infinitely many
geometrically distinct homoclinics if only the action of Z × Z/2 is taken into account. Here our
Theorem 2.1 gives a better result, indeed it states that the number of orbits with respect to the
action of Z × G is infinite as well.
We illustrate this situation with the following example (cf. [3, Example 9.1a]): Let ρ 0 : G →
O(N ) be an orthogonal representation of G in R N and let
ρ(g) :=
ρ0 (g)
0
0
ρ0 (g)
!
.
Then ρ is a symplectic representation of G in R 2N . If N is even and G = SO(N ), then Z/2 ⊂
G (where Z/2 is represented by the matrices ±I) and the above conclusion applies provided
H(gp, gq, t) = H(p, q, t) for all g ∈ G, p, q ∈ R N .
Remark 2.3 If the Hamiltonian system (1) is autonomous (i.e. H = H(z)), then Φ is invariant
with respect to the representation of R × G given by ((s, g)z)(t) = g(z(t + s)). Hence the correct
definition of geometrically distinct solutions in this case is that they are not in the same R × G–
orbit. Although Theorem 2.1 guarantees the existence of infinitely many critical Z × G–orbits, all
of them may very well be contained in a single R × G–orbit. So our multiplicity result is of no
interest for autonomous systems. A similar observation (for G = {e}) has been made by several
authors, see e.g. [8, 9, 19].
3
Properties of the functional
Recall that E = H 1/2 (R, R2N ) is the space of functions z ∈ L2 (R, R2N ) whose Fourier transform
ẑ satisfies
Z
(1 + |ξ|2 )1/2 |ẑ(ξ)|2 dξ < ∞.
R
This is a Hilbert space under the inner product
(z, v) :=
Z
R
(1 + |ξ|2 )1/2 ẑ(ξ) · v̂(ξ) dξ;
4
a more convenient inner product will be introduced below. Let
(z, v)G :=
Z
(gz, gv) dg,
G
where dg is the normalized Haar measure, and let
(4)
Φ(z) =
1
2
Z
R
(−J ż − Az) · z dt −
Z
F (z, t) dt =:
R
1
(Lz, z)G − ψ(z).
2
Then (gz, gv)G = (z, v)G (i.e. the representation of G in (E, (., .) G ) is orthogonal) and L is equivariant. It is easy to see from Plancherel’s formula that L is a bounded selfadjoint operator. Moreover,
it follows from (H1) that −iξJ − A is invertible with (−iξJ − A) −1 uniformly bounded with respect
to ξ ∈ R. Therefore L is invertible in E. A more detailed argument may be found in Stuart [20,
d
Section 10] where it is also shown that σ(−J dt
− A) is unbounded both from above and from below
1
2N
in H (R, R ). Hence E = Y ⊕ W , where Y, W are infinite-dimensional L–invariant subspaces
of E and the quadratic form (Lz, z)G is negative definite on Y and positive definite on W . Let
P : E → Y and Q : E → W be the orthogonal projections and define
hz, vi := (L(Q − P )z, v)G .
The inner products (., .), (., .)G and h., .i are equivalent and the spaces Y, W are orthogonal with
respect to (., .)G and h., .i. Since (k ∗ z, k ∗ v)G = (z, v)G and (Lz, z)G is Z–invariant, L is Z–
equivariant. It follows that Y and W are Z×G–invariant. Moreover, if k.k is the norm corresponding
to the inner product h., .i, then
1
1
Φ(z) = kQzk2 − kP zk2 − ψ(z),
2
2
where ψ is as in (4). We summarize the above facts in the following
Proposition 3.1 Suppose (H1) is satisfied. Then the representation (3) of Z×G in E is orthogonal
with respect to the inner product h., .i. Moreover, E = Y ⊕ W , where Y, W are orthogonal, Z × G–
invariant and
Z
(−J ż − Az) · z dt = kQzk2 − kP zk2 .
R
It follows from (2) and (H3) that |Fz (z, t)| ≤ c0 (|z|+|z|p−1 ) for some c0 . Since E is continuously
embedded in Ls (R, R2N ) for each s ∈ [2, ∞) (see e.g. [20, Lemma 10.4]), the same argument as in
[22, Lemma 3.10] shows that ψ ∈ C 1 (E, R) and
0
hψ (z), vi =
Z
R
Fz (z, t) · v dt
(by duality we consider ψ 0 as an element of E). Therefore
(5)
hΦ0 (z), vi =
Z
R
(−J ż − Az − Fz (z, t)) · v dt,
and z is a critical point of Φ if and only if it is a solution of (1). Moreover, since z ∈ L s (R, R2N )
for all s ∈ [2, ∞), Fz (z(.), .) ∈ L2 (R, R2N ). So it follows from (1) that z ∈ H 1 (R, R2N ) whenever
Φ0 (z) = 0. Hence z(t) → 0 as |t| → ∞. Suppose z n * z. Then zn → z in Lsloc (R, R2N ) for
s ∈ [2, ∞) and Φ0 (zn ) * Φ0 (z). We have shown
5
Proposition 3.2 Assume (H1)–(H6). Then Φ ∈ C 1 (E, R), Φ0 is given by (5) and is weakly
sequentially continuous. Moreover, z ∈ E is a homoclinic solution of (1) if and only if z 6= 0 and
Φ0 (z) = 0.
In the next proposition we describe some further properties of the functional Φ.
Proposition 3.3 Assume (H1)–(H6).
(i) For each sufficiently small r > 0, b := inf ∂Br ∩W Φ > 0, where Br := {z ∈ E : kzk < r}.
(ii) Let W0 ⊂ W , dim W0 < ∞. Then Φ(z) → −∞ whenever z ∈ Y ⊕ W0 , kzk → ∞.
Proof (i) By (2) and (H3), for each ε > 0 there is c ε > 0 such that 0 ≤ F (z, t) ≤ ε|z|2 + cε |z|p .
Hence by the Sobolev embedding theorem, 0 ≤ ψ(z) ≤ C(εkzk 2 + cε kzkp ), where C is independent
of ε. Since ε was chosen arbitrarily, ψ(z) = o(kzk 2 ) as z → 0. Keeping in mind that Φ(z) =
1
2
2 kzk − ψ(z) for z ∈ W , we obtain the conclusion.
(ii) By (H4), F ≥ 0 for all z and F (z, t) ≥ c 1 |z|γ for some c1 > 0 and all |z| ≥ 1. Hence for
each δ > 0 there is c2 > 0 such that F (z, t) ≥ c2 |z|γ − δ|z|2 . Let z = y + w ∈ Y ⊕ W0 . Since there
exists a continuous projection from the closure of Y ⊕ W 0 in Lγ (R, R2N ) to W0 , we obtain after
choosing a sufficiently small δ,
Z
1
1
kwk2 − kyk2 − c2
|z|γ dt + δ
2
2
R
Z
2
2
≤ c3 kwk − c4 kyk − c5
|w|γ dt,
Φ(z) ≤
Z
R
|z|2 dt
R
where c3 , c4 , c5 > 0. Since dim W0 < ∞ and γ > 2, Φ(z) → −∞ as kzk → ∞.
2
Since the functional Φ is invariant with respect to the action of the (non–compact) group Z,
the Palais–Smale condition is not satisfied. Below we shall analyse the behavior of Palais–Smale
sequences. The arguments are known and follow closely [15] (and to large extent also [9, 10]);
therefore we omit the details and sometimes only point out the differences with the above–mentioned
work.
Recall that a sequence {zn } is called a Palais–Smale sequence at the level c ((P S) c –sequence
for short) if Φ(zn ) → c and Φ0 (zn ) → 0 as n → ∞.
Lemma 3.4 Assume (H1)–(H6). Let {zn } ⊂ E be a (P S)c –sequence. Then {zn } is bounded and
c ≥ 0.
Proof We just need minor modifications of the proof of Lemma 1.5 in [15]. By (H4),
(6)
1
c + 1 + kzn k ≥ Φ(zn ) − hΦ0 (zn ), zn i ≥ c1
2
Z
R
zn · Fz (zn , t) dt ≥ 0
for almost all n. It follows from (H4)–(H6) that for an appropriate c 2 > 0,
|Fz (z, t)|2 ≤ c2 z · Fz (z, t) if |z| ≤ 1
Therefore
c + 1 + kzn k ≥ c3
Z
and |Fz (z, t)|q ≤ c2 z · Fz (z, t) if |z| ≥ 1.
2
|zn |≤1
|Fz (zn , t)| dt +
6
Z
q
|zn |>1
!
|Fz (zn , t)| dt ,
where c3 > 0; hence
c3
Z
c3
Z
and
|zn |≤1
|zn |>1
2
!1/2
≤ (c + 1 + kzn k)1/2
q
!1/q
≤ (c + 1 + kzn k)1/q .
|Fz (zn , t)| dt
|Fz (zn , t)| dt
Let zn = yn + wn . By the Hölder inequality and the Sobolev embedding theorem,
2
0
kyn k = −hΦ (zn ), yn i −
Z
R
yn · Fz (zn , t) dt ≤ kyn k 1 + c4 (c + 1 + kzn k)1/2 + c5 (c + 1 + kzn k)1/q
for almost all n. This, together with a similar inequality for w n , implies that {zn } is bounded.
Finally, by (6), Φ(zn ) → c ≥ 0.
2
Remark 3.5 If {zn } is a sequence such that Φ0 (zn ) → 0 and Φ(zn ) is bounded above, then the
argument of Lemma 3.4 shows that {zn } is bounded. So a posteriori {zn } is a PS–sequence and we
may assume that Φ(zn ) → c.
Lemma 3.6 Assume (H1)–(H6). If {zn } is a (P S)c –sequence, then either c = 0 and zn → 0 after
passing to a subsequence, or c > 0 and there exist ε > 0, r > 0 and a sequence {a n } ⊂ R such that
kzn kL2 ((an −r,an +r),R2N ) ≥ ε for almost all n.
The argument is the same as in Lemma 1.7 in [15] and is therefore omitted. A crucial role in
the proof is played by the fact that if {z n } is bounded in E and
lim sup
Z
a+r
n→∞ a∈R a−r
|zn |2 dt = 0
for some r > 0, then zn → 0 in Ls (R, R2N ) for each s ∈ (2, ∞). This is a special case of a result
due to P.L. Lions, see e.g. [22, Lemma 1.21] (in [22] the space is H 1 (RN ) but it easy to see by
inspection that the argument remains valid for E).
Denote z̃n = kn ∗ zn , where kn ∈ Z. Since Φ is Z–invariant, {z̃n } is a (P S)c –sequence whenever
{zn } is. An immediate consequence of the above lemma is
Lemma 3.7 Assume (H1)–(H6). If Φ admits a (P S) c –sequence {zn } for some c > 0, then there
exist kn ∈ Z such that passing to a subsequence, z̃ n * z 6= 0 and Φ0 (z) = 0.
We will need the following representation theorem which describes Palais–Smale sequences:
Theorem 3.8 Assume (H1)–(H7). Let {z n } be a (P S)c –sequence for Φ, where c > 0. Then there
exist l critical points z i ∈ K(Φ) \ {0} and l sequences of integers k ni such that, up to a subsequence,
l
X
i
i
kn ∗ z → 0,
zn −
i=1
l
X
Φ(z i ) = c
i=1
and
|kni − knj | → ∞ if i 6= j.
7
The key steps in the proof are Lemma 3.7 and the following
Lemma 3.9 Assume (H1)–(H7). Let {zn } be a (P S)c –sequence for Φ such that zn * z 6= 0; then
(i) Φ(zn ) − Φ(zn − z) → Φ(z),
(ii) Φ0 (zn − z) → 0.
The proofs of this lemma and of Theorem 3.8 follow by repeating the argument of Proposition
4.2 in [15] (see also [9, 10]). We would like to emphasize that it is here the assumption (H7) is used
(in the verification of (ii), see (4.20) of [15]). It is easily seen from Lemma 3.6 that K(Φ)∩Φ α0 = {0}
for some α > 0. This fact was essential in the concluding part of the proof of Lemma 4.2 in [15].
4
Index and pseudoindex
Let E be a separable Hilbert space, E = Y ⊕ W an orthogonal decomposition and P : E → Y ,
Q : E → W the corresponding orthogonal projections. Given a complete orthonormal system
{ej }∞
j=1 in Y , we define a new norm by


kzkτ := max kQzk ,

∞
X
j=1


2−j |hej , P zi| .

In what follows we shall use the prefix τ to distinguish the topology induced by this norm from the
original topology. Clearly, kQzk ≤ kzk τ ≤ kzk.
Let A be a closed subset of E. A map h : A → E will be called τ −locally finite-dimensional if
each point z ∈ A has a τ −neighborhood N z such that h(Nz ∩A) is contained in a finite-dimensional
subspace of E. A map f = I − h : A → E will be called admissible if it is τ −continuous (i.e.
τ
τ
f (zn )→f (z) whenever zn →z) and h is τ −locally finite-dimensional. More details on these notions
may be found in Section 2 of [15]. In particular, it was shown there that if f is an admissible map,
then f is continuous in the original topology and if B is a closed, convex and bounded subset of
Y ⊕ W0 , where dim W0 < ∞, then B is τ −compact. Also, if {zn } is a bounded sequence, then
τ
zn →z if and only if P zn * P z and Qzn → Qz.
In this and in the next section we assume that G = Z/p, where p is a prime. As we shall see in
Section 6, Theorem 2.1 can be reduced to the case of such G. A representation of Z/p in E will be
called fixed point free if E Z/p = 0.
Suppose W0 is a finite-dimensional subspace of W and E 0 := Y ⊕ W0 . We shall need the
following theorem of Borsuk–Ulam type:
Theorem 4.1 Let ρ and ρ̃ be two fixed point free representations of Z/p in E 0 which leave Y and
W0 invariant. Suppose U is a bounded ρ–invariant neighborhood of the origin in E 0 and f : Ū → E0
is an admissible map, equivariant in the sense that f (ρ(g)z) = ρ̃(g)f (z) for all g ∈ Z/p, z ∈ E 0 .
If f −1 (0) is τ −compact and f (Ū ) ⊂ E1 = Y ⊕ W1 , where W1 is a proper subspace of W0 , then
f −1 (0) ∩ ∂U 6= ∅.
Proof Suppose f −1 (0) ∩ ∂U = ∅. For each z ∈ f −1 (0) there exists a τ −neighborhood Ñz of z
which is mapped by h into a finite-dimensional subspace E z of E0 . We may assume that Ez is
ρ–invariant. Let
[
Nz :=
ρ(g)Ñz ;
g∈Z/p
8
then Nz is an invariant neighborhood of O(z) and h(N z ) ⊂ Ez . Proceeding in this way for all
O(z) ⊂ f −1 (0) we obtain a τ –open covering of f −1 (0). By the τ –compactness of f −1 (0) there
S
exist points z1 , . . . , zm such that f −1 (0) ⊂ N := m
i=1 Nzi ∩ U . Clearly, N is open and h(N ) is
contained in an invariant finite-dimensional subspace L of E 0 , and we may assume L = YL ⊕ W0 .
Set NL := N ∩ L and fL := f |L . Since f −1 (0) ⊂ N , fL−1 (0) ∩ ∂NL = ∅. Hence the Brouwer degree
deg(fL , NL , 0) is well-defined. Since fL (N̄L ) ⊂ YL ⊕ W1 and YL ⊕ W1 is a proper subspace of L,
deg(fL , NL , 0) = 0 contradicting the fact that deg(f L , NL , 0) ≡ 1 (mod p) (see e.g. Bartsch [2]). 2
Suppose that
(7)
ρ is an orthogonal fixed point free representation of Z/p in E and Y, W are ρ–invariant.
(8)
Φ ∈ C 1 (E, R) is ρ–invariant, Φ(0) = 0 and Φ(z) → −∞ whenever kzk → ∞, z ∈ Y ⊕ W 0
and dim W0 < ∞.
Let
Σ := {A ⊂ E : A closed and ρ−invariant}.
Now we proceed to define an index and a pseudoindex for sets in Σ.
Definition 4.2 Let f : E → E. Then f ∈ H if:
(a) f is a homeomorphism (with respect to the original topology of E).
(b) f is equivariant (in the sense that f (ρ(g)z) = ρ(g)f (z)) and admissible.
(c) f (Φc ) ⊂ Φc for all c ≥ −1.
Suppose p > 2.
Definition 4.3 Let B ∈ Σ \ ∅. The Z/p–index of B, denoted i(B), is the smallest integer k for
which there exists a fixed point free representation ρ̃ of Z/p in C k and a map ϕ ∈ C(B, Ck \{0}) such
that ϕ(ρ(g)z) = ρ̃(g)ϕ(z). If such a map does not exist for any k, then i(B) = +∞; furthermore,
i(∅) = 0.
One can show that i satisfies the usual properties of an index. This can be done either by
adapting the standard proofs as given e.g. in [16, 17] or by noting that i(B) = A–genus(B), where
A is the unit circle in C and A–genus is defined in [3] (see in particular Proposition 2.9 there and
the comments preceding it). Let us note for further reference that if A ∈ Σ is compact and 0 ∈
/ A,
then i(A) < ∞ and i(A) = i(N ) for each sufficiently small neighborhood N ∈ Σ of A.
If p = 2, then there is only one orthogonal fixed point free representation ρ (corresponding to
the antipodal action of Z/2), ϕ is equivariant if and only if it is odd and C k in the definition above
should be replaced by Rk . So in this case i(B) is nothing else than Krasnoselskii’s genus.
Definition 4.4 Let r > 0 be fixed and so small that inf B̄r Φ > −1. The pseudoindex i∗ (A), where
A ∈ Σ, is defined by
i∗ (A) := min i(f (A) ∩ ∂Br ∩ W ).
f ∈H
Note that since ρ is orthogonal, ∂Br ∈ Σ; hence f (A) ∩ ∂Br ∩ W ∈ Σ and i∗ is well-defined.
Our pseudoindex is similar to the one that has been introduced by Benci [5] but there are also
some differences; in particular, our class H of homeomorphisms is a semigroup and not a group as
required in [5].
9
Lemma 4.5 Let A, B ∈ Σ.
(i) If A ⊂ B, then i∗ (A) ≤ i∗ (B).
(ii) If h ∈ H, then i∗ (h(A)) ≥ i∗ (A).
(iii) i∗ (A ∪ B) ≤ i∗ (A) + i(B).
Proof (i) is obvious.
(ii) If f ∈ H, then f ◦ h ∈ H and
i∗ (A) ≤ min i(f ◦ h(A) ∩ ∂Br ∩ W ) = i∗ (h(A)).
f ∈H
(iii) Let f ∈ H. Then
i∗ (A ∪ B) ≤ i(f (A ∪ B) ∩ ∂Br ∩ W ) ≤ i(f (A) ∩ ∂Br ∩ W ) + i(f (B))
by the subadditivity and monotonicity of the index. Since f (B) is homeomorphic to B, i(B) =
i(f (B)). Hence i∗ (A ∪ B) ≤ i(f (A) ∩ ∂Br ∩ W ) + i(B) and i∗ (A ∪ B) ≤ i∗ (A) + i(B).
2
We show that there exist sets of arbitrarily large pseudoindex. Let p > 2. Since all irreducible
fixed point free representations of Z/p are 2–dimensional (see e.g. [7, Section II.8], [11, I.2.7]),
there exists a sequence {Wk } of invariant subspaces of W such that dim W k = 2k and Wk ⊂ Wk+1 .
Let Ek := Y ⊕ Wk .
Lemma 4.6 i∗ (Ek ) ≥ k.
Proof Assume by contradiction that i ∗ (Ek ) = l with 0 ≤ l < k. Then there exists f ∈ H such that
i(f (Hk ) ∩ ∂Br ∩ W ) = l. Let U := f −1 (Br ) ∩ Ek and B := f −1 (B̄r ) ∩ Ek . Since f is equivariant,
then U and B are invariant, furthermore, B is τ −closed and U is open in E k . Since Φ(z) → −∞
as z ∈ Ek and ||z|| → ∞, then there exists Rk > r such that for all z ∈ Ek , ||z|| ≥ Rk , we have
Φ(z) ≤ −1; hence U ⊂ BRk (indeed, Φ(f (z)) > −1 whenever z ∈ U ). Furthermore, Ū ⊂ B,
f (B \ U ) ⊂ ∂Br and since f (0) = 0, 0 ∈ U . Assume that f (E k ) ∩ ∂Br ∩ W 6= ∅ (the other case
is simpler). There exists an equivariant map ϕ : f (E k ) ∩ ∂Br ∩ W → Cl \ {0} ⊂ Ck−1 \ {0}.
Since Ck−1 is isomorphic to Wk−1 , we may assume ϕ : f (Ek ) ∩ ∂Br ∩ W → Wk−1 \ {0} (where the
representation of Z/p in Wk−1 is the one inherited from Ck−1 ).
Let ϕ∗ : B̄r → Wk−1 be an equivariant extension of ϕ to B̄r . Consider a map f¯ : B →
Ek−1 given by f¯(z) = P f (z) + ϕ∗ (Qf (z)); f¯ is admissible and it is equivariant in the following
sense. Let ρ̃ denote the (fixed point free) representation of W k−1 inherited from Ck−1 and define
¯
ρ̂(g)(y + w) = ρ(g)y + ρ̃(g)w. Then f(ρ(g)z)
= ρ̂(g)f¯(z). If f¯(z) = 0, then f (z) ∈ W , Qf (z) = f (z)
and ϕ∗ (f (z)) = 0. Since for z ∈ B \ U, ϕ∗ (f (z)) = ϕ(f (z)) 6= 0, then f −1 (0) ∩ (B \ U ) = ∅ and in
particular f¯−1 (0) ∩ ∂U = ∅. But on the other hand, f¯−1 (0) is τ −compact (because it is τ −closed
and contained in the τ −compact set B̄Rk ∩ Ek ), hence Theorem 4.1 implies f¯−1 (0) ∩ ∂U 6= ∅. 2
If p = 2, then we take Wk to be k–dimensional and the argument is somewhat simpler (and has
been given in [15], see Lemma 4.8 there).
5
A deformation lemma
Let Φ be the functional described in Section 3. Recall that E = Y ⊕ W and Q : E → W is the
orthogonal projection. Assume G = Z/p, where p is a prime, the representation ρ of G in R 2N
10
is symplectic, fixed point free and H is equivariant. Then Φ is invariant and E Z/p = 0. In this
section we shall need the following two additional conditions:
K(Φ) \ {0} = OZ (K), where K is a compact set.
(9)
If F := QK, then (k1 ∗ F) ∩ (k2 ∗ F) = ∅ whenever k1 6= k2 .
(10)
The reason for introducing the conditions (9)–(10) is that (as we shall see) they are sufficient for
a deformation lemma to hold and they are certainly satisfied if K(Φ) consists of finitely many
Z × G–orbits. This together with a minimax argument will lead to a contradiction showing that
the number of critical orbits is in fact infinite.
We assume that K(Φ) 6= {0} (the other case is simpler). For all k ∈ Z and b > a > 0 let
Ua := Y ⊕
[
k∈Z
{w ∈ W : d(w, k ∗ F) < a}
Ta,b (k) := Y ⊕ {w ∈ W : d(w, k ∗ F) ∈ (a, b)},
where d(x, A) = inf kx − yk. For l ∈ N, k̄ = (k 1 , . . . , k l ) ∈ Zl and z̄ = (z 1 , . . . , z l ) ∈ Kl , let
y∈A
k̄ ∗ z̄ := k 1 ∗ z 1 + . . . + k l ∗ z l and k̄ ∗ K := k1 ∗ K + . . . + k l ∗ K.
Given a sequence {h̄n } ⊂ Zl , we say that {h̄n } diverges or h̄n → ∞ if |hin − hjn | → ∞ as n → ∞ for
all i 6= j. Let λ := min kwk. Since K(Φ) ∩ Y = {0}, λ > 0. In order to prove a deformation lemma,
w∈F
we give a lower bound for kΦ0 k in a suitable set; for this purpose we adapt some ideas from [1, 18]
to the present context.
Lemma 5.1 Suppose that Φ satisfies (9)–(10).
(a) There exists r0 > 0 such that if k ∈ Z \ {0} then
d(k ∗ F, F) :=
min
(w 1 ,w 2 )∈(k∗F ,F )
kw1 − w2 k ≥ 3r0 .
(b) µρ := inf{kΦ0 (z)k : z ∈ Ur0 \ Ūρ } > 0 for all ρ ∈ (0, r0 ).
(c) If {zn } is a Palais-Smale sequence such that Φ(z n ) is bounded away from 0 and kQzn+1 −
Qzn k < r20 for all n, then there exists k ∈ Z such that d(z n , k ∗ K) → 0; in particular, for all δ we
have zn ∈ Uδ whenever n is large.
Proof (a) Set
c̃ = lim
min
n→∞ (z 1 ,z 2 )∈F 2
kz 1 − n ∗ z 2 k =
√
2 min kzk > 0 ,
z∈F
where the second equality holds because hz 1 , n ∗ z 2 i → 0 uniformly with respect to z 1 , z 2 ∈ F as
n → ∞. Then the inequality d(F, n ∗ F) < c̃/3 holds only for a finite number of integers and (a)
follows by the compactness of F. We choose r 0 < λ2 .
S
(b) It follows from (a) that Ur0 \ Ūρ = k∈Z Tρ,r0 (k) and the sets Tρ,r0 (k) are disjoint. If µρ = 0
for some ρ, then by the Z−invariance of the functional there exists a sequence {z n } ⊂ E such that
zn ∈ Tρ,r0 (0) and Φ0 (zn ) → 0. As Tρ,r0 (0) ⊂ Φc for some c, then the sequence {zn } is Palais-Smale
according to Remark 3.5, and by Theorem 3.8 there exist l critical points z 1 , . . . , z l ∈ K and a
sequence {k̄n } ⊂ Zl , k̄n → ∞, such that, up to a subsequence
kzn − k̄n ∗ z̄k → 0,
11
therefore, for large n we have k̄n ∗ z̄ ∈ T ρ ,2r0 (0), i.e.
2
ρ
< d(Q(k̄n ∗ z̄), F) < 2r0 .
2
We show that these inequalities lead to a contradiction: indeed, as k̄n → ∞, up to a subsequence
we have kni → k i ∈ Z ∪ {+∞} ∪ {−∞} for n → ∞ and k i ∈ Z for at most one value of i. If l = 1
and either kn1 → k 1 6= 0 or kn1 → ±∞, then d(Q(k̄n ∗ z̄), F) = d(Q(kn1 ∗ z 1 ), F) ≥ 3r0 for almost all
n; if l = 1 and kn1 → 0, then d(Q(k̄n ∗ z̄), F) = d(Q(kn1 ∗ z 1 ), F) → 0. If l > 1, then choose i such
that kni → ∞ and note that limn→∞ d(Q(k̄n ∗ z̄), F) ≥ kQz i k ≥ λ > 2r0 .
(c) By Theorem 3.8 there exist l critical points z 1 , . . . , z l ∈ K, a sequence {h̄k } ⊂ Zl , h̄k → ∞,
and a subsequence {znk } such that kznk − h̄k ∗ z̄k → 0. If l = 1, then (c) holds: indeed, since
kQzn+1 − Qzn k < r20 , it follows from (a) and (b) that d(Qz n , k 1 ∗ F) < r0 for some k 1 and almost
all n. Hence h̄k = k 1 for k large and znk → k 1 ∗ z 1 .
We shall show that l ≥ 2 cannot occur. By the definition of diverging h̄k and the fact that
kQzk > 2r0 whenever z ∈ K, we have d(Q(h̄n ∗ z̄), h̄k ∗ F) ≥ 2r0 for a fixed arbitrary k and
large n, say n ≥ nk . So for all k there exists m ≥ nk such that d(Qzm , h̄k ∗ F) > r0 . Since
kQzn+1 − Qzn k < r20 and kznk − h̄k ∗ z̄k < r20 whenever k is large, for such k there exists m k ≥ nk
with d(Qzmk , h̄k ∗ F) ∈ ( r20 , r0 ). But this is impossible because {z mk } is a PS-sequence and for all
sequences {k̄n } ⊂ Zl (l ≥ 2) such that k̄n → ∞ as n → ∞ we have
0
lim inf inf kΦ (z)k : d(Qz, k̄n ∗ F) ∈
n→∞
r0
, r0
2
>0.
To prove
this claim by contradiction, assume that there is a sequence {z n } with d(Qzn , k̄n ∗ F) ∈
0
r0
0
1
l0
2 , r0 , Φ (zn ) → 0 and k̄n → ∞. By Theorem 3.8, there exist l critical points (u , . . . , u ) = ū ∈
0
Kl and a sequence {h̄n } such that h̄n → ∞ and, up
to a subsequence, kzn − h̄n ∗ ūk → 0. Hence
for n large enough d(Q(h̄n ∗ ū), k̄n ∗ F) ∈ r40 , 2r0 . If l 6= l0 or |hin − knj | → ∞ for some i and
all j, then limn→∞ d(Q(h̄n ∗ ū), k̄n ∗ F) ≥ kQui k ≥ λ > 2r0 . So l = l0 and up to a subsequence
hin − kni → ai ∈ Z for each i (possibly after relabelling the i’s in k̄n ). Then
lim d(Q(h̄n ∗ ū), k̄n ∗ F)2 =
n→∞
l
X
i=1
d(Q(ai ∗ ui ), F)2 .
Since the distances on the right-hand side are either 0 or exceed 2r 0 , the above limit is not in
[ r20 , 2r0 ].
So far we have proved that l is necessarily equal to 1 and consequently {z n } has a subsequence
znk → k 1 ∗z 1 . The same argument shows that any subsequence of {z n } has a subsequence converging
to an element of k 1 ∗ K (with the same k 1 ). Hence the conclusion.
2
Choose 0 < δ < r0 /4 and r > 0 such that inf B̄r Φ > −1. Set µ := inf{kΦ0 (z)k : z ∈ Ur0 \ Uδ }
and b := inf ∂Br ∩W Φ; then µ, b > 0 by respectively Lemma 5.1 (b) and Proposition 3.3 (i).
Lemma 5.2 Let N := E \ Uδ . There exists a Z/p−equivariant vector field V : N → E satisfying
hΦ0 (z), V (z)i ≥ 0 for all z ∈ N ∩ Φ−1 and hΦ0 (z), V (z)i > 1 for all z ∈ N ∩ Φb/2 . Moreover, V is
τ −locally τ −Lipschitz continuous, locally Lipschitz continuous and τ −locally finite-dimensional.
τ
Proof Recall from Section 4 that if {z n } is a bounded sequence, then zn → z if and only if
τ
P zn * P z and Qzn → Qz. Moreover, if zn → z and zn ∈ Φc , then {P zn } is necessarily bounded,
12
so zn * z. Using Fatou’s lemma and weak lower semicontinuity of the norm it follows therefore
that Φc is closed for each c (cf. [15]).
For z ∈ N ∩ Φb/2 let
2Φ0 (z)
.
ω(z) :=
kΦ0 (z)k2
Since Φ0 is weakly sequentially continuous, the function
v 7→ hΦ0 (v), ω(z)i ∈ R
τ
is τ −continuous on Φ−1 , i.e. if vn ∈ Φ−1 and vn →v, then hΦ0 (vn ), ω(z)i → hΦ0 (v), ω(z)i. Therefore
z has a τ −open neighborhood Uz ⊂ E such that
hΦ0 (v), ω(z)i > 1
(11)
for all v ∈ Uz ∩ Φ−1 ; we may assume Uz is contained in a τ −ball of radius smaller than δ/2.
Additionally we let U0 := Φ−1 (−∞, b/2). The set U0 is τ −open (since Φb/2 is τ −closed).
The family {Uz }z∈Φb/2 ∪ {U0 } is a τ −open covering of the space (N, τ ). Therefore it has a
τ −locally finite τ −open refinement {N j }j∈J . Let {λj }j∈J be a τ −Lipschitz continuous partition
of unity subordinated to the cover {N j }j∈J . If Nj ⊂ Uzj for some zj ∈ N , then we set ωj := ω(zj ),
and if Nj ⊂ U0 , we take ωj := 0. For any z ∈ N define
(12)
V (z) :=
1 X X −1
g (λj (gz)ωj )
p g∈Z/p j
and note that V is Z/p−equivariant and the sum over j is τ −locally finite, therefore V is τ −locally
finite-dimensional. Since for each j there is a constant L j such that |λj (z 0 )−λj (z 00 )| ≤ Lj kz 0 −z 00 kτ ≤
Lj kz 0 − z 00 k, V is τ −locally τ −Lipschitz continuous and locally Lipschitz continuous.
2
By (11), hΦ0 (z), V (z)i ≥ 0 for all z ∈ N ∩ Φ−1 and hΦ0 (z), V (z)i > 1 for z ∈ N ∩ Φb/2 .
Lemma 5.3 Suppose Φ satisfies (9)–(10). Then for all 0 < ε < min{δµ/4, b/2} there exists a map
f ∈ H such that f (Φc+ε \ U3δ ) ⊂ Φc−ε whenever c ≥ b.
Proof We adapt an argument from [1]. Let V be the vector field defined in Lemma 5.2 and let
ψ : E → [0, 1] be a Z/p−invariant τ −Lipschitz continuous function satisfying
ψ(z) =
(
1, z ∈
/ U2δ and Φ(z) ≥ b/2
0, z ∈ Uδ .
Consider the flow η defined by the Cauchy problem
(13)
(
dη
ds
= −ψ(η)V (η)
η(0, z) = z.
Since k kτ ≤ k k, ψ is also Lipschitz continuous and (13) admits a unique solution η(·, z) in a suitable
right neighborhood of s = 0; assume for the moment that for all z ∈ E such a neighborhood is the
half-line [0, +∞) and define f (z) := η(2ε, z).
To prove that f ∈ H, we note first that (a) and (c) of Definition 4.2 are obviously satisfied.
Since ψV is Z/p−equivariant, so is f = η(2ε, ·) [16]. Finally, since ψV is τ −locally τ −Lipschitz
continuous and τ −locally finite-dimensional, it follows from Proposition 2.2 of [15] that η is an
admissible homotopy; hence f is an admissible map.
13
Let z ∈ Φc+ε \ U3δ . We claim that η(s, z) ∈
/ U2δ for any s ∈ [0, 2ε]. Indeed, if η(s, z) ∈ U 2δ
for some s, then there exist 0 ≤ s1 < s2 ≤ 2ε such that η(s1 , z) ∈ ∂ U3δ , η(s2 , z) ∈ ∂ U2δ and
η(s, z) ∈ U3δ \ U2δ whenever s1 < s < s2 . Let v = η(s, z) for such s and choose j ∈ J, g ∈ Z/p with
gv ∈ Nj . Since each neighborhood Nj is contained in a τ −ball of radius smaller than δ/2, then
the point zj which enters in the definition of ωj = ω(zj ) satisfies ||Qzj − Q(gv)|| < δ/2; therefore
zj ∈ U4δ \ Uδ , kΦ0 (zj )k ≥ µ and ||ωj || ≤ 2/µ. Hence
(14)
Finally
kV (v)k ≤
1 X X
2 X X
2
λj (gv) kωj k ≤
λj (gv) = .
p g∈Z/p j
µp g∈Z/p j
µ
Z
δ ≤ kη(s2 , z) − η(s1 , z)k = s2
s1
4ε
ψ(η)V (η)ds
≤ µ < δ,
and this contradiction proves the claim. Since ψ = 1 and hΦ 0 (z), V (z)i ≥ 1 on Φb−ε \ U2δ , it follows
that if z ∈ Φc+ε \ U3δ and η(σ, z) ∈ Φb−ε for some σ ∈ [0, 2ε], then
Φ(η(σ, z)) − Φ(z) =
Z
σ
0
d
Φ(η(s, z))ds =
ds
Z
σ
0
hΦ0 (η(s, z)), ηs (s, z)ids ≤ −σ
and Φ(η(2ε, z)) ≤ c − ε.
To complete the proof of the lemma we still have to show that the solution η of (13) exists
for all initial data z ∈ E and all s ≥ 0; by contradiction, assume there exists z ∈ E such that
the corresponding flow η is defined only on [0, S), S < +∞, then lim sup s→S − kηs (s, z)k = +∞.
Consider the following Cauchy problem:
(
(15)
dϕ
dt
= −ψ(ϕ)X(ϕ)
ϕ(0, z) = z ,
where X(z) is defined by
X(z) :=
V (z)
.
kV (z)k
Since sup kϕt k ≤ 1, then either there exist t0 > 0, w ∈ E such that ψ(w)V (w) = 0 and ϕ(t, z) → w
as t → t0 or ϕ(t, z) is defined for all t ∈ [0, +∞). The curves η(·, z) and ϕ(·, z) are equal up to a
reparametrization, therefore the first possibility cannot occur (otherwise η(s, z) → w as s tends to
some s0 ≤ S and η(s, z) = w for s ≥ s0 ). So ϕ(t, z) exists for all t ≥ 0. Furthermore,
ϕ(t, z) = η
and
Z
+∞
0
Z
t
0
kV (ϕ(r, z))k −1 dr, z
kV (ϕ(t, z))k −1 dt = S < +∞ .
By the above equality, for all ε > 0 there exists a sequence t n → +∞ such that |tn − tn−1 | < ε and
kV (ϕ(tn , z))k → ∞. (To see this, consider the sequence x n := nε/2 and let
cn :=
P
Z
xn+1
xn
kV (ϕ(r, z))k −1 dr;
then cn → 0 because S = n cn , therefore for all n there exists tn ∈ [xn , xn+1 ] such that cn =
−1
r0
r0
ε
2 kV (ϕ(tn , z))k .) Let ε < 4 : since kϕt k ≤ 1, kϕ(tn , z) − ϕ(tn−1 , z)k < 4 . The sequence
14
zn := ϕ(tn , z) satisfies kzn − zn+1 k < r40 and kV (zn )k → ∞. For each n there exist points
yj ∈ N ∩ Φb/2 such that V (zn ) is given by a formula like (12) with ω j = 2Φ0 (yj )/kΦ0 (yj )k2 . If
kΦ0 (yj )k ≥ 4/kV (zn )k for all indices j, then arguing as in (14) we get the contradiction kV (z n )k ≤
1
0
2 kV (zn )k. Hence we can find vn := yjn and gn such that kΦ (vn )k < 4/kV (zn )k → 0 and kQvn −
−1
−1
Q(gn zn )k < δ/2. Using Z/p−invariance it follows that kQg n vn − Qgn+1
vn+1 k < δ + r40 < r20 and
0
−1
−1
−1
Φ (gn vn ) → 0. Moreover, Φ(gn vn ) is bounded and Φ(gn vn ) ≥ b/2. By Lemma 5.1 (c) we infer
vn ∈ Uδ/2 for n large, hence zn ∈ Uδ and ψ(zn ) = 0, which contradicts S < +∞.
2
We note that if K(Φ) = {0}, then it follows from Lemma 3.7 that there are no (P S) c –sequences
with c > 0. Hence kΦ0 (z)k ≥ µ > 0 for z ∈ Φb/2 , so the conclusion of Lemma 5.3 remains valid
with U3δ replaced by the empty set, and the proof is simpler.
6
Proof of Theorem 2.1
We first reduce Theorem 2.1 to the following special case:
Theorem 6.1 Suppose that H satisfies (H1)–(H7), ρ is a symplectic representation of Z/p in R 2N ,
where p is a prime, and H is invariant with respect to ρ. If (R 2N )Z/p = 0, then there is no compact
set K with the property that K(Φ) \ {0} = O Z (K) and if F := QK, then (k1 ∗ F) ∩ (k2 ∗ F) = ∅
whenever k1 6= k2 . In particular, (1) has infinitely many geometrically distinct homoclinic solutions.
To show that the above result implies Theorem 2.1 we adapt an argument of Bartsch and Clapp
[4]. Since the representation ρ is admissible, we have K ⊂ H ⊂ G, where K is a normal subgroup
of H and (R2N )H = 0, V := (R2N )K 6= 0. Moreover, H/K is solvable, hence there exist subgroups
S0 ⊂ S1 ⊂ . . . ⊂ Sr = H/K such that S0 is a torus, Si−1 is normal in Si and Si /Si−1 ∼
= Z/pi for
i = 1, . . . , r. We shall distinguish two cases.
Case 1. V Sj = 0 and V Sj−1 6= 0 for some j ≥ 1.
Let π : H → H/K be the projection and H0 := π −1 (Sj−1 ). Then Sj−1 = H0 /K and (E K )Sj−1 =
E H0 . Denote Φ̃ = Φ|E H0 . Since V Sj−1 = (R2N )H0 , we may assume that (R2N )H0 = R2M for some
M and E H0 = H 1/2 (R, R2M ). Since Φ̃ is π −1 (Sj )/H0 –invariant, π −1 (Sj )/H0 ∼
= Sj /Sj−1 ∼
= Z/pj
2M
S
/S
S
j
j−1
j
and (R )
= V = 0, Theorem 6.1 applies to Φ̃. By the principle of symmetric criticality
[22, Theorem 1.28], all critical points of Φ̃ are also critical for Φ.
Suppose now that (1) has finitely many geometrically distinct critical orbits. Let O Z×G (z1 ), . . . ,
OZ×G (zm ) be those orbits which intersect with E H0 . We may assume zj ∈ E H0 . Let

and wj = Qzj . Then
K := 

F =
m
[
j=1
m
[
j=1

OG (zj ) ∩ E H0

OG (wj ) ∩ E H0 .
Clearly, K is compact, K(Φ)\{0} = OZ (K) and, by the definition of geometrically distinct solutions,
(k1 ∗ F) ∩ (k2 ∗ F) = ∅ whenever k1 6= k2 (the number of distinct orbits in F may be less than m
because some OG (wj ) may coincide). This is a contradiction to Theorem 6.1.
Case 2. V S0 = 0.
Since S0 = T q (the q–dimensional torus, q ≥ 1), we have {e} = T 0 ⊂ T 1 ⊂ . . . ⊂ T q = S0 and
15
j
j−1
V T = 0, V T
6= 0 for some j ≥ 1. Let K0 := π −1 (T j−1 ) (where again π : H → H/K). Then
j−1
T
= K0 /K, V Tj−1 = (R2N )K0 and (E K )Tj−1 = E K0 . Hence similarly as in Case 1, we may
assume (R2N )K0 = R2M for some M , E K0 = H 1/2 (R, R2M ) and Φ̃ := Φ|E K0 is π −1 (T j )/K0 –
invariant. Furthermore, π −1 (T j )/K0 ∼
= Tj /Tj−1 ∼
= S 1 and (R2M )Tj /Tj−1 = V Tj = 0. Since
iθ
ipθ
iθ
1
Z/p = {e : θ ∈ R, e = 1} ⊂ {e : θ ∈ R} = S , there is a prime p such that (R2M )Z/p = 0.
Hence we can conclude as in Case 1.
Now it remains to prove Theorem 6.1. Suppose that a compact set K 6= ∅ with the properties
stated in the theorem exists (if K = ∅, the argument is simpler). Then (9)–(10) are satisfied. For
each k ≥ 1 let
dk := ∗ inf sup Φ(z).
i (A)≥k z∈A
The representation of Z/p in E induced by ρ is orthogonal, fixed point free and leaves Y and W
invariant. This and Proposition 3.3 (ii) imply (7)–(8). Hence by Lemma 4.6 d k is well-defined
and clearly dk ≤ dk+1 . Since i∗ (A) ≥ k implies that f (A) ∩ ∂Br ∩ W 6= ∅ for each f ∈ H,
Φ(z) ≥ Φ(f (z)) ≥ b for some z ∈ A according to Proposition 3.3 (i). So d k ≥ b.
Choose δ, ε satisfying the assumptions of Lemma 5.3 and set c := sup z∈U3δ Φ(z). Suppose that
dk > c for some k and let f ∈ H be as in Lemma 5.3. By Lemma 4.5, i ∗ (f (Φdk +ε )) ≥ i∗ (Φdk +ε ) ≥ k.
Now if z ∈ U3δ , then Φ(z) ≤ c, so Φ(f (z)) ≤ c < dk , and if z ∈ Φdk +ε \ U3δ , then Φ(f (z)) ≤ dk − ε.
This contradicts the definition of dk . So b ≤ dk ≤ c and dk → d¯ ≤ c. Since 0 ∈
/ F and F is compact,
i(F) = m < ∞. It is easy to see that if δ is small enough, then the sets Y ⊕ {w ∈ W : d(w, k ∗ F) ≤
3δ} are disjoint and have index m; it follows that i( Ū3δ ) = m. By the definition of dk and Lemmas
4.5, 5.3,
k ≤ i∗ (Φdk +ε ) ≤ i∗ (Φdk +ε \ U3δ ) + i(Ū3δ ) ≤ i∗ (Φdk −ε ) + m;
¯ ≥ d,
¯ a contradiction.
therefore i∗ (Φdk −ε ) ≥ k−m and dk −ε ≥ dk−m . Letting k → ∞ we obtain d−ε
References
[1] G. Arioli, F. Gazzola, S. Terracini, Multibump periodic motions of an infinite lattice of particles,
Math. Z. 223 (1996) 627-642
[2] T. Bartsch, A simple proof of the degree formula for Z/p–equivariant maps, Math. Z. 212
(1992) 285-292
[3] T. Bartsch, Topological Methods for Variational Problems with Symmetries, Springer-Verlag
1993
[4] T. Bartsch, M. Clapp, Critical point theory for indefinite functionals with symmetries, J. of
Funct. Anal. 138 (1996) 107-136
[5] V. Benci, On critical point theory for indefinite functionals in the presence of symmetries,
Trans. of the A.M.S. 274 (1982) 533-572
[6] N. Bourbaki, Algebra I, Springer-Verlag 1989
[7] T. Bröcker and T. tom Dieck, Representation of Compact Lie Groups, Springer-Verlag 1985
[8] V. Coti Zelati, I. Ekeland, E. Séré, A variational approach to homoclinic orbits in Hamiltonian
systems, Math. Ann. 288, (1990) 133-160
16
[9] V. Coti Zelati, P.H. Rabinowitz, Homoclinic orbits for second order Hamiltonian systems
possessing superquadratic potentials, J. of the AMS 4 (1991) 693-727
[10] V. Coti Zelati, P.H. Rabinowitz, Homoclinic type solutions for a semilinear elliptic PDE on
Rn , Comm. Pure Appl. Math. 45 (1992) 1217-1269
[11] T. tom Dieck, Transformation Groups, de Gruyter 1987
[12] Y.H. Ding and M. Girardi, Infinitely many homoclinic orbits of a Hamiltonian system with
symmetry, preprint
[13] Y.H. Ding and M. Willem, Homoclinic orbits of a Hamiltonian system, preprint
[14] H. Hofer and K. Wysocki, First order elliptic systems and the existence of homoclinic orbits
in Hamiltonian systems, Math. Ann. 288, (1990) 483-503
[15] W. Kryszewski, A. Szulkin, Generalized linking theorem with an application to semilinear
Schrödinger equation, Adv. Diff. Eq., to appear
[16] J. Mawhin and M. Willem, Critical point theory and Hamiltonian systems, Springer-Verlag
1989
[17] P.H. Rabinowitz, Minimax Methods in Critical Point Theory with Applications to Differential
Equations, CBMS 65, Amer. Math. Soc., Providence, R.I. 1986
[18] E. Séré, Existence of infinitely many homoclinic orbits in Hamiltonian systems, Math. Z. 209,
(1992) 27-42
[19] E. Séré, Looking for the Bernoulli shift, Ann. Inst. H.P. 10, 5, (1993), 561-590
[20] C.A. Stuart, Bifurcation into spectral gaps, Supplement to Bull. Belgian Math. Soc. 1995
[21] K. Tanaka, Homoclinic orbits in a first order superquadratic Hamiltonian system: Convergence
of subharmonic orbits, J. Diff. Eq. 94 (1991) 315-339
[22] M. Willem, Minimax Theorems, Birkhäuser 1996
17