Behavioral Ecology Vol. 11 No. 6: 676–685
Nest mate recognition in ants with complex
colonies: within- and between-population
variation
Robin J. Stuarta and Joan M. Herbersb
Department of Biology, University of Vermont, Burlington, VT 05405, USA
The ability to recognize kin is widespread, and especially important in highly social organisms. We studied kin recognition by
assessing patterns of aggression within and between nests of the ant Leptothorax longispinosus. Colonies of this species can be
fractionated into subunits, a condition called polydomy. The problem of recognizing relatives is therefore more complex when
those relatives can live in two or more different places. We hypothesized that spatial subdivision may have resulted in a stronger
genetic component to kin recognition than in cases where colonies live in a single location. To test our hypothesis we assessed
recognition capabilities for two populations of this ant that differ in the complexity of their colonies. In a New York, USA,
population, polydomy is very common, and colonies also can have multiple queens. By contrast, a population in West Virginia,
USA, has colonies that typically are monogynous and rarely are polydomous. We conducted introductions of ants between
different nests collected in the same neighborhood, with self-introductions and alien introductions as controls. Nests from the
two populations showed corresponding differences in their aggression towards intruders. For New York nests, the extent of
genetic similarity was the single best predictor of aggression, whereas for West Virginia nests aggression was jointly influenced
by genetic similarity and spatial distance. In both populations, we found nest pairs for which aggression was nonreciprocal; these
probably reflect recognition errors by one of the nests. After the ants were maintained in the laboratory for 3 months, their
aggression scores rose and fewer recognition errors were made. Thus nest-mate and colony-mate recognition in this species are
mediated primarily by endogenous cues (genetic similarity); the importance of exogenous cues for nest mate recognition
depends on the population’s social system. Key words: nest mate recognition, social insects, ants, colony structure. [Behav Ecol
11:676–685 (2000)]
O
ne of the most successful derivatives from Hamilton’s
landmark work on inclusive fitness theory (1964) is the
field of kin recognition. Hamilton’s theory suggested that selection favoring behaviors that benefit kin could also produce
the ability to discriminate between kin and non-kin. Indeed,
a voluminous literature (reviews in Fletcher and Michener,
1987; Hepper, 1991; Sherman et al., 1997) has confirmed that
kin recognition shapes social behavior in a wide range of organisms. The field of kin recognition explores three categories of proximate questions that must be carefully disentangled: signal production and/or acquisition; signal perception;
and behavioral responses to perceived signals (Gamboa et al.,
1991; Sherman et al., 1997). Here we investigate the source
of recognition signals for a social insect with complex colony
structure. We show that two ant populations with different
social structures have correspondingly different sources of
recognition cues which reflect their ecological backgrounds.
Therefore, kin recognition should be studied with explicit reference to the ecological context within which the recognition
behavior is expressed.
The hymenoptera comprise a continuum from solitary life
to tightly-structured family groups (Wilson, 1971). In the highly social forms, colonies are family groups living in a central
nest location. Because the nest represents the focus for most
interactions, students of social insects focus on the ability to
Address correspondence to J. M. Herbers, who is now at the Department of Biology, Colorado State University, Fort Collins, CO
80523. E-mail: [email protected]. R. J. Stuart is now at the
University of Florida, Citrus Research and Education Center, 700 Experiment Station Road, Lake Alfred, FL 33850, USA.
Received 3 November 1999; revised 18 May 2000; accepted 25 May
2000.
2000 International Society for Behavioral Ecology
recognize nest mates rather than on broader kin recognition
abilities. The distinction can be illustrated by the behavior of
honeybees which reproduce by binary fission during swarming; the daughter colonies retain close kin relationships, but
they acquire distinct recognition cues within a few days of
swarming (Breed et al., 1998). Here we introduce another
complication by considering insects whose colonies can be
physically subdivided; when a colony is fractionated into multiple nesting sites, the ability to recognize colonial relatives
must transcend the confines of the nest itself.
Recognition behavior for social insects typically involves active antennation of individuals seeking entrance to the nest;
when a stranger is recognized, agonistic behavior ensues. Aggression towards non-nest mates is provoked almost exclusively by chemical cues that are learned shortly after eclosion
from the pupal life stage (Alloway and Hare, 1989; Gamboa,
1995; Smith and Breed, 1995). Two species have been especially well-studied with respect to their nest mate discrmination abilities, the honey bee Apis mellifera (Breed, 1998; Getz,
1991) and the paper wasp Polistes fuscatus (Gamboa, 1995).
Both have sophisticated recognition abilities based on olfactory cues emanating from the exoskeletal cuticle. Cuticular
hydrocarbons serving as signals for nest mate recognition are
absorbed by the insects from their nest environment (comb
wax and paper, respectively [Breed et al., 1995; Gamboa et al.,
1996]). Because nest materials derive from sources outside the
colony, subtle differences in the resource bases used by different colonies result in different blends of cuticular hydrocarbons. Thus exogenous signals have priority for nest mate
recognition in these species, but there is evidence that endogenous cues (i.e., genetic differences) can mediate recognition as well (Breed, 1998; Gamboa et al., 1996; but see
Downs and Ratnieks, 1999).
The literature on nest mate recognition in ants is scattered,
Stuart and Herbers • Nest mate recognition in ants
677
Table 1
Size of the database we analyzed
Total number of tests
Season of
collection
No. of
nests in plot
No. of
nests tested
Round
one
New York
1
2
3
4
5
6
7
8
9
10
11
12
TOTAL
Spring
Spring
Spring
Summer
Summer
Summer
Autumn
Autumn
Autumn
Winter
Winter
Winter
22
3
87
19
20
35
7
23
28
16
31
112
403
7
0
15
16
16
16
6
19
19
11
13
33
171
54
0
78
126
102
91
42
110
119
72
103
215
1112
West Virginia
1
2
3
4
5
6
7
8
9
10
11
12
TOTAL
Spring
Spring
Spring
Summer
Summer
Summer
Autumn
Autumn
Autumn
Winter
Winter
Winter
3
8
18
20
7
7
18
5
7
15
6
5
119
3
5
14
17
7
4
17
3
7
15
6
4
102
12
30
84
151
55
20
125
12
56
117
41
20
723
Plot
with only a few studies on any given species (Breed and Bennett, 1987; Jaisson, 1991). Even so, the ability of ants to recognize nest mates has been well-established, and olfactory
cues mediate that recognition; both endogenous and exogenous cues have been described (Beye et al., 1998; Breed et
al., 1992; Crosland, 1990; Morel et al., 1990; Stuart, 1988a).
An additional complication for studying nest mate recognition
in some ant species is their complex colony structure. Colonies can consist of physically discrete local nesting units, sometimes separated from each other by several meters, a condition termed polydomy (Alloway et al., 1982). Spatial substructuring in polydomous colonies requires workers to move frequently among different nest sites to maintain intra-colony
communication (Herbers, 1986; Stuart, 1985), and identifying
colony boundaries in these species requires a synthesis of data
from behavioral aggression tests and genetic characterization
(Banschbach and Herbers, 1996).
In some species, polydomy can lead to fission reproduction.
Should a colony containing more than one queen be separated into subunits, any subunit containing a queen can become independent over time. This process is functionally
equivalent to swarming in honeybees, but with a longer time
scale. In honeybees, new recognition cues to differentiate between nest mates are acquired within a few days of swarming
(Breed et al., 1998). By contrast, polydomy that leads to fission
reproduction is relatively passive, with new colonies gradually
becoming behaviorally independent over time (Crozier et al.,
1984; Pedersen and Boomsma, 1999).
We hypothesize that for polydomous species, a genetic component to nest mate recognition should predominate. Exogenous cues acquired from environmental sources should be
less important in these species, because they could interfere
Round
two
128
74
89
42
98
119
550
12
30
84
133
56
20
126
12
57
120
42
20
712
with recognition between colony members living in different
domiciles. To test this hypothesis, we examined nest mate recognition in two populations of the tiny forest ant Leptothorax
longispinosus, which exhibit important differences in social
structure. Colonies in a New York population vary in queen
number, with some nests having no queens, others having
one, and still others having more than one; this syndrome of
variable queen number is known as facultative polygyny (Herbers, 1984). By contrast, colonies in a West Virginia population rarely have multiple queens. In New York, colonies split
into multiple nesting units, and there is an annual cycle of
colony fractionation and re-coalition called seasonal polydomy; this accounts for a high frequency of queenless nests,
which are in fact colony subunits. In West Virginia, colonies
are rarely polydomous, and queenless nests tend to be orphans (Herbers and Stuart, 1996a). These differences in social structure cause a cascade of other effects, notably on reproduction and sex ratios (Herbers and Stuart, 1996b), and
so we suspected they might differ in their nest mate recognition as well. In particular, we hypothesized that complex
colony structure in the New York population would be accompanied by an enhanced role for endogenous sources of nest
mate recognition cues there.
METHODS
Nests of L. longispinosus were collected from two sites in
1987–1989. The New York site (NY) is located in the E.N.
Huyck Preserve (Albany County) and the West Virginia population (WV) at Watoga State Park (Pocahontas County). Our
study sites and the ant communities living therein have been
Behavioral Ecology Vol. 11 No. 6
678
described elsewhere (Herbers, 1989; Herbers and Stuart,
1996a,b).
At each site, a series of 49-m2 quadrats was set up and then
completely excavated and mapped; three quadrats excavated
in each of the four seasons provided the ants for our tests.
Nests of L. longispinosus were bagged and returned to the
laboratory, where they were settled into 10 mm ⫻ 2 mm glass
tubes. These nest tubes were placed in nest boxes and maintained according to standard conditions (Herbers, 1984). The
occupants were censused periodically and fed a combination
of ant food (Bhatkar and Whitcomb, 1970) and frozen fruitflies, with water provided ad libitum. The spatial structure and
genetics of these collections have been explored in detail elsewhere (Herbers, 1989; Herbers and Stuart, 1996a).
Within each plot, we identified a series of focal nests, using
only nests that had at least ten workers. For each nest, we
identified its nearest neighbors in the plot (up to the sixth
nearest neighbor). We then conducted trials between each
focal nest and its neighbors, with focal nests receiving intruders from test neighbors as described below. Because we tested
within plots, many nests served as focal nests and as neighbors
to other focal nests. We also had two sets of controls: selfintroductions and introduction to/from ‘‘alien’’ nests that
were collected at least 100 m from the focal nest. We therefore
had four classes of tests: (1) nests collected from the same
plot (pairwise introductions); (2) self-introductions; (3) introduction of alien workers into focal nests; and (4) introduction
of ants from focal nests into alien nests. Our experiment,
then, conformed to a split–split–split plot design (focal nests
within plots within seasons within site), with repeated measures (neighbors) at the lowest level.
Within 3 weeks of collection, the first round of aggression
tests was initiated. We repeated aggression tests 3 months after
collection, to ascertain whether signals used for colony recognition changed over time; these tests were designated collectively as round two. In both rounds, the protocols were
identical. A single worker from a donor nest was selected; we
used foragers outside the nest if possible, and only used workers with dark exoskeletons to ensure we had older workers.
We chilled the ant to anesthetize her, and then placed a loop
of polyester thread around her alitrunk (Stuart, 1986). The
thread was glued to a thin stick such that the tethered ant
could be gently walked into the nesting tube of a recipient
nest. The donor ant was left in the recipient nesting tube until
either 3 min had elapsed or until a clear aggressive response
had been elicited. A different donor worker was used for each
introduction, and a given nest was not designated as recipient
more than twice on the same day.
During trials, we videotaped the ensuing behavior both of
the donor ant and of workers in the recipient nest, characterizing the interaction along three axes of grooming, biting,
and stinging. From those raw observations an aggression score
was developed on a scale of zero to four. A score of zero meant
the introduction was completely amicable, with the donor entering and being groomed by the recipients. A score of one
meant very mild aggression, with a short bout of dragging the
donor ant or a recipient opening her mandibles. A score of
two indicated escalated aggression with open mandibles directed toward the donor, often accompanied by dragging or
biting. If there was prolonged dragging, with actual biting and
attempts to sting, a score of three was given. Scores higher
than three were given for attempts to maim or kill the donor
ant. All experiments were done blind to mitigate observer
bias: one investigator tracked the identity of donor and recipient nests, while a second scored the ensuing behavior.
Because we tested within plots, many nests served as focal
nests and as neighbors to other focal nests. After all data had
been collected, we characterized interactions between a pair
Figure 1
Because density was different in our two study sites, distances
between neighbors were lower in New York (solid bars) than in
West Virginia (open bars).
of nests as friendly (both introductions with scores less than
two); hostile (both introductions scoring two or higher); or
asymmetric (one introduction scoring below two and the reciprocal test scoring two or above). Clearly, the reciprocal tests
were not independent, nor were the repeated observations
between focal nests and their neighbors. For the smallest data
sets (plots with seven or fewer nests) we were able to use matrix correlation methods to correct for nonindependence.
With denser plots, however, our focal nest design produced
many missing cells in distance matrices, thereby precluding
use of matrix correlations. Rather, in order to achieve independence within our data sets, we used each pair of nests only
once in statistical analysis, randomly assigning the status of
donor and recipient to members of the pair. Therefore our
fundamental unit for data analysis was a pair of nests, and any
given pair appeared only once in the data.
Data on allozyme variability at five polymorphic loci were
used to infer genetic structure within and between nests. Because allozyme markers have low resolution, we combined information from all five loci to compute a genetic similarity
index (for details see Herbers and Stuart, 1996a). This index
varied from zero (genetic profiles completely incompatible
between two nests) to one (genetic profiles identical between
the two nests). We included these values, worker and queen
numbers in both donor and recipient nest, and spatial location of nests in our database.
RESULTS
Table 1 shows the size of the database we built from aggression
tests. Between sites, New York, had significantly higher nest
density than West Virginia (K-W test, G ⫽ 8.2, 1 df, p ⬍ .01),
yielding significant differences in distances to nearest neighbors as well (Figure 1; ANOVA, effect of site p ⬍ .01). Nests
in WV were typically isolated from their nearest neighbors by
more than a meter; in NY only about 60 cms separated the
average nest from its nearest neighbor. Because differences in
density introduced differences in distance to neighbors, we
are careful to distinguish neighbor rank from absolute spatial
distance below.
Logistically, we were able to conduct a second round of
aggression tests for only half the sampled plots from NY; thus
comparisons between round one and round two are available
Stuart and Herbers • Nest mate recognition in ants
679
Figure 2
Map of nest locations on a 7 m ⫻ 7 m plot that was fully excavated,
with results of introductions between different nests. On this
relatively uncrowded plot there were four nests (A–D), and all six
reciprocal tests were conducted. The numbers indicate aggression
scores for an introduction from the distal nest into the proximal
nest. For example, the introduction of a worker from A into D
elicited aggression of two, while the reciprocal introduction elicited
aggression of score 2.5.
only for summer and autumn collections. By contrast, we were
able to conduct both rounds of tests for all the plots collected
in West Virginia (Table 1). Totals of 171 nests from New York
and 102 nests from West Virginia were used; respectively, 1662
and 1435 tests were conducted, including self-introductions,
pairwise tests, and tests with aliens. To our knowledge, these
represent the largest sample sizes for aggression tests available
for any ant species.
Overall patterns
Raw data for an uncrowded plot are shown in Figure 2. For
this plot, we conducted all six possible pairwise introductions
between nests, and the bi-directional arrows in Figure 2 show
those aggression scores. We also conducted self-introductions
and tests with aliens for these nests (data not shown), yielding
a total of 20 tests altogether in this plot (Table 1). The data
in Figure 2 illustrate that in general introductions were symmetric; if the introduction of a worker from nest i elicited
aggression from workers in nest j, usually we recorded aggression for the reverse introduction as well. On that particular plot, we found one friendly pair (BC), four hostile pairs
(AB, AD, BD, CD), and one asymmetric pair (AC).
Table 2 gives statistics for the four classes of introductions,
with data pooled over all tests. Self-introduction controls, by
which a worker was tethered and reintroduced into her own
nest, were almost uniformly friendly (Table 2). With rare exceptions, workers were allowed re-entry to their own nest without the slightest display of hostility, and usually with considerable grooming (medians of zero in both sites, and arithmetic means very close to zero). Thus our manipulation of
workers by tethering did not appear to induce aggressive behavior as artifacts. Conversely, our controls that included
‘‘aliens’’ typically elicited hostile behavior (Table 2). There
were a few friendly introductions involving aliens from each
population (minimum scores in Table 2), but these were rare.
Median scores were 3.0 in NY and 2.5 in WV for introductions
involving aliens, showing that aggression was high between
ants living in different parts of the forest. Taken together, all
these results support our contention that aggression reflected
the degree of familiarity between ants. Our experimental protocol, by which we gently introduced ants from one nest into
the entrance of the focal nest, mimicked the natural situation
when guard ants encounter returning foragers. Individuals
seeking entrance to a nest must pass antennal inspection, and
presumably must present the appropriate combination of olfactory cues. We had very low frequencies of apparent errors
in our control trials involving self-introductions and introductions into/from alien nests, which strongly suggests that our
observations reflected natural behavior. We proceed then to
interpret results from the pairwise trials in that light.
Introduction of ants from different nests collected from the
same plot elicited a range of reactions (pairwise interactions
in Table 2). On average, pairwise interactions were much
more aggressive than self-introductions (Friedman’s tests, p ⬍
.0001), yet in both sites perfectly friendly introductions of ants
into other nests were observed (minimum scores of 0 for pairwise tests). There was no difference between the sites in average pairwise aggression scores (medians of 2.5 in both NY
and WV).
In the NY population, introductions to/from alien nests
Table 2
Summary of results for the first round of aggression tests
a
N of
tests
Mean score
(⫾ s.e.)
Median
Minimum Maximum score
New York
Self-introductions
Pairwise tests
Introduction into alien nesta
Introduction of an alien
141
599
139
140
0.08
2.35
2.60
2.66
⫾
⫾
⫾
⫾
0.029
0.036
0.054
0.044
0
0
1.0
0
3.0
3.5
3.5
3.5
0
2.5
3.0
3.0
West Virginia
Self-introductions
Pairwise tests
Introduction into alien nesta
Introduction of an alien
98
330
114
116
0.01
2.40
2.44
2.57
⫾
⫾
⫾
⫾
0.009
0.049
0.059
0.059
0
0
0.5
0
0.5
4.0
3.5
3.5
0
2.5
2.5
2.5
Alien nests came from at least 100 m away from focal nests, and ‘‘pairwise’’ refers to tests between
nests collected from the same plot.
680
Behavioral Ecology Vol. 11 No. 6
elicited stronger aggression on average (median of 3.0) than
did pairwise introductions (median of 2.5; K-W test, H⬘ ⫽
10.9, 1 df, p ⬍ .0001). On average, then, pairwise interactions
were friendlier in NY than were alien interactions. Conversely,
both groups had identical median scores of 2.5 for the WV
population (Table 2); tests involving aliens and pairwise tests
provoked the same levels of aggression on average. We interpret the difference between sites to reflect their different social structures. Because polydomy is common in NY but rare
in WV, pairwise tests in NY had a higher frequency of zeroes.
Effects of distance and genetic similarity
The aggression scores we collected represented up to six pairwise introductions for each focal nest. Each focal nest was
collected from a particular plot, which was part of a seasonal
collection within each site. Unfortunately, we were unable to
normalize the aggression scores via standard transformations,
and thus could not use the hierarchical parametric approach
that fit our experimental design.
Our principal goal was to sort out the effects of distance
and genetic similarity on aggression scores. We first examined
data from six plots small enough to allow all pairwise interactions. For those plots, we constructed three matrices, one
with the aggression data, a second with spatial distance, and
the third with genetic similarity data; we then tested for correlations among the matrices via Mantel tests (cf. Manly,
1995). Those tests showed a significant positive correlation
between distance and aggression for one plot (G ⫽ 2.12 for
WV 11, p ⬍ .05), and significant negative correlations between
genetic similarity and aggression for two plots (G ⫽ ⫺2.34 for
NY 7 and G ⫽ ⫺2.18 for WV 11, both p ⬍ .05). Therefore,
even for small data sets with low power (since Mantel tests are
conservative), we found evidence for effects of both genetic
and spatial distance on the aggression between ant nests. For
larger datasets, we were unable to perform Mantel tests due
to the large number of missing cells. Rather, we applied nonparametric methods and used each pair of nests only once.
Below we report on patterns for data pooled across plots and
seasons, because preliminary analysis showed such pooling
did not obscure important patterns.
In Figure 3 we show average aggression scores as a function
of neighbor rank (nearest neighbor has rank one, secondnearest has rank two, and so on). Aggression changed with
neighbor rank in both sites (Figure 3a; K-W tests, H⬘ ⫽ 13.26
and 13.75 respectively, each with 5 df; p ⬍ .05 for each), due
to lower aggression towards the first nearest neighbor. Aggression scores among higher neighbors were no different
(nonparametric multiple contrasts; in both sites p ⬍ .05 for
differences between the first neighbor and all others; p ⬎ .05
for differences among neighbors two through six). That is,
focal nests were less aggressive towards their closest neighbor
than to other nests in the vicinity. We saw the same pattern
in round two, which was conducted several months later (Figure 3b): aggression towards the nearest neighbor was significantly less than aggression towards further neighbors, but
there were no differences among second through sixth neighbors (K-W tests with multiple contrasts; p ⬍ .05 for the first
neighbor, and p ⬎ .05 among higher-ranking neighbors).
Thus reduced aggression was associated with close spatial
proximity.
Overall, genetic similarity values were higher in NY than
those in WV (K-W test, H⬘ ⫽ 47.9, 1 df, p ⬍ .0001); this result
reflects colony subdivision in NY. Neighbor rank was not correlated with genetic similarity in either population, however
(Spearman’s rank correlations; in NY ␳ ⫽ ⫺0.07 and in WV
␳ ⫽ ⫺0.06; for both p ⬎ .05). That is, the first nearest neighbor was no more similar or dissimilar genetically to the focal
nest than more distant neighbors. On the other hand, genetic
Figure 3
Aggression scores changed with neighbor rank in both sites and in
both rounds of tests. To analyze these data, we used a given pair of
nests only once; as a result, sample sizes are sometimes smaller for
close neighbors (where pairs tended to be each others’ neighbors)
than for further neighbors. (A) Aggression scores from round one,
shortly after collection. (B) Aggression scores from round two, after
3 months of laboratory domestication. Solid bars, New York; open
bars, West Virginia.
similarity was inversely related to absolute spatial distance between nest pairs in New York (Spearman’s ␳ ⫽ ⫺0.248, p ⬍
.0001); no such correlation existed for West Virginia (␳ ⫽
⫺0.058, p ⬎ .05). We interpret the fact that absolute spatial
distance but not neighbor rank was correlated with genetic
similarity in NY to mean that polydomous colonies intercalated. To be sure, colony subunits were closer to each other than
two randomly chosen nests, but they were not necessarily each
other’s closest neighbors. By contrast, the lack of any correlation between spatial distance and genetic similarity in WV
reflected the low frequency of colony subdivision there.
The above analysis shows that we could examine the effects
of spatial distance and genetic similarity independently for
WV aggression data. For NY data, however, the two were confounded. To tease them apart, we analyzed the aggression data
via nonparametric partial correlation analysis (Siegal and Castellan Jr., 1988). The coefficients in Table 3 reinforce our conclusion above that distance and genetic similarity were confounded in NY (partial correlation ⫽ ⫺0.225, p ⬍ .001) but
Stuart and Herbers • Nest mate recognition in ants
681
Table 3
Partial correlation coefficients and sample sizes
New York
Distance
Genetic
similarity
0.071
(473)
0.110
(201)
⫺0.225b
(473)
⫺0.133b
(473)
⫺0.164a
(201)
Distance
Round one
aggression
Round two
aggression
West Virginia
Distance
Genetic
similarity
0.126a
(303)
0.245c
(308)
⫺0.046
(303)
⫺0.170b
(303)
⫺0.283c
(308)
0.01 ⬍ p ⬍ .05.
0.001 ⬍ p ⬍ .01.
c p ⬍ .001.
a
b
not in WV (partial correlation ⫽ ⫺0.046, p ⬎ .05). In NY,
only genetic similarity gave a significant partial correlation
with aggression in round one (Table 3). By contrast, both
distance and genetic similarity were correlated with round
one scores in WV. The same pattern was observed for round
two aggression (Table 3). These results, which are based on
large sample sizes and thus have high power (Table 3), show
that for the NY population only genetic similarity influenced
aggression scores, whereas in WV both distance and genetic
similarity did.
Effects of demography
We wondered whether demographic factors affected our aggression scores. Because we had multiple tests with each focal
nest, we avoided pseudoreplication by using only aggression
towards the first nearest neighbor for this analysis. We found
no effects of queen number or worker number in the donor
nest on aggression shown to an intruder from that nest (Kruskal-Wallis tests, p ⬎ .05). That is, the demographic background of an intruder worker’s home did not affect how she
was treated by a recipient nest. Similarly, we found no effect
of worker number of the recipient nest on aggression: small
and large nests alike reacted similarly to intruders. However,
we did find an effect of queen number in the recipient nest
for both populations (Figure 4); workers in queenless nests
were less aggressive towards intruders than were workers in
monogynous and polygynous nests (K-W tests; H⬘ ⫽ 7.44, 2 df
for NY; and H⬘ ⫽ 6.44, 1df for WV; both p ⬍ .05). In general,
Figure 4
Aggression increased significantly with the number of queens
residing in the recipient nest. Numbers refer to the total number of
pairwise tests conducted between first-rank neighbors. Solid bars,
New York; open bars, West Virginia.
then, nest demography did not influence aggression, with the
important exception of recipient queen number.
Reciprocity of interactions
Many nests served as both donor and recipient to each other;
that is, we had a number of pairs for which we could assess
reciprocity of aggression. These reciprocal tests gave the most
complete picture of between-nest aggression possible, and so
we examined them in detail. In general the aggression of ants
in nest j towards an introduced ant from nest i was correlated
with the aggression of ants in nest i towards an introduced
ant from nest j (Kendall’s ␶ ⫽ 0.24 for NY and 0.21 WV, p ⬍
.0001 for both). Those correlations provide support for our
interpretation that we were in fact measuring some aspect of
nest mate recognition in our assay.
We characterized nest pairs as friendly, hostile, or asymmetric in each round (Table 4). Most pairwise interactions
were hostile, and fewer than 10% were friendly. We were surprised to find that about 16% of all interactions were asymmetric; that is, introduction of i into j was peaceful whereas
the reverse was not. These asymmetries were equally common
Table 4
Frequencies of outcomes in aggression tests
a) Pairwise interactions
Round one
WV
NY
Round two
WV
NY
b) Tests with aliens
Round one
WV
NY
Round two
WV
NY
Friendly
Hostile
Asymmetric
G-tests
21 (6.4%)
49 (8.3%)
252 (77.1%)
445 (75.0%)
54 (16.5%)
99 (16.7%)
G ⫽ 1.09
p ⬎ .05
12 (3.7%)
17 (6.3%)
291 (89.5%)
220 (81.2%)
22 (6.8%)
34 (12.5%)
G ⫽ 8.5
p ⬍ .05
1 (0.9%)
1 (0.8%)
93 (81.6%)
122 (89.0%)
20 (17.5%)
14 (10.2%)
G ⫽ 2.86
p ⬎ .05
0 (0%)
0 (0%)
107 (99.1%)
61 (87.1%)
1 (0.9%)
9 (12.9%)
G ⫽ 11.9
p ⬍ .001
Behavioral Ecology Vol. 11 No. 6
682
in the two sites for round one (Table 4; G-test, p ⬎ .05), and
we return to them below.
Those pairs that were friendly in round one tended to be
first neighbors to each other (data not shown; G-tests with five
df; for NY, G ⫽ 38.9 and for WV, G ⫽ 30.3; both p ⬍ .001).
Indeed, the average distance between friendly pairs was considerably smaller than the average distance between hostile
pairs (Figure 5; ANOVA, p ⬍ .001 for both sites in round
one). We reclassified pairs based on their interactions in
round two (Table 4) and found the same patterns: nests that
were friendly tended to be near neighbors, and were significantly closer to each other than hostile or asymmetric pairs
(Figure 5; ANOVA, p ⬍ .001 for both sites). The data in Figure
5 also confirmed our earlier inferences: friendly pairs were
significantly more similar genetically than hostile or asymmetric pairs (K-W tests; H⬘ ⫽ 15.9 for NY and 14.8 for WV;
both with 2 df and p ⬍ .001). The same pattern was observed
in round two but was significant only in WV (H⬘ ⫽ 26.3, 2 df,
p ⬍ .0001; in NY, H⬘ ⫽ 4.93, 2 df, p ⬎ .05).
We were intrigued by pairs of nests that gave asymmetric
aggression scores. In these pairs, one nest was friendly to an
intruder but the reciprocal introduction yielded an aggressive
response. We first looked to see if particular nests were overrepresented in this category: from the frequency of asymmetries (Table 4) we generated a Poisson distribution, and then
compared the number of times a particular nest was classified
as part of an asymmetric pair to that distribution. The distribution of asymmetries per nest conformed to Poisson expectation (data not shown), meaning that individual nests were
involved in asymmetric interactions according to chance expectation. We examined our data further to see if demographic characteristics could distinguish asymmetric pairs from others, but found no such effects (G-tests on queen numbers and
ANOVA on log-transformed worker numbers; in all cases p ⬎
.05). Taken together, our data show that asymmetric pairs
could not be distinguished in any way from mutually hostile
pairs.
To explore the problem of asymmetric pairs more closely,
we categorized interactions with alien ants (Table 4). Only
one test involving aliens in each site was friendly, and those
occurred in round one (Table 4). This result confirms that
our tests with aliens were solid controls. Even so, tests involving aliens were asymmetric in 10.2% of the NY trials and in
17.5% of the WV trials, proportions that were not significantly
different (G test, p ⬎ .05). Asymmetries involving aliens were
rarer than those involving pairs (compare results of pairwise
interactions with results of tests with aliens, Table 4; G ⫽ 20.0
in NY and 7.4 in WV, both with 2 df; p ⬍ .0001 and p ⬍ .05,
respectively). In round two, asymmetries involving aliens were
very rare for the WV data, but represented 12.9% of all the
NY trials.
Round one versus round two
Finally, we compared interactions of nest pairs tested right
after collection and again after three months of laboratory
domestication (Figure 6). There was an overall positive correlation between aggression scores for the same pairs of nests
between rounds (Spearman’s ␳ ⫽ 0.24 in NY and 0.42 in WV;
p ⬍ .001 for both sites), but scores were higher in round two
than in round one (sign tests, p ⬍ .05 for both sites). Furthermore, median aggression between pairs was higher in
round two (Figure 6; K-W tests; in NY H⬘ ⫽ 10.6 and in WV
H⬘ ⫽ 49.2 with 1 df, p ⬍ .01 for both). For WV nests, both
sets of control introductions (involving aliens and self-introductions) also yielded higher aggression scores in round two
than they had in round one (p ⬍ .05); for the NY nests, however, aggression in control trials did not change between
round one and round two (p ⬎ .05).
A pattern of increased aggression after lab domestication
was evident from the classification of individual interactions
as well (Table 4). The proportion of hostile pairwise interactions rose in round two (Table 4, p ⬍ .01 for each site), and
there was a higher frequency of hostility for WV pairs than
for NY pairs (p ⬍ .05). Similarly, tests involving aliens in round
two provoked a higher frequency of hostile interactions than
had been seen in round one (Table 4). Further analysis
showed that only 59% of pairs that were characterized as
friendly in round one were still friendly 3 months later (Figure 7); one-fourth became hostile, and the remainder showed
asymmetric behavior in round two. Furthermore, 84% of all
nests that showed asymmetric aggressive behavior in round
one became hostile after a lapse of three months, and only
3% became friendly. Only one pair of nests that had been
hostile in round one became friendly in round two, and only
8% made the transition to being asymmetric (Figure 7). The
transitions frequencies in Figure 7 deviate strongly from
chance expectation (G ⫽ 163.4, 2 df, p ⬍ .0001), with friendly
pairs remaining friendly and hostile pairs remaining hostile
at a higher than expected frequency. By contrast, asymmetric
pairs remained asymmetric at a rate lower than chance expectation (G ⫽ 7.5, 2 df, p ⫽ .02), with the majority becoming
hostile (Figure 7). The overwhelming pattern then was for
pairs of nests to become more hostile to each other after 3
months had elapsed in the laboratory.
DISCUSSION
The ability to discriminate kin, while widespread, need not
imply the operation of kin selection (Barnard, 1991; Grafen,
1990). Indeed, confirming that kin selection is the best explanation for kin discrimination requires rigorous testing of alternatives (Pfennig et al., 1999), and evidence linking kin discrimination to kin selection is surprisingly thin (Sherman et
al., 1997). Even so, we can safely look to species with welldeveloped social systems that have themselves been shaped by
kin selection in order to infer the evolutionary basis of kin
recognition. Not surprisingly, then, highly social species that
live in family groups have been especially well-studied with
respect to their ability to recognize kin (Crozier and Pamilo,
1996). Our work fits squarely into the large body of research
on social insects by demonstrating that ecological factors affecting colony structure can exert strong influences on how
kin recognition is achieved.
Most authors have characterized recognition systems within
a single population. We have shown here that between-population variation in nest mate recognition gives important insight to the ecology of recognition systems (see also Morel et
al., 1990). Our use of two populations that differ in social
structure uncovered important differences in the way they use
cues to distinguish between nest mates and strangers. Our
West Virginia and New York populations differ in nest density,
frequency of polydomy, and prevalence of polygyny. These differences produce higher within-nest relatedness and lower between-nest relatedness in WV relative to NY (Herbers and Stuart, 1996a). Furthermore, colonies are more tightly packed in
NY, which produces spatially intercalated polydomous colonies. As a consequence, genetic similarity was of overwhelming importance to predicting between-nest aggression there.
Use of exogenous cues to distinguish nest mates in NY would
lead to high error rates: environmentally derived cues are
most similar over short distances, and the intercalation of polydomous colonies could lead to odor similarity among nonrelatives and dissimilarity among relatives. Thus the combination of polydomy and tight colony packing should have selected for endogenous cues having priority, as we found. Conversely, in WV ants used cues correlated both with genetic
Stuart and Herbers • Nest mate recognition in ants
683
Figure 5
Distances were lowest (A) and
genetic similarity was highest
(B) for friendly pairs of nests
than for those that were hostile
or showed asymmetric aggression. Data from round one are
given in top panels and data
for round two in bottom panels. Sample sizes are given in
Table 4.
Figure 7
Transitions of interactions between round one and round two. For
each pair of nests that was tested, we characterized their interaction
as friendly, asymmetric, or unfriendly in each round, and then
examined the data for state changes. The strong majority of
transitions were to a more aggressive state in round two than had
been observed in round one. Data were pooled across populations,
since the patterns of transitions were similar.
Figure 6
Overall aggression scores for introductions between nests shortly
after collection (round one) and 3 months later (round two). The
top panel reports data from tests involving aliens; the middle panel
for nests collected within 6 m of each other; and the bottom panel
for workers re-introduced into their own nests.
similarity and distance between nest pairs. Given that colonies
have simple structure there and are widely spaced, exogenous
cues have greater relevance for nest mate recognition.
Despite the differing priority of endogenous cues between
the two populations, exogenous cues must have contributed
to aggression both in NY and WV. After 3 months of lab domestication, overall aggression rose and pairs that had been
friendly became hostile. We interpret that clear pattern to
show that repeated interactions in the field, which we prevented in the lab, are crucial for maintaining recognition
among subunits of polydomous colonies. Thus, while endogenous cues are important, they must be reinforced with ex-
Behavioral Ecology Vol. 11 No. 6
684
ogenous cues. Clearly, an understanding of population structure is essential for interpreting data on aggression between
non-nest mates.
In general, environmental influences are extremely important for predicting aggressive interactions between colonies of
social insects (Smith and Breed, 1995). Nesting material is an
important source of recognition cues for wasps and bees, but
rarely plays a role in producing recognition cues for ants
(Gamboa, 1995; Smith and Breed, 1995; but see Heinze et al.,
1996). Demographic variables can influence nest mate recognition; for example, variable queen number has been
linked to differential aggression in Leptothorax lichsteini (Provost, 1989) and Solenopsis invicta (Morel et al., 1990), but was
excluded as important for Leptothorax ambiguus (Stuart,
1991) and Rhytidoponera confusa (Crosland, 1990). Similarly,
colony size (i.e., worker number) was correlated with aggression in L. ambiguus (Stuart, 1991) but not R. confusa (Crosland, 1990). We have shown that in general, nest demography
had few effects on nest mate recognition in Leptothorax longispinosus. The exception was that queenless nests, presumably subunits of polydomous colonies, were less aggressive to
neighbors than nests containing queens. Again, a thorough
understanding of the social system is necessary to interpret
demographic effects on recognition systems.
Hostile versus friendly behavior among pairs of nests was
most strongly dependent on two factors, distance and genetic
similarity. These two were themselves strongly correlated in
our NY population, which makes its nest mate recognition
system similar to that of F. pratensis (Beye et al., 1997, 1998).
Studies on a European Leptothorax species (Heinze et al.,
1996) also implicated spatial distance between nests; whether
or not genetic similarity was confounded with distance in that
study is unknown. Certainly, our discovery of inter-population
differences in the importance of distance and genetic similarity mandates a thorough exploration of their priority in future
studies.
The pairs of nests for which we recorded asymmetric interactions are of special interest. While we cannot rule out the
possibility that those asymmetries are artifacts, the data suggest otherwise. We were unable to identify any demographic,
spatial, or genetic variable associated with asymmetric pairs.
This contrasts with paper wasps, for which genetic relationships predict asymmetric aggressive behavior; for example,
nieces tolerate aunts but aunts do not tolerate nieces in Polistes fuscatus (Bura and Gamboa, 1994). Furthermore, the vast
majority of pairs classified as asymmetric in round one were
classified as hostile in round two (Figure 7; p ⫽ .02). We therefore suspect that asymmetries represented mistakes of recognition by the tolerant nest of the pair. Our interpretation that
most asymmetries involved recognition mistakes is reinforced
by their presence among tests that included aliens (Table 4);
any nest not showing aggression to an alien intruder clearly
made a mistake. Whether or not such errors occur in the field
at the rates we found them in the laboratory awaits future
study.
Between-nest aggression in our study species shows many
similarities to its close relatives L. curvispinosus and L. ambiguus (Stuart, 1987a,b,c, 1988a,b, 1991, 1992). Stuart (1988a)
has established that L. curvispinosus uses a collective odor
system to recognize nest mates, and our data echo many results of his experiments. There are some important differences, however, among the species. Most striking, Stuart (1987c)
found that aggression between nests of L. curvispinosus waned
over time in the laboratory, which he ascribed to the loss of
environmentally-based colony recognition odors as nests were
kept on a uniform diet. We found, on the other hand, that
aggression increased as nests were held in the lab, which we
also ascribe to the loss of collectively-held colony-specific
odors. Nest mate recognition cues still important shortly after
collection (round one) apparently disappeared by round two
(Figure 6). To explain these differences, we postulate that for
L. longispinosus, repeated interactions among nest subunits
are needed to maintain colony integrity in the field. Exchange
of recognition cues among workers from different nest subunits is maintained after colonies break up in the spring, either by grooming or trophollaxis, both of which have been
observed in laboratory reconstructions of polydomous colonies (Herbers and Tucker, 1986). When that exchange of cues
is interrupted by physical segregation in the laboratory, colony
recognition between nest subunits can no longer be effected,
and introductions at a later date elicited aggressive behavior.
We have shown here that recognition systems in these little
ants reflect their ecological background. Nest mate recognition in L. longispinosus appears to be but one dimension of
a complex social system that responds to environmental parameters. Our results show that a full understanding of social
ecology can allow accurate predictions about how nest mate
recognition is mediated. Indeed, we assert that future studies
of nest mate recognition in social insects are incomplete unless they are placed squarely within the context of the species’
social organization.
We thank Vickie Backus, Laura Snyder, and Linda Prince for assistance with fieldwork and aggression experiments. Chris DeHeer, Susanne Foitzik, Tim Judd, and an anonymous reviewer made many
suggestions for improving the manuscript. This work was supported
by grants from Vermont EPSCoR and the National Science Foundation.
REFERENCES
Alloway TM, Buschinger A, Talbot M, Stuart R, Thomas C, 1982. Polygyny and polydomy in three North American species of the ant
genus Leptothorax Mayr (Hymenoptera: Formicidae). Psyche 89:
249–274.
Alloway TM, Hare JF, 1989. Experience independent attraction to
slave-maker ant larvae in host-species ant workers (Leptothorax longispinosus; Hymenoptera: Formicidae). Behaviour 110:93–105.
Banschbach VS, Herbers JM, 1996. Complex colony structure in social
insects: I. Ecological determinants and genetic consequences. Evolution 50:285–297.
Barnard C, 1991. Kinship and social behaviour: the trouble with relatives. Trends Ecol Evol 6:310–312.
Beye M, Neumann P, Chapuisat M, Pamilo P, Moritz RFA, 1998. Nest
mate recognition and the genetic relatedness of nests in the ant
Formica pratensis. Behav Ecol Sociobiol 43:67–72.
Beye N, Neumann P, Moritz RFA, 1997. Nest mate recognition and
the genetic gestalt in the mound-building ant Formica polyctena.
Insectes Soc 44:1–73.
Bhatkar A, Whitcomb WH, 1970. Artificial diet for rearing various
species of ants. Florida Entomol 53:217–232.
Breed MD, 1998. Recognition pheremones in the honey bee. Bioscience 48:463–470.
Breed MD, Bennett, B, 1987. Kin recognition in highly eusocial insects. In: Kin recognition in animals. (Fletcher DJC, Michener CD,
eds). Chichester: John Wiley; 243–285.
Breed MD, Garry MF, Pearce AN, Hibbard BE, Bjostad LB, Page RE
Jr, 1995. The role of wax comb in honey bee nest mate recognition.
Anim Behav 50:489–496.
Breed MD, Leger EA, Pearce AN. Wang YJ, 1998. Comb wax effects
on the ontogeny of honey bee nest mate recognition. Anim Behav
55:13–20.
Breed MD, Snyder LE, Lynn TL. Morhart JA, 1992. Acquired chemical
camouflage in a tropical ant. Anim Behav 44:519–523.
Bura EA, Gamboa GJ, 1994. Kin recognition by social wasps: asymmetric tolerance between aunts and nieces. Anim Behav 47:977–
999.
Crosland M, 1990. The influence of the queen, colony size, and worker ovarian development on nest mate recognition in the ant Rhytidoponera confusa. Anim Behav 39:413–425.
Stuart and Herbers • Nest mate recognition in ants
Crozer RH, Pamilo, P, 1996. Evolution of social insect colonies. Oxford: Oxford University Press.
Crozier RH, Pamilo P, Crozier YC, 1984. Relatedness and microgeographic genetic variation in Rhytidoponera mayri, an Australian aridzone ant. Behav Ecol Sociobiol 15:143–150.
Downs SG, Ratnieks FLW, 1999. Recognition of conspecifics by honeybee guards uses nonheritable cues acquired in the adult stage.
Anim Behav 58:643–648.
Fletcher DJC, Michener CD, 1987. Kin recognition in animals. New
York: John Wiley.
Gamboa GJ, 1995. Kin recognition in social wasps. In: Natural history
and evolution of paper-wasps. (Turillazzi S, West-Eberhard MJ, eds).
Oxford: Oxford University Press; 161–177.
Gamboa GJ, Grudzien TA, Espelie KE, Bura EA, 1996. Kin recognition
pheremones in social wasps: combining chemical and behavioural
evidence. Anim Behav 51:625–629.
Gamboa GJ, Reeve HK, Holmes WG, 1991. Conceptual issues and
methodology in kin-recognition research: a critical discussion. Ethol 88:109–127.
Getz W, 1991. The honey bee as a model kin recognition system. In:
Kin recognition. (Hepper P, ed). Cambridge: Cambridge University
Press; 358–412.
Grafen A, 1990. Do animals really recognize kin? Anim Behav 39:42–
54.
Hamilton W, l964. The genetical evolution of social behavior. J Theor
Biol 7:1–52.
Heinze J, Foitzik S, Hippert A, Hölldobler B, 1996. Apparent dearenemy phenomenon and environment-based recognition cues in
the ant Leptothorax nylanderi. Ethology 102:510–522.
Hepper P, 1991. Kin recognition. Cambridge: Cambridge University
Press.
Herbers JM, 1984. Queen-worker conflict and eusocial evolution in a
polygynous ant species. Evolution 38:631–643.
Herbers JM, 1986. Nest site limitation and facultative polygyny in the
ant Leptothorax longispinosus. Behav Ecol Sociobiol 19:115–122.
Herbers JM, 1989. Community structure in north temperate ants: temporal and spatial variation. Oecologia 81:201–211.
Herbers JM, Stuart RJ, 1996a. Multiple queens in ant nests: impact
on genetic structure and inclusive fitness. Am Nat 147:161–187.
Herbers JM, Stuart RJ, 1996b. Patterns of reproduction in southern
versus northern populations of Leptothorax ants (Hymenoptera:Formicidae). Ann Entomol Soc Amer 89:354–360.
Herbers JM, Tucker CW, 1986. Population fluidity in Leptothorax longispinosus (Hymenoptera: Formicidae). Psyche 93:217–229.
Jaisson P, 1991. Kinship and fellowship in ants and social wasps. In:
Kin recognition. (Hepper P, ed). Cambridge: Cambridge University
Press; 60–93.
Manly BFJ, 1995. The statistics of natural selection. London: Chapman
and Hall.
685
Morel L, VanderMeer RK, Lofgren CS, 1990. Comparison of nest mate
recognition between monogyne and polygyne populations of Solenopsis invicta (Hymenoptera: Formicidae). Ann Ent Soc Amer 83:
642–647.
Pedersen J, Boomsma JJ, 1999. Genetic analysis of colony structure in
polydomous and polygynous ant populations. Biol J Linn Soc 66:
115–144.
Pfennig DW, Collins JP, Ziemba RE, 1999. A test of alternative hypotheses for kin recognition in cannibalistic tiger salamanders. Behav Ecol 10:436–443.
Provost E, 1989. Social environmental factors influencing mutual recognition of individuals in the ant Leptothorax lichsteini Bondr. (Hymenoptera: Formicidae). Behav Proc 18:35–59.
Sherman PW, Reeve HK, Pfennig DW, 1997. Recognition systems. In:
Behavioural ecology. (Krebs JR, Davies NB, eds). Oxford: Blackwell
Science; 69–96.
Siegal S, Castellan NJ Jr, 1988. Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill.
Smith BH, Breed MD, 1995. The chemical basis for nest mate recognition and mate discrimination in social insects. In: Chemical
ecology of insects 2. (Carde RT, Bell WJ, eds). New York: Chapman
and Hall; 287–317.
Stuart RJ, 1985. Spontaneous polydomy in laboratory colonies of the
ant Leptothorax curvispinosus Mayr (Hymenoptera: Formicidae).
Psyche 92:71–81.
Stuart RJ, 1986. Use of polyester fibers to mark small leptothoracine
ants (Hymenoptera: Formicidae). J Kansas Entomol Soc 59:566–
568.
Stuart RJ, 1987a. Individual workers produce colony-specific nest mate
recognition cues in the ant Leptothorax curvispinosus. Anim Behav
35:1062–1069.
Stuart RJ, 1987b. Nest mate recognition in leptothoracine ants: testing
Fielde’s progressive odor hypothesis. Ethol 76:116–123.
Stuart RJ, 1987c. Transient nest mate recognition cues contribute to
a multicolonial population structure in the ant Leptothorax curvispinosus. Behav Ecol Sociobiol 21:229–235.
Stuart RJ, 1988a. Collective cues as a basis for nest mate recognition
in polygynous leptothoracine ants. Proc Nat Acad Sci 85:4572–4575.
Stuart RJ, 1988b. Development and evolution in the nest mate recognition systems of social insects. In: Evolution of social behavior
and integrative levels. (Greenberg G, Tobach E, eds). Hillsdale,
New Jersey: Erlbaum; 177–195.
Stuart RJ, 1991. Nest mate recognition in leptothoracine ants: testing
for effects of queen number, colony size, and species of intruder.
Anim Behav 42:277–284.
Stuart RJ, 1992. Nest mate recognition and the ontogeny of acceptability in the ant Leptothorax curvispinosus. Behav Ecol Sociobiol
30:403–408.
Wilson EO, 1971. The insect societies. Cambridge: Belknap.