A C o l l a b o r a t i o n M e c h a n i s m on Positive I n t e r a c t i o n s
in Multi-agent Environments
K e i M a t s u b a y a s h i * and M a r i o T o k o r o +
Department of Computer Science, Keio University.
3-14-1 Hiyoshi, Yokohama 223 Japan
Abstract
In m u l t i - a g e n t e n v i r o n m e n t s where agents independently generate and execute plans to
satisfy their goals, the resulting plans may
sometimes overlap. In this paper, we propose
a c o l l a b o r a t i o n mechanism using social law,
t h r o u g h w h i c h r a t i o n a l agents can s m o o t h l y
delegate and receive the execution of the overl a p p i n g parts of plans in order to reduce the
cost of plan execution. Also, we consider col
l a b o r a t i o n w i t h agents t h a t do not abide by
social law, t h a t is, self-centered agents. Simu l a t i o n results show t h a t our mechanism also
has the p r o p e r t y of balancing t he cost of plan
execution and shows f l e x i b i l i t y towards selfcentered agents.
1
Introduction
In m u l t i - a g e n t e n v i r o n m e n t s where agents generate and
execute plans w h i c h satisfy their o w n goals, plans are
generated independently, thus two relationships [ M a r t i a l , 1992] exist among plans. T h e first is called a negative relationship and exists between plans w h i c h a t t e m p t
to use the same resource at the same t i m e . T h i s rel a t i o n s h i p w i l l cause conflicts. Such conflicts must be
avoided or resolved because they prevent actions f r o m
being p r o p e r l y executed. T h e other relation is called a
positive relationship and exists between plans w h i c h i n clude o v e r l a p p i n g subplans, subplans consisting of b o t h
c o m m o n actions and i n d i v i d u a l actions. W h e n there is
a positive relationship and one agent delegates the exec u t i o n of the o v e r l a p p i n g subplan to the other agent, it
can reduce the cost of execution.
A lot of research has been done on negative relationships, such as conflict avoidance, by using a social law
[Shoham and T e n n e n h o l t z , 1992] and conflict resolution
based on game theory [ Z l o t k i n and Rosenschein, 1990].
However, despite the i m p o r t a n c e of a positive relationship for t h e agents' effective execution of plans, not much
has been made [Foulser et al., 1991] to utilize this, except to reduce costs by deleting the overlapping actions
346
Distributed AI
in a d o m a i n consisting of only one agent [Hayes, 1989]
and for multi-agent systems where i r r a t i o n a l agents delegate and receive plans [ M a r t i a l , 1992] and for m u l t i - a g e n t
systems where agents m u t u a l l y construct a c o l l a b o r a t i v e
plan [Osawa and T o k o r o , 199l] .
In this paper we propose a c o l l a b o r a t i o n mechanism
by which r a t i o n a l agents can delegate and receive subplans in a positive relationship so t h a t the cost of ex
ecution is reduced. If an agent delegates its actions to
the other agent, it can reduce the execution cost greatly
and benefit f r o m i t . On the other h a n d , when an agent
receives actions, the cost increases a l i t t l e even t h o u g h
the reduction in the cost of the sender is larger t h a n the
gain in its o w n cost. As r a t i o n a l agents can delegate actions but may not be w i l l i n g to receive, it is very h a r d
to collaborate in a positive relationship. However, if an
agent w h i c h receives actions is guaranteed to be able to
delegate some actions in the f u t u r e , t h e n even a r a t i o n a l
agent can collaborate in a positive r e l a t i o n s h i p . We propose a mechanism based on a social law (the guarantee)
in w h i c h agents must balance the costs of received actions. W i t h this mechanism, r a t i o n a l agents can collab
orate t h r o u g h the expected cost based on game theory.
Moreover, there are t w o other benefits arising f r o m
this law: (1) j u s t as agents can expect another agent's
p r o p o s i t i o n (strategy) at negotiation on delegation, they
can avoid conflicts of strategies and can collaborate
s m o o t h l y ; and (2) the macroscopic p r o b l e m t h a t arises
when the load of the whole m u l t i - a g e n t system is concentrated on an i r r a t i o n a l agent can be solved w i t h this
microscopic cost-balancing law.
A l t h o u g h the image related to the benefit of such a
law is i m p l i c i t in m a n y AI works, the social law was
only recently formalized and discussed [Shoham and T e n nenholtz, 1992]. However, this w o r k does not consider
agents w h i c h do not abide by t h e law. W h e n t r e a t i n g
a multi-agent e n v i r o n m e n t as an open d i s t r i b u t e d envir o n m e n t , it is q u i t e n a t u r a l to t h i n k t h a t such agents
exist. Therefore, our mechanism considers c o l l a b o r a t i o n
w i t h self-centered agents. It is a very simple s o l u t i o n for
agents to not collaborate w i t h self-centered agents. Nevertheless, it is problematic t h a t an agent w h i c h except i o n a l l y d i d not abide by the law for some reason, cannot
collaborate any more, and t h a t agents could benefit by
c o l l a b o r a t i n g w i t h the self-centered agents. T h e r e f o r e ,
agents should collaborate in a more r a t i o n a l and fiexi-
ble way: they collaborate if they can benefit and do not
collaborate when they cannot. O u r mechanism allows
agents to collaborate in this way.
S i m u l a t i o n results show t h a t w i t h our mechanism,
r a t i o n a l agents can reduce their cost by delegati n g / r e c e i v i n g actions. T h e results also show t h a t the
mechanism has the i m p o r t a n t properties of flexibility
and guarantee cost-balancing.
T h e o u t l i n e of the rest of the paper is as follows. In
the next section, some of the assumptions are presented
before discussion. O u r mechanism is proposed in Sections 3 and 4. In Section 3, t w o evaluation functions
are defined using t h e game theory. Also, the social law
and the m o d e l of agents are described. In Section 4, the
other p a r t of our m e c h a n i s m is described. In Section 5,
some examples are presented. E x p e r i m e n t a l results are
shown and discussed in sections 6 and 7, respectively.
2
3
Definitions
3.1
C h a n g e o f cost b y d e l e g a t i o n
T h e change in agent A's subplan, w h i c h has a favor relationship to B's subplan and the associated cost t h r o u g h
d e l e g a t i o n / r e c e p t i o n , is shown in Figure 1 where some
actions are indicated as circles.
For instance, if a subplan of agent A (a) consists of
three actions: "go to store" then "buy f o o d " and finally
"come back," and t h a t of agent B consists of: "go to
store" then " b u y j u i c e " and finally "come back," t h e n
the subplans are in a favor relationship ( p a r t l y overlapp i n g ) . If A receives the execution of B's s u b p l a n , the
replanned subplan (b) t h a t A m u s t execute is: "go to
store" then " b u y f o o d " and " b u y j u i c e " ( B ' s action) and
finally "come back" and " h a n d B j u i c e " (action to adjust
A's initial plan).
Assumptions
A s s u m p t i o n s on the agents' plans and actions are as follows:
1. Each agent has a plan to achieve its o w n goal, and
doesn't know t h e plans of other agents'.
2. Subplans (plans w h i c h achieve sub goals) consist of
sequences of actions.
In order to express costs, we use the f o l l o w i n g n o t a tion:
3. T h e meanings of actions and associated costs are
predefined and c o m m o n to all agents.
•
: A ' s cost for e x e c u t i n g a subplan w h i c h has
a favor relationship to B's s u b p l a n .
In order to concentrate on the a l g o r i t h m of delegat i o n / r e c e p t i o n decision, we assume the f o l l o w i n g :
•
: T h e cost of A ' s actions when executed by
B when delegated (e.g., " b u y f o o d " ) .
4. A g e n t s can find w h i c h parts (subplans) of their plan
can be delegated.
•
: T h e cost of A ' s a d d i t i o n a l actions in the
case of d e l e g a t i o n / r e c e p t i o n (e.g., " h a n d B j u i c e " ) .
Positive relationship t h a t is i m p l i c i t l y contained between plans is d i v i d e d i n t o t w o relationships 1 [ M a r t i a l ,
1992]: a s u b s u m p t i o n relationship and a favor relation
If A receives the execution of B's actions, A ' s overall
cost is the cost of its o w n actions
plus the cost
of the delegated actions f r o m B
and the cost of
a d j u s t i n g a plan to a new p l a n
On the other h a n d , if A delegates the execution of its
subplan, the subplan w i l l become ( c ) . In this case, A ' s
overall cost is only the cost of a d j u s t i n g a p l a n
ship.
3.2
5. Agents can discover other agents w i t h which to neg o t i a t e delegation.
6. N e g o t i a t i o n is carried out between t w o agents only.
If the actions of an agent's plan are the same or subsume another agent's actions, the plans are in a subs u m p t i o n relationship and one agent does not need to
execute its plan if the other agent executes its o w n p l a n .
If an agent's plan p a r t i a l l y overlaps w i t h another agent's
p l a n and is n o t in a s u b s u m p t i o n relationship, the plans
are in a favor r e l a t i o n s h i p . In this case, an agent does
not need to execute its plan if another agent is w i l l i n g
to receive the plan and to execute its own plan together
w i t h t h e n o n - o v e r l a p p i n g p a r t o f the received plan.
N o w , we assume the f o l l o w i n g 2 :
7. A g e n t s collaborate when their plans are in a favor
relationship.
1
Martial divided the relationships in more detail.
I t is not necessary for agents to collaborate in a subsumption relationship because it is enough for an agent to check
whether its actions were already done after execution of the
other agent's plan.
2
Social law for delegation
Social law is formalized in [Shoham and T e n n e n h o l t z ,
1992] and is treated as the pair of actions and c o n d i t i o n s
about states t h a t p r o h i b i t t a k i n g a c t i o n . I n t u i t i v e l y , social law / is the law by which agents must balance the
costs of the individual received actions.
Definition 1 (Social law for delegation)
l is defined to obey the next constraints.
(a)
Social law
If an agent receives more actions than the other
agent, it cannot receive actions at the next negotiation.
(b) If an agent delegates more actions than the other
agent, it cannot delegate actions at the next negotiation.
W h e n agents abide by
gate actions and benefits
even if it receives actions
even a r a t i o n a l agent can
this law, each agent can delein the f u t u r e according to (a),
and loses by t h a t . Therefore,
propose the strategy "receive."
Matsubayashi and Tokoro
347
Though this law may not be the only one that makes
agents collaborate on positive relationships, the search
for such a law is beyond the scope of this paper. However, this law has the following two advantages: (1) as
(a) and (b) are exclusive, agents can avoid the conflicts
of strategies proposed in the negotiation, or collaborate
smoothly, and (2) the microscopic law defined between
two agents can balance the agents' costs of executing
received actions in a whole system.
3.3
Expected payoff
We use payoff in game theory as the increase in cost arising when an agent proposes its strategy at negotiation
and defines the expected payoff. The strategy consists of
two propositions: "delegate" and "receive" 3 Therefore,
agent A's strategy p is a mixed strategy (p 1 , P2) where p1
is A's probability of proposing strategy "delegate" and
is that of the reception. Also, the other agent
B proposes a mixed strategy
expressing the
same idea. Please note that the expected payoff is not
the expected cost but the expected increase in cost.
Because of the existence of social law /, the strategy
that is selected by A not only influences the strategy
of B in the future, but also influences the cost of A in
the future. Hence, the expected payoff is the sum of the
expected payoff of current negotiation, present expected
payoff (Ep(p, /)) and the negotiation in the future, future
expected payoff (EF(P, /)).
However, other agents (say, agent B) may not abide
by the law. W i t h the reliability of B (Pi(B)), i.e., the
probability of B abiding by the law, the expected payoff,
is defined as follows:
D e f i n i t i o n 2 ( E x p e c t e d p a y o f f of agent A)
3.3.1
W h e n t h e o t h e r agent abides b y t h e law
If the other agent, B, abides by the law, A's payoff of
current negotiation is expressed in the next payoff matrix. Entries '—' (undefined) represent cases that cannot
happen because both agents abide by the law.
Table 1: A's payoff (increase in cost)
Table 2: A's payoff in the future
3
There is also one other strategy: "do nothing," in which
agents must execute the subplan for themselves. This strategy is taken when the expected payoff is not less than 0.
348
Distributed Al
Matsubayashi and Tokoro
349
T h e costs for agent B are taken for the same table w i t h
B replacing A. However, t h e cost of the subplan (25-30)
and the cost of the delegated p a r t of the plan (5-12) was
r a n d o m l y selected and was not necessarily common to
b o t h agents. T h e results are t h e average of 5000 s i m u lations.
6.1
negotiation is n o t cost-consuming (Cneg = 2), t h e agent
continues to collaborate w i t h t h e self-centered agent, because it can reduce its cost more t h a n w h e n d o i n g it alone
(i) b u t less t h a n when in s i t u a t i o n ( i i ) .
The cost of B
Table 3: Cost of actions for agent A
R e d u c t i o n o f cost
Figure 3 shows the relation between the cost for executing a subplan and the frequency of negotiation for
delegation.
The cost of A
Figure 4: Sum of the cost of the plan received
Curves in the figures are the result of the next conditions
except for (iii b)).
(i)
B does not exist (agent A has to execute the
actions alone),
(ii)
B uses our mechanism,
(iii a)
B is self-centered (B's strategy is only "dele(iii b)
(iv)
B is self-centered
B is the same as (iii a) till the 30-th negotiation, after then, it is the same as (ii).
Although the cost doesn't decrease when agent A executes the actions alone (i), A can reduce the cost when
the other agent uses our mechanism (ii). When the other
agent is self-centered (iii), the cost is at first a little bit
more than when the other agent uses our mechanism
from the start. This is caused by a decision made by
both flipping a coin when strategies collide and usage of
the "do nothing" strategy when A cannot benefit from
it. However, when the delegation precedes with
(a cost-consuming negotiation), as the agent using our
mechanism does not come to collaborate with the selfcentered agent, the cost approaches the cost of (i). When
350
Distributed Al
In the case where the other agent changes its mind
and decides to use our mechanism rather than being
self-centered (iv), the same result obtained for the selfcentered agent is obtained at first. If the other agent
has decided to use our mechanism, it returns the benefit
obtained up till then in order to balance the individual
received cost. Therefore, the cost is extremely reduced
at that time, and approaches the cost of (ii).
Furthermore, we simulated the case when
and
are different from 15 and 0.07. W i t h a less than 15,
result (iii) presents the largest change among all simulations. The agent decides not to collaborate with the selfcentered agent earlier. If
is greater than 0.07, result
(iv) represents the largest change. The cost when the
other agent has come to use our mechanism is smaller.
The reason for this is that as
larger,
larger so
that the agent comes to collaborate earlier on the social
law. But if
is smaller, the agent will propose the
"do nothing" strategy so that the "delegate" strategy
would be less used and the cost is not reduced.
6.2
Cost b a l a n c i n g of r e c e i v e d a c t i o n s
Figure 4 shows the received cost of both agents. Curves
(ii)-(iv) in the figure are the same conditions as before.
When both agents use our mechanism (ii), the costs
received are balanced. When the other agent is selfcentered (iii), the agent receives too many actions because of the flexibility of the mechanism. However, this
loss cannot continue to increase because A does not get
to participate in the negotiation on delegation if A thinks
that it cannot benefit from it. Also, in the case of the
self-centered agent which changes its mind and decides
to use our mechanism in the course of negotiation, the
received costs are finally balanced (iv).
7
Discussion
We discuss some issues related to our mechanism.
T h e social law w h i c h guarantees t h a t an agent can
delegate its actions in the f u t u r e even if it receives the
actions of another agent, n o t o n l y enables r a t i o n a l agents
to reduce t h e i r cost by c o l l a b o r a t i n g , b u t also enables our
m e c h a n i s m to have t w o i m p o r t a n t properties: a g u a r a n tee of balanced costs for i n d i v i d u a l l y received actions
a n d an avoidance of conflicts in strategies.
As o u r microscopic law, defined between t w o agents,
enables t h e agents to balance t h e received costs between
t h e m so t h a t t h e costs are macroscopically balanced in a
w h o l e m u l t i - a g e n t system. W i t h t h i s law, a n i r r a t i o n a l
agent t h a t m a y receive t o o m a n y actions w o u l d not be
a b o t t l e n e c k of the whole system if other agents thatdelegated tasks to it w a i t e d u n t i l they finished.
W i t h agents A and B a b i d i n g to this law, if A receives
more than
A proposes "deleg a t e " and B proposes "receive." Therefore, the strategies do not conflict, so t h a t agents can easily reach an
agreement. In [Shoham and T e n n e n h o l t z , 1992], social
law is used to constrain actions to avoid conflicts. It
became unnecessary for agents to negotiate for conflict
r e s o l u t i o n . However, agents m u s t negotiate to exchange
i n f o r m a t i o n a b o u t plans when they collaborate in a posi t i v e r e l a t i o n s h i p . O u r mechanism used the law to con
s t r a i n t h e strategy of n e g o t i a t i o n so as to avoid conflict.
Moreover, we considered self-centered agents, or
agents t h a t do not abide by the law. T h i s is a case
w h i c h is not considered in [Shoharn and T e n n e n h o l t z ,
1992]. It is very simple for agents not to collaborate
w i t h self centered agents after they realize w h a t k i n d of
agents they are dealing w i t h . However, it is p r o b l e m a t i cal t h a t an agent w h i c h exceptionally does not abide by
the law for some reason, cannot collaborate any more.
A l s o , p r o b l e m a t i c a l is the fact t h a t agents do not col
l a b o r a t e w i t h self-centered agents even when benefit can
be o b t a i n e d . O u r mechanism is flexible: r e l i a b i l i t y P 1 is
g r a d u a l l y decreased every t i m e the strategies conflict, so
t h a t t h e chance to collaborate w i t h a real self-centered
agent is decreased and agents collaborate only if they
can benefit f r o m i t . However, agents w i t h our mechan i s m m a y receive a l i t t l e more action t h a n a self-centered
agent because of this flexibility.
Here, we consider the s i t u a t i o n where the bank and
post office are in the same direction and b o t h agents
p l a n to go to b o t h places. Because their plans are "go
to b a n k " and "go to post office" and " r e t u r n , " the res u l t i n g plan of our mechanism is t h a t one agent goes to
b o t h places. Nevertheless, if the places are in different
directions, the r e s u l t i n g plans are that each agent goes to
a different place. T h e l a t t e r result is obtained t h r o u g h
t w o n e g o t i a t i o n s because their plans are "go to b a n k "
and " r e t u r n " and "go t o post office" and " r e t u r n . " O u r
m e c h a n i s m also provides such an o p t i m a l s o l u t i o n .
In [ Z l o t k i n and Rosenschein, 1990], flipping a coin is
used to resolve conflicts, b u t the purpose for our work is
to avoid conflict. Since t h e m e n t i o n e d work is designed
for c o l l a b o r a t i o n in negative relationships, the conflict in
goals m u s t be resolved. On the other h a n d , as our work
is directed to p o s i t i v e relationships, the conflicts are not
necessarily resolved (in this case, agents cannot reduce
costs).
In a related piece of research [Martial, 1992], the proposed algorithm focuses on the optimization of plans of
agents. Therefore, agents are willing to receive the actions even if they cannot benefit from them, and the
received costs would not be balanced. Our mechanism
does not require such irrational agents; each agent can
rationally and flexibly select its own strategies.
8
Conclusion
The utilization of positive relationships has not received
much attention in D A I . We proposed a mechanism using
a social law by which rational agents can reduce the cost
of their execution by delegating and receiving actions.
W i t h this law, agents can avoid conflicts of strategies
in a negotiation and reach an agreement easily. Furthermore, we considered the case in which agents do not
abide by the law (self-centered agents) and the case in
which agents that are self-centered at first come to abide
by the law eventually. Simulation results show that by
using our mechanism, rational and flexible collaboration
is possible with these kinds of agents.
We intend to use a more complex model of agents,
and consider communication costs and collaboration
with multiple agents by introducing market mechanisms.
Also, we will introduce a mechanism based on incomplete information for more detail, as stated in [Kakehi
and Tokoro, 1993].
Acknowledgments
We would like to thank T. Hirotsu, R. Kakehi, V. Vasconcelos and I. Satoh for their helpful comments.
References
[Foulser et al., 1991] D. E. Foulser, M. L i , and Q. Yang.
A Quantitative Theory for Plan Merging. In AAAI91, 1991.
[Hayes, 1989] C. C. Hayes. A Model of Planning for Plan
Efficiency: Taking Advantage of Operator Overlap. In
IJCAI-89, 1989.
[Kakehi and Tokoro, 1993] R. Kakehi and M. Tokoro. A
Negotiation Protocol for Conflict Resolution in Multiagent Environments. In fCICIS '93, 1993.
[Martial, 1992] F . V . M a r t i a l . Coordinating Plans of Autonomous Agents. Lecture Notes in Artificial Intelligence, Subseries of LNCS. Springer-Verlag, 1992.
[Osawa and Tokoro, 1991] E. Osawa and M. Tokoro.
Collaborative Plan Construction for Multiagent Mutual Planning. In MAAMAW '91, 1991.
[Shoham an d Tennenholtz, 1992] Y.Shoham
and M. Tennenholtz. On the synthesis of useful social
laws for artificial agent societies (preliminary report).
In AAAI-92, 1992.
[Zlotkin and Rosenschein, 1990] G. Zlotkin and J. S.
Rosenschein. Negotiation and Conflict Resolution in
Non-Cooperative Domains. In AAAI-90, 1990.
Matsubayashi and Tokoro
351