Test in Multi-Agent Systems
© LES/PUC-Rio
Background
• The WWW and
applications:
the
mobile
devices
– collection of distributed components
– input from a variety of sources
– adapt according to users preferences.
• Agent technology:
– prominent way to implement such systems
– uses abstractions of a higher level.
© LES/PUC-Rio

new
Agents – A new Level of Abstraction
Problem Domain
R1
R2
R3
R4
R5
R7
R6
R8
How?
What?
files
functions
Which agents?
modules
Procedural
Object
Attribute
Method
Class
Object Oriented
Agent
Organization
Roles
Messages
Protocols
Agent Oriented
© LES/PUC-Rio
Agent Definition
• Agent:
System
– is an autonomous entity
– that executes a plan
– to achieve a goal
– based on its beliefs (attributes)
– communicates asynchronously (message based)

Garcia, A., Lucena, C., Cowan D. Agents in Object-Oriented Software Engineering. Software Practice &
Experience, Elsevier, 34 (5), pp. 489-521, 2004.
• Autonomy
– Different set of actions may achieve the same goal
© LES/PUC-Rio
Agent Technology: Benefits and Challenges
• On the one hand:
– Facilitates
to
requirements.
address
current
application
• On the other hand:
– new challenges to software
consequently, to maintainability.
testability
• Tests:
– aims at uncovering regression bugs.
– serve as live documentation.
© LES/PUC-Rio
and
Obstacles to Software Testability
• Testability:
– Controlling the test input (controllability)
– Observing the
(observability)

output
of
the
unit
under
test
Voas, J., and Miller, K.. Software Testability: The New Verication. IEEE Software, 12
3:1728, 1995.
• Obstacles:
– More than one “expected” behavior for an agent
– Test input: data
conversations.
+
messages
© LES/PUC-Rio
from
concurrent
MAS Development Approaches
• Traditionally adopted:
– prototyping process
– ad-hoc approaches to assure system quality.
• Few works define
supporting tool:
a
testing
– Agile
– Agile Passi
– SUnit
– JAT
© LES/PUC-Rio
process
with
SUnit: Test Driven Development of MultiAgent Systems
Ali Murat Tiryaki, Sibel Oztuna, Oguz Dikenelli,
R. Cenk Erdur
© LES/PUC-Rio
Approach
• Test driven based development approach that
supports iterative and incremental MAS
construction
• It implements a JUnit-based framework for
writing automated tests for agent behaviors and
interactions between agents.
– Mock agents to model the organizational aspects of
the MAS
– It was implemented on to of Seagent (Hierarchical
Task Network)
© LES/PUC-Rio
Test Driven Development
• Test driven development (TDD)
– Style of software development in which
development is driven by automated tests.
– Cycle: developer firstly writes functional tests for a
unit/task, writes the code that passed the written
tests, and then, the code is refactored to improve
the design
• Supports the iterative and incremental
construction of task
© LES/PUC-Rio
Agent Oriented Test Driven Development
• Scenario: It defines an elementary process within MASs
which produce a net value for the initiator.
• Unit: Agent. Agents cooperatively execute the
functionalities of the scenario at hand. Each agent has its
own tasks based on the responsibilities within the scenario.
Agents interact using an agent communication language
which has a higher level semantics.
A Sample HTN Structure
© LES/PUC-Rio
Iterative cycle of AOTDD
© LES/PUC-Rio
SUnit Testing Framework
• It provides a framework for test driven plan
development
• It supports the developer to create and execute
tests in a uniform and an automatic way during
the plan implementation
• It consists of three sub modules
– Structural Test Module
– Action Test Module
– Flow Test Module
© LES/PUC-Rio
Structural Test Module
• This module assists the developer by defining
subtasks and by determining the interrelationship of the subtasks which compose the
HTN plan.
• For every single complex task in a plan, a
structural test case should be implemented
Assert methods
© LES/PUC-Rio
Action Test Module
• Action is a primitive tasks that generates an
outcome or a message to another agent in the
platform.
• This module is designed to examine outcomes or messages
in respect to the user defined provision sets for the primitive
tasks.
Assert methods
© LES/PUC-Rio
Flow Test Module
• This module is designed to evaluate outcomes or
outgoing messages of the tasks in respect to the
incoming messages or predefined message sets
The Class Diagram
of the Flow Test
Module
Assert methods
© LES/PUC-Rio
Creating a Unit Test
• SeagentStructuralTestCase should be extended to
validate the plan structure.
• SeagentActionTestCase have be extended to be able to
write assert methods for primitive task(s).
– check the results of the primitive task(s) such as outcomes
and/or generated messages.
• SeagentFlowTestCase check the correctness of OWL
concept’s type and value with the expected ones.
© LES/PUC-Rio
Creating a Unit Test - Example
• Traveler tries to organize a holiday plan which includes hotel
booking and transportation details. The aim of the scenario
is to arrange the cheapest holiday plan by selecting the
proper accommodation & transportation options based on
the traveler preferences.
© LES/PUC-Rio
Creating a Unit Test - Structural Test
public void testLevelOne(){
assertDisinheritanceSender(disinheritOK,arrangeArrival);
assertProvisionSender(arrangeAccomodation,
provisionResInfoForAcc,getReservationInfo;
assertProvisionSender(arrangeArrival,
provisionResInfoForArr,arrangeAccomodation);
}
© LES/PUC-Rio
Creating a Unit Test – Flow Test
public class FullIsolatedTest extends SeagentFlowTestCase {
…
public MessageSet assignMessageSet() {
MessageSet mySet= new MessageSet();
FIPAMsg aMessage = new FIPAMsg();
…
mySet.addMessage(aMessage);
…
return mySet;
}
public void testForFirstLevel(){
assertActionOutcome(arrangeAccomodation, reservationInfo);
assertActionExecuted(arrangeArrival);
assertMessageArrived(aMessage);
}
© LES/PUC-Rio
JAT: A Test Automation Framework for
Multi-Agent Systems
Roberta Coelho, Elder Cirilo, Uirá Kulesza
Arndt von Staa, Awais Rashid, Carlos Lucena
Approach
• It defines an approach for building and running
MAS tests.
• It relies on Aspect Oriented Programming to:
– Control the test input
– Observe the output units under test
• It was implemented on top of JADE:
– Middleware for agent-based applications developed
in Java.
© LES/PUC-Rio
JADE Agent Testing (JAT) Framework
• JAT enables:
– the creation of JUnit-style tests for JADE systems.
• JAT is used:
– as a generic testing framework for JADE developers
– same as JUnit is for Java developers
© LES/PUC-Rio
Fault Model
Agent-related new abstractions  new sources of bugs
• Message ordering
• Message content
• Increases on message delay
• Faults in agents beliefs
• Faults in internal procedures
© LES/PUC-Rio
Supporting Ideas
1. Using Agents to test Agents:
• Mock Agent:
• adaptation of the mock object concept
• a fake implementation of a real agent
• sends messages to an agent under test (AUT) and
• checks its response:
• content, delay, type, ordering.
(A)
method call
(B)
Class
send message
method output
message response
Tester
Agent A
© LES/PUC-Rio
Supporting Ideas
2. Monitoring Agents:
– Agents are autonomous
– The developer need to know agents state
• ex: to answer: output == expected output?
– Monitor Aspect observes and stores information
about agents state transitions (ex: running, dead).
© LES/PUC-Rio
Supporting Ideas
2. Monitoring agents: (cont.)
• Agent state machine is a crosscutting abstraction;
• spread over the application and platform modules;
• can be represented as an Aspect (XPI);
public aspect AgentStateMachine {
pointcut created():call(public
ContainerController.createNewAgent(..));
pointcut from_created_to_running(Agent o):
execution(protected void Agent.setup())
&& target(o);
pointcut from_running_to_dead (Agent o):
execution(public void Agent.doDelete())
&& target(o);
...
}
© LES/PUC-Rio
running
created
waiting
dead
Supporting Ideas
2. Monitoring agents: (cont.)
privileged public aspect Monitor {
...
before(Agent o):
AgentStateMachine.from_created_to_running(o){
includeInRunningAgents(o);
}
public void waitUntilAgentIsDead (String mockAgentID){
while (!isDead(mockAgentID) && (MAXTIMEOUT > timeSpent)){
wait(20000);
...
}
}
...
}
© LES/PUC-Rio
Supporting Ideas
3. Synchronizing Mock Agents
– Mock agents partially implement a test scenario
– We need to define the order Mocks Agents should
interact with the AUT.
– Synchronization  crosscutting concern
– Synchronizer Aspect: orchestrates the test scenario.
Mock Agents
Synchronizer
© LES/PUC-Rio
JAT Framework: The Architecture
JADE Middleware
JUnit Framework
<<crosscuts>>
<<crosscuts>>
A
Aspects
<<uses>>
<<crosscuts>>
<<crosscuts>>
M
S
JAT Framework
Legend:
M
Monitoring Concern
A
Agent State Machine Concern
© LES/PUC-Rio
S
Synchronization Concern
JAT Framework: Dynamics
AgentStateMachineXPI
© LES/PUC-Rio
JAT Framework: Dynamics
AgentStateMachineXPI
1
© LES/PUC-Rio
JADETestCase: Code Example
public class BookTradingTesteCase extends JADETestCase {
public void testBookTradingScena1(){
//Load Data Repositories
…
startAUT("seller","BookSeller");
startMockAgent("buyer1","BookBuyerMockAgent",FIRST);
startMockAgent("buyer2","BookBuyerMockAgent",SECOND);
//Blocks until interaction finishes
Monitor.aspectOf(). waitUntilTestHasFinished("buyer1");
Monitor.aspectOf(). waitUntilTestHasFinished("buyer2");
//Check Expected Result:
assertInteractionOK("buyer1");
assertInteractionOK("buyer2");
//Check Expected Result:
Object belief= getBelief(“seller”,“catalogue”);
assertEquals(expectedBelief,belief);
}
...
}
© LES/PUC-Rio
JAT Framework: Dynamics
JADETestCase
:
JADEMockAgent
:
AUT: JADEAtent
:Monitor
Monitoring
DataRepository
1. testMethod
()
:Synchronizer
2. loadInteraction
Order( )
3. start(params)
4. start(params )
5. state = crosscuts( )
6. add(state)
7. waitUntilTestFinishes ( )
8. send(msg)
9. send(msg)
10.check(msg)
11. setResult
()
12. state =crosscuts( )
13.add(state)
14. notifyTestFinished ()
16. assertions( )
Legend:
join point intercepted
by Monitor aspect
join point intercepted
by Synchronizer aspect
© LES/PUC-Rio
object
aspect
Case Studies Results: Detected faults
Detected Faults
Fault Type
BT
EC
AC
All
-
2
-
2
-
2
-
2
On Interaction Protocol
3
6
4
13
Mismatching Protocols
1
-
-
1
Incomplete Main Flow
-
2
1
3
Missing Alternative Flows
2
4
3
9
1
1
1
3
Boundary values
-
-
1
1
Logic Mistakes
1
1
-
2
4
9
5
19
On MAS Constraints
Unimplemented MAS constraint
On Internal Procedures
TOTAL
© LES/PUC-Rio
Case Studies Results: Detected faults
Detected Faults
Fault Type
BT
EC
AC
All
-
2
-
2
-
2
-
2
On Interaction Protocol
3
6
4
13
Mismatching Protocols
1
-
-
1
Incomplete Main Flow
-
2
1
3
Missing Alternative Flows
2
4
3
9
1
1
1
3
1
1
-
2
5
19
On MAS Constraints
Unimplemented MAS constraint
On Internal Procedures
Interaction
protocols
Boundary
valuesonly define the main execution
flow. Thus,
the developer needs to decide: what 1
Logic Mistakes
1
should TOTAL
an agent do when it receives an unexpected
4
9
message format?
© LES/PUC-Rio
Case Studies Results: Detected faults
Detected Faults
Fault Type
BT
EC
AC
All
-
2
-
2
-
2
-
2
On Interaction Protocol
3
6
4
13
Mismatching Protocols
1
-
-
1
Incomplete Main Flow
-
2
1
3
Missing Alternative Flows
2
4
3
9
1
1
1
3
1
1
-
2
5
19
On MAS Constraints
Unimplemented MAS constraint
On Internal Procedures
Interaction
protocols
Boundary
valuesonly define the main execution
flow. Thus,
the developer needs to decide: what 1
Logic Mistakes
1
should TOTAL
an agent do when it receives an unexpected
4
9
message format?
•
Mocks are useful to examine how agents behave under
exceptional scenarios.
© LES/PUC-Rio
Case Studies Results: Effectiveness of test cases
Fault Type
BT
EC
AC
All
On Interaction Protocol
16/16
23/23
21/21
60/60
Message Ordering
6/6
10/10
8/8
24/24
Message Timing
6/6
10/10
7/7
23/23
Message Content
4/4
3/3
6/6
13/13
4/4
1/1
2/2
7/7
4/4
1/1
2/2
7/7
5/5
7/2
4/4
15/10
Corrupt Return and
Parameter values
2/2
6/2
2/2
8/4
Logic Mistakes
3/3
1/0
3/3
7/6
24/25
31/26
27/27
83/78
On Beliefs
Corrupt Belief
On Internal Procedures
TOTAL
•
Faults Injected/Detected
Justifies the need to complement JAT Tests with conventional
OO unit tests (agent internal procedures).
© LES/PUC-Rio