International Journal of Innovations & Advancement in Computer Science
IJIACS
ISSN 2347 – 8616
Volume 3, Issue 10
December 2014
Designing metrics to evaluate the performance of Agent
Oriented Software
1,2
Sapna Mahar1 , Pradeep Kumar Bhatia2
Computer science and Engineering, Guru Jambheshwer University of Science & Technology,
Hisar, Haryana, INDIA
Abstract
The characteristics of a software agent, such as social ability, proactivity, reactivity, adaptability, mobility and
goal orientation, depend mainly on their execution performance; and it is therefore not easy to measure them
by using just static product metrics. The static metric as well as dynamic metrics are to be used to evaluate the
global performance of agent in combination with working level of product performance metric. The present
paper proposes four new agent oriented metrics, which could be use to enhance the quality of software agent.
These metrics are Operational Time of an Agent, Response Time of an Agent, Invariant Count and Event
Occurrence Factor. These metrics are then implemented on some programs based on JADE platform.
Keywords: - Software Agent, Agent Metrics, Agent Software Quality.
1. Introduction
Many metrics have been researched, developed and put to use to determine the quality of software. The
development of software metrics has grown in tandem with that of software engineering. For different
software development models, different types of metrics have been constantly researched and developed to
provide useful measurements for programming paradigms. Metrics are designed to evaluate the quality of
software, which allows the easier detection of possible error or bad design. Metrics, on the basis of program
design, can be grouped as Static Metrics and Dynamic Metrics. The static metrics are collected during the
analysis of software code whereas dynamic metrics are collected during the execution of software. Software
Project Performance Metrics can also group into three main categories i.e. product, process and resources.
Each of the metrics categories emphasizes on different aspects of software engineering. Product metrics
measure the properties of software product, such as its documentation, design and performance, regardless of
its development stage.
Agent Oriented Software is mainly based on object oriented. In contrast to object oriented software,
agents are self-contained computational systems. Each agent runs independently and interacts with other
agents or external systems to achieve its design objectives. The similarity between object and agent suggests
that the object oriented metrics can be applied on the agent. Object oriented metrics focus on the object, class,
information hiding, inheritance, polymorphism and coupling. Agent metrics emphasize in measuring the
characteristics of agent such as social ability, autonomy, reactivity, adaptability, intelligence, learning,
proactivity, goal-orientation and mobility. Therefore, to measure the overall quality of an agent a complete
metrics suite should include a subset of OO metrics and a set of pure Agent metrics. The complexity of agent
can be determined by measuring the characteristic of agent. The special characteristics of Agent oriented
software engineering can also affect the agent quality and more emphasis should be placed in measuring them.
The paper is proposing four new metrics one of them is static metric which is applicable on product
design level and the other three are dynamic which are categorized under product working level.
The paper is organized in 7 sections, starting with introduction. Section 2 includes some related work
based on agent oriented metrics. Section 3 discusses three previous metrics already been defined. Section 4
defines a set of proposed metrics. Section 5 analyzes them and 6 evaluates these metrics with the help of few
programs. Section 7 concludes the paper followed by references.
2. Related work
86
Sapna Mahar, Pradeep Kumar Bhatia
International Journal of Innovations & Advancement in Computer Science
IJIACS
ISSN 2347 – 8616
Volume 3, Issue 10
December 2014
A whole set of performance metrics related to the product, process and resources technologies and
components of the software agent and multi agent system was defined[1][2].The product performance metrics
measure the agents and the system in terms of their design, description and working level. These metrics then
examines and finds out as to how these metrics could be applied into the new agent- oriented software [3].
Some of these metrics were categorized according to agent’s characteristics and applied to evaluate them.
Researches for determining and evaluating software agent’s quality considering an agent’s characteristics like
social ability[4], autonomy[5] and pro-activity[6] has been done. They established the attributes associated with
these characteristics and set out the measure for agent’s global evaluation. A set of measures was proposed for
evaluating the reactivity property considering its associated attributes at various levels [8]. A set of measures of
agent oriented software considering mobility characteristic which has been decomposed into three attributes
[9]
. Some measures was analyzed and categorized for attributes for intelligence characteristics of agent [10]. All
of these researches has been done on product performance metrics and the measures used were based on
design and working level of agent system. After going through many literatures we propose some metrics
which can enhance the performance of agent and hence improve its quality.
3. Existing Metrics
Some of the existing metrics, which motivates us for designing new measures, are Agent Life span,
Response for message and Variable Density.
a. Agent Lifespan
It measures the time duration that agent has spent in the system. There is no standard time for which
the agent remains alive. Sometimes, the agent, waiting for resources and services might cause it to stay in
the system for a longer time before it accomplishes its goals and terminate [3][9]. So, we have proposed
Operational Time of an Agent.
b. Response for Message
It measures the amount of messages that are invoked in response to a message received by the agent.
To process the incoming message, new messages might be sent to another agent. This metrics is dynamic
and product working metrics. It promotes the social ability and reactivity of the agent. In this, metric
measures the number of messages considered, but not the time consumed by them [4]. So, our next
proposed metric is Response Time of an Agent.
c. Variable Density
It uses the number and data type of variables to determine the agent’s internal states. Large number of
internal states requires more computation to maintain the values. It is a static and product design metrics
and promotes the autonomy and intelligence of agent [5]. In contrast to this metric, we considered those
variables that cannot be changed i.e. Invariant Count.
4. Proposed Measures
a. Operational Time of the Agent/Operating Time of Agent(OTA)
Operational Time of the Agent can be defined as the actual time; an agent spends, in performing
operations. This metric is an enhancement of the Agent Life Span, which includes the total time an agent
remains alive in the system. Sometimes, an agent has to wait for some resources or response from another
agent, which is likely to affect the performance of the agent.
OTA = ALS - WT
(1)
Where, OTA is the Operational time of the Agent, ALS is Agent Life Span & WT is waiting time of the
Agent.
87
Sapna Mahar, Pradeep Kumar Bhatia
International Journal of Innovations & Advancement in Computer Science
IJIACS
ISSN 2347 – 8616
Volume 3, Issue 10
December 2014
This metric is dynamic one and measures the working level of an agent i.e. lesser the operational time
greater will be the efficiency of the agent. Hence, it improves the performance of the agent. It also affects
the reactivity of the agent.
b. Response Time of the Agent(RTA)
The inter-agent communication also assesses the quality of an agent. So, it is evitable that agent
metric should measure the Communication performance. Communication Setup Time metrics was
proposed to measure the time required to set up the communication.
Response Time of the Agent can be defined by the time taken by an agent while responding to some
another agent or system. The time taken is generally very low i.e. in micro or milliseconds.
RTA = Time required in giving response(2)
This metric can be assumed to be an extension of Response for a Message, which defines the number of
messages invoked in response of a message.
This metric is also dynamic one and measures the working level of an agent i.e. lesser the response time
greater will be the efficiency of the agent. It affects reactivity and social ability of the agent.
c. Invariant Count
It is equally important to measure the quality of attributes, which includes the Knowledge Usage,
Knowledge Update and data types of the variables. These reflect the overhead and efficiency of the agent
and usability of internal information.
The proposed metrics can be defined as the total number of final variables in an agent.
IC=
(3)
th
Where, IC is Invariant Count and FVi is the i Variable declared as Final.
This metric enhances the autonomy; and rest of the system cannot access its variables. The autonomy of
agent can be roughly estimated by static product metric. This metric belongs to design level and enhances
the concurrency control mechanism and maintain data consistency.
Note: This metrics can be applicable on those programming languages, which uses Final keyword to use
constant variable; and for rest of programming languages programming the constant variables can be
considered.
d. Event Occurrence Factor(EOF)
An event-driven application is a program that is written in response to the actions generated by user or
system. This approach is typically used for most modern software application. In agent oriented software,
during the lifetime of an agent, it has to respond to different situations and has to handle different events.
This ability of the agent to generate events should be measured to improve the responsiveness, throughput
and flexibility of the agent oriented software.
The proposed metric can be defined as the total number of events occurred in a software agent.
EOF =
(4)
Where, EOF is Event Occurrence Factor, n is the number of classes, m is the number of event in a
class and
is the event the jth event in ith class.
This metric is dynamic one and measures the working level of an agent and therefore affects the
interactivity of the agent. The behaviour of an agent varies from one situation to another, so it is very
difficult to determine the exact performance of an agent.
5. Analysis of proposed measures
88
Sapna Mahar, Pradeep Kumar Bhatia
International Journal of Innovations & Advancement in Computer Science
IJIACS
ISSN 2347 – 8616
Volume 3, Issue 10
December 2014
From the above definition and general information about the proposed metrics, we can categorize them on
two bases i.e. on the basis of execution and on the basis of product performance, as shown in Table 1.
OTA
RTA
EOF
IC
Execution based
Static
Dynamic
Metric
Metric




Product Performance Metrics
Design
Working Level
Level




Table1. Metrics classified according to execution of agent and product performance
Further, the metrics also affect some of the characteristics of agent as shown in Table 2. IC affects the
Autonomy and to certain extent the Intelligence as well, while RTA affects the Social Ability, Reactivity and
Mobility of the agent.
Social
Ability
OTA
RTA
EOF
IC
Autonomy
Intelligence Proactivity Reactivity Mobility









Table2. Metrics affecting various characteristics of software agent
6. Experimental Evaluation
We have applied the above mentioned metrics on some java programs developed on JADE platform. These
metrics are calculated manually on a set of programs implementing various agents, which includes
BookSellerAgent (P1), BookBuyerAgent (P2), TopicMessageSender (P3), TopicMessageReceiver (P4) and
ThanksAgent (P5). P1 and P2 communicate with each other regarding book shopping. P3 communicates with
P4 regarding Topic Messages. P5 creates two another agent and send greetings to them.
1. P1 agent registers itself and adds a book with its price to the catalogue. After that, it serves queries and
purchase orders from the buyer agents.
2. P2 agent searches the known seller agents with the required book. If it finds more than one sellers, it
chooses the best one on the basis of price. After that, it sends the purchase order to the seller and buys the
book.
3. P3 agent periodically sends the messages about a given topic.
4. P4 agent registers itself to receive messages about that topic sent by P3.
5. P5 agent first registers with the DF and creates a list of agents, and each of them registers with the DF,
and then P5 agent sends a message of greetings to them and awaits for an answer to the greetings and at
last thanks the agents those have answered.
The results for Operational Time of Agent are depicted in Figure 1, where from it is clear that P1 consumes
highest time whereas P4 takes the least. Higher be the OTA, lesser an agent will be efficient. The decreasing
order of efficiency as well as reactivity and mobility is P1, P5, P2, P3 and P4.
89
Sapna Mahar, Pradeep Kumar Bhatia
International Journal of Innovations & Advancement in Computer Science
IJIACS
ISSN 2347 – 8616
Volume 3, Issue 10
December 2014
Operational Time of Agent
6
5
4
OTA
3
2
1
0
P1
P2
P3
P4
P5
Fig1. Measures for Operational Time of Agent
The results for the Response Time of Agent are depicted in Figure 2, wherefrom it is clear that P2 has highest
response time whereas P3 takes the least. The efficiency of an agent is inversely proportional to the response
time. P3 and P4 are more social, reactive and mobile as compared to P1 and P2.
Response Time of Agent
80
60
RTA
40
20
0
P1
P2
P3
P4
Fig2. Measures for Response Time of Agent
The results for Invariant Count are depicted in Figure 3, from where it is clear that P5 has highest ID and P2
has zero IC. The decreasing order of autonomy is as P5, P1, P3 &P4 and P2.
Invariant Count
3
2.5
2
IC
1.5
1
0.5
0
P1
P2
P3
P4
P5
Fig3. Measures for Invariant Count
The results for Event Occurrence Factor are depicted in Figure 4, which shows that P2 and P3 are event driven
programs whereas P1, P4 and P5 are not. So, we can say that P2 and P3 are more reactive and proactive as
compared to rest of three.
90
Sapna Mahar, Pradeep Kumar Bhatia
International Journal of Innovations & Advancement in Computer Science
IJIACS
ISSN 2347 – 8616
Volume 3, Issue 10
December 2014
Event Occurance Factor
1
0.8
0.6
EOF
0.4
0.2
0
P1
P2
P3
P4
P5
Fig4. Measures for Event Occurrence Factor
7. Conclusion and Future Work
From the review of various literatures, it could be concluded that some of the important parameters from
which the performance of an agent can be assessed are missing. These parameters can be operational
performance of agent, time required to response back to agent or environment, concurrency control and event
driven approach. All these play a very important role in analyzing the behavioral performance of an agent,
which is the key point of the research. Keeping all these things in view, four metrics could be proposed. These
metrics have been applied on some programs based on JADE platform to evaluate the performances thereof.
The future work will be to implement these metrics on some real projects of agent software.
References
[1] C. Wille, R. Dumke, S. Stojanov, “Quality Assurance in Agent-Based Systems Current State and
Open Problems”, Preprint No. 4, Fakultät für Informatik, Universität, Magdeburg.
[2] R. Dumke, R. Koeppe, and C. Wille, “Software Agent Measurement and Self-Measuring AgentBased Systems”, Preprint No. 11, Fakultat für Informatik, Otto-von-Guericke - Universitat
Magdeburg, 2000.
[3] K. Shin, “Software Agents Metrics”, a Preliminary Study & Development of a Metric Analyzer,
Project Report No. H98010, Dept. Computer Science, School of Computing, National University of
Singapore, 2004.
[4] F. Alonso, J. L. Fuertes, L. Martínez and H. Soza, “Measuring the Social Ability of Software Agents,”
Proc. of the Sixth International Conference on Software Engineering Research, Management and
Applications, Prague, Czech Republic, pp. 3–10, 2008.
[5] F. Alonso, J. L. Fuertes, L. Martínez and H. Soza, “Towards a Set of Measures for Evaluating
Software Agent Autonomy”, Proc. of the 7th Joint Meeting of the European Software Engineering
Conference and ACM SIGSOFT Symposium on the Foundations of Software, 2009.
[6] F. Alonso, J. L. Fuertes, L. Martínez and H. Soza, “Measuring the Proactivity of Software Agents”,
Proc. of the 5th International Conference on Software Engineering Advances, IEEE, 2010.
[7] F. Alonso, J. L. Fuertes, L. Martínez and H. Soza, “Evaluating Software Agent Quality: Measuring
Social Ability and Autonomy”, Innovations in Computing Sciences and Software Engineering, pp.
301-306, 2010.
[8] N. Sivakumar, K.Vivekanandan, “Measures for Testing the Reactivity property of a Software Agent”,
International Journal of Advanced Research in artificial Intelligence, Vol 1, 9, pp.26-33, 2012.
[9] S. Mahar, Pradeep Kumar Bhatia, “Measuring Mobility of Software Agent”, International Journal of
Computer Information & Computation Technology, Vol 3, 3, 221-224, 2013.
[10] S. Mahar, Pradeep Kumar Bhatia, “Measuring the Intelligence of Software Agent”, International
Journal of Innovative Science, Engineering & Technology, Vol. 1 Issue 6, 2014.
[11] http://jade.tilab.com/
91
Sapna Mahar, Pradeep Kumar Bhatia