A Formal Approach to Verify Software Scalability
Requirements using Set Theory and Hoare Triple
Alim Ul Gias∗ , Mirza Rehenuma Tabassum∗ , Amit Seal Ami∗ , Asif Imran∗ , Mohammad Ibrahim∗ ,
Rayhanur Rahman∗ and Kazi Sakib†
Institute of Information Technology, University of Dhaka
Ramna, Dhaka–1000
∗ {bit0103,
bit0129, bit0122, bit0119, bit0117, bit0101}@iit.du.ac.bd
† [email protected]
Abstract—Scalability is the ability of a system to handle
variation in execution environment and continuing to function
in order to meet user needs. For ensuring scalability, it is
important to verify that programmers are writing code that
can scale. However, verifying scalability from code level has
its own limitations as it did not receive adequate attention
from researchers. This paper proposes a formal approach to
verify scalability from the code level using set theory and Hoare
triple. The method denotes variables and functions involving
scalability through set notations. Hoare triple is used to measure
the performance fulfillment with varying workload by a code
segment given that certain code quality measure like caching
or data compression is applied. The methodology is presented
by means of an algorithm which strictly inhibits to passover
a specific scalability requirement and requires to re-apply a
quality measure until a specific requirement is being satisfied.
The approach is applied for developing a real life online ticketing
system and results show that it provides stable response time over
a wide range of user requests. This indicates that the proposed
approach is capable of ensuring scalability by verifying it from
system’s code.
Index Terms—Software Scalability, Verification, Formal
Method, Set Theory, Hoare Triple
I. I NTRODUCTION
Scalability is an expected requirement of a software system
characterizing it’s ability to fulfill quality goals like performance, throughput, etc. when characteristics of execution
environment vary over a expected range [1]. It can be crucial
for systems like e-commerce and social networking sites, web
services and online ticketing systems which deal with growing
number of users. These systems tend to experience unpredictable and extensively varying degrees of requests, especially
during events such as breaking news, national occasions and
Denial of Service (DoS) attacks [2].
Developing a scalable web application involves careful
planning about coding so that it can make the application
work properly under demanding workloads [3]. Therefore,
it is essential to verify scalability from the code level for
ensuring that the system can scale elastically when it is needed.
Moreover, in case of some organizations inconsistent system
performances due to lack of scalability can be fatal from the
business perspective. Thus, during the development of those
systems it is important to apply strictness to ensure none of
the scalability requirements can remain unimplemented.
Although several works have been done on scalability
framework and evaluation method [4], [5], [6], [7], [8], it
is hard to find a significant work that specifically focuses
on verifying scalability from the code level. State of the art
works focus on architectural and design aspect [4], prediction
of resources to make the system scalable [5] and scalability
elicitation and analysis in requirement engineering phase [7],
[6], [8]. However, it is arguable that existing works can be
used to verify that whether the code can scale and impose the
strictness regarding the implication of scalability.
This paper proposes a formal approach to verify the software scalability from the code level. The approach uses
set theory [9] to define multiple sets of variables and their
mapping functions involving system scalability. These sets
include system’s load extensive states, code quality measures
like caching or data compression, performance expectation
over a specific range of requests, state-quality compatibility
pairs and code segments. Mapping functions include map
from system states to code segments and compatibility pair
to performance fulfillment with varying user requests. A
performance requirement is fulfilled if its respective Hoare
triple [10] including code quality measure, code segment and
performance expectation holds. The whole procedure is represented by an algorithm that strictly inhibits to leave a specific
requirement without being implemented and requires to reapply certain quality measure until the expected performance
goal is being achieved.
The approach was applied to develop the prototype of an
online ticketing system. Before the implementation, requirement analysis was performed to identify load extensive states,
code quality measures and expected performance goals with
respect to specific number of user requests. The prototype is
tested using a software named ab1 which is a HTTP server
benchmarking tool provided by Apache. The software was
used to generate a large amount of user traffic varying the
number of requests at a time from 100 to 1000 for a total
period of 10 second keeping the concurrency level to 100.
Results show that the system is responding within the expected
response time of 15 ms and 30 ms with minor fluctuations
due to multiple web server processes and caching limitations.
1 httpd.apache.org/docs/2.2/programs/ab.html
These results assure that the proposed methodology is capable
of ensuring scalability from the code.
The rest of the paper is organized as follows: Different
frameworks for scalability analysis and elicitation is discussed
in Section II. The proposed approach is defined in Section
III. Section IV presents the case study of an online ticketing
system on which the proposed approach is being applied.
Lastly, this paper is concluded with a discussion about few
observations visualized from results of the case study and
future research directions.
II. L ITERATURE R EVIEW
Scalability is a major concern for software development
but not much work has been done on it. Earlier, scalability
as a requirement was overlooked while developing software
systems for lack of techniques for eliciting and testing scalability requirements [7]. Most of the works focused on building
framework for scalability. Some of the significant works is
going to be discussed briefly in this section.
Scalability was analyzed for Admission Control Server
Based Resource Allocation (ACS-RA) in [4]. It was then
compared with Resource Reservation Protocol (RSVP) [11],
[12] approach. The work was done by analyzing a framework
for reserving resources according to the RSVP model in a
network cloud. They analyzed the scalability in the worst
case scenario which is a single centralized Admission Control
Server for a given differentiated services cloud [13], [14]. They
showed that ACS-RA is better than RSVP approach in the
aspect of scalability. However, investigation on the comparison
of the user plane performances was expected in their work.
Assuming a single centralized ACS to have a worst case
scenario proved their claim that ACS-RA architecture scales
better than RSVP successfully but some other scenarios should
be added.
Lloyd G. Williams et al. presented Quantitative Scalability Evaluation Method (QSEM) to directly quantify the
scalability of a software system [5]. Their method identifies
specific quantity of resources needed to be used for ensuring
scalability. They have shown results from a case study after following seven steps of QSEM. It uses straightforward
measurements of maximum throughput at different amount
of processors or nodes of the targeted system. Identifying
critical use cases, selecting representative scalability scenarios,
determining scalability requirements, planning measurement
experiments, performing measurements and presenting the
result are the steps of QSEM to complete the scalability
analysis. The steps of their method shows how to evaluate
scalability from the very first phase with identifying critical
use cases. Their evolution method can measure scalability
requirement quantitatively which can surely help to figure out
the actual requirement and outcome. This approach is good to
verify scalability from use case to application level but a code
verification step could make this approach more attractive.
As scalability is not defined and understood clearly, a
framework for accurately characterizing and analyzing scalability requirements is presented in [6]. Resolving intuitions,
ambiguities and inconsistencies of scalability is the primary
goal of that framework. Authors claimed that their framework
precisely detects dependency relationships of scalability and
describe important steps of scalability analysis. The framework
treats scalability as a multi-criteria optimization problem. They
used principles of microeconomics to support their analysis.
Scalability has not been given the systematic treatment it
deserved claimed in [7]. A framework for characterization
and analysis of software system scalability for addressing the
mentioned issue has been proposed. Application of the popular
method goal-oriented requirements engineering (GORE) [15],
[16] is described with a case study of a real word software
system in the paper. The study found appropriateness of GORE
as well as it’s limitations which are shown in their result. They
claimed that GORE lacks techniques to elaborate goal models
with respect to scalability goals. Though the advantages of
using GORE are discussed quite elaborately, drawbacks are
not discussed properly.
Leticia Duboc et al. worked for elaborating specific scalability requirements through identifying goal-obstacles to satisfy
those requirements [8]. Their method systematically elaborate
and analyze scalability requirements. Result of their research
was presented from a case study in which the method was
applied to a complex, large-scale financial fraud detection system. Method of this research was formulated from KAOS [17],
[18] goal-oriented requirements engineering method. They
used KAOS for the elaboration of goal models, management
of conflicts between goals and selecting alternative system
designs. The key issues they identified in their work can help
to fix the developers view towards scalability requirements.
The review of state of the art works shows that though
multiple issues have been considered in proposing a method
for scalability prediction, elicitation and analysis, none of the
work offer an approach to verify scalability from the code
level. However, in order to have consistent system performance
over varying workloads it is essential to write scalable code
and a formal approach should be provided for its verification.
This paper introduces one such approach for verifying scalability using set theory and Hoare triple and strictly requires to
fulfill all scalability requirements.
III. F ORMALIZING S CALABILITY V ERIFICATION
We define a set of states which will receive excessive load
as S = {s1 , s2 , ....., sn }. Let C = {c1 , c2 , ....., cn } denote the
set of code segments that eventually forms the whole system.
We also define a mapping function ψ : S → C that will
relate a specific load extensive state to a code segment. The
set of certain code quality measures that should be applied for
allowing the code to scale is defined as Q = {q1 , q2 , ....., qm }.
It is obvious that all quality measures will not be applicable
to a specific state. Thus we define a set of compatibility
pairs where each pair include a specific state and quality
measure. This is denoted as Ω = {ω1 , ω2 , ....., ωp } where
ωt = {si , qj }, i ≤ n and j ≤ m. We define the set
of expected performance outputs with varying workloads as
K = {k1 , k2 , ....., kp } where kt is a output of the expected
function ξ with respect to ωt which means kt = ξ(ωt ).
In order to fulfill a specific scalability requirement, each
compatibility pair has to satisfy the function φ : Ω → {0, 1}.
The output of the function is defined as follows:
φ(si , qj ) =
1
0
if the Hoare triple holds
otherwise
A Hoare triple of the form {Q} C {K} will hold if after
executing ci ∈ C derived from si ∈ ωt using ψ, the expected
performance output kt = ξ(ωt ) is satisfied given that qj ∈ ωt
is implied.
Algorithm 1 illustrates certain steps that need to be performed for verifying the system scalability using the proposed
formal approach. The algorithm requires the scalability requirement specification as an input and it will ensure that all
requirements are satisfied. Initially load extensive states S,
code quality measures Q, compatibility pairs Ω and expected
outcomes K will be defined. During the implementation phase,
C and mapping function ψ will be defined. After finishing the
implementation it has to be ensured that all Hoare triple hold.
If a Hoare triple does not hold then the code chunk must be
redefined with respect to code quality measures until the triple
holds.
Algorithm 1 Scalability Verification
Require: Scalability requirement specification
Ensure: All the scalability requirements are satisfied
1: Begin
2: Identify load extensive states S
3: Define code quality measures Q and compatibility pairs Ω
4: Define expected outcomes K
5: As the system being implemented define C and mapping function ψ
6: while |Ω| 6= 0 do
7:
Use ψ to get ci from si
8:
if {qj } ci {kt } holds then
9:
Ω ← Ω \ {ωt }
10:
else
11:
redefine ci with respect to qj
12:
end if
13: end while
14: State that all the scalability requirements are satisfied
15: End
IV. C ASE S TUDY
A case study was conducted on an online ticketing system
to illustrate how the proposed method can be used to verify
system scalability. This section discusses the details of that
case study including the system scenario, implication of the
proposed approach on developing the prototype of the ticketing
system and results obtained after load testing the prototype for
measuring system consistency.
A. System Scenario
The user will initially have a view of the tournament fixture
for the current season. The fixture will show the match number,
teams, venues, date and time of matches. An option will be
provided beside the matches yet to occur, enabling the user
to buy ticket. After choosing a match to buy ticket, the user
will be given the option to choose the gallery, block number,
row and the seat number. After choosing the options the price
of the ticket (according to the option) will be shown. If the
user selects the ticket to buy then s/he will be considered as
a buyer and asked for providing necessary information which
are name, address and mobile phone number.
An SMS having a secret code will be send to the submitted
mobile number. The buyer will then be provided the option
to submit that code and if the code is correct the submitted
information will be stored in the database. A unique serial
number will be generated to keep track of the buyer and s/he
will be taken to complete the payment. After receiving the
confirmation that the money has been paid, a receipt in the
PDF format, having the generated serial will be provided by
the system. The buyer can use that receipt to collect his/her
ticket from the nearest ticket collection center.
B. Exercising the proposed approach
After analyzing the whole process of this online ticketing
system using the approach of [5], some of the states are
considered as critical which must be scalable. It is showed
by the system scenario that several requests from users and
options providing by systems can be programmed in such
a way which can ensure the expected performance. Data
compression and caching are the two techniques implemented
here for scalability. Only two techniques are used to keep it
simple for this case. Code sections of critical states have been
fetched where cache and compression are applied.
Scalability is verified by the algorithm showed in earlier
section with Hoare Triple and Set Theory. Code sections with
the selected states are mapped and techniques are implemented
upon them. Different states have different quality measures.
Data compression and caching are the two quality measures
where related states are coupled to form the compatibility pair.
Expected outcomes for the compatibility pairs are made from
the user statistics and surveys as discussed in [5].
Critical states are first taken and related quality measures are
mapped. Viewing fixture by the user, gallery selection, ticket
buying option provided by the system, adding user information
and ticket purchasing information of users, updating ticket
availability and generating receipt are the critical states of
online ticketing system. The states which affects the systems
scalability is referred as critical states. Two of the critical states
are shown here as representing states.
Fixture viewing state can be cached to ensure scalability for
varying execution environment. The assumptions were based
on statistics that 10,000 users may request to view fixture
per seconds with a 56 Kbps connection. Test was done in
Local area network which indicates that the requests should
be served within 15 ms. When the state is sf and the quality
measure cache is qc , it makes the compatibility pair ωm .
Expected outcome km should be within 15 ms. {qc } cf {km }
hold, if the outcome after test is less than 15 ms.
Generating receipt is another critical state where a receipt
of the purchase is provided to the user. Compression is applied
TABLE I: Formal Scalability Verification of View Fixture and Generate Receipt Modules
Name of the state
Quality Measure (Q)
Code Chunk (S → C)
Expected Outcome (K)
Hoare Triple
View Fixture (sf )
Cache (qc )
(cf )
Response time within 15 ms up to
10,000 (km )
{qc } cf km
in this state to compress the receipt and pass over the network.
If a 311 KB receipt file becomes 259 KB after compression,
then it should take around 30 ms per request which was stated
in the requirements specification document. The compatibility
pair ωn is made with the state sr and qp . {qp } cr {kn } hold, if
expected outcome kn is less than 30 ms. Table I summarizes
the formal verification of mentioned two states.
C. Results
The prototype of the system was developed using PHP
and MySQL Database on 32 bit Linux Mint 15 machine
with Intel Core-i3 processor, 4GB RAM. It was hosted using
Apache server and load tested using the benchmarking tool
ab provided by Apache. Two modules - View Fixture and
Generate Receipt were implemented each having two versions.
In case of version one for the View Fixture module, caching
was applied following the proposed approach and the other
version involved no caching. Similarly, in case of Generate
Receipt module a version included data compression and the
other did not. User requests were generated through ab by
varying the number of requests at a time from 100 to 1000
for a total period of 10 second keeping the concurrency level
to 100. The response times in each of these four cases are
reported in Table II and III.
The graph of Table II and III is shown in Figure 1 and 2
respectively. From the graph it is observed that the response
time in case of caching is much more less than caching being
not applied. However, in case of compression though the
same thing is suppose to happen, the opposite phenomenon
is observed. This is detailed discussed in Section V
Generate Receipt (sr )
Compression (qp )
(cr )
Response time within 30 ms up to
10,000 (kn )
{qp } cr {kn }
at the same time and processed other types of data similar to
any real application server. As a result, random spikes occurred
throughout the observation. However, overall time required per
requests remained consistent.
In Figure 1b, performance in terms of no cache is depicted.
Here, the observation is that time required to serve requests
was high right from the start. The peaks are observed at
specific number of requests, such as 400, 600 and 800. The
reason is that because of lack of cache, the time require
to process requests was always high. The decreased value
is observed because the number of requests was not made
continuously, but discretely. To explain, values taken started
from 200 requests, with increment of 200 requests per round,
to 1000 requests. It is also observed that with the number
of requests handled, the time required to handle requests
increased as well, making this approach inappropriate for
scaling.
In Figure 2a and 2b, the performance is shown in terms of
time required per request for compressed and non compressed
data. However, unlike known observations, it is observed that
performance of compressed data is not good compared to
uncompressed data. The reason is that the experiment was
conducted in Local Area Network, where there is low latency
for communication between client and server end. As a result,
the time required to compress data for each request became
significant. There are also several spikes observed randomly,
which occurred due to the real system like property of our
environment serving multiple types of request simultaneously
instead of just allocating processor for providing web services.
V. D ISCUSSION
VI. C ONCLUSION AND F UTURE W ORK
An important observation can be found from Figure 1a.
Here, it is observed that even though the time required to
serve request remains somewhat consistent, there are several
significant spikes in terms of time required. This phenomenon
is observed because when a request is submitted at first, it
is processed to be stored in cache for future usage. As a
result, time required for handling further requests within the
same axes are shown to be smaller comparatively. This can be
seen in the Figure 1a where number of requests was between
200-400 range, where initial time required was from 13.5 to
14.5 unit times, whereas further requests required around 13
unit times only. The other spikes observed, particularly within
600-800 number of requests range and 800-1000 number of
requests range however, are not observed in cases of initial
time required. This happened because the system, hosting the
web server, was tasked for serving multiple types of requests
This paper proposes a formal approach that utilizes Set
theory and Hoare triple to ensure proper exercise of each
scalability requirements. The importance of verifying scalability from the code level was highlighted as it can make
applications consistent with respect to performance goals
despite rapid increase of user requests. The proposed method
applies a strictness that none of the scalability requirements
can be skipped and allows re-applying code quality measures,
like data compression and caching, until fulfilling certain
performance goals.
For experimental purpose, an online ticketing system was
developed using traditional approach without following the
proposed method. Requests varying within a range of 1001000 with constant concurrency of value 100 were sent to
the application for a time period of 10 second. Another application was developed that followed the proposed approach
TABLE II: Performance Comparison of Two Versions of the Generate Receipt Module
Number of Req. at a
Time
100
200
300
400
500
600
700
800
900
1000
Caching
Req. per Second
8282.82
7777.96
7768.76
6797.78
7976.41
7699.57
7158.67
7806.78
7481.52
7351.29
Applied
Time per Req. (ms)
12.073
12.857
12.872
14.711
12.537
12.988
13.969
12.809
13.366
13.603
No Caching Applied
Req. per Second
Time per Req. (ms)
176.98
565.027
176.76
564.783
177.29
564.051
176.75
565.764
177.39
563.743
177.09
564.676
176.64
566.113
177.47
563.468
177.19
564.37
176.22
567.468
TABLE III: Performance Comparison of Two Versions of the Generate Receipt Module
Number of Req. at a
Time
100
200
300
400
500
600
700
800
900
1000
Compression Applied
Req. per Second
Time per Req. (ms)
3614.46
27.667
3955.13
25.284
3704.21
26.996
3654.21
26.312
3841.4
26.032
3957.48
25.269
4095.89
24.415
3890.11
25.706
3917.67
25.525
3679.44
27.178
567
566.5
14.5
Time per Request (ms)
Time per Request (ms)
15
No Compression Applied
Req. per Second
Time per Req. (ms)
5392.12
18.546
5485.54
18.23
5368.53
18.627
5469.02
18.285
5533.1
18.073
5493.3
18.204
5531.74
18.078
5542.59
18.042
5501.72
18.176
5430.53
18.414
14
13.5
13
12.5
12
0
200
400
600
800
Number of Requests at a Time
1000
566
565.5
565
564.5
564
178
563.5
8200
8000
7800
7600
7400
Requests per Second
7200
7000
6800
177.5
563
0
200
177
400
Requests per Second
600
176.5
800
Number of Requests at a Time
(a) Caching Applied
1000
176
(b) No Caching Applied
Fig. 1: Performance visualization in terms of caching being applied and not applied
28
19
Time per Request (ms)
Time per Request (ms)
27.5
27
26.5
26
25.5
25
18.8
18.6
18.4
18.2
4100
24.5
4000
24
0
3900
200
400
3800
600
Number of Requests at a Time
800
3700
1000
3600
(a) Compression Applied
Requests per Second
5600
5550
18
5500
0
200
5450
400
600
Number of Requests at a Time
5400
800
5350
1000
5300
(b) No Compression Applied
Fig. 2: Performance visualization in terms of compression being applied and not applied
Requests per Second
and similar number of requests were sent. The response time
was recorded for each of those cases and results show that
the later system significantly outperforms the first one. These
results illustrate that the proposed approach can be used to
ensure scalability of a system from it’s code.
The proposed method considers fulfilling each scalability
requirement as a binary function and functions specific to
all requirements must return true for a successful exercise of
the approach. A further research challenge could be applying
specific weight to each scalability requirement and consider
the approach as a Fuzzy system. Work has to be done on how
to assign those weight and obtaining the thresh-hold value
based on which the success of the approach will be measured.
ACKNOWLEDGEMENT
This research has been supported by the University Grant
Commission, Bangladesh under the Dhaka University Teachers
Research Grant No-Regi/Admn-3/2012-2013/13190.
We want to express our gratitude to Mr. Shah Mostafa
Khaled, Assistant Professor, Institute of Information Technology, University of Dhaka for helping us in modeling the
proposed method.
R EFERENCES
[1] D. S. Rosenblum, “Software system scalability: concepts and techniques,” in Proceedings of the 2nd India software engineering conference. ACM, 2009, pp. 1–2.
[2] C. Olston, A. Manjhi, C. Garrod, A. Ailamaki, B. M. Maggs, and
T. C. Mowry, “A scalability service for dynamic web applications,” in
Proceedings of the 2nd Biennial Conference on Innovative Data Systems
Research, 2005, pp. 56–69.
[3] A. Otto, “Write scalable code,” Code Project [Online], 2011.
[4] A. Detti, M. Listanti, S. Salsano, and L. Veltri, “Supporting rsvp
in a differentiated service domain: an architectural framework and a
scalability analysis,” in International Conference on Communications,
vol. 1. IEEE, 1999, pp. 204–210.
[5] L. G. Williams and C. U. Smith, “Qsemsm: Quantitative scalability
evaluation method,” in Proc. CMG, 2005.
[6] L. Duboc, D. Rosenblum, and T. Wicks, “A framework for characterization and analysis of software system scalability,” in Proceedings of the
the 6th joint meeting of the European software engineering conference
and the ACM SIGSOFT symposium on The foundations of software
engineering. ACM, 2007, pp. 375–384.
[7] L. Duboc, E. Letier, D. S. Rosenblum, and T. Wicks, “A case study in
eliciting scalability requirements,” in Proceedings of 16th International
Requirements Engineering. IEEE, 2008, pp. 247–252.
[8] L. Duboc, E. Leiter, and D. S. Rosenblum, “Systematic elaboration of
scalability requirements through goal-obstacle analysis,” IEEE Transactions on Software Engineering, vol. 39, no. 1, pp. 119–140, 2013.
[9] B. Potter, D. Till, and J. Sinclair, An introduction to formal specification
and Z. Prentice Hall PTR, 1996.
[10] C. Stirling, “A generalization of owicki-gries’s hoare logic for a concurrent while language,” Theoretical Computer Science, vol. 58, no. 1, pp.
347–359, 1988.
[11] L. Zhang, S. Deering, D. Estrin, S. Shenker, and D. Zappala, “Rsvp:
A new resource reservation protocol,” IEEE Network, vol. 7, no. 5, pp.
8–18, 1993.
[12] G. R. Mallofre, “Resource reservation protocol (rsvp),” in Seminar on
Transport of Multimedia Streams in Wireless Internet, Department of
Computer Science University of Helsinki, Finland. Citeseer, 2003.
[13] E. Nyberg, S. Aalto, and R. Susitaival, “A simulation study on the
relation of diff-serv packet level mechanisms and flow level qos requirements,” in International Seminar on Telecommunication Networks
and Teletraffic Theory, 2002.
[14] R. Balmer, F. Baumgarter, T. Braun, and M. Gunter, “A concept for
rsvp over diffserv,” in Proceedings of Ninth International Conference
on Computer Communications and Networks. IEEE, 2000, pp. 412–
417.
[15] A. van Lamsweerde, “Goal-oriented requirements enginering: a
roundtrip from research to practice [enginering read engineering],” in
Proceedings of 12th International Requirements Engineering Conference. IEEE, 2004, pp. 4–7.
[16] A. Lapouchnian, “Goal-oriented requirements engineering: An overview
of the current research,” Technical Report http:// www.cs.toronto.edu/
∼alexei/ pub/ Lapouchnian-Depth.pdf , University of Toronto, 2005.
[17] R. Darimont, E. Delor, P. Massonet, and A. van Lamsweerde,
“Grail/kaos: an environment for goal-driven requirements engineering,”
in Proceedings of the 19th International Conference on Software engineering. ACM, 1997, pp. 612–613.
[18] W. Heaven and A. Finkelstein, “Uml profile to support requirements
engineering with kaos,” IEE Proceedings-Software, vol. 151, no. 1, pp.
10–27, 2004.
All in-text references underlined in blue are linked to publications on ResearchGate, letting you access and read them immediately.