Lecture 2
Software Testing
2.1 System Testing
2.2 Component Testing ( Unit testing )
2.3 Test Case Design
2.4 Test Automation
The Testing Process
 System testing
 Testing of groups of components integrated to create a
system or sub-system
 The responsibility of an independent testing team
 Tests are based on a system specification
 Component testing (Unit testing )
 Testing of individual program components
 Usually the responsibility of the component developer
(except sometimes for critical systems)
 Tests are derived from the developer’s experience
The software testing process
2.1 System testing
 Involves integrating components to create a
system or sub-system
 May involve testing an increment to be delivered
to the customer
 Two phases:
 Integration testing - the test team have access to the
system source code. The system is tested as components
are integrated
 Release testing - the test team test the complete system
to be delivered as a black-box
2.1.1 Incremental Integration Testing
Repeated again
A
T1
A
T1
T2
A
T2
T2
B
T3
B
T3
B
C
T3
T4
C
T4
D
Test sequence 1
T1
Test sequence 2
Test sequence 3
T5
2.1.2 Release Testing
 The process of testing a release of a system
that will be distributed to customers
 Primary goal is to increase the supplier’s
confidence that
requirements
the
system
meets
its
 Release testing is usually black-box or
(functional testing)
 Based on the system specification only;
 Testers do not have knowledge of the system
implementation
Black-box testing
Testing scenario-example
A studen t in Scotland is studying A merican History and has been asked to writ e a paper
on ÔFrontier mentalit y in the American West from 1840 to 1880 Õ.To do this, she need s to
find sourc es from a range o f libraries. She logs on to the LIBSYS system and uses the
search facilit y to discove r if she can acce ss original documents from t hat tim e. She
discover s sour ces in va rious US un iversit y li braries and down loads copies of some of
these. However , for one document, she needs to have confirmation from her un iver sit y
that she is a genu ine studen t and that use is f or non- commercial pu rposes. The s tudent
then use s the facil ity in LIBSYS that can reque st such permis sion and registers her
reques t. If granted, the document will be down loaded to the registered li braryÕs server
and printed for her . She receives a message f rom LIBSYS telli ng her that she will receive
an e-mail message when th e printed docu me nt is ava il able for coll ection.
Frontier mentality is some type of trial without judges and
courts, was the dominate situation in America!!!!!!
Testing scenario-example
(what to test?)
1.
Test the login mechanism using correct and incorrect logins to check
that valid users are accepted and invalid users are rejected.
2.
Test the search facility using different queries against known sources to
check that the search mechanism is actually finding documents.
3.
Test the system presentation facility to check that information about
documents is displayed properly.
4.
Test the mechanism to request permis sion for downloading.
5.
Test the e-mail response indicating that the downloaded docume nt is
available.
Use cases
 Use cases can be a basis for deriving the tests for a
system
 They help identify operations to be tested and help
design the required test cases
 From an associated sequence diagram, the inputs
and outputs to be created for the tests can be
identified.

Example(ATM system):visit http://www.mathcs.gordon.edu/courses/cs211/ATMExample/UseCases
.html#Startup
Performance testing
 Part of release testing may involve testing the
emergent properties of a system, such as
performance and reliability
 Performance tests usually involve planning a
series of tests where the load is steadily
increased until the system performance
becomes unacceptable
Stress testing
 Exercises the system beyond its maximum design load.
Stressing the
come to light
system
often
causes
defects
to
 Stressing the system test failure behaviour
Systems should not fail disastrously.
Stress testing checks for unacceptable loss of service or
data( server recovery time).
 Stress testing is particularly applicable to distributed
systems that can exhibit severe degradation as a
network becomes overloaded.
2.2 Component testing (Unit testing )
 Component or unit testing is the process of
testing individual components in isolation.
 It is a defect testing process.
 Components may be:
 Individual functions or methods within an object;
 Object classes with several attributes and methods;
 Composite components with defined interfaces used to
access their functionality.
2.2.1 Object class testing
 Complete test coverage of a class involves
 Testing all operations associated with an object;
 Setting and interrogating all object attributes;
 Exercising the object in all possible states.
 Inheritance makes it more difficult to design
object class tests as the information to be
tested is not localised.
Weather Station Software Example
 It is a package of software controlled
instruments which collects data, performs some
data processing and transmits this data for
further processing.
 The instruments include air and ground
thermometers, an anemometer, a wind vane, a
barometer and a rain gauge. Data is collected
periodically.
 When a command is issued to transmit the
weather data, the weather station processes and
summarises the collected data. The summarised
data is transmitted to the mapping computer
when a request is received.
Weather station object interface-example
WeatherStation
identifier
repo rtWeather ()
calibrate (instruments)
test ()
startup (instruments)
sh utdo wn (instruments)
Weather station testing example-cont
 Need to define test cases for the operations:
reportWeather,
calibrate,
test,
startup
shutdown
 Using a state model:
identify sequences of state transitions to be
tested
The event sequences to cause these transitions
2.2.2 Interface testing
 Objectives are to detect faults due to
interface errors or invalid assumptions
about interfaces.
 Particularly important for object-oriented
development as objects are defined by their
interfaces.
Interface testing-cont
Interaction
between
classes could
not be tested.
But the net
result could be
sensed via the
main interface
Interface errors
 Interface misuse
 A calling component calls another component
and makes an error in its use of its interface
(e.g. parameters in the wrong order).
 Interface misunderstanding
 A calling component embeds assumptions
about the behaviour of the called component
which are incorrect ( may be implemented by
different person ).
 Timing errors
 The called and the calling component operate
at
different
speeds
and
out-of-date
information is accessed.
2.3 Test Case Design
 Involves designing the test cases (inputs and
outputs) used to test the system
 The goal of test case design is to create a set
of tests that are effective in validation and
defect testing
 Design approaches:
 Requirements-based testing;
 Partition testing;
 Structural testing
2.3.1 Requirements Based Testing
 A
general principle of requirements
engineering is that requirements should be
testable.
 Requirements-based testing is a validation
testing technique where you consider each
requirement and derive a set of tests for that
requirement.
LIBSYS requirements-example
R1
The user shall be able to search either all of the initial set of databases or select a
subset from it.
R2
The system shall provide appropriate viewers for the user to read documents in the
document store.
R3
Every order shall be allocated a unique identifier (ORDER_ID) that the user shall
be able to copy to the accountÕs permanent storage area.
LIBSYS requirements based testing cases example-cont
1

2

3

4

5

Initia te user search for searches fo r items that are known to
be present and known not to be present, whe re the set of
databases include s 1 database.
Initia te user searches fo r items that are known to be present
and known not to be present, whe re the set of databases
include s 2 databases
Initia te user searches fo r items that are known to be present
and known not to be present whe re the set of databases
include s more than 2 databases.
Select one d atabase from t he set of databases and initi ate
user searches fo r it ems that are known to be present and
known not to be present.
Select more than on e database from the set of databases
and initiate searches for items that are known to be present
and known not to be present.
2.3.2 Partition Testing
 Input data and output results often fall into
different classes where all members of a class
are related
 Each of these classes is an equivalence
partition or domain where the program
behaves in an equivalent way for each class
member
 Test cases should be chosen from each
partition
Equivalence partitioning
Equivalence partitions
Search routine specification
procedure Search (Key : ELEM ; T: SEQ of ELEM;
Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition
-- the sequence has at least one element
T’FIRST <= T’LAST
Post-condition
-- the element is found and is referenced by L
( Found and T (L) = Key)
or
-- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Search routine - input partitions
 Inputs which conform to the pre-conditions
 Inputs where a pre-condition does not hold
 Inputs where the key element is a member
of the array
 Inputs where the key element is not a
member of the array
Testing guidelines(sequences)
 Test software with sequences which have
only a single value
 Use sequences of different sizes in different
tests
 Derive tests so that the first, middle and last
elements of the sequence are accessed
 Test with sequences of zero length(null)
Search routine - input partitions
( 6 Testing cases )
Sequ ence
Single value
Single value
More than 1 value
More than 1 value
More than 1 value
More than 1 value
Inp ut sequ ence (T)
17
17
17, 29, 21, 23
41, 18, 9, 31, 30, 16, 45
17, 18, 21, 23, 29, 41, 38
21, 23, 29, 33, 38
Eleme nt
In sequence
Not in sequence
First element in sequence
Last element in sequence
Middle eleme nt in sequence
Not in sequence
Key (Key)
17
0
17
45
23
25
Output (Found, L)
true, 1
false, ??
true, 1
true, 7
true, 4
false, ??
2.3.3 Structural Testing
 Sometime called white-box testing
 Derivation
of test
program structure
cases
according
to
 Knowledge of the program is used to identify
additional test cases
 Objective
is to exercise all program
statements (not all path combinations)
Binary search - equivalent partitions
 Pre-conditions satisfied, key element in array
 Pre-conditions satisfied, key element not in array
 Pre-conditions unsatisfied, key element in array
 Pre-conditions unsatisfied, key element not in array
 Input array has a single value
 Input array has an even number of values
 Input array has an odd number of values
Binary search equivalent partitions
Binary search - ( 8 test cases)
Inp ut array (T)
17
17
17, 21, 23, 29
9, 16, 18, 30, 31, 41, 45
17, 18, 21, 23, 29, 38, 41
17, 18, 21, 23, 29, 33, 38
12, 18, 21, 23, 32
21, 23, 29, 33, 38
Key (Key)
17
0
17
45
23
21
23
25
Output (Found, L)
true, 1
false, ??
true, 1
true, 7
true, 4
true, 3
true, 4
false, ??
2.3.3.1 Path testing
 It is a white-box testing strategy.
 The objective of path testing is to ensure
that the set of test cases is such that each
path through the program is executed at
least once
 The starting point for path testing is a
program flow graph( not DFD ) that shows
nodes representing program decisions and
arcs representing the flow of control
 Statements with conditions are therefore
nodes in the flow graph
Binary search flow graph-example
Program decision
Flow of control
program flow graph
Binary search flow graph example- cont
(Independent paths)
 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14
 1, 2, 3, 4, 5, 14
 1, 2, 3, 4, 5, 6, 7, 11, 12, 5, …
 1, 2, 3, 4, 6, 7, 11, 13, 5, …
 Test cases should be derived so that all of these paths
are executed
 A dynamic program analyser may be used to check that
paths have been executed
2.4 Test Automation
 Testing
is an expensive process phase. Testing
workbenches provide a range of tools to reduce the time
required and total testing costs.
 Systems such as Junit support the automatic execution
of tests. ( get a look at http://junit.sourceforge.net/ )
 Most testing workbenches are open systems because
testing needs are organisation-specific.
 They are sometimes difficult to integrate with closed
design and analysis workbenches.
Topics in Metrics for Software Testing
Quantification
•One of the characteristics of a maturing
discipline is the replacement of art by science.
• Quantification was impossible until the right
questions were asked.
 Computer Science is slowly following the
quantification path.
• There is skepticism because so much of what we want
to quantify it tied to erratic human behavior.
Software quantification
 Software Engineers are still counting lines of
code.
• This popular metric is highly inaccurate when
used to predict:
– costs
– resources
– schedules
Software quantification(
Cont.)
• Lines of Code (LOC) is not a good measure
software size.
• In software testing we need a notion of size when
comparing two testing strategies.
• The number of tests should be normalized to
software size, for example:
– Strategy A needs 1.4 tests/unit size.
Asking the right questions
 When can we stop testing?
• How many bugs can we expect?
• Which testing technique is more effective?
• Are we testing hard or smart?
• Do we have a strong program or a weak test
suite?
• Currently, we are unable to answer these
questions satisfactorily.
Metrics taxonomy
 Linguistic Metrics: Based on measuring
properties of program text without
interpreting
what the text means. – E.g., LOC.
• Structural Metrics: Based on structural
relations between the objects in a
program. – E.g., number of nodes and
links in a control flowgraph.
Lines of code (LOC)
 LOC is used as a measure of software complexity.
• This metric is just as good as source listing weight
if we assume consistency w.r.t. paper and font
size.
• Makes as much sense (or nonsense) to say:
– “This is a 2 pound program”
• as it is to say:
– “This is a 100,000 line program.”
Lines of code paradox
 Paradox: If you unroll a loop, you reduce
the complexity of your software ...
• Studies show that there is a linear
relationship between LOC and error rates for
small programs (i.e., LOC < 100).
• The relationship becomes non-linear as
programs increases in size.
Halstead’s program length
Example of program length
Example of program length
Structural metrics
 Linguistic complexity is ignored.
 Attention is focused on control-flow and
data-flow complexity.
 Structural metrics are based on the
properties of flowgraph models of programs.
Cyclomatic complexity
 McCabe’s Cyclomatic complexity is defined as: M
= L - N + 2P
• L = number of links in the flowgraph
• N = number of nodes in the flowgraph
• P = number of disconnected parts of the
flowgraph.
Examples of
cyclomatic complexity
Pitfalls
• if ... then ... else has the same M as a loop!
• case statements, which are highly
regular structures, have a high M.
• Warning: McCabe’s metric should be
used as a rule of thumb at best.
Rules of thumb based on M
 Bugs/LOC increases discontinuously for M > 10
 M is better than LOC in judging life-cycle
efforts.
 Routines with a high M (say > 40) should be
scrutinized.
 M establishes a useful lower-bound rule of
thumb for the number of test cases required
to achieve branch coverage.
Assignment 2
 Halstead software science metrics was developed by
Maurice Halstead, who claimed they could be
used to evaluate the mental effort (E) , time
required to create a program (T), and Predicted
number of bugs (B).
 These metrics are based on four primitives listed
below:
n1 = number of unique operators
n2 = number of unique operands
N1 = total occurrences of operators
N2 = total occurrences of operands
Assignment 2-cont








Software science metrics are listed below:
Vocabulary: n = n1 + n2
N1 = n1 log2 n1
N2 = n2 log2 n2
Program Length N = N1 + N2
Program volume: V = N * log2n = (N1 + N2 ) log2( n1 + n2)
Effort required to generate the program ( as a measure of
text complexity ) is :
E = (n1 N2 N log2n)/ (2n2)
Time required to generate the program is T = E/18
Predicted theoretical initial number of bugs B = (E) 2/3/3000
Assignment 2-cont
 Given that a software application contains 2048
operators and 1024 operand
(a) Compute V,E,T, theoretical B
(b) Assume that Three different practical
feedbacks indicated that the practical founded
bugs was as follows:
2000 bugs in the first feedback
825 bugs in the second feedback
100 bugs in the third feedback
Give your comment for each feedback
Assignment 2
 Halstead software science metrics was developed by
Maurice Halstead, who claimed they could be
used to evaluate the mental effort (E) , time
required to create a program (T), and Predicted
number of bugs (B).
 These metrics are based on four primitives listed
below:
n1 = number of unique operators
n2 = number of unique operands
N1 = total occurrences of operators
N2 = total occurrences of operands
Assignment 2-cont








Software science metrics are listed below:
Vocabulary: n = n1 + n2
N1 = n1 log2 n1
N2 = n2 log2 n2
Program Length N = N1 + N2
Program volume: V = N * log2n = (N1 + N2 ) log2( n1 + n2)
Effort required to generate the program ( as a measure of
text complexity ) is :
E = (n1 N2 N log2n)/ (2n2)
Time required to generate the program is T = E/18
Predicted theoretical initial number of bugs B = (E) 2/3/3000
Assignment 2-cont
 Given that a software application contains 2048
operators and 1024 operand
(a) Compute V,E,T, theoretical B
(b) Assume that Three different practical
feedbacks indicated that the practical founded
bugs was as follows:
2000 bugs in the first feedback
825 bugs in the second feedback
100 bugs in the third feedback
Give your comment for each feedback