Chapter 6: Testing and
Integration: Part Two
Ronald J. Leach
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
1
Unit Testing
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
2
What is a Unit?
• Part of a system
• The term “component” can include a unit or a
collection of units
• Components are combined into systems
• But what is a “system”?
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
3
Unit Testing
• Unit testing can be performed using white-box
or black-box methods or both. Items tested
include:
– Interfaces
– Local and global data structures
– Execution paths (normal execution)
– Error-handling paths, especially non-local gotos
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
4
Coupling-based Testing
• Based on the premise that system errors occur
most frequently at unit boundaries
• There are many types of coupling (presented
in increasing order of coupling)
• Test in the places where errors are most likely
to occur.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
5
Content Coupling
• Content coupling is the highest level of
coupling between modules
• One module modifies or relies on the internal
local data of another module.
• Changing one module’s data (location, type,
timing) will change the dependent module.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
6
Content Coupling, cont.
Unit 1
• …
• Exception occurs
• …
Unit 2
• …
• …
• Exception is handled
• …
Copyright Ronald J. Leach, 1997, 2009,
2014
7
Content Coupling, cont.
• Systems with content coupling can be tested
using recovery testing ideas.
• The flow of control is managed in a systematic
way by exception handlers.
• Ideally, exception handlers bring software
execution back to a “safe state.”
• There may be problems if multiple content
coupling occurs in a distributed system.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
8
Content Coupling, cont.
• Content coupling can also cause a disaster in
systems.
• The problems may be very hard to test.
• In languages such as C, using the system calls
setjmp and longjmp makes problems hard to
test, locate, and fix.
• Avoid these non-local GOTO statements
whenever possible.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
9
Common Coupling
• Two modules share the same global data or
global variable.
• Changing the shared resource implies
changing all the modules using it.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
10
Common Coupling, cont.
Unit 1
• …
• Use global data
• …
Unit 2
• …
• …
• Use global data
• …
Global data
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
11
Control Coupling
• One module controlling the logic of another,
by passing it information on what to do (e.g.
passing a what-to-do flag).
• Execution of a loop or program branch
depends upon the parameter.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
12
Control Coupling, cont.
Unit 1
• …
• parameter sent
• …
Unit 2
• …
• …
• if (parameter > 0)
do this
• else
do that…
Copyright Ronald J. Leach, 1997, 2009,
2014
13
Stamp Coupling
• Modules share a composite data structure
(such as a record) but use only part of it (such
as a field)
• This may change the way a module reads a
record because a field, which the module
doesn't need, has been modified.
• Also known as data-structured coupling.
• Sometimes considered as data coupling
discussed later).
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
14
Stamp Coupling, cont.
Unit 1
• …
• Multi-field record sent
• …
Unit 2
• …
• …
• One field of multi-field
record is used
• …
Copyright Ronald J. Leach, 1997, 2009,
2014
15
Data Coupling
• Modules share data through parameters.
• Parameters may be used in a calculation, but do not
affect logical flow (loops, branches) of receiving
module.
• Differs from stamp coupling since each piece of data
sent is used.
– Some researchers require parameter type to not
be a structured type such as a field.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
16
Data Coupling, cont.
Unit 1
• …
• Parameter sent
• …
Unit 2
• …
• …
• Parameter is used, but
not for control of
program unit.
• …
Copyright Ronald J. Leach, 1997, 2009,
2014
17
External Coupling
• Two use data that is external to both.
• The external data may have a specified
format, communications protocol, or device
interface.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
18
External Coupling, cont.
Unit 1
• …
• Use externally specified
data
• …
Unit 2
• …
• …
• Use externally specified
data
• …
• …
Communications such as a specified socket
Copyright Ronald J. Leach, 1997, 2009,
2014
19
Message Coupling
• A very low level of coupling.
• Modules do not communicate with each other
through parameters.
• Modules use a public interface, such as
messages or events, to exchange parameterless messages.
• More prevalent in object-oriented systems.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
20
Message Coupling, cont.
Unit 1
• …
• Use external message
• …
Unit 2
• …
• …
• Use external message
• …
• …
Message such as a mouse click
Copyright Ronald J. Leach, 1997, 2009,
2014
21
No Coupling
• The lowest form of coupling
• Modules do not appear to communicate with one
another at all.
• Concerns:
– Each module might use up system resources,
making it hard for the other module to execute.
– Very hard to find such errors without extreme
stress testing
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
22
No Coupling, cont.
Unit 1
• …
• …
• …
Unit 2
• …
• …
• …
• …
Available memory, virtual memory limits
Copyright Ronald J. Leach, 1997, 2009,
2014
23
Testing Object-Oriented Systems
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
24
OOT Strategy
• Class testing for objects is the equivalent of
unit testing for conventionally developed
modules and program units.
• operations within the class are tested
• the state behavior of the class is examined
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
25
OOT Strategy, cont.
Integration of objects into systems is often done
using any three different strategies
– thread-based testing integrates the set of classes
required to respond to one input or event
– use-based testing integrates the set of classes
required to respond to one single use case
– cluster testing integrates the set of classes
required to demonstrate one collaboration
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
26
Random Testing
• identify operations applicable to a class and define
constraints on their use
• identify a minimum test sequence
– an operation sequence that defines the minimum
operational history of the class (object)
• generate many random test sequences
– exercise other (more complex) class instance
operational histories
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
27
OOT Partition Testing
Partition Testing reduces the number of test cases
required to test a class in much the same way as
equivalence partitioning for conventional software
• state-based partitioning
– This is used to categorize and test operations
based on their ability to change the state of a class
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
28
OOT Partition Testing
• attribute-based partitioning
– This is based on the ability to categorize and test
operations based on the object’s attributes that
they use.
• category-based partitioning
– This is based on the ability to categorize and test
operations based on the generic function that
each one performs.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
29
Inter-Class Testing
• For each client class, use the list of class operators to
generate a series of random test sequences.
– The operators selected will send messages to
other server classes.
• For each message that is generated, determine the
collaborator class and the corresponding operator in
the server object.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
30
Inter-Class Testing, cont.
• For each operator in the server object (that
has been invoked by messages sent from the
client object), determine the messages that it
transmits.
• For each of the messages, determine the next
level of operators that are invoked and
incorporate these into the test sequence
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
31
Patterns
• Patterns are an extremely influential idea in
software design.
• The idea of developing software by matching
its specification to predefined and pre-coded
patterns was popularized greatly by a book by
Erich Gamma, Richard Helm, Ralph Johnson,
and John Vlissides.
• The book is often called the “Gang of Four
Book.”
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
32
Pattern Matching
• Matching an existing problem to an existing
pattern is very time-consuming and is often
part of a component selection process.
• The effort may best fit into a systems
engineering, or requirements engineering
process.
• It can also occur in the design process.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
33
Pattern Testing
• This is the major advantage of using patterns.
• Since the pattern has been pre-coded and has
been tested so well that it can be considered
to be certified as a reusable component, it is
not necessary to test the pattern.
• The testing necessary is to make certain that
other software components (generally these
are objects) coordinate with the component
that the pattern represents.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
34
O-O Test Case Design
Suggestions are provided by Ed Berard of EVB:
• Each test case should be uniquely identified and
should be explicitly associated with the class to be
tested,
• The purpose of the test should be stated.
• A list of testing steps should be developed for each
test and should contain:
– A list of specified states for the object to be
tested
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
35
O-O Test Case Design, cont.
– A list of messages and operations that will be
exercised as a consequence of the test.
– A list of exceptions that may occur as the object
is tested.
– A list of external conditions (changes in the
environment external to the software that must
exist in order to properly conduct the test).
– Include supplementary information that will aid in
understanding or implementing the test.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
36
O-O Testing Methods
Fault-based testing
– The tester looks for plausible faults (defects in the
system implementation that may result in
failures).
– To determine whether these faults exist, test cases
are designed to exercise the design or code.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
37
O-O Testing Methods
Class Testing and the Class Hierarchy
• All derived classes must be tested, even
though they are obtained from inheritance.
• Inheritance can make the testing process
much longer, because of the large number of
possibilities.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
38
O-O Testing Methods
Scenario-Based Test Design
• This concentrates on what the user does, not
what the product does.
• These tasks are captured in a collection of use
cases that users must perform
• These use cases are then used as tests.
• Potential user errors, known as “misuse
cases,” also can lead to tests.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
39
System Testing
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
40
What’s after unit testing?
• Both white-box and black-box testing are
frequently used to test relatively small
program modules – this was called “unit
testing.”
• After integration of tested components
together – whether top-down, bottom-up,
sandwich, or big-bang – a system is created.
• The system is probably too big to test properly
by white-box methods.
• What do we do?
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
41
Systems
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
42
Systems
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
43
System Testing Approaches
• The focus is on black box testing
– probably, just a subset.
• The most common approach is to test the
interfaces between program units.
• The software design specified the nature of
these interfaces.
• Focus on the most likely problem areas.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
44
System Errors
• Make sure that data intended to be passed
from one unit to another actually got there.
• Check automatically using a debugger, to
make sure that critical data was available at
the debugger’s breakpoints.
• An older, simpler method when no debugger
is present
– simply echo the data, check on its type and value.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
45
Validation Testing
Validation testing makes certain that the
software satisfies its written requirements.
• The focus is often on functional software
requirements
• Requirements are frequently functional
– (Upon input x, the system shall do y within z
seconds.)
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
46
System Testing
• System testing follows unit testing and is often
called “testing in the large.”
• Focus is on system integration
• This requires that the components of the
system have all been tested adequately.
• It also requires that the individual
components have been integrated
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
47
Security Testing
• Verifies that protection mechanisms built into
the system will prevent penetration both from
expected sources (hackers) and unexpected
ones (rogue employees)
• Generally requires a security expert as part of
the testing team (and, ideally, as part of the
requirements and design teams)
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
48
Performance Testing
• Performance testing is done most often on
systems.
• Developers often do their own analysis of
smaller units.
• Small units can be tested by benchmarks
• Such analysis may include algorithm analysis
and temporal logic
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
49
Performance Testing
• The run-time performance of software on
specified inputs must be within limits
specified by the requirements for system
performance.
• Limits can be hard, soft, or average
• System loads are usually specified in this form
of testing
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
50
Stress Testing
• Examine the throughput and reliability of the
system if there are unusually high demands on
the system’s resources.
– Protect against denial-of-service attacks.
• May be done using benchmarking software.
• May be done in conjunction with security
analysis testing and performance testing.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
51
Warnings
• Emulator test results can mislead
• Example: The Apple iPhone OS version 3.0 (Late July
2009)
–
–
–
–
Software degrades battery life severely.
To be fixed in next release
Some suggestions require users to change habits
Hardware cache size increased on the iPhone 3G to
handle increased data transfer rates
• Virtualization often provides misleading timing
information
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
52
Alpha Testing
• Here the focus is on the use of a relatively
early version of a system.
• The system is given to a select group of
customers to determine the utility of the
system by potential users.
• Users also provide feedback on the problems
not found by the development organization’s
internal software testers.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
53
Alpha Testing, cont.
• Considerable new coding and testing are
expected during this phase.
• Beware!
– Poorly tested products can anger potential
customers, even if they are getting the software
for free.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
54
Beta Testing
• Here the focus is on the use of a more solid
version of a system to many more customers
than in alpha testing.
• Customers still help determine the utility of
the system.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
55
Beta Testing, cont.
• Customers still provide feedback on the
problems not found by the development
organization’s internal software testers.
• Customers expect higher quality of software
released under beta test than under alpha
test.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
56
Acceptance Testing
• Occurs near to system delivery, before a client
or customer accepts a new system.
• A process to obtain confirmation from a
Subject Matter Expert (SME) that the system,
with any modifications or additions, meets the
requirements.
• Ideally, the owner or client of the system
tested will conduct the acceptance test.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
57
Usability Testing
• The aim is to observe people using the
product in order to discover errors and
eliminate ambiguity.
• Develop a mental model of both the
software’s users and how the software will
interact with this model.
• Users may be novices or experts, in this
software or computers generally.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
58
Why Test Usability?
• Software that is hard to use never is successful
in the marketplace.
• Developers and designers often fall in love
with technical achievements and do not
consider the problems of “average” users.
• Usability is big business.
– Jakob Nielsen’s company charges $70,000+ for a
full day seminar on usability.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
59
Usability Testing
• Observe people using the product in order to
discover errors and eliminate ambiguity.
• Create a mental model of users.
• How much time, and how many steps, are
required for people to perform certain tasks?
• How many mistakes did people make
– fatal or recoverable?)
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
60
Why model users?
• Some are highly experienced and want to
tailor their environment to their work style.
• Others want to use the software right out of
the box (or right after the download!)
• Some prefer to be guided by hierarchical
menus, even if the menus are deeply nested.
• Others prefer to have the ability to use
keystroke or mouse shortcuts.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
61
Imperfect Solution
$ rm –r * .jpg
User probably meant
rm –r *.jpg
Why not prompt the user before every deletion?
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
62
Imperfect Solution, cont.
Why not prompt the user before every deletion?
Are you sure?
Are you sure?
Are you sure?
Are you sure?
Prompting users for every deletion can be
annoying, too.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
63
Usability Laboratories
• At companies
• At universities
• May be available for a fee
– Students test software usability
– External users test software usability
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
64
Labs usually have:
• video cameras
• software for monitoring keystrokes and mouse inputs
• facilities for tracking eye movement focused on a
monitor screen
• expert professional analysis of results of data
collection
• usability reports
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
65
Testing without Laboratories
• Select a fixed group of people, generally its
employees or their families, to test usability.
• Perhaps even crowd testing
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
66
Testing Plans
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
67
Elements that affect test plans
• There should be a precise list of the
requirements that must be tested.
• Without this list, there is no way to tell if the
requirements have been met.
• The test plan must be consistent with the
goals of the organization.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
68
Test Plan Elements, cont.
• If the most important goal of the organization
is to have a minimal test of the software’s
essential features as determined by the
marketing department, the test plan should
test only those features as a first priority.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
69
Test Plan Elements, cont.
• After all the most essential set of
requirements have been tested, the next set
of test cases should consider the cases that
most users are likely to want.
• Test software features unlikely to be
encountered in practice if time permits.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
70
Test Plan Elements, cont.
• The test plan should specify the operational
environment.
• For PCs: test minimal, maximal and the most
common configurations.
• The plan must allocate a sufficient amount of
time to be used for testing (at each phase)
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
71
Test Plan Elements, cont.
• The set of personnel available for testing must
be known and part of the plan.
• The plan must ensure that there are adequate
hardware resources available for testing.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
72
Test Plan Elements, cont.
• The plan must make sure that there are
adequate software resources available for
testing, usually in the form of test drivers and
testing tools.
• There must be a sufficient amount of available
free computing cycles resources that are to be
used for testing.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
73
Test Plan Elements, cont.
• The plan must address the available tools that
can be used for testing data analysis.
• The plan must take into account any existing
standards and practices manual.
• The plan must consider the type of software
(object oriented, procedural, mixed).
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
74
Regression Testing and
Configuration Management
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
75
Regression Testing
• Unfortunately, the term “regression testing” is
somewhat misleading.
• In psychology, the term refers to a person
whose behavior falls back to a previous, often
much younger or immature state.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
76
Regression Testing
• In software testing, a regression occurs when
previously working software functionality
stops working after some change to the
software.
• The most common method of regression
testing is to re-run previously executed tests
and check that previously fixed faults have not
returned.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
77
What happens after?
• Suppose we need to change the software to fix an
observed error.
• Once it has been changed and passes the regression
test, the changed version is the one that we will
probably work with.
– What if a new, fatal error is found?
– Can we go back to a previous version?
– Yes, if proper configuration management tools and
techniques are used.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
78
What to do with Test Data
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
79
What do we do with test data?
• At this point you have done some testing.
• You have reported the errors you found in
some sort of database or organization-specific
reporting system.
• Once the developers fixed the errors, they
submitted the source code again for testing.
• You find new errors and report them.
• Why is the reporting procedure so formal?
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
80
Test data usage
• The answer lies in the way that senior
management uses, or should use, test data.
• The test database is not just a repository for
data about the status of the software’s
testing.
• It contains information that can greatly
improve the organization’s software
development process if used efficiently.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
81
The Test Results Database
• The test results database is a tool for
managers.
• It is also a useful tool for software designers.
• How?
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
82
Test Results Database, cont.
• One field always present in a test database is
the severity of each error found during
testing.
• Errors with high severity will always be fixed
before ones with low severity.
• The database can be used to help with
scheduling software development effort.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
83
The Test Database
• Most software products, even in safety-critical
applications, are released or deployed with
some residual errors.
• What are these errors?
• Where is information on these errors?
– The test database!
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
84
The Test Database, cont.
• Other kinds of information can be found in the
database, too.
• Most reporting databases have fields for the
date and time when the error was placed in
the system and the date and time that the
error was corrected.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
85
The Test Database, cont.
• The difference indicates how long problems
remain unresolved.
• Problems with high severity, limit how soon
software can be released.
• Other problems can be found by examining
the database carefully, also.
• A subcontractor or employee whose work is
more free of errors than others may be
producing high quality work
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
86
Using the Test Database Example
• A NASA system had become very expensive to
modify to increase its functionality for new
applications.
• What was the cause?
– Unclear, the development team and
application domain had not changed from
previous projects.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
87
Using the Test Database Example
• The offending software component had never
been reengineered to make use of more
modern software tools.
Reengineer!
• The solution was to reengineer the offending
module.
• After this, the software far exceeded quality
standards in terms of defects/KLOC.
• Further analysis of the database showed that
no other changes were considered necessary.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
89
Assessments of Quality
•
•
•
•
There should always be a defect test DB
It can illustrate quality measures
One common measure: defects/KLOC
Other measurements are used
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
90
Software Quality Assurance
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
91
Software Quality Assurance
Software QA includes:
• Process definition and standards
• Formal (technical) reviews
• Measurement
• Analysis and reporting
• Planning and review
• Standards (ISO 9001, etc.)
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
92
Six-Sigma Software
• The term “six sigma” refers to six standard
deviations
• Approximately 3.4 defects per million
occurrences
• Extremely difficult to achieve.
• Compare to 2 defects/KLOC
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
93
Formal Reviews
• Plan (materials, participants, place)
• Assign roles (facilitator, discussion leader,
examiner, scribe, etc)
• Participants prepare their roles
• Present system overview at meeting
• Find defects at meeting
• Rework to fix defects found
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
94
Formal Reviews
• Follow-up inspection for defects found
• (Also known as Fagan Inspections)
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
95
A Manager’s Viewpoint on
Software Testing
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
96
A Manager’s Viewpoint
• Managers do not want any unpleasant
surprises in the testing and integration phase.
• The testing process should proceed in an
organized, systematic way.
• Managers want to review the test plans and
some of the test data before testing starts.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
97
A Manager’s Viewpoint, cont.
• Insure that the requirements traceability
matrix is checked, perhaps leaving the
responsibility for such testing to the QA team.
• Metrics data are kept and used during testing
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
98
A Manager’s Viewpoint, cont
• Managers will usually have troublesome
modules identified at this stage, with
reallocation of resources if necessary.
• The integration process must proceed in a
systematic way.
• Managers require regression testing during
integration since modules may be changed .
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
99
A Manager’s Viewpoint, cont.
• Managers will arrange for metrics data to be
collected and used during integration.
• This will also help to identify modules that are
troublesome at this stage.
• Managers will also want the process to be
efficient, with testing proceeding on schedule.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
100
A Manager’s Viewpoint, cont.
• A minimal set of testing tools includes:
– Test drivers
– Stub generators
– Test harnesses (to execute functions and
objects without creating wrapper code)
– Test data management tool
– Regression tester
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
101
A Manager’s Viewpoint, cont.
– Path analyzer (to determine all decision
points)
– Static analyzer (to determine anomalies in
the source code’s organization)
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
102
A Manager’s Viewpoint, cont.
• For many software systems, especially those in
safety-critical environments, a fault injection
tool will also be used.
• Such software tools deliberately place faults
within code to determine if faults in
subsystems can be isolated, or if the effects of
the faults propagate throughout the software
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
103
The Agile Case Study: Testing and
Integration
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
104
Agile Testing
• Agile testing is a practice that follows the socalled “agile manifesto,” emphasizing testing
from the perspective of customers who will
use the system.
• Agile testing is not a separate phase in a life
cycle.
• Agile testing deemphasizes more defined
testing procedures.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
105
Agile Testing, cont.
• Because of the lack of more formally defined
testing procedures, agile development is not
employed on most government or commercial
software projects.
• Agile development is almost never used on
safety-critical systems.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
106
Agile testing, cont.
• The agile testing process tests newly
developed code iteratively until sufficient
quality is achieved, at least from the
perspective of end users.
• The goal is to have the entire project team,
including testers and end users, work toward
demonstrable quality.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
107
Testing is embedded in the agile
process
• Developers work hand-in-hand with
customers during development.
• Regression testing is universally used.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
108
Testing in our case study
• One goal was to minimize testing by using
COTS products, even ones not often used in
this application domain (such as MatLab).
• Regression testing was constantly used.
• Our case study did not use test-driven
development (defined on the next slide)
extensively.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
109
Test-driven development
• Automated tests are created, after which
software is written to minimally pass the test.
• The tests are created with the input of the
customers working with the agile process
software developers.
• This extends the idea of a requirements
traceability matrix and testable requirements.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
110
The Major Continuing Software
Project: Testing
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
111
A high-level view
• The entire software system is comprised of
modules, many obtained from other sources.
• Software utilities are considered as black
boxes
• Only interfaces need to be checked. The same
approach when testing the “glueware” used to
combine these utilities.
• Black-box testing is a natural choice for this.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
112
White-box testing?
• This is appropriate when there are many
execution paths and complex internal logic.
• This describes the software interfacing to the
operating system for the linking of names of
input files to later routines.
• This was to be done using command-line
arguments.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
113
Choices
For small units, choose one of:
• White-box testing
• Black-box testing
For subsystems (larger units):
• Choose procedural testing for procedurally
developed subsystems
• Choose object-oriented testing for O-O
developed subsystems
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
114
Oobject-oriented methods
• Must be applied to the back end of the
system.
• The main testing for this subsystem are
completeness of the objects and the proper
assignment of default values to member
functions.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
115
The Major Continuing Software
Project: Integration
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
116
Top-down or bottom-up or other?
• Start with a top-down approach.
• Check user interface to see if file names are
handled properly.
• Check to see if the first subsystem recognizes
the names of input files and the appropriate
directories .
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
117
Integrate objects bottom-up
• The back end of the software system consists
of O-O modules for output data storage and
for display.
• Integrate these objects bottom up
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
118
Combine the system’s front and
back ends
• A hybrid integration approach.
• Ideally, very few steps are necessary.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
119
Check the RTM for completeness
• The Requirements Traceability Matrix lists
each requirement and a test for it.
• Make sure that all requirements are actually
implemented and tested for.
Copyright Ronald J. Leach, 1997, 2009,
2014, 2015
120