Conquering Complex and Changing Systems
Object-Oriented Software Engineering
Chapter 9,
Testing
Terminology




Reliability: The measure of success with which the observed
behavior of a system confirms to some specification of its
behavior.
Failure: Any deviation of the observed behavior from the
specified behavior.
Error: The system is in a state such that further processing by
the system will lead to a failure.
Fault (Bug): The mechanical or algorithmic cause of an error.
There are many different types of errors and different ways how
we can deal with them.
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
2
What is this?
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
3
Erroneous State (“Error”)
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
4
Algorithmic Fault
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
5
Mechanical Fault
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
6
How do we deal with Errors and Faults?
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
7
Modular Redundancy?
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
8
Patching?
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
9
Testing?
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
10
Declaring the Bug
as a Feature?
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
11
Examples of Faults and Errors

Faults in the Interface
specification
 Mismatch between what the
client needs and what the
server offers
 Mismatch between
requirements and
implementation

Algorithmic Faults
 Missing initialization
 Branching errors (too soon,
too late)
 Missing test for nil
Bernd Bruegge & Allen Dutoit

Mechanical Faults (very
hard to find)
 Documentation does not
match actual conditions or
operating procedures

Errors




Stress or overload errors
Capacity or boundary errors
Timing errors
Throughput or performance
errors
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
12
How to Deal with Errors

Error prevention (Fault Avoidance) - before system is released:





Use good programming methodology to reduce complexity
Use version control to prevent inconsistent system (configuration mgmt)
Apply verification to prevent algorithmic bugs
Review of code by inspection team
Error detection (Fault detection) - while system is running:
 Testing: Create failures in a planned way
 Debugging: Start with an unplanned failures
 Monitoring: Deliver information about state. Find performance bugs

Error recovery (Fault Tolerance) - recover from failure while
system is running:
 Data base systems (atomic transactions)
 Modular redundancy
 Recovery blocks
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
13
Some Observations

It is impossible to completely test any nontrivial module or any
system
 Theoretical limitations: Halting problem
 Practial limitations: Prohibitive in time and cost


Testing can only show the presence of bugs, not their absence
(Dijkstra)
Often more effective to have independent tester
 Programmer often sticks to the data set that makes the program
work
 A program often does not work when tried by somebody else.
 Don’t let this be the end-user!
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
14
Testing Activities
Subsystem
Code
Subsystem
Code
Unit
Test
Unit
Test
Tested
Subsystem
Tested
Subsystem
Requirements
Analysis
Document
System
Design
Document
Integration
Test
Integrated
Subsystems
Functional
Test
User
Manual
Functioning
System
Tested Subsystem
Subsystem
Code
Unit
Test
Bernd Bruegge & Allen Dutoit
All tests by developer
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
15
Testing Activities ctd
Client’s
Understanding
of Requirements
Global
Requirements
Validated
Functioning
System PerformanceSystem
Test
Accepted
System
Acceptance
Test
User
Environment
Installation
Test
Usable
System
Tests by client
Tests by developer
User’s understanding
Tests (?) by user
Bernd Bruegge & Allen Dutoit
System in
Use
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
16
Quality Assurance encompasses Testing
Quality Assurance
Usability Testing
Scenario
Testing
Fault Avoidance
Verification
Prototype
Testing
Product
Testing
Fault Tolerance
Atomic
Transactions
Configuration
Management
Modular
Redundancy
Fault Detection
Reviews
Walkthrough
Debugging
Inspection
Testing
Component
Testing
Bernd Bruegge & Allen Dutoit
Integration
Testing
System
Testing
Correctness
Debugging
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
Performance
Debugging
17
Testing Concepts


Component – part of system that can be isolated for testing
Test case – set of inputs and expected results that tries to cause
failures and detect faults
 Black box tests – input/output behavior of component
 White box tests – internal structure of component


Test stub – partial implementation of components on which the
tested component depends
Test driver – partial implementation of a component that
depends on the tested component
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
18
Testing Concepts (cont)

Correction – a change made to a component, to repair a fault
 A correction can introduce new faults! Then what?

Problem Tracking
 Documentation of every failure, error, and fault
 Used with configuration management to find the new faults

Regression testing
 After a change, re-execute all prior tests to ensure that previously
working functionality has not been affected

Rationale Maintenance
 Documentation of rationale of the change
 Developers can avoid new faults by reviewing assumptions and
rationales of the other developers
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
19
Component Testing

Unit Testing:
 Individual subsystem
 Carried out by developers
 Goal: Confirm that each subsystem is correctly coded and carries
out the intended functionality

Integration Testing:
 Groups of subsystems (collection of classes) and eventually the entire
system
 Carried out by developers
 Goal: Test the interfaces among the subsystems
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
20
System Testing

System Testing:
 The entire system
 Carried out by developers
 Goal: Determine if the system meets the requirements (functional
and global)

Acceptance Testing:
 Evaluates the system delivered by developers
 Carried out by the client. May involve executing typical
transactions on site on a trial basis
 Goal: Demonstrate that the system meets customer requirements
and is ready to use

Implementation (Coding) and testing go hand in hand
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
21
Unit Testing

Informal:
 Incremental coding

Static Analysis:





Hand execution: Reading the source code
Walk-Through (informal presentation to others)
Code Inspection (formal presentation to others)
Automated Tools checking for
 syntactic and semantic errors
 departure from coding standards
Dynamic Analysis:
 Black-box testing (Test the input/output behavior)
 White-box testing (Test the internal logic of the subsystem or object)
 Data-structure based testing (Data types determine test cases)
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
22
Black-box Testing

Focus: I/O behavior. If for any given input, we can predict the
output, then the module passes the test.
 Almost always impossible to generate all possible inputs ("test
cases")
 Need to reduce number of test cases:

Equivalence Testing
 Divide input conditions into equivalence classes
 Choose test cases for each equivalence class. (Example: If all
negative inputs should behave the same, just test one negative)

Boundary Testing
 Special case of equivalence testing
 Select elements from the “edges” of equivalence classes
 Developers often overlook special cases at the boundary

Example: use of >= instead of >
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
23
White-box Testing


Focus: Thoroughness (Coverage). Every statement in the
component is executed at least once.
Path testing
 Exercise all possible paths through a piece of code
 Start with flow graph, and graph all possible paths (including all
choices at branch statements and loops)
 Limitation: Doesn’t always detect things that are missing.

State-based testing
 Focuses more on object-oriented systems
 Derive test cases with start state, transition stimuli, and expected
end state
 Difficulty: must include sequences to put system in desired start
state before transitions are tested
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
24
White-box Testing Example: Determining the Paths
FindMean (FILE ScoreFile)
{ float SumOfScores = 0.0;
int NumberOfScores = 0;
1
float Mean=0.0; float Score;
Read(ScoreFile, Score);
2 while (! EOF(ScoreFile)) {
3 if (Score > 0.0 ) {
SumOfScores = SumOfScores + Score;
NumberOfScores++;
}
5
Read(ScoreFile, Score);
4
6
}
/* Compute the mean and print the result */
7 if (NumberOfScores > 0) {
Mean = SumOfScores / NumberOfScores;
printf(“ The mean score is %f\n”, Mean);
} else
printf (“No scores found in file\n”);
9
}
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
8
25
Constructing the Logic Flow Diagram
Start
1
F
2
T
3
T
F
5
4
6
7
T
F
9
8
Exit
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
26
Comparison of White & Black-box Testing

White-box Testing:
 White-box testing often tests what is done, instead of what should be
done
 Cannot detect missing use cases

Black-box Testing:
 Potential combinatorial explosion of test cases (valid & invalid data)
 Often not clear whether the selected test cases uncover a particular
error
 Does not discover extraneous use cases ("features")


Both types of testing are needed
White-box testing and black box testing are the extreme ends of a testing
continuum.
Bernd Bruegge & Allen Dutoit
Object-Oriented Software Engineering: Conquering Complex and Changing Systems
27