ISQB Software Testing
Section Meeting
10 Dec 2012
Why testing is necessary
• Increasing complexity of software
• Cost of software failures
ARIANE 5, Flight 501 Failure, Report by the
Inquiry Board
"The internal [...] software
exception was caused during
execution of a data
conversion from 64-bit floating
point to 16-bit signed integer
value. The floating point number
which was converted had a value
greater than what could be
represented by a 16-bit signed
integer. [...].“
Test and Quality
• Quality: the degree to which a component,
system or process meets specified
requirements and/or user/customer needs
and expectations.
• Software quality: the totality of functionality
and features of a software product that bear
on its ability to satisfy stated or implied needs.
ISO 9126 Software Quality
Software Quality Characteristics:
• Functionality:
– Specified or expected functions (behavior): suitability, accuracy, interoperability, compliance,
security
• Reliability:
– Maintenance of a certain level of performance: maturity, fault tolerance, recoverability
• Usability:
– Effort needed for use, user-friendliness: understandability, learnability, operability
• Efficiency:
– Ratio of level of performance and used resources: time behaviour, resource behaviour
• Maintainability:
– Effort to perform defined changes: analysability, changeability, stability, testability
• Portability:
– Ability to transfer to a different environment (SW, HW, etc.): adaptability, installability,
conformance, replaceability
Testing principles
• Testing can show the presence of defects but
not their absence
• Exhaustive testing is impossible
• Early testing
• Defect clustering
• Pesticide paradox
• Testing is context dependent
• Absence-of-errors fallacy
Fundamental Test Process
Test Level Acceptance Testing
Test Level System Testing
Test Level Integration Testing
Test Level Component Testing
Planning
C
o
n
t
r
o
l
Analysis &
Design
Implementation
& Execution
Exiting
criteria
Test cycle 1...n
Testing
Closure
Test Cases Specification
Logical and Concrete Test Cases
• Example: Logical test case
• Example: Concrete test case
Test Oracle
After each executed test case it has to be
decided whether there is a failure or not:
• the observed result/outcome is compared to
the expected result/outcome
• therefore the expected result/outcome has to
be determined in advance for each test case
• this expected outcome has to be gathered
from an appropriate source: the test oracle
Testing in
Software Development Models
Overview of Test Types
Static Testing
• Review
– Inspection,
walkthrough ...
• Static Analysis
– Control flow analysis
– Data flow analysis
– Metrics
– Coding standards
Control Flow Analysis Example
CERN recommended
static analysis tools
https://security.web.cern.ch/security/recommendations/en/code_tools.shtml
Test design techniques
Equivalent Class Partitioning
Set of Input values
Equivalent class
Combinations of equivalence classes for building test cases:
• All possible combinations of all valid equivalence classes build valid
test cases. Even with a few parameters this can lead to a high number
of test cases. For this reason, often only frequent combinations are
considered
• An invalid input equivalence class should only be combined with valid
input equivalence classes
Boundary Value Analysis: Example
• A function calculates z, the sum of two integer
numbers x and y
• For the input the following ranges are defined:
– x: [0 .. 50]
– y: [20 .. 100]
Parameter
Valid classes
Invalid class
x
0,50
-1,51
y
20,100
19,101
Structures of Decision Tables
Statement Testing
• Statement
coverage
– Every instruction
(statement) is
executed at least
once
• Decision coverage
– Every branch is
executed at least
once
Path Coverage
• Path coverage
– Each path is executed at least once
Error guessing Testing
• Examples of potential errors out practice
– "pathological" situations:
•
•
•
•
–
–
–
–
–
Empty input fields, blanks in strings
Repeated access (many times) to the same data
No more space available, limits
Negative numbers, etc.
Situations that are explicitly forbidden
Situations that "never arise"
Special characters (a, s, €, etc.)
Abuse and misunderstandings by the user
Change from and to daylight saving time, leap year
Exploratory Testing
Continuing learning process
• Become acquainted with the product
• Learn how errors in software arise and which
effect those errors have
• Discover the weaknesses of the software
• Learn how to test the software
• Test the software
• Report the problems
• Create new test cases on the basis of what you
just have learned
Testing Organization Forms
Balance Between
Test Cost and Failure Cost
Early Failure Detection as a Goal
Testing Management Tasks
•
•
•
Test Management
Test Organization
Test Planning and Estimation
–
•
Risk Analysis
Test Progress Monitoring
–
Failure based metrics
•
–
Test case based metrics
–
Test object based metrics
•
•
–
code-coverage, dialogue-coverage, etc.
test effort, on schedule milestones, etc.
Test Control
–
Corrective measures have to be taken:
•
•
•
•
•
•
•
number of blocked tests, number of successful tests, etc.
Test process based metrics
•
•
number of failures found, per release, per class, per failure status,etc.
Requesting additional resources
Updating of the test schedule
Re-assigning priorities
Initiation of additional test cycles
Configuration Management
Incident Management
Norms and Standards
THANKS FOR ATTENDING!