TEST
TEIN
Introduction to Testing
1
Testing - a Definition
13
Test Levels
2
Errors are part of Human Nature
14
Structural Testing
3
Human Error Probability
15
Functional Testing
4
Spectactular Errors
16
Test Objectives
5
The Testing Dilemma
17
Test Processes
6
The Origins of Testing
18
Test Planning
7
Organisational Aspects of Testing
19
Test Techniques
8
Legal Aspects of Testing
20
Test Effort Distribution
9
Economical Aspects of Testing
21
Software Error Distribution
10
Test Costs and Benefits
22
Software Error Detection Probability
11
Test Approaches
23
Testing as a Profession - Minus Points
12
Test Methods
24
Testing as a Profession - Plus Points
TEST
TEIN-1
Testing - a Definition
... Testing implies empirically demonstrating that a given piece
of software fulfills a given set of requirements in a given
environment under certain circumstances. Testing is intended
to both validate and to verify system behavior. To verify is to
confirm that the software matches its specs. To validate is to
confirm that the software functions in a real live environment.
... Testing is only one of several approaches to ensuring the
quality of a software system, approaches which include,
among others, reviews, inspections, audits, static analysis and
simulation. Testing is the empirical approach to ensuring
software quality. These are all regarded as Validation und
Verification (V&V) activities.
TEST
TEIN-2
Errors are part of Human Nature
The human being is unfortunately an imperfect creature, which carries over its own
imperfections into the products it creates. Humans, in their current state of
evolution, are unsuitable for complex software development, no matter in what
language and at which level of abstraction.
Humans say what they do not mean and write what they do not intend to write. The
human communication pattern is imprecise. Even the communication between
mind and mouth, as well as between mind and the extremities, is distorted. Thus
every human is to a certain extent handicapped, some more, others less.
Transmission disorders occur in what the human being perceives in his
environment. He confuses what he thinks he hears or sees with what really is heard
or seen. His sensors are biased by his expectations and prevent him from being
totally objective. (See Kant "Kritik der reinen Vernunft").
It can not be expected from a human being, as a biologically imperfect creature, to
produce software without errors. Only a biologically altered organism, i.e. a
genetically reengineered creature, could be expected to produce error free
software..
Therefore it can never be expected to have error free software. The question is, how
error free the software has to be to fulfill its purpose.
TEST
TEIN-3
Probability of human error in Software
Of 100 lines, that a normal human writes or draws at least
1,5 will be erroneous.
Of those the human will recognize some 50% leaving
0,7 erroneous lines.
In a program or document with 1000 lines that will make for
7 errors.
In a program or document with 10,000 lines that will amount to
70 errors.
The larger and complexer the program is, the more errors there
will be..
As long as humans produce software there will be errors
because to erre is human.
Error detection is the purpose of
Error free software is an illusion!
TEST
TEIN-4
Spectacular Software Errors
Spectacular faults in the past have lead to the
emergence of testing as as an unavoidable activity
within the life-cycle of a software system. Without
testing, there is no way to ascertain the presence or
the absence of such errors.
The Venussonde was lost because of a comma
instead of a point.
DO 1000 I = 1,15
DO 1000 I = 1.15
The rising moon was detected as being an
incoming enemy rocket.
The second Space-Shuttle-Start was interrupted
because of a synchronizsation error in the five
shuttle processors just before takeoff.
The Ariane-5 Spacecraft exploded at takeoff time
because of a register overflow in an untested reused
component.
The Deutsche Bank lost 8 Million German marks
because of a rounding error in the Euro Conversion.
TEST
TEIN-5
The Testing Dilemma
Why have‘nt you
fed the hay to my
horse yet???
First I have to
remove the needles,
otherwise it will choke.
TEST
TEIN-6
The Origins of Testing as a Branch of
Software Engineering
The first technical conference on software testing
occurred at the:
Univ. of North Carolina in July 1972
Other conferences followed in both the U:S.A. and in
Europe. The first major European test conference took
place in London in 1978 Two of the speakers were Ed
Miller and Harry Sneed. Sneed reported on Prüfstand,
the first German module testing tool.
By 1980, test technology had become a recognized
branch of software engineering. The U.S. Ballistic
Missile Defense system was the first to be
systematically tested using automated test tools –
RXTV tool set from General Research Corp.
The first commercial test laboratory was established
by Ed Miller and Harry Sneed to test components of
the German Integrated Transport System (ITS)
in Budapest, Hungary in the spring of 1978
TEST
TEIN-7
Organisational Aspects of Testing
From the viewpoint of the organisation there are three tests
the test by the developer himself,
the test by an independent tester and
the test by the user.
Each Test consists of the following steps:
Organizing the test resources
Selecting the proper test strategy
Planning the test activities
Specifing the test cases
Executing the tests
Evaluating the test results
Testing is a project in its on right, i.e., a special project within the
development, maintenance, migration or integration project.
TEST
TEIN-8
Legal Aspects of Testing
The overall standard for evaluating the quality of
software products is the ISO Standard 9126.
The IEEE Standard 829 prescribes the proper
system test documentation. .
The IEEE Standard 1008 prescribes procedures for
software unit testing.
Responsible standardisation gremiums are:
ISO/EMC - int. Standards Organisation
ANSI - American National Standard institute
IEEE - Int. Electronic & Electrical Engineers
NBS - National Bureau of Standards
ASQF- American Standard Quality Foundation
The ASQF is authorized to examine and to certify
testers. Only certified testing companies can be held
liable for software reliability.
TEST
TEIN-9
Economical Aspects of Testing
No Testing
With Test
Zero Test Costs
1% Erroneous
+ 100% Test Costs
25%
Erroneous
75%
Correct
6% of the Results will
be wrong
24%
corrected
75%
Correct
0,25% of the Results
will remain wrong
TEST
TEIN-10
Test Costs and Benefits
100%
25%
%Error Probability
% of Development Effort
50%
10%
5%
5%
Test effort
1%
0%
TEST
TEIN-11
Test Approaches
Developer Test
Tester Test
Systemtest
User Test
Systemtype
Programtest
Standalone-PC
Trial & Error
Mainframe
Dialoge Test
Batch Test
System Test
Live Test
Embedded
Module Test
Integration Test
Hardware Test
Process Control
Module Test
Integration Test
Process Test
Client/Server
Client Test
Server Test
Frontend Test
BackendTest
Live Test
Internet
Webclient Test
Network Test
Usage Test
-
Acceptance Test
-
TEST
TEIN-12
Test Methods
Test Methods according to the American
National Bureau of Standards Publication for
"Software-Quality Assurance":
Requirement Analysis
Object Integration Test
Audit of the Req. Specs
Component Integration
Test
Design Review
Testplan Review
Code Inspection
Module/Class Test
Component Test
Funktion Test
Black-Box Test
White-Box Test
Interface Test
Load Test
Regression Test
System Test
Acceptance Test
Installation Test
Beta Test
TEST
TEIN-13
Test Levels
Testing requires knowledge of the software test object at some level.
One can acquire that knowledge
 by using the software without knowing how it is constructed
(corresponds
to a trial run, e. g. the test drive of a new
automobile).
This level of testing is termed
Functional Testing = Black-Box Testing.
One can also acquire that knowledge
 by examining the inner structure of the test object (corresponds to
the
inspection of the motor when buying a new automobile).
This level of testing is termed
Structural Testing = White-Box-Testen.
One can also acquire that knowledge
 by discovering how the parts of the system interact with one
another
This level of testing is termed
Integration Testing = Grey-Box-Testen.
TEST
TEIN-14
Structured Testing
Structured Testing is the most wide spread method of WhiteBox Testing. This method is based on the extraction of test paths
and test parameters from the structure of the source code.
There are various intermediate approaches between the pure
Black-Box and the pure White-Box test approach.
There is a fine distinction between debugging and testing although
both activities are similar.
Debugging is directed toward the removal of errors
Testing is directed toward the identification of errors
Both activities are performed by the developer.
TEST
TEIN-15
Functional Testing
Functional Testing (Black-Box-Testing) takes place without any
knowledge of how the software is constructed. It is focused on
the usage of the product.
For functional testing, the proper test data must be
selected.The preparation of the test data must be planned from
the beginning.
Functional Testing is not sufficient for Systems with a complex
internal construction (e. g. self made technical services). Each
subsystem must be tested independently of the others.
Every Test is a test against somethimg, a comparison with
another object. There must be an oracle to determine whether
the behaviour of a software system is correct or not.
Functional testing should be made by an independent tester.
The developer is too biased to recognize his own faults.
TEST
TEIN-16
Test Objectives
Each Test is a test against something. The actual behavior of
a program is compared with the expected behavior. For that,
the expected behavior must be defined.
Program
As a rule, a program can be tested against
the specification,
the user‘s Handbook,
the previous program and
the operational profile.
Oracle
TEST
TEIN-17
Test Processes
A Test consists of a series of overlapping steps
Test Management
Test Planing
Test Design
Test Specification
Test Execution
Test Evaluation
Test Documentation
TEST
TEIN-18
Test Planning
Which data?
Where from?
Input data
Where to?
What is correct?
Test paths
Program
Output data
Faults
With what probability?
Which paths are to be traversed? (cases)
How long may a transaction last? (Time)
What data are required? (Data sets)
When can one stop? (End Criteria)
Testing without specific test objectives is similar to shooting in the dark.
Finding an error is purely coincidental.
Random testing produces random results, which cannot be controlled.
TEST
TEIN-19
Test Techniques
Technique
Manu/Auto.
Stat/Dyn
Struct/Funct
Proving
Walkthroughs
Inspections
Design Reviews & Audits
Simulation
Desk Checking
Peer Review
Executable Specifications
Automated Unit Testing
Load Testing
Error Prediction
Cause & Effect Graphs
Design-based Testing
Coverage directed Testing
Complexity directed Testing
Compiler supported Analysis
Data Flow Analysis
Control Flow Analysis
Symbolic Execution
Instrumentation
Combined Techniques
both
manu
manu
manu
auto
manu
manu
auto
auto
manu
manu
both
manu
auto
auto
auto
auto
auto
auto
auto
auto
static
dynamic
static
static
dynamic
both
both
dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
both
both
static
static
static
static
dynamic
both
both
both
both
both
functional
structural
structural
functional
functional
functional
functional
functional
functional
structural
structural
structural
structural
structural
structural
structural
both
From the NBS Spec. Publ. 500-75 Validation, Verification and Testing of Computer
Software, U.S. Dept. of Commerce, 1981
TEST
TEIN-20
Test Effort Distribution
20%
System
design
40%
Component Design
Module Design
60%
Coding
20%
Module Test
Integration Test
20%
System
test
Test Effort
40%
TEST
TEIN-21
Software Error Distribution
ErrorType
Frequency
12%
14%
24%
28%
8%
9%
11%
13%
7%
8%
9%
10%
16%
18%
12%
COMP
LOGIC
RESULT
INIT
DEF
INTER
I-O
Other
TRW-Experience
GRC-Experience
According to "Error Detection Using Path Testing And Static Analysis" in Computer Vol. 12. Nr. 8.
TEST
TEIN-22
Error Detection Probability
according to a Study of Howden
Error Detection Rate
Error Types
Test Method
Average
COMP LOGIC RESULT INIT DEF INTER I-O
Code Inspection
25%
20%
25% 25%
Static Analysis
25%
20%
50%
Path Analysis
25%
40%
Result Comparison
25%
50%
Load Test
40%
18%
14%
25%
25%
20%
12%
50%
Integration Test
Function Test
LOAD
50%
25%
16%
25% 25%
9%
50% 50%
21%
25% 50%
9%
TEST
TEIN-23
Testing as a Profession - Minus Points
Testing is a dirty job, one has to clean up the mess of others.
Testing is not creative and offers no possibilty of self fullfillment.
Testing is a dependent activity and encompasses many
detailed tasks.
Testing is not a highly respected job, testers are looked down
upon.
Testing tools are complicated and difficult to use.
Testing methods are often project specific and not portable.
Testing is a destructive activity - one is out to destroy the
system.
Testing can become terribly boring!
TEST
TEIN-25
Testing as a Profession - Plus Points
Testing is a challenge, a difficult task which requires
creativity and lots of self discipline.
Testing is important, because managers and users need to
gain confidence in the product. Developers should be
thankful that someone finds there errors before the users
do.
Testing is an activity, which demands both domain
knowledge and technical savay.
Testing can be interesting, since the technology is still
highly underdeveloped, thus providing the opportunity for
innovation. There is a tremendous need for better tools.
Testing is unavoidable - someone has to do it.
Testing offers an employment opportunity for old
developers, who are no longer able to keep pace with the
youngsters.