Testing strategy of DILIGENT
4D Soft Ltd.
Introduction
Testing of complex systems is very difficult and
necessitates a detailed plan of the testing methods,
strategy and adequacy criteria to be applied.
New techniques require new testing methods as well.
Real study of lack of test planning:
Web application required of serving 10 000 user in
parallel. After installation only 3 users could access in
the same time.
One of the most critical effect on testing is the stability of
the specification and the design of application
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
2
Summary of testing process
Comprehensive testing plan - the fundamental
guidance for the whole testing process
Contains:
starting conditions of present plan – accepted
specification documents
subject of the testing
testing organisation
Selecting methods and tools for different testing phases
defines the constraints
risk management
elaboration of quality assurance plan
defines the exit criterion
Start date: Jan. 2005 – end date: Apr. 2005
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
3
Elaboration of testing plan – Subject of
the testing
Defining the subject of testing
Diligent’s services
Interface testing with gLite
(all the relevant interfaces with Diligent,
we do not test gLite)
Interface testing with other systems
Specification documents define the subjects more
precisely
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
4
Elaboration of testing plan -Test
management
Setting up the testing organisation, test management
István Forgács: project manager
Responsibilities: general project leader, professional
guidance, deciding on strategic testing issues
Éva Takács
Responsibilities: technical coordinator, relationship
with Diligent and other organisations involved in the
project, relationship with development (tutorial on
unit testing, etc for development)
Klára Tauszig
Responsibilities: administrative competencies
Test team
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
5
Elaboration of testing strategy - Test
methodology for the different testing phase
Test planning – we planned to use category-partition
method for all phases of testing. This eases test planning
and gives a general view of testing. Maybe the original
method should be extended to involve performance, load
and stress, etc. testing
Unit testing – we propose test driver and coverage tools
to be applied. All-decision test data adequacy criterion
seems to be enough and a minimum of 90% coverage of
each unit, but the goal is 100%. Black box and white box
testing together.
Integration and component testing – black box testing.
Test automation would be useful.
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
6
Test methodology for the different
testing phase - continued
System testing – it covers lots of testing activities on the
entire system. Functional testing is essential. Load and
performance testing is critical. Traditional testing tools
probably cannot be applied. New methods and may be
home tools are necessary. DILIGENT and EGEE testing
effort can be united.
Acceptance testing – the only criterion is that different
team and methods are to be applied.
Regression testing. Also very critical.
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
7
Elaboration of testing strategy - Using of
testing tools
Large projects such as DILIGENT necessitate of using
testing tools
The selection of testing tools is a difficult task since there
are numerous tools but most of them is not applicable at
all.
4D Soft gives advice for the selection of unit testing tools,
and methodology for unit testing. The main criteria are:
Easy-to-use test driver and stub functions
Test code and data should be separated
New test data are possible to be inserted by the tester
without any recompilation of the system
Simple coverage analyser promoting all-decision
criterion
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
8
Elaboration of testing strategy - Selecting of
testing tools
Only some tools should be selected for evaluation
Only downloadable tools are to be selected
Existing experience is taken into consideration
Goals and requirements should be clearly identified and
quantified
Requirements are usability, learning curve, price, etc.
Real-life applications instead of toy programs should be
applied for tool evaluation
Multi-functional tools have an advantage over more
different tools
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
9
Elaboration of testing strategy - Test
automation
Test automation means test repetition and using
testing tools.
Test automation adds an additional cost of 25%,
therefore we apply testing tools where intensive
regression testing is necessary. This is almost
always the case, but we should investigate the
components of the system one by one carefully.
If test automation is necessary, but no applicable
tools on the market, home tools should be
implemented.
Test planning can be automated only in part.
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
10
Elaboration of testing strategy - Selection of
testing criteria
indicates when the testing can be finished
Unit testing
all the program statements should run at least once – too
week
all the program branches should run at least once appropriate
all the conditions should run at least once – too strong
Component and system testing
The total number of test cases is an appropriate criterion
(between LOC/10 and LOC/20)
MTBF
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
11
Elaboration of testing strategy - Testing
environment
Currently there are several open questions:
Which are the services\functions that can be tested
on one computer?
Is it possible to develop a minimal environment at
4D Soft site
A more distributed testing environment for Diligent
Nr of sites (minimum (3?), average, maximum
Nr of computers at each site?
Is there or will be any Grid testing environment for
DILIGENT?
Test environment should be specified in common
with the design team
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
12
Elaboration of testing strategy - Test
documents
It would be reasonable to use the same tool for bug
reporting than used by the EGEE team
Using gLite we may find failures in it, since correct
program does not exist
Detailed test documents are generated by the testing
tools.
Final test evaluation documents are made manually
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
13
Elaboration of testing strategy – Walk through
of documents
Methods for error discovery
Walk-through: (T1.1-Functional and Architectural
description)
Tests the documents integrity, understandability,
clarity and self-consistence
Usability for the test cases design
What kind of test cases can be designed having
these documents?
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
14
Elaboration of testing strategy - Risk
analysis
We should assume the main risks specific to
DILIGENT
defining risk matrix
Contingency planning for the critical risks
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
15
Elaboration of testing strategy - Risk
analysis
Generally, the main risks of testing are as follow:
risk of the project – changing the specification and
the design on the fly, the code does not conform
with the original plan, etc.
The budget of testing is insufficient – almost
always the case
Late delivery
risk of the resources (knowledge, human factors)
risk of technology (new, without experience)
human related risks (missing co-operation)
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
16
Test planning
Test plans defines the test specification and then the
test cases for different testing phases (unit,
integration, system, etc.)
Method used for designing test plans:
Category partitioning
Basic function: functionality that corresponds to a
use-case
Category: basic entity of a function that is
expressed in the related design document
Choice: different states of a category
Advantage: test case set is reduced from “universe”
, no important test cases are left out
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
17
Designing of the test plans – using
specification
Test case design method used: category partitioning
Required documents that serve as input for designing test
cases are the following:
Functional test cases – functional specification (T1.1)
Integration test cases – architectural specification (T1.2)
---------------------------------------------------------------Unit test cases – based detailed design documents
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
18
Tutorial for developers
The focus is on unit testing (methods, tools)
Guidelines on:
Defining test specification
Generating black box test cases from test
specification
Running black box test cases
Checking coverage by the testing criterion
Selecting white box test data
Selecting unit testing tools
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
19
Component and integration test
Performed on a single computer
All the unit tested components will be tested using test
cases defined by category-partitioning method
Component’s integration will be tested together with the
component test
We intend to apply EBIT technique
Components are tested in a predefined order
according to the data flow
Testing tool could be applied
Some test code should be inserted into the code
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
20
DILIGENT testing emphasizing Grid
Traditional functional testing requires complex
functionality to be validated involving distributed
environment
Load, stress and performance testing require new
methods
We try to extend category-partition method to plan test
cases for all testing phases above. Since the method is
very general we believe that it is possible.
We also develop an in house tool the automate categorypartition planning
We try free load testing tools to see whether existing
tools are applicable
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
21
Regression testing
Traditional regression testing executes all the test cases
and investigates those cases for which the original and
the executed results are different
Selective tools are not available on the market
We should plan regression for different test phases. Only
tests that requires more than two iterations is well worth
automating
If possible we apply static regression testing, new tools
are to be developed at 4D Soft
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
22
Static and selective regression
testing
Static regression testing determines all the (output)
variables that are influenced by a given code modification
The programmer select which variables are intentionally
affected, the other influences are due to program defect,
and should be fixed
Only tests that influences the required variables are
executed, thus few test cases should be executed
Now the goal is that the original and the modified result be
different.
If this is not the case the test case should be transformed
to be effective.
DILIGENT-EGEE Interaction - Content and Metadata Management
and Testing meeting CERN, 16th December 2004
23