Doc. Identifier:
Note
81920514
Date: 31/07/2017
Subject:
TESTING, VALIDATION AND RELEASE PROCESS
Author(s):
Andre Formica, Bob Jones, David Foster, Gabriel Zaquine, Zdenek Sekera
Distribution:
WP Managers
CONTENTS
CONTEXT: ........................................................................................................................................................... 1
1.
INTRODUCTION ........................................................................................................................................ 1
2.
RELEASE PROCESS DESCRIPTION ..................................................................................................... 2
2.1. SOFTWARE DEVELOPMENT ........................................................................................................................... 3
2.2. INTEGRATION AND SOFTWARE BUILD ........................................................................................................... 4
2.3. SOFTWARE TESTING............................................................................................................................... 5
2.4. BETA RELEASE ......................................................................................................................................... 5
2.5. PRODUCTION RELEASE.......................................................................................................................... 6
3.
TEST AND VALIDATION PROCESS ...................................................................................................... 6
3.1 WP DEVELOPMENT & UNIT TESTS ................................................................................................................ 6
3.2 INTEGRATION AND BUILD VALIDATION ......................................................................................................... 7
3.3 GRID VALIDATION ......................................................................................................................................... 7
3.4 APPLICATION VALIDATION ............................................................................................................................ 8
3.5 SYSTEM VALIDATION ..................................................................................................................................... 8
3.6 VERSION RELEASE ......................................................................................................................................... 8
4.
TEST ENVIRONMENT ............................................................................................................................. 9
5.
CERTIFICATION TESTBED .................................................................................................................... 9
6.
DETAILED PROCESS OF TESTING AND TESTBED ENVIRONMENTS ...................................... 11
CONTEXT:
This document is the result of extensive discussions between EDG and LCG projects with the aim of
combining resources in order to achieve the high quality grid middleware for the benefit of scientific
applications.
It represents our current thinking. The details may evolve over time as experience with the day-to-day
application of the approach is gained.
1. INTRODUCTION
This document describes the release process as a cycle of well defined every day activities
(development-integrating-testing), application testing and the system certification leading to the
next software release. It introduces the role of a release manager as an essential element of the
release process and establishes a need for a new testbed called the certification testbed. Most
importantly it defines the role of the TSTG (the testing group) as well as the ATG (Application
Testing Group) as the driving force in establishing the high quality system release. The quality
IST-2000-25182
PUBLIC
1 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017
group (QAG) representative makes sure this process will be followed inside each WP and input
and output from each steps are conforming to the required quality level.
Quality needs to be present at every point in the software creation and release process. It cannot be
added post facto. Therefore the process needs to be defined and the quality issues then dealt with
at every point.
The process for release 1.2 will not be interrupted nor changed, the version 1.2 will be released as
planned, but the new process presented below will now be executed on the next release of the
EDG software (assumed to be 1.4). This will enable the roles, responsibilities and test plans to be
finalised. The process will be reviewed once it has been applied to release 1.4.
Section 2 describes the software development and release process, and section 3 the testing and
validation process embedded within this. In section 4 we overview the test environment
comprising 3 testbeds for development, certification and production. The certification testbed will
be new and is to be created by LCG. Section 5 gives detail of this new testbed. Finally section 6
provides a tabular summary of all the stages in the proposed testing and validation process.
2. RELEASE PROCESS DESCRIPTION
The understanding of the release process for the GRID software is a necessary prerequisite to
understanding the testing requirements specified later in this paper.

The release date for the next release is determined by the project management at the end of
the previous release cycle. Every effort has to be spent to meet this date. If, however, the
technical obstacles appear that make this impossible, the new decision will be taken by the
project management on the suggestion of the release manager.

An incremental approach making use of as many automated steps as possible will be adopted.

The release manager is the person leading the release process. He is responsible for bringing
the new version of the software to the high quality expected by the project management.
The release manager has the duty and the authority to:
o
Consult with the various groups within the project, namely ITeam, Applications
groups, WP6 testing group, etc., to determine the release details and propose them to
the project management.
o
Determine priorities for the release process, including testing, bug fixing, feature
development (within the current release cycle) to ensure the planned release date is
met.
o
Determine the new issues to be addressed by the user support group as a result of
planned new features or changed behaviour of existing software as a result of possible
software changes.
IST-2000-25182
PUBLIC
2 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017
o
Stimulate the development of the new tests or test suites for testing the newly
developed features or improving the existing tests to better cover the changing
software.
o
Determine the most suitable date for the beta test period (typically 2-3 weeks before
the release date, see below) and make sure the special beta test period rules are
enforced.
o
Organize the release of the software (such as to make sure all files/CD’s are properly
tested, all documentation is included)
Make sure the release date is met.
o
If for any reason this appears not feasible, he has to warn the project management so
that an appropriate decision can be taken in the interests of the project. This may
include changing of the release date or dropping some features/fixes in favour of
maintaining the original date.
o

Verification that bugs entered into bugzilla bug tracking system are addressed by
software changes included in the release.
The problems/bugs identified at every stage of the release cycle are immediately fed back to
the development team for corrections using the standard GRID bug-tracking mechanism, the
bugzilla. Thus bugzilla is the mechanism for tracking changes (all changes, including new
features) and the most appropriate communication medium among all release cycle
participants.
The GRID software release process consists of the following activities that are completely
intertwined and alive throughout the cycle, till the release date:
2.1. SOFTWARE DEVELOPMENT
Consists of adding new features, patches, fixes (commonly: mods) to the main software tree.
Important note: the documentation including man pages, user guides, manuals, configuration,
and other documentation is an integral part of the SW development.

Mods are added daily into the main software tree(s).
This allows for much more rigorous control since problems are discovered early in the
cycle.

The added mod can fix only the problem described in its corresponding bugzilla report
entry.
This allows for much better understanding of exactly what has been changed in the
software.

The feature mod can be added only if agreed beforehand by the release manager
(who in turn will discuss the inclusion with the project management as part of the
decision process to set up the release date.
This prevents mushrooming of various features that were never meant to be part of a
release.
IST-2000-25182
PUBLIC
3 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017

It is required that every mod to be added to the tree has passed the peer review for the
quality of the code, completeness (including documentation and the test case, if
appropriate) and its adherence to the submission rules above.
This often discovers problems with the code before it ever gets in the tree.
The peer review simply means that a developer finds a colleague willing to review his
code. It is a moral obligation of every developer to submit to this requirement.

Every mod has to have a corresponding update in the bugzilla report (to be also
verified during the peer review).

Every mod has to be unit tested by the developer.
It is totally unacceptable to submit a fix that has not been tested, for obvious reasons.
The software development and its unit testing is the responsibility of each WP, unit testing
is performed on dedicated development machines or, possibly, on development testbeds if
the nature of the problem requires it.
NOTE: Anyone should be able to run the unit tests. This allows others to verify that the code
satisfies the defined unit tests.
2.2. INTEGRATION AND SOFTWARE BUILD
The complete GRID software is built nightly to test the integrity of all modifications added the
previous day. This ensures:

the software can be built and packaged without obvious errors

the software passes the automated unit tests

The easy identification of the offending mod if the build fails since there are only a
limited number of modifications that were added since the last build.

Easy regression to the previous version when the build fails catastrophically.

Much easier identification of the run-time failures that may be discovered during the
testing phase by the testing group (see below), because the changes are known,
documented and controlled.

The SW build should include building of all standard configurations agreed upon by
the EDG management taking into account the needs of HEP experiments via the Grid
Deployment Board (GDB) and representatives from WP9 and WP10, as part of the
daily testing.

The build testing is based on a set of tests. The tests are to be collected or developed
and maintained by the ITeam. They will be run automatically (i.e. not by hand). The
ITeam must check the results of the build testing and take the appropriate action to
ensure the build remains clean.

The daily SW builds also ensure the up-to-date completeness and consistency of the
software tree.
The software build testing is the responsibility of the ITeam group.
IST-2000-25182
PUBLIC
4 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017
2.3. SOFTWARE TESTING

The software built over night is next tested against a series of tests to guarantee the
backward compatibility or to expose a new behavior.

The backward compatibility tests verify the basic functionality of the software.

It is necessary to have the testing activities as automated as possible, however, it is
recognized, that the automated suites of tests has to be completed by certain tests
performed manually (e.g. downing a node/site or breaking the internet connection
during the test run).

Every problem should be immediately reported back to developers through the
standard communication mechanism (the bugzilla)

The test suites will be modified with time as the new aspects of testing appear
necessary (e.g. a new feature has been added or a better way of testing has been
found).

This testing is an everyday activity; it is the single most important activity leading
to the solid, stable, usable software.
The TSTG group shares with the LCG certification group the overall responsibility for
this testing and the overall maintenance of tests. In other words, as new features are added
to the software, TSTG should add appropriate tests to their test suite.
2.4. BETA RELEASE
Eventually the software will be ready for the beta release. From now on the current software
tree (which becomes now the “release tree”) is considered frozen and a new tree (the
“development tree” is forked as a base for the next release.
The certification group, together with the ATG (Application Testing Group) and the TSTG
(testing group) now runs extensively the real-world applications which should stress the GRID
software and expose the remaining functional as well as performance issues.
From now on the development cycle as described in pts.2.1-2.3 above remains the same,
except that checking modifications into the release tree becomes far more restricted and
only the critical mods, fixing specific bugs or performance problems may be checked in.
The critical mods that are included into the release tree are also included at the same time into
the development tree (to ensure the desired synchronization between both trees). It is the
entire responsibility of every author (developer) of a mod to take care of this synchronization.
Indeed, it is not generally possible to do this automatically by some kind of a script since the
same module could have been previously modified by somebody else as a result of new
feature development (remember, this is the development tree) so the developer has to be aware
of all possible implications.
All other non-critical mods together with possibly new features go into the development tree.
The developer is responsible for resolving all possible conflicts.
The release manager is the authority making decisions about the criticality of all mods.
IST-2000-25182
PUBLIC
5 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017
The forking of a development tree marks the start of the release cycle for the next version. But
the main development focus remains on fixing the remaining critical problems in the release
tree.
The beta-release period lasts normally about 2-3 weeks but can be adjusted as required by
circumstances. It ends by the new production release of the GRID software.
2.5. PRODUCTION RELEASE
The production release is the final release of the version destined for the general public usage.
The release tree is archived. This is necessary for the maintenance reasons. The binary as well
as source versions are prepared, together with all user and system documentation and made
ready for the release.
The certification group is responsible for this part of the release process.
3. TEST AND VALIDATION PROCESS
The overall process is summarised below and takes into account the following main points:
 The inputs and outputs for each stage.
 The formal exit criteria, the tests performed.
 The team responsible and team leader.
 The expected duration.
 The hardware/testbed platform.
 The communication mechanisms with other teams.
 The actions when problems are found.
 The signoff procedure when everything is ok.
3.1 WP DEVELOPMENT & UNIT TESTS
Initial unit testing is performed entirely within the WPs themselves. Typically software
components are produced by each WP together with a test plan and testing software as
appropriate. All the testing procedures for a particular unit are documented. These unit tests
must be completed successfully before handing over the software for certification testing.
Such unit tests will either be performed within the WP dedicated testbed resources, or on the
development testbed. In this latter case WPs would have scheduled access to the development
testbed as required.
The WP performs their tests as necessary every day before leaving their software with the
current day modifications in a repository to be picked up by the integration group that will
follow with the complete night builds, see below.
IST-2000-25182
PUBLIC
6 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017
It is also mandatory that the WP supplies a developer guide for each component as well as the
test suite to the testing group which will repeat them as part of the broader testing (see 2.3
above and chapters below).
3.2 INTEGRATION AND BUILD VALIDATION
In this step individual components are assembled on the development testbed and integrated
into a complete candidate release. Once assembled the candidate release is then tested by the
integration test suite, assembled and maintained by the ITeam.
Any problems found will be fed back to WP via their ITeam representatives and bugzilla. The
WP will then deliver a corrected version, again unit-tested. This process is repeated every day
throughout the release development cycle.
The ITeam builds all possible standard configurations (with different compilers, libraries,
etc…) for all architectures as defined by EDG management together with the Grid
Deployment Board (GDB) and tests them with appropriate tests made either by them or
supplied by the WPs as a result of their unit testing described above.
Note: GDB may request support for certain platforms but this requires negotiation since each
platforms implies resources are required.
A test subset can be automated (e.g. via scripts etc.) and integrated as part of the nightly build
procedure as non regression tests.
All failures are reported via bugzilla to each WP for correction.
3.3 GRID VALIDATION
The testing activity of TSTG should be oriented to insure correct installation and configuration
of the new software release, and then to functional and destructive testing as described in the
TSTG plan. Service specific tests should be integrated in the test framework. This framework
can be used also for the basic functionality tests of ITeam.
The Grid validation consists of testing:

basic grid functionality

grid services

security

information

resource brokering

data catalogue, data replication

connectivity
IST-2000-25182
PUBLIC
7 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017

configurability

error recovery
This everyday activity is currently performed by the testing group (TSTG) on the on the
certification testbed and perhaps exceptionally on the development testbed. The hope is that
a good fraction of the tests are automated and can form the basis of the acceptance tests for a
tagged release.
The testing group also has a number of testbed sites that are not integrated in any official
testbed. They are used by the TSTG, in particular, for installation and destructive tests.
3.4 APPLICATION VALIDATION
Functionality tests performed by the ATG using application groups’ applications and test
scenarios. These tests correspond to the validation tests currently being organized by EIP
(informally known as the Loose Cannons). Representatives from WP9 & WP10 must also be
included in this task and have significant representation.
These tests are performed by the ATG on the certification testbed.
3.5 SYSTEM VALIDATION
Tests with regional centers, as agreed by the GDB on complete grid system functionality in
geographically distributed centers. These include regression, load and performance tests.
This activity will be performed by the LCG certification team together with the ATG often
using ATG test suites to verify the grid functionality in a normal user environment.
The system validation is performed on the certification testbed.
3.6 VERSION RELEASE
Scheduled introduction into production. Transfer of documentation and training to User and
operations support, final training and publication of end-user documentation.
This activity will be performed by the LCG certification team.
IST-2000-25182
PUBLIC
8 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017
4. TEST ENVIRONMENT
The test procedure specified in this document requires the creation of the "new" test environment (as
opposed to “dev” and “pro”). This new environment is called “certification testbed”. This is to be a
stable environment on which the EIP will work and which becomes the candidate certified software
environment
The following schema describes the testbed environment and tests activities:

Development Machines: dedicated work package-specific resources for the unit testing of
code. May be provided in conjunction with WP6 or various sites.

Development Testbed: existing EDG development testbed. Support during office hours.

Certification Testbed: (New). To be created in conjunction with LCG. Support during office
hours.

Application/Production Testbed: based on existing EDG testbed but needs extra resources
and sites as well as a round the clock support.
Testing Activities
“WP specific”
testbeds
“Development”
testbed
WPs add unit
tested code to
CVS repository
Run nightly build
& auto. tests
Install on cert. Testbed
& run back. compat.
tests
WPs
Candidate public
release
for use by apps.
TSTG
no
Any
Errors?
Fix problems
“Application”
testbed
“Certification”
testbed
Any
Errors?
yes
yes
ITeam
Fix problems
no
Candidate beta Release
For testing by apps.
yes
Office hours
Apps
ATG
Any
Errors?
no
24x7
B.Jones– July 2002 - n° 2
5. CERTIFICATION TESTBED
IST-2000-25182
PUBLIC
9 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017
The new testbed, called the certification testbed will be established together with the GDB (Grid
Deployment Board) and the CERN Fabric activity for the following purpose:

To allow for the broad GRID system software testing under different hardware and
software configurations (matching the definition of “standard configurations” used above).

To allow for testing of interoperability of different versions of GRID software as required
by GRID users.

To allow for emulation at CERN of geographically dispersed GRID sites.
This is required because it allows for simplified testing of various communication issues
inside CERN before going for more involved tests (see next point). Thus the
geographically dispersed sites (next point) will be used for testing of exclusive features
impossible to emulate locally. This will significantly decrease the required availability
time for the external collaborators.

To include several (3-5) geographically dispersed sites that will participate in GRID
validation
The participating sites should agree to be ready for testing on a days notice and should
actively participate.
The establishing of the certification testbed is of utmost importance for the success of the GRID
project.
IST-2000-25182
PUBLIC
10 / 11
Doc. Identifier:
Note
81920514
Date: 31/07/2017
6. DETAILED PROCESS OF TESTING AND TESTBED ENVIRONMENTS
.
Stage
Input
Output and formal
exit criteria
Tests Performed
Team
Leader
Typical
Duration
Test
Environment
Commun
ication
Problem
Actions
1.
WP mod
WP mod tested
Unit Test Plan
WP
WP Quality
Assurance
representati
ve
Every
day
WP
Development
machines
Meetings
/Mail
Bugzilla
Unit Tests
2.
Integration
and
Build
Validation
3.
Grid
Validation
4.
Application
Validation
5.
System
Validation
6.
Version
release
Development
Testbed
All WP unit
tested
release
candidates
Integrated
Software Build
Build,
Deployment
and Basic Tests
ITEAM
ITEAM
Leader
(Charles)
Every
day
Development
Testbed
Meetings
/Mail
Bugzilla
Integrated
Software Build
Build
validated
against test plan
Integration Test
Plan
(Installation,
Grid Tests)
TSTG
TSTG
leader/
Every
day
Certification
Testbed
Meetings
/Mail
Bugzilla
Application
Test Plan
ATG
Till next
release
date
Certification
Testbed/
Meetings
/Mail
Bugzilla
Build validated
against test plan
Build passed App
Quality Measures
LCG
Certificat
ion
Zdenek
Sekera
Frank
Harris/
Ingo
Augustin
Application/
Production
Testbed
Build
passed
App
Quality
Measures
Build Ready for
Production
System
Plan
Test
LCG
Certificat
ion
Zdenek
Sekera
Till next
release
Certification
Testbed
Meetings
/Mail
Bugzilla
Build Ready for
Production
Certified Build
Release
Test
Plan
(Documentation
, packaging etc)
LCG
Certificat
ion
Zdenek
Sekera
2 weeks
Application/
Meetings
/Mail
Bugzilla
Production
Testbed
Production
Environment
(LCG)
IST-2000-25182
PUBLIC
11 / 11
Meetings/M
l