P
Presentation
Paper
Bio
R
E
S
E
N
T
A
T
I
O
T7
Return to Main Menu
Thursday, Dec 7, 2000
Efficient Mobile Network
Acceptance Testing
Jorg Paul
International Conference On
Software Testing, Analysis & Review
DEC 4-8, 2000
• COPENHAGEN, DENMARK
N
telecommunications
mmannesmann
mobilfunk
Efficient Mobile Network Acceptance Testing
Jörg Paul
Thomas Gärtner
Mannesmann Mobilfunk GmbH
Am Seestern 1
D-40547 Düsseldorf
Germany
[email protected]
Ericsson GmbH
Fritz-Vomfeldestr. 26
D-40547 Düsseldorf
Germany
[email protected]
Dr. Dieter Kreuer
Ericsson Eurolab Deutschland
GmbH
Ericsson-Allee 1
D-52134 Herzogenrath
Germany
[email protected]
1
telecommunications
mmannesmann
mobilfunk
Contents
Contents
1. Mobile Network Acceptance Testing
2. The NEXT/AIMS Tool Platform
3. The GSM R8 Project
4. Results
5. Conclusions
2
telecommunications
mmannesmann
mobilfunk
Basic
BasicStructure
Structure of
ofaaGSM
GSMNetwork
Network
SSS
PSTN
HLR
GMSC
MSC
VLR
OSS
AuC
MSC
EIR
BSS
BSC
BTS
MS
BTS
BSC
BTS
MS
3
BTS
OMC
telecommunications
mmannesmann
mobilfunk
1.
1. Mobile
Mobile Network
Network Acceptance
Acceptance Testing
Testing
4
t
lo
u
ol
/R
A
FS
U
ni
tT
es
tin
g
Fe
at
ur
e
Te
st
in
Fu
g
nc
tio
n
Te
st
in
g
Sy
st
em
Te
st
Ty
pe
A
cc
ep
ta
nc
e
Te
st
• different releases for SSS, BSS and OSS at least
shipped once a year by the vendor (Ericsson)
• Test phases for the new software:
telecommunications
mmannesmann
mobilfunk
Type
Type Acceptance
Acceptance Testing
Testing
• any new HW or SW release tested before integration
into the commercial network („no harm to the network“)
• joint test activity between vendor and operator
• testing done in the actual live network configuration
• SUT not tested standalone, but as a part of the full GSM
network
• emphasis on end-to-end functional testing
• both black-box and white-box testing performed
5
telecommunications
mmannesmann
mobilfunk
Growth
Growthof
ofNetwork
Network Complexity
Complexity
SMSC
VRU
VRU
SGSN
IN SCP
CTI
W-MSC
VMS
ACD
WAP
MSC
PBX
VoIP
AuC
Billing
EIR
VLR
PSTN
HLR
BSS
6
telecommunications
mmannesmann
mobilfunk
Conflict
Conflict Situation
Situation for
for Testing
Testing
Quality
Testing
Costs
Time
7
telecommunications
mmannesmann
mobilfunk
Network
Network Complexity
Complexity and
and Test
Test Coverage
Coverage
40
Complexity
35
30
Test Coverage
with Test Automation
25
TAcc duration
20
Test Coverage
15
without Test Automation
10
1990
1992
1994
1996
8
1998
2000
telecommunications
mmannesmann
mobilfunk
Automated
Automated Testing:
Testing: Our
Our goals
goals
• resource savings due to automation of regression
testing
• speed-up of test execution
• more efficient test plant utilisation (e.g. at nights and
weekends)
• higher reproducibility of automated testing
• enable testers to concentrate on the new features of a
release
9
telecommunications
mmannesmann
mobilfunk
3.
3. The
The NEXT/AIMS
NEXT/AIMS Tool
Tool Platform
Platform
• original target application: Statistical Usage Testing
• end-to-end functional testing of mobile networks
• both black-box and white-box testing possible
• suitable platform for automation of Type Acceptance
testing performed by Ericsson and Mannesmann
Mobilfunk
10
telecommunications
mmannesmann
mobilfunk
Principle
Principle of
of NEXT/AIMS
NEXT/AIMS
NEXT
Commands
AIMS
Server
Actions
AIMS
Hardware
SUT
Calls
Results
Signals
MS
SUT Status
SRS
SUT Status
TENUX
Commands
Commands
11
telecommunications
mmannesmann
mobilfunk
Integrated
Integrated Automated
Automated Testing
Testing Platform
Platform for
for GSM
GSM
12
telecommunications
mmannesmann
mobilfunk
4.
4. The
The GSM
GSM R8
R8 Project
Project
GSM R8
CSS
PSS
BSS
13
OSS
telecommunications
mmannesmann
mobilfunk
Automated
Automated Test
Test Objects:
Objects:
SMS
3%
Other
Tests
14%
Call Tests
50%
Data / Fax
33%
14
telecommunications
mmannesmann
mobilfunk
Automated Test Objects (continued):
•
•
•
•
•
•
•
•
Call Handling
Interworking with BSS and GSM R7
Supplementary Services
Charging
Statistics and Traffic Measurements
Data and Fax Tests
Short Message Service
APZ Regression Tests
15
telecommunications
mmannesmann
mobilfunk
5.
5. Results
Results
Grade of Automation: Test Objects
24%
24%
52%
fully automated TOs
fully manual TOs
mixed TOs
16
telecommunications
mmannesmann
mobilfunk
Test Case Execution Times
0:55:00
5:04:22
5:04:22
5:04:22
0:50:00
0:45:00
0:40:00
Minimum
0:35:00
Mean
0:30:00
Maximum
0:25:00
0:20:00
0:15:00
0:10:00
0:05:00
An
no
un
ce
AP
m
ts
Z
R
eg
re
ss
C
n
al
lB
ar
rin
Ba
gs
si
c
C
al
lH
nd
C
ha
rg
in
g
Fa
x/
D
Fr
at
au
a
d
Pr
ev
In
nt
te
n
rw
rk
R
In
7/
trw
R
8
M
SC
St
/B
at
SC
is
t.
Su
bs
tm
.
O
ve
ra
ll
0:00:00
17
Test Object
telecommunications
mmannesmann
mobilfunk
Number of Repetitions
100%
90%
89.0%
80%
70%
60%
50%
40%
30%
20%
8,0%
10%
2,7%
0,0%
0,3%
0%
1
2
3
4
18
5
0,0%
6
19
23:00-24:00
22:00-23:00
21:00-22:00
20:00-21:00
19:00-20:00
18:00-19:00
17:00-18:00
16:00-17:00
15:00-16:00
14:00-15:00
13:00-14:00
12:00-13:00
11:00-12:00
10:00-11:00
09:00-10:00
08:00-09:00
07:00-08:00
06:00-07:00
05:00-06:00
04:00-05:00
03:00-04:00
02:00-03:00
01:00-02:00
00:00-01:00
mmannesmann
mobilfunk
telecommunications
Execution Time Distribution
14%
12%
10%
8%
6%
4%
2%
0%
telecommunications
mmannesmann
mobilfunk
Implementation Progress
120%
Coded TCs
Verified TCs
100%
80%
60%
40%
20%
21/00
19/00
17/00
15/00
13/00
11/00
09/00
07/00
05/00
03/00
01/00
51/99
49/99
47/99
45/99
43/99
0%
Calendar Week
20
telecommunications
mmannesmann
mobilfunk
Grade of Automation: Test Cases
20% ne w
Te s ts (m anual)
41% m anual
Re g re s s io n
Te s ts
80% m ix e d
Re g re s s io n
Te s ts
59%
auto m ate d
Re g re s s io n
Te s ts
21
telecommunications
mmannesmann
mobilfunk
Relative Costs Comparison
100%
automated Test
manual Test
80%
66%
60%
40%
34%
16%
20%
3%
0%
STP hrs/TC
mhrs/TC
TAcc duration/TC
22
total R8 TAcc
duration
telecommunications
mmannesmann
mobilfunk
6.
: What
6. Conclusions
Conclusions:
What we
we reached
reached ...
...
• Length of Type Acceptance reduced by 30%
! shorter time-to-market
• more efficient test plant utilisation
! automated testing 24 hours a day
• resource savings during execution phase
! test cases normally run unattended by the tester
23
telecommunications
mmannesmann
mobilfunk
What
What we
we learned
learned ...
...
• stability of the test configuration is critical
• manual testing in parallel to automated testing should be
avoided
• no significant speed-up of test case execution
(SUT was the limiting factor)
24
Increasing the Efficiency of Mobile Network
Acceptance Testing by Test Automation
Jörg Paul, Dr. Dieter Kreuer, Thomas Gärtner
Joerg Paul graduated in physics at the Aachen University of Technology in 1995. At university
he mainly worked in the fields of theoretical elementary particle physics and computational
physics. In 1996 he joined the German GSM network operator Mannesmann Mobilfunk. Since
then he has been working in several software testing and test automation projects. He is now
project leader for the introduction of automated testing into the joint type acceptance project
GSM R8 of Mannesmann Mobilfunk and Ericsson.
Email address: [email protected]
1
Increasing the Efficiency of Mobile Network Acceptance Testing by
Test Automation
Jörg Paul, Dr. Dieter Kreuer, Thomas Gärtner
Mannesmann Mobilfunk GmbH
Am Seesstern 1
40547 Düsseldorf
Germany
Tel.: +49 211 533 3799
Fax: +49 211 533 3804
[email protected]
Ericsson Eurolab Deutschland
GmbH
Ericsson-Allee 1
52134 Herzogenrath
Germany
Tel.: +49 2407 575 447
Fax.: +49 2407 575 651
[email protected]
Ericsson GmbH
Fritz-Vomfelde Str. 26
40547 Düsseldorf
Germany
Tel.: +49 211 534 2118
Fax.: +49 211 534 2356
[email protected]
Abstract:
In the last decade, operators and manufacturers of mobile networks had to deal with a rapid growth of
network complexity, which is still going on. Accordingly, the number of services offered to the customer
and the number of network elements and interfaces incorporated in the network strongly increase from year
to year, while the release cycles decrease. A lot of testing is performed to find the software defects of the
new releases before commercial service. Both manufacturers and network operators are facing the problem
how to perform an ever-increasing amount of test cases in a decreasing time interval. In order to solve this
problem, the test automation tool platform NEXT/AIMS has been developed by Ericsson in close cooperation with Mannesmann Mobilfunk. NEXT/AIMS was designed to automate end-to-end functional
testing that is performed in system or acceptance test. Already in the acceptance test of the current Ericsson
software release GSM R8, which is the very first application of the tool, a grade of test automation of 47%
has been reached. By means of the automated testing approach a considerable amount of human resources
can be saved in the test execution phase, which could therefore be shortened by 30%. In this paper we are
going to describe the NEXT/AIMS tool platform and the results drawn from its first practical application in
the GSM R8 type acceptance project.
1. Introduction
Since the first operators of digital mobile networks have started service, a rapid technical development
has taken place. One example for the evolution of digital mobile networks is the Global System for Mobile
Communications (GSM) (see [1] and [2]. GSM is a standard for wireless mobile networks which offers
various kinds communication services to the fully mobile customer, like telephony, data / fax transmission,
short message service (SMS) or Intelligent Network (IN) services. Currently, packet switching technologies
like GPRS (General Packet Radio Service) are being introduced into the GSM networks [3]. The coming
third generation mobile telephony standard UMTS (Universal Mobile Telephony System) will be designed
to fully support mobile multimedia applications like video conferencing [4].
The introduction of many new services leads to a more and more complex network. Not only do the
single network elements become more complex due to the increasing number of supported features, but also
the number of different network elements needed to provide a broad range of services is growing steadily.
The rapidly growing network complexity leads to a real challenge for testers. In spite of the growing
number of features, the release cycles of hardware and software for the network even tend to become
shorter due to the strong demands of the very dynamic mobile communications market which promotes
competition between the operators. It is very important to find remaining software defects before the
product is released to commercial service. Therefore, both the manufacturers and the operators of mobile
networks have to perform a number of test cases which is nearly exponentially growing in a time frame
2
which is constant or even decreasing. If the testing efficiency cannot be increased significantly, a loss of test
coverage and also test quality cannot be avoided. It will no longer be possible to test neither all the new
functions of the system under test (SUT) nor the old functions (regression tests) with sufficient depth under
the existing time constraints.
In addition to the various tests performed by the vendors (e.g. system test), Mannesmann Mobilfunk as
an operator of one of the largest GSM networks world-wide, performs a thorough acceptance test of any
new component to be introduced into the operative network. This test activity is called Type Acceptance
and will be performed for each new or changed component in advance of the integration into the live
network. The goal of the Type Acceptance is to assure that any new hardware or software component will
not damage the commercial network. Therefore, it is crucial that Type Acceptance testing is performed in a
test environment which ideally should be identical to the real commercial network. Otherwise, there is no
way to find faults related to special parameter settings, external interfaces or details of the configuration
used in the field. Those faults usually are not visible in an isolated laboratory configuration.
In general, a Type Acceptance project is a joint project between Mannesmann Mobilfunk and the
corresponding vendor of the SUT. As Ericsson is one of the main vendors of GSM network elements used
by Mannesmann Mobilfunk, there are several large Type Acceptance projects run by both companies every
year. Therefore, both Mannesmann Mobilfunk and Ericsson are facing the problem of an exploding number
of test cases, which can no longer be compensated by additional human resources. As a consequence, both
companies jointly have been working on methods to increase the efficiency of both system and acceptance
testing significantly since 1996.
The introduction of test automation turned out to be a very promising measure to optimise testing in a
way that test coverage and quality can be kept sufficiently high. A large number of test cases performed in a
Type Acceptance are end-to-end functional tests, which verify the system functionality from the customer’s
point of view. Therefore, one of the key difficulties of the project was to develop at test system that
simulates the mobile end-user and is able to execute high-level functional tests with real GSM terminal
equipment (black-box testing). But at the same time, a large number of test cases require internal
information about the status of the SUT (white-box testing). In order to develop such a test system, it was
necessary to combine the experience and know-how of both the network operator and the manufacturer.
In 1999 both companies decided to implement automated testing in the Type Acceptance of the Ericsson
switching software release CME201 R8 CSS. The goal of the project was to automate as many of the
existing regression test cases as possible. The NEXT/AIMS tool platform used for the project is described
in section 2 of this paper. In section 3 we are going to describe the test scope for the GSM R8 test
automation project, and section 4 summarizes the results we obtained.
2. The NEXT/Aims platform
The automated test case execution during the type acceptance was based on a tool platform developed by
Ericsson in close co-operation with Mannesmann Mobilfunk.
To understand the architecture and implementation of the tool platform, one must consider, that its actual
target application is automated black box testing focussing on Statistical Usage Testing (StUT) [7]...[10].
The intent of StUT is to describe real user behaviour via a stochastic traffic model. The tool platform can
then run a simulation of the model and thereby automatically generate a large number of test cases that are
immediately executed on the system test plant. Apart from checking the correctness of the execution, quality
figures such as Mean Time to Failure (MTTF) or other Quality of Service Figures can be determined.
3
Performing a performance test on a telecom system requires putting the exchanges under a considerable
traffic load. Load generators are the only means of generating such a high load. Additionally, a Type
Acceptance test platform requires control of the terminal equipment to access all the functions available to
the users. This means controlling actual mobiles, modems, etc. Furthermore, it must be possible to enter
operator commands, because some tests require activation/deactivation of services, or checking of internal
switch status information. Such commands are usually entered into the exchange via a dedicated interface in
a particular man-machine language (MML). Finally, some global execution control must be in place to
synchronise the actions of the test tools on the various interfaces. For instance, a test case might set up a
configuration via the MML interface, initiate a time-variant background traffic load via the load generator,
and at the load peaks perform calls with real mobiles.
Our test platform is shown in figure 1. On top, the global execution is controlled by a software tool
called NEXT (Network Executor), which has access to a repository of test cases (TC’s). NEXT controls a
load generator MGTS and another tool AIMS (Air interface and mobile subscriber simulator, consisting of
hardware and software components) to access real terminal equipment. Fax and data terminals are not
directly controlled via AIMS, yet (this is planned for the upcoming release), but since nearly 40% of the
automated R8 Type Acceptance test cases consist of fax/data test cases, a preliminary solution was
implemented involving a fax/data server. This server has several modems connected (mobile, PSTN, ISDN)
and runs both a fax and a data server program that use the modems to perform all kinds of fax and data
transmissions.
Ericsson telecom exchanges feature an MML interface provided by a unit called the “IOG” (Input/Output Group). NEXT has an interface to the IOG, too, allowing to send MML commands and to receive
printouts. Finally, NEXT can control a tool dubbed PCM-LIG (PCM Link Interrupt Generator) which can
perform controlled interrupts of the PCM links between exchanges, thus simulating hardware failures.
Figure 1: Integrated automated testing platform for GSM
The system under test is a small GSM network consisting of the nodes MSC (Mobile Switching Centre),
BSC (Base Station Controller) and BTS (Base Transceiver Station). NEXT is directly connected to the
MSC and BSC nodes via their IOG MML interfaces. The commercially available load generator MGTS is
4
used to put load onto the signalling links of the MSC exchange (A-G). The AIMS tool interfaces mobiles or
fixed telephones (BL = Both way Line circuit, simulating PSTN phones, or phones on a Private Access
Branch Exchange or PABX). PABX’s have a 2 Mbit/s PCM connection to the Primary Rate Access
Interface, PRA, of the MSC. Using coaxial cables, mobile radio signals are fed to the base stations via a
Semiconductor Roaming Simulator, SRS, simulating the air interface between mobiles and base stations.
Lacking a commercial solution, AIMS was created by our group to control the keypad of a mobile (or
any other telephone), to read out its display (if supported by the telephone), and to monitor power, ringing,
and in-band signalling tones. The latter tones are decoded and recognised, as are test tones sent by AIMS to
continuously monitor voice connections (voice checks). Furthermore, AIMS can send and recognise DTMF
tones, which can e.g. be recorded as a substitute for unrecognisable voice announcements. Finally, AIMS
controls the attenuation in the SRS. The SRS has 4 inputs for mobile HF signals and 4 outputs for base
station HF cables. To connect more mobiles, it is either possible to combine the HF signals of several
mobiles on a single input, or to use several SRS units connected in cascade. AIMS is presented in more
detail in [5].
The NEXT tool, which controls the test execution, has also been created by our project group. NEXT is
based on Petri nets enhanced with a command language. Blocks of commands, executed upon occurrence of
a Petri net transition, can be included in each transition, e.g. to control external tools. Token values can be
used as parameters for the commands in the command blocks. Token values are also used to import signals
from the connected test tools (e.g. a token 4 showing up in a place named ringing might signify, that mobile
4 is ringing). NEXT allows designing, executing, and monitoring test models. Many more details on NEXT
and examples of its Petri nets can be found in [5] and [6].
For testers who merely wish to execute existing test programs, we created the NEXT Graphical User
Interface for Testers (GUI-T). Within GUI-T, test models are merely files. GUI-T allows executing a test
suite, i.e. a series of sequential or parallel batches of tests. GUI-T also manages test resources (such as
mobiles, IOG-connections etc.), so that test models can pick from the pool of available resources whatever
they need during execution. This allows a flexible resource allocation during parallel execution.
Prior to execution, test cases to be used with the platform are designed and tested by Ericsson testers.
This is actually the most time consuming task, involving the creation of reusable modules, splitting existing
test instructions into small, reusable pieces, and running the test case under different fault situations. This
effort is usually higher than the preparation of test cases for manual execution, but pays off with repeated
execution during the Type Acceptance and other test phases.
A test case with NEXT consists of an initialisation step, an execution step, and a restoration step. The
initialisation step serves to connecting to the test tools and claiming the resources. In many test cases,
certain parameters in the system under test are also checked and modified here. In the execution step, test
commands are sent to the system under test and the received signals are compared with expected results.
Any deviation is evaluated as a fault and the test case directly jumps to the restoration step. Here, where the
test case eventually ends up even with no detected faults, all the modifications of the initialisation step are
undone and the test resources are released again.
When Ericsson testers have finished designing a set of test cases, test experts from Mannesmann
Mobilfunk are invited to validate the execution of the test cases and to approve their correctness – or to
disapprove it and to demand modifications. As watching the test execution for hundreds of test cases is very
time consuming, the more viable way of exchanging and validating execution logs was often adopted,
especially when test cases occurred in many variations of a common basic test case.
A test case approved by Mannesmann Mobilfunk is not to be touched anymore, which can always be
verified by employing a configuration management (CM) tool. Approved test cases are stored in a suitable
repository. Prior to a test campaign in the TAcc, GUI-T is used to combine test cases from this repository
into a test suite for successive execution. Test cases that do not disturb and are not disturbed by the
execution of other test cases may be specified to run in parallel, if the available test resources allow so.
Otherwise, test case execution is sequential. Test resources are assigned to test cases executed in parallel on
a first come-first serve basis with no optimisation, which we found to be no problem, as very few test cases
actually allow parallel testing activities at all.
5
For each test case, two log files are automatically created during execution: the Automated Test Report
(ATR) which reports on the execution steps, expected and observed results including resource and timing
information. The test case designers need to program the output, which goes into the ATR. The other log is
the Automated Test Log (ATL), where all the messages exchanged with connected tools are stored in the
order of occurrence, with no influence of the tester. This log is sometimes useful for fault tracing or proving
the correctness of a test case. During and after the execution, both files can be opened and browsed from
within GUI-T. The ATR contains the test result (passed or failed) which can also be viewed on-line in GUIT during the test suite execution.
3. Test scope of the GSM R8 CSS project
The GSM standard is still being enhanced every year. Most of the vendors of switching systems and base
station systems built so-called "releases" at least once a year. These releases incorporate all new and
enhanced features and usually consist of software and hardware components. The current release developed
by Ericsson is named GSM R8. GSM R8 contains various updates for the different GSM subsystems, the
Switching Subsystem (SSS), the Base Station Subsystem (BSS) and the Operation and Maintenance
Subsystem (OSS). Due to the introduction of GPRS, the SSS is in fact split up into the classical Circuit
Switching Subsystem (CSS) and the Packet Switching Subsystem (PSS), which has a similar structure as a
router-based IP network.
After GSM R8 had undergone the Ericsson software quality assurance process, it entered the Type
Acceptance process. The aim of the Type Acceptance is to find the remaining software defects in the
software and to verify that the new software release to be installed will not damage or disturb the network
operated by Mannesmann Mobilfunk. In a network which is as complex as a GSM network, many of the
defects found in the testing process are in some way related to the network configuration. In order to find
the errors that would most likely occur in operation, any new software release has to be tested in the live
network configuration, otherwise it might be impossible to find these defects. That means that the same
software configuration has to be established, the same market corrections and adaptations have to be loaded
and the same exchange properties and parameter settings have to implemented as in the field. Furthermore,
an acceptance test for any GSM network element (e.g. an MSC) is never performed standalone. All the
other network elements of the full GSM chain have to be present, too, and many of the Type Acceptance
Tests are end-to-end functional tests involving real terminal equipment.
Figure 2 shows a sketch of an example test configuration of the GSM R8 Type Acceptance. Although the
SUT is the MSC, the test configuration is in fact a small GSM network, which contains three MSC nodes
and three Base Station Controllers (BSC). The MSC is part of the CSS and fulfils no packet switching
functionality yet. In order to trace the mobile customer who is roaming through the network, the HLR
(Home Location Register) and the VLR (Visitor Location Register) are needed. The HLR is a database that
contains the basic subscription information of the customer, e.g. his service profile, and permanently is
updated with the subscriber’s status (attach/detach, present location etc.). The VLR database stores the
information of all subscribers that are roaming in the specific MSC area the VLR is responsible for.
Normally, MSC and VLR are realized on the same physical network node. The BSC is responsible for the
maintenance of the base transceiver stations (BTS), locating the mobile stations, rate adaptation and data
compression of voice calls (TRC, transcoder rate adaptor centre). It also handles the signalling and payload
traffic between the mobile stations and the switching system.
During the execution phase of a Type Acceptance project, the test configuration needs to be changed
quite often, because certain test objects in general need a very special environment to be tested. However, in
a test environment that changes frequently, configuration faults are likely to occur, which sometimes causes
problems that are extremely hard to detect. In order to find these configuration faults in advance of testing,
which is always a tedious and time-consuming job if done manually, we designed a special test script (Petri
net) to check the set-up of the network automatically. The script can be run easily before any test session
without additional manual effort. By means of the Configuration Check, a lot of problems could be detected
and solved even before testing actually started.
6
Figure 2: Sketch of an example test configuration of the R8 project.
For an operator of a mobile network it is extremely important, that all the features of the network that are
directly visible to the customer are free of errors. Therefore, a major part of the tests performed during Type
Acceptance are end-to-end functional tests with an emphasis on black box testing techniques. In order to
find exactly the errors the customer would encounter, the tester often takes the customer's point of view and
verifies the services offered to the user under all relevant initial conditions. Every time the software
changes, these services have to be tested again in order to assure that no defects have entered the software
(regression testing). Starting with the first GSM releases the pool of regression tests has grown
tremendously. Already in the last years it has turned out to be impossible to perform all the regression tests
and the tests for the new functionality in the given time frame. With the NEXT/AIMS tool platform we have
managed to automate 47% of the regression tests performed for the CSS part of the GSM R8 Type
Acceptance.
The customer accesses the network through the mobile station (MS). One crucial result of former
acceptance tests was that it is not sufficient to simulate the mobile phone. Although the interface between
the mobile station and the GSM network is a standardised interface, the communication between the mobile
and the network is a non-trivial matter and caused a lot of troubles in the past. In contrast to methods
applied in protocol conformance testing (see [11]), end-to-end functional testing is always performed with
commercially available mobile phones. Therefore, NEXT/AIMS accesses the SUT via the same interfaces
that are used by the customers of the network operator.
In addition to this, many test cases require the ability of NEXT to interact with the SUT via the
command line interface of the SUT. The behaviour of the network nodes is examined by sending commands
and analysing printouts, like it is done in the operation and maintenance centres of a network operator. In
addition to the black box tests, there are also tests checking single functions of the SUT with respect to
internal status information, which can be accessed via the command line interface to the SUT. These tests
are implemented as white box tests.
The tests, which are performed, cover the whole range of user interaction and operation and maintenance
procedures. This is necessary to check, if operation and maintenance influences or disturbs traffic handling.
7
For GSM R8 we have concentrated on the regression tests of former releases, because these tests have to be
repeated in every Type Acceptance project. For GSM R8, which is the first application of our tool, we have
restricted ourselves to the CSS part only. But in the BSS area there is also a huge potential for test
automation, which might even be higher than in the CSS, due to the various BTS types and combinations
that have to be tested with nearly identical test scenarios. In the framework of GSM R9 we are going to
implement automated testing in the BSS area, too.
3.1 Test Objects Selected for Automation
At the moment there are three basic services provided to the GSM customer, which are telephony,
circuit-switched data & fax and the Short Message Service (SMS). The SMS enables the user to send
simple text messages with a maximal length of 160 characters from the mobile phone to another mobile or
the Internet. The basic GSM data service has a maximum bandwidth of 9.6 kbit/s, which has been extended
recently by the introduction of High-Speed Circuit Switched Data (HSCSD, see [12]). In addition to these
basic services GSM offers, there is a large number of so-called Supplementary Services, which are defined
in the GSM recommendations [13]. Services like call forwarding or call barring are just two examples of the
Supplementary Services, which can be de-/activated, configured and utilised by either the customer or the
network operator (e.g. for barring of outgoing calls for a specific subscriber). The corresponding test object
Supplementary Services covers most of the services, which are available for the customers. The test tool
performs the typical actions of GSM users and checks the results by monitoring the phones and the
behaviour of the network elements (nodes).
All the possible interactions of the customer with these three basic GSM services combined with the
Supplementary Services form a large and very important group of test cases in every Type Acceptance. In
total 69% of all test cases, which were performed during the acceptance test for GSM R8 CSS were related
to basic telephony, the short message service or data/fax calls.
As the NEXT/AIMS tool platform was especially designed to perform these types of tests without human
interaction, we have managed to automate a large fraction of them. In total, 47% of all test cases which were
executed in the acceptance test for GSM R8 CSS were performed fully automatically, and 92% of these test
cases contained either voice, fax or data calls or short message transmission.
Especially the test object mobile fax and data transmission has proven to be an ideal candidate for
automated testing, because there are various modes to operate the data service all of which have to be
tested. This results in a set of test cases that have to be repeated several times with different parameter
settings of either the terminal equipment or the network. The test tool checks all provided fax and data
services, e.g. all data rates, connection elements and compression modes used for all possible call types
(mobile originating, mobile terminating and mobile- mobile calls). There are analogue (PSTN) and digital
(ISDN) modems and terminal adapters used to test the whole range of equipment, which is used by the
subscribers. In total, 33% of the test cases we performed automatically were either data or fax test cases.
The short message service (SMS) has become more and more popular among the customers of mobile
operators. The test object Short Message Service tries to cover all traffic scenarios related to the point-to1
point SMS . Examples for basic scenarios are sending messages from one mobile phone to another one,
which are received immediately, sending multiple messages, and receiving messages after attaching the
mobile station. Short messages are transmitted in a signalling channel of the Common Channel Signalling
System No. 7 (see [14]). For this reason, testing the SMS sometimes requires to analyse the signalling
information, which is exchanged between the network and the mobile station. At the moment this is not
possible with the NEXT platform, which provides no protocol analyser functionality yet. Therefore, the
protocol information was checked manually by using commercially available protocol analysers. Compared
to the fax and data test object, the set of SMS test cases is quite small (less than 5% of all automated test
cases).
1
The point-to-multipoint SMS transmission was not included in our test automation program yet. This
scenario will be implemented with the next release.
8
Although, the SMS traffic volume in the network has increased tremendously during the last three years,
the telephony service is still the most important service offered by today’s mobile network operators. For
this reason, all the different aspects of voice services are tested very carefully by Mannesmann Mobilfunk
for any upcoming software release. One of the most simple test objects is Basic Call Handling, which tests
the fundamental GSM call scenarios for different subscribers who are roaming through the test network.
For a GSM network operator it is not sufficient to verify that all the services work very well. It is
essential that also the charging functionality works properly. In GSM, the charging is done by the operator’s
billing platform, which processes the call records written by the MSC. These call records contain the
essential information for each call set-up in a specific MSC area (regardless whether it is voice, data or fax),
for example the A- and B-party numbers, the starting time and the call duration. The records are stored on a
hard disk of the Input/Output Group (IOG) of the MSC and are periodically transferred to the billing
system. In the Charging test object, all relevant call types are performed fully automatically by the tool
platform and the corresponding call records are saved. As NEXT neither has the ability to check if the
generated call records are correct, nor has an interface to the operator’s billing platform, the call records
still have to be analysed manually or transferred to the billing system where the post-processing is made.
In order to be able to plan the capacity of the network, the operator needs detailed information about the
traffic volume carried by each network node and the amount of signalling messages processed by the
signalling network. The Ericsson switch provides this information by a large number of counters, which
measure the individual events relevant for the later analysis. The test object Statistics and Traffic
Measurements checks whether these counters are correct and can be relied upon. For this purpose, a large
set of call scenarios is performed fully automatically by NEXT/AIMS. The corresponding traffic
measurements are stored by the MSC and transferred for post-processing. As NEXT/AIMS is not able to do
the post-processing itself, the data still has to be checked by the testers.
After a new release was tested successfully, it has to be rolled out in the whole network. Usually, the
BSC is not updated in the same night as the MSC. Therefore it is necessary to check whether the new MSC
release is compatible to the old BSC release and vice versa. When upgrading a network, it is neither
possible nor achievable to update the whole network in one step. In consequence, there is a period where
both release versions are used. The Interworking test object proves that all services can interwork between
all network elements.
As described above, NEXT provides a command line interface to the system under test (e.g. the MSC),
the so-called TENUX interface. Via TENUX it is possible to execute maintenance commands on the SUT
or to obtain result printouts, which enables us to perform also test objects which are beyond the scope of
2
simple call tests, for example the APZ Regression Tests. The APZ is the core of an Ericsson exchange. It
consists of a pair of central processors, at least one pair of support processors and up to 1024 regional
processors controlling up to 16 control modules. The test proves central functions like software upgrades,
hardware replacements and alarm reception. Since all parts of an AXE (the hardware platform all Ericsson
GSM nodes are based on) are redundant, nearly all maintenance works should be possible without traffic
disturbances.
Although we have reached a grade of automation of 57% for the regression tests that were performed in
the acceptance test of the release GSM R8 CSS, there are still a lot of test cases that could not be automated
with current release of the tool platform, for example the signalling tests. The signals which are exchanged
by the different GSM nodes are rather complex and have to be decoded using a protocol analyser. The most
common protocol analysers on the market have no open interface to send commands and receive result
printouts. In consequence, the test automation of these tests had to be postponed.
Other tests probably will never be automated at all. These are usually tests which require some kind of
external interaction which cannot easily be triggered by a test automaton, e.g. manipulating the hardware of
the SUT in order to check if the hardware fault detection and recovery works properly. In principle, one
could imagine that in the future even these tests could be automated, too, but it is questionable if the benefit
gained by this would justify the high implementation effort.
2
APZ is not an acronym.
9
4. Results from the R8 CSS Project
4.1 Preparation Phase
The GSM R8 activities related to automated testing started in spring 1999, long before even defining the
set of test cases to be executed, because adequate time for programming the test cases was needed. The
baseline for selecting test cases for automation was the previous release, GSM R7, so the test descriptions
from this release were investigated for potential test cases for the GSM R8 automated TAcc. This meant of
course, that new features of GSM R8 would not be subject to automated testing, but this restriction was in
accordance with Mannesmann Mobilfunk’s and Ericsson’s plans to only apply automated testing in
regression tests. On the one hand, this means that test descriptions are already available as instructions for
the designers of automated test cases. On the other hand, human testers got the opportunity to get
acquainted with the new features by manually testing them, which will help in programming automated test
cases for the subsequent release.
A team of EDD3 and MMO4 test experts thus walked through all the test cases of the former GSM R7
TAcc and identified a large number of test cases that could be automated with the existing NEXT/Aims test
automation platform. In relative numbers, based on the final selection of test cases used in the R8 TAcc
(including the new features), 52% of the test objects had no automation potential at all. 24% could be even
completely automated, with a remaining 24% of test objects with a mixed population of automated and
manual tests. In terms of test cases, the R8 TAcc had a 20% fraction of new test cases (that were excluded
from test automation by edict) and an 80% fraction of regression tests. From the regression test cases, 59%
could be automated, while 41% had to be performed manually. In terms of the total number of test cases,
this amounts to 47% test automation vs. 53% manual testing. This limit was set mainly by the number of
features supported by the test platform but also the nature of some test cases as mentioned above.
The programming activities for the automated tests began in October 1999. Initially, the progress was
not directly visible as the test case designers at EDD first built a set of modules that would be used later by
the individual test cases. Therefore, the test case development started very slowly. The result, however, was
a super-generic test case implemented with NEXT that was turned into an individual test case by just
feeding it with the right parameters, which vastly accelerated the implementation of test cases. It was stated
by the designers that coding (not including testing!) such a test case was in the order of magnitude of 15-30
minutes (although we have taken no measurements on this). However, coding was the least time consuming
activity during the preparation phase. Test descriptions had to be read and understood (some needed to be
completely rewritten, as they were too cryptic). Test code needed to be verified itself in the test plant, which
usually caused a significant overhead for configuring the test plant and the automated test platform
properly. Disturbance by parallel testing activities was a major problem. Sometimes, manual testers
reloaded the switch software or re-plugged network connections while an automated test suite was running,
which of course led to a total failure. Adding to the problems was the replacement of some of the switches
with new hardware components that were not all received and installed by the original schedule. Proper
resource scheduling and communication between the different test teams can definitely be improved in
future automated TAcc projects.
But also the test platform developed at another Ericsson subsidiary, EED5, caused some setbacks.
Parsing the MML-printouts of the switches proved to be difficult, as the thousands of possible printouts
were originally meant for human testers rather than automated analysis. Thus, quite often a newly
encountered printout required a modification of the parser. Finally, the new fax/data platform was not ready
on time and had a few bugs that were not detected at EED due to the limited capability of the system test
plant at EDD with respect to fax/data features.
Figure 3 shows the progress in coding and testing according to the weekly reports by the test case design
team. As stated above, the design of test cases started with a delay due to the common modules being
3
Ericsson GmbH, Düsseldorf, Germany
Mannesmann Mobilfunk GmbH, Düsseldorf, Germany
5
Ericsson Eurolab Deutschland GmbH, Herzogenrath, Germany
4
10
developed
first.
Progress stagnated
120%
around the turn of
Coded TCs
the year due to the
100%
Verified TCs
holiday
season.
The
speed
of
80%
coding then picked
up rapidly with a
60%
very sharp increase
in week 8. This
40%
was due to the
fax/data test cases,
20%
which
0%
encompassed
roughly a third of
all the test cases,
Calendar Week
and which were
only variations of 4
Figure 3: Progress during test case development
basic test cases
with different parameters (transmission rates etc.), so that they could easily be obtained by copy-pasting
from the initial test cases with a few minor modifications. In the end, the more difficult test cases were
implemented, and the speed of coding slowed down again. There was even a reduction in the number of test
cases (with the 100% mark being based on the final set), as a few test cases were taken out of the R8 TAcc
(features not supported).
21/00
19/00
17/00
15/00
13/00
11/00
09/00
07/00
05/00
03/00
01/00
51/99
49/99
47/99
45/99
43/99
Implementation Progress
It was also found that some test cases that were already coded had duplicates in other in test objects, so
that they could be removed. Figure 4.1 also shows the progress in verifying the TCs, which was initially
hampered by the above mentioned problems with test plant scheduling. The graph also shows a small drop
in the number of test cases in week 16, as MMO demanded a change in the way the fax/data test cases were
verified, so that the affected test cases needed to be re-tested. The reports ended in week 21, short of the
verification reaching 100%, as a few test cases could not be run successfully and were taken out of the
TAcc. It was later found that this was in fact a problem of the test plant configuration rather than the test
platform.
Not plotted is the validation time, which was spent both by MMO and EDD employees. Validation
means that the MMO-experts are demonstrated the correct behaviour of the test cases. Due to the large
number of test cases and the limited availability of system test plant (STP) time as well as of the MMO
experts, most of the test cases were validated by EDD handing logfiles to MMO.
Not taking into account the initial creation of modules, which was a one-time activity, the average effort
to implement, verify, and validate one test case was close to 7:50 hours or roughly one working day. This
includes the whole of the daytime activities that were done during the test case development. It also includes
the move from GSM R7 to GSM R8, which required modifications of some test cases and re-testing of all
those that were not directly implemented on GSM R8. As mentioned, the coding work only comprised less
than half an hour of this time. No matter what language is used for implementing the test cases, a significant
overhead can be expected in practice for similar projects, considering the first application of automated
testing and the related problems.
4.2 TAcc Phase
Then came the busy phase of the TAcc itself. In theory, one could create a huge test suite of all the
automated test cases and run it on the system under test. Reality was quite different, though. During the
TAcc period, there were more than 50 executions of test suites that varied in length from a single test case
to a whole test object. The main reason was that the system was reconfigured many times during the type
11
acceptance and it was never possible to run but a few test objects on one particular configuration.
Configuration changes could not be avoided, on the other hand, as the test plant resources were scarce and
so it was not possible to set up a generic configuration that would suite all the test objects equally well.
Execution Time Distribution
14%
12%
10%
8%
6%
4%
2%
22:00-23:00
20:00-21:00
18:00-19:00
16:00-17:00
14:00-15:00
12:00-13:00
10:00-11:00
08:00-09:00
06:00-07:00
04:00-05:00
02:00-03:00
00:00-01:00
0%
Figure 4: Distribution of TC Execution over the day
4.21Testing Efficiency
It is quite interesting to consider the hours, during which the automated tests were run. Figure 4 plots the
relative number of test cases that were started during each of the daytime hours. While there is a clear peak
centred in the early afternoon, as would be expected for office activities, the remainder of the time was used
approximately evenly for running the test cases. Thus, automated testing is clearly a 24h activity. Tripling
the available time for testing by using the test resource around the clock is the major advantage of test
automation.
4.22 Speed of Execution
An
no
u
AP nc
em
Z
R
eg ts
re
C
s
al
l B sn
Ba ar
rin
si
gs
c
C
al
lH
nd
C
ha
rg
in
g
F
Fr ax/D
au
d ata
Pr
In
e
te
rw vn
tn
rk
In
trw R7
/R
M
SC 8
St
at
/B
is
S
t.
Su C
bs
tm
.
O
ve
ra
ll
How long do test cases run? Figure5 provides an idea. For most of the automated test objects, execution
times are plotted showing the fastest, the average, and the longest execution time. The overall values are
shown at the outer right. The
execution times vary in a wide
Test Case Execution Times
range, and contrary to the
5:04:22
5:04:22
5:04:22
0:55:00
expectations of a layman, they
Minimum
0:50:00
are in the order of minutes to
0:45:00
Mean
hours rather than seconds.
0:40:00
Maximum
The limiting factor here is the
0:35:00
system under test, not the
0:30:00
execution speed of the test
0:25:00
platform. Some Charging test
0:20:00
cases require less than a
0:15:00
minute to be executed (make a
0:10:00
call and check the charging
0:05:00
information), while a certain
0:00:00
Interworking test runs for
more than 5 hours, as it shall
test whether a call can be
performed with such a long
Test Object
duration. The average test
case takes in the order of 15
Figure 5: Execution times vs. Test Objects
12
minutes. A human tester is not much slower, here, but first, the automated testing runs without any human
attendance at all, while manual testing usually involves at least one or two Ericsson testers and an MMO
auditor. Secondly, as we have seen above, the automated test may run during nighttime hours and over the
weekend. The MMO auditor only needs to review the logs, and he will only have a closer look at the logs of
test cases that end with the attribute “FAILED”.
4.23 Quality of Automated Testing
It is very difficult to compare the quality of the automated and manual test in terms of the detected faults.
On one hand, automated testing is more sensitive to faults, as it pays attention even to small deviations e.g.
of the pulse duration of the busy tone. On the other hand, the automated test cases were based on manual
tests from the preceding TAcc, so the potential to find errors was the same. Or not quite the same, as only
regression tests were automated, that is, test cases for features that were already used in the last software
release so that most of bugs should have been fixed meanwhile. The result was as follows: from all the
detected unspecified behaviours, automated testing found 34%. As we have seen, the fraction of automated
test cases was 47% of all the test cases. One might have expected a similar fraction for the detected
unspecified behaviours, but as manual testing was more focussed on the new, error-prone features, this
expectation is not met. The only conclusion that can be drawn from these figures is that automated testing
does in fact find faults, but whether it is better or worse at finding them compared to manual testing cannot
be decided on the basis of the available data.
4.24 Cost Comparison
If the execution speed of automated tests is not that high, have there been any savings? Figure 6 answers
this with a clear yes. The leftmost three bars indicate the relative durations of an automated test case
compared to a manual one as measured in test plant usage (STP hours), manhours, and TAcc duration
hours. The rightmost bar compares the actual duration of the automated and the manual TAcc testing parts.
These figures need some more elucidation. The STP hours are a plain comparison on a per test case
basis, how long the execution of an automated test case is compared to the manual execution. To obtain the
manual figure, which is normalised to 100% here, the number of manually executed TAcc test cases was
divided by the number of office hours. The figure for the automated testing is based on the average length
of a test run according to the test logs. It is hard to compare the figures on an equal basis as the time for
configuring the test plant and putting together the test suites was not measured here. But the trend is clear:
the manual test plant usage is in the same order of magnitude as the automated test plant usage. The system
under test determines the speed of
Relative Costs Comparison
execution, not the
tester
or
test
100%
machine.
80%
automated Test
The
manhours
per
test
case
were
manual Test
60%
obtained by multiplying the above
34%
figure by the mean
40%
number of persons
16%
involved. In the case
20%
of manual testing,
3%
these are at least
0%
two,
one
from
STP hrs/TC
m hrs/TC
TAcc
total R8 TAcc
MMO and one from
duration/TC
duration
Ericsson. In the case
Figure 6: Costs of automated vs. manual testing
of
automated
testing, there only
needs to be one person involved in putting together the test suite and starting it. It was estimated that a tester
66%
13
only needs to be attending the tests for 1/10 of the time. Therefore, the relative amount of man-time for
automated testing is vastly smaller than for manual testing.
What do the testers do during the execution? Mostly, they are at home and possibly sleeping. Many of
the automated tests were performed outside the usual office-hours. This is shown in the third bar. The
automated testing figure just represents the average duration of an automated test case, while the manual
100% bar is based on the actual duration of the TAcc, including weekends and none-office-hours. As
manual testing is limited to office hours, the time efficiency is roughly 4 times lower than for automated
testing, and so the 66% from the STP hour bar become 16% when the actual TAcc duration is considered.
This was based on a per test case basis. The rightmost bar shows the actual relation between the manual
R8 SSS TAcc, based on the number of TAcc days, and the automated one, based on the sum of all the
periods during which test cases were run (including repetitions and periods of less than two hours between
test executions to account for preparation times). The numbers of executed test cases were roughly equal, so
the bars should have the same length. However, due to the more efficient resource usage of automated
testing, there is a threefold increase in speed compared to manual testing. And this was achieved with a very
small team of testers, compared to a much larger team of testers involved in manual testing.
How about the effort spent on designing the test cases? We have no exact figures on the amount of hours
spent for preparing the manual tests, but roughly, the automation of the tests required about the same
amount of hours or at most twice as much as preparing the manual tests. However, one should not base a
cost comparison on these figures, as this was the first set-up of a huge database of automated tests. This
initial effort occurs only once for automated testing. After that, only the new test cases from the former
TAcc which enter the regression test pool need to be automated. We have seen that this fraction is only
about 20% of a TAcc with a potential for automation of about 50%. Thus, only 10% of the new test cases
will need to be automated for the next phase, as opposed to 47% that were automated within this project.
Hence, the effort for the next phase will definitely be smaller with automated testing than with manual
testing. More gain is expected, when test cases from earlier phases (system test etc.) can be taken advantage
of, and when the test case database is not only employed during TAcc, but also for testing of correction
releases of the GSM software.
4.3 Additional Benefits
There are a number of benefits of automated testing that cannot be easily measured, but which should
nevertheless not be forgotten. One of them is the reproducibility of automated test cases. In our
investigations at Ericsson, we have found that some faults only occur under certain timing conditions. A
manual tester may by chance find such a fault, but when he tries to verify, whether he just hit the wrong key
on the mobile or whether he really encountered a fault, he may not be able to reproduce the same behaviour
due to a slightly different timing. With automated testing, the execution is timed very precisely and hence
time critical faults are reproduced with much better accuracy.
Furthermore, automated testing does most of the boring tester tasks. Writing reports and executing test
cases that mostly pass is not very challenging. Automated testing only leaves the identification of the source
of a fault to the testers, which not just improves his or her efficiency, but also creates a more motivating
working climate. The same is true for the test case designers, who worship programming more than writing
test instructions.
Another advantage of automated testing is that the skill of the persons running the test suites doesn’t
need to be high. Setting up a test suite and running it is a task for a student or a newly employed person.
Experts are only needed for programming the test cases. If the test instructions are detailed enough, they
even only need to be consulted occasionally by the test case designers, which are more software designers
than testers. Experts in GSM are more difficult to find nowadays, so automated testing can help lessen the
strain on these valuable resources as well.
Besides benefits, there is also one drawback, though. A very important factor for automated testing is
trust, which has to be gained from the testers. When test cases would not pass during the preparation phase,
testers not involved in test case programming always blamed the test platform first. In nearly all the cases,
14
the fault could later be traced to the system under test, but this had to be demonstrated by executing the
same test case manually. The test platform was actually running very stable with only two crashes in 2
months. It may take a while of practical experience until all testers fully acknowledge the benefits of test
automation.
5. Conclusions
In the Type Acceptance project for the release Ericsson GSM R8 CSS we have successfully used the
NEXT/AIMS tool platform to automate 47% of the test cases that were executed. As the automated test
cases can run fully automatically with only a small human effort required for starting and supervising the
test campaign, we have therefore saved considerable human resources during test execution. Of course, as it
was the first application of the tool platform, a large effort had to be spent for implementation, testing and
validation of test cases. But once programmed, the test cases can be used in all the following acceptance
tests and the initial investment pays off. Furthermore, we have reduced the length of the Type Acceptance
execution phase by 30%. Therefore automated testing contributes to a reduction of the time-to-market for a
new release.
However, we have learned that automated testing did not speed up the test case execution significantly.
Although the tool itself is able to execute test cases much faster than any human being, the limiting factor is
the SUT, which has to react to the actions initiated by the test case, e.g. by executing commands or sending
result printouts. Therefore the automation gain lies in the fact that automated test cases can run unattended
by the tester, which increases the efficiency of test plant utilisation. However, the stability of the test
configuration has turned out to be critical for test automation. A severe configuration fault, which was not
detected before the test started, can easily ruin the whole test campaign, which is normally detected in the
next morning. The same statement holds true for manual testing in parallel to a running automated test
campaign, which therefore should be avoided. If you keep this in mind and organise the test sessions in a
way that automated testing can run undisturbed by other activities in the test plant, test automation can be a
great success for a company.
6. References
[1] S. H. Redl, M. K. Weber, M. W. Oliphant: “An Introduction to GSM”, Artech House, 1995
[2] M. Mouly, M. B. Pautet: “Current Evolution of the GSM System”, IEEE Personal Communications
Magazine, October 1995
[3] ETSI specification GSM 03.60: “Digital Cellular Communications System (Phase 2+); General Packet
Radio Service”
[4] The UMTS Forum: “The Path towards UMTS – Technologies for the Information Society”, UMTS
Forum Report No. 2, 2000
[5] Dieter Kreuer, Simon Hoff, Jürgen Sauermann: “Facing the Challenge of Growing Telephone Network
Complexity with Automated Testing,” Proc. EuroStar 98 conference, Munich, 1998.
[6] Dieter Kreuer: “Applying Test Automation to Type Acceptance Testing of Telecom Networks: A Case
Study with Customer Participation,” Proc. Automatic Software Engineering ASE 99, Cocoa Beach,
FLA, USA, Oct. 1999.
[7] R.C. Linger: “Cleanroom Software Engineering for Zero-Defect Software,” Proc. of the 15th
International Conference on Software Engineering, IEEE Computer Society Press, Baltimore, 1993.
[8] J.D. Musa: “Operational Profiles in Software Reliability Engineering,” IEEE Software, March 1993,
pp. 14-32.
[9] J.A. Whittaker M.G. Thomason: “A Markov Chain Model for Statistical Software Testing,” IEEE
Transactions of Software Engineering, October 1994, pp. 812-24.
15
[10] Michael R. Lyu (Editor), Handbook of Software Reliability Engineering, McGraw-Hill, 1995, pp. 75657.
[11] ITU-T Recommendation X.290: “OSI Conformance Testing Methodology and Framework for Protocol
Recommendations for ITU-T Applications”
[12] ETSI specification GSM 02.34: “Digital Cellular Communications System (Phase 2+); High Speed
Circuit Switched Data – Stage 1”
[13] ETSI specification GSM 02.80: “Digital Cellular Communications System (Phase 2+); Supplementary
Services – General Aspects”
[14] ITU-T Q.700: “Introduction to ITU-T signalling System No. 7”
16
Thursday 7 December 2000
T7
Efficient Mobile Network
Acceptance Testing
Jorg Paul
Jorg Paul graduated in physics at the Aachen University of Technology in 1995. At
University he mainly worked in the fields of theoretical elementary particle physics and
computational physics. In 1996 he joined the German GSM network operator
Mannesmann Mobilfunk. Since then he has been working in several software testing and
test automation projects. He is now project leader for the introduction of automated
testing into the joint type acceptance project GSM R8 of Mannesmann Mobilfunk and
Ericsson.