The benefits of modeling in a large-scale test
integration project: A case study.
Graham Bath, Principal Consultant, T-Systems Test Factory .
Structure of the presentation.
Setting the context
ƒ
ƒ
ƒ
Test Factory
Project Characteristics
Motivation for change
How our testing approach was improved by using models
ƒ Phases of change
Conquest 2008
13.08.2007
2
Context:
What is the T-Systems Test Factory?
What kinds of projects are we involved in?
Test Factory International.
T-Systems is a part of Deutsche Telekom AG.
Test Factory is a part of T-Systems.
„at work“
„around and
about“
„at home“
Conquest 2008
13.08.2007
4
Test Factory International.
What we do.
Automation
Test consulting
Test tools
Test
management
Test
infrastructure
management
System test
System
integration test
Captive customers
External customers
Conquest 2008
13.08.2007
5
Test Factory International.
Principal responsibilities.
Specification/
Design/
Implem. n
(Supplier n)
Module
FunctionalTest 1
Test* 1
(Supplier 1) (Supplier 1)
Module
Test n
(Supplier n)
*incl. bilateral Interfaces
Functional
Test* n
(Supplier n)
Operations
Quality Gate: eExit criteria
Specification/
Design/
Implem. 1
(Supplier 1)
Test Factory
Quality Gate: Entry criteria
Development
Functional
System
integration test
Operational
System
integration test
Operation
Hotfix-Tests
"Transition to Operation"
Conquest 2008
13.08.2007
6
Test Factory International.
Growth.
2008
2007
2005
1993
First independent
testing
organization
started
1999
Build up of
Specialisation in
Telecommunicatio
ns branch
Development
Offshore/
Nearshore
2002
Start Offshore/
Nearshore
Founding of
international Test
Factory with points
of production in
Germany,
Hungary, India.
Partnership with
Cognizant
Strong growth in
Germany
Development of industry branches
(P&H, financial, telematic)
Conquest 2008
13.08.2007
7
T-Systems Test Factory.
Project characteristics.
The IT-Landscape of our customer is distinguished by the following essential points:
ƒ
Complex application landscape (e.g. Interoperability of up to 100 Applications)
ƒ
Many functional and operational requirements on these applications:
ƒ
Mostly highly-complex business processes, which traverse a large number of applications.
ƒ
Extremely large data volumes (e.g. up to 40 m. customer data records), which have to be
kept consistent over several applications.
ƒ
Frequent introduction of new requirements and products at short notice.
ƒ
Decentralized development of individual applications by different suppliers.
Conquest 2008
13.08.2007
8
Starting
point
Testing Approach: Before 2007.
On-shore
Customer
Requirements
(Text)
Testanalysis
Text
Testspecification
Testexecution
Test cases TWB
„ Text-based requirements with varying quality
„ Established ISTQB-Standard test process
„ Results of Test Analysis and Specification manually inserted into the
test management tool “Test Factory Workbench” (TWB)
„ This was not a satisfactory solution for working with our offshore organization
Conquest 2008
13.08.2007
9
Motivation for change.
Slow Know-How transfer – high communication effort.
Initial Situation
Documentation
ƒ Expert knowledge only through
„Local heroes“
ƒ Text-based and high-level
ƒ German documentation must
be translated
Standardization
ƒ Strong bonding within teams
ƒ Weak interaction between teams
ƒ Informal communication
Results
Â
Â
Misunderstandings
Time consuming and low quality knowledge
transfer
Time, costs, and quality loss
Efforts to standardize restricted
Responsibilities difficult to define, especially
in offshore projects
Conquest 2008
13.08.2007
10
Motivation for change.
Slow Know-How transfer – high communication effort.
Initial Situation
Documentation
ƒ Expert knowledge only through
„Local heroes“
ƒ Text-based and high-level
ƒ German documentation must
be translated
Standardization
ƒ Strong bonding within teams
ƒ Weak interaction between teams
ƒ Informal communication
Results
Â
Â
Misunderstandings
Time consuming and low quality knowledge
transfer
Time, costs, and quality loss
Efforts to standardize are slowed down
Responsibilities difficult to define, especially
in offshore projects
Conquest 2008
13.08.2007
11
Improving the testing approach
using models
Model-Driven Approach.
Models as Knowledge Carriers.
A model-based approach offers:
Knowledge management und documentation:
ƒ Minimized impact of language barriers through visualization
ƒ Ability to present information easily at different
levels of abstraction
Process Standardization:
ƒ Standard Notation (e.g. UML)
ƒ Offers a controllable division of responsibilities
ƒ High automation potential
Conquest 2008
13.08.2007
13
Current
situation
Model-Driven Approach: Phase 1.
Models in UML 2.0
Mostly Onshore
Customer
Requirements
(Text)
Testanalysis
Mostly
Offshore
Testexecution
Testspecification
Test cases
TWB
„ Text-based requirements are depicted as UML models (mostly activity diagrams)
„ The modeling (in Enterprise Architect) is supported by Test Factory guidelines
„ Models are attached in the test management tool (TWB)
„ Cooperation started with TU Munich
Conquest 2008
13.08.2007
14
What we needed to do.
Ensure Tool Support
ƒ Adapt and extend the test management tool to enable
models to be attached
Establish Modeling Guidelines
ƒ UML is powerful and versatile!
ƒ At first this flexibility can be a disadvantage
ƒ Modeling guidelines and coaching are essential.
A pilot project was conducted
ƒ
Ca. 30 complex business processes were implemented
using the Modeling Guidelines
Conquest 2008
13.08.2007
15
Phase 1: The benefits.
Offshoring is “enabled”
ƒ Models can be used instead of translated documents
Communications greatly improved
ƒ Less tacit knowledge – fewer misunderstandings
Staff motivation
ƒ Complex teams better integrated and more involved
Efficiencies
ƒ Up to 50% efficiency increase in some areas compared to an
approach without models
Conquest 2008
13.08.2007
16
What next?
Improving the testing approach
using models
Conquest 2008
13.08.2007
17
Primary impulse for the next phase.
Integrating with the customer
ƒ Customer also decided to introduce a model-based approach
as standard using MID-Innovator
ƒ Potential for common interface formats such as Domain
Specific Language (DSL)
ƒ Potential for making our testing-specific notations available to
our customer.
ƒ An optimum integration of the tool landscape with the
customer can be achieved and common central model
repository established.
ƒ Maximize customer “buy-in” in the testing process
ƒ Achieve Win – Win situations
Conquest 2008
13.08.2007
18
Model-Driven Approach: Phase 2 objectives.
Mostly
Offshore
Model in MID/Innovator
Customer
Requirements
(Model)
Testanalysis
Testexecution
Testspecification
TF-Generation
TWB
„ Establish an interface with the customer to enable the “seamless”
acceptance of requirements into the testing domain using models
„ Enable test cases to be generated from the test models and and imported
into the test management tool (TWB)
„ Establish the new approach as a Test Factory “best practice” (Roll-Out,
Change Management, Wiki “Test Factory Framework”)
Conquest 2008
13.08.2007
19
Benefits (so far).
Better understanding and transparency of complex
business processes
Ability to create test case instances
Conquest 2008
13.08.2007
20
High level overview of test model* using MID-Innovator
* for a specific
test sequence
Conquest 2008
13.08.2007
21
Benefits (so far).
Better understanding and transparency of complex
business processes
Ability to create test case instances
Conquest 2008
13.08.2007
22
Creation of test case instances
Instance for a
particular attribute value
(attribute = product,
value = T-ISDN)
Complete logic
Instance
Conquest 2008
13.08.2007
23
Test case execution: before/after.
Previous procedure
New procedure
................ ........................
.....
.......... .......... for T-ISDN ......
............ ........ ..............
1
Test case A
1
Test case instance A
2
Test case D
2
Test case instance D
3
Test case C
3
Test case instance C
4
Test case X
4
Test case instanceISDN
X
ISDN
DSL
Conquest 2008
13.08.2007
24
Expected benefits.
Less modeling effort by the testing organization
ƒ Test Factory only adds testing-specific aspects to the
models received
Stronger and smoother integration of the requirements process
with the testing process
Efficiency potential of increased automation
ƒ Test case generation on the basis of models
ƒ A basis for automated test execution is created
Conquest 2008
13.08.2007
25
References:
„Fallstudie: Modellierung eines großen Testintegrationsprojekts“,
Modelling Magazine, Oktober 2008 (www.mid.de)
„The benefits of modeling in a large-scale test integration project: A case Study“
Testing Experience magazine, September 2008 (www.testingexperience.com)
Many thanks!