ERP Testing Challenges– Need for a Stable Test Environment
Presented At
STEP AUTO 2011 – ERP testing Conference – Dec 8, 2011
by
A. Raghunath, Senior Project Manager – Package testing
Pinaki Sen Sarma, Project Manager – Package testing
© 2011 Infosys Ltd.
1
Table of Contents
• Nuances of ERP Testing & related discussion
• Typically how QA environment is provided
• Environment transition activity
• Types of environment issues
• Environment Issue Impacts
• Recommended activities for improved environment stability
• Conclusion
2
Nuances of ERP Testing
But are these enough?
Niche
Skill
set
Integration
of multiple
modules
What are the critical
success factors other
than skill set?
Do we take anything for
granted?
The bigger
picture – End
to end
perspective
There may be quite a few contributing factors,
but for our discussion, we will focus on THE
NEED FOR A STABLE TEST ENVIRONMENT
3
Typically how QA environment is provided
• The instances for QA environment (Stage environment) could be created in multiple
ways:
• Copy of past release environment
• Copy of production
• Hardware of the QA environment is almost never at par with the production
environment because of the cost factor involved!
4
Environment transition activity
• Environment related verifications done during transition phases are:
5
i.
Pre-flow activity
ii.
IT Verification
iii.
Smoke Test
iv.
Ramp-up
Types of environment issues and associated impacts
a. Concurrent Program (CP) Issue
b. TIBCO Issue
c.
Data-sync Issue
d. Performance Issue
e. Unplanned Environment Downtimes
Teams
Issue
QA
Development
Infra
DBAs
Concurrent Program Issue
Yes
Yes
Yes
Yes (Occasionally)
TIBCO Issue
Yes
Yes
No
No
Performance Issue
Yes
Yes
No
Yes (Occasionally)
Data-sync Issue
Yes
Yes
No
Yes
Environment Downtime
Yes
Yes
Yes
Yes
6
Environment Issue Impact (Contd..)
Environment issues account for the most number of defects in a majority of the releases we
have handled this far. Following is the Pareto analysis of one such release:
200
Pareto Analysis
184
180
100% 100%
99.46%
98.75%
97.85%
94.27%
90%
88.53%
Around one-third (33%) of the total
defects were environment related!
82.26%
160
75.63%
140
140
80%
70%
120
58.06%
100
60%
98
50%
80
40%
32.97%
60
30%
37
40
35
32
20%
20
20
5
4
2
0
10%
0%
Environment
Config/Setup
Code Related
7
Not a defect
System/Backend
data
Design
Documentation
Total
Cum %
'Invalid Test
Script
Reqt not in Pjt
Scope
User-Error data
Enter
Environment Issue Impact (Contd..)
There can be a wide range of issues which fall under the environment issue category :



CDB & Concurrent Program issues
constitute 26% of the environment
related defects
Many of the CDB issues were pertaining
to replication of data from DB1 -> DB2
Less frequent sub root-causes (like
Cross Functional Issue, Invalid Trigger,
SABRIX Issue etc ) have been clubbed
under Misc category.
8
Environment Issue Impact (Contd..)
It’s not just the environment issues per-se that affects testing, but also the frequency of
recurrence of the same. While some of them are one-time issues, there are quite a few which
recur very frequently, resulting in more effort wastage than originally anticipated:
9
Environment Issue Impact (Contd..)
We had captured data for effort loss across 2 consecutive releases and following were our findings:
Quarterly Release
Total Available Hrs
Actual Impacted Hrs
Release 1 – Core Projects
Release 1 – Project B
Release 1 – Regression
Total Release 1
Total % Effort Lost for Release 1
Release 2 – Core Projects
Release 2 – Project B
Release 2 – Regression
Total Release 2
Total % Effort Lost for Release 2
6050
3778
1144
10972
610
880
110
1600
15%
320
472
309
1100
19%
2100
1900
1900
5900
We need a stable
test environment!
Converting the above effort loss into dollar values shows the significant impact of an unstable
environment on the overall cost.
Release
Release 1
Release 2
10
Total QA Cost
700 K
400 K
Actual Loss
(%)
15 %
19 %
Actual Loss
(‘000s of $$)
105 K
76 K
Recommended activities for improved environment stability
• Strengthening the environment transition process
• Data-sync verification during instance copy
• Dedicated Infra/DBA support to monitor environment issues
• More hardware support for testing instances
• Robust resolution mechanism for QA issues (w.r.t environment)
• Creation of focused teams for environment monitoring
• Separate performance test environment
• Looking at new paradigms like Cloud computing / Virtualization as options for creating the test instance
11
Conclusion
Initial drivers for cost reduction – Offshoring
Looking for additional efficiencies – one key area is a stable QA environment
Adequate attention, proactive monitoring and other factors can reduce cost between 15
– 20%
Newer Paradigms like Cloud computing / virtualization of QA environment will help in
driving additional efficiencies.
12
THANK YOU!
© 2011 Infosys Technologies Ltd.
13