Test Strategy Evolution Throughout the Lifecycle
Chris Comey &
Davidson Devadoss
We no longer ridicule Darwin
Are Evolution principles relevant and
applicable in the modern IT work
space?
Survival of the fittest
What is the fittest?
In hindsight, that which survives
Is this relevant for software testing?
Why Test Strategies Exit 1)
2)
3)
4)
5)
6)
7)
8)
To define a test framework & approach
To ensure the key activities are considered
To integrate the test function with the other teams
To ensure testing needs are highlighted
To communicate and raise awareness
To set ownership, roles & responsibilities for critical tasks
To gain agreement for the above – sign off
To address RISK!
Who works in a static environment?
• The world changes around the project
• Change is driven by technology, integration, changing requirements and test results
• Assumptions made are proven or disproven
• Motivational factors and business priorities change
What are testers good at?
•
•
•
•
•
•
Analysis Attention to detail
Ability to adapt to changing priorities
Focusing on priority
Assessing risk and scoping test coverage
Reporting time, cost, quality
Example 1 : Integrating Legacy Systems
Test Strategy developed
I/F
New
I/F
I/F
Legacy
Delivery Strategy
Development Team (x2)
Support Team
Infrastructure
Design
Test Team
Next version
under build
Q-Gate
•Build env
•Deploy code
•Shakedown
Test Team
Latest version
under system test
Q-Gate
•Sys Test Complete
•C/M defects fixed
•Good enough 2 Integrate
Integration
Support env
Integrated
Environment
Integration
Testing
Code Delivery Strategy
Next version
under build
Rel 3
Rel 2
Test Team
Rel 1
Environments
Integration
Support Env
Integration
Testing
What went wrong?
• Initial issues stabilising the first environment, meant:
–
–
–
–
System Testing did not start as scheduled
Development started to put releases ‘on the shelf’ – test backlog
Pressure to deliver to the Integration stage started to build
First release not fit for testing to start, 2nd dev release ready, 2nd
environment being prepared
• Analysis showed:
– 3 teams delivering individually (infrastructure, application code, Content, integrating to stable legacy interfaces )‐ No Link testing
• Decision
– Focus on getting 1 release ready for Integration (warts and all)
Some progress!
• One release tested enough to go for integration
• Still struggling to build 2nd (and now 3rd) system test environment
Observations
• No link testing meant infra, new code and new data content were not integrating prior to system test starting (so no test progress)
• Ownership of defects being disputed (code, env, config, etc)
• No/poor release notes – Release content mismatches found in testing
• No configuration management meant stable environments could not be reproduced from scratch
• Double/treble keying of fixes required
• Test Environment support provided by Live support (when available!)
version 1/2/3
under build
version 1&2 under
system test
V1 Integration
Support env
V1 Integrated
Environment
Observations
1)
2)
3)
4)
5)
6)
7)
Problems were seen by management as being issues within testing, when they were actually much wider!
The test strategy looked inwards at testing within the test team, it did not define the dependencies on other teams in enough detail
Testing relies heavily on the basic processes being in place – release management, environment management, change management, configuration management, defect management, all of which need to be agreed with the relevant parties.
Due to the number of teams involved there was a reluctance to deviate from the agreed test strategy – “we would have to go through sign off again!”
There appeared to be a reluctance to ‘do the right thing’, only what had been agreed (if I do what was agreed I am safe)
A ‘good’ test strategy may not be the ‘right’ test strategy
There was a “lessons learned” document from a previous project that detailed nearly all of the issues being experienced!
Example 2 : Phased release – offshore Development
Project Objectives: To deliver a new system to an extremely demanding client that is not prepared (too busy) to fully document their requirements.
Key points:
•Off shore development team
•Project approach to provide incremental deliveries for a series of ‘solution show & tells’ to ensure the solution is fit for purpose (Agile delivery).
Project Test Stages
•
•
•
•
•
Unit Test – Developer
Component Testing – Independent Developer
Component Integration Testing – Lead Developer
Smoke Test (QA Test env) – Lead Developer
Functional System Testing – Test Team
– Basic Functionality
• System Integration Testing – Test Team
– Critical End‐to End Scenarios
– Non‐Critical End‐to End Scenarios
• Non Functional Testing – Test Team
• UAT Stage – Business Team
The identified Risks at the Mid‐point review
Identified risks pre System Test stage
Component Integration Testing •Risk 1. There is a risk here that what works in Dev QA may not work in QA Test
this would extend the Smoke testing stage in QA.
Identified risks & mitigating actions
Smoke Test
•Risk 2 There is a risk that the source of any issues will be difficult to identify as Development do not have access to QA Test environment due to data sensitivity issues. This could mean that items that work in ‘Dev QA’ but not in ‘QA Test’ are difficult to isolate. Identified risks & mitigating actions
Development Testing overall ‐
•Risk 3 There is a risk that the Dev testing will complete successfully but critical issues will be found when the system is tested in an E2E manner. This is due to the lack of documented Business Processes which detracts from system context for development of the system. Identified risks & mitigating actions
• Risk 4 There is a risk that Dev will use all the time allocated for testing in order to deliver a higher quality product when the project would have preferred to take the delivery earlier with a lower intrinsic quality level. Potentially this means higher priority E2E defects will be raised later in the lifecycle potentially endangering the project end date.
Identified risks & mitigating actions
• Risk 5 .There is a risk that the project will be delayed striving for perfection and delivering low priority fixes in numerous releases that detract effort from the important defects and risk introducing new higher priority defects. This will prevent the stage from completing and the next stage from starting.
Identified risks & mitigating actions
Critical End‐to End Scenarios
•Assuming Risk 3 has been mitigated earlier in the lifecycle then any issues here will be as a result of environmental issues, deployment or configuration issues or True E2E integration issues.
Non‐Critical End‐to End Scenarios –
•By definition any issues discovered during test execution of lower risk scenarios are lower impact. As all defects will be assessed and categorised these will be dealt with in the normal manner. Identified risks & mitigating actions
UAT stages –
•Risk 6 There is a risk that the business users will not accept the way in which the system fulfils the business requirements. This would trigger Change Requests late in the project lifecycle and is a major risk to delivery.
Identified risks & mitigating actions
• Risk 7 There is a risk that the user will identify a new or missing business processes during UAT that will trigger a change request late in the life cycle.
What was learned?
1. Test Strategies should be written for the situation
2. A good strategy may not be the right strategy!
3. The world changes around the project, your strategy may have been right but is now incorrect due to changing external factors
4. Good practice put in place at the start of the project can have a negative impact later if not revisited and assessed ‘in the current situation’
5. Even if you involve the users at each stage this does not guarantee success
What was learned?
6. Detail the testing dependencies and inputs required (specially if they are outside your control)
7. Lessons learned are not learned if the learning is not applied!
8. Ensure the strategy complements the project objectives rather than replacing them 9. Schedule Test Strategy review points in the project plans – the world changes and so should we
10. Use the RBT approach to validate the test strategy throughout the test lifecycle Questions?
Who has worked on a programme or Project that has been scrapped?
Was that a ‘project failing’ or
changing external factors?
Don’t be a Dodo!
TSG
Bringing Value to:
• Organisations and People;
• Systems and Developments;
• Business & Corporate Objectives
Testing Solutions Group Ltd
5th Floor 117-119 Houndsditch
London EC3A 7BT
Tel: +44 (0) 20 7469 1500
Fax: +44 (0) 20 3411 8637
Web: www.testing-solutions.com