V0.1
31/10/2016
Wholesale Market Reform
CMOS 2.3 Exit Criteria
10th January 2017
Executive Summary
The purpose of this document is to set out for review a set of Exit Criteria to be applied to
the MOSL CMOS 2.3 release.
This criteria only covers MOSL and it excludes Market Participant criteria.
The Test Exit Criteria will be used as the gateway between the transition of the CMOS 2.3
release from the Market Participant Sandpit (MPS) to the Production Environment. This
Test Exit Criteria is designed to capture the current risk position and it will be used to
determine when the testing is complete. This may include the acceptance of concessions.
Note: although this criteria covers the defects uncovered by the Market Participant and
MOSL’s testing, it isn’t intended to replace the criteria defined by MPs or MOSL set out
within their own test strategies. Specifically, MOSL retain the responsibility of ensuring
CMOS is fit for purpose, and each MP has a responsibility of ensuring CMOS integrates
successfully with their own internal systems and processes, through their own internal
quality assurance processes.
Copyright © United Utilities Water Limited 2017
2
Exit Criteria (MOSL)
In the context of the following criteria, the set of "tests" refers to the testing scope covered
by both MPs (covering their internal systems and processes) and MOSL (covering the
CMOS product), including and not limited to SIT and UAT. The criteria is as follows:
• A confirmed baselined version of the application and any associated data and
configuration items was delivered into the Market Participant Sandpit;
• 100% of all in-scope tests have been attempted and all defects encountered have been
logged and triaged with MOSL, complete with the agreed assigned priority and severity;
• All defects delivered as part of the CMOS 2.3 Release Note have been retested;
• All fixed defects delivered during the testing as hotfixes have been retested;
• The impact of de-scoped tests (including de-scoped defect fixes) is understood and
captured;
• No defects exist with a priority of P1 or business severity of S1 (see, “Appendix A” for
definitions);
• The impacts of any P2 or S2 defects is understood and accepted;
• Outstanding P2 or S2 defects have an agreed (CIO Forum) release for delivery;
• The impact of P3 and P4 defects are understood and accepted;
• If test cases or defects are outstanding, an acceptable technical or procedural
workaround has been identified with an acceptable agreed residual risk;
Copyright © United Utilities Water Limited 2017
3
Exit Criteria (MOSL)
• Test Completion Report is updated with the results of the testing and is signed-off
within MOSL;
• A tested back-out plan is available and in place;
• Release Notes, deployment, and back-out guides have been updated for use in live;
• Any test constraints (e.g. missing end-points requiring the use of stubs and / or “test
only” configuration changes) are captured and shared;
• It is agreed any P1 or S1 defects occurring in production after the release date are
resolved within 10 days of go-live.
Copyright © United Utilities Water Limited 2017
4
Appendix A - Definitions
Note: the Market Participant User Forum will need to agree how to convert observations
to defects.
Defect priorities and severities:
Priority
P1 - Urgent
P2 - High
P3 - Medium
P4 - Low
Description
Blocking defects (i.e. technical showstoppers) experienced by more than one Market Participant. No
testing is possible.
Highest priority for non-blocking defects. This indicates areas of the testing cannot be performed or
the functional area is significantly degraded; however, progress can continue in other areas.
Indicates a defect which should be fixed.
Indicates little urgency is needed for a defect fix
Severity
S1 - Business Critical
Description
Blocking defects (i.e. Business showstoppers) experienced by more than one Market
Participant. The Business cannot operate, can operate at considerable risk (e.g. reputational,
financial) or cost, there is impractical effort required.
S2 – Serious
Areas of the Business cannot operate or can operate only for days at considerable risk or cost
with significant effort required.
S3 - Degraded Service Business workarounds are required at risk and cost. Note increased volumes of minor
defects collectively may become a serious issue.
S4 - No Service Impact Business workaround available at an acceptable risk and cost.
Copyright © United Utilities Water Limited 2017
5