Consultation on the further overlap of
Systems Integration Testing and Interface
Testing for DCC Release 1.3 – Proposal to
Secretary of State and summary of
consultation responses
Date:
12 September 2016
Classification:
DCC Public
1
Proposal
On 1 September 2016, DCC published a consultation seeking Parties’ views on proposals
to request the further overlapping of elements of Systems Integration Testing (SIT) and
Interface Testing (IT) for DCC Release 1.3 (R1.3), in a manner which would not affect the
ability of a User to conduct:
i.
ii.
2
User Entry Process Testing (UEPT),or
the ability of the DCC to conduct SIT UAT (User Acceptance Testing).
Background
The purpose of Systems Integration Testing (SIT) is to demonstrate that the component
parts of the DCC Systems can comply with the requirements of sections E, G and H of the
SEC, and that Registration Data Providers (RDPs) can comply with the requirements in
section E of the SEC. In complying with these sections DCC must have regard for the
Technical and Procedural requirements supporting each section of the SEC referred to
above, as set out in the Testing Baseline Requirements Document (TBRD).1
SIT comprises a period of SIT Solution Testing (SIT ST) and a period of SIT User
Acceptance Testing (SIT UAT) which culminates in the publication of a Test Completion
Report on a Region-by-Region and RDP-by-RDP basis confirming that the DCC is
capable of meeting all requirements set out in the TBRD.
The SEC recognises the potential requirement for the overlap of SIT and IT. Section T3.6
of the SEC states that prior to the start of Interface Testing, DCC can request to the
Secretary of State that Interface Testing should commence from some point during SIT,
providing that DCC first sets out its analysis of the benefits and risks of doing so, following
consultation with SEC Parties.
The SEC also accommodates the concept of transitional testing through numerous
releases of DCC functionality. Section T3.36 enables overlap between any ‘Additional
Systems Integration Testing’ and ‘Additional Interface Testing’, which are the defined
terms in the SEC for R1.3 SIT and R1.3 IT respectively.
DCC undertook a consultation on changes to the plan for R1.3 delivery on 7 July and on
31 August submitted a revised plan for R1.3 for approval by the Secretary of State. This
plan incorporates (and is dependent on) an overlap of SIT ST with IT, and it is this overlap
on which DCC consulted on 1 September.
In this consultation, DCC proposed that R1.3 SIT ST and R1.3 IT should only overlap in
respect of functionality that does not impact Testing Participants in UEPT in respect of a
particular user role. This principle is illustrated in the following diagram:
1
In accordance with SEC T2.3, DCC must demonstrate compliance with TBRD in respect of R1.2 requirements before it can exit SIT.
The extant TBRD is available here: https://www.smartenergycodecompany.co.uk/sec/the-developing-sec
Further overlap of Systems
Integration Testing and Interface
Testing
DCC Public
Page 2 of 6
3
Consultation responses
DCC sought general responses from stakeholders on the overlap proposals in the
consultation, as opposed to asking specific questions.
In total 5 stakeholders responded to the consultation, all of whom were Large Supplier
Parties. A summary of the themes raised across responses is provided below.
3.1
Managing parallel activities and resource contention
Respondents raised concerns over the impact that the additional overlap of testing phases
carries risks and could lead to a plan which ultimately takes longer to deliver. One
respondent questioned the success of control points as mitigating steps in the plan to
date. Several respondents noted the issue of DCC’s testing resources being stretched;
expressing concern that DCC would be running so many testing activities in parallel at this
point in the delivery programme. It was suggested that the monitoring of progress against
defined criteria with tolerance thresholds would provide further assurance to Testing
Participants. One respondent noted concerns that testing the scope of R1.3 would be
more technically complex than R1.2, given the focus on pre-payment functionality, and
that testing should not be overlapped because of this.
Further overlap of Systems
Integration Testing and Interface
Testing
DCC Public
Page 3 of 6
DCC response
Whilst noting the concerns raised regarding the use of control points, DCC considers that
their use in the plan remains the most effective way to track progress across both
releases, including for R1.3 SIT and IT, and to assess the dependencies and constraints
that exist between activities. As noted previously in DCC consultations regarding
Programme delivery, control points provide the opportunity to manage the inherent
uncertainty in the programme, and where possible to mitigate risks in a way that does not
impact on DCC Live.
DCC has established and will maintain the agreed testing thresholds set out in its Testing
Approach documents, which have previously been subject to consultation and approved
and DCC considers that these are the appropriate levels to maintain throughout SIT and
Interface Testing for R1.3. We have been sharing important testing information as part of
our engagement with stakeholders, including via weekly SIT progress reports (and joint
SIT/IT progress reports where test phases overlap) and weekly teleconferences to the
SEC Panel’s Testing Advisory Group and the Smart Metering Delivery Group and regular
presentations to its supporting specialist sub-groups. DCC commits to continuing this
engagement as we move through the testing phases for R1.3.
Whilst noting that the pre-payment Service Requests under test for R1.3 are more
complex for Testing Participants than those tested in R1.2, there are other factors which
make R1.3 less burdensome. These include that most of the enduring solution will have
been successfully tested in R1.2, including the core motorway, the security solution and
the Service Management System. There will be no testing with RDPs, or of the CPL.
Additionally, whilst the first time proving of the security trust model took extensive time to
resolve in Release 1.2, this should not be the case in R1.3. On balance, we do not
consider it necessary to amend the planned overlap on this basis.
3.2
Building on lessons learned and the risk of overlap impacting the
overall delivery of R1.3
It was noted by respondents that building on lessons learned in R1.2 testing was
important, with one respondent noting that test statistics suggested a high number of high
severity defects were outstanding when UEPT for R1.2 commenced, implying that testing
was being undertaken against a potentially unstable environment. It was also noted that
this approach may lead to the need for protracted End-to-End testing, ultimately leading to
a live date for R1.3 which is later than if SIT ST and IT had not been overlapped.
DCC response
UEPT for R1.2 commenced with a sub-set of Service Requests which had completed
testing in SIT without any outstanding defects. However, problems were discovered with
the manner in which Communications Hubs had been installed in the Northern Region test
labs, and a defect was discovered that impacted the ability of the Communications Hubs
to communicate with the emulator in the UIT environment in this test lab. DCC is
committed to utilising learning from these problems, and ensure that they don’t reoccur in
R1.3.
DCC considers the proposed overlap of testing stages an important means by which to
incorporate flexibility in the programme plan, across the range of testing activities. We
Further overlap of Systems
Integration Testing and Interface
Testing
DCC Public
Page 4 of 6
have made use of, and will continue to make use of testing statistics to inform our
approach. Through R1.2 testing DCC has evolved better ways of working in defect triage
and remediation and we plan to build on that learning for R1.3, not only to identify key
issues, but also to identify and build on quick wins to improve progress.
Further overlap of Systems
Integration Testing and Interface
Testing
DCC Public
Page 5 of 6
4
Next steps
After taking into consideration consultation responses received, it remains DCC’s view
that is it appropriate to request the further overlapping of elements of Systems Integration
Testing (SIT) and Interface Testing (IT) for R1.3, in a way which would not affect the
ability of a User to conduct User Entry Process Testing (UEPT or the ability of the DCC to
conduct SIT UAT (User Acceptance Testing).
In accordance with Sections T3.6, and T3.36 of the SEC, DCC is submitting this report,
along with consultation responses, to the Secretary of State for consideration.
Subject to the Secretary of State’s conclusions regarding this proposal, DCC will continue
to engage with stakeholders to finalise the planning of any overlap, including sharing
information on prioritising overlapped functionality with the TAG, and where necessary
updating the relevant test documentation to support the revised approach.
Further overlap of Systems
Integration Testing and Interface
Testing
DCC Public
Page 6 of 6