National Renal Dataset
Full Operational Standard
Appendix G7
National Datasets Service
National Renal Dataset
Changes to Implementation Plan and Lessons
Learned from Operational Testing
Changes to Implementation Plan and Lessons Learned through Operational Testing.
Purpose of this document
The purpose of this document is to report on the
changes to the implementation plan and lessons
learned from operational testing.
VERSION HISTORY
Version
Date
Brief Summary of Change
Owner’s Name
0.1
12/06/08
Initial draft
T Holdsworth
0.2
21/07/2008
Incorporation of comments from peer review
T Holdsworth
0.3
22/07/2008
Inclusion of comments from Project Board
A Roe
1.0
22/07/2008
Finalised version
A Roe
For more information on the National Datasets Service
status of this document, please The Information Centre for Health and Social Care
contact:
1, Trevelyan Square
Leeds
LS1 6AE
Date of Issue
Reference
Telephone:
E-mail:
0845 300 6016
[email protected]
Internet:
www.ic.nhs.uk
PR139 Renal Datasets
© Copyright 2008 The Information Centre for Health and Social Care, National Datasets Service.
All Rights Reserved.
This work remains the sole and exclusive property of The Information Centre and may only be
reproduced where there is explicit reference to the ownership of The Information Centre. This
work may only be reproduced in a modified format with the express written permission of The
Information Centre.
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
1
Copyright ©2008 The NHS Information Centre
Changes to Implementation Plan and Lessons Learned through Operational Testing.
Contents
1.
BACKGROUND .....................................................................................................................................................3
2.
REVIEW OF MACRO LEVEL TESTING PLAN ..............................................................................................3
2.1.
3.
SCOPE CHANGES ...............................................................................................................................................3
TESTING SITES.....................................................................................................................................................3
3.1.
SUPPORT - LOGGING OF OPERATIONAL TESTING ISSUES .....................................................................................4
4.
SYSTEM DEVELOPMENT ..................................................................................................................................5
5.
DATA CAPTURE AND COLLATION ................................................................................................................6
6.
OUTCOMES ...........................................................................................................................................................6
7.
APPENDIX A – NRD SHARE POINT SCREENSHOTS ...................................................................................7
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
2
Copyright ©2008 The NHS Information Centre
Changes to Implementation Plan and Lessons Learned through Operational Testing.
1. Background
Lessons learned review of the Operational Testing of the National Renal Dataset (NRD) against
original plans and operational issue log.
2. Review of Macro Level Testing Plan
The “Macro Level Testing Plan version 1.1” which was based on the “Datasets Testing Strategy
version 1.0” covers the operational testing planned.
Macro_Level_Testing
_Renal_1.1.doc
2.1.
Scope Changes
The original scope for the project was for the data to be collected in a clinical setting on
operational systems and be transmitted to the UK Renal Registry (UKRR) and UK Transplant
(UKT) and then on to the Secondary Uses Service (SUS). The scope was amended to exclude
SUS and that the end point of the data journey would be UKRR and UKT. The Macro Level
Testing Plan was amended to reflect this.
Learning points

It is important to review and amend plans when the overall scope has changed.
3. Testing Sites
After a state of readiness review it was decided to proceed with Derby as the sixth testing site.
The task of supplier liaison and co-ordination for delivery of system upgrades was to be carried
out by the UK Renal Registry. This was part of the Service Level Agreement (SLA) between the
IC and UKRR. This was agreed as an important role in ensuring appropriate technical and
content quality of the system upgrades for the dataset implementation; however it meant that the
IC project team were one step removed from system specification discussions. The Clinical
Director of the renal unit at Derby agreed that collection could begin before the end of March 2008
of a sample of data items focussed on Haemodialysis (HD) and Peritoneal Dialysis Access.
However the IC project team were not involved in discussion between the system supplier and
UKRR.
On a site visit to Derby it was found that the system, Vital Data (provided by Vital Pulse) had not
been developed specifically to accommodate the NRD testing collection but already did capture
data items for Haemodialysis Access and Peritoneal Dialysis Access, some with slightly different
value sets. Given that the systems underlying database is based on PROTON there were no
significant technical limitations to development rather issues with the scheduling of work with the
developer.
The renal unit at Norwich were keen to take part in the operational testing from the outset of the
project however preparation at that site suffered as there was little communication between UKRR
and Mediqal (issue OP01). The IC facilitated the communication as development of the eMED
Renal system at Norwich was at risk of not being ready in time and this would have excluded the
unit from participation in the operational testing. If this had occurred the testing would not have
met the agreed minimum specification of five sites and two systems.
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
3
Copyright ©2008 The NHS Information Centre
Changes to Implementation Plan and Lessons Learned through Operational Testing.
Involvement of the UKRR was of fundamental importance and the technical development work
required to achieve the operational testing could not have been completed without their input.
Both the UKRR and The IC project team understand the importance of a clear and unambiguous
SLA and achievement of this requires detailed discussion. The timescales planned for the work
were acknowledged as being ambitious by members of the Project Board and the technical
complexity of the work required to turn the data items into operational system items proved to be a
significant factor in the development timescales.
Learning points

Operational implementation of data items is a very complex process and there needs to be
flexibility to allow for changes to the planned work and timescales associated with the
development process

The close involvement of technical specialists in the subject area, in this case UKRR, is
vital to successful delivery of operational testing.

Consideration needs to be given as to whether to delegate system development to third
parties; the rationale was one of patient safety and quality assurance. For future projects
the IC should be a stakeholder in the specification for third party systems in partnership
with expert domain agencies.

Being closer to the development process will improve project information flows in terms of
communicating issues/progress reports to aid decision making.

Having direct access to third party suppliers gives two way benefits, as well as the project
beam being informed directly of the current position from the supplier, the supplier in turn
benefits from being able to ask the IC questions directly.

Contact with sites should be at all levels, sometimes the wrong impression of what is
actually happening at a site can be given from liaising at a high level only. Liaising with a
range of staff informs new lines of questioning to understand sites local business
processes and issues.
3.1.
Support - logging of operational testing issues
Support was given by site visits, telephone support and an external Share point website, screen
shots from this site are included as Appendix A and the issue log is embedded in this document
for reference.
NRD-Sharepoint-issu
e log.doc
Appendix A – NRD Share point screenshots
The majority of support required was around the new developments to PROTON and issues in
understanding the implementation in the software. The IC project team had a copy of the system
specification but no guidance notes relating to the software are available. The project team were
provided with a series of screen shots before the start of the testing and these were circulated to
the sites using the PROTON system.
The Share point website was used for documents, templates and operational testing issues.
There were changes within The IC relating to the setting up of external users, this involved users
filling in complicated automated registration forms for access. This change introduced by the IT
department was very complicated and customer unfriendly. After internal representations the IT
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
4
Copyright ©2008 The NHS Information Centre
Changes to Implementation Plan and Lessons Learned through Operational Testing.
department loaded the user list manually. In terms of using the website for a central location for
issues, this was largely successful with key stakeholders having access to the latest version.
Learning points

More can be gained from the use of a Share point website to support testing.
important that issues surrounding the use of such websites are resolved.

Helpdesk processes should be agreed with supplier, administrators and sites before
operational testing commences.

Training materials should be developed in conjunction with developers, suppliers and local
business processes.
It is
4. System Development
This task was assigned to UKRR and the project team at The IC had little information as to what
the changes would look like until these were released to trust testing sites. No test system was
available for the PROTON system to be demonstrated and developed safely. The acceptable
value lists in PROTON were generic and could allow invalid items to be entered.
When PROTON went live with the changes there were a number of issues which impeded the
collection of data (issues OP02, 05, 06, 07, 09). This was mitigated by a prompt response from
the developer. On site visits it was found that the renal unit at Southmead Hospital, Bristol already
had significant collection of Vascular Access on their legacy screens. With hindsight it may have
been more appropriate to develop these local screens for use in the testing phase. The result of
having these screens meant there were duplicate values to be entered on the system for some
data items causing an unnecessary confusion and additional burden on staff.
Mediqal has a test system for development and user sites have access to a test instance of the
application for testing and appraising internal business processes. This is a safer development
process although it is time consuming and new release timetables can be difficult to plan as in
spite of risk management assessment, the nature of development means that unforeseen issues
occur and must be managed to ensure the release is fit for purpose.
Learning points







Test systems should be available for software demonstration and development.
Training materials should be developed in conjunction with developers/specifiers and
business processes.
Sufficient time must be allowed to enable suppliers to thoroughly test new versions before
releasing to live systems.
Where there is already existing ‘pioneering’ collection on systems this should be given full
consideration for further development given the collection experience.
Acceptable value lists should be tailored to the field currently in focus
Cross validation checks across fields should be applied to improve the quality of data
collected.
Systems should be analysed for duplication of collection, to avoid inconsistencies and
unnecessary burden where possible.
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
5
Copyright ©2008 The NHS Information Centre
Changes to Implementation Plan and Lessons Learned through Operational Testing.
5. Data Capture and Collation
Sites had individual SLA and also individual agreed collection specifications based on the new
data items in the NRD. These specifications had considerable overlap across sites to provide
consolidation of data capture. These were agreed with clinical directors however when frontline
staff reviewed the site agreed collections a number of differences were found in what they could
actually collect as part of a business process and what systems could technically collect. There
was wide variation between sites, in part due to the local configuration of systems. Despite
inclusion of a specification within the SLA between The IC and UKRR, some misunderstandings
occurred in developing the UKRR database to receive data from sites.
This misunderstanding resulted in less data items actually being transmitted to the UKRR, which
affected the prescribed items in the testing specification. This issue was picked up as part of the
regular visits by the Datasets Testing Manager to UKRR. The issue was mitigated by extracting
data direct from PROTON for analysis at The IC. One site failed to collect data to their SLA
specification as the planned collection of the extra items was to use overtime for the staff involved
(issue OP12). Even though the costs were to be reimbursed by the National Datasets Service,
the protocol at the Trust dictated that staff in the unit could not work any overtime.
Learning points

Agreement on the data collection should involve staff at all levels where possible, it is not
that there is any intention to mislead but sometimes detailed areas of work are not widely
understood across all staff groups.

Regular visits to check on progress and that plans are being interpreted correctly proved
vital while there is still time and resource available for the direction to be amended and
mitigated.

Sign off of individual items is required to confirm development according to specification.

There should be overlapping collections as contingency for sites failing to collect items as
agreed. This can be for reasons unrelated as to the clinical/technical viability of collection
i.e. a Trust overtime ban.
6. Outcomes
The lessons learned from the operational testing have been discussed by the Project Board and
stakeholders in the National Renal Dataset. The lessons have been used to inform the risk log for
implementation of the NRD as a Full Operational Standard and will also be used to inform future
dataset developments by the National Datasets Service.
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
6
Copyright ©2008 The NHS Information Centre
Changes to Implementation Plan and Lessons Learned through Operational Testing.
7.
Appendix A – NRD Share point screenshots
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
7
Copyright ©2008 The NHS Information Centre
Changes to Implementation Plan and Lessons Learned through Operational Testing.
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
8
Copyright ©2008 The NHS Information Centre
Changes to Implementation Plan and Lessons Learned through Operational Testing.
Version No: 1.0
Date: 22.07.2008
Author: Tim Holdsworth
9
Copyright ©2008 The NHS Information Centre