Information Systems Services
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach,
timelines and resources needed to successfully validate an IT delivery project. An example user test plan
accompanies this guide in a separate document, and a template for use by user testers is also available.
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,
Contents
1.
User Testing Environment ............................................................................................ 3
1.1
Overview ................................................................................................................. 3
1.2
Test Data ................................................................................................................ 5
1.3
Production Data Migration ...................................................................................... 5
1.4
Testing staff ............................................................................................................ 6
1.5 Types of test ............................................................................................................... 7
1.6 Test Environment ....................................................................................................... 7
2.
User Test Plans ............................................................................................................ 8
2.1
Test Phases ............................................................................................................ 8
2.2
Test Scenarios ........................................................................................................ 8
2.3
Test Cases ............................................................................................................. 8
2.4
Test Scripts ............................................................................................................. 9
2.5
Remedial Actions Log ............................................................................................. 9
Information Systems Services
Page 2 of 10
Version 1.0 Updated- DM
Document1
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,
1. User Testing Environment
1.1
Overview
Regardless of what type of project you are working on, you want to catch errors as early in
the lifecycle as possible. There are projects that purposefully blast through the Analysis,
Design and Coding Phases with a philosophy that they will fix all the problems in testing.
This usually leads to disastrous outcomes.
The test plan includes the expectations for testing that will occur during the entire project.
Developers should be responsible for unit testing only to ensure that their work is quality
work. Integrated testing and system testing should be done by the technical team
(sometimes developers are included in this group). The User Acceptance testing needs to
be completed by the users or customers. Their test scripts based on the system’s
requirements should be developed to ensure the system will meet user needs
The following diagram shows how the testing of the system fits in with the rest of the
development process. Notice that the system is being developed while the user
acceptance test plans are being written. This ensures that final testing is done
independently of those who have developed the system. Both the IT and User testing
components of the process are highlighted in yellow.
Quality checkpoints may be added to this process flow diagram to ensure that the test
plans and scripts are in sync with the software as it’s developed.
Information Systems Services
Page 3 of 10
Version 1.0 Updated- DM
Document1
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,
Users / customers
design Test
Phases, Scenarios
& Cases.
Unit Test
Integrated /
System Test
Information Systems Services
Page 4 of 10
Version 1.0 Updated- DM
Document1
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,
1.2
Test Data
Before testing can commence, it may be necessary to ensure a minimum set of data is
present in the test system. This may take the form of ‘reference’ or ‘transaction’ type data.
Reference data takes the form of information used repeatedly across the system such as
finance codes or members of staff. Transaction type data takes the form of one off pieces
of information such as individual sales orders. Typically, functions which produce
transaction data (such as creating an order) will require reference data before they can be
performed, whereas reporting functions will rely on both types of data to work effectively.
Once the logical test cases have been decided upon, the type of data required to support
the test can be identified. For example, if a test case is identified which requires a report to
be run to show sales figures for a period, it follows that sales figures must exist within the
reporting system in order for them to appear on the report. Identifying types of data
required to support the logical tests helps to confirm that the design matches the user’s
expectation in the same way as the definition of the test cases themselves does. For
example, stating that warehouse location data is required for the correct execution of a
‘create sales order’ test might set alarm bells ringing for the designer if they thought this
data was not used in the creation of a sales order.
Once the design of the system is complete, the logical test data needs to be refined into
actual defined data items in the same way as the logical Test Cases are refined into actual
user actions (listed in the Test Scripts). So for example, a logical data item might be ‘a
General Ledger (GL) code in the Office & Travel category’. This might be converted into an
actual data item of ‘GL code 54400 Photocopy Consumables’. Defining data items in this
way allows them to be re-used throughout the Test Scripts in a controlled way as the
predicted consequences of the tests take effect on the data. Using the example of the GL
code above, three test scripts might be planned:

Script to test reporting on Office & Travel costs

Script to test reporting on Consumable costs

Script to test a change the category of a GL code.
If the category change test were planned to take place between the two reporting tests,
using the 54400 GL code in all three tests would show the effect of the update test in the
execution of the final reporting test.
1.3
Production Data Migration
In addition to testing the functionality of the system being implemented, in some cases,
data will need to be migrated from an existing system into the new system. Tests will need
to be devised to ensure that this migration is done correctly. There are a number of test
which the IT department will run as a first check (for example a check that the number of
Information Systems Services
Page 5 of 10
Version 1.0 Updated- DM
Document1
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,
records in the new system matches that in the old system), but more complex validation
will need to be undertaken by end users. Often this will take the form of running reports in
the old and new systems and comparing the reports. Any discrepancies found could be
due to a number of reasons such as errors in migration, errors in the new reports, changes
in the way the reports represent the data, and so great care must be taken to ensure
comparisons are being made like for like. It is likely however, that testing of this type will
be dependant on the correct functioning of any new reports, and so the importance of
thorough testing of these reports prior to this stage should be clear.
1.4
Testing staff
The testing of a system implementation or change is an extremely important part of the
software development process. Unfortunately, for a system of any significant size, the
amount of user testing required is also likely to be quite extensive and time consuming.
However, since most staff have great demands on their time during their normal working
days, finding the time to commit to an extensive testing process can prove very
challenging. Further difficulties are caused by the very nature of this process in that it is
likely to be an iterative process with pauses in the testing being required as technical
rework is done to resolve faults as they are found.
All of this means that it is very important to identify staff who are willing to perform this
testing role as early as possible to ensure they are able to schedule time well in advance.
Furthermore, once this commitment is made, it is important for the testers to understand
the importance of starting their testing promptly, as identifying faults at the end of the test
period will cause delays as work will be required to remedy the faults and test once again.
Testing can be broken down into three types of activity - writing test plans, writing test
scripts and performing the tests
The difference between test plans and test scripts is explained in Section Two, but can be
summarised by saying that test plans represent what the system should do and test scripts
represent how the system should do it. This means that in theory, each of these activities
can be done by different groups of people and at different time:
Activity
When to do it
Who to do it
Writing test plans
During system design
Business staff familiar with the business
functions the system will affect.
Writing test scripts
During system build
Business staff familiar with how the system
has been designed to allow users to
perform the business functions.
Executing test scripts
After system build
End users who will be performing the
business functions – not necessarily
involved in the system design to date.
Information Systems Services
Page 6 of 10
Version 1.0 Updated- DM
Document1
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,
1.5 Types of test
Fundamentally, there are two types of test which should be considered: functional and
non-functional tests. Functional tests, as the name implies, check that the functionality
supplied to the user is fit for purpose as defined in the user requirements and design
documentation. Non-functional tests consider aspects beyond the basic ‘can a user do z, y
and z’ type tests, and instead focus on things of a more environmental nature. Examples
of these types of test could be internationalisation tests, recovery (from system failure)
tests, security tests, stress tests and endurance tests. It is quite possible that the User
Test staff are not easily able to perform these tests themselves, but it is important that they
satisfy themselves that these aspects have been tested. For example, a non-functional
test may be to check that a web system can handle 1,000 users accessing the system at
the same time. This may require specialist software to simulate which is not
available/useable by User Testers, and so specialist/technical staff may be required to
prove that this scenario can be handled by the system.
1.6 Test Environment
In most cases, a dedicated test system will be provided in which to perform the testing.
However, if the changes being tested are part of an existing ‘live’ system it is likely that
there are other developments planned for other areas of the system. Each of these
developments is likely to require testing and it may not be possible to do this
simultaneously as they may interfere with each other. For example, an error identified in
development A may require the change which introduced it to be removed. If development
B is built on top of development A, removing A may well affect the way B works. Care will
be taken therefore to schedule such changes so that they do not overlap but any delays in
the testing of one of these developments will therefore have a knock on effect on
subsequent developments. If the staff required for the subsequent developments have a
limited window in which to implement them, it is quite possible that they will become
unavailable and it will become impossible to implement the change.
Information Systems Services
Page 7 of 10
Version 1.0 Updated- DM
Document1
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,
2.
User Test Plans
2.1
Test Phases
The user acceptance test plan is divided into a series of phases which break the project
into more easily managed steps. These phases should be at a very high level to represent
the system functionality in a totally implementation independent way. In other words, the
technology used to solve the user’s problems does not need to be considered when
deciding on the test phases.
For example, if the system to be tested is intended to allow people to purchase goods, the
functions required of the system at a very high level might be:




2.2
Shopping on the system
Delivering the goods
Banking
Reporting
Test Scenarios
These Test Phases are then broken down into a series of Test Scenarios which emulate
typical usage of the system. So the ‘Shopping on the system’ phase of testing could
include the scenarios:



Can a user access the system
Can goods be setup onto the system ready for purchase
Can a user buy goods
It is important to note that both the Test Phases and Test Scenarios can be identified early
on in the development process. They describe what the system is intended to do, rather
than how it is intended to do it. This means that the people commissioning the system
should be able to define both of these elements of the test plan once the requirements
have been defined and so it is not necessary to wait for the system’s detailed design to be
completed.
2.3
Test Cases
The final step in defining the ‘logical’ tests required, is to define the details within the Test
Scenarios to ensure that every possible eventuality is considered. For example, it is not
simply enough to ensure that an authorised user can access the system, it is equally
important to ensure that unauthorised users cannot access the system.
Not only is the production of the User Test Cases an important part of the test process
itself, it is also an important way of affirming the design of the system is fully understood by
the developers. The people actually writing the code to implement the system are unlikely
Information Systems Services
Page 8 of 10
Version 1.0 Updated- DM
Document1
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,
to be the same people as the analysts who have developed a good understanding of the
business functions involved. Giving them a full and comprehensive list of things the system
should do will therefore act as a safety check to ensure they haven’t missed anything
when interpreting the system design.
2.4
Test Scripts
Once the system has been designed, and it is clear how the system will work to implement
the requirements, the Test Cases can be refined further. The User Test Scripts can now
be produced to detail the steps that a user of the system will need to go through in order to
achieve the goals laid out in the Test Cases. As this is system dependant, a detailed
System Design will be required to allow the level of detail necessary to be understood by
the Test Script author.
The Test Scripts should describe the precise steps that a user should go through in order
to test the element in question. This will enable the Test Scripts to be given to users who
have not been trained in the system. If this were not done, the user training would have to
be done before the system had been fully tested, running the risk of bugs manifesting
themselves during training. This could have a disastrous effect on end user’s perception of
the system, potentially leading to a low take up of the system and ultimate project failure.
During the execution of these Test Scripts the testers may identify areas where the system
is not performing according to its design (bugs), in which case the Test Script will be
marked as having failed, and an entry placed on the Remedial Actions Log for the
development team to investigate and attempt to resolve.
2.5
Remedial Actions Log
In addition to bugs identified during the testing as outlined in Section 2.4 above, the testers
may come up with things which would make the system better, but were not included in
the system specification. It may be that these are things which were considered during the
design but excluded for good reason, or things that were omitted but would genuinely
improve the way the system works. Examples of the first type (i.e. considered but
excluded) could be things that add a slight overhead to a manager, but result in a
significant effort saving for a large number of system users. In this case, the extra
managerial effort might have been considered worth the large saving of effort totalled
across all of the other users.
Whatever the type of suggested change, these should be recorded on the Remedial
Actions Log along with genuine system faults. The appropriate group within the
development team can then examine all of them together and consider what action to take
based on their severity and effort to resolve. Examples of remedial actions may be:

Do nothing – this is not actually a bug, but a misunderstanding the system design

Do nothing – this is a genuine bug, but the cost of repair outweighs the benefit
Information Systems Services
Page 9 of 10
Version 1.0 Updated- DM
Document1
User Testing Guidelines
This guide contains information on the best practice approach to user testing, explaining the approach, timelines and resources needed to
successfully validate an IT delivery project. An example user test plan accompanies this guide in a separate document,

Fix – genuine bug which needs fixing. Must be prioritised against other fixes

Change – not in original specification but considered worth doing. Must be
prioritised against other changes and fixes.

Retain – genuine bug but not critical enough to delay ‘go-live’. An action plan for
resolving it must be defined (e.g. fix during post go live support or second phase
development)

Retain – not in original specification but considered worth doing at a later date. May
become another project, or a later phase of the current one.
Information Systems Services
Page 10 of 10
Version 1.0 Updated- DM
Document1