Insight Consulting Ltd.
http://www.insight.ie
Risk-based testing – a common language for project stakeholders
We have probably all heard the term ‘black box testing’. This means that test design is based on the
required external behaviour of a system rather than knowledge of its internals. Unfortunately, in many
projects, the term could be better applied to how testing is perceived by stakeholders such as project
managers, developers and customers! Testing activities are often just a ‘black box’ into which we put
the software and out of which comes ‘better’ tested software (and some form of sign-off from the test
or ‘quality’ group that we get after applying an appropriate amount of pressure).
There are a number of consequences of this situation:
 We may not be getting the best return from our testing efforts. This is because there is little
influence and communication from project stakeholders to help ensure that there is effective
testing focused on important issues.
 There is poor visibility and reporting on the progress of testing in a manner understandable to all
project stakeholders
 The decision about when to stop testing and release the product is based on a subjective sign-off
by the test group1 whereby they take on the responsibility. This is often highly inappropriate as the
decision to release is really a business decision and testing is frequently ‘squeezed’ anyway.
 In general there is a lack of appreciation of the value-add of testing. This has significant
implications in terms of not utilising testing resources effectively as their role is misunderstood.
Even more significantly the morale of testers is impacted often resulting in high attrition rates.
So how does software process improvement help us solve these ‘project challenges’? There are some
useful elements in the CMM, and more so in the CMMI, that identify practices that relate to planning,
verification/validation strategies and risk management. However, one still needs to apply these in a
practical manner to really solve these project challenges. One must make a stronger link between risks
and testing to avail of the benefits of ‘risk-based testing’.
Risk-based testing typically involves:
 Identifying and analysing product risks that can be addressed by testing. This is best done in
collaboration with customers/users who can provide business risks and developers who can
provide system/technical risks. Examples of business risks include critical functions/features that
the users need to do their job. System/technical risks could include core system functions,
performance, security or other issues that are critical from a system operational viewpoint.
 Developing a testing strategy that can mitigate these prioritised risks. This may involve assigning
critical features to be tested in particular stages or iterations of testing (ranging from static testing
such as peer reviews to dynamic testing such as functional system testing).
 Designing tests within each test stage that extensively check the allocated high risk elements with
less testing of lower risk elements. The result is a prioritised set of test cases agreed by project
stakeholders to address the most important product risks.
 Executing the tests in order of priority.
 Reporting progress on the basis of risks addressed and residual risks remaining. This could also be
expressed in terms of the benefits and product objectives that become available as the risks
blocking them are addressed by testing.
Testing therefore provides information to make a risk-based decision about release – a decision that
should be made when the benefits available outweigh the remaining risks.
Involving project stakeholders in influencing testing gains understanding, support and provides
valuable advice for the testing effort. Testing focuses on finding faults in parts of the system where the
impact and likelihood are greatest. Lower risk areas receive less testing or indeed may be deemed by
project stakeholders to be out of scope for testing. Any released faults in these parts of the system are
by definition less likely and of lower consequence. If testing gets squeezed we have at least addressed
the most critical risks and significantly the decision to release is now based on information relevant to
project stakeholders i.e. benefits available and residual risks. The role of testing is to provide this
information.
Indeed the test group is often given a title containing the word ‘quality’ indicating their responsibility
for product quality!
1
 2002 Insight Consulting Ltd.