National Research Center
«Kurchatov Institute»
Modern approach to software
verification and validation
for safety-critical systems
IX International Forum «ATOMEXPO 2017»
«Intellectual Automation and Electrical Engineering»
Authors: Kalinushkin A., Musihin A., Tsarev V.
Speaker: Tsarev V.
Moscow, June 21st 2017
Software Reliability
The link between software reliability and verification/validation activities
 Software reliability is defined as a “set of attributes related to the ability of a
computer program to maintain operation quality in the set conditions over a set
period of time”
Software attributes
Stability
Software reliability
Quantitative
characteristics
Reliability assessment
Qualitative
characteristics
Reliability assurance
Error tolerance
Recoverability
2
Software Reliability
The link between software reliability and verification/validation activities
Algorithms
Interpretation
media
Error
migration
Errors
3
Software Reliability
The link between software reliability and verification/validation activities
The complexity of quantitative software reliability analysis is explained by the following
factors:
 Absence of a correlation between the presence of errors in the software and its
reliability (an error present in the software does not necessarily lead to a failure);
 Unpredictable results of error correction;
 Impossible to provide 100% test coverage;
 Very complex classification of failures into software, software-hardware and
hardware failures;
 The existing approaches to software reliability quantitative assessment are based on
building software models, but these models are actually an extrapolation of the
testing process to the real operating environment and are riddled with a number of
serious application restrictions, which reduces the usefulness and accuracy of the
resulting data;
 There isn’t a single universally accepted quantitative measure of the software
reliability.
4
Software Reliability
The link between software reliability and verification/validation activities
Conclusions
 A computer program cannot be proven absolutely fault-free in theory, and
therefore 100% freedom from failures is not feasible.
 It is not currently possible to perform quantitative assessment of software
reliability, so quality assurance efforts must be concentrated on the development,
testing and verification stage.
 In absence of an objective ability to assess software reliability and assign a
recommended quantitative failure probability, a justification of software quality
can be achieved through testing and verification.
5
Software Verification
Definitions. IAEA glossary on safety issues. Terminology Used in Nuclear Safety and Radiation Protection. IAEA, Vienna, 2007.
Verification
The process of determining whether the quality or features of a product or
service meet the prescribed, predefined or required criteria. Verification is
closely related to quality assurance and quality control.
Validation
Confirmation, based on objective evidence, that the requirements developed
for specific goal and use or application have been met.
6
Software Verification
Definitions. Summary
• Software verification is a process for determining whether a software
product meets the requirements imposed on it at all lifecycle stages of the
software system being designed.
• An additional task of verification is to localize the errors introduced during
development or modification of the software product.
• Verification is an integral part of the lifecycle of a software product; it is
important to note that verification objectives also include control of the
results.
• Quality assurance issues are closely related to the verification and validation
processes, and considering them in separation is not appropriate.
7
Regulatory Documents
Russian norms and rules in the area of nuclear energy
NP-001-15. General provisions on ensuring safety of nuclear stations
3.1.16. If a safety-critical system is implemented using programmable digital devices,
the respective norms, rules and methods shall be established and applied for
development, testing and verification of the programmable digital devices and
software throughout the system lifecycle, particularly during the software
development phase. All development must be subjected to the quality assurance
system.
NP-026-16. Requirements to control systems important for safety of nuclear stations
14. With regards to operating results, verification must be conducted at all lifecycle
stages of a safety-critical control system. All inconsistencies identified during the
verification shall be documented and eliminated.
8
Regulatory Documents
Basic standards
• The following standards are used as basic standards when conducting
software verification:
• IEC 60880 “NUCLEAR POWER STATIONS. Safety-critical control and
management systems. Software for computer systems performing
Category A functions”;
• IEC 62138 “NUCLEAR POWER STATIONS. Safety-critical control and
management systems. Software for computer systems performing
Category B and C functions”
• Despite the high detail level of software verification activities described in
IEC 60880 and IEC 62138 standards, it was decided to modify the standards
using IAEA requirements and a number of international standards described
below.
9
Regulatory Documents
IAEA documents
• SSG-39. Design of Instrumentation and Control Systems for Nuclear Power
Plants
• SSG-2. Specific Safety Guide. Deterministic Safety Analysis for Nuclear Power
Plants
• TECHNICAL REPORTS SERIES 384. Verification and Validation of Software
Related to Nuclear Power Plant Instrumentation and Control.
10
Regulatory Documents
Additions to basic standards
• Recommendations on software verification are presented in Item 8 and
Annex E4 of IEC 60880.
• The issues of verification are also considered in detail in IEC 60880;
nevertheless, we believe it is advisable to complement IEC 60880 provisions
with IEEE 1012 recommendations, “System and Software Verification and
Validation”, for the following reasons:
• IEEE 1012 standard establishes the form and the content of verification
plans and reports;
• It also contains instructions on choosing verification methods to ensure
the verification is as comprehensive as possible;
• It also contains instructions on testing.
11
Verification Process
Generalized structure as per IEC 60880.
Technical assignment
SW requirements development
V
Software requirements
Software design development
V
Software design
Evaluating requirements
for completeness and correctness
Evaluating compliance
with the technical assignment
Evaluating design
for completeness and correctness
Evaluating compliance
of the software requirements
Evaluating the code
for completeness and correctness
Software coding
Evaluating compliance
with the software design
V
Software modules code
Testing
Software integration
Integrated software
V
Evaluating testing
completeness
Evaluating compliance with the software
requirements and design
Integration
testing
Evaluating testing
completeness
T
R
A
C
E
A
B
I
L
I
T
Y
12
Verification Process
Generalized structure as per IEC 60880, Items 8.1.8, 12-13.
Requirements
Regulatory documents
Input data
N lifecycle process
Verification
Negative
outcome
Output data
Positive
outcome
N+1 lifecycle process
13
Verification Activities
Developing a verification plan
Developing a verification plan includes determining the list of activities
required for performing verification and respective procedures,
methods, techniques and facilities to determine whether the software
product meets the requirements imposed at various stages of the
software system development.
14
Verification Activities
Correlation between verification plan sections and regulatory requirements
Document section
Definitions and abbreviations
Verification
overview
Critical analysis
Instruments, techniques and
methods
Verification Activities
Requirements to testing documentation
Regulatory document
IAEA Glossary
IEC 61508-4
IEC 12207
IEEE 1471
NP-001-15
IEC 61508-1
IEC 61226
IEEE 1012
IAEA TECHNICAL REPORTS SERIES 384
IEEE 1012
SSG-2
SSG-39
IEC 60880
IEC 62138
IAEA TECHNICAL REPORTS SERIES 384
IEEE 829-1998
15
Verification Activities
Critical analysis of the verification subject
Regulatory
document
NP-001-15
IEC 61226
Description
Elements of safety systems where isolated failures in case of a design basis accident result in a violation of
design basis limits established for those design basis accidents
Element of normal operation
Control element of a safety system
a) Functions required to meet safe controlled state with the aim of preventing design basis accidents developing
to unintended consequences or to mitigate the consequences of such accidents;
с) Functions required to provide information and control abilities to undertake certain non-automated actions
with the aim of achieving safe controlled state.
Regulatory
document
IEC 61508-1
Regulatory
document
IEEE 1012
Conclusion
Description
Low request intensity operating mode
Description
Failure to perform the functions partially, or substantial financial and social losses.
Class 2NU
Category А
Average
probability of
function failing to
respond to
request
Safety integrity
level
10-4 – 10-3
3
Event probability
Software integrity
level
SIL 3
4
16
Verification Activities
Verification techniques
Purpose
Purpose
Purpose
Expert review
The purpose of this technique is to find violations and errors, mainly of predetermined
types, in the software documentation. As a rule, expert review is used for documents
containing project details, code, testing scenarios or testing results.
Walkthrough
The purpose of this technique is to check completeness and consistency of the documents
developed as a result of project activities (e.g., technical design), or requiring high
justification levels (e.g., testing plans); and also to identify errors at all stages of software
development. The method must be adapted to documentation developed by the group.
Traceability analysis
This technique provides assurance that input requirements to the performance indicators
were met accurately by verifying their traceability in the output documentation; and also
to ensure that all critical requirements were determined and complied with throughout the
development lifecycle, i.e. from requirements analysis to final testing.
17
Verification Activities
Assigning roles
Role
Verification group leader
Primary document author
Secondary document author
Reviewer(s)
Tasks
Preparing and conducting the assessment,
holding meetings, assigning deadlines for
performance of works, recording defects
identified, control over readiness of input
data for assessment and correction of errors
found after the assessment.
Presenting basic provisions of the primary
document and answering evaluation team
members’ questions in this regard.
Presenting to the inspection members key
ideas underlying the author’s interpretation
of the primary document, answering any
questions related to the secondary document.
Analysis of the secondary document for
compliance with the primary document with
the aim of identifying any inconsistencies.
18
Verification Activities
Evaluation process
Expert review
Activity type
Planning
Roles involved
Verification
group leader
Content
Evaluating the readiness of the primary and secondary document for evaluation using
the following criteria:

Documents have been developed;

The content is presented in a clear format;

Sufficient detail level of the documents.
Identifying evaluation participants and their roles.
Identifying interactions between the document authors and reviewers.
Preparation
Documentation
review
Modification
Follow-up
Verification
group leader
List of documents as per Table 4, and checklist of questions for inspection are
distributed to the evaluation group members to work with.
Verification
group leader
Organizing interaction between the author(s) of the secondary document and
reviewers.
Preparing a report in accordance with the checklist of inspection questions.
Preparing an error report.
Modifying the secondary document on the basis of document review procedure,
using the error report.
Reviewer(s)
Secondary
document author
Verification
group leader
Preparing answers to verification group members’ comments.
Evaluating the modification results, to make sure all errors found were corrected and
no new errors were introduced.
Making a decision on subsequent review of the documents.
19
Verification Activities
Evaluation process
Walkthrough
Activity type
Planning
Preparation
Roles involved
Content
Evaluating the readiness of the primary and secondary document for evaluation using the following
criteria:

Documents have been developed;
The content is presented in a clear format;
Verification group leader 

Sufficient detail level of the documents.
Identifying evaluation participants and their roles.
Scheduling meetings.
Verification group leader List of documents as per “List of documents by stages” table for the respective stage, and checklist of
questions for inspection are distributed to the evaluation group members to work with.
Identifying the goals of the evaluation, key questions and issues reviewers should pay attention to.
Recording any inconsistencies between the primary and the secondary document, describing errors, their
Verification group leader localization and classification.
Preparing a report in accordance with the checklist of inspection questions.
Joint evaluation (in
Primary document
the form of a
author
meeting)
Modification
Follow-up
Preparing an error report.
Presenting the most substantial provisions of the primary document and answering participants’
questions in this regard.
Secondary document
author
Presenting key ideas and techniques used in the secondary document and explains why certain solutions
were chosen over others and how the provisions of the primary document were implemented; answering
participants’ questions.
Reviewer(s)
Analysis of the documents in accordance with the inspection checklist, taking part in the discussion,
preparing recommendations on eliminating errors if found.
Secondary document
author
Modifying the secondary document on the basis of joint evaluation results, using the error report.
Evaluating the modification results, to make sure all errors found were corrected and no new errors were
Verification group leader introduced.
Making a decision on subsequent joint evaluation of the documents.
20
Verification Process
Software requirements verification
Documentation for this stage
Software requirements specification verification
Input documents
Software requirements specifications
Interface requirements specifications
Procedure documentation for software configuration management
Checked for compliance with
Software/hardware requirements specification
Quality plans
Applicable standards
Output documents
Software requirements verification report
Configuration management analysis report
Traceability analysis report
Error report
21
Verification Process
Software requirements verification
Tasks
Configuration management
process analysis
Software requirements
specification analysis
Techniques
Expert review
ID
A
B
C
•
◊
◊
4
•
•
•
5
•
•
•
6
•
◊
◊
7
Walkthrough
Interface requirements
specification analysis
Traceability analysis
Category
Traceability analysis
22
Verification Process
Software requirements verification
23
Verification Process
Software requirements verification
Output document forms for this stage
Document
Verification report form
Error report form
Developers feedback response form
Traceability analysis report form
Inspection questions checklist
Annex
reference
А 1.1
А 1.2
А 1.3
А 1.4
А 1.6
24
Thank you for attention!
25