Safety Critical Systems
and
Certification Issues
DO-178B
Airborne Standard
SC190 / WG-52 Application
Guidelines For RTCA
DO-178b/ED-12b
RTCA
SC-167
EUROCAE
WG-12
CNS/ATM
Community
SC-190 / WG-52
Cast Position
Papers
CAST
(Certification
Authority
Software
Team)
Avionics
Industry
SC-190 Products
3
DO-178B in a Nutshell
Highly Process Oriented
Requires Traceability
 Requirements
 High
Level Design
 Detailed Design
 Source Code
 Test Procedures
 Test results
Test & Test & Test & Test and …..
4
DO-178B Safety Levels
Level A

Catastrophic

Failure Prevents Continued Safe Flight and Landing
Level B

Hazardous/Severe-Major

Potential Fatal Injuries to a Small Number of Occupants
Level C

Major

Discomfort to Occupants or Possible Injuries
Level D

Minor

5
Increased Crew Workload
Development Team
What is COTS?
How can objectives of DO-178B be
satisfied, using COTS?
There is much variation in applicants for
COTS certification credit
Is DO-178B clear on the interpretations?
Product is COTS
+
Certification Evidence
Available as COTS
Together these satisfy safety objectives
6
Testing without
Source Code
Wrappers to
Validate Parameters
Application
Commercial O/S
This cannot be trusted
unless O/S is verified
7
Special Considerations
Use of Service History for Certification
Air Traffic Control system
Worked Correctly in US for years
Transferred to U.K.
Actual Plane 2
Plane 1
Displayed Plane 2
Greenwich Meridian
8
Use of Service History
Developed under a less stringent standard (Military?)
Used for 4 years
Safety
Critical
System
Problems tracked
Quality Good!
Residual Error
Dead Code(Unintended Function)
Is this system safe for the next 4 years? At Level A, B, C?
We can bound inputs, but we cannot check internal states
without “looking inside”
9
Black Box Testing
No single failure should prevent
“Continuous safe flight and landing.”
Statistical testing cannot show absence
of a single state that will cause a
failure
Software has discontinuities
There is no foundation for statistical reasoning
about software faults or safety
Software does not follow
Gaus/Normal Distribution
10
Verification Team
examples of issues
 What
are low level requirements? How can
they be tested
 Data and Control flow coupling
 Use of higher level test results for lower
level requirements
 What is the intent of structural coverage?
 Traceability of source to object code for
structural coverage
 What is statement, decision, condition and
MCDC coverage testing (Modified Condition/Decision Code)
 Verification tool qualification
11
 etc..
Coverage Analysis
at Level B and C
Statement Coverage
Decision Coverage
 Entry
Points
 Exit Points
 All Decisions
 All Outcomes
12
Fixing anomalies
example for level B/Library
Filter
Compiler
Object Code
B := 3;
A := Filter (B);
X := X + A;
Source level coverage required Even for Filter
13
Boundary Level testing
not enough!
A := B; -- A and B are packed Boolean arrays
Run-time call to:
Bit_Block_Move (From, To, Size); -- size in bits
Min, Mid, Max in combination gives
27 (useless) test cases
5
Bits size
 32 Bit size
 67 Bit Size
 From
Interesting test cases based on
actual code
i.e. White Box Testing
overlaps To
 To overlaps From
14
Coverage at Level A
 Coverage
required at Machine Code level or
 Show source to object code traceability
and test at source level or
 Use different language/compilers
and use voting system
 MCDC testing required
 each
condition must have effect on outcome
 Tools
which modify source for traceability
problem at level A
 Mitigation
method : use 3 different compilers
15
(Now Look At Conditional Statements)
Conditions/Decisions
Conditions
if A=B and C or D<3 then
Decision
Boolean Variable
Boolean Operators
16
What are
Conditions/Decisions
Ada :
if (A=B and C) or D<3 then
C:
if ((A==B) & C ) | (D<3) then
C:
if ((A==B) * C) + (D<3) then
C++ :
if ((A==B) and C) or (D<3) then
MCDC Coverage Requires all Branches
AND all Conditions Be Covered
17
More Boolean
Conditions
Ada :
X := (A=B and C) or D<3;
if X then -- X is Boolean
Cannot hide from
Testing Obligations
C:
X = ((A==B) * C) + (D<3));
if X then /* X can be any Integer
‘*’ and ‘+’ are Boolean Operators!
18
Condition coverage
Ada :
X := (A=B and C) or D<3;
if X then -- X is Boolean
Coverage of “Basic-Block”
may not capture condition
results
19
Avoiding MCDC Testing
Modified Condition/Decision Code
Use Ada’s short-circuit conditions:
if A=0 and then B< 2 and then C>5 then
Or in C write:
if A== 0 && B < 2 && C < 5 {
20
Why short-circuit
conditions eliminate
MCDC
if A=0 and then B< 2 and then C>5 then
if A=0 then
if B<2 then
if C>5 then
P;
end if;
end if;
end if;
MCDC not required for this code
21
Testing strategy
must evaluate conditions
if A=0 and then B< 2 and then C>5 then
if A=0 then
if B<2 then
if C>5 then
P;
end if;
end if;
end if;
BUT !!!
Testing must show that
each ‘then’ part has
been tested True and False
MCDC not required for this code
22
Inlining code
 If
decisions/conditions introduced
 Decisions
must be identified and verified (level B)
 Conditions must be identified and verified (level
A)
 Verification
may be done by analysis
 Traced
to derived requirements
 ensure safety is not compromised
 Code may be “deactivated”
 As
inlined code depends on local state it may
be very hard to test the conditions in
accordance with standards requirements
 Intent - absence of unintended funtion
23
 Dead code not allowed
Use of Tools
 Tool
Qualification is required if tool replaces
a step of development process
 Development tools
 Examples
- Compiler, Design to code generator
 May introduce an error
 In general - NOT qualified, not trusted
 Verification
tools
 Examples
- Coverage analyser
 May conceal an error
 May be qualified - Trusted for verification
purposes
 Additional
verification process required if
the tool is not trusted24
CNS/ATM Process Integration
Information matrix
 Regulators
 Committees
 Standards
Bodies
 Standard
Software Integrity Assurance
Standards vs. Software Development
Standards
Relationships between DO-178B
and IEC 61508 25
Ground Based
Community
 Communications/
Navigation/Surveilance
and Air Traffic Management - in the loop
 Ground Based Systems affect airborne
software
 DO-178B addresses airborne only
 Guidance being prepared to encompass
needs of CNS/ATM community (SC-190
committee)
 Standards tightening up
26
Certification Standards
are Improving
 “Holes”
in documents are being fixed
 Understanding of Certification Requirements
is spreading
 Industry and Certification Authorities are
collaborating on Guidance materials
 It will get more difficult to “shop around” for
a more lenient signature
27
( Legal)
Safety Systems
Laws
Regulations
Standards
Guidelines
PROCESS
Case Law
Precedence
Interpretations
Standards
Guidelines
Traceability
Visibility
EVIDENCE / RECORD
Confidence / Safety
28
When Is Software Safe
29
When Is Software Safe
We Don’t Know !!
30
What is our best guess
about the safety
 When
applicable processes have been
followed
 When we have verified the code “from
within”
 When this has been checked
and checked
and checked
and checked
and checked
and checked
31
The FAA/JAA Rules
are Strict
To Date:
“no fatalities have been attributed to
Software Failure”
Have we been lucky?
Have we been safe?
32