Acceptance Testing Strategy Guided by
Evolutionary Operational Profiles
Authors: Maria de Fatima Mattiello-Francisco
Ana Maria Ambrosio
National Institute for Space Research
Edgar Toshiro Yano
Aeronautical Institute of Technology
Industry Track ISSRE2007
[email protected]
Motivation
Acceptance Process
GROUND SYSTEM CONCEPTION FOR SACI-1
Service Profiles
Test Strategy
Results
The target is the Payload Embedded
Software on board X-Ray Astronomy Satellite
Telecommand
Satellite Telemetry
&
Payload Telemetry
Confidence on critical software operation
takes into account:
System’s stakeholders satisfaction
Antenna Control
Principal Investigator
Transmitter/Receiver
Stakeholders involved in
Development & Operation Processes
TC
TM
INTERNET
Expected testing quality
TM
Quality is related Test Techniques
to Test Strategies Environmental facilities
which includes
Testing time duration
Data Base SERVER
Scientific Community
2
Motivation Acceptance Process
Mission Integration
Stages
Service Profiles
Test Strategy
Results
ECSS- European Cooperation
for Space Standardization
Instrument
Subsystem
System
CDR – Critical Design Review
QR – Qualification Review
AR – Acceptance Review
Motivation Acceptance Process
Service Profiles
Test Strategy
Results
Mission Integration Stages
The target software
is integrated with
other mission's
components in the
three mission stages
Instrument
(Payload)
Subsystem
System
(Satellite Platform) (Space & Ground Segment)
During each stage the payload software will be an encapsulated
component being validated under the different tester’s point of view
Acceptance Process
A formal testing to determine whether or not a systems satisfies
acceptance criteria based on the software problems reports resulting from
verification and validation activities [IEEE-Std1059-1993]
In our approach the acceptance criteria is based on the number of faults
revealed by the services execution during each integration stages.
Motivation Acceptance Process
Service Profiles
Test Strategy
Results
Testers involved in the Mission Integration Stages
Instrument
Payload
U1 U2 U3 U4
Subsystem
Satellite Platform
System
Space & Ground Segment
U2 U3 U4 U5
U2 U3 U4 U5 U6 U7
U1- Payload Software Engineer (focus on software requirement validation)
U2 – Payload Engineer (focus on interfaces performance)
U3 – Principal Investigator (Scientist focus on operational modes)
U4 – Software Engineer (Regression Test )
U5 – On Board Data Handling Computer Engineer (focus on communication)
U6 – Mission System Engineer
U7 – Ground Segment Engineer (Mission Center Operation)
Testers with different needs carry on the mission integration stages
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
Operational Profiles
External user-oriented test approach – specifies the intended usage of the
system in terms of operations and their occurrence probabilities [MUSA93]
Services
Space system operations are service-oriented. A set of on-board
performing functions is always related to one service purpose.
On-board embedded software is a component that collaborates on service
provider by executing services activities, being destination of service
requests and the source of service reports [ECSS-E-70-2003].
Operation problems can be anticipated by TESTERS performing
service-oriented test cases during the acceptance testing
6
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
The proposed Evolutionary Operational Profile
Aggregates to Musa’s approach the service operational profiles,
adding the concept of service and the usage of system architectural elements
Such new approach aims at supporting a test strategy for embedded space
software acceptance along the evolutionary
integration stages
Denotes the growth in complexity and autonomy of the
environment with which the embedded software interacts
as far the integration stages evolves.
At each integration stage the environmental facilities
are incrementally substituted by the real subsystems.
Software functionality does not change but service
operation does due to the test scripts evolution
7
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
SERVICE PROFILES
Is a concept which depends on software behavior in terms
of operational mode and software architecture
Services are performed within system modes by component
and their interfaces.
They are conceived taking into account the following steps
•
Identify software operational modes profile
• Identify software architectural profile
• List the SERVICES to be performed
• Establish relationship between element and services concerning
USAGE INTENSITY
• Calculate the service operational profiles
8
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
Mode profile
Is defined as the probability of occurrence of each operational mode
in the system use.
A service might be provided by more than one software operational
mode and its occurrence probabilities being different in each mode.
Software operations and services are performed within operational
modes and by architectural elements.
Case Study: MIRAX Payload Software SWPDC operational modes
9
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
Architectural Element profile
Case Study: MIRAX Payload Software - SWPDC Architecture
10
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
Architectural Element profile
Given a category of architectural element, one can attribute probabilities of
usage of ei in system operation lifetime as a whole.
The occurrence probability
estimation depends on high
knowledge of the application and
historical data from previous
experience in similar projects
SERVICE USAGE MATRIX
The binary cell
a ij indicates the
relative use of the
architectural
element by each
service
11
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
Service Profiles Calculation
Service profile is defined as the occurrence probability of each service
in terms of using a particular element.
Where weight wij represents the degree
of usage intensity and/or criticality of the
element e i on such service q j
MODE USAGE INTENSITY MATRIX
12
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
Test Strategy
Based on SERVICES PROFILES PSe, one selects the number of TEST
CASES Ntc provided by U testers per service in order to match the previous
total time T allocated for each mission integration stage
Ek
There is a set of services Si that exercise better
some elements Ek than other. Therefore,
priority on performing particular test cases is
recommended whenever reduction on testing
duration are imposed to the testers Uj
Uj
f ( S4,U2,E1) = ???
Si
The test strategy effectiveness will be demonstrated in
our case study with data from the mission integration
stage at Instrument level provided by the testers. All test
cases associated to the mentioned services were
executed in order to evaluate this approach. 13
Motivation
Ntc provided
by tester per
service
Acceptance Process
U1
Service Profiles
Instrument
U2
Test Strategy
U3
Results
U4
A
101
636
46
88
= 870
Considering that the average time of 2 test cases execution (including analyses) is
1 hour for group A, this stage will require 435 hours to be concluded.
If the available time is 220 hours (50%) only, our test strategy approach reduces
respectively the number of test cases as showed in group B:
B
Reduction
Percentage
72
72%
285
45%
29
63%
54
61%
= 440
60%
One observes that the reduced test set (60%) is not proportional to the
schedule constraints (50%). There is a gain in terms of test set to be performed.
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
Faults detected by A and B
A
B
U1
U2
U3
U4
147 failures
74 failures
A particular system failure can be detected by several Test Cases provided
by different testers. In such case study it was observed that 147 failures
detected by the test set A were caused by the following faults:
7 bad configuration on test execution
2 source code malfunction
7 document inconsistencies
The 74 failures detected by
the reduced test set B were
caused by the same faults
The results showed that redundancies in terms of test cases coverage are
given by the service-oriented test case generation. Such redundancy allows to
choose test case which covers intrinsically more than one service, without
losing test efficiency.
15
Motivation
Acceptance Process
Service Profiles
Test Strategy
Results
Questions???
Financial support to the QSEE Project
Quality in Space Application Embedded Software
http://www.cea.inpe.br/qsee/
16