Table of Contents
ABSTRACT
4
1
5
1.1
1.2
2
INTRODUCTION
IFT BASICS
AMOS SATELLITES FAMILY
THE FUNCTIONAL TESTS OPTIMIZATION (FTO) APPROACH
5
7
8
2.1 INTRODUCTION
2.2 FTO MATHEMATICAL APPROACH
2.2.1 MST ALGORITHM
2.2.2 WEIGHTED-DFS ALGORITHM
2.2.3 FTO COMPLEXITY
2.3 FTO WORKFLOW
2.3.1 MODELING SYSTEM'S BEHAVIOR
2.3.2 DETERMINING WEIGHTS FOR STATE TRANSITIONS
8
10
10
13
14
14
14
14
3
17
3.1
3.2
4
FTO VALIDATION
GENERAL
AMOS-3 FTO PROCESS VALIDATION
REFERENCES
17
17
17
Abstract
In today’s Space industry the role of the verification and validation (V&V) process has taken
center stage. This is primarily due to the fact that in today’s competitive market where
projects are forced to cut down time to market, the systems engineer’s job of fully verifying
and validating the system has become an almost “mission impossible”. A previous paper
introduced a novel V&V plan, such that will enable the system’s engineer to fully validate
and verify the system while taking into consideration his minimal and dwindling resources.
This approach has it’s roots in what is known in the industry as the IFT (Integrated
Functional Testing), where the functionality of the fully integrated system is tested. The
testing is performed against a testing matrix which is a combination of the requirements
matrix and the system’s software capabilities. The IFT approach was tested on a variety of
GEO and LEO satellites including ”AMOS-1”, ”AMOS-2”, ”OFEQ-5”and ”TECSAR”.
Despite of its many advantages the IFT approach still has a challenge of choosing the proper
scenarios to be simulated. It was demonstrated that a successful implementation of IFT
depends on close familiarity with operating methods of the system both at the user level and at
the engineering level. It was found that an insufficient familiarity with the satellite systems and
operational logic might result in choosing a wrong set of scenarios, thus reducing the
confidence level of satellite’s functionality.
The FTO confronts this issue by implementing models and tools from a mathematical branch
of ’graph theory’. By mapping the scenarios selection problem into two dimensional graph
domain containing nodes and edges, it is possible to mathematically ensure the selection of the
minimal set of scenarios that will cover all system states and pass through the most critical
transitions between these states. Furthermore, the FTO approach enables the deterministic
calculation of the confidence level of a selected set of scenarios that will be used in Satellite’s
V&V. This calculation enables a simple tradeoff where the number of tested scenarios can be
adjusted in a way that will optimally fit the project constraints (time and money) while
immediately reflecting the changes into the confidence score of the functional tests process.
FTO ensures that the minimal set of scenarios will cover all the system states using the most
important transitions. It is fully integrated into some promising system verification
approaches such as IFT (Integrated Functional Tests).
The implementation of FTO algorithm as a software tool will automate the test and
verification process by shortening test periods, cutting testing costs and producing an
objective measure of system functionality confidence.
1 Introduction
1.1 IFT BASICS
Quis Custodiet Ipsos Custodes ??
A jet airplane is checked by the test pilots,
and not the engineers that built it.
Who should check the satellite???
Integrated Functional Tests (IFT) is a test approach where the functionality of the fully
integrated system is tested. The testing is performed against a testing matrix which is a
combination of the system requirements matrix and the system software (space and ground
segments) capabilities.
The main purpose of IFT is to verify the implementation of the system requirements for the
integrated system - hardware and software (End-to-End). IFT implements two powerful rules:
• The entire functionality of the system is implemented
in and performed by the software
• It is possible to find a set of Eigen Scenarios that will
spread all the scenarios space
IFT is based on dividing the scenarios into four groups, according to scenario logic which is
based upon the system’s logic:
a. Nominal scenarios are designed to verify the logic with the nominal sequence of
events. These scenarios cover the operation of the system from pre-launch to service.
b. Fault scenarios are designed to check the logic response to faults, that are designed to
be dealt with autonomously, hence do not affect the system’s performance.
c. Survivability scenarios are designed to check the logic response to faults that degrade
system’s performance and only intervention by the operator can solve.
d. Pinpoint scenarios are defined to perform functional checks that cannot be tested in
the course of a normal test scenario.
Three main advantages were discussed:
• IFT approach minimized risk to the Satellite and satellite’s units.
• The IFT by-product is ‘End-to-End’ verification and validation, including Ground
segment, Space segment and the interfaces between the segments.
• IFT gives the system engineer high confidence level of system’s readiness, due to
comprehensive coverage of system elements with a small number of tests (about 30
tests for more than 1000 elements).
Yet, there were still challenges that had to be addressed:
1. Successful implementation depends on close familiarity with operating
methods of the system.
a. User level (user in the loop)
b. Engineering level
2. Choosing a wrong set of Eigen scenarios, may cause major glitches in the
coverage.
1.2 AMOS SATELLITES FAMILY
IAI\MBT – SPACE satellites are successfully proven systems, based on 20 years experience
in space industry and a 100% success of in orbit missions (LEO and GEO), operating to the
satisfaction of all customers.
AMOS family satellites are small communication satellites developed and operated by
IAI\MBT – Space Division.
The purpose of AMOS satellites is to provide telecommunication and broadcasting services
in several coverage areas in the Middle East, Europe and the east cost of the US.
The satellites operate during sunlight and during eclipse without performance degradation.
AMOS satellites are co-located at 4°W in the Geostationary Orbit and enables "Hotspot"
capabilities for this slot.
In about three years from now, AMOS-3 will be placed in collocation with AMOS-1 and
AMOS-2, at 4ºW. AMOS-3 will be the first Israeli communication satellite that will be
inserted directly to orbit (direct to GEO).
AMOS-3 will provide, beside of the regular Ku-band communication capabilities in the
Middle East, Europe and the east cost of the US, an additional Ka-band and Steerable beams,
both in Ku and KA bands.
AMOS 2
1,350
160
1,800
1,374
650
11/14
75W
Ku
72
3 (Shaped)
AMOS 1
800
100
1,200
996
471
7/9
34W
Ku
72
2
Orbital Slot
Ownership
Launcher
AMOS 3
1,800
250
2,400
1,270
810
13 Ku + 2Ka
95W + 105W
Ku + Ka
72 Ku, 500 Ka
3 Ku +1 Sku
1Ka + 1SKa
4ºWest
SPACECOM
ZENIT 3SLB
4ºWest
SPACECOM
SOYUZ
Launch Date
End Of 2007
27/12/2003
4ºWest
IAI
ARIANE
44L
16/05/1996
Payload Power [Watt]
Payload Weight [Kg]
Solar Array Power [Watt]
Launch Mass [Kg]
Dry Mass [kg]
No. of Xponders
TWT Power
RF Band
Xponder BW [MHz]
No. of Beams
Table 1- 1 AMOS Family Evolution
2 The Functional Tests Optimization (FTO) Approach
2.1 INTRODUCTION
Functional Tests Optimization (FTO) is a suggested methodology that will allow system
engineers to get a recommendation for an optimized set of scenarios which gives a maximum
coverage of the system with minimum effort.
All complex systems have numerous "states" which are distinguished from each other by the
values of a set of properties or attributes. A typical state is strongly correlated to a certain
"path" namely by the sequence of states that the system encountered before reaching it.
The following graph represents a simplified graphic representation of satellite states (graph
nodes), transitions (edges) and scenarios (possible paths between any set of nodes using
transitions).
Course Cruise
5
2
1
5
3
GEO Fine
4
GEO Emergency
5
1
3
1
SKM
3
TNM
Figure 2-1. Simple State Machine
The figures include five states (Course Cruise, GEO Emergency, GEO Fine, SKM and TNM)
and 11 transitions. Each transition (edge) is directed and weighed. The edges are directed
since, naturally, transitions are not a symmetric process. The weight of each weight
represents the expectancy of the transition and will be explained in details in section 2.2.2.
For practical reasons it is convenient to grade the edges with highest expectancy by the
lowest weight and the edges with lowest expectancy with the highest weight. Thus the
graphic representation of the Simple State Machine is re-calculated by subtracting each edge's
weight from the highest number and adding one to the result (in order to prevent zero
weights). The resulting graph is:
Course Cruise
1
4
5
1
3
GEO Fine
2
GEO Emergency
1
5
3
5
SKM
3
TNM
Figure 2-2. State Machine after weight adjustment
A more detailed example, represented Amos-3 satellite state machine is presented at section
3.2.
2.2 FTO MATHEMATICAL APPROACH
The mathematical approach is based on two graph algorithms: Maximum Spanning Trees
(MST) and Depth First Search (DFS) over weighted directed graphs. These graphs are
comprised of nodes which are the various possible states and weighted edges which are the
transitions between states.
We intend to find a minimal set of transitions that will simulate the most important scenarios
while passing through all the system states. First we separate the problem into two different
sub problems:
1. Find the minimal set of edges that must be tested during IFT.
2. Find minimal set of paths that go through the edges found in sub problem 1.
The first sub problem is addressed using MST algorithm. Since MST gets an undirected
graph as an input the graph must be converted to an undirected format before the MST
algorithm is applied. For more details see section 2.2.1.
The second sub problem is addressed using a standard DFS algorithm. For more details see
section 2.2.2
2.2.1 MST ALGORITHM
During this research the MST algorithm that we chose is PRIM's algorithm which is:
1. Select Smallest Edge. Add it’s nodes to the tree.
2. Select one edge with smallest weight that connects the tree it to an outside node.
3. Repeat step 2 until all nodes are in the tree.
As explained the graph must first be converted into an undirected format. This is done by
multiplying each node and separating the incoming edges from the outgoing ones. The
following figure demonstrates the transformation when performed on the system shown in
figure 2.2.
Course Cruise
IN
1
GEO Fine
IN
2
GEO Emergency
IN
Course Cruise
OUT
1
GEO Fine
OUT
4
1
GEO Emergency
OUT
3
3
5
5
SKM
IN
5
SKM
OUT
3
TNM
IN
TNM
OUT
Figure 2-3. State Machine after separation of nodes to IN and OUT pairs
In figure 2-3 the direction of each edge is clear even without the arrow head. Thus, writing
the graph without the arrows does not cause any information loss.
Course Cruise
IN
1
GEO Fine
IN
GEO Emergency
IN
1
GEO Fine
OUT
4
2
Course Cruise
OUT
1
GEO Emergency
OUT
3
3
5
5
5
SKM
IN
SKM
OUT
3
TNM
IN
TNM
OUT
Figure 2-4. State Machine after erasing all arrow heads
After erasing all arrow heads, the MST algorithm can be performed over the graph presented
in figure 2-4. Note that since we converted all expectancies values the MST algorithm will
choose the minimal set of edges that had the highest expectancies in figure 2-1.
The following figure demonstrates the edges selected by the MST algorithm.
Course Cruise
IN
1
GEO Fine
IN
2
GEO Emergency
IN
Course Cruise
OUT
1
GEO Fine
OUT
4
1
GEO Emergency
OUT
3
3
5
5
SKM
IN
5
SKM
OUT
3
TNM
IN
Figure 2-5. The bold edges were selected by the MST algorithm
TNM
OUT
After choosing the edges that should be tested during IFT the graph can return to ot's original
representation after removing the edges not in the MST.
Course Cruise
5
2
5
3
GEO Fine
4
GEO Emergency
5
3
1
SKM
3
TNM
Figure 2-6. Original State Machine after removal of redundant edges
It is possible to calculate the Confidence of the IFT by dividing the weight of the remaining
edges by the total weight of the graph edges as shown in figure 2-1.
∑ Weights 29
edges∈MST
=
= 93.5%
Confidence =
Weights
31
∑
all edges
2.2.2 Weighted-DFS Algorithm
The DFS is an algorithm for traversing a graph. The algorithm starts at one node and explores
as far as possible along each path before backtracking. The current application uses are two
modifications to the basic DFS algorithm.
1. Since the graph discussed here is weighed, when the algorithm reaches a certain node
which has more than a single edge going out of it, it chooses the edge with the highest
weight to follow.
2. The last edge of a tree is allowed to create a loop.
The following figure demonstrates the paths (scenarios) selected by the revised DFS for the
graph presented in figure 2-6.
Course Cruise
5
5
2
3
GEO Fine
4
GEO Emergency
5
3
1
SKM
3
TNM
Figure 2-7. Scenarios Selected by the Weighed DFS algorithm
By following each path the following scenarios were chosen for further testing:
1. Course Cruise → GEO Emergency → GEO Fine → SKM → TNM → GEO Fine
2.
GEO Fine → GEO Emergency → TNM
3.
GEO Emergency → Course Cruise
4. Course Cruise → GEO Fine
2.2.3 FTO COMPLEXITY
The Mathematical representation of the complexity of FTO can be written as:
O(FTO ) = O(MST ) + O(DFS ) = O(E ⋅ log V + V + E )
Where:
E – Number of Edges
V – number or nodes
2.3 FTO WORKFLOW
The following passage describes the workflow needed in order to implement the FTO
algorithm:
1. Model the System's behavior by building a state machine.
2. Determine the weights of each transition.
3. Select transition needed for simulation (using MST)
4. Build scenarios comprised of transitions (Using weighed-DFS)
2.3.1 MODELING SYSTEM'S BEHAVIOR
There are many ways of modeling system's behavior. The methodology chosen by us was the
Embedded Computer Systems Analysis and Method (ECSAM)[1]
The State chart shall include:
a. Major system states definition.
b. Transition between states
c. User triggers implemented as system inputs.
d. Logic Events implemented as state transitions and system
outputs.
2.3.2 DETERMINING WEIGHTS FOR STATE TRANSITIONS
One of the setbacks of the IFT method is the need to know and understand system's
operational concept, hence we were challenged to come up with an unbiased objective
method of giving weights for each state transition.
The following sections describe the process of calculating the edges weights.
2.3.2.1 Weight Definition
To answer this question we used the classical approach of Reliability engineering, meaning
the weight formula is:
TW A→ B = max{Pj ⋅ S j }
Where:
TWA→B – Transition weight from State 'A' to State 'B'
Pj – The Probability of an individual transition from State 'A' to State 'B' to occur.
Sj – The Severity had the transition Pj occurred.
max{ Sj } – Maximum severity of possible transitions from State 'A' to State 'B'.
2.3.2.2 Evaluating the Transition Probability
In order to determine what is the probability of a transition to occur we first need to
understand the causes for a transition to happen.
Transitions between states are due to:
• Hardware Faults.
o Each hardware fault dealt by Satellite’s logic is connected to a
specific subsystem.
o Each subsystem has a known calculated or measured reliability.
o The probability of a transition occurrence can be simulated
using the above data
• Control Algorithm Faults
o Inability to keep the system within predefined limits.
o Positive feedbacks.
o Unexpected Sensors behavior.
o Unexpected Actuators behavior.
• User Faults
o Data inputs from the user to the system can lead to errors
• Telecommands.
o Mission rules dictate enforcing state transitions on the system.
The FTO methodology suggests that:
1.
The probability of a State transition which is related to Hardware faults
should be calculated as: Pj = 1 − R(Hardware ) .
2.
The probability of a State transition which is related to Control algorithm
faults should be calculated as:
∑ attitude related faults Or P = ∑ faults det ected during ATP .
Pj =
j
∑ faults
∑ tests performed
Note:
If the system is based on a previous system, the first approach should be taken, if it's a new
system than the ATP approach should be taken.
3.
The probability of a State transition which is related to User faults should
be calculated as: Pj = U [0...1] a random number with a uniform
distribution.
4.
The probability of a State transition which is related to Telecommands
should be calculated as: Pj = 1
From all of the above we can see that: 0 ≤ Pj ≤ 1
2.3.2.3 Evaluating the Transition Severity
The Severity of a transition must be strongly connected to the System's Mission, hence the
following scaling is suggested:
•
Sj = 1 for transitions to and from ‘Transient’ states.
•
Sj = 3 for transitions to ‘Survivability’ states.
•
Sj = 4 for transitions to ‘Operational’ states.
•
Sj = 5 for transitions to ‘Shutdown’ states.
Definitions:
1. Operational state is every state that allows the system to perform in such a
way that all its missions and goals are fulfilled.
2. Survivability state is every state that allows the system graceful degradation in
a manner that only part of system's mission and goals are fulfilled, and can be
recovered with user's interference.
3. Transient state is the state that the system uses in order to go from Survival to
Operations and vise versa.
4. Shutdown state is the non-operative state of the system, a sort of final stop.
From all of the above we can see that: 0 ≤ S j ≤ 5
3 FTO Validation
3.1 GENERAL
The test-case platform chosen for evaluating FTO is the AMOS-3 communications satellite.
AMOS-3 is a highly sophisticated communications satellite, which will be launched by the
end of 2007. AMOS-3 was designed and built using the experienced gathered during previous
AMOS programs, meaning AMOS-1 and AMOS-2.
AMOS-3 was a perfect candidate because we can compare FTO results to the IFT results
which will be performed by AMSO-3 developers, hence give a quantitative proof of the
method validity and effectiveness
3.2 AMOS-3 FTO PROCESS VALIDATION
During our case study we will use the AMOS-3 operational logic state machine to create the
transitions weights matrix reflecting the expectancies of moving from any state to its
neighbors.
After identifying all existing edges a weight shall be calculated for each edge. The calculation
shall be performed according to the process defined in section 2.3.2.
The set of scenarios will be compared to a set of scenarios suggested by engineers from
various disciplines. The overlap between the scenarios chosen by the FTO and the scenarios
chose by engineers will be used to assess the validity of the FTO.
4 References
[1] Systems Modeling & Requirements Specification using ECSAM, An Analysis
Method for Embedded and Computer-Based Systems, by Jonah Z. Lavi and
Joseph Kudish, ISBN: 0-932633-45-5.
[2] The IFT Approach, E. Sagi and M. Pariente, INCOSE_IL conference, 2005
The Authors
Raz Tamir, Ph.D. Computer Science,
MsC. Aerospace Engineering
AMOS-4 Satellite Chief System Engineer
Raz started working in the space industry in 2002 as an observation satellite system engineer.
In this role Raz led the development and implementation of various satellite systems.
Between 1995 and 2002 Raz specialized in Unmanned Air Vehicles (UAV) engines
development.
Raz is the CEO of INSA, The Israeli NanoSatellite Association
Email: [email protected]
Phone: 972-3-5314631
Meidad Pariente, M.E. Systems Engineering
AMOS-3 Satellite Chief System Engineer
Meidad started working in the space industry in 1995 as AMOS-1 Satellite operator,
Co-managed the IFT of AMOS-2, was the System Engineer of the first 2 phases of project
VENµS (joint venture of Israel Space Agency and CNES) today acts as a Chief System
Engineer in AMOS-3 project.
Meidad is the Spokesperson of INSA, The Israeli NanoSatellite Association
Email: [email protected]
Mobile: 052-8301385