Flexible and Intelligent Learning Architectures for SoS (FILA-SoS)
Volume 4 – Architecture Assessment Model
Technical Report SERC-2015-TR-021-4.4
February 19, 2015
Principal Investigator: Dr. Cihan H. Dagli, Missouri University of Science and Technology
Project Team
Co-PI: Dr. David Enke, Missouri S&T
Co-PI: Dr. Nil Ergin, Penn State University
Co-PI: Dr. Dincer Konur, Missouri S&T
Co-PI: Dr. Ruwen Qin, Missouri S&T
Co-PI: Dr. Abhijit Gosavi, Missouri S&T
Dr. Renzhong Wang
Missouri S&T Graduate Students
Louis Pape II, Siddhartha Agarwal and Ram Deepak Gottapu
SERC 2015-TR-021-4.4 – Volume 4
1
Flexible and Intelligent Learning Architectures for SoS (FILA-SoS)
Volume 1: Integrated Model Structure
Volume 2: Meta-Architecture Generation Multi-Level Model
Volume 3: Fuzzy-Genetic Optimization Model
Volume 4: Architecture Assessment Model
Volume 5: Cooperative System Negotiation Model
Volume 6: Non-Cooperative System Negotiation Model
Volume 7: Semi-Cooperative System Negotiation Model
Volume 8: Incentive based Negotiation Model for System of Systems
Volume 9: Model for Building Executable Architecture
Volume 10: Integrated Model Software Architecture and Demonstration FILA-SoS
Version1.0
Volume 11: Integrated Model Structure FILA-SoS Version 2.0
Volume 12: Complex Adaptive System-of-System Architecture Evolution Strategy
Model for FILA-SoS Version 2.0
Volume 13: On the Flexibility of Systems in System of Systems Architecting: A new
Meta-Architecture Generation Model for FILA-SoS Version 2.0
Volume 14: Assessing the Impact on SoS Architecture Different Level of
Cooperativeness: A new Model for FILA-SoS Version 2.0
Volume 15: Incentivizing Systems to Participate in SoS and Assess the Impacts of
Incentives: A new Model for FILA-SoS Version 2.0
Volume 16: Integrated Model Software Architecture for FILA-SoS Version 2.0
Volume 17: FILA-SoS Version 1.0 Validation with Real Data
SERC 2015-TR-021-4.4 – Volume 4
2
Copyright © 2015 Stevens Institute of Technology, Systems Engineering Research Center
The Systems Engineering Research Center (SERC) is a federally funded University Affiliated Research
Center managed by Stevens Institute of Technology.
This material is based upon work supported, in whole or in part, by the U.S. Department of Defense
through the Office of the Assistant Secretary of Defense for Research and Engineering (ASD(R&E)) under
Contract Numbers: H98230-08-D-0171 and HQ0034-13-D-004.
Any views, opinions, findings and conclusions or recommendations expressed in this material are those of
the author(s) and do not necessarily reflect the views of the United States Department of Defense nor
ASD(R&E).
No Warranty.
This Stevens Institute of Technology and Systems Engineering Research Center material is furnished on an
“as-is” basis. Stevens Institute of Technology makes no warranties of any kind, either expressed or
implied, as to any matter including, but not limited to, warranty of fitness for purpose or merchantability,
exclusivity, or results obtained from use of the material. Stevens Institute of Technology does not make
any warranty of any kind with respect to freedom from patent, trademark, or copyright infringement.
This material has been approved for public release and unlimited distribution.
SERC 2015-TR-021-4.4 – Volume 4
3
Executive Summary
Multi-faceted systems of the future will entail complex logic and reasoning with many levels of reasoning
in intricate arrangement. The organization of these systems involves a web of connections and demonstrates
self-driven adaptability. They are designed for autonomy and may exhibit emergent behavior that can be
visualized. Our quest continues to handle complexities, design and operate these systems. The challenge in
Complex Adaptive Systems design is to design an organized complexity that will allow a system to achieve
its goals. This report attempts to push the boundaries of research in complexity, by identifying challenges
and opportunities. Complex adaptive system-of-systems (CASoS) approach is developed to handle this
huge uncertainty in socio-technical systems.
Although classically (Dahmann, Rebovich, Lowry, Lane, & Baldwin, 2011) four categories of SoS are
described in literature namely; Directed, Collaborated, Acknowledged and Virtual. However, there exist
infinitely many SoS on the edges of these categories thus making it a continuum. Many SoS with different
configurations can fill this gap. These four types of SoS vary based on their degree of managerial control
over the participating systems and their structural complexity. The spectrum of SoS ranges from Directed
SoS that represents complicated systems to Virutal SoS that are complex systems.
Acknowledged SoS lie in between this spectrum. This particular SoS is the focal point of our research
endeavor. Acknowledged SoS and Directed SoS share some similarities such as both have (Dahman &
Baldwin, 2011) SoS objectives, management, funding and authority. Nevertheless, unlike Directed SoS,
Acknowledged SoS systems are not subordinated to SoS. However, Acknowledged SoS systems retain their
own management, funding and authority in parallel with the SoS. Collaborative SoS are similar to
Acknowledged SoS systems in the fact that systems voluntarily work together to address shared or common
interest.
Flexible and Intelligent Learning Architectures for SoS (FILA-SoS) integrated model is developed in this
research task provides a decision making aid for SoS manager based on the wave model. The model
developed called the FILA-SoS does so using straightforward system definitions methodology and an
efficient analysis framework that supports the exploration and understanding of the key trade-offs and
requirements by a wide range system-of-system stakeholders and decision makers in a short time. FILASoS and the Wave Process address four of the most challenging aspects of system-of-system architecting:
1. Dealing with the uncertainty and variability of the capabilities and availability of potential
component systems;
SERC 2015-TR-021-4.4 – Volume 4
4
2. Providing for the evolution of the system-of-system needs, resources and environment over
time;
3. Accounting for the differing approaches and motivations of the autonomous component system
managers;
4. Optimizing system-of-systems characteristics in an uncertain and dynamic environment with
fixed budget and resources;
Some of the highlights of FILA-SoS are listed in terms of its capabilities, value added to systems
engineering, ability to perform “What-if Analysis”, modularity of integrated models, its potential
applications in the real world and future additions to the current version.
FILA-SoS has a number of unique capabilities such as integrated model for modeling and simulating SoS
systems with evolution for multiple waves. It also has modularity in the structure where the models can be
run independently and in conjunction with each other. Besides there are a couple of different models for
both architecture generation and SoS behavior and various individual system behavior negotiation models
between SoS and individual systems. In terms of value added FILA-SoS aids the SoS manager in future
decision making. It also helps in understanding the emergent behavior of systems in the acquisition
environment and impact on SoS architecture quality. FILA-SoS serves as an artifact to study the dynamic
behavior of different type of systems (non-cooperative, semi-cooperative, cooperative). It enables us to
identify intra and interdependencies among SoS elements and the acquisition environment. FILA-SoS can
provide a “What-if” Analysis depending on variables such as SoS funding and capability priority that can
be changed as the acquisition progresses through wave cycles. It has the ability to simulate any architecture
through colored petri nets. In addition, it can simulate rules of engagement & behavior settings: all systems
are non-cooperative, all systems are semi-cooperative, and all systems are cooperative or a combination.
Some of the potential applications include modeling a wide variety of complex systems models such as
logistics, and cyber-physical systems. It also acts as a test-bed for decision makers to evaluate operational
guidelines and principles for managing various acquisition environment scenarios. Future Capabilities that
are currently in progress are extending the model to include multiple interface alternatives among systems
and incorporation of risk models into environmental scenarios.
The project reports span 17 volumes. Each report describes the various aspects of the FILA-SOS integrated
model.
Volume 1 is the Integrated Model Structure report for FILA-SoS Version 1.0. It provides a short description
of all independent models that make up the FILA-SoS integrated model. Integrated FILA-SoS developed
is tested in three notional System-of-Systems namely; Toy Problem for Aircraft Carrier Performance
SERC 2015-TR-021-4.4 – Volume 4
5
Assessment, ISR (intelligence surveillance and reconnaissance) and SAR (search and rescue). FILA-SoS
integrated model is currently being validated with a real life data from a medium sized SoS. The results of
this validation are given in volume 17.
Volume 2 describes Meta-Architecture Generation Multi-Level Model. The multi-level meta-architecture
generation model considers constructing an SoS architecture such that each capability is provided by at
least one system in the SoS and the systems in the SoS are able to communicate with each other. Secondly,
it has multiple objectives for generating a set of SoS architectures namely; maximum total performance,
minimum total costs and minimum deadline. Finally, the model establishes initial contracts with systems
to improve performances.
Volume 3 illustrates the second meta-architecture generation model known as the Fuzzy-Genetic
optimization model. This model is based on evolutionary multi-objective optimization for SoS architecting
using genetic algorithms and four key performance attributes (KPA) as the objective functions. It also has
a type-1 fuzzy assessor for dynamic assessment of domain inputs and that forms the fitness function for the
genetic algorithm. It returns the best architecture (meta-architecture) consisting of systems and their
interfaces. It is a generalized method with application to multiple domains such as Gulf War
Intelligence/Surveillance/Reconnaissance Case, Aircraft Carrier Performance Assessment Case and
Alaskan Maritime Search and Rescue Case.
Volume 4 describes an Architecture Assessment Mode that can capture the non-linearity in key
performance attribute (KPA) tradeoffs, is able to accommodate any number of attributes for a selected SoS
capability, and incorporate multiple stakeholder’s understanding of KPA’s. Assessment is based on a given
meta-architecture alternative. This is done using type-1 fuzzy sets and fuzzy inference engine. The model
provides numerical values for meta-architecture quality.
Volumes 5, 6, and 7 describe the systems negotiation models. The negotiation can be based on any
quantitative issue such as amount of funding required, deadline to join SoS and performance level requests.
Volume 5 specifically describes the Cooperative System Negotiation Model. The systems following this
model behave cooperatively while negotiating with the SoS manager. The model of cooperative behavior
is based on agent preferences and the negotiation length. Each system agent has two inherent behaviors of
cooperativeness: Purposive (normal behavior) and Contingent (behavior driven by unforeseen
circumstances). The approach models the tradeoff between the two behaviors for the systems. A fuzzy
weighted average approach is used to arrive at the final proposed value.
SERC 2015-TR-021-4.4 – Volume 4
6
Volume 6 goes on to describe the Non-Cooperative System Negotiation Model in which systems behave
in their self-interest while negotiating with the SoS coordinator. A mathematical model of individual
system’s participation capability and self-interest negotiation behavior is created. This methodology is an
optimization-based generator of alternatives for strategically negotiating multiple items with multiple
criteria. Besides, a conflict evaluation function that estimates prospective outcome for identified alternative
is proposed.
The third and last system negotiation model is described in Volume 7, which illustrates the SemiCooperative System Negotiation Model. It exhibits the capability of being flexible or opportunistic: i.e.,
extremely cooperative or uncooperative based on different parameter values settings. A Markov-chain
based model designed for handling uncertainty in negotiation modeling in an SoS. A model based on
Markov chains is used for estimating the outputs. The work assigned by the SoS to the system is assumed
to be a ``project’’ that takes a random amount of time and a random amount of resources (funding) to
complete.
Volume 8 explains the SoS negotiation model also called the Incentive Based Negotiation Model for System
of Systems. This model is based on two key assumptions that are to design a contract to convince the
individual systems to join the SoS development and motivate individual systems to do their tasks well.
Game theory and incentive based contracts are used in the negotiation model that will maximize the welfare
for parties involved in the negotiation. SoS utility function takes into account local objectives for the
individual systems as well as global SoS objective whereas the incentive contract design persuades
uncooperative systems to join the SoS development.
Volume 9 illustrates the process of building Executable Architectures for SoS. The operations of the SoS
is a dynamic process with participating system interacting with each other and exchange various kinds of
resources, which can be abstract information or physical objects. This is done through a hybrid structure of
OPM (Object process methodology) and CPN (Colored petri nets) modeling languages. The OPM model
is intuitive and easy to understand. However, it does not support simulation, which is required for accessing
the behavior related performance. This is achieved by mapping OPM to CPN, which is an executable
simulation language. The proposed method can model the interactions between components of a system or
subsystems in SoS. In addition, it can capture the dynamic aspect of the SoS and simulate the behavior of
the SoS. Finally, it can access various behavior related performance of the SoS and access different
constitutions or configurations of the SoS which cannot be incorporated into the meta-architecture
generation models of Volume 2 & 3.
SERC 2015-TR-021-4.4 – Volume 4
7
Volume 10 elucidates the Integrated Model Software Architecture and Demonstration based on the models
described above. Volume 11 and thereon the reports are aimed at the upcoming newer version 2.0 of FILASoS. Volume 11 provides Integrated Model Structure for FILA-SoS Version 2.0 that could be implemented
in a new software environment.
Volume 12 provides a model to answer the first research question “What is the impact of different
constituent system perspectives regarding participating in the SoS on the overall mission effectiveness of
the SoS?”. It is named the Complex Adaptive System-of-System Architecture Evolution Strategy Model
and is incorporated in FILA-SoS Version 2.0. This volume describes a computational intelligence based
strategy involving meta-architecture generation through evolutionary algorithms, meta-architecture
assessment through type-2 fuzzy nets and finally its implementation through an adaptive negotiation
strategy.
Volumes 13 and 14 provide two different approaches to answer the second research question “How do
differing levels of cooperativeness in participating in the SoS impact the ability and timeliness of a group
to agree on a SoS or system architecture? Or impact the ability to effectively use the architecture already
in place?”.
Volume 13 is termed the Flexibility of Systems in System of Systems Architecting: A new MetaArchitecture Generation Model for FILA-SoS Version 2.0. The research question is answered through an
alternative technique to meta-architecture generation besides the one described in Volume 2.
Volume 14 proposes a new method for Assessing the Impact on SoS Architecture Different Level of
Cooperativeness. Second research question is answered through a model that allows different levels of
cooperativeness of individual systems.
Volume 15 is an extension of previous systems negotiation models based on incentivizing and is aptly
called Incentivizing Systems to Participate in SoS and Assess the Impacts of Incentives: A new Model for
FILA-SoS Version 2.0. It also provides an approach to answer the third research question “How should
decision-makers incentivize systems to participate in SoS, and better understand the impact of these
incentives during SoS development and effectiveness?”. This model is based on the fact that providing
incentives only depending on the outcome may not be enough to attract the attention of the constituent
systems to participate in SoS mission. Therefore, this model extends the approach as described in Volume
8 while considering the uncertainty in the acquisition environment. The incentive contract is designed based
on the objectives of the SoS and the individual systems. Individual system’s objective is to secure highest
SERC 2015-TR-021-4.4 – Volume 4
8
incentives with minimal effort while the SoS manager’s goal is to convince individual systems to join the
SoS development while maximizing its own utility.
Volume 16 gives an overview of the integrated model architecture in version 2.0 of the software. It includes
all old and new models previously mentioned.
Finally, Volume 17 describes the validation of the FILA-SoS Version 1.0 with a real life data provided by
MITRE Corporation by from a moderately sized SoS.
SERC 2015-TR-021-4.4 – Volume 4
9
List of Publications Resulted and Papers Submitted from FILA-SoS
Research
1. Wang, R., Agarwal,S., & Dagli, C. (2014). Executable System of Systems Architecture
Using OPM in Conjunction with Colored Petri Net: A Module for Flexible Intelligent &
Learning Architectures for System of Systems, In Europe Middle East & Africa Systems
Engineering Conference (EMEASEC).
2. Ergin, N. K.,(2014), Improving Collaboration in Search and Rescue System of
Systems, Procedia Computer Science, Volume 36, Pages 13-20.
3. Agarwal, S., & Dagli, C. H. (2013). Augmented Cognition in Human–System Interaction
through Coupled Action of Body Sensor Network and Agent Based Modeling. Procedia
Computer Science, 16, 20-28.
4. Acheson, P., Dagli, C., & Kilicay-Ergin, N. (2013). Model Based Systems Engineering
for System of Systems Using Agent-based Modeling. Procedia Computer Science, 16,
11-19.
5. Agarwal, S., Pape, L. E., & Dagli, C. H. (2014). A Hybrid Genetic Algorithm and Particle
Swarm Optimization with Type-2 Fuzzy Sets for Generating Systems of Systems
Architectures. Procedia Computer Science, 36, 57-64.
6. Agarwal, S., Pape, L. E., Kilicay-Ergin, N., & Dagli, C. H. (2014). Multi-agent Based
Architecture for Acknowledged System of Systems. Procedia Computer Science, 28, 110.
7. Agarwal, S., Saferpour, H. R., & Dagli, C. H. (2014). Adaptive Learning Model for
Predicting Negotiation Behaviors through Hybrid K-means Clustering, Linear Vector
Quantization and 2-Tuple Fuzzy Linguistic Model. Procedia Computer Science, 36, 285292.
8. Agarwal,S., Wang, R., & Dagli, C., (2015) FILA-SoS, Executable Architectures using
Cuckoo Search Optimization coupled with OPM and CPN-A module: A new MetaArchitecture Model for FILA-SoS, France, Complex Systems Design & Management
(CSD&M) editor, Boulanger, Frédéric, Krob, Daniel, Morel, Gérard, Roussel, JeanClaude, P 175-192 . Springer International Publishing.
SERC 2015-TR-021-4.4 – Volume 4
10
9. Pape, L., Agarwal, S., Giammarco, K., & Dagli, C. (2014). Fuzzy Optimization of
Acknowledged System of Systems Meta-architectures for Agent based Modeling of
Development. Procedia Computer Science, 28, 404-411.
10. Pape, L., & Dagli, C. (2013). Assessing robustness in systems of systems metaarchitectures. Procedia Computer Science, 20, 262-269.
11. Pape, L., Giammarco, K., Colombi, J., Dagli, C., Kilicay-Ergin, N., & Rebovich, G.
(2013). A fuzzy evaluation method for system of systems meta-architectures. Procedia
Computer Science, 16, 245-254.
12. Acheson, P., Dagli, C., & Kilicay-Ergin, N. (2013). Model Based Systems Engineering
for System of Systems Using Agent-based Modeling. Procedia Computer Science, 16,
11-19.
13. Acheson, P., Dagli, C., & Kilicay-Ergin, N. (2014). Fuzzy Decision Analysis in
Negotiation between the System of Systems Agent and the System Agent in an AgentBased Model. arXiv preprint arXiv:1402.0029.
14. Kilicay-Ergin, N. H., Acheson, P., Colombi, J. M., & Dagli, C. H. (2012). Modeling
system of systems acquisition. In SoSE (pp. 514-518).
15. Acheson, P., Pape, L., Dagli, C., Kilicay-Ergin, N., Columbi, J., & Haris, K. (2012).
Understanding System of Systems Development Using an Agent-Based Wave Model.
Procedia Computer Science, 12, 21-30.
16. Konur, D., & Dagli, C. (2014). Military system of systems architecting with individual
system contracts. Optimization Letters, 1-19.
17. Agarwal et al., 2015 Flexible and Intelligent Learning Architectures for SoS (FILASoS): Architectural evolution in Systems-of-Systems, 2015 Conference on Systems
Engineering Research.
18. Ergin, D., & Dagli, C., Incentive Based Negotiation Model for System of Systems
Acquisition. ( Accepted by Systems Engineering Journal)
19. Wang, R., & Dagli, C., Search Based Systems Architecture Development Using
Holistic Approach (Accepted to IEEE Systems Journal with minor revisions)
SERC 2015-TR-021-4.4 – Volume 4
11
Table of Contents
List of Publications Resulted and Papers Submitted from FILA-SoS Research ........................................... 10
1
Introduction ................................................................................................................................ 15
1.1
Motivation for research .............................................................................................................. 15
1.2
System of System Challenges...................................................................................................... 18
1.3
How does FILA-SoS address SoS pain points............................................................................... 21
2
Overview of the FILA-SoS integrated model............................................................................... 25
2.1
Definition of Variables for SoS .................................................................................................... 25
2.2
Independent modules of FILA-SOS ............................................................................................. 29
3
Architecture Assessment ............................................................................................................ 32
3.1
Acknowledged SoS ...................................................................................................................... 32
3.2
Proposed Modeling Approach .................................................................................................... 34
3.3
SoS Attributes ............................................................................................................................. 36
4
Proposed Method for Developing an Sos Evaluation Model ..................................................... 39
4.1
Use Case Model of the Domain Independent Method ............................................................... 39
4.2
Domain Independent Model Development ............................................................................... 42
4.2.1
Establishing A Vision of The SoS.......................................................................................... 43
4.2.2
Understanding Stakeholders Views .................................................................................... 49
4.2.3
Review of the Method Steps ............................................................................................... 54
4.2.4
Architecture Space Exploration........................................................................................... 57
4.3
5
Individual Systems’ Information ................................................................................................. 58
4.3.1
Cost, Performance and Schedule Inputs of Component Systems ...................................... 58
4.3.2
Membership Functions ....................................................................................................... 59
4.3.3
Mapping Attribute Measures to Fuzzy Variables ................................................................ 60
4.3.4
Non-Linear Trades in Multiple Objectives of SoS................................................................ 62
4.4
Combining SoS Attribute Values Into an Overall SoS Measure .................................................. 63
4.5
Exploring the SoS Architecture Space with Genetic Algorithms ................................................. 66
4.6
Combining the Fuzzy Approach with the GA Approach .............................................................. 67
Concluding Remarks ................................................................................................................... 69
Cited and Related References ..................................................................................................................... 70
SERC 2015-TR-021-4.4 – Volume 4
12
List of Figures
Figure 1 Schematic drawing of four classical types of SoS based on Degree of Control and Degree of
Complexity .................................................................................................................................................. 16
Figure 2 ISR System-of-Systems for testing FILA-SoS.................................................................................. 23
Figure 3 SAR System-of-Systems for testing FILA-SoS ................................................................................ 23
Figure 4 Assessing the Performance of Aircraft Carrier for testing FILA-SoS ............................................. 24
Figure 5 The Wave Model of SoS initiation, engineering, and evolution ................................................... 26
Figure 6 Integrated modules within FILA- SoS ............................................................................................ 29
Figure 11. SoS evaluation method determines the fitness of each architecture, or (system + interface) SoS
arrangement, from the meta-architecture and domain dependent information ...................................... 35
Figure 14. Use case diagram for developing a SoS Architecture; dashed lines are for information; solid
lines are ‘responsible for,’ or active involvement; this portion of the effort excludes the
NegotiateAchievableSoS use case............................................................................................................... 40
Figure 15. Domain Independent Process Method for SoS Model building ............................................... 43
Figure 16. Sample OV-1 for Ballistic Missile Defense (ASN/RDA, 2009) .................................................... 46
Figure 17. Domain independent method to gather domain dependent data for SoS architecture model
development and analysis, with output of a ‘good’ architecture at the lower right ................................. 49
Figure 18. Given an Evaluation Model that depends on a chromosome from the meta-architecture, the
genetic algorithm can optimize within a multidimensional space of attributes ........................................ 55
Figure 19. Matlab Fuzzy Toolbox printout of membership function shapes used in this analysis ............ 60
Figure 20. Map from fuzzy variable on horizontal axis to probability of detection on left ....................... 61
Figure 21. Attribute values, mapped to fuzzy variables............................................................................. 62
Figure 23. Example of nonlinear SoS fitness versus Affordability and Performance, based on membership
function shapes and combining rules ......................................................................................................... 63
Figure 24. Exploring the meta-architecture - 25 chromosomes, 22 systems, Example 1.......................... 67
Figure 25. Exploring the meta-architecture to map membership function edges, Example 2 .................. 68
Figure 26. Exploring large populations to set the membership function edges ........................................ 68
SERC 2015-TR-021-4.4 – Volume 4
13
List of Tables
Table 1 System of Systems and Enterprise Architecture Activity ............................................................... 18
Table 2 List of SoS and component systems’ variable meanings within the meta-architecture ................ 40
Table 3 Example SoS evaluation model building questionnaire for creating an AV-1................................ 44
Table 4 Example AV-2, Integrated Dictionary ............................................................................................ 53
Table 5 Example of a few powerful Fuzzy Inference Rules for combining attribute values ....................... 64
SERC 2015-TR-021-4.4 – Volume 4
14
1 Introduction
1.1 Motivation for research
In the real world, systems are complex, non-deterministic, evolving, and have human centric capabilities.
The connections of all complex systems are non-linear, globally distributed, and evolve both in space and
in time. Because of non-linear properties, system connections create an emergent behavior. It is imperative
to develop an approach to deal with such complex large-scale systems. The approach and goal is not to try
and control the system, but design the system such that it controls and adapts itself to the environment
quickly, robustly, and dynamically. These complex entities include both socioeconomic and physical
systems, which undergo dynamic and rapid changes. Some of the examples include transportation, health,
energy, cyber physical systems, economic institutions and communication infrastructures.
In addition, the idea of “System-of-Systems” is an emerging and important multidisciplinary area. An SoS
is defined as a set or arrangement of systems that results when independent and useful systems are integrated
into a larger system that delivers unique capabilities greater than the sum of the capabilities of the
constituent parts. Either of the systems alone cannot independently achieve the overall goal. System-ofSystems (SoS) consists of multiple complex adaptive systems that behave autonomously but cooperatively
(Dahman, Lane, Rebovich, & Baldwin, 2008). The continuous interaction between them and the
interdependencies produces emergent properties that cannot be fully accounted for by the “normal” systems
engineering practices and tools. System of Systems Engineering (SoSE), an emerging discipline in systems
engineering is attempting to form an original methodology for SoS problems (Luzeaux, 2013).
Since SoS grow in complexity and scale with the passage of time it requires architectures that will be
necessary for understanding and governance and for proper management and control. Systems architecting
can be defined as specifying the structure and behavior of an envisioned system. Classical system
architecting deals with static systems whereas the processes of System of Systems (SoS) architecting has
to be first done at a meta-level. The architecture achieved at a meta-level is known as the meta-architecture.
The meta-architecture sets the tone of the architectural focus (Malan & Bredemeyer, 2001). It narrows the
scope of the fairly large domain space and boundary. Although the architecture is still not fixed but metaarchitecture provides multiple alternatives for the final architecture. Thus architecting can be referred to as
filtering the meta-architectures to finally arrive at the architecture. The SoS architecting involves multiple
systems architectures to be integrated to produce an overall large scale system meta-architecture for a
specifically designated mission (Dagli & Ergin, 2008). SoS achieves the required goal by introducing
SERC 2015-TR-021-4.4 – Volume 4
15
collaboration between existing system capabilities that are required in creating a larger capability based on
the meta-architecture selected for SoS. The level of the degree of influence on individual systems
architecture through the guidance of SoS manager in implementing SoS meta-architecture can be classified
as directed, acknowledged, collaborative and virtual. Acknowledged SoS have documented objectives, an
elected manager and defined resources for the SoS. Nonetheless, the constituent systems retain their
independent ownership, objectives, capital, development, and sustainment approaches. Acknowledged SoS
shares some similarities with directed SoS and collaborative SoS. There are four types of SoS that are
described below:
Figure 1 Schematic drawing of four classical types of SoS based on Degree of Control and Degree of Complexity
–
–
Virtual
•
Virtual SoS lack a central management authority and a centrally agreed upon
purpose for the system-of-systems.
•
Large-scale behavior emerges—and may be desirable—but this type of SoS must
rely upon relatively invisible mechanisms to maintain it.
Collaborative
•
In collaborative SoS the component systems interact more or less voluntarily to
fulfill agreed upon central purposes.
SERC 2015-TR-021-4.4 – Volume 4
16
–
–
Acknowledged (FILA-SoS integrated model is based on Acknowledged SoS)
•
Acknowledged SoS have recognized objectives, a designated manager, and
resources for the SoS; however, the constituent systems retain their independent
ownership, objectives, funding, and development and sustainment approaches.
•
Changes in the systems are based on collaboration between the SoS and the system.
Directed
•
Directed SoS’s are those in which the integrated system-of-systems is built and
managed to fulfill specific purposes.
•
It is centrally managed during long-term operation to continue to fulfill those
purposes as well as any new ones the system owners might wish to address.
•
The component systems maintain an ability to operate independently, but their
normal operational mode is subordinated to the central managed purpose.
This research is based on Acknowledged SoS. The major objectives of the reasearch are:
–
To develop a simulation for Acknowledged SoS architecture selection and evolution.
–
To have a structured, repeatable approach for planning and modeling.
–
To study and evaluate the impact of individual system behavior on SoS capability and
architecture evolution process.
The dynamic planning for a SoS is a challenging endeavor. Department of Defense (DoD) programs
constantly face challenges to incorporate new systems and upgrade existing systems over a period of time
under threats, constrained budget, and uncertainty. It is therefore necessary for the DoD to be able to look
at the future scenarios and critically assess the impact of technology and stakeholder changes. The DoD
currently is looking for options that signify affordable acquisition selections and lessen the cycle time for
early acquisition and new technology addition. FILA-SoS provides a decision aid in answering some of the
questions.
This volume gives an overview of a novel methodology known as the Flexible Intelligent & Learning
Architectures in System-of-Systems (FILA-SoS). Some the challenges that are prevalent in SoS architecting
and how FILA-SoS attempts to address them is explained in the next section.
SERC 2015-TR-021-4.4 – Volume 4
17
1.2 System of System Challenges
All these recent developments are helping us to understand Complex Adaptive Systems. They are at the
edge of chaos as they maintain dynamic stability through constant self-adjustment and evolution. Chaos
and order are two complementary states of our world. A dynamic balance exists between these two states.
Order and structure are vital to life. Order ensures consistency and predictability and makes the creation of
systems possible. However, too much order leads to rigidity and suppresses creativity. Chaos constantly
changes the environment creating disorder and instability but can also lead to emergent behavior and allows
novelty and creativity. Thus, sufficient order is necessary for a system to maintain an ongoing identity,
along with enough chaos to ensure growth and development. The challenge in Complex Adaptive Systems
design is to design an organized complexity that will allow a system to achieve its goals. SoS is a complex
systems by its nature due to the following characteristics that are component systems are operationally
independent elements and also managerially independent of each other. This means that component systems
preserve existing operations independent of the SoS. SoS has an evolutionary development and due to the
large scale complex structure shows an emergent behavior. Emergence means the SoS performs functions
that do not reside in any one component system.
2012 INCOSE SoS working group survey identified seven ‘pain points’ raising a set of questions for
systems engineering of SoS which are listed in Table 1 (Dahman, 2012).
Table 1 System of Systems and Enterprise Architecture Activity
Pain Points
Question
Lack of SoS Authorities &
What are effective collaboration patterns in systems of systems?
Funding
Leadership
What are the roles and characteristics of effective SoS leadership?
Constituent Systems
What are effective approaches to integrating constituent systems into a SoS?
Capabilities & Requirements
How can SE address SoS capabilities and requirements?
Autonomy, Interdependencies &
How can SE provide methods and tools for addressing the complexities of SoS
Emergence
interdependencies and emergent behaviors?
Testing, Validation & Learning
How can SE approach the challenges of SoS testing, including incremental
validation and continuous learning in SoS?
SoS Principles
SERC 2015-TR-021-4.4 – Volume 4
What are the key SoS thinking principles, skills and supporting examples?
18
The importance and impact on systems engineering of each pain point is illustrated below:
i.
Lack of SoS Authorities & Funding and Leadership pose several and severe governance and
management issues for SoS. This conditions has a large impact on the ability to implement systems
engineering (SE) in the classical sense to SoS. In addition, this problem affects the modeling &
simulation activities.
ii.
Constituent Systems play a very important role in the SoS. As explained earlier usually they have
different interests and ambitions to achieve, which may or may not be aligned with the SoS..
Similarly models, simulations and data for these systems will naturally have to be attuned to the
specific needs of the systems, and may not lend themselves easily to supporting SoS analysis or
engineering
iii.
Autonomy, Interdependencies & Emergence is ramifications of the varied behaviors and
interdependencies of the constituent systems making it complex adaptive systems. Emergence
comes naturally in such a state, which is often unpredictable. While modeling & simulation can aid
in representing and measuring these complexities, it is often hard to achieve real life emergence.
This is due to limited understanding of the issues that can bring up serious consequences during
validation.
iv.
The overall Capability of the SoS and the individual systems capability needs may be high level
and need definition in order to align them with the requirements of the SoS mission. The SoS
mission is supported by constituent systems, which may not be able (or willing) to address them.
v.
Testing, Validation & Learning becomes difficult since the constituent systems continuously keep
evolving, adapting, as does the SoS environment which includes stakeholders, governments, etc.
Therefore creating a practical test-bed for simulating the large dynamic SoS is a challenge in itself.
Again modeling & simulation can solve part of the problem such as enhancing live test and
addressing risk in SoS when testing is not feasible; however, this requires a crystal clear
representation of the SoS which can be difficult as discussed in earlier points.
vi.
SoS Principles are still being understood and implemented. Therefore, the rate of success is yet to
be addressed formally. This poses some pressure on the progress of SoS engineering. Similarly,
there is an absence of a well-established agreeable space of SoS principles to drive development
and knowledge. This constricts the effective use of potentially powerful tools.
SERC 2015-TR-021-4.4 – Volume 4
19
The DoD 5000.2 is currently used as the acquisition process for complex systems. Schwartz (2010)
described this process as an extremely complex systemic process that cannot always constantly produce
systems with expected either cost or performance potentials. The acquisition in DoD is an SoS problem that
involves architecting, placement, evolution, sustainment, and discarding of systems obtained from a
supplier or producer. Numerous attempts undertaken to modify and reform the acquisition process have
found this problem difficult to tackle because the models have failed to keep pace with actual operational
scenarios. Dombkins (1996) offered a novel approach to model complex projects as waves. He suggested
that there exists a major difference in managing and modeling traditional projects versus complex projects.
He further illustrated his idea through a wave planning model that exhibits a linear trend on a time scale;
on a spatial scale, it tries to capture the non-linearity and recursiveness of the processes. In general, the
wave model is a developmental approach that is similar to periodic waves. A period, or multiple periods,
can span a strategic planning time. The instances within the periods represent the process updates. A
recently proposed idea (Dahman, Lane, Rebovich, & Baldwin, 2008) that SoS architecture development for
the DoD acquisition process can be anticipated to follow a wave model process. According to Dahman DoD
5000.2 may not be applicable to the SoS acquisition process. Acheson (2013) proposed that Acknowledged
SoS be modeled with an Object-Oriented Systems Approach (OOSA). Acheson also proposes that for the
development of SoS, the objects should be expressed in the form of a agent based model.
The environment and the systems are continuously changing. Let there be an initial environment model,
which represents the SoS acquisition environment. As the SoS acquisition progresses through, these
variables are updated by the SoS Acquisition Manager to reflect current acquisition environment. Thus, the
new environment model at a new time has different demands. To fulfill the demands of the mission a
methodology is needed to assess the overall performance of the SoS in this dynamic situation. The
motivation of evolution are the changes in the SoS environment (Chattopadhyay, Ross, & Rhodes, 2008).
The environmental changes consist of:
•
SoS Stakeholder Preferences for key performance attributes
•
Interoperability conditions between new and legacy systems
•
Additional mission responsibilities to be accommodated
•
Evolution of individual systems within the SoS
Evaluation of architectures is another SoS challenge area as it lends itself to a fuzzy approach because the
criteria are frequently non-quantitative, or subjective (Pape & Dagli, 2013), or based on difficult to define
or even unpredictable future conditions, such as “robustness.” Individual attributes may not have a clearly
defined, mathematically precise, linear functional form from worst to best. The goodness of one attribute
SERC 2015-TR-021-4.4 – Volume 4
20
may or may not offset the badness of another attribute. Several moderately good attributes coupled with
one very poor attribute may be better than an architecture with all marginally good attributes, or vice-versa.
A fuzzy approach allows many of these considerations to be handled using a reasonably simple set of rules,
as well as having the ability to include non-linear characteristics in the fitness measure. The simple rule set
allows small adjustments to be made to the model to see how seemingly small changes affect the outcome.
The methodology outlined in this research and technical report falls under a multi-level plug-and-play type
of modeling approach to address various aspects of SoS acquisition environment: SoS architecture
evaluation, SoS architecture evolution, and SoS acquisition process dynamics including behavioral aspects
of constituent systems.
1.3 How does FILA-SoS address SoS pain points
The first pain point is Lack of SoS Authorities & Funding which begs a question “What are effective
collaboration patterns in systems of systems?”
Since there is lack of SoS Authority but more so persuasion involved in the workings of a SoS, systems are
allowed to negotiate with the SoS manager. Deadline for preparation, funding and performance required
to complete the mission are some of the issues that form the negotiation protocol. Besides different
combination of behavior types assigned to the systems can help us gauge the best effective collaboration
patterns in systems of systems after the end of negotiations.
The leadership issues pose the question, “What are the roles and characteristics of effective SoS
leadership?” This is addressed by incorporating views from multiple stakeholders while assessing the
architecture’s quality. In addition, we maintain that the characteristics are similar to what an Acknowledged
SoS manager would have while distributing funds and resources among systems for a joint operation. The
SoS manager also has the opportunity to form his decision based on most likely future scenarios, thus
imparting him an edge as compared to other models. This will improve the process of acquisition in terms
of overall effectiveness, less cycle time and integrating legacy systems. Overall, the role of the leadership
is presented a guide than someone who would foist his authority.
The third pain point question, “What are effective approaches to integrating constituent systems into a SoS?
is addressed below. A balance has to be maintained during acquisition between amount of resources used
and the degree of control exercised by the SoS manager on the constituent systems. The meta-architecture
generation is posed as a multi-objective optimization problem to address this pain point. The constituent
systems and the interfaces between them are selected while optimizing the resources such as operations
cost, interfacing cost, performance levels etc. The optimization approach also evaluates the solutions based
on views of multiple stakeholders integrated together using a fuzzy inference engine.
SERC 2015-TR-021-4.4 – Volume 4
21
How can SE address capabilities and requirements? is the fourth pain point and is answered in this
paragraph. Organizations that acquire large-scale systems have transformed their attitude to acquisition.
Hence, these organizations now want solutions to provide a set of capabilities, not a single specific system
to meet an exact set of specifications. During the selection process of systems it is ensured that, a single
capability is provided by more than one system. The idea is to choose at least one systems having unique
capability to form the overall capability of the SoS.
The fifth pain point on autonomies, emergence and interdependencies is one of the most important
objectives of this research. This objective can be described as “How can SE provide methods and tools for
addressing the complexities of SoS interdependencies and emergent behaviors?”. Each system has an
autonomous behavior maintained through pre-assigned negotiation behaviors, differ operations cost,
interfacing cost and performance levels while providing the same required capability. The interfacing
among systems is encouraged to have net-centric architecture. The systems communicate to each other
through several communication systems. This ensures proper communication channels. Together the
behavior and net-centricity make it complex systems thus bringing out the emergence needed to address
the mission.
FILA-SoS is an excellent integrated model for addressing the complexities of SoS interdependencies and
emergent behaviors as explained in the above paragraphs.
As for the sixth pain point on testing, validation and learning goes, FILA-SoS has been tested on three
notional examples so far the ISR, Search and Rescue (SAR) and the Toy problem for Aircraft Carrier
Performance Assessment. For ISR (refer to Figure 2) a guiding physical example is taken from history.
During the 1991 Gulf War, Iraqi forces used mobile SCUD missile launchers called Transporter Erector
Launchers (TELS) to strike at Israel and Coalition forces with ballistic missiles. Existing intelligence,
surveillance, and reconnaissance (ISR) assets were inadequate to find the TELs during their vulnerable
setup and knock down time. The “uninhabited and flat” terrain of the western desert was in fact neither of
those things, with numerous Bedouin goat herders and their families, significant traffic, and thousands of
wadis with culverts and bridges to conceal the TELs and obscure their movement.
SERC 2015-TR-021-4.4 – Volume 4
22
Figure 2 ISR System-of-Systems for testing FILA-SoS
A Coast Guard Search and Rescue (SAR) (Figure 3) SoS engineering and development problem is selected
for serving the Alaskan coast. Detailed information about this case study can be found in Dagli et al (2013).
There is increasing use of the Bering Sea and the Arctic by commercial fisheries, oil exploration and
science, which increases the likelihood of occurrence of possible SAR scenarios.
Figure 3 SAR System-of-Systems for testing FILA-SoS
The toy problem for assessing the performance of the aircraft carrier involves multiple systems such as
satellites, uav’s and ground station that support the aircraft carrier to fulfill the mission (refer to Figure 4).
The results have been obtained for multiple waves of the evolution process for all the examples.
SERC 2015-TR-021-4.4 – Volume 4
23
Figure 4 Assessing the Performance of Aircraft Carrier for testing FILA-SoS
These example discussed above clearly show the domain independence of FILA-SoS.
FILA-SoS is a novel method of making sequential decisions over a period for SoS development. The goal
is to apply the integrated model to dynamically evolve SoS architecture and optimize SoS architecture,
design and validate through simulation tools. The integrated model structure can be applied to various
application areas including development of dynamic water treatment SoS architecture, development of
dynamic Air Traffic Management SoS, and development of autonomous ground transport SoS. FILA-SoS
has a number of abilities that make it unique such as:
 Aiding the SoS manager in future decision making
 To assist in understanding the emergent behavior of systems in the acquisition environment and impact
on SoS architecture quality
 To facilitate the learning of dynamic behavior of different type of systems (cooperative, semi-cooperative
, non-cooperative)
 Identifying intra and interdependencies among SoS elements and the acquisition environment
 Modeling and application to a wide variety of complex systems models such as logistics, cyber-physical
systems and similar systems
 Acting as a Test-bed for decision makers to evaluate operational guidelines and principles for managing
various acquisition environment scenarios
 Appropriate to model SoS that evolve over a period of time under uncertainties by multiple wave
simulation capability
SERC 2015-TR-021-4.4 – Volume 4
24
2 Overview of the FILA-SoS integrated model
In this section an overview of FILA-SoS is described. The model developed called the FILA-SoS is using
straightforward system definitions methodology and an efficient analysis framework that supports the
exploration and understanding of the key trade-offs and requirements by a wide range system-of-system
stakeholders and decision makers in a short time. FILA-SoS and the Wave Process address four of the most
challenging aspects of system-of-system architecting:
i.
Dealing with the uncertainty and variability of the capabilities and availability of potential
component systems.
ii.
Providing for the evolution of the system-of-system needs, resources and environment over
time.
iii.
Accounting for the differing approaches and motivations of the autonomous component
system managers.
iv.
Optimizing system-of-systems characteristics in an uncertain and dynamic environment
with fixed budget and resources
2.1 Definition of Variables for SoS
This list comprises of the notation for variables used to solve the Acknowledged SoS architectural evolution
problem:
C: The overall capability (the overall goal to be achieved by combining sub-capabilities)
𝑐𝑗 : j ∈ J, J= {1, 2,…, M}: Constituent system capabilities required
𝑠𝑖 : i ∈ I, I= {1, 2,…, N}: Total number of systems present in the SoS problem
Let 𝑨 be a 𝑁 x 𝑀 − 𝑚𝑎𝑡𝑟𝑖𝑥 𝑜𝑓 𝑎𝑖𝑗 𝑤ℎ𝑒𝑟𝑒
𝑎𝑖𝑗 = 1 𝑖𝑓 capability 𝑗 is possessed by system 𝑖
𝑎𝑖𝑗 = 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
𝑃𝑖 : Performance of system 𝑖 for delivering all capabilities ∑𝑗 𝑎𝑖𝑗
𝐹𝑖 : Funding of system 𝑖 for delivering all capabilities ∑𝑗 𝑎𝑖𝑗
𝐷𝑖 : Deadline to participate in this round of mission development for system 𝑖
𝐼𝐹𝑖𝑘 𝑖𝑠 𝑡ℎ𝑒 interface between systems 𝑖 𝑎𝑛𝑑 𝑘 s.t. s≠ 𝑘, k ∈ I
𝐼𝐶𝑖 : The cost for development of interface for system 𝑖
𝑂𝐶𝑖 : The cost of operations for system 𝑖
SERC 2015-TR-021-4.4 – Volume 4
25
𝐾𝑃𝑟 : r ∈ R, R= {1, 2,…, Z}: The key performance attributes of the SoS
𝐹𝐴: Funding allocated to SoS Manager
p= {1, 2,…, P}: number of negotiation attributes for bilateral negotiation
𝑡𝑚𝑎𝑥 : Total round of negotiations possible
𝑡 : Current round of negotiation (epochs)
𝑡𝑚𝑎𝑥 : Total round of negotiations possible
𝑆𝑜𝑆
𝑉𝑝𝑖
(𝑡): The value of the attribute 𝑝 for SoS manager at time 𝑡 for system 𝑖
𝑆
𝑉𝑝𝑖
(𝑡): The value of the attribute 𝑝 for system 𝑖 owner at time t
𝑇𝑄: Threshold architecture quality
The model also involves a list of stakeholders such as the Acknowledged SoS manager, system
owners/managers, SoS environment etc.
Figure 5 The Wave Model of SoS initiation, engineering, and evolution
FILA-SoS follows the Dahmann’s proposed SoS Wave Model process for architecture development of the
DoD acquisition process as depicted in Figure 5. FILA-SoS addresses the most important challenges of SoS
architecting in regards to dealing with the uncertainty and variability of the capabilities and availability of
potential component systems. The methodology also provides for the evolution of the system-of-system
needs, resources and environment over time while accounting for the differing approaches and motivations
of the autonomous component system managers. FILA-SoS assumes to have an uncertain and dynamic
environment with fixed budget and resources for architecting SoS. The overall idea being to select a set of
SERC 2015-TR-021-4.4 – Volume 4
26
systems and interfaces based on the needs of the architecture in a full cycle called the wave. Within the
wave, there may be many negotiation rounds, which are referred to as epochs. After each wave, the systems
selected during negotiation in the previous wave remain as part of the meta-architecture whilst new systems
are given a chance to replace those left out as a result.
Processes involved in the wave model and their analog in FILA-SoS can be explained through the first
stage of Initializing the SoS. In terms of initializing, wave process requires to understand the SoS objectives
and operational concept (CONOPS), gather information on core systems to support desired capabilities.
This starts with the overarching capability 𝐶 desired by Acknowledged SoS manager and defining the 𝑐𝑗 or
sub-capabilities required to produce capability 𝐶 and 𝐹𝐴, funding allocated to SoS Manager. These also
form the input to the FILA-SoS for the participating systems 𝑠𝑖 . FILA-SoS requires 𝑡𝑚𝑎𝑥 the number of
negotiation cycles, selection of the meta-architecture modelling procedure and system negotiation models
assigned to participating systems.
The second stage is called the Conduct_SoS_Analysis. For the Wave process, it represents starting an initial
SoS baseline architecture for SoS engineering based on SoS requirements space, performance measures,
and relevant planning elements. For FILA-SoS the baseline architecture is called as the meta-architecture.
Meta-architecture is basically picking up the systems 𝑠𝑖 and their respective capabilities 𝑎𝑖𝑗 . Metaarchitecture modelling requires the values for 𝐾𝑃𝑡 , the key performance attributes of the SoS, 𝑃𝑖
(Performance of system 𝑖) , 𝐹𝑖 (Funding of system 𝑖 ), and 𝐷𝑖 deadline to participate in this round of mission
development for system 𝑖 which is assumed to be the total for all capabilities possessed by system 𝑖. The
cost for development of a single interface for system 𝑖, 𝐼𝐶𝑖 and 𝑂𝐶𝑖 the cost of operations for system 𝑖 is
also needed at this stage of the model. The next step is the Develop/ Evolve SoS. In this case in terms of the
Wave process essential changes in contributing systems in terms of interfaces and functionality in order to
implement the SoS architecture are identified. Within FILA-SoS this signals the command to send
connectivity request to individual systems and starting the negotiation between SoS and individual systems.
This stage requires the number of negotiation attributes 𝑃 for a bilateral negotiation between Acknowledged
SoS manager and each systems 𝑖 selected in the meta-architecture and 𝑡𝑚𝑎𝑥 which denotes the total round
of negotiations possible.
The next phase is Plan SoS Update in Wave process. In this, phase the architect plans for the next SoS
upgrade cycle based on the changes in external environment, SoS priorities, options and backlogs. There is
an external stimulus from the environment, which affects the SoS architecture. To reflect that in FILA-SoS
determines which systems to include based on the negotiation outcomes and form a new SoS architecture.
Finally, the last stage in Wave process is Implement SoS Architecture which establishes a new SoS baseline
SERC 2015-TR-021-4.4 – Volume 4
27
based on SoS level testing and system level implementation. In the FILA-SoS the negotiated architecture
quality is evaluated based on 𝐾𝑃𝑟 , key performance attributes of the SoS. If the architecture quality is not
up to a predefined quality or 𝑇𝑄 the threshold architecture quality the Acknowledged SoS manager and
systems 𝑖 selected in the meta-architecture go for renegotiations. Finally the process moves on to the next
acquisition wave. The evolution of SoS should take into account availability of legacy systems and the new
systems willing to join, adapting to changes in mission and requirement, and sustainability of the overall
operation. FILA-SoS also has the proficiency to convert the meta-architecture into an executable
architecture using the Object Process Model (OPM) and Colored Petri Nets (CPN) for overall functionality
and capability of the meta-architecture. These executable architectures are useful in providing the muchneeded information to the SoS coordinator for assessing the architecture quality and help him in negotiating
better.
Some of the highlights of FILA-SoS are described in terms of its capabilities, value added to systems
engineering, ability to perform “What-if Analysis”, modularity of integrated models, its potential
applications in the real world and future additions to the current version. The most important capability of
FILA-SoS is it being an integrated model for modeling and simulating SoS systems with evolution for
multiple waves. Secondly, all models within FILA-SoS can be run independently and in conjunction with
each other. Thirdly, there are two model types that represent SoS behavior and various individual system
behaviors. Finally, it has the capacity to study negotiation dynamics between SoS and individual systems.
The value added by FILA-SoS to systems engineering is it aids the SoS manager in future decision making,
can help in understanding the emergent behavior of systems in the acquisition environment and its impact
on SoS architecture quality. Besides, it has three independent systems behavior models, which are referred
to as cooperative, semi-cooperative and non-cooperative. These behavior models are used to Study the
dynamic behavior of different type of systems while they are negotiating with SoS manager. In addition,
FILA-SoS assists in identifying intra and interdependencies among SoS elements and the acquisition
environment.
FILA-SoS also can facilitate a “What-if” Analysis using variables such as SoS funding and capability
priority that can be changed as the acquisition progresses though wave cycles. The parameter setting for all
negotiation models can be changed and rules of engagement can be simulated for different combinations of
systems behaviors.
Potential Application of FILA-SoS include complex systems models such as logistics, cyber-physical
systems. In addition, it can act as test-bed for decision makers to evaluate operational guidelines and
principles for managing various acquisition environment scenarios. While the future capabilities that we
SERC 2015-TR-021-4.4 – Volume 4
28
would like to be included are extending the model to include multiple interface alternatives among systems
and incorporation of risk models into environmental scenarios
2.2 Independent modules of FILA-SOS
The FILA-SoS has a number of independent modules that are integrated together for meta-architecture
generation, architecture assessment, meta-architecture executable model, and meta-architecture
implementation through negotiation. An overall view is presented in Figure 6.
Figure 6 Integrated modules within FILA- SoS
All the independent models are listed below for reference:
1. Meta-Architecture Generation Model
7. Executable Architecting Model: OPM &
CPN
2. Architecture Assessment Model
8. Overall Negotiation Framework
3. SoS Negotiation Model
4. System Negotiation
Cooperative
Model:
Non-
5. System Negotiation Model: Cooperative
6.
System Negotiation
Cooperative
SERC 2015-TR-021-4.4 – Volume 4
Model:
Semi-
29
The first meta-architecture generation method is fuzzy-genetic optimization model (Pape, Agarwal,
Giammarco & Dagli, 2014). This model is based on evolutionary multi-objective optimization for SoS
architecting with many key performance attributes (KPA). It also has a type-1 fuzzy assessor for dynamic
assessment of domain inputs and that forms the fitness function for the genetic algorithm. It returns the best
architecture (meta-architecture) consisting of systems and their interfaces. It is a generalized method with
application to multiple domains such as Gulf War Intelligence/Surveillance/Reconnaissance Case and
Alaskan Maritime Search and Rescue Case.
The second meta-architecture generation model is based on multi-level optimization (Konur & Dagli,
2014). In this model, architecting is done in two rounds: the first being the initiating the SoS by selecting
the systems to be included in the SoS and then improving the SoS’s performance by allocating funds to
participating systems. The model is generic based on multiple attributes such as maximum performance,
minimum cost and minimum deadline. It based on a Stackelberg game theoretical approach between the
SoS architect and the individual systems.
The particle swarm optimization (Agarwal, Pape, & Dagli, 2014) technique for meta-architecture
generation is similar to fuzzy-genetic model. Except for the fact that evolutionary optimization technique
in this case is based on swarm intelligence. In addition, there are some new key performance attributes used
to calculate the architectures quality. Cuckoo search optimization (Agarwal, Wang, & Dagli, 2014) based
meta-architecture is again anew biologically inspired method of optimization. It has been shown that it in
certain cases it performs better than PSO.
The first architecture assessment method is based on type-1 fuzzy logic systems (FLS) (Pape et al., 2013).
The Key Performance Parameters (KPP) chosen are performance, affordability, flexibility, and robustness.
It can capture the viewpoints of multiple stakeholders’. It can also accommodate any number of KPPs.
Another architecture assessment method is based on type-2 fuzzy modular nets (Agarwal, Pape & Dagli,
2014). The attributes used for evaluation were Performance, Affordability, Developmental Modularity,
Net-Centricity and Operational Robustness. Type-1 fuzzy sets are able to model the ambiguity in the input
and output variables. However, type-1 fuzzy sets are insufficient in characterizing the uncertainty present
in the data. Type-2 fuzzy sets proposed by Zadeh (1975) can model uncertainty and minimize its effects in
FLS (Mendel & John, 2002).
It is not possible to implement such meta-architecture without persuading the systems to participate, hence
to address the issue a negotiation model is proposed based on game theory (Ergin, 2104). It is an incentive
based negotiation model to increase participation of individual systems into Search and Rescue SoS. The
SERC 2015-TR-021-4.4 – Volume 4
30
model provides a strategy for SoS management to determine the appropriate amount of incentives necessary
to persuade individual systems while achieving its own goal. The incentive contract is designed based on
the objectives of the SoS and the individual systems. Individual system’s objective is to secure highest
incentives with minimal effort while the SoS manager’s goal is to convince individual systems to join the
SoS development while maximizing its own utility. Determining the incentives for individual systems can
be formulated as a multi-constraint problem where SoS manager selects a reward for the individual system
such that the reward will maximize SoS manager’s expected utility while satisfying the constraints of the
individual systems
Another negotiation model based on clustering and neural networks is developed (Agarwal, Saferpour &
Dagli, 2014). This model involves adapting the negotiation policy based on individual systems behavior
that is not known to the SoS manager. The behavior is predicted by clustering the difference of multi-issue
offers. Later the clustered data is trained using supervised learning techniques for future prediction.
Individual systems providing required capabilities can use three kinds of negotiation models based on their
negotiation strategies non-cooperative Linear Optimization model, cooperative fuzzy negotiation model,
and Semi-cooperative Markov chain model (Dagli et al., 2013).
Executable architectures are generated using a hybrid of Object Process Methodology (OPM) and Colored
Petri Nets (CPN) (Agarwal, Wang, & Dagli, 2014), (Wang, Agarwal, & Dagli, 2014), and (Wang & Dagli,
2011). To facilitate analysis of interactions between the participating systems in achieving the overall SoS
capabilities, an executable architecture model is imperative. In this research, a modeling approach that
combines the capabilities of OPM and CPN is proposed. Specifically, OPM is used to specify the formal
system model as it can capture both the structure and behavior aspects of a system in a single model. CPN
supplements OPM by providing simulation and behavior analysis capabilities. Consequently, a mapping
between OPM and CPN is needed. OPM modeling supports both object-oriented and process-oriented
paradigm. CPN supports state-transition-based execution semantics with discrete-event system simulation
capability, which can be used to conduct extensive behavior analyses and to derive many performance
metrics. The following section presents the fuzzy genetic architecture generation model.
SERC 2015-TR-021-4.4 – Volume 4
31
3 Architecture Assessment
3.1 Acknowledged SoS
Very few SoS are under the sort of tight central control typically attributed to military organizations by
nonmilitary members. On the continuum of degree of central control of SoS described in the SE Guide for
SoS, ranging from extremely tight to near anarchy, almost no SoS exist at either of the far ends of the scale
(Director Systems and Software Engineering, OUSD (AT&L), 2008). Most SoS exist near the center of
the scale, as acknowledged SoS; where there is some recognized central authority, but not complete,
centralized control, authority, or budget. Even in the military, authority is broadly delegated.
The definition of an acknowledged SoS is an overlay on existing component systems that have independent
existence outside the SoS. They have their own architectures, missions, stakeholders, and funding sources
(Bergey, 2009). Moreover, successful managers of acknowledged SoS understand that existing component
systems work best if they are perturbed as little as possible to meet the new requirements necessary to
contribute to the SoS. That is, if the component systems’ architectures are left to the systems engineering
and architecture professionals at the systems’ hierarchy level. It is in the best interest of the SoS manager
to coordinate and guide individual systems rather than attempt to issue commands to them. On one hand,
the component (existing, independently managed) systems have no need to accede to a SoS manager’s
requests/demands, nor to officially report through SoS management staff teams. On the other hand, there
may be numerous reasons to cooperate with the SoS manager’s desired changes. These include the
opportunity (or excuse) to break open their architecture to make those minor adjustments required to join
the SoS – this could allow an opportunity to fix some ongoing problems that do not on their own account
justify such ‘breakage.’ Another reason might be to stretch out the life of the program (and its contractors)
with fresh, new tasking, when otherwise the system would be approaching its end of life, decommissioning
and disposal. A system might already be planning to make changes to its architecture that could easily
accommodate the desirable SoS changes, but the new SoS opportunity could be a bonus source of funds or
the basis of further upgrades to make the system even more relevant in its own domain in the future.
This modeling framework is offered for an acknowledged SoS, where each component system is a fully
functioning, independently funded and managed system (represented by the SPO). A high-level (SoS level)
official envisions an opportunity to achieve a needed, new capability by using combinations of existing
systems in a new way, such that component systems can be left largely unchanged, or incorporated with
relatively minor changes. This acknowledged SoS approach is only useful if it can achieve the new
capability for either or both of the following:

A reduced cost compared to designing a separate, new “purpose built” system, and/or
SERC 2015-TR-021-4.4 – Volume 4
32

A reduced time to field the new capability.
Defense Secretary Rumsfeld famously said “…you go to war with the army you have, not the army you
might want or wish to have at a later time” (Suarez, 2004). Therefore, the concept of this acknowledged
SoS meta-architecture is that the major capabilities are built into the systems already, but small, quick
changes can be made to interfaces to enhance those existing capabilities when used in a cooperative manner.
The proposed architecture is a novel, binary system and interface architecture, which will be called the
meta-architecture throughout this document. It will guide the SoS architecture development through a wave
model evolution in capability over time (Dahmann, Rebovich, Lane, Lowry, & Baldwin, 2011) with
incremental improvements after it starts operation.
The new capabilities being sought in the SoS are achieved by combining mostly existing system capabilities
and/or adding new capabilities that arise in conjunction with other systems (i.e., through new interfaces)
(CJCSI 6212.01F, 12 Mar 2012). If simply throwing more systems (with their individual capabilities) at
the problem were sufficient, there would be no need to create the SoS. Therefore, all successful
acknowledged SoS architectures need to invest in the relationships (and interfaces) between the systems
comprising the SoS. Further, improvements in SoS attributes of interest such as performance, affordability,
reliability, etc., must also arise from the interfaces, otherwise there is no advantage over simply adding
individual systems’ capabilities. The nature of the acknowledged SoS implies that the SoS manager does
not have absolute authority to command system participation (nor interoperability changes), but must
“purchase” the component systems’ participation and modifications not merely with funds but also through
persuasion, the strength of the vision of the SoS, quid pro quos, the bully pulpit, appeals to good sense, and
whatever other means are legitimate and effective (Director Systems and Software Engineering, OUSD
(AT&L), 2008). Individual systems remain free to decide not to participate in the SoS development,
although that choice may cost those systems something, too. That cost would not come from the SoS
budget.
Alternately, some of the desired systems may not be available to the SoS during a particular operational
period of need. They may be down for maintenance, assigned to a higher priority mission, or geographically
distant on their day-to-day missions, therefore not able to contribute. Some capabilities and interfaces may
already exist, meaning they are free and fast for development, but they still may have a significant cost to
operate in a fielded SoS. This may be a reason not to ask them to participate. Some systems may have
enough capability that the SoS can tap their spare capability while they pursue their original tasks, so they
are essentially free to operate. Other capabilities may need minor (compared to a new start major program)
development efforts, either within a system or by developing a new interface with another system. The
SERC 2015-TR-021-4.4 – Volume 4
33
performance capabilities of the SoS will generally be greater than the sum of the capabilities of its parts
(Singh & Dagli, 2009). If this were not the case, there would be no need for the SoS. Changing the way
the systems interact with no modifications typically would not improve the SoS capabilities; it would simply
add more systems with their individual capabilities. Systems engineering in the overall context of the SoS
must address all the attributes of disparate groups of systems, as well as any crucial issues which affect
their behavior.
An instance of an acknowledged SoS might be a military command and control SoS that has transitioned
from a tightly knit group of a few systems to an acknowledged SoS that now includes many more previously
independent systems. This could be due to a change in the implementation or the importance of the missions
currently being supported, or of a change of importance and increase in complexities of potential crosscutting (new) SoS capabilities (Dahmann, Baldwin, & Rebovich, 2009). Another acknowledged SoS might
be a regional Air Traffic Control (ATC) system that crosses national boundaries. National ATCs are
independent, but find it in their interest to cooperate and interface with the regional ATC.
One way to develop better tools for predicting performance is to use proposed new tools on a very simple
model, where the results can be calculated independently. Exploring the working of a tool on simple models
can build confidence that the tool does what it is intended to do. Another way to build confidence is to
choose a model that can be extended in a very straightforward manner to more complex situations. Real
SoS may have very complex architectures, but at the most basic level, they may be boiled down to ‘are the
systems here or not, and which of them interface with each other.’ (If they do not interface with each other,
they are not a SoS, but simply a collection of systems.). This simple model of the SoS is really a metaarchitecture.
3.2 Proposed Modeling Approach
The model is a decision making aid for the SoS manager. It does not so much find the best solution to
designing a SoS, as help the manager explore the influence of the various constraints on the shape of a
reasonable solution. The method starts, as shown in Figure 7, from the SoS context and goals, using the
simplified binary meta-architecture including the full range of candidate systems and their interfaces.
Guided interviews uncover the SoS purpose, characteristics of candidate systems, key attributes that
characterize the SoS and methods for measuring the SoS in each of these attributes. The key attributes
generally lend themselves to linguistic characterization and ranges of measures that may be handled through
fuzzy logic. A subset of the characteristic capabilities of the component systems is discovered and
documented. Estimated costs, schedule and performance goals are established for the systems, interfaces,
and SoS as a whole. When the models are combined in the SoS model, it is ‘end-around’ checked for
SERC 2015-TR-021-4.4 – Volume 4
34
consistency, and typically needs to be adjusted in trial runs until it is self-consistent. The completed model
can assess any proposed SoS architecture within the meta-architecture for its KPAs, and provide an overall
assessment of the SoS for its ability to satisfy its numerous constraints. The constraints and their
combination may be described in the rules of a fuzzy inference system. Seeing the results of the
characterization of the KPAs, the other inputs, the combinations of systems and buildup of SoS capabilities
from the component systems is the most useful part of the method for the SoS manager and the stakeholders.
Variations of all inputs, assumptions, rules, etc. may be examined to identify the most influential
characteristics of the problem to insure the formulation of the problem and solutions are proper and helpful.
Stakeholders,
Values,
Preferences,
Resources
Systems,
Capabilities,
Costs,
Schedules
System &
Interface
MetaArchitecture
The SoS
Vision and
CONOPS
Modular
Capability
Combination
Algorithms
A Fuzzy
SoS
Evaluation
Model
Fuzzification
of SoS
Attributes,
Criteria, Rules
Figure 7. SoS evaluation method determines the fitness of each architecture, or (system + interface) SoS
arrangement, from the meta-architecture and domain dependent information
Figure 7 shows generalized steps for how to derive the set of attributes by which to evaluate the fitness of
a selected arrangement of the systems and their interfaces to provide required capabilities to the SoS.
Attributes desirable in the completed SoS architecture are elicited from stakeholders through linguistic
analysis of guided interviews with stakeholders. Having developed the attributes of interest, the possible
ranges of evaluation in each attribute are separated into an agreeable number of gradations of goodness or
badness (defining the membership functions for fuzzy sets) with some overlap due to ambiguities in
linguistic representation. The relative value of combinations of performances in each attribute is developed
into fuzzy rules through a series of stakeholder hypothetical tradeoff exercises. The multiple objective
optimization (MOO) problem of finding a good architecture to achieve acceptable values of the several
attributes of the SoS may be solved by finding an architecture that maximizes the single fuzzy SoS fitness
evaluation. The method regards the independent variable to be a chromosome with randomly positioned
ones and zeroes in it, and the dependent variable to be the SoS fitness or overall quality. Exploring the
architecture ‘space’ by evaluating a few hundred chromosomes with varying percentages of ones provides
SERC 2015-TR-021-4.4 – Volume 4
35
insight into whether a solution is possible. The fuzzy membership function edges and the rules may need
to be adjusted to find a set of tunable parameters that closes on itself, i.e., one that finds any solution. In
this case, a genetic algorithm approach is used to find a near optimal arrangement from the metaarchitecture if one exists. (It is certainly possible to design a problem for which no acceptable solutions
exist.) Combining all these steps into an organized method has not previously been applied to SoS. Because
of the many simplifications in the method, it is not expected to provide final solutions directly, but to give
insight into behaviors of possible real solutions in response to changes in rules, definitions of capabilities,
performance models, membership function shapes, environment, budgets, etc. that drive aspects of the
development and evolution of SoS.
3.3 SoS Attributes
Systems engineers call the areas of engineering design that require detailed knowledge and detailed analysis
tools ‘specialty engineering’ areas (INCOSE, 2011). These types of areas may also be called attributes of
SoS. Just as a measure of ‘reliability’ or ‘availability’ may require very detailed analyses at many levels
within a system design, but result in a single overall number to characterize the design in that specialty area,
the attributes of a SoS may require detailed analyses, but result in a single characterizing number. The
attributes or specialty areas are sometimes be called ‘-ilities;’ they are the subject of continuing, intense
research, especially in the area of SoS. Large lists of the attributes, many with several definitions, are being
catalogued and organized in several on-going efforts (Mekdeci, Shah, Ross, Rhodes, & Hastings, 2014)
(Ross, Rhodes, & Hastings, 2008). Just as that single number characterizing a system in a specialty area
may have numerous conditions limiting its applicability, the attribute measures characterizing the SoS will
probably be valid over a limited range of scenarios. To understand the implications of a particular measure,
one needs to know about all those conditions. Simply presenting that data in an intelligible format is a
challenge. Finally, since the specialty engineering areas typically have well-known algorithms and
procedures for evaluating combinations of subsystems that are easily extended to combinations of systems,
this effort attempts to deal with more appropriately SoS specific attributes. These SoS attributes might be
described as the ones which depend more heavily on the SoS architecture, which is detailed in the
chromosome.
A key feature of the attributes of either systems or SoS is that they frequently pull in different directions.
For example, improving speed may reduce range, both key attributes of overall technical performance.
Improving reliability may increase cost, thereby reducing acquisition affordability, but possibly increasing
operations and maintenance affordability. Numerous other candidate attributes of SoS exert pulls along
different directions in the multi-dimensional design or architecture space. The selected architecture must
satisfy the most unhappy stakeholder enough to avoid a veto. The stakeholders’ concerns are represented
SERC 2015-TR-021-4.4 – Volume 4
36
in the attributes selected to grade the value of the proposed architectures. The models used to evaluate the
attributes must be fully described and open to stakeholders so they can assure themselves the competition
among architectures is fair. The weighting between attributes must be open and fair as well.
Pitsko and Verma (Pitsko & Verma, 2012) describe four principles to make a SoS more adaptable. They
spend a large part of their time describing what adaptable means to various stakeholders, that different
stakeholders may continue to have slightly different concepts of what adaptability means, that the definition
is probably dynamic – changing over time, and that this ambiguity likely will apply to many other SoS
attributes. Schreiner and Wirthlin discuss a partial failure to fully model a space detection SoS architecture,
but learned a lot about how to improve the approach the next time they try it (Schreiner & Wirthlin, 2012).
The point is that people are not modeling according to a well-developed theory of SoS and then reporting
on the success or failure: they are still attempting to define the theory.
There are numerous approaches in the literature attempting to describe useful attributes, as well as how to
measure them, to help understand or predict the value of various architectural arrangements. These include
evolvability and modularity almost as complementary attributes (Clune, Mouret, & Lipson, 2013), while
Christian breaks evolvability into four components described as extensibility, adaptability, scalability and
generality (Christian III, 2004). Christian introduces the concept of complexity to overlay on these
attributes because a ‘too simple’ system cannot evolve. Kinnunen reviews at least four definitions of
complexity (Kinnunen, 2006) before offering his analysis of one related to the object process methodology
of Dori. Mordecai and Dori extend that model to SoS specifically for interoperability (Mordecai & Dori,
2013). Fry and DeLaurentis also discussed measuring netcentricity (interoperability within the SoS), noting
the difficulty of pushing the commonly used heuristics too far, because the Pareto front exists in multiple
dimensions (Fry & DeLaurentis, 2011), not just two as commonly depicted. Ricci et al. discuss designing
for evolvability of their SoS in a wave model and playing it out several cycles in the future, evaluating cost
and performance (Ricci, Ross, Rhodes, & Fitzgerald, 2013). Because SoS are complex, there are many
ways to look at them, with no dominant theory yet. This is why this direction of research is interesting and
worth pursuit (Acheson, et al., 2012).
Slightly different definitions for some of the SoS attributes were chosen for this work, especially for
flexibility and robustness. Lafleur used flexibility in the operational context of changing a system after
deployment (Lafleur, 2012), in a way in a different way than Deb and Gupta’s classic notion of robustness
(Deb & Gupta, Introducing Robustness in Multi-Objective Optimization, 2006) by shifting the optimum
(defined as narrowly better performance), rather than accepting lower performance across a wider front.
Singer used robustness in a different operational context (Singer, 2006), that of losing a node in a network,
SERC 2015-TR-021-4.4 – Volume 4
37
rather more like losing a system or an interface from the SoS as described here. Gao et al. discussed a
concept of robustness as the ability to withstand hacker attacks for networks of networks with varying
degrees of interconnectedness (Gao, Buldyrev, Stanley, & Havlin, 2011). The concept of the flexibility
attribute used here is more attuned to giving the SoS manager flexibility during development, when
selecting systems to supply all the desired capabilities. This falls right in line with some of the thinking of
recent discussions of resilience and sensitivity analyses, although they use the terms resiliency or robustness
for it (Smartt & Ferreira, 2012) (Yu, Gulkema, Briaud, & Burnett, 2011) (Jackson & Ferris, 2013). The
point is that there are many possible ways to describe the attributes of SoS, depending on circumstances,
organizations, and stakeholders’ preferences. Many of these ways depend directly on the architecture of
the system of interest. This dependency on interconnectedness fits into the framework of the architecture
meta-model used here. If an attribute does not depend on the SoS architecture in any way, then it will not
be useful to help select between potential architectures. It is not necessary that a useful ranking algorithm
be very accurate in its relationship to the measured attribute, only that it be pretty highly correlated to reality
and nearly monotonic in its ranking. That is sufficient to be useful in this approach.
For purposes of this research effort, the following key attributes for a family of ISR SoS were defined by a
group of subject matter experts (SMEs) during the SERC research task RT-44 (Dagli, et al., 2013):

Performance: Generally, the sum of the performance in required capabilities of the individual
component systems, with a small boost in performance due to increased coordination through
interfaces.
This is explained further in section Error! Reference source not found. on
netcentricity.

Affordability: Roughly the inverse of the sum of the development and operation costs of the SoS.
The performance delta above is applied in a different way to the affordability to change its shape
as a function of the number of interfaces, but also to be somewhat related to superior performance.

Developmental Flexibility: This is roughly the inverse of the number of sources that the SoS
manager has for each required sub capability. If a required capability is available from only one
component system, then the SoS manager’s flexibility is very small; they must have that system as
part of the SoS. On the other hand, if each capability is available from multiple systems within the
SoS, the manager has far more developmental flexibility.

Robustness: This is the ability of the SoS to continue to provide performance when any individual
participating system and all its interfaces is removed. Generally, having a very high performing
system as part of your SoS is a good thing; however, if that system is ever absent, the performance
SERC 2015-TR-021-4.4 – Volume 4
38
of the SoS may be degraded substantially. Therefore, it may be useful to have the contributions of
the individual system capabilities more widely dispersed, so that the loss of one system does not
represent as great a loss to the SoS (Pape & Dagli, 2013).
4 Proposed Method for Developing an Sos Evaluation Model
4.1 Use Case Model of the Domain Independent Method
The method for developing an architecture evaluation model of an SoS is the same regardless of domain.
The steps of the method are shown in the use case summary diagram of Figure 8. The SoS manager is a
key player, along with the SoS stakeholders, in forming a vision of the desired, acknowledged SoS
capabilities. Information from potential component systems also contributes to the SoS vision. The vision
of the SoS informs the model facilitator for exploring ways to model the desirable SoS attributes. This may
include what fraction of the system capabilities the SoS will require, defining the meaning of the attributes
and SoS missions in context, and establishing trade space limits to explore within the SoS meta-architecture.
Other inputs include estimated costs for modification and operation of the systems within the SoS, which
ideally would come from system stakeholders or SMEs, but usually start as estimates from the SoS
manager. The modeler works with the model facilitator and various SMEs to develop attribute evaluation
models that depend on the meta-architecture structure. These individual attribute evaluation models are
combined through a fuzzy logic rule based system to assess the overall SoS. With this assessment tool,
sample architectures represented in the meta-model may be evaluated for relative fitness as an entire SoS.
This fitness assessment tool is precisely what is needed by a GA to sort the better architectures within a
mutating population of trial chromosomes searching out the meta-architecture space. Sensitivity analyses
can be run by the modeling team in consultation with the SoS manager. The consensus SoS design may
then be presented by the SoS manager to the SPO managers for negotiation about any minor changes
required to join the SoS. The documentation developed during the modeling effort is even more important
for SoS explanation than for the legal and regulatory prescriptions of the DoDAF for official Program of
Record (POR) systems, because the SoS is outside the pre-existing design and training of the component
systems. Results of the negotiations also need to be well documented, because SMEs may provide
additional information to the negotiations, and stakeholders will want to know what capabilities their
systems agree to provide to the SoS.
SERC 2015-TR-021-4.4 – Volume 4
39
Figure 8. Use case diagram for developing a SoS Architecture; dashed lines are for information; solid lines are
‘responsible for,’ or active involvement; this portion of the effort excludes the NegotiateAchievableSoS use case
The list of data required, and the variable names used throughout this effort, for the generic SoS model is
shown in Error! Reference source not found.. This is a simplified, binary model of the systems’ presence
or absence from the SoS, and the non-directed interfaces between each pair of systems.
Table 2 List of SoS and component systems’ variable meanings within the meta-architecture
Name or description of variable
Name of SoS:
Number of potential systems:
Number of types of systems:
Names of system types:
Number of component capabilities:
Names of component capabilities:
Binary meta-architecture upper
triangular matrix:
Individual systems of the SoS
SERC 2015-TR-021-4.4 – Volume 4
Expression
sos
m
t
sys_typi : i ϵ {1,…t}
n
sys_capi : i ϵ {1,…n}
Aij : i ϵ {1,…m}, j ϵ {i,…m}
1
2
3
4
5
6
7
Aij : i ϵ {1,…m}, j =i , also simetimes
written as Aii , or simply Ai
8
40
Name or description of variable
Expression
Aij : i ϵ {1,…m}, j > i , and
Ajk = 1, Aik = 1, Aii =1, Ajj=1, Akk = 1 ,
where Akk is any communications system 9
C
10
Feasible interface
SoS main capability:
SoS performance in
capability:
its
large
Component capabilities of systems:
Performance of a particular system in
its key capability:
Estimated funding to add an interface
to an individual system:
Deadline for developing new
interface(s) on a system:
Estimated funding for operation of all
the participating systems during an
SoS operation:
Function describing the advantage of
close collaboration within a SoS as a
function of participating systems and
interfaces:
Function for combining system
capabilities into SoS capability C:
Number of individual attributes the
stakeholders want to evaluate the SoS
over:
Attribute names to evaluate SoS
architectures against
(e.g., cost,
performance, flexibility):
Number of gradations of each
Attribute that become Fuzzy
Membership Functions (FMF):
Fuzzy membership function names
within each attribute (granulation = a,
attribute = b):
PSoS
11
cij :: i ϵ {1,…n capabilities}, j ϵ {1,…m
systems} (binary matrix)
12
PiSs : i ϵ {1,…m}, Ss is each system
13
FIFiSs : i ϵ {1,…m}, Ss is each system
DiSs : i ϵ {1,…m}, Ss is each system
14
15
FOPiSs : i ϵ {1,…m}, Ss is each system
16
F (Aii, Aij, j≠i, ) : i ϵ {1,…m}, j ϵ {i,…m}
17
𝐶 = ∑𝑛𝑘 ∑𝑚
𝑖 𝐴𝑖𝑖 𝑐𝑘𝑖
18
g
19
Attk : k ϵ {1,…g attributes}
20
hk : k ϵ {1,…g attributes}
21
FMFab a ϵ {1,…hk gradations},
{1,…g attributes}
b ϵ
22
Boundab a ϵ {1,…h+1}, b ϵ {1,…g}
Fuzzy
membership
function a=1 is lower bound of universe of
boundaries (cross over points) for discourse, a ϵ {2,…h+1} is upper bound
each of b SoS attributes:
of FMF(a-1)b because Matlab can’t handle
matrix subscripts of zero
23
Overall SoS performance in an
( ∑𝑛𝑘 ∑𝑚
𝑖 𝐴𝑖𝑖 𝑐𝑘𝑖 ) * F (Aii, Aij, j≠i, )
Attribute
24
𝑛 𝑚
Ss
Total cost of developing and using an 𝑇𝐶 = ∑𝑗 ∑𝑖 𝐴𝑖𝑗 FIF𝑖
Ss
SoS
+ ∑𝑛𝑘 ∑𝑚
25
𝑖 𝐴𝑖𝑖 FOP𝑖
SERC 2015-TR-021-4.4 – Volume 4
41
Name or description of variable
Parameters for controlling the GA:
Mutation Rate
Number in Population
Number of Generations
Expression
Delta
P
G
26
Figure 9 shows an alternate view of the method as a process flow with emphasis on the individual steps,
without concern for who performs them.
4.2 Domain Independent Model Development
The SoS model includes all the information available to it from the sources gathered from the participants
identified in Figure 8, but it still must be cast in terms of the binary participation model of the metaarchitecture.
The first step, regardless of domain, is to identify the reasons for the SoS and the desired capabilities. The
SoS manager, and the facilitator, must always develop some background and vocabulary within the domain
so that meaningful discussions may be held among stakeholders. At this point one can begin to create
domain specific models of development schedules, costs, performance, and other attributes to be used in
evaluating an SoS architecture. The steps of the general method, however, are the same regardless of the
domain of the model as shown in Figure 9. Many modeling approaches in the literature assume the
architecture is already defined. This is similar to SE methods that assume the requirements are well defined
– nice and clean, but neither realistic nor adequate. The DoDAF, Ver. 2.02, to its credit, begins at the proper
place when it describes a domain independent six-step process for how to build an architecture model for a
large DoD system:
1. Determine Intended Use of Architecture
2. Determine Scope of Architecture
3. Determine Data Required to Support Architecture Development
4. Collect, Organize, Correlate, and Store Architectural Data
5. Conduct Analyses in Support of Architecture Objectives
6. Document Results in Accordance with Decision-Maker Needs (ASD(NII), 2010)
SERC 2015-TR-021-4.4 – Volume 4
42
Figure 9. Domain Independent Process Method for SoS Model building
This research extends the DoDAF system oriented model to SoS, adding detail on how to create, document
and use a similar model building process for an SoS. This will form a basis to help designers and managers
choose SoS architectures more wisely in the future.
The DoDAF viewpoints may be extended to the buildup of any SoS (military, civil or commercial) in nearly
exactly the same way it is intended to be used to document the vision, plans, capabilities, and workings of
a complex weapons system.
4.2.1 Establishing A Vision of The SoS
A SoS is by definition a group of independently capable systems, collaborating for a greater purpose, in
other words, to deliver a larger capability. Within some range, systems may be present in varying numbers
(or not at all) for a particular application of the SoS. The concept for the SoS must be articulated, captured,
and agreed to among the stakeholders in relation to this variability in participation. Some SoS, after being
developed, are on stand-by until called on to perform; others may implement a new capability that is actually
operating all the time. The ideal SoS provides an acceptable range of capabilities over a broad range of
compositions. Typically, the SoS manager (or management group) creates a vision statement to guide
development of the concept for the SoS. The vision includes a high level description of the goals of the
SoS, the potential types of participants and their capabilities, and the mission(s), threat(s), and a description
of how the SoS arrangement will improve existing capabilities, or provide new ones. The architecture
model of the SoS captures this vision but also provides the framework for decomposing the vision to
SERC 2015-TR-021-4.4 – Volume 4
43
manageable components as well as for building up the SoS out of legacy, new, or modified systems. The
SoS manager must start with information like that shown in Error! Reference source not found. that
corresponds to the DoDAF AV-1 Overview and Summary Information. (Corresponding roughly to Step 1
of the DoDAF 2.02 6-Step Architecture Development Process.)
Table 3 Example SoS evaluation model building questionnaire for creating an AV-1
Overarching
Purpose Of SoS
Unique Value Of
SoS
SoS Measures Of
Effectiveness
Issues That Might
Limit Effectiveness
SoS Features That
Might
Greatly
Increase
SoS
Effectiveness
Desired
Effectiveness
Rom
Budget:
Development
Rom
Budget:
Operations
Desired Schedule
Attributes Of The
SoS/Range Limits
For
Fuzzy
Evaluation
Capabilities
Of
Contributing
Systems
Component Legacy
Systems
A DoDAF OV-1 style description is often helpful; text should accompany it
What makes it better than simply adding another legacy system
How will you know how good it is?
Are changes of procedure necessary? Are there legal, regulatory, or
bureaucratic impediments to the creation of the SoS?
Can changes in procedures help? What is the innovation?
What would be considered really good, what’s adequate, what’s inadequate?
What might be ‘tradeable’ – Suggestions for fuzzy rules, e.g., is extra
performance worth more budget? Is extra flexibility worth more? How
much? Is lack of flexibility OK? etc.
How do they combine?
Type/
Capabilities
Category
Time to Costs $M- Notes
Develop/ Dev
and (Incompatibilities,
Equip
Ops
Constraints,
Characteristics, etc.)
etc.
4.2.1.1 An Operational View OV-1 High Level Operational Concept
The type of information that the SoS manager must have for the ‘Vision of the SoS’ is at least one example
of how the SoS would be used (or a list of examples with all their context). The example must discuss
expected participants in a rough picture (whether in graphics or text) of what the SoS will do in operation.
Initial draft of this information may be summarized in one or two pages as shown in Error! Reference
SERC 2015-TR-021-4.4 – Volume 4
44
source not found. for the All Viewpoint. This may be expanded to the OV-1 Operational Overview that
describes how the system will be used in slightly greater detail. It can be a graphic with accompanying text
showing the overall concept of use of the SoS as shown in Figure 10. Every term used in the descriptions
is defined in the All View 2, the Integrated Dictionary (AV-2). Major component systems, data or resource
exchanges, and effects are depicted iconically to present an overall, high level impression of how the SoS
may be dispatched, controlled, employed, and recovered, for example, as shown in Figure 10. For a SoS,
support is normally presumed to be supplied by the system operators in their continuing independent
missions, unless significant changes are imposed by the SoS configuration. Major mission segments are
shown are shown in the OV-1. The unifying SoS Integrated Dictionary (AV-2) is started with the OV-1,
building outward so that all terms, components, activities, and interfaces are defined in one place.
Tracking the sources of definitions is more necessary for an SoS than for a system. Differences in usage
of similar terms between component system stakeholders, model developers and operational users should
be flagged in the AV-2 by noting multiple definitions for the same or similar terms within their proper
contexts. This is significantly important in an acknowledged SoS because the nominally independent
component systems may have their own unique acronyms, terms or usages. The OV-1 establishes the scope
of the SoS. A key component of most SoS is mission flexibility – the ability to pivot to different postures
or missions as conditions change. A discussion of the range of likely activities of the SoS should be
included in the textual explanation of the OV-1, or even as multiple graphics for different missions if that
is part of the SoS charter. The OV-1 of a SoS must also include a discussion of priority between the SoS
mission versus the original and continuing missions of the systems, and should also include a generalized
discussion of how deeply the SoS architecture will be allowed to control the component systems. That is,
to what extent major interfaces enabling the SoS need to be controlled by (or at least communicated to) the
SoS manager through the architecture, versus where existing systems may continue control of their own
configurations. (An extension of Step 2 of DoDAF 2.02 to handle the SoS.)
SERC 2015-TR-021-4.4 – Volume 4
45
Figure 10. Sample OV-1 for Ballistic Missile Defense (ASN/RDA, 2009)
4.2.1.2 Collecting Descriptive Domain Information
Identifying the numerous stakeholders and their concerns, and gathering data about component systems and
their missions are key parts of developing the required domain knowledge to build a SoS. This process step
is the same regardless of domain. The method is domain independent, but the data gathered is now domain
dependent data. An initial rough level of knowledge is needed to allow a facilitator to make plans for
stakeholder interviews. Identification of key discussion points and possible areas of tradable concepts
within the early SoS construct are made at this point. However, until detailed discussions with the
stakeholders are held, one must not jump to conclusions about what is valuable or tradable, nor even what
the SoS framework looks like. Facilitated discussions with the stakeholders must draw out the following
features of the SoS:
•
Desired and composable capabilities, with expected or desired levels of performance
•
Concepts of operation for the desired new SoS capabilities
•
Likely scenarios for the employment of the SoS
•
Key performance parameters, with expected or desired levels of performance
•
Possible algorithms to combine component capabilities into SoS capabilities
•
Shared (as well as conflicting) judgments about potential evaluation criteria for SoS attributes
SERC 2015-TR-021-4.4 – Volume 4
46
•
Relative ranking of, and expected values of, attributes of the SoS by groups of stakeholders
•
Rough estimates of cost, schedule, and performance deltas for required minor changes to
existing systems to achieve desired SoS interfaces, or performance
•
And to get an overall ‘feel’ for how the SoS might work in practice.
An important part of developing a SoS architecture is to define all the component systems’ ownership,
missions, and priorities in case some capabilities must be ‘hijacked’ to support the SoS. Identifying all
affected stakeholders is the second part of the facilitation exercise. In the Pentagon, this is called ‘staffing
a position paper.’ Since SoS normally include systems both from multiple domains, as well as across a
range of stages of their life cycle, affected stakeholder identification requires careful and extensive
coordination. As the stakeholders are identified, they should be placed in a hierarchy of command, tasking,
and funding chains. This network is the basis of the Organizational Relationships Viewpoint (OV-4), which
serves as an excellent template for SoS in any domain, not only military ones. In normal DoDI 5000.02
system development, this is nominally within one service, and most of the relationships are obvious. In a
SoS, whether military, civil, or commercial, this effort may require special attention and care to achieve
successful coordination across major organizational boundaries (Director Systems and Software
Engineering, OUSD (AT&L), 2008). Major concerns of each stakeholder should be discovered, recorded,
tracked and updated over time, to aid in the coordination of initial tasking as well as changes to the goals
of the SoS over its lifecycle. An ideal place for this information is in the OV-4 part of the DoDAF.
Capability managers (or at least communities of interest) may be defined in the Capability Taxonomy
viewpoint (CV-2). These are cross referenced and mapped in the Capability Dependencies viewpoint (CV4) among the component systems and stakeholders. An ideal SoS would have a variety of ways to achieve
each of its required capabilities, perhaps with varying efficiencies. Having only a single way to achieve a
required capability is an exceedingly poor way to design a SoS; due to the independent nature of the
component systems’ missions, there is no guarantee that all possible systems will always be made available
to the SoS. The concerns expressed in terms of the US DoD programs are equally applicable to any complex
set of entrenched bureaucracies, such as companies in supply chains, divisions of corporations, or elements
of intergovernmental enterprises.
The desired capabilities of the SoS, as well as those of the component systems must be carefully defined
and accounted for both as a function of participating numbers of systems but also over time, as the SoS
plans to mature. An ideal architecture should handle not only incremental improvements over time as
capabilities evolve, but also a range of numbers of component systems. This accounts not only for
technological improvements but also for the availability of systems. The number of systems can change on
SERC 2015-TR-021-4.4 – Volume 4
47
any particular day due either to logistic availability or to higher priorities outside the SoS. The attribute
models of the SoS must be developed as functions of these variables. A SysML approach could allow
parametric definition of capabilities and effectiveness to be explicitly built into the model.
Other
approaches may require additional math models, which ideally will be based on architectural data from the
SoS model and the participation represented in the meta-architecture model.
Linguistic analysis (‘computing with words’) (Singh & Dagli, "Computing with words" to support multicriteria decision-making during conceptual design, 2010) of the stakeholder discussions allows one to
deduce a set of attributes, potential membership function shapes, and rules for combining attribute values
to create an overall SoS fitness evaluation. It may be necessary to iterate definitions of membership function
shapes and rules to get a reasonable set that works together. Working together here means that the attribute
measures do not overlap, nor correlate too well, among themselves (i.e., they are orthogonal, or nearly so).
If they were duplicative, it would tend to give too much weight to a subset of issues, instead of optimizing
over the broadest range of attributes.
Attribute characteristics and desirable ranges identified in the linguistic analysis are combined with fuzzy
evaluation and a set of rules to derive a meta-architecture based, overall fitness value from the participating
systems and interfaces. Level setting and model checking runs may need to be performed to insure the
story is self-consistent. Then the model can be sampled for stakeholders’ validation. These steps are as
shown in Figure 11 and Error! Reference source not found.
The following variables are identified for the SoS model development:
SERC 2015-TR-021-4.4 – Volume 4
48
Figure 11. Domain independent method to gather domain dependent data for SoS architecture model
development and analysis, with output of a ‘good’ architecture at the lower right
4.2.2 Understanding Stakeholders Views
A DoD acknowledged SoS is a very large, complex endeavor. SoS by definition create cross-functional
organizations. They bring together functions that may have been built up through separate, large systems
(and their program offices) that were developed over many years for many reasons, and only recently appear
to have the potential to improve the effectiveness of a process or create a new capability by the joining
together of these previously disparate systems. The new capability is highly desired, but not of overriding
importance in the acknowledged SoS.
Many stakeholders are inevitably involved in a SoS.
The
stakeholders include at least the following recursive classes of interested parties:

Component Systems (System Program Offices (SPOs) in the DoD or management agencies or
corporations, and all the single system stakeholders that they represent)

The SoS Manager or management agency

Payers/funders (typically Congressional Committees, DoD, and Services for military systems, but
also finance offices of other state or federal agencies, or CEOs of corporations)

Congressional committees/watchdog agencies

National or Theater Command Authority for military
SERC 2015-TR-021-4.4 – Volume 4
49

Users/beneficiaries of the SoS

Operators of the SoS

Competitors of the SoS

Enemies/threats/targets of the SoS

Allies of the U.S.

Press/public opinion
A similar list could be made for other types of SoS in the civil or commercial domains. Occasionally
individual stakeholders may be members of several groups simultaneously. Additional stakeholders may
be professional organizations, industry groups, standards organizations, municipalities, rulemaking
agencies, shareholders of corporations, charities, entrenched bureaucracies, unions, non-governmental
organizations, etc. ‘Due diligence’ is the term for doing the work to identify the stakeholders of a proposed
SoS, their degree of influence, and their level of concern about changes to their existing systems to make
the SoS work.
4.2.2.1 Relationships To Established Decompositions: Task Lists, Joint
Capability Areas, ISO Standards
When the domain is military, the Universal Joint Task List, the Service specific task lists, and the Joint
Capability Areas provide excellent vocabulary for defining the missions and capabilities required for
military tasks, independent of the systems used to achieve them (Joint Staff, 2010) (
[email protected], 2009). This vocabulary of capabilities and tasks (activities in UML or SysML
style modeling) is aggregated in the SoS AV-2 and model behavior definitions, so that each time that a
term, word or concept is used in the architecture, reports, or performance models, it is consistent and clear
to every stakeholder or participant. Other domains than military typically have manuals, corporate, industry
or government standards, scholarly, or professional guidance documents, or even textbooks to provide this
background of vocabulary and definitions. The ISO-10303 series of standards is another source of
guidance, particularly AP233, Systems Engineering Data Representation. In fact, there are usually so many
possible sources that it is highly advisable to maintain source tracking within the AV-2, with priorities
assigned to each source to prevent confusion when a term is overloaded by multiple definitions depending
on context.
The DoD task lists contain suggested very high level definitions of measures of effectiveness for evaluating
the performance of the capabilities. These are potentially valuable sources for determining membership
SERC 2015-TR-021-4.4 – Volume 4
50
function shapes and edge values. These are typically ‘improved upon’ for specific system solutions, but
they serve as an excellent starting point for drafting evaluation criteria for the SoS, especially in
performance. For the first pass through a fuzzy analysis, the membership function shapes are not too
important; triangular or trapezoidal shapes work well enough to get started. At the preliminary stage of
analyzing choices with crude, initial models, it is more important to get the terminology, ordering, and trade
space rules agreed to among the stakeholders than to have highly accurate membership function shapes.
Other ‘-ilities’ models may contribute to SoS attributes – reliability, availability, affordability, survivability,
flexibility, adaptability, agility, ability to be redirected, autonomy, precision, among many others (Mekdeci,
Shah, Ross, Rhodes, & Hastings, 2014), may be useful in evaluating characteristics of a particular SoS
meta-architecture. SoS attributes should be created through reasonable extrapolations of the component
systems’ capabilities to each area, with a small improvement factor for self-coordination. (If the SoS has
no advantage over the simple sum of component systems’ capabilities, then there is no need for the SoS –
simply send more systems to do the task.) By the time one has defined the vision, capabilities, stakeholders,
components, and measures of effectiveness, there should be enough of a basis to decide what additional
data will be required to develop the architecture evaluation models. (Step 3 of DoDAF 2.02.)
4.2.2.2 Capability Improvement of A Proposed SoS
The concept in the FILA-SoS for the buildup and even emergence of capabilities within the SoS is that
capabilities are brought to the SoS basically intact by the component systems as currently existing.
Typically, the SoS improves the sum of the individual component system capabilities by a change in the
way they work together to provide some unique or even greatly improved capability. Assume the interfacing
of those systems together in a new way can be made to improve performance by a small multiplier for each
connection. This is a typical approach introduced as the concept of netcentricity by Alberts, Garska and
Stein in the late 1990s (Alberts, Garstka, & Stein, 1999). This is equivalent to the small delta in
performance for each used-feasible interface. It is a greatly simplified notion to regard the performance
improvement to be a simple exponential function of the number of interfaces; there is undoubtedly a plateau
effect on the lower end whereby a minimum number of systems must be interfaced to be able to see the
effect. On the high end there is no doubt also a limit to improvement by the introduction of the concepts
of information overload, latency, and bandwidth limits. The simplifying assumption that more interfaces
is better is nevertheless quite reasonable over a broad range between the two extremes, especially since it
is limited to a small fraction for each interface.
4.2.2.3 Decomposition of Capabilities To Functions And Logical Views
The high level capabilities described in the AV-2 and OV-1 can be decomposed to lower level actions
and/or functions allocable to the potential component systems. This continues iteratively, exactly as in
SERC 2015-TR-021-4.4 – Volume 4
51
normal/standard systems engineering, until both a functional hierarchy and behavioral description can be
attributed to component systems. Some systems may need upgrades to be compatible with the SoS
architecture. The phasing and organization of the capabilities must be agreed to by both the systems and
the SoS manager, with performance, funding and schedules. The time phasing of capabilities development
is shown in the Capability Phasing Viewpoint (CV-3). If some systems’ capabilities were to be ready before
others and they could be used together, the timeframes would be noted and this would become a Capability
Vision Viewpoint (CV-1) that shows how the deployed capability is built up (ASN/RDA, 2009). Mapping
capability development to operational activities is shown with the Capability to Operational Activities
Mapping Viewpoint (CV-6). If some activities are not possible without the developing capabilities, then
there will be some operational changes over time, as well. Some functions may be logically grouped
because they can be reused to support other missions; some might be grouped because they are unique to
the SoS mission and configuration. Training and tactics may have to be developed to use new capabilities,
or even to get the systems to work together operationally if the systems don’t already do so in existing, joint
missions. These constraints may be shown in several of the capability views, but especially in the
Capability Dependencies (CV-4) and Capability to Organizational Development (CV-5) viewpoints.
The decomposition of capabilities to functions, and the aggregation of functions to higher levels of
abstraction, eventually to capabilities, are inverse processes.
Sometimes it is easier to decompose
downward, other times it is easier to aggregate upward. This depends on what information is available
when one starts the process. The important point is to fill in the Capability Taxonomy Viewpoint (CV-2),
so that it is complete and makes sense to all stakeholders (or at least is accepted by all) as the operative
definition for the SoS. The capability taxonomy is a subset of the Integrated Dictionary definitions, with
the addition of the item’s location within the hierarchy. Naturally, it is best to think through the implications
of the definitions for the whole lifecycle of the SoS. This also implies that the vision should be sufficient
to sustain a lifecycle view for the SoS, not merely the initial use of it. In practice, this sufficiency of vision
is rare.
Many SoS, in spite of being complicated arrangements, are also started as quick reaction responses to
environmental changes. Therefore, many SoS are short time frame exercises. “Make the required changes
quickly, and get them deployed!” is the prevailing attitude in this case. In this extreme, there is scant
thought given to planned upgrades, phased deployment, or building for growth. Here, all the changes or
new interfaces need to be developed in one time period (usually a budget period, called an epoch in FILASoS). When there is planning for development over several epochs, the ‘in-work’ interfaces are regarded
as not participating until their delivery epoch.
SERC 2015-TR-021-4.4 – Volume 4
52
The development of the Architecture Views must be an ongoing, continually updated process extending
throughout the program life cycle. The simplest cocktail napkin draft to the most detailed, data base driven,
multiply approved, fully vetted, graphical interface control definition should be documented within a
“model,” as the single source of data. Many of the remaining DoDAF viewpoints can be derived from basic
Operational Activity Model Viewpoints (OV-5b) activity diagrams if they contain both activities allocated
to swim lanes (denoting the various participants/elements/actors) and sequenced data flows between
elements. This is an addition to the basic (minimalist) definition of an activity diagram, but adding these
two items is an important step in defining how the SoS will operate. One can vary the amount of back up
text residing in each object in the model. This is dependent on the amount of detail required and available
at each stage of the architecture development. However, the AV-2 works most brilliantly if two conditions
are fulfilled: all participants assiduously define their terms in it, and a facilitator continually edits its
contents for clarity and consistency. Consistency is sustained if the rest of the documentation uses the AV2.
An architecture of the SoS will exist, whether or not it is defined, planned, or understood. It will be a far
more useful architecture (and a better SoS) if the architecture is developed intentionally, and well
documented. That the documentation might be in a standard, organized framework, and maintained
throughout the life of the SoS in a central repository, could make it useful to new hires, visitors, and the
engineers and managers attempting to upgrade, maintain, or use the SoS in the future. (Step 4 and 6 of
DoDAF 2.02.)
The Integrated Dictionary (AV-2) is the authoritative source for definitions of all elements of the
architecture or program descriptions. All acronyms, terms of art, and important concepts must be defined
there, and the source of the definition is maintained to give context for understanding arcane, duplicative,
or cross program usages (frequent occurrences in SoS). Example architectural element definitions are
shown in Error! Reference source not found..
Table 4 Example AV-2, Integrated Dictionary
Phrase
Acronym Definition
Computing
Infrastructure
Readiness
CIR
SERC 2015-TR-021-4.4 – Volume 4
Source
Provides the necessary computing infrastructure and DoD
related services to allow the DoD to operate according to v2.0
net-centric principles. It ensures that adequate
processing, storage, and related infrastructure services
are in place to respond dynamically to computing needs
and to balance loads across the infrastructure.
IEA
53
Phrase
Acronym Definition
Source
Concept of Operations
A clear and concise statement of the line of action chosen Std I/F UCS
by a commander in order to accomplish his mission.
Nato
STANAG
4586-3
Conceptual
Model
The required high-level data concepts and their DoDAF 2.02
relationships.
Data DIV-1
Condition
The state of an environment or situation in which a DoDAF 2.02
Performer performs.
Confidentiality
Assurance that information is not disclosed to DoD
unauthorized entities or processes.
v2.0
Configuration
A characteristic of a system element, or project artifact, INCOSE Sys
describing their maturity or performance.
Eng Hndbk
v3.2.1
IEA
4.2.2.4 Conducting Analyses of SoS Behavior
The SoS manager and development facilitator must at this point be doing some mental estimation of where
the required SoS capabilities could be obtained and for what cost. They must be designing questions to
elicit both responses and thought from the stakeholders about what could be of value in building the SoS.
The stakeholders extend both up and down the chain of responsibilities with the SoS manager in the middle.
Are there potential multiple sources for most required capabilities? Are there new ways of putting pieces
together in different ways to accomplish necessary tasks or functions? Do new technologies allow for
anything more easily that previously envisioned? What if something did work in a new way, how much
better would it be? What functional relationships could be described to evaluate the SoS? What ranges of
values of performance would be outstanding, pretty good, acceptable, poor, or awful? Answering these
questions will allow models to be built that will allow new designs of a SoS to be evaluated. (Step 5 of
DoDAF 2.02. Step 6 is documenting the viewpoints in a self-consistent model, which is done during all
the previous steps.)
4.2.3 Review of the Method Steps
Yet another way to look at this model development process is shown in Figure 12, using the binary
participation meta-architecture model as a starting point. A vision of the SoS, facilitated stakeholder
discussions, produces a plethora of linguistic terms and definitions. Linguistic analysis of these discussions
may be used to distill the SoS attributes that are important to the stakeholders. Linguistic analysis also may
SERC 2015-TR-021-4.4 – Volume 4
54
be used to establish ranges of values for the attributes that are considered excellent, good, or very bad, as
well as the strength of the stakeholders, feelings about each of these ranges. The modeler gets to play a
role at this point by writing trial algorithms that work on the meta-architecture to deliver an initial trial
measure of each attribute. These measures should depend significantly on the meta-architecture, because
they are used to evaluate the goodness of the architecture of the SoS. Given this information, establishing
membership functions to fit the fuzzy evaluation measures is relatively easy. Rules for combining attribute
valuations are also developed from stakeholder interviews and discussions. The rules are embodied in a
Mamdani fuzzy inference system or fuzzy associative memory in the form seen in Error! Reference source
not found. in section 4.4. The measures are used to improve the selection of the SoS architecture within
the genetic algorithm approach. The optimized architecture is then proposed for implementation and
negotiation between the component systems and SoS manager. The negotiations require a reasonably good
starting point to have any chance of success, and that is what this research is designed to provide – the
starting architecture for the agent based modeling part of the problem. The system negotiations are the key
to getting a realizable, implementable architecture for the SoS, because the systems cannot be forced into
the SoS in the case of an acknowledged SoS being analyzed in this effort.
•Systems and their interfaces are present (1), or not (0)
Binary Meta- •An instance of the meta-architecture is a "chromosome" representing one particular arcchitecture
Architecture
•Facilitated interviews to draw out input data and value judgments from key stakeholders
Stakeholder •Model building and validation iterations proceed toward consensus
Discussions
Evaluation
Model
Genetic
Algorithm
•Fuzzy SoS attributes created from stakeholder concerns, performance algorithms of collaborating systems,
and advantages from interfacing
•Fuzzy model can evaluate multiple attributes for each SoS chromosome to arrive at an overall SoS "fitness"
•Can explore large volumes of the potential architecture space
•Can optimize with respect to many attributes using overall fuzzy fitness
Figure 12. Given an Evaluation Model that depends on a chromosome from the meta-architecture, the genetic
algorithm can optimize within a multidimensional space of attributes
It is important to devise a method to visualize how the component systems’ capabilities (ci) of various
architecture instantiations come together to create the SoS capabilities (C). This helps during the level
setting exercises, but is vital to describing both the approach and the results to stakeholders as well. Finally,
there must be clear explanations of the limitations of the modeling approach. The numerous simplifications
SERC 2015-TR-021-4.4 – Volume 4
55
mean that the model is not likely to match reality very well in detail; the best this model can do is match in
terms of broad, general trends in comparing high level architectural impacts between different approaches
to constructing the SoS.
For every SoS there will be requirements for component system and capability descriptions. Capabilities
of each system are denoted by ci, and the way those capabilities are combined into the SoS capability C
must be described and captured in the model. When the information is gathered and organized, then the
domain specific model is described, but the fact that there must be some way to build up the required
capability as a module in the model is domain independent. For our meta-architecture, there may be sets
of capabilities from each system, a combination algorithm to describe how the SoS capability is built up
from the systems, and costs for development of either new capabilities or interfaces, with schedules and
costs for operations of the systems in the SoS. More detailed models could be used, for example if the cost
of discovering and codifying new doctrine or tactics, and training in the new configurations is known or
can be estimated. Other desirable attributes and ways of measuring and combining them may be discovered
during the stakeholder discussions; these may be added to the SoS evaluation, but there must be some
process such as this regardless of domain. Initial draft runs of the model may also lead the modelers to
changes and improvements in the model modules. This method of discovering the domain dependent data
that goes into the model is domain independent. It should work for any domain.
4.2.3.1 Choosing the SoS Key Attributes
The facilitators and model builders need some basic knowledge of the domain of the SoS. Without that,
they will not be able to ask intelligent questions of the stakeholders and SMEs. Conversely, if the
facilitators know too much about a domain, they may unconsciously pre-select a solution, biasing the way
they ask questions. An example questionnaire form for directing the interviews with stakeholders, is shown
in Error! Reference source not found.. This is only a very high level starting point; it should be adjusted
for any specific SoS application.
Questions such as those in Error! Reference source not found. are intended to elicit from the stakeholders
the key attributes (or key performance parameters (KPPs)) that they care about for the development and use
of the SoS. The questions are asked from the point of view, and with the intention, of developing relatively
simple evaluation algorithms that depend strongly on the participating systems and their interfaces and how
they interconnect in operation. At the initial stage of development, these algorithms may be fairly
approximate, using rough estimates. The goal is to have some kind of broadly feasible architecture with
which to begin the analysis leading to negotiations between SoS manager and individual systems for an
agreed to SoS. It is fully expected that individual systems’ performance, cost, schedule and other attributes
would be adjusted during negotiations. On the other hand, attribute evaluation algorithms are designed to
SERC 2015-TR-021-4.4 – Volume 4
56
be modular, so that if better models become available, they may be substituted in at any time. Well
documented, traceable information trails are invaluable when reviewing, improving, correcting, or
extending the models (or the SoS itself). These documented traces are well described by DoDAF style
views.
4.2.3.2 Domain Model Data
There is a great deal of information potentially available about the components of the SoS – each system
may be complex in its own right. The architect must find a better way to share SoS data with analysts and
stakeholders than dozens or hundreds of columns of numbers. Color coded and textured graphs, multiple
and rotatable viewpoints into multi-dimensional data, slices, smart filtering, correlations, time series
analyses, and animations may all be used to aid the understanding of the very large sets of data produced
by SoS modeling (Yi, Kang, Stasko, & Jacko, 2007). Both large scale trends and significant but tiny
artifacts in the data must be easily and quickly discoverable in the way the data is conveyed to reviewers.
One needs to be careful of the color palette chosen to display results, since different display or printer
devices may represent them differently, sometimes in surprising ways. Some in the audience will usually
be color blind to various degrees, as well. One must be careful not to assume that color coding data artifacts
makes them obvious to others. (For those interested, the website http://www.vischeck.com/examples/
simulates the way colorblind people see for people with normal color vision, and suggests alterations to
color palettes that allow more people see an image in a similar way).
4.2.4 Architecture Space Exploration
This modeling method uses a genetic algorithm (GA) approach to explore the architecture space. A
population of chromosomes is evaluated and sorted to select the better ones for propagation to future
generations with genetic modifications. The ones and zeroes in the chromosomes are generated at random
during the first generation. This is normal GA procedure. However, to avoid getting an average 50% ones
in the entire initial generation of chromosomes, a bias is applied to the random number generator so that
the probability of any bit in a chromosome of that generation being a one depends on the chromosome’s
position in the population. The number of chromosomes in a population for one generation of the GA is
variable. Typically a few tens to a few hundred chromosomes are used in the population in each generation.
For the initial generation, chromosome number 1 has only a few ones, with mostly zeroes. The last
chromosome in a population has mostly ones with only a few zeroes. Typically, low numbers of ones in
the chromosome (meaning participating systems and/or interfaces) is associated with lower cost and lower
performance. The other attributes could be better or worse, depending on their definitions. Higher numbers
of participating systems and interfaces are usually associated with higher performance and higher cost
(equation 24 and 25 in Error! Reference source not found.). Costs/affordability and overall performance
SERC 2015-TR-021-4.4 – Volume 4
57
are almost universally necessary SoS evaluation attributes. Since there is normally a desire for higher
performance and lower cost, one hopes for a sweet spot between the extremes, where you can get adequate
performance, and adequate affordability (nearly the inverse of cost) as well as acceptable values of the other
attributes.
A key feature of the method is to do an exploration of the architecture space with a few thousand sample
chromosomes, which cover a large range of participating systems and interfaces. Each of the chromosome’s
attribute evaluations is plotted against the membership functions for that attribute. The membership
function shapes and/or the algorithms for evaluating the attributes may need to be adjusted several times in
an iterative process that may include discussions with stakeholders to arrive in acceptable SoS model. This
is explained further in section 4.6
The meta-architecture and associated data model being proposed so far contains many features that mimic
real-life:

There may be multiple copies of the same system

There may be slight differences between the otherwise similar systems

Each system may have multiple capabilities

There is a minimum number of component capabilities required to make up the SoS capability.
If a proposed architecture does not have the minimum capabilities, a penalty is tacked on to its performance,
to enhance the chances of discarding its chromosome in the fitness comparisons at each generation of the
genetic algorithm. No population member or bit position is pre-selected to be discarded before evaluating
it for all attributes.
4.3 Individual Systems’ Information
4.3.1 Cost, Performance and Schedule Inputs of Component Systems
The models used here treat the cost of developing a new capability or adding an interface separately from
the cost of operating the system during a deployment of the SoS. In real life these costs are potentially paid
for from different pots of money, such as acquisition vs. operations budget lines in DoD, or current vs.
future funds. The development cost is normally a onetime cost, while the operations cost of the system is
a continuing cost each time the SoS is called into action. Performance enhancement is normally on the
order of a few percent for the modification requested to fit into the SoS. For adding an interface, it might
require a new radio and antennas to be installed on a vehicle, or extending the software data base of
SERC 2015-TR-021-4.4 – Volume 4
58
messages that can be handled by an existing system on a vehicle. There can be significant costs for a minor
modification to accomplish retesting of functions that might be affected by any changes to fielded systems
(called regression testing), in addition to testing of the change itself. Whatever the change, in addition to
time to develop and test the change, the system hosting it will be ‘down for maintenance’ during the
installation of the change. The time to develop, install and test a change is the development time. This is
generally one epoch, or time period, in the wave model. If a capability already exists, such as ability to use
a specific radio on a platform, the development cost and time for that system for that capability will be zero.
However, development of the other end of the interface on a different system may still be required and will
count toward the cost of the interface. Some complex modifications might take two or more epochs to
develop. In this case, since the development is not complete at the end of the first epoch, it is as if the
system chose not to participate, because it delivers no capability yet. However, one is still spending funds
on that development, for which one receives nothing until the development is complete. The bottom of
Error! Reference source not found. shows a simplified template to gather the estimated individual system
cost and performance data.
4.3.2 Membership Functions
Membership functions (MF) map the fuzzy values to the real world values and show the fuzziness of the
boundaries between the granularities or grades within each attribute. The Matlab Fuzzy Toolbox has a
number of built in shapes for membership functions. Triangular, trapezoidal, and the Gaussian smoothed
corners of trapezoidal shapes are available among others; only the Gaussian rounded trapezoidal shape
shown in Figure 13 was used in this analysis. It is very common when evaluating large projects to have a
band of acceptability for each grade in each attribute. A familiar example is the Contractor Performance
Assessment Reporting System (CPARS). It assigns one of five colors to a series of common measures of
project status. It is also common to have multiple reviewers provide a grade in each area, which is then
averaged to get the final grade (Department of the Navy, 1997). This is intended to avoid the issue of
unconscious bias or error of interpretation of the data by a single reviewer on a borderline issue. This
process is very similar to the mathematically more precise fuzzy logic process. Other MF shapes show
similar characteristics, but the nonlinearities in the output surface display the concepts better with the
slightly rounded MF shapes shown in Figure 13. All the variables in the Fuzzy Tool Box are scaled the
same way. In real space, a further scaling is required to the individual variables. The MFs cross each other
at the 50% level between each of the numbers in the granularity scale from 1 to 4. For this fuzzy inference
system (FIS), 1 = Unacceptable, 2 = Marginal, 3= Acceptable, and 4 = Exceeds (expectations) for each
attribute:
Performance, Affordability, (Developmental) Flexibility, and Robustness.
There is no
requirement that the scaling be the same for different attributes. In fact, the Matlab Fuzzy Tool Box allows
the MFs to be scaled to real values, and it might have been clearer to use that facility, but the graphical user
SERC 2015-TR-021-4.4 – Volume 4
59
interface (GUI) for changing the values is rather tedious, so a method to apply the scaling outside the GUI
was developed. The process of translating real values to fuzzy values is called fuzzification or fuzzifying.
Multiple criteria are combined through the rules in fuzzy space, and the output fuzzy value is de-fuzzified
to a crisp value for the SoS assessment. In this fuzzification scheme, values are rounded to the nearest
integer value for each fuzzy gradation. In the example in Figure 13, a fuzzy value of 2.35 would round to
2, and not coincidentally, would fall on the sloping line for Marginal membership at a value of about 65%,
higher than on the line for Acceptable membership of about 35%. The next section discusses how the real
values are mapped to the fuzzy scale.
Figure 13. Matlab Fuzzy Toolbox printout of membership function shapes used in this analysis
4.3.3 Mapping Attribute Measures to Fuzzy Variables
The generic membership function range is the ‘universe of discourse.’ This typical range, shown in section
4.3.2, must be mapped to the real world values of the domain specific SoS. The mapping can be done inside
Matlab so that Figure 13 would be scaled in real units, but that requires working in a tedious GUI. It can
also be done by mapping the key shape points of the scaled MF to the real world values. In a real problem,
this mapping of ranges for each attribute would come from the problem definition and the stakeholders’
beliefs and desires discovered during the model building step of the method. Examples could be estimated
values for cost of the SoS, or performance in terms of square miles searched per hour, or tons of freight
delivered per day in another type of problem. The probability of success, or the number of shipments, or
other attributes would have desired thresholds that define the levels of performance in each attribute, such
as: unacceptable, marginal, acceptable, or exceeds expectations in a four part granularity for each attribute.
The judging criteria may take on a wide variety of terminology and of forms, depending on the domain.
Any degree of granularity is possible. An even number of gradations were chosen in this instance to avoid
SERC 2015-TR-021-4.4 – Volume 4
60
the possibility of an evaluation question being answered in the middle. Odd numbers of gradations tend to
allow stakeholders to answer too many valuation questions disproportionately in the middle during the
interview process, while even numbers of gradations force the choices to be above or below average. This
depends on one’s problem and particular stakeholders, of course.
Figure 14. shows a typical mapping between real world values on the left, and the fuzzy variable on the
bottom. Note that there is no requirement for the mappings to be linear. Figure 15shows affordability and
robustness mapped to their fuzzy values. All the attribute values need to be a matched set, for a matched
set of attribute models. In this case, robustness depends on the range of the values of performance.
Therefore, if the maximum performance doubles due to a change in the model, then the real world
robustness map would need to change as well. The real world values for affordability are dollars, and
robustness is the maximum loss of performance when removing any system, but they are mapped as
negative values here, because less is better. This allows the fuzzy attributes to be plotted as monotonically
increasing. Minor kinks in the mapping lines show that the slopes of the membership function maps do not
need to be constant.
Performance
0.6
0.5
Likelihood of detection/day
0.4
0.3
0.2
0.1
Unacceptable
Marginal
Adequate
Exceeds
0
1
1.5
2
2.5
3
3.5
4
Figure 14. Map from fuzzy variable on horizontal axis to probability of detection on left
SERC 2015-TR-021-4.4 – Volume 4
61
ROBUSTNESS
AFFORDABILITY
0
0
-50
1
1.5
2.5
0
1
4
-60
-100
2
-190
-200
4
-0.15
-0.3
-0.4
-155
3
-0.2
-110
-150
-250
3.5
-0.45
-0.6
-250
-0.75
-0.8
-300
Figure 15. Attribute values, mapped to fuzzy variables
4.3.4 Non-Linear Trades in Multiple Objectives of SoS
Fuzzy logic can be used to fit highly nonlinear surfaces even with a relatively small rule base. The
commonly cited problem of dimensionality for fuzzy logic systems in fitting arbitrarily large input sets
(Gegov, 2010) does not arise in this problem because the number of inputs are small – limited to the KPAs
of the SoS design problem. The combination of membership function shapes and combining rules allows
one to fit quite nonlinear surfaces in the several required dimensions of this problem. Furthermore, the
input variables are generally monotonic, increasing in value from the fuzzy value of ‘worst’ to ‘best.’ All
the membership functions used in this effort (input and output) have been scaled from 1 to 4 for simplicity
of display in this document, but that scaling is purely arbitrary. The actual scaling is through linguistic
variables discovered through the interactions of the facilitator, SMEs, and stakeholders. They are typically
terms such as “very bad,” “good,” “excellent,” etc. For most attributes, there is a further mapping of the
linguistic terms, such as ‘excellent affordability is a cost between $8M and $10M,’ or ‘acceptable
affordability is cost between $10M and $12M.’ If the attribute evaluation elements can be categorized in
such fuzzy terms as this, then relatively simple rules for combining them can result in a straightforward
overall SoS evaluation from the resultant fuzzy inference system or fuzzy rule based system.
SERC 2015-TR-021-4.4 – Volume 4
62
Figure 16. Example of nonlinear SoS fitness versus Affordability and Performance, based on membership function
shapes and combining rules
4.4 Combining SoS Attribute Values Into an Overall SoS Measure
A Mamdani fuzzy inference system allows the combination of as many input attributes as desired (Fogel,
2006). Each attribute is equivalent to an objective or dimension in a multi-objective optimization problem.
Gegov expanded this concept to include networks of fuzzy systems, to cover deep and complicated
problems with many dimensions (Gegov, 2010), and uncertainties extending to Type II fuzzy sets.
Nevertheless, if rules of the form discussed below (which are symmetrical), are combined with rules of the
form ‘if attribute one and attribute four are excellent, but attribute five is marginal, then the SoS is betterthan-average,’ etc., which allows for asymmetry or non-uniform weighting among attributes, then very
complex evaluation criteria may be described for the SoS. Using membership function shapes other than
those shown in Figure 13 also allows considerable tuning of the mapping of input attribute values
(depending on the SoS architecture or chromosome structure in the model) to the output of the overall SoS
quality or fitness.
A Mamdani Type I fuzzy rule set may also be called a Fuzzy Associative Memory (FAM) to combine the
attribute values into the overall SoS fitness score. Attribute measures are converted to fuzzy variables from
the mappings explained in section 4.3.2, the rules are followed to form a fuzzy measure for the SoS
architecture (represented by a chromosome). That measure may be de-fuzzified back to a crisp value for
final comparison in the GA through an equivalent mapping in the output space. The rules should be kept
simple for two reasons: primarily it is easier for the analyst to understand and to explain them to the
stakeholders, but also because a few rules within the fuzzy logic system can be very powerful in defining
the shape of the resulting surface. Still, some sensitivity analysis can be done on the rule sets, and results
SERC 2015-TR-021-4.4 – Volume 4
63
of minor changes in the rules may be displayed for comparison, all other things being kept the same. Rules
are typically of the form: ‘if all attributes are good, then the SoS is superb,’ ‘if all attributes except one are
superb, then the SoS is still superb,’ ‘if any attribute is completely unacceptable, then the SoS is
unacceptable.’ A dozen or so of these rules can give an excellent estimate of the stakeholders’ intentions,
including significant nonlinearities and complexity (Gegov, 2010). The Mamdani FIS allows satisfying
two contradictory rules simultaneously by simply including them both in the calculation of the resultant
output value.
The linguistic form of some of these rules may be easier to express than the mathematical form. For
example, ‘if any attribute is unacceptable, then the SoS is unacceptable’ can be expressed linguistically as
a single sentence, but mathematically a separate rule for each attribute is tested alone to implicate the
unacceptability of the SoS. If the rule can be expressed as a single sentence linguistically, as in Error!
Reference source not found., it will be counted as only one rule. The rules come out of linguistic analysis
of the stakeholder interviews, with some normative smoothing by the facilitator. At worst, if consensus
cannot be reached on a rule statement among the stakeholders, a version of the analysis with the rule
expressed both ways can be compared for sensitivity to that rule. This approach can also help explain the
issue to the stakeholders.
Table 5 Example of a few powerful Fuzzy Inference Rules for combining attribute values
Five Plain Language Rule
If ANY single attribute is Unacceptable, then SoS is Unacceptable
If
ALL of the attributes are Marginal, then the SoS is Unacceptable
If
ALL the attributes are Acceptable, then the SoS is Exceeds
If (Performance AND Affordability ) are Exceeds, but (Dev. Flexibility and Robustness) are Marginal,
then the SoS is Acceptable
If ALL attributes EXCEPT ONE are Marginal, then the SoS is still Marginal
RULES for ISR
If Clauses
If all attributes are Unacceptable
If all attributes are Marginal
If all attributes are Acceptable
If Performance and Affordability are
Exceeds
If all attributes Except Robustness
are Marginal
SERC 2015-TR-021-4.4 – Volume 4
and Flexibility and Robustness are
NOT Unacceptable
and Robustness is NOT Exceeds
Then
the SoS is Unacceptable
the SoS is Unacceptable
the SoS is Exceeds
the SoS is Acceptable
the the SoS is Marginal
64
If all attributes Except Flexibility are
Marginal
If all attributes Except Affordability
are Marginal
and Flexibility is NOT Exceeds
and
Affordability
Unacceptable
is
the the SoS is Marginal
NOT
the SoS is Acceptable
The Surfaces look like this:
With Performance vs. Robustness looking the same as the figure on the right for Flexibility.
RULES for SAR
If Clauses
If all attributes are Unacceptable
If all attributes are Marginal
If all attributes are Acceptable
If Performance and Affordability are
Exceeds
If all attributes Except Robustness
are Marginal
If all attributes Except Flexibility are
Marginal
If all attributes Except Affordability
are Marginal
and Flexibility and Robustness are
Marginal
and Robustness is Acceptable
and Flexibility is NOT Exceeds
and
Affordability
Unacceptable
is
Then
the SoS is Unacceptable
the SoS is Unacceptable
the SoS is Exceeds
the SoS is Acceptable
the the SoS is Marginal
the the SoS is Unacceptable
NOT
the SoS is Acceptable
Affordability and performance look the same as the previous rule set, but Flexibility and Robustness both
change shape:
SERC 2015-TR-021-4.4 – Volume 4
65
Flexibility depended on the distribution of capabilities among systems (set by the input domain data sheet,
which varied between domains), Robustness was the best remaining performance function after removing
each system - all it did was use the performance function repeatedly after removing each system
successively. Affordability also depended on the input domain data, also adjusted independently for each
domain.
4.5 Exploring the SoS Architecture Space with Genetic Algorithms
Having developed a method of evaluating architectures based on presence or absence of any combination
of systems and interfaces within the meta-architecture, this evaluation may be used as the fitness measure
for selection for propagation to a new generation within an evolutionary algorithm.
One class of
evolutionary algorithm is the genetic algorithm (GA). The key feature of a GA approach is to evaluate the
overall fitness of a series of chromosomes in a ‘population.’ One then sorts the chromosomes by their
fitness, and proceeds to a next generation through mutations, crossovers, or ‘sexual reproduction’ of a
fraction of the better fitness chromosomes in that generation. Mutation rates, crossover points, special rules
for certain sections of the chromosome (genes), or deciding which parents are combined, can all be varied
as part of the GA approach.
The GA first generation starts with a population of random arrangements of chromosomes built from the
meta-architecture, which spans the search space, then sorts them by fitness. A fraction of the better
performing chromosomes is selected for propagation to the next generation through mutation and/or
transposition. A few poorly performing chromosomes may also be included for the next generation, to
avoid the danger of becoming stuck on a purely local optimum, although proper selection of mutation and
transposition processes can also help avoid this problem.
SERC 2015-TR-021-4.4 – Volume 4
66
4.6 Combining the Fuzzy Approach with the GA Approach
In order that the GA work with any string of bits within the meta architecture, the algorithms for evaluating
each attribute must work for any string of bits. The results of individual attribute evaluations may take on
a large range of values. When the desired and tradable values of the attributes, and the algorithms for
evaluating them, are determined from the SoS stakeholder interviews, the range of values of each attribute
is pre-determined. The entire range of values is the ‘universe of discourse.’ In each dimension or attribute,
the entire range is mapped contiguously to the granularity described by the membership functions. There
is no guarantee that any arrangement of systems and interfaces will be found to be acceptable. Because this
effort was to develop and explore the method, and the example SoS were largely fictional, all the model
parameters could be adjusted to find examples that would work. The key to this adjusting process was to
plot the attribute evaluations against the number of ones in the chromosome.
Biasing the random number generator to produce a population of chromosomes with varying numbers of
ones allowed an exploration of chromosomes from various regions of the meta-architecture. By iterating
adjustments of the attribute membership function edges against a population of randomly generated (but
biased in the number of ones) initial populations of chromosomes, an acceptable picture of the SoS behavior
could be determined.
Figure 17. Exploring the meta-architecture - 25 chromosomes, 22 systems, Example 1
SERC 2015-TR-021-4.4 – Volume 4
67
Figure 18. Exploring the meta-architecture to map membership function edges, Example 2
When a few hundred chromosomes are present in the exploratory population, one can get a very good idea
of the shape of the behavior of the meta-architecture space as a function of the number of interfaces between
systems of the SoS, shown in Figure 19. More systems and interfaces generally leads to more of all the
attributes: performance, flexibility, robustness, but to more cost as well (= less affordable). However, one
can also see that the trends are noisy, and not perfectly correlated in the ISR model shown in Figure 19.
The exploration phase allows the setting of the membership function edges to take advantage of the
variability in the evaluation functions to drive the GA search toward regions that look more likely to
produce a decent compromise from among the competing attributes.
Figure 19. Exploring large populations to set the membership function edges
SERC 2015-TR-021-4.4 – Volume 4
68
One needs to be in a reachable region of the SoS attribute space, or the universe of discourse, defined by
the membership function edges in the fuzzy inference system. It is of little value to have an architecture
that produces $100M solutions when the only acceptable value is less than $50M. Therefore, some level
setting of expectations, tuning of algorithms, and of the input domain data may all be necessary to reach a
reasonable “space” within which to attempt optimization with the GA. This is the function of the
exploration phase of the process.
5 Concluding Remarks
The research presented has been based on Type-I fuzzy sets. There are certain benefits of using a type-2
Fuzzy Logic system for our particular research problem. Type-2 fuzzy sets allow us to handle linguistic
uncertainties, as typified by the adage “words can mean different things to different people” (Karnik,
Mendel & Liang, 1999). The benefits achieved are mentioned as follows: During collection of rules by
surveying experts, it is very likely to get diverse answers from each expert, based on the locations and
spreads of the fuzzy sets associated with antecedent and consequent terms. This leads to statistical
uncertainties about locations and spreads of antecedent and consequent fuzzy sets. Such uncertainties can
be incorporated into the descriptions of these sets using type-2 membership functions. In addition, experts
often give different answers to the same rule-question; this results in rules that have the same antecedents
but different consequents. In such a case, it is also possible to represent the output of the FLS built from
these rules as a fuzzy set rather than a crisp number. This can also be achieved within the type-2 framework.
Finally the uncertainty during training also exists on both the antecedents and consequents. If we have
information about the level of uncertainty, it can be used when we model antecedents and consequents as
type-2 sets. This is presented in the Volume 12.
SERC 2015-TR-021-4.4 – Volume 4
69
Cited and Related References
[email protected]. (2009, Jan 12). Joint Capability Areas. Washington DC.
Abo-Sinna, M. A., & Baky, I. A. (2007). Interactive balance space approach for solving multi-level multi-objective
programming problems. Information Sciences, 177, 3397–3410.
Abraham, A., & Jain, L. (2005). Evolutionary Multiobjective Optimization. Evolutionary Multiobjective Optimization,
1-6.
Acheson, P. (2010). Methodology for Object Oriented System Architecture Development. IEEE Systems Conference.
Acheson, P., Dagli, C., & Kilicay-Ergin, N. (2013, March). Fuzzy Decision Analysis in Negotiation between the
System of Systems Agent and the System Agent in an Agent Based Model. International Journal of Soft
Computing and Software Engineering [JSCSE]. doi:10.7321/jscse
Acheson, P., Pape, L., Dagli, C., Kilcay-Ergin, N., Columbi, J., & Haris, K. (2012). Understanding System of Systems
Development Using an Agent-Based Wave Model. Complex Adaptive Systems, Publication 2. 12, pp. 21-30.
Washington D.C.: Procedia Computer Science.
Agarwal, S., & Dagli, C. H. (2013). Augmented Cognition in Human–System Interaction through Coupled Action of
Body Sensor Network and Agent Based Modeling. Conference on System Engineering Research, (pp. 2028).
Agarwal, S., Pape, L. E., & Dagli, C. H. (2014). GA and PSO Hybrid with Type-2 Fuzzy Sets for Generating Systems
of Systems Architectures. Procedia Computer science.
Agarwal, S., Saferpour, H. R., & Cihan, C. H. (2014). Adaptive Learning Model for Predicting Negotiation Behaviors
through Hybrid K-means Clustering, Linear Vector Quantization and 2-Tuple Fuzzy Linguistic Model.
Procedia Computer Science, 36, 285-292.
Agarwal, S., Wang, R., & Dagli, C. H. (2014). Executable Architectures Using Cuckoo Search Optimization Coupled
with OPM and CPN-A Module: A New Meta-Architecture Model for FILA SoS. In Complex Systems Design
& Management (pp. 175-192). Paris: Springer International Publishing.
Ahn, J.-H. (2012). An Archietcture Description method for Acknowledged System of Systems based on Federated
Architeture. Advanced Science and Technology Letters 05.
Alberts, D. S. (2011). The Agility Advantage: A Survival Guide for Complex Enterprises and Endeavors. Washington
DC: Center for Advanced Concepts and Technology.
Alberts, D. S., Garstka, J. J., & Stein, F. P. (1999). Network Centric Warfare: Developing and Leveraging Information
Superiority, 2nd Edition. Washington DC: C4ISR Cooperative Research Program.
Alfaris, A. (2009). The Evolutionary Design Model (EDM) for the Design of Complex Engineered Systems: Masdar
City as a Case Study. Massachusetts Institute of Technology.
Alves, M. J., Dempe, S., & Júdice, J. J. (2012). Computing the pareto frontier of a bi-objective bi-level linear problem
using a multiobjective mixed-integer programming algorithm. Optimization, 61(3), 335–358.
Amberg, M. (1996). Modeling Adaptive Workflows in Distributed Environments. Proc. of the 1st Int. Conf. on
Practical Aspects of Knowledge Management. Basel.
Anandalingam, G., & Friesz, T. L. (1992). Hierarchical optimization: An introduction. Annals of Operations Research,
34(1), 1-11.
Arnold, A., Boyer, B., & Legay, A. (2012). Contracts and Behavioral Patterns for System of Systems: The EU IP
DANSE Approach.
SERC 2015-TR-021-4.4 – Volume 4
70
Arnold, A., Boyer, B., & Legay, A. (2012). Contracts and Behavioral Patterns for System of Systems: The EU IP
DANSE Approach.
ASD(NII), D. (2009). DoD Architecture Framework Version 2.0 (DoDAF V2.0). Washington DC: Department of
Defense.
ASD(NII), D. (2010). DoD Architecture Framework Version 2.02 (DoDAF V2.02). Washington DC: Department of
Defense.
ASN/RDA. (2009). Net-Ready Key Performance Parameter (NR-KPP) Implementation Guidebook, Ver. 1.
Washington DC: Department of the Navy.
Bac, M., & Raff, H. (1996). Issue-by-issue negotiations: the role of information and time preference. Games and
Economic Behavior, 125-134.
Baky, I. A. (2009). Fuzzy goal programming algorithm for solving decentralized bi-level multi-objective
programming problems. Fuzzy Sets and Systems, 160, 2701–2713.
Baky, I. A. (2010). Solving multi-level multi-objective linear programming problems through fuzzy goal
programming approach. Applied Mathematical Modelling, 34, 2377–2387.
Bergey, J. K. (2009). US Army Workshop on Exploring Enterprise, System of Systems, System, and Software
Architectures.
Blanchard , B. S., & Fabrycky , W. J. (2010). Systems Engineering and Analysis. Upper Saddle River, NJ: Prentice
Hall.
Bonabeau, E. (2002). Agent based Modeling: Methods and Techniques for Simulating Human Systems,. Proceedings
of the National Academy of Sciences, Vol. 99, 7280-7287.
Bouleimen, K., & Lecocq, H. (2003). A new efficient simulated annealing algorithm for the resource-constrained
project scheduling problem and its multiple mode version. European Journal of Operational Research,
149(2), 268-281.
Bradley, S. P., Hax, A. C., & Magnant, T. L. (1977). Applied Mathematical Programming. Addison-Wesley.
Brucker, P., Drexl, A., Möhring, R., Neumann, K., & Pesch, E. (1999). Resource Constrained Project Scheduling:
Notation, Classification, Models, and Methods. European Journal of Operational Research, 112(1), 2-41.
Cai, Z., & Wang, Y. (2012). Combining Multiobjective Optimization with Differential Evolution to Solve Constrained
Optimization Problems. IEEE Transactions on Evolutionary Computation, 1(16), 117-134.
Cara, A. B., Wagner, C., Hagras, H., Pomares, H., & Rojas, I. (2013). Multiobjective Optimization and comparison
of Nonsingleton Type-1 and Singleton Interval Type-2 Fuzzy Logic Systems. IEEE Transactions on Fuzzy
Systems, 21(3), 459-476.
Chattopadhyay, D., Ross, A. M., & Rhodes, D. H. (2008). A framework for tradespace exploration of systems of
systems. 6th Conference on Systems Engineering Research. Los Angeles CA.
Christian III, J. A. (2004). A Quantitative Approach to Assessing System Evolvability. Houston: NASA Johnson Space
Center.
CJCSI 6212.01F. (12 Mar 2012). Net Ready Key Performance Parameter (NR KPP). Washington DC: US Dept of
Defense.
Cloutier, R. J., Dimario, M. J., & Polzer, H. W. (2009). Net Centricity and System of Systems. In M. Jamshidi, System
of Systems Engineering (pp. 150-168). Hoboken NJ: John Wiley & Sons.
SERC 2015-TR-021-4.4 – Volume 4
71
Clune, J., Mouret, J.-B., & Lipson, H. (2013). The evolutionary origins of modularity. Proceedings of the Royal
Society - B, 280, 2863.
Coello, C. A., & Lamont, G. B. (2004). An Introduction to Multi-Objective Evolutionar Algorithms and Their
Applications. Applications of Multi-Objective Evolutionary Algorithms, 1, 1-28.
Coello, C. A., & Mezura-monte, E. (2005). A Simple Multimembered Evolution Strategy to Solve Constrained
Optimization Problems. IEEE Transactions on Evolutionary Computation, 9(1), 1-17.
Cohon, J. L. (1985). Multicriteria Programming: Brief Review and Application. In G. J. S (Ed.). New York: Academic
Press.
Coleman, J. W., Malmos, A. K., Larsen, P. g., Peleska, J., Hains, R., Andrews, Z., . . . Didier, A. (2012). COMPASS
Tool Vision for a System of Systems Collaborative Development Environment. Proceedings of the 7th
International Conference on System of System Engineering, IEEE SoSE 2012.
Contag, G., Laing, C., Pabon, J., Rosenberg, E., Tomasino, K., & Tonello, J. (2013). Nighthawk System Search and
Rescue (SAR) Unmanned Vehicle (UV) System Development. SE4150 Design Project, Naval Postgraduate
School.
Crossley, W. A., & Laananen, D. H. (1996). Conceptual Design of Helicopters via Genetic Algorithm. Journal of
Aircraft, 33(November-December), 1062-1070.
Dagli, C. H., & Wang, R. (2013). Developing a holistic modeling approach for search-based system architecting.
Procedia Computer Science, 16, 206-215.
Dagli, C. H., Singh, A., Dauby, J., & Wang, R. (2009). Smart Systems Architecting: Computational Intelligence
Applied to Trade Space Exploration and System Design. Systems Research Forum, 3(2), 101–120.
Dagli, C., Ergin, N., Enke, D., Giammarco, K., Gosavi, A., Qin, R., . . . Pape, L. (2013). An Advanced Computational
Approach to System of Systems Analysis & Architecting using Agent Based Behavioral Modelin. Systems
Engineering Research Center. Hoboken NJ: SERC.
Dahmann, J. (2012). INCOSE SoS Working Group Pain Points. Proc TTCP-JSA-TP4 Meeting. Retrieved from
http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Documents/NDIA-SE-MS-SoS_2013-0820_Dahmann.pdf
Dahmann, J. (2014). System of System Pain Points. INCOSE International Symposium. Las Vegas: INCOSE.
Dahmann, J., Baldwin, K. J., & Rebovich, G. (2009). Systems of Systems and Net-Centric Enterprise Systems. 7th
Annual Conference on Systems Engineering Research. Loughborough.
Dahmann, J., Lane, J., Rebovich, G., & Baldwin, K. (2008). A model of systems engineering in a system of systems
context. Proceedings of the Conference on Systems Engineering Research. Los Angeles CA.
Dahmann, J., Rebovich, G., Lane, J. A., Lowry, R., & Baldwin, K. (2011). An Implementers’ View of Systems
Engineering for Systems of Systems. Proceedings of IEEE International Systems Conference. Montreal.
Dauby, J. P., & Dagli, C. H. (2011). The canonical decomposition fuzzy comparative methodology for assessing
architectures. IEEE Systems Journal, vol. 5, no. 2, 244-255.
Dauby, J. P., & Upholzer, S. (2011). Exploring Behavioral Dynamics in Systems of Systems. In C. (. Dagli, Complex
Adaptive Systems, Procedia Computer Science, vol 6 (pp. 34-39). Elsevier.
Deb, K. (2000). An efficient constraint handling method for genetic algorithms. Computer Methods Applied
Mechanics and Engineering(186), 311-338.
Deb, K. (2001). Multi-objective Optimization using Evolutionary Algorithms. 16(Wiley).
SERC 2015-TR-021-4.4 – Volume 4
72
Deb, K., & Gupta, H. (2006). Introducing Robustness in Multi-Objective Optimization. Evolutionary Computation,
14(4), 463-494.
Deb, K., & Sinha, A. (2009). Solving bilevel multi-objective optimization problems using evolutionary algorithms. In
M. F.-K. Ehrgott, Evolutionary Multi-Criterion Optimization. Lecture Notes in Computer Science (pp. 110–
124). Berlin: Springer.
DeLaurentis, D., Marais, K., Davendralingam, N., Han, S. Y., Uday, P., Fang, Z., & Gurainiello, C. (2012). Assessing
the Impact of Development Disruptions and Dependencies in Analysis of Alternatives of System-of-Systems.
Hoboken NJ: Stevens Institute of Technology, Systems Engineering Research Center.
Department of the Navy. (1997). Contractor Performance Assessment Reporting System (CPARS). Washington DC.
Diaz, E., Tuya, J., & Blanco, R. (2003). Automated Software Testing Using a Metaheuristic Technique Based on Tabu
Search. Proceedings of the 18th IEEE International Conference on Automated Software Engineering, (pp.
310–313).
Díaz, E., Tuya, J., Blanco, R., & Dolado, J. J. (2008). A Tabu Search Algorithm for Structural Software Testing.
Computers & Operations Research, 35(10), 3052–3072.
Director Systems and Software Engineering, OUSD (AT&L). (2008). Systems Engineering Guide for Systems of
Systems. available from http://www.acq.osd.mil/se/docs/SE-Guide-for-SoS.pdf.
Djavanshir, G. R., Alavizadeh, A., & Tarokh, M. J. (2012). From System-of-Systems to Meta-Systems: Ambiguities
and Challenges. In A. V. Gheorghe, System of Systems.
Dolado, J. J. (2000). A Validation of the Component-Based Method for Software Size Estimation. IEEE Transactions
on Software Engineering, 26(10), 1006 –1021.
Dolado, J. J. (2001). On The Problem of the Software Cost Function. Information and Software Technology, 43(1),
61–72.
Dombkins, D. (2007). Complex Project Management. South Carolina: Booksurge Publishing.
Dombkins, D. H. (1996). Project Managed Change: The Application of Project Management Techniques to Strategic
Change Programs. Australian Graduate School of Management, Center for Corporate Change. Sydney:
University of New South Wales. Retrieved from www.academia.edu/4978870/Wave_Planning__Project_Managed_Change_1996_
Dudek, G., Jenkin, M. R., Milios, E., & Wilkes, D. (1996). A taxonomy for multi-agent robotics. Autonomous Robots,
3(4), 375-397.
Dudenhoeffer, D., & Jones, M. (2000). A Formation Behavior for Large Scale Micro-Robot Force Deployment.
Proceedings of the 32nd Conference on Winter Simulation.
Dutta, P. K. (1999). Strategies and Games Theory & Practice. The MIT Press.
Eichfelder, G. (2010). Multiobjective bilevel optimization. Mathematical Programming, 123(2), 419-449.
Epperly, T. G. (1995). Global Optimization of Nonconvex Nonlinear Programs Using Parallel Branch and Bound.
Citeseer.
Faratin, P., Sierra, C., & Jennings, N. R. (1998). Negotiation decision functions for autonomous agents. Robotics and
Autonomous Systems 24, 159-182.
Farmani, R., & Wright, J. A. (2003). Self-Adaptive Fitness Formulation for Constrained Optimization. IEEE
Transactions on Evolutionary Computation, 7(5), 445-455.
SERC 2015-TR-021-4.4 – Volume 4
73
Fatima, S. S., Wooldridge, M., & Jennings, N. (2004). An Agenda-based Framework for Multi-issue negotiation.
Artificial Intelligence, 152, 1-45.
Flanagan, D., & Brouse, P. (2012). System of Systems Requirements Capacity Allocation. Procedia Computer
Science, 8, 112-117.
Fleming, P. J., & Fonseca, C. M. (1995). An overview of evolutionary algorithms in multiobjective optimization.
Evolutionary Computation, 3(2), 1-16.
Fleming, P. J., & Fonseca, C. M. (1998). Multi-objective Optimization and Multiple Constraint Handling with
Evolutionary Algorithms-Part I. IEEE Transactions on Systems, Man, and Cybernetics- Part A: Systems and
Humans, 28(1), 26-37.
Fogel, D. (2006). Evolutionary Computation. New Jersey: John Wiley and Sons.
Fry, D. N., & DeLaurentis, D. A. (2011). Measuring Net-Centricity. Proceedings of the 6th International Conference
on System of Systems Engineering. Albuquerque.
Gao, J., Buldyrev, S. V., Stanley, H. E., & Havlin, S. (2011, January). Networks formed from interdependent networks.
Nature Physics, 8, 40-48. doi:10.1038/NPHYS2180
Gao, Y., Zhang, G., & Lu, J. (2009). A fuzzy multi-objective bilevel decision support system. International Journal
of Information Technology and Decision Making, 8(1), 93–108.
Garrett, R. K., Anderson, S., Baron, N. T., & Moreland, J. D. (2009). Managing the Interstitials, a System of Systems
Framework Suited for the Ballistic Missile Defense System. Systems Engineering, 14(1), 87-109.
Garvey, P. R., & Pinto, C. A. (2009). Introduction to functional dependency network analysis. Second International
Symposium on Engineering Systems. Cambridge MA: MIT.
Ge, B., Hipel, K. W., Wang, K., & Chen, Y. (2013). A Data-centric Capability-Focused Approach for System-ofSystems Architecture Modeling and Analysis. Systems Engineering, 16(3), 363-377.
Gegov, A. (2010). Fuzzy Networks for Complex Systems. Berlin: Springer.
Giachetti, R. E. (2012). A Flexible Approach to Realize an Enterprise Architecture. Procedia Computer Science, 8,
147-152.
Glover, F. (1989). Tabu Search—Part I. ORSA Journal on Computing, 1(2), 190–206.
Glover, F. (1989). Tabu Search—Part II. ORSA Journal on Computing, 2(1), 4-32.
Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. Machine Learning, 3(23), 95-99.
Gonzalez-Zugasti, J. P., Otto, K. N., & Baker, J. D. (2000). A Method for Architecting Product Platforms. Research
in Engineering Design(12), 61-72.
Gries, M. (2004). Methods for Evaluating and Covering the Design Space during Early Design Development.
Integration, the VLSI Journa, 38(2), 131-183.
Guariniello, C., & DeLaurentis , D. (2014). Communications, information, and cyber security in Systems-of-Systems:
Assessing the impact of attacks through interdependency analysis. Procedia Computer Science; Conference
on Systems Engineering Research. Redondo Beach CA: Elsevier.
Haimes, Y. Y. (2012). Modeling Complex Systems of Systems with Phantom System Models. Systems Engineering,
15(3), 333-346.
SERC 2015-TR-021-4.4 – Volume 4
74
Han, S. Y., & DeLaurentis, D. (2013). Development Interdependency Modeling for System-of-Systems (SoS) using
Bayesian Networks: SoS Management Strategy Planning. Procedia Computer Science, Volume 16, 698–707.
Atlanta.
Hansen , P., Jaumard , B., & Savard, G. (1992). New branch-and-bound rules for linear bilevel programming. SIAM
Journal on Scientific and Statistical Computing, 13(5), 1194–1217.
Hassan, R., & Crossley, W. (2004). Discrete Design Optimization under Uncertainty: A Generalized Population-Based
Sampling Genetic Algorithm. AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics & Materials
Conference. Palm Springs.
Hassan, R., de Weck, O., & Springmann, P. (2004). Architecting a Communication Satellite Product Line. 22nd
International Communication Satellite Systems Conference. Monterey, CA.
He, Z., Yen, G. G., & Zhang, J. (2013). Fuzzy-Based Pareto Optimality for Many-Objective Evolutionary Algorithms.
IEEE Transactions of Evolutionary Computation, PP(99), 1-16.
Henson, S. A., Henshaw, M. J., Barot, V., Siemieniuch, C. E., Dogan, H., Lim, S. L., . . . DeLaurentis, D. (2013).
Towards a Systems of Systems Engineering EU Strategic Research Agenda. IEEE International Conference
on System of Systems Engineering. Maui.
Herroelen, W., De Reyck, B., & Demeulemeester, E. (1998). Resource-constrained project scheduling: a survey of
recent developments. Computers & Operations Research, 25(4), 279-302.
Hill climbing. (n.d.). Retrieved October 24, 2013, from http://en.wikipedia.org/wiki/Hill_climbing
Holland, J. H. (1973). Genetic Algorithm and the Optimal Allocation of Trials. SIAM Journal on Computing, vol 2,
June 1973, 88-105.
Hunt, B., Lipsman, R. L., & Rosenberg, J. M. (2001). A guide to MATLAB: For beginners and experineced users.
New York: Cambridge University Press.
III, J. A. (2004). A Quantitative approach to Assessing System Evolvability. Systems Engineering, 770, 1-16.
INCOSE. (2011). SYSTEMS ENGINEERING HANDBOOK v. 3.2.2. San Diego: INCOSE.
Jackson, S., & Ferris, T. L. (2013). Resilience Principles for Engineered Systems. Systems Engineering, 16(2), 152164.
Jackson, S., & Ferris, T. L. (2013). Resilience Principles for Engineered Systems. Systems Engineering, 16(2), 152164.
Jia, L., Wang, Y., & Fan, L. (2011). Uniform Design Based Hybrid Genetic Algorithm for Multiobjective Bilevel
Convex Programming. Proceedings of the Seventh International Conference on Computational Intelligence
and Security, (pp. 159-163).
Johnston, W., Mastran, K., Quijano, N., & Stevens, M. (2013). Unmanned Vehicle Search and Rescue Initiative.
SE4150 Design Project, Naval Postgraduate School.
Joint Staff. (2010). CJCSM 3500.04C, UNIVERSAL JOINT TASK LIST (UJTL). Washington DC: Department of
Defense.
Jonker, C. M., Robu, V., & Treur, J. (2007). An Agent Architecture for Multi-attribute Negotiation Using Incomplete
Preference Information. Auton Agent Multi-agent System, 15, 221-252.
Karnik, N. N., Mendel, J. M., & Liang, Q. (1999). Type-2 fuzzy logic systems.Fuzzy Systems, IEEE Transactions on,
7(6), 643-658.
SERC 2015-TR-021-4.4 – Volume 4
75
Kilicay Ergin, N., & Dagli, C. H. (2008). In M. (. Jamshidi, System of Systems: Innovations for the 21st Century (pp.
77-100). Wiley & Sons, Inc.
Kilicay-Ergin, N., Enke, D., & Dagli, C. (2012). Biased trader model and analysis of financial market dynamics.
International Journal of Knowledge-based Intelligent Engineering Systems, Vol. 16, 99-116.
Kinnunen, M. J. (2006). Complexity Measures for System Architecture Models. Cambridge: MIT: System Design and
Management Program Masters Thesis.
Kirkpatrick;, S., Gelatt, C. D., & Vecchi, M. P. (1983). Optimization by Simulated Annealing. Science, 220(4598),
671–680.
Koch, P. N., Evans, J. P., & Powell, D. (2002). Interdigitation for Effective Design Space Exploration Using iSIGHT.
Structural and Multidisciplinary Optimization, 23(2), 111–126.
Konur, D., & Dagli, C. (2014). Military system of systems architecting with individual system contracts. Optimization
Letters, October.
Konur, D., & Golias, M. M. (2013). Cost-stable truck scheduling at a cross-dock facility with unknown truck arrivals:
A meta-heuristic approach. Transportation Research Part E, 49(1), 71-91.
Kraus, S. (1996). An Overview of Incentive Contracting. Artificial Intelligence, 83, 297-346.
Kraus, S. (2001). Automated negotiation and decision making in multiagent environments. Easss 2086, 150-172.
Krothapali, N. K., & Deshmukh, A. V. (1999). Design of Negotiation Protocols for Multi-agent Manufacturing
Systems. International Journal of Production Research, 37(7), 1601-1624.
Lafleur, J. M. (2012). A Markovian State-SpaceFramework for Integrating Flexibility into Space System Design
Decisions. Atlanta: Georgia Institute of Technology School of Aerospace Engineering Doctoral Thesis.
Lamar, B. W. (2009). Min-additive utility functions. MITRE Corporation. MITRE Corporation. Retrieved from
http://www.mitre.org/sites/default/files/pdf/09_0383.pdf
Lane, J. A., & Bohn, T. (2012). Using SysML Modeling to Understand and Evolve System of Systems. Systems
Engineering, 16(1), 87-98.
Li, C., & Chiang, T.-W. (2013). Complex Neurofuzzy ARIMA Forecasting - A New Approach Using Complex Fuzzy
Sets. IEEE Transactions on Fuzzy Systems, 21(3), 567-584.
Li, M., Lin, D., & Wang, S. (2010). Solving a type of biobjective bilevel programming problem using NSGA-II.
Computers and Mathematics with Applications, 59(2), 706-715.
Lin, Y., Cunningham III, G. A., Coggshall, S. V., & Jones, R. D. (1998, September). Nonlinear System Input Structure
Identification: Two Stage Fuzzy Curves and Surfaces. IEEE Transactions on Systems, Man, and Cybernetics
- Part A: Systems and Humans, 28(5), 678- 684.
Lopes, F., Wooldridge, M., & Novais, A. C. (2008). Negotiation among autonomous computational agents: principles,
analysis and challenges. Artificial Intelligence Review 29, 1-44.
Lu, J., Zhang, G., & Dillon, T. (2008). Fuzzy multi-objective bilevel decision making by an approximation kth-best
approach. Multiple-Valued Logic and Soft Computing, 14, 205–232.
Luzeaux, D. (2013). System-of-Systems and Large-Scale Complex Systems Architecting. Complex Systems Design
& Management. Paris. Retrieved from http://www.csdm2013.csdm.fr/IMG/pdf/2nd_keynote_Speaker__Dominique_LUZEAUX.pdf
Maier, M. W., & Rechtin, E. (2009). The Art of Systems Architecting, 3rd ed. Boca Raton: CRC Press.
SERC 2015-TR-021-4.4 – Volume 4
76
Malan, R., & Bredemeyer, D. (2001, August). Software Architecture Resources. (BREDEMEYER CONSULTING)
Retrieved Dec 2014, from http://www.bredemeyer.com/pdf_files/NonFunctReq.PDF
Maskin, E. S. (1996). Theories of the Soft Budget Constraint. Japan & the World Economy, 8, 125-133.
Masud, A. S., & Hwang, C. L. (1979). Multiple Objective Decision Making – Methods and Applications. In Lecture
Notes in Economics and Mathematical Systems (Vol. 164). Springer.
Mekdeci, B., Shah, N., Ross, A. M., Rhodes, D. H., & Hastings, D. (2014). Revisiting the Question: Are Systems of
Systems just (traditiponal) Systems or are they a new class of Systems? Cambridge, MA: Systems
Engineering Advancement Research Initiative (SEAri).
Mendel, J. M. (2013). On KM Algroithms for Solving Type-2 Fuzzy Set Problems. IEEE Transactions on Fuzzy
Systems, 21(3), 426-446.
Mendel, J. M., & John, R. I. (2002). Type-2 Fuzzy Sets Made Simple. IEEE Transactions on Fuzzy Systems, 10(2),
117-127.
Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E. (1953). Equation of State
Calculations by Fast Computing Machines. The Journal of Chemical Physics, 21(6), 1087–1092.
Michalewicz, Z., & Schoenauer, M. (1996). Evolutionary Algorithms for Constrained Parameter Optimization
Problems. Evolutionary Computation, 4(1), 1-32.
Miettinen, K. M. (1999). Nonlinear Multi-Objective Optimization. Boston: luwer Academic Publishers.
Migdalas, A., Pardalos, P. M., & Värbrand, P. (1998). Multilevel Optimization: Algorithms and Applications.
Dordrecht: Kluwer Academic Publishers.
Min, B., & Chang, S. H. (1991). System Complexity Measure in the Aspect of Operational Difficulty. IEEE
Transactions Nuclear Science, 38(5), 1035-1039.
Mordecai, Y., & Dori, D. (2013). I5: A Model-Based Framework for Architecting System-of-Systems Interoperability,
Interconnectivity, Interfacing, Integration, and Interaction. International Symposium of the International
Council on Systems Engineering (INCOSE). Philadelphia.
Mostafavi, A., Abraham, D. M., Noureldin, S., Pankow, G., Novak, J., Walker, R., . . . George, B. (2012). "Risk-based
Protocol for the Inspection of Transportation Construction Projects undertaken by State Departments of
Transportation. Journal of Construction Engineering & Management, ASCE, 139(8), 977-986.
Osman, M. S., Abo-Sinna, M. A., Amer, A. H., & Emam, O. E. (2004). A multi-level non-linear multi-objective
decision-making under fuzziness. Applied Mathematics and Computation, 153(1), 239-252.
Pape, L., & Dagli, C. (2013). Assessing robustness in systems of systems meta-architectures. Complex Adaptive
Systems. Baltimore.
Pape, L., & Dagli, C. (2013). Assessing robustness in systems of systems meta-architectures. Procedia Computer
Science, Complex Adaptive Systems. 20, pp. 262-269. Baltimore: Elsevier.
Pape, L., Agarwal, S., Giammarco, K., & Dagli, C. (2014). Fuzzy Optimization of Acknowledged System of Systems
Meta-architectures for Agent based Modeling of Development . Conference on Systems Engineering
Research. Redondo Beach CA.
Pape, L., Giammarco, K., Colombi, J., Dagli, C., Kilicay-Ergin, N., & Rebovich, G. (2013). A fuzzy evaluation method
for system of systems meta-architectures. Procedia Computer Science, Conference on Systems Engineering
Research (CSER’13). 16, pp. 245-254. Atlanta: ScineceDirect, Elsevier.
SERC 2015-TR-021-4.4 – Volume 4
77
Parsons, S., & Wooldridge, M. (2000). Languages for Negotiation. Proceedings of the Fourteenth European
Conference on Artificial Intelligence (ECAI-2000).
Pedrycz, W., Ekel, P., & Parreiras, R. (2011). Fuzzy Multicriteria Decision Making; Models, Methods and
Applications. West Sussex: John Wiley & Sons.
Pieume , C. O., Marcotte , P., Fotso , L. P., & Siarry. , P. (2011). Solving bilevel linear multiobjective programming
problems. American Journal of Operations Research, 1, 214–219.
Pitsko, R., & Verma, D. (2012). Principles for Architecting Adaptable Command and Control. New Challenges in
Systems Engineering and Architecting. St. Louis.
Räihä, O. (2008). Applying Genetic Algorithms in Software Architecture Design. University of Tampere.
Ravindran, A., Ragsdell, K. M., & Reklaitis, G. V. (2006). Engineering Optimization Methods and Applications.
Hoboken, NJ: John Wiley & Sons.
Rela, L. (2004). Evolutionary Computing in Search-Based Software Engineering. Lappeenranta University of
Technology.
Ricci, N., Ross, A. M., Rhodes, D. H., & Fitzgerald, M. E. (2013). Considering Alternative Strategies for Value
Sustainment in Systems-of-Systems (Draft). Cambridge MA: Systems Engineering Advancement Research
Initiative.
Rios, L. M., & Sahinidis, N. V. (2009). Derivative-Free Optimization: A Review of Algorithms and Comparison of
Software Implementations. Advances in Optimization II.
Rosenau, W. (1991). Coalition Scud Hunting in Iraq, 1991. RAND Corporation.
Ross, A. M., Rhodes, D. H., & Hastings, D. E. (2008). Defining Changeability: Reconciling Flexibility, Adaptability,
Scalability, Modifiability, and Robustness for Maintaining System Lifecycle Value. Systems Engineering,
246 - 262.
Ross, S. (2003). Introduction to Probability Models, 8th Edition. Academic Press.
Rostker, B. (2000, July 25). Iraq's Scud Ballistic Missiles. Retrieved Sep 12, 2013, from Iraq Watch:
http://www.iraqwatch.org/government/US/Pentagon/dodscud.htm
Runarsson, T. P., & Yao, X. (2000). Stochastic Ranking for Constrained Evolutionary Optimization. IEEE
Transactions on Evolutionary Computation, 4(3), 284-294.
Russell, S., & Norvig, P. (2009). Artificial Intelligence: A Modern Approach. Prentice Hall.
Sanz, J. A., Fernandez, A., Bustince, H., & Herrara, F. (2013). IVTURS: A Linguistic Fuzzy Rule-Based Classification
System Based On a New Interval-Valued Fuzzy Reasoning Method with Tuniing and Rule Selection. IEEE
Transactions on Fuzzy Systems, 21(3), 399-411.
Schreiner, M. W., & Wirthlin, J. R. (2012). Challenges using modeling and simulation in architecture. Procedia
Computer Science, 8, 153-158.
Schwartz, M. (2010). Defense acquisitions: How DoD acquires weapon systems and recent efforts to reform the
process. Washington DC: LIBRARY OF CONGRESS CONGRESSIONAL RESEARCH SERVICE.
Selva, D., & Crawley, E. F. (2013). VASSAR: Value Assessment of System Architectures using Rules. IEEE
Aerospace Conference (pp. 1-21). Big Sky MT: IEEE.
Shi, X., & Xia, H. S. (2001). Model and Interactive Algorithm of Bi-Level Multi-Objective Decision-Making with
Multiple Interconnected Decision Makers. Journal of Multi-criteria Decision Analysis, 10, 27-34.
SERC 2015-TR-021-4.4 – Volume 4
78
Shtub, A., Bard, J. F., & Globerson, S. (1994). Project Management. New Jersey: Prentice Hall.
Simpson , T. W., & D’souza, B. S. (2002). Assessing Variable Levels of Platform Commonality within a Product
Family Using a Multiobjective Genetic Algorithm. 9th AIAA/ISSMO Symposium on Multidisciplinary
Analysis and Optimization. Atlanta, GA.
Singer, Y. (2006). Dynamic Measure of Network Robustness. IEEE 24th Conference of Electrical and Electronic
Engineers in Israel.
Singh, A., & Dagli, C. H. (2009). Multi-objective Stochastic Heuristic Method for Trade Space Exploration of a
Network Centric System of Systems. 3rd Annual IEEE International Systems Conference, 2009. Vancouver,
Canada.
Singh, A., & Dagli, C. H. (2010). "Computing with words" to support multi-criteria decision-making during
conceptual design. Systems Research Forum, 85-99.
Smartt, C., & Ferreira, S. (2012). Constructing a General Framework for Systems Engineering Strategy. Systems
Engineering, 15(2), 140-152.
Smartt, C., & Ferreira, S. (2012). Constructing a General Framework for Systems Engineering Strategy. Systems
Engineering, 15(2), 140-152.
SPEC Innovations. (n.d.). Model-Based Systems Engineering Tools. Retrieved August 20, 2014, from
https://www.innoslate.com/systems-engineering/
Suarez, R. (2004, Dec 9). Troops Question Secretary of Defense Donald Rumsfeld about Armor. Retrieved Apr 14,
2014, from PBS NewsHour: http://www.pbs.org/newshour/bb/military-july-dec04-armor_12-9/
Sumathi, S., & Surekha, P. (2010). Computational Intelligence Paradigms: Theory & Applications Using MATLAB.
Boca Raton FL: CRC Press.
Talbot, F. B. (1982). Resource-constrained project scheduling with time-resource tradeoffs: The nonpreemptive case.
Management Science, 28(10), 1197-1210.
Taleb, N. N. (2004). Fooled by Randomness. New York: Random House Trade Paperbacks.
Thompson, M. (2002, December 23). Iraq: The Great Scud
http://www.time.com/time/magazine/article/0,9171,1003916,00.html.
Hunt.
Time
Magazine,
pp.
Trans-Atlantic Research and Education Agenda in Systems of Systems (T-AREA-SOS) Project. (2013). The Systems
of Systems Engineering Strategic Research Agenda. Loughborough: Loughborough University.
Wall, M. B. (1996). A genetic algorithm for resource-constrained scheduling. Cambridge, MA: Doctoral dissertation,
Massachusetts Institute of Technology.
Wang, J.-Q., & Zhang, H.-Y. (2013). Multicriteria Decision-Making Approach Based on Atanassov's Intuitionistic
Fuzzy Sets With Incomplete Certain Information on Weights. IEEE Transactions on Fuzzy Systems, 21(3),
510-515.
Wang, R., & Dagli, C. H. (2011). Executable system architecting using systems modeling language in conjunction
with colored Petri nets in a model‐driven systems development process. Systems Engineering, 14(4), 383409.
Wang, R., Agarwal, S., & Dagli, C. (2014). Executable System of Systems Architecture Using OPM in Conjunction
with Colored Petri Net: A Module for Flexible Intelligent & Learning Architectures for System of Systems.
Europe Middle East & Africa Systems Engineering Conference (EMEASEC). Somerset-West, Western Cape,
South Africa.
SERC 2015-TR-021-4.4 – Volume 4
79
Wang, Y., Cai, Z., Guo, G., & Zhou, Y. (200). Multiobjective Optimization and Hybrid Evolutionary Algorithm to
Solve Constrained Optimization Problems. IEEE Transactions Systems, Man, and Cybernetics- Part B:
Cybernetics, 34(3), 560-575.
Wanyama , T., & Far, B. H. (2007). A Protocol for Multi-agent Negotiation in a Group choice Decision Making
Process. Journal of Network and Computer Applications, 30, 1173-1195.
Wappler, S. (2007). Automatic Generation of Object-Oriented Unit Tests Using Genetic Programming. Berlin:
Universitätsbibliothek der Technischen Universität.
Wappler, S., & Wegener, J. (2006). Evolutionary Unit Testing of Object-Oriented Software Using Strongly-Typed
Genetic Programming. Proceedings of the 8th annual conference on Genetic and evolutionary computation,
(pp. 1925–1932).
Warfield, J. N. (1973). Binary Matrices in System Modeling. IEEE Transactions on Systems, Man, and Cybernetics,
SMC-3(No. 5, September), 441-449.
Weiss, W. (2008). Dynamic Security: An Agent-based Model for Airport Defense. Proceedings of the Winter
Simulation Conference.
Woldesenbet, Y. G., Yen, G. G., & Tessema, B. G. (2009). Constraint Handling in Multiobjective Evolutionary
Optimization. IEEE Transactions on Evolutionary Computation, 13(3), 514-525.
Yi, J. S., Kang, Y., Stasko, J. T., & Jacko, J. A. (2007, Nov/Dec). Toward a Deeper Understanding of the Role of
Interaction in Information Visualization. IEEE Transactions on Visualization and Computer Graphics, 13(6),
1224 - 1231.
Yu, O.-Y., Gulkema, S. D., Briaud, J.-L., & Burnett, D. (2011). Sensitivity Analysis for Multi-Attribute System
Selection Problems in Onshore Environmentally Friendly Drilling (EFD). Systems Engineering, 15(2), 153171.
Yu, T.-L., Yassine, A. A., & Goldberg, D. E. (2007). An information theoretic method for developing modular
architectures using genetic algorithms. Research in Engineering Design, 18(2), 91-109.
Zadeh, L. A. (1975). Fuzzy logic and approximate reasoning. Synthese, 30(3-4), 407-428.
Zhan , G., Lu, J., & Ya, G. (2008). An algorithm for fuzzy multi-objective multi-follower partial cooperative bilevel
programming. Journal of Intelligent & Fuzzy Systems, 19, 303–319.
Zhan, G., Lu, J., & Ya, G. (2008). Fuzzy bilevel programming: multi-objective and multi-follower with shared
variables. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 16, 105-133.
Zhang, G., & Lu, J. (2010). Fuzzy bilevel programming with multiple objectives and cooperative multiple followers.
Global Optimization, 47, 403-419.
Zhang, T., Hu, T., Jia-wei, C., wang, Z., & Guo, X. (2012). Solving Bilevel Multiobjective Programming Problem by
Elite Quantum Behaved Particle Swarm Optimization. Journal of Applied Mathematics, 2012.
doi:10.1155/2012/102482
Zhang, T., Hu, T., Zheng, Y., & Guo, X. (2012). An Improved Particle Swarm Optimiza-tion for Solving Bilevel
Multiobjective Programming Problem. Journal of Applied Mathematics,, 2012. doi:10.1155/2012/626717
Zheng, Y., Wan, Z., & Wang, G. (2011). A fuzzy interactive method for a class of bilevel multiobjective programming
problem. Expert Systems with Applications, 38, 10384–10388.
Zhou, A., Qu, B.-Y., Li, H., Zhao, S.-Z., Suganthan, P. N., & Zhang, Q. (2011). Multiobjective Evolutionary
Algorithms: A Survey of the State of the Art. Swarm and Evolutionary Computation(1), 32-49.
SERC 2015-TR-021-4.4 – Volume 4
80
SERC 2015-TR-021-4.4 – Volume 4
81