“An Empirical Investigation of System
Changes to Frame Links between
Design Decisions and Ilities”
J. Clark Beesemyer, Adam M. Ross, and Donna H. Rhodes
Massachusetts Institute of Technology
CSER 2012
St. Louis, 19-22 March 2012
Motivation
“Rep. John Mica called on the agency to "reform" and "become...a thinking,
risk‐based, flexible agency that analyzes risks, sets security standards and
audits security performance.”
“Defense Secretary Panetta: "The US joint force will be smaller and it will be
leaner. But it will be more agile, more flexible, ready to deploy quickly,
innovative and technologically advanced.“
… “the Defense Department and the Office of the Director of National
Intelligence pledged to foster an industrial base that is 'robust, competitive,
flexible, healthy, and delivers reliable space capabilities on time and on
budget.'"
Quotes from AIAA Daily Launch, 20 Jul 2011 – 13 Feb 2012
Ross, Beesemyer, and Rhodes 2012
seari.mit.edu
© 2012 Massachusetts Institute of Technology
2
Motivation
• High impact early decisions usually made with
incomplete system knowledge
• Decision makers may improve their capacity to
discriminate between system concepts and design
choices by measuring a system’s “ilities” such as
changeability, scalability, and survivability
– These system properties (ilities) can be ambiguous,
imprecise, and hard to validate or verify
• Increased interest in ilities requires better
understanding of the change mechanisms and more
precise language
Looking at empirical cases of system changes and associated ilities in a
structured manner may lead to more effective design approaches.
seari.mit.edu
© 2012 Massachusetts Institute of Technology
3
Ilities as Outcomes
Design
Principle
Path
Enabler
Change
Mechanism
Ility
Characteristic
• Ilities are performance characteristics made possible ultimately
by the design principles used in the early phases of design
• Descriptive research aims to look at many of these cases for
trends and insight that may be used in future designs
Targeted
Modularity
seari.mit.edu
Software
packaged
add-ons
(apps)
Downloading
from
AppStore
© 2012 Massachusetts Institute of Technology
Flexibility
Versatility
Extensibility
Evolvability
4
The Change Option
WP-2011-1-2 Ross, A.M., and Rhodes, D.H.,
“Anatomy of a Change Option: Mechanisms
and Enablers”, <seari.mit.edu/papers.php>
S2
S1
$.initial_cost
M1
S3
S4
Path
Enabler
Restrictive
Epoch
(num_uses>1)
S2
M1
M1
S1
S1
M1
M1
M1
$.carry_cost
$.execution_cost
Start
date
Expiry
date
Time
The following factors characterize a change option:
•
•
•
•
•
•
Name
Start date
Expiry date
Possible end states = f(start state)
Initial cost = cost of path enabler?
Carry cost = f(now, execution date, expiry
date)
•
•
•
•
•
•
Execution cost = f(end state, epoch)
Reusability = number of times it can be
executed
Valid epochs
Valid lifecycle phase
Pre-requisites for execution (e.g. agent)
Change “type” (e.g. flexible, adaptable, etc.)
Change options can be compared and valuated on this basis
seari.mit.edu
© 2012 Massachusetts Institute of Technology
5
Generalizing the Change Statement:
A Prescriptive Basis
(From Ross,
Rhodes, and
Hastings 2008)
1
2
3
4
5
6
7
8
9
10
10 dimensional basis for specifying “changeability”-related ilities
WP-2011-1-2 Ross, A., Beesemyer, J., and Rhodes, D., “A Prescriptive Semantic Basis for System Lifecycle Properties”, seari.mit.edu/papers.php
seari.mit.edu
© 2012 Massachusetts Institute of Technology
6
Generalizing the Change Statement:
A Prescriptive Basis
10 dimensional basis for specifying “changeability”-related ilities
1
2
Dim #
1
2
3
4
5
6
7
8
9
10
3
4
Category Name
Cause
Context
Entity
Aspect
Phase
Agent
Parameter Type
Effect
Potential States
Valuable
5
6
7
8
9
10
# Levels
4
3
4
4
4
4
3
5
4
256
WP-2011-1-2 Ross, A., Beesemyer, J., and Rhodes, D., “A Prescriptive Semantic Basis for System Lifecycle Properties”, seari.mit.edu/papers.php
seari.mit.edu
© 2012 Massachusetts Institute of Technology
7
Captured Ilities
Prescriptive Semantic Basis for Changeability-type Ilities
In response to “cause” in “context”, desire “agent” to make some “change” in “system” that is “valuable”
Cause
Context
System
Agent
Change
Valuable
Why
Where
What
What
When
Who
What
What
What
What
When
When
For What
For What
Cause
Context
Entity
Aspect
Phase
Agent
Param Change
Type
Effect
(Scale)
Effect
(Amount)
Potential
States
Timing
Span
Resources
Benefit
perturbation
specificity
abstraction
aspect
LC phase
executes
param type
level
set
target range
reaction
duration
cost
utility
disturbance
shift
none
any
circumstantial
general
any
architecture
design
system
any
form
function
operations
any
pre-ops
ops
inter-LC
any
internal
external
either
none
any
level
set
any
bigger
smaller
not-same
same
any
more
less
not-same
same
any
one
few
many
any
sooner
later
always
any
shorter
longer
same
any
less
more
same
any
more
less
same
any
(choose one)
(choose one)
any
any
shift
shift
shift
any
any
any
any
any
disturbance
any
any
any
any
any
shift
any
any
circumstantial
circumstantial
general
any
any
any
any
any
circumstantial
any
any
any
any
any
any
seari.mit.edu
any
any
system
system
architecture
any
any
any
any
any
any
any
any
any
any
any
any
function
operations
any
form
any
any
any
any
form
any
any
form
any
any
any
any
any
ops
ops
ops
ops
inter-LC
any
any
any
ops
any
ops
ops
any
any
ops
any
any
any
any
any
none
any
internal
external
any
any
any
any
any
any
any
any
either
any
set
set
any
level
any
any
any
level
set
set
any
any
any
any
set
any
any
any
any
same
same
any
not-same
not-same
not-same
any
any
any
any
not-same
not-same
any
not-same
any
not-same
not-same
same
any
any
not-same
not-same
any
not-same
not-same
any
any
not-same
not-same
more
not-same
any
Ility Name
many
many
any
few
any
any
any
any
many
any
any
any
any
any
any
any
any
© 2012 Massachusetts Institute of Technology
any
any
any
any
any
any
any
any
any
any
any
any
any
sooner
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
shorter
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
any
same
Functional Versatility
Operational Versatility
Robustness
Classical Passive Robustness
Evolvability
Adaptability
Flexibility
Scalability
Optionability
Modifiability
Survivability
Reconfigurability
Agility
Reactivity
Extensibility
Changeability
Value Robustness
8
Change Database
• To capture changes from many different systems and various
domains
– Currently holding ~100 changes from ~45 systems
• Developed from Ility Framework previously discussed
• Takes system and change information in a categorical manner
to determine ilities present in changes
seari.mit.edu
© 2012 Massachusetts Institute of Technology
9
Change Database
This research aims to find a better means of determining which
ilities are present in different system changes and map those
ilities to various design principles. When stakeholders identify
an ility as a desired property of a design, the ultimate goal of
added value to the system from having this ility represents one
end of this relationship.
seari.mit.edu
© 2012 Massachusetts Institute of Technology
10
Change Database
Path Enablers are
instantiations of
DPs that facilitate
CEs in a system.
Design Principles
are the heuristics
designers follow
when making
design choices.
seari.mit.edu
www.militaryfactory.com
Ilities are properties
shown during these
changes that equate
to more or less value
for stakeholders.
Change Mechanisms
give the system
options to change if
necessary given
conditions and
performance
© 2012 Massachusetts Institute of Technology
11
Change Database
Change
database
seari.mit.edu
© 2012 Massachusetts Institute of Technology
XXXXXXXXXX
12
Change Database
System
Information
XXXXXXXXXX
Source and
Cost Info
(if available)
seari.mit.edu
© 2012 Massachusetts Institute of Technology
13
Change Database
Specified
Parameter
XXXXXXXXXX
Path
Enablers
Ilities
Identified
seari.mit.edu
© 2012 Massachusetts Institute of Technology
14
Change Database
XXXXXXXXXX
Change
Information
seari.mit.edu
© 2012 Massachusetts Institute of Technology
15
Cluster Analysis
COOLCAT
• Clustering algorithm made to handle
categorical data, (Fu et al.)
• Model based clustering (entropy model)
– Where entropy is a measure of dissimilarity
• Basic Method
– Turn categorical data into nominal dataset
– Specify number of clusters
– Run piecewise entropy to initialize first 2 clusters w/
highest entropy (most different)
– Max(min(entropy)) of remaining records for each new
cluster to specified number of clusters
– Fill remaining records into clusters by minimizing
system entropy for each record
seari.mit.edu
© 2012 Massachusetts Institute of Technology
16
Preliminary Insights
Heuristic-based clusters
COOLCAT clusters
Cluster 1; Size = 28
Entropy = 5.616; System Entropy = 5.88
Finite 4
Cause
Shift19
None 1
Shift28
Cause
Context
Circumstantial 5
General 10
Circumstantial 14
Context
Entity
Architecture
7
Design
4
System
External
Level 5
Set23
Agent
External
18
20 4
Internal
Finite
Cause
0.4
None
Shift
Cause
0
0.1
0.2
0.4
0.3
Circumstantial 47
Finite
24
0.5
0.7
0.6
0.9
0.8
None
py
y
Agent
External 52
System11
Design5
0
0.1
Level
0.3
0.4
0.5
0.6
0.8
0.9
13
Phase
E (ops)
Inter-LC
0.2
Param_Type
Level
9
14
Set
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.2
Many12
0.4
0.6
One 5
0.8
Operations2
Many5
0.4
0.6
None 1
Form 20
Aspect
0.8
0
0.2
0.4
0
0.2
Set
Many18
0.4
0.6
0.8
0.6
0.8
1
Shift19
Form 16
2
5
Function
Operations
E (ops)23
External 22
Agent
Level 5
0
Internal 1
Set18
Few 5
Potential_States
1
General 11
System 20
Aspect
Param_Type
One 10
0.8
Level
Few 7
Potential_States
1
Set1
Circumstantial 12
Phase
Set5
0.6
0.4
EntityDesign3
Internal 10
Many14
0.2
Cause Finite 4
Context
E (ops)24
Potential_States
1
Shift11
Level 19
Param_Type
0.8
17
Cluster 6; Size = 23
Entropy = 5.384; System Entropy = 4.003
Operations4
External 14
Agent
Few 4
0
1
One 6
0.6
8
Few 4
Potential_States
0.8
System 24
Phase
Set5
0.6
0.4
Inter-LC8
External 25
Param_Type
Set
0.2
2
3
Function
Operations
E (ops)17
Agent
None 2
Level 3
Param_Type
None 2
System 14
Form 20
Aspect
Phase
Many5
Few
0
General 13
Entity
Architecture 2 Design9
11
Level
E (ops)4
Internal 2
Circumstantial 12
Context
Many17
0.2
0.4
0.6
One 1
0.8
1
Military vs non-military
Control the number of clusters (2-n)
Large population vs small population
Mass produced vs unique
Can compare mental clusters to COOLCAT clusters.
Long lifespan vs short
COOLCAT will find similarities more complex than simple data mining.
For example, the mental models show that shorter lifecycles tend to be
more evolvable. COOLCAT models yield an evolvable cluster that
is almost equal lifecycles but much more military and space oriented.
seari.mit.edu
1
Shift25
Cause
General 1
Internal 5
Operations3
4
tential_States
Agent
Circumstantial 24
Context
Entity
Level 4
0.2
0.4
Finite 12
Cause
General 5
External 9
Param_Type
0.2
Shift2
Operations9
External 8
System 3 4
Form 1
Aspect
Cluster 5; Size = 24
Entropy = 4.555; System Entropy = 4.003
Inter-LC9
Phase
One 1
Many11
Potential_States
0
0
1
Design
Param_Type
Phase
Few 17
Potential_States
1
Design6
Form 7
Agent
Few 11
0.8
Shift9
Circumstantial 4
Entity Architecture 3
Aspect
Potential_States
0.6
Operations1
Internal 6
Level 17
Param_Type
Many1 One 3
0.4
Cause
Context
External23
Agent
Few
0
19
Cluster 4; Size = 25
Entropy = 5.87; System Entropy = 4.604
E (ops)15
Shift2
Agent
Circumstantial 4
Entity
E (ops)17
External 11
Phase
Finite 2
Cause
Context
Form 16
Aspect
Cluster 4; Size = 9
Entropy = 4.656; System Entropy = 4.003
10
Internal 14
Level 36
Potential_States
1
System 14
6
Aspect
Cluster 3; Size = 4Form
Entropy = 4.434; System Entropy = 4.003
Shift11
System 17
Agent
Few 10
0
One
0.8
Circumstantial 14
Context
Circumstantial 17
Phase
Set14
Param_Type
Operations6
Operations2
Inter-LC11
External 14
Agent
1
0.6
Finite 13
Cause
1
Finite 6
Entity
Form 12
Phase E (ops)3
Potential_States
Form17
Aspect
0.8
Context
Architecture 14
Entity
One 12
0.7
Many
0.4
3
Cluster 3; Size = 15
Entropy = 6.046; System Entropy = 4.604
One 11
0.6
Cause
General 14
Aspect
0.2
None 2
Cluster 2; Size = 17
Entropy = 2.196; System Entropy = 4.003
Shift14
Cause
Set
Many31
2
Entity
Design1
33
Few 24
Few
External 22
Agent
Param_Type
10
0.2
Set20
Many27
0.4
Cluster 1; Size = 14
Entropy = 2.436; System Entropy = 4.003
General9
tential_States
E (ops)36
Phase
Set15
0
Level 43
0.2
Context
Param_Type
Architecture 7
General
7
Inter-LC13
34
Inter-LC12
External 15
1
13
Function
Operations
Internal 19
Few 25
0
Internal 15
Operations2
E (ops)63
ntial_States
2
13
Function
Operations
Shift
External 42
Param_Type
E (ops)54
ential_States
Entity
System
Phase
Circumstantial14
Context
E (ops)3
Agent
System 62
Agent
Form 52
General 1
System 36
Form 34
Aspect
1
Param_Type
56
Form 49
Phase
py
Shift21
0.8
38
Circumstantial
Aspect
51
1
Aspect
Cause Finite 2
0.6
1
Entity
Design1
General 20
6
Many
10
6 One 2
EntityArchitecture
Design
Few 16
Potential_States
Set10
Context
Shift24
Circumstantial 35
Entity
Operations3
Form 12
Aspect
Phase
47
Context
Level 14
Param_Type
One 3
Cluster 2; Size = 63
Entropy = 5.997; System Entropy = 5.88
2
None 1
Context
Architecture 15
Entity
Many10
0.2
Inter-LC7
Finite 11
Cause
General 15
Context
28
Few 15
ntial_States
0
E (ops)17
Cluster 2; Size = 36
Entropy = 3.923; System Entropy = 4.604
Shift15
Cause
Inter-LC20
E (ops)
Agent
Operations4
Cluster 1; Size = 15
Entropy = 2.686; System Entropy = 4.604
System 2
1
4
Function
Operations
8
Param_Type
Phase
Design9
Form 23
Phase
13
Form 20
Aspect
General 23
Architecture 17
Entity
Aspect
© 2012 Massachusetts Institute of Technology
17
1
Significance
• 2 clusters
• 6 clusters
– Expected system entropy
5.8508 using COOLCAT
– Expected system entropy
3.6795 using COOLCAT
– Exp Sys Ent 1000 trials
using random clustering:
– Exp Sys Ent 1000 trials
using random clustering:
80
120
70
100
60
80
50
40
# of trials
# of trials
60
40
30
20
20
10
0
7.3
7.4
7.5
7.6
7.7
7.8
7.9
0
6.7
6.8
seari.mit.edu
6.9
7
7.1
7.2
7.3
7.4
7.5
7.6
Expected system entropy
Expected system entropy
© 2012 Massachusetts Institute of Technology
18
7.7
Conclusion
• Systems can use similar path enablers to achieve many
different states, and these changes may be characterized by
multiple ilities
• Despite desire to implement ilities in designs, a clear method is
still lacking for identifying “good” designs with respect to
preferred ilities
• Possible to classify / decompose system changes in a 10dimensional semantic basis to describe the change mechanism
• Ilities are attributed not a priori, but as an outcome to the change
• Enables decision makers to create better requirements that are
clear, concise, and verifiable.
Clarity in language is important as we look for design insights in
groups of change mechanisms present in many systems.
seari.mit.edu
© 2012 Massachusetts Institute of Technology
19
Questions?
seari.mit.edu
© 2012 Massachusetts Institute of Technology
20