Model-Based Requirement Definition for
Instrument Systems
Matthew William Smith, David W. Miller
June 2014
SSL #3–14
2
Model-Based Requirement Definition for
Instrument Systems
Matthew William Smith, David W. Miller
June 2014
SSL #3–14
This work is based on the unaltered text of the thesis by Matthew William Smith
submitted to the Department of Aeronautics and Astronautics in partial fulfillment
of the requirements for the degree of Doctor of Philosophy at the Massachusetts
Institute of Technology.
4
Model-Based Requirement Definition
for Instrument Systems
by
Matthew William Smith
Submitted to the Department of Aeronautics and Astronautics
on May 22, 2014, in partial fulfillment of the
requirements for the degree of
Doctor of Philosophy in Aeronautics and Astronautics
Abstract
Instrument systems such as imagers, radars, spectrometers, and radiometers are important to users in the astronomy, Earth science, defense, and intelligence communities. Relatively early in the development cycle, performance requirements are defined
at the top level and allocated to various subsystems or components. This is a critical step, as poor requirement definition and resulting requirement instability has
historically led to increased cost and, in some cases, program cancelation. Defining
requirements for instrument systems is uniquely challenging in part due to the divide
between system users (e.g. scientists) and system designers (e.g. engineers). The two
groups frequently differ in terms of background, objectives, and priorities, and this
disconnect often leads to difficulty in evaluating and resolving requirement trade-offs.
The objective of this thesis is to develop a model-based approach to requirement
definition that addresses the above challenges. The goal of the methodology is to
map science objectives to a set of top-level engineering requirements in a manner that
enables traceability in the requirement hierarchy and facilitates informed trades across
the science/engineering interface. This is accomplished by casting the requirement
definition process as an optimization problem. First, an executable instrument model
is created to capture the forward mapping between engineering decisions and science
capability. The model is then exercised to find an inverse mapping that produces
multiple sets of top-level engineering requirements that all meet the performance
objectives. A new heuristic optimization algorithm is developed to carry out this task
efficiently and exhaustively. Termed the Level Set Genetic Algorithm (LSGA), this
procedure identifies contours of equal performance in the design space using an elitepreserving selection operator to ensure convergence, together with a global diversity
metric to ensure thorough exploration. LSGA is derivative-free, parallelizable, and
compatible with mixed integer problems, making it applicable to a wide variety of
modeling and simulation scenarios.
As a case study, the model-based requirement definition methodology is applied
to the Regolith X-ray Imaging Spectrometer (REXIS), an instrument currently in development at MIT and scheduled to launch on NASA’s OSIRIS-REx asteroid sample
5
return mission in the fall of 2016. REXIS will determine the elemental composition
of the target asteroid by measuring the solar-induced fluorescence spectrum in the
soft x-ray regime (0.5–7.5 keV). A parametric model of the instrument is created to
simulate its end-to-end operation, including x-ray propagation and absorption, detector noise processes, pixel read-out, signal quantization, and spectrum reconstruction.
After validating the model against laboratory data, LSGA is used to identify multiple sets of top-level engineering requirements that meet the REXIS science objectives
with regard to spectral resolution. These results are compared to the existing baseline
requirement set, providing insights into the alternatives enabled by the model-based
approach. Several additional strategies are presented to quantify and mediate requirement trades that may occur later in the development cycle due to science creep or
engineering push-back. Overall, these methods provide a means of synthesizing and
then evaluating top-level engineering requirements based on given science objectives.
By doing so in a comprehensive and traceable manner, this approach clarifies the
trade-offs between scientists and engineers that inevitably arise during the design of
instrument systems.
Thesis Committee Chair:
David W. Miller
Professor of Aeronautics and Astronautics
Thesis Committee Member:
Sara Seager
Professor of Earth, Atmospheric and Planetary Sciences
Thesis Committee Member:
Kerri L. Cahoy
Assistant Professor of Aeronautics and Astronautics
Thesis Committee Member:
Rebecca A. Masterson
REXIS Instrument Manager
6
Acknowledgments
This work was funded by the NASA Jenkins Pre-Doctoral Fellowship Program, the
NASA Space Technology Research Fellowship Program (grant #NNX11AM63H) and
the REXIS program (grant #NNG12FD70C).
7
8
Contents
1 Introduction
19
1.1
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
1.2
Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
1.2.1
The Importance of Good Requirements . . . . . . . . . . . . .
23
1.2.2
The Difficulty of Good Requirements . . . . . . . . . . . . . .
27
1.3
Scope
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31
1.4
Research Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
1.5
Thesis Roadmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
2 Literature Review
35
2.1
Requirement Definition Processes . . . . . . . . . . . . . . . . . . . .
35
2.2
Model-Based Design Methodologies . . . . . . . . . . . . . . . . . . .
37
2.3
Project-Specific Approaches to Requirements . . . . . . . . . . . . . .
41
2.4
Research Gap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
46
3 Model-Based Requirement Definition Methodology
3.1
3.2
47
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
48
3.1.1
Desired Methodology Properties . . . . . . . . . . . . . . . . .
48
3.1.2
Steps for Model-Based Requirement Definition . . . . . . . . .
50
3.1.3
Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . .
53
3.1.4
Star Camera Sample Problem . . . . . . . . . . . . . . . . . .
53
Step 1: Quantify User Objectives . . . . . . . . . . . . . . . . . . . .
56
3.2.1
56
Data Product . . . . . . . . . . . . . . . . . . . . . . . . . . .
9
3.3
3.4
3.5
3.2.2
Figure of Merit . . . . . . . . . . . . . . . . . . . . . . . . . .
57
3.2.3
Performance Requirement . . . . . . . . . . . . . . . . . . . .
57
3.2.4
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
58
Step 2: End-to-End Model Creation . . . . . . . . . . . . . . . . . . .
58
3.3.1
Requirement Variables and Vectors . . . . . . . . . . . . . . .
59
3.3.2
Identifying Requirement Variables . . . . . . . . . . . . . . . .
61
3.3.3
Model Implementation . . . . . . . . . . . . . . . . . . . . . .
67
3.3.4
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
75
Step 3: Generate Requirement Vectors (Optimization)
. . . . . . . .
76
3.4.1
Goal Programming . . . . . . . . . . . . . . . . . . . . . . . .
79
3.4.2
Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . .
81
3.4.3
Level Sets and Diversity Preservation . . . . . . . . . . . . . .
84
3.4.4
Level Set Genetic Algorithm (LSGA) . . . . . . . . . . . . . .
89
3.4.5
Star Camera Requirement Vectors . . . . . . . . . . . . . . . . 111
3.4.6
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Step 4: Select a Requirement Vector . . . . . . . . . . . . . . . . . . 115
3.5.1
Local Sensitivity Metrics . . . . . . . . . . . . . . . . . . . . . 115
3.5.2
Derivative-Free Local Sensitivity Metric
3.5.3
Choosing A Robust Requirement Set . . . . . . . . . . . . . . 122
4 REXIS
4.1
. . . . . . . . . . . . 117
125
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.1.1
Instrument Overview . . . . . . . . . . . . . . . . . . . . . . . 127
4.1.2
Concept of Operations . . . . . . . . . . . . . . . . . . . . . . 131
4.1.3
Overview of CCD X-Ray Spectroscopy . . . . . . . . . . . . . 132
4.2
User Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
4.3
End-to-End Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.3.1
Measurement Pipeline . . . . . . . . . . . . . . . . . . . . . . 137
4.3.2
Model Implementation . . . . . . . . . . . . . . . . . . . . . . 147
4.3.3
Model Validation . . . . . . . . . . . . . . . . . . . . . . . . . 163
10
4.3.4
Baseline Performance and Single-Axis Studies . . . . . . . . . 169
4.4
Requirement Sets through LSGA . . . . . . . . . . . . . . . . . . . . 172
4.5
Requirement Trades
4.6
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
. . . . . . . . . . . . . . . . . . . . . . . . . . . 176
5 Conclusions
185
5.1
Thesis Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
5.2
Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
5.3
Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
A Multiobjective Considerations
193
A.1 Multiobjective LSGA . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
A.2 Comparison to NSGA-II . . . . . . . . . . . . . . . . . . . . . . . . . 195
11
12
List of Figures
1-1 Requirement flow down. . . . . . . . . . . . . . . . . . . . . . . . . .
21
1-2 System life cycle cost and cost impact of changes . . . . . . . . . . .
24
1-3 SBIRS cost program estimates over time . . . . . . . . . . . . . . . .
27
1-4 V-model of the system development life cycle. . . . . . . . . . . . . .
32
3-1 Methodology input and output . . . . . . . . . . . . . . . . . . . . .
47
3-2 Steps in the model-based requirement definition methodology . . . . .
51
3-3 Star camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54
3-4 HAS2 CMOS imager . . . . . . . . . . . . . . . . . . . . . . . . . . .
55
3-5 End-to-end model . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
59
3-6 Model of functional decomposition using requirement variables . . . .
61
3-7 Star camera measurement pipeline . . . . . . . . . . . . . . . . . . . .
63
3-8 Star camera imaging system pipeline . . . . . . . . . . . . . . . . . .
64
3-9 HAS2 sensor operation . . . . . . . . . . . . . . . . . . . . . . . . . .
65
3-10 Star camera centroid processing algorithm . . . . . . . . . . . . . . .
66
3-11 Simulated star camera windows. . . . . . . . . . . . . . . . . . . . . .
71
3-12 Centroid accuracy as a function of star brightness . . . . . . . . . . .
72
3-13 Stars in field of view as a function of brightness . . . . . . . . . . . .
73
3-14 Star camera simulation output versus number of Monte Carlo samples
74
3-15 Monte Carlo error (MCE) estimate versus number of samples . . . . .
76
3-16 Contours of N0.1 over the requirement space . . . . . . . . . . . . . .
78
3-17 General genetic algorithm (GA). . . . . . . . . . . . . . . . . . . . . .
83
3-18 Basic GA and the problem of diversity collapse . . . . . . . . . . . .
85
13
3-19 Crowding metric for an initial population. . . . . . . . . . . . . . . .
92
3-20 Spread component of the diversity metric . . . . . . . . . . . . . . . .
94
3-21 Spacing component of the diversity metric . . . . . . . . . . . . . . .
95
3-22 Diversity metric illustrated for various cases . . . . . . . . . . . . . .
97
3-23 One generation of LSGA . . . . . . . . . . . . . . . . . . . . . . . . .
98
3-24 Replacing members in the level set . . . . . . . . . . . . . . . . . . . 105
3-25 LSGA level set solutions for test function f1 . . . . . . . . . . . . . . 107
3-26 LSGA metrics for function f1 . . . . . . . . . . . . . . . . . . . . . . 108
3-27 LSGA with and without level set diversity enhancement
. . . . . . . 109
3-28 Test function f2 (x) and LSGA results . . . . . . . . . . . . . . . . . . 110
3-29 Test function f3 (x) and LSGA results . . . . . . . . . . . . . . . . . . 111
3-30 Test function f4 (x) and LSGA results . . . . . . . . . . . . . . . . . . 112
3-31 Diversity metrics for test function f4 (x) . . . . . . . . . . . . . . . . . 112
3-32 LSGA results for the star camera sample problem. . . . . . . . . . . . 113
3-33 Diversity metrics for the star camera sample problem. . . . . . . . . . 114
3-34 Monte Carlo samples for sensitivity analysis . . . . . . . . . . . . . . 118
3-35 Scatter plots for sensitivity analysis . . . . . . . . . . . . . . . . . . . 119
3-36 Star camera sensitivity analysis results. . . . . . . . . . . . . . . . . . 123
4-1 REXIS mounting location on OSIRIS-REx . . . . . . . . . . . . . . . 126
4-2 Simulated X-ray fluorescence spectrum for the asteroid Bennu . . . . 128
4-3 Major components of REXIS . . . . . . . . . . . . . . . . . . . . . . . 129
4-4 REXIS focal plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
4-5 REXIS concept of operations. . . . . . . . . . . . . . . . . . . . . . . 132
4-6 Sample CCD spectrum . . . . . . . . . . . . . . . . . . . . . . . . . . 134
4-7 Effect of varying spectral resolution . . . . . . . . . . . . . . . . . . . 135
4-8 Overall REXIS measurement pipeline . . . . . . . . . . . . . . . . . . 138
4-9 REXIS spectrometer measurement flow . . . . . . . . . . . . . . . . . 139
4-10 Charge measurement process within CCDs . . . . . . . . . . . . . . . 140
4-11 CCID-41 layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
14
4-12 Data flow within REXIS on-board electronics . . . . . . . . . . . . . 144
4-13 REXIS event grade definitions . . . . . . . . . . . . . . . . . . . . . . 145
4-14 REXIS ground processing flow . . . . . . . . . . . . . . . . . . . . . . 146
4-15 REXIS simulation block diagram. . . . . . . . . . . . . . . . . . . . . 148
4-16 Simplified CCID-41 layer structure . . . . . . . . . . . . . . . . . . . 151
4-17 Absorption coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . 152
4-18 Number of parallel and serial transfers per pixel . . . . . . . . . . . . 157
4-19 Sample CCD pixel windows . . . . . . . . . . . . . . . . . . . . . . . 158
4-20 Simulated REXIS histogram . . . . . . . . . . . . . . . . . . . . . . . 159
4-21 Simulated REXIS spectrum . . . . . . . . . . . . . . . . . . . . . . . 160
4-22 REXIS radiation environment. . . . . . . . . . . . . . . . . . . . . . . 163
4-23 REXIS simulation convergence . . . . . . . . . . . . . . . . . . . . . . 164
4-24 Dark current model validation . . . . . . . . . . . . . . . . . . . . . . 165
4-25 Spectral resolution versus X-ray energy . . . . . . . . . . . . . . . . . 166
4-26 Spectral resolution versus temperature . . . . . . . . . . . . . . . . . 168
4-27 Radar plot of baseline REXIS requirements . . . . . . . . . . . . . . . 170
4-28 REXIS single-axis studies . . . . . . . . . . . . . . . . . . . . . . . . 171
4-29 LSGA diversity metric for REXIS simulation run . . . . . . . . . . . 173
4-30 Three REXIS requirement sets produced by LSGA . . . . . . . . . . 174
4-31 Requirement set corner cases . . . . . . . . . . . . . . . . . . . . . . . 175
4-32 Allocating performance margin equally among engineering requirements.177
4-33 Changes to relax REXIS engineering requirements . . . . . . . . . . . 179
4-34 Requirement changes to accommodate engineering push-back . . . . . 180
4-35 Changing requirements to increase science performance. . . . . . . . . 182
A-1 Multiobjective level surfaces . . . . . . . . . . . . . . . . . . . . . . . 194
A-2 Multiobjective level set optimization . . . . . . . . . . . . . . . . . . 196
A-3 Level curves for NSGA-II comparison. . . . . . . . . . . . . . . . . . . 197
A-4 Comparison of NSGA-II and LSGA . . . . . . . . . . . . . . . . . . . 198
15
16
List of Tables
1.1
Science/engineering divide . . . . . . . . . . . . . . . . . . . . . . . .
30
3.1
Quantified user objectives for the star camera problem . . . . . . . .
58
3.2
Requirement variables for the star camera problem . . . . . . . . . .
67
3.3
Star camera model input parameters . . . . . . . . . . . . . . . . . .
68
3.4
LSGA optimization parameters . . . . . . . . . . . . . . . . . . . . .
90
3.5
Star tracker requirement vectors . . . . . . . . . . . . . . . . . . . . . 114
4.1
REXIS Design Parameters . . . . . . . . . . . . . . . . . . . . . . . . 131
4.2
X-ray lines measured by REXIS . . . . . . . . . . . . . . . . . . . . . 135
4.3
REXIS spectral resolution requirements . . . . . . . . . . . . . . . . . 136
4.4
REXIS simulation input parameters . . . . . . . . . . . . . . . . . . . 149
4.5
Electron-hole pair libration energies . . . . . . . . . . . . . . . . . . . 153
4.6
CCD test parameters vs. REXIS flight parameters . . . . . . . . . . . 167
4.7
REXIS requirement variables, baseline values and side bounds . . . . 169
4.8
Requirement values for Figure 4-32. . . . . . . . . . . . . . . . . . . . 177
4.9
Requirement values for Figure 4-33. . . . . . . . . . . . . . . . . . . . 178
4.10 Requirement values for Figure 4-35. . . . . . . . . . . . . . . . . . . . 181
17
18
Chapter 1
Introduction
Instrument systems—telescopes, radars, spectrographs, sounders, etc.—are of critical
value to users in the astronomy, Earth science, defense, and intelligence communities. In general, such systems sample a portion of the electromagnetic spectrum
and produce data products containing the measurement of interest, often after some
post-processing steps. These data are then used to deepen our knowledge and understanding of the physical universe, the climate of our own planet, localized severe
weather, the impact of human development, military activities, geopolitical events,
and other aspects of the environment that we seek to characterize from afar. Instrument systems have therefore become an indispensable source of information for
users across multiple industries, environments, and regions of the electromagnetic
spectrum.
Defining good requirements is a critical step in the development of any complex
system. During this process, overall mission objectives are distilled into mission
requirements, top-level engineering requirements, subsystem requirements, and on
down into lower levels of design until “build-to” specifications have been produced.
These early decisions with respect to requirements determine a large percentage of
downstream system cost. Failure to produce good requirements often results in requirement instability and late design changes that increase program cost and add
schedule delays.
Requirement definition for instrument systems is especially challenging due to a
19
divide that often exists between users of the system such as scientists and designers of the system such as engineers. Moreover, current error budgeting techniques
that are typically used for observation system requirement definition are inadequate.
Traditional methods often focus on sensitivity around a point design (rather than exploring the entire design space) and fail to capture important effects such as software
processing, systematic noise sources, and instrument calibration errors.
This thesis presents a model-based requirement definition methodology for instrument systems that attempts to remedy these drawbacks. At the center of the approach
is an end-to-end system model that serves as a mapping between engineering design
choices and science data products. After building and validating the model, it is
exercised using a new purpose-built heuristic algorithm to locate the set of top-level
engineering requirements that meet science performance objectives. There are many
such sets, all of which perform equally well from a science perspective, therefore the
final step in the methodology is to conduct a sensitivity analysis to determine which
set of requirements is most robust to change. This is motivated by the fact that
requirement changes are historically a major source of cost and schedule growth for
aerospace systems.
1.1
Overview
Requirement definition is the process of translating stakeholder goals and preferences
into technical specifications on a system and its constituent subsystems. This allocation or flow down of high level preferences into lower level requirements results in a
series of “shall” statements that capture what the system must do in order to meet
the stakeholders’ expectations.
The requirements allocation process is shown schematically in Figure 1-1. The
mission objectives are top-level qualitative statements describing the goals of the
project. For example, the mission objective of the Regolith X-ray Imaging Spectrometer (REXIS) instrument—described in greater detail below—is to measure the global
elemental abundance of the asteroid Bennu. Each mission objective is then translated
20
Figure 1-1: Flow down of requirements from qualitative mission objectives to quantitative mission requirements to system- and subsystem-level engineering requirements
(adapted from [1]).
into one or more quantitative mission requirements, which capture the science goals
using numerical values instead of prose. The mission requirement corresponding to
the global abundance objective is to measure the elemental ratios Mg/Si, Fe/Si, and
S/Si to within 20% of that for a CI chondrite). With mission requirements defined,
one proceeds to define engineering requirements, first at the top level and then on
downward. The identifiers L0, L1, etc. are often used as shorthand to refer to requirements at a given level of decomposition.
Of particular importance is the boundary between the user domain and designer
domain, shown as a red dashed line in Figure 1-1. Mission objectives and mission
requirements both express the goals of the user through qualitative and quantitative
means, respectively. Engineering requirements (L2 and below in Figure 1-1), on the
other hand, are under the purview of designers who are responsible for creating a
solution that satisfies the user’s objectives and can be implemented in hardware and
software. In the case of a science mission the users typically consist of the Principal
21
Investigator and science team, while the designers are the engineers constructing the
instrument.1 As will be discussed below, one of the fundamental challenges of defining
requirements for instrument systems is the need to do so across the user/designer
(scientist/engineer) boundary.
The NASA Systems Engineering Handbook [1] distinguishes between several different types of requirements.2 Functional requirements define what functions need
to be done to accomplish the objectives. Performance requirements define how well
the system needs to perform the functions. There are also constraints that capture
externally-imposed bounds on certain engineering or scientific choices, and interface
requirements that specify how the system will interact (mechanically, electrically,
thermally, etc.) with other systems or subsystems. This thesis is concerned primarily
with performance requirements.
The language used to discuss requirements can be organization-dependent. The
DoD often refers to key performance parameters (KPPs) when discussing high-level
requirements or objectives for overall system performance. Specifically, KPPs are defined as “those attributes or characteristics of a system that are considered critical or
essential to the development of an effective military capability” [140]. KPPs must be
testable to enable feedback from test and evaluation efforts during system validation.
Related to KPPs, the NASA Systems Engineering Handbook [1] identifies three
other technical measures: MOEs, MOPs, and TPMs.
Measures of effectiveness
(MOEs) are high-level measures of operational success that are closely related to
achievement of mission goals. MOEs are in the user domain; they define requirements from the customer/stakeholder perspective in a solution-neutral way. This
thesis will use the previously defined terms “mission objectives” and “mission requirements” instead of “MOEs”. Measures of performance (MOPs) are functional
attributes of the design that are important to achieving the mission objectives (i.e.
1
This thesis will occasionally use the terms “user”/“designer” in place of “scientist”/“engineer”
to reflect the fact that not all users of remote sensing and observation systems are scientists per se
(e.g. military operators and planners are the primary users of DoD spacecraft and instruments).
2
The NASA Systems Engineering Handbook discusses a variety of more specialized requirement
categories that are not discussed in this thesis (e.g. reliability requirements, safety requirements,
human factors engineering requirements, etc.).
22
MOEs), but that do not measure the MOEs directly. Typically multiple MOPs contribute to any single MOE. In the language of this thesis MOPs are equivalent to the
top-level engineering requirements. Finally, technical performance measures (TPMs)
are critical performance or success metrics that are tracked by the project during
the design and implementation phase. TPMs are usually selected from the MOEs
and MOPs, and serve as a barometer of how the system is performing as it matures
throughout development.
1.2
Motivation
Almost all recent space acquisition efforts, including several advanced instrument systems, have encountered worrisome cost growth and schedule delays. In many cases
these problems have been traced to deficiencies in the requirement definition and
allocation process. The methodology described in this thesis improves requirement
definition through the use of modeling, optimization, and sensitivity analysis to explicitly link designer (engineer) concerns with those of users (scientists).
This section highlights the theoretical and practical importance of good requirements, with emphasis on downstream impacts on the design/implementation phases
of development. The focus then turns to the underlying reasons why requirement
definition is often difficult. The model-based requirement definition methodology is
a direct response to these challenges.
1.2.1
The Importance of Good Requirements
Instrument systems are typically developed in a systems engineering context in which
top-level requirements are developed and then flowed down (“allocated”) to subsystem layers during the design phase (see Figure 1-2). This requirement definition process can have far-reaching consequences on the success of an observation and remote
sensing program. Programs with poorly defined requirements are at risk of requiring
design changes later in the development cycle than would otherwise be necessary. Additional design changes can also result from requirements instability or “requirements
23
creep” in which new requirements are added later and without rigorous justification
or analysis. Engineering re-work as a result of either poorly crafted original requirements or requirements creep will consume additional resources in terms of time, cost,
or both. In short, good requirements are essential if a program is to stay within its
cost and schedule baselines.
(a)
(b)
Figure 1-2: Factors underlying the schedule and cost consequences of poor requirement definition: (a) commitment of life cycle cost over time and (b) the cost impact
of changes over time. Both figures are from Blanchard [8].
Conceptually, the outsized role of requirement definition in determining overall
success is due to the way the costs of engineering decisions are distributed over the
program life cycle. Figure 1-2(a) shows notional cumulative expenditures over time
(dashed line). The graph also shows the percentage of cost committed at a certain
time in the program (solid line). This is equivalent to the cost-reduction opportunity
at a given point. During the system design and development phase, the opportunity for cost reduction is high because very little cost has actually been committed.
Likewise, the cumulative cost is low. Projected cumulative cost increases relatively
quickly as a result of design work and is accompanied by a sharp reduction in cost
reduction opportunity as engineering decisions “lock in” expenditures. We see that
a majority of the overall system cost is determined by choices made early in the development process. Equivalently—and importantly—the greatest opportunity for life
cycle cost reduction occurs during the system design and development phase when
requirements are being defined and allocated.
24
Figure 1-2(b) shows what happens when those “locked in” engineering decisions
are revised later in the development cycle. The plot shows the cost of design changes
as a function of time. Ideally, the majority of design changes are made during the
preliminary design phase, resulting in a cost peak during that phase of the program.
In practice, however, there is a tendency to keep requirements as non-specific as
possible, particularly lower-level requirements. As Blanchard [8] points out, engineers
often do not want to be forced into design decisions earlier than necessary. This
results in delaying low-level requirements specification until the detailed design phase
or possibly even the production phase. Design changes become much more costly to
incorporate downstream due to the need for re-work and the unintended impacts on
other aspects of the system. This situation can be avoided by defining requirements
appropriately early in the program life cycle (earlier than the current practice). This
is a challenging proposition but one that the methodology outlined in this thesis
attempts to solve.
Many high performance observation systems across DoD and NASA have experienced cost growth, schedule delays, or outright cancelation as a result of requirements
creep or other requirements-related problems. This is the conclusion of several studies
over the past decade by various government-chartered entities. In a 2008 report titled
Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and
Benefits for Future Air Force Acquisition [19] a committee of the National Research
Council (NRC) analyzed the root causes of program failure in Air Force projects,
with special attention to the early phases of the product life cycle. They observed
that thoroughly defined and stable requirements early in the program are essential
to overall success. One of the main conclusions was that “beginning development
without a complete list of stable requirements is one of the key ‘seeds of failure’ [...]
It is important to complete requirements trade-offs prior to the development phase.”
The NRC study came five years after another relevant report authored by a joint
task force of the Defense Science Board (DSB) and Air Force Scientific Advisory
Board (SAB) on the acquisition of national security space programs [46]. The task
force was directed to characterize problems in the DoD’s space enterprize by looking
25
at underlying causes and systemic issues such as cost growth and schedule delays
that impact space programs. Their conclusions were similar to those of the NRC
committee, finding “requirements definition and subsequent control, or lack thereof,
to be a dominant driver of cost increases, schedule delays, and incurred mission risk.”
Both the NRC and DSB/SAB studies also provide useful insight at the level of
individual programs. One example is the Space Based Infrared System (SBIRS), a
constellation of stand-alone spacecraft and hosted payloads that will provide a ballistic missile early warning capability replacing the legacy Defense Support Program
(DSP) satellites [129]. SBIRS has had a troubled development history. The full-scale
development contract was awarded in 1996 and the first geosynchronous satellite was
not launched until 2011 [10]. The program experienced two Nunn-McCurdy breaches3 ,
one in 2002 and a second in 2005.
The NRC committee found that one of the root causes for issues plaguing SBIRS
was “a high degree of requirements instability after Milestone B (or its equivalent),
driven by shifting missile defense strategies and tactical requirements changes.” [19]
The DSB/SAB task force reached a similar conclusion and published data showing
that added requirements caused between $1.1B and $1.3B in additional program cost
(see Figure 1-3).
In addition to the works highlighted above, the interested reader is referred to several Government Accountability Office (GAO) studies that discuss cost and schedule
overruns in space missions, including the role of requirement definition. For example,
in the 2003 report Improvements Needed in Space Systems Acquisition Management
Policy [101] the authors identify requirement changes to the DoD Advanced Extremely High Frequency (AEHF) communications satellite as causing major design
modifications costing hundreds of millions of dollars. Similar conclusions to the NRC
and DSB/SAB report were reached in the 2005 study Stronger Development Practices and Investment Planning Needed to Address Continuing Problems [102]. Other
relevant GAO reports include GAO-11-239 [103] and GAO-11-590 [104].
3
A Nunn-McCurdy breach occurs when unit cost exceeds the baseline value by 25 percent or
more, requiring congressional recertification.
26
Figure 1-3: SBIRS program cost estimates as a function of time from [46]. Note the
cost impact of requirements growth. Estimates include the President’s budget, Most
Probable Cost (MPC), Estimate at Completion (EAC), and Program Office Estimate
(POE).
1.2.2
The Difficulty of Good Requirements
Having shown the importance of good requirements, we now turn to the fact that
defining and allocating requirements is a challenging exercise. Indeed, many of the
cases cited above are proof of the difficulty. Instrument systems are particular in
that there is often a fundamental divide between those who use such systems (e.g.
scientists) and those who design them (e.g. engineers). Scientists and engineers are
often mentioned in the same breath, however these two groups are very different in the
context of instrument systems. Scientists reside in the “user” domain; they specify
the top-level measurement goals and performance metrics, and are the end users of
data products generated by the system. Engineers reside in the “designer” domain;
they are responsible for technical design, implementation, and test. A divide between
the two groups is often the source of tension for projects, and can arise for a variety
of reasons:
• Goals and priorities Scientists and engineers typically have very different
goals and priorities in mind. Scientists desire to maximize system performance
and produce scientifically meaningful results. Engineers aim to produce feasible
designs that balance performance with cost, mass, power, and other engineeringrelated quantities. These two objectives are frequently in opposition.
27
• Language Engineers are often unfamiliar with the specialized vocabulary used
by scientists and vice versa. A word that is commonplace to one group of
individuals may be completely unfamiliar to the other. Moreover, one word
may have two or more completely different meanings depending on which group
is using it.
• Background Scientists and engineers often bring divergent perspectives to the
project as a result of different academic or professional backgrounds. This could
be in terms of work environment (academia vs. industry vs. government), organizational structure (small research laboratory vs. large matrix organization),
and experience navigating various political and/or bureaucratic environments.
• Analysis tools Scientists and engineers often use their own unique set of tools
that may be unfamiliar to those outside their domain. This could either be in
terms of specific software packages or hardware products, or more generally
the approaches used to solve problems. A scientist may prefer to scrutinize a
problem in depth using analytical models based on first principles. An engineer,
on the other hand, may prefer numerical methods or experimentation. The
reverse could also be true.
Effective requirement definition necessarily requires bridging the user/designer
divide as science objectives are allocated to various engineering-level specifications
(recall the horizontal dashed line in the requirements flow diagram in Figure 1-1).
A lack of coordination and understanding between scientists and engineers results
in poorly defined requirements or time-consuming design iteration between the two
groups.
The scientist/engineer divide has been noted anecdotally by those experienced in
the field and also has been recognized in the literature. One of the earliest studies was
the 1972 work of Holland and Stead [61] who analyzed the communication differences
of scientists and engineers in a technical organization building a multi-disciplinary
scientific instrument. They interviewed employees, who attributed communication
difficulties between the two groups to “natural differences between scientists and
28
engineers.” The authors then used records of written communication to quantify these
differences in terms of response length, words per sentence, and choice of words. In
line with the language barrier noted above, the word choice case showed that scientists
and engineers did not generally use the same vocabulary for evaluating a common
situation.
More recent work has focused on similar issues in the context of software development in research settings. In a 2005 work, Segal [124] studies a group of software
engineers (i.e. designers) and research scientists (i.e. users) who are together developing an instrument to be sent into space. The analysis focuses on requirement
definition, with special attention to the cultural differences that complicate the effort. The employees in the study had in place a “traditional” top-down requirement
definition procedure that relied on early specification of science requirements by the
scientists, followed by requirement allocation, code implementation by the engineers,
and requirement verification. In practice, however, requirement definition turned out
to be an iterative process, with the requirements constantly in a state of change.
The scientists would defer on specifying requirements, much to the frustration of the
software engineers. One of the scientists noted as much in the following quote:
[...] being like a bunch of scientists, we [thought] we could change everything up until the last minute ... [the software engineers] were just saying
“Sort the requirements out now! Do it now! You haven’t got time!” [124]
Segal also discusses the difference in vocabulary and writing style between scientists
and engineers, and how this led to challenges in producing a requirements specification
document. As a result, the requirements and specification documents failed to foster
a complete and shared understanding between the two groups.
There are several other relevant examples in the literature. Segal followed the
above study with another report in 2008 [125] that discussed similar issues. Among
them is the contrast between engineers, who are accustomed to following an “upfront”
process (where requirements are defined at the start of a project) versus scientists,
who often use an “emergent” process (in which requirements emerge over time via
29
iteration). The author summarizes the conflict using the dialogue shown in Table 1.1.
While the exchange is notional and somewhat tongue-in-cheek, it is based on interviews with scientists and engineers and accurately reflects the nature of many
requirements discussions between the groups.
Table 1.1: Notional dialogue between an engineer and a scientist on requirements
generation (from [125]).
Engineer
I need your
requirements.
Scientist
Sorry, haven’t quite
worked out what they
are yet.
I need your
requirements NOW.
Ooh, wouldn’t it be
interesting if we
tried this...
Give me your
requirements!
Just have to work
out what’s going on
here...
(sigh)
(sigh)
The theme of engineer/scientist conflict in software development is further explored by Killcoyne and Boyle [72], who are concerned primarily with the life sciences.
Gilmore [50] discusses the European Extremely Large Telescope (E-ELT) project and
the tendency of scientists to seek systems that maximize performance. These demands, he points out, are often in tension with questions of technical and cost feasibility. Avnet [87] considers knowledge transfer within a multidisciplinary team; his
results are discussed in greater detail in the literature review.
The authors of the NRC report discussed above recognize the important role of
bridging the user/designer divide in the context of requirement definition, stating that
“tight partnership and collaboration between the users/sponsors and the developer
30
are critical to stabilize requirements in the formative phase of a program.” [19]
Several sources discuss the use of modeling and simulation (M&S) during early
project phases and the ways in which M&S could be used to enhance requirement
definition. The NRC report [19] emphasizes M&S as an important element in the early
development of complex systems, particularly in the management of requirements
changes and performance verification. The following excerpt from that report does a
good job of capturing the motivation behind this thesis:
Modeling, simulation, and analysis will play a major role in identifying
key design parameters—the “long poles in the tent”—which are associated
with potential risk, sensitivity, or uncertainty and on which the ability of
the design to meet important requirements will depend. Whether the parameter is processing throughput or response time or the acquisition cost
per removal-free operating hour, early identification provides guidance to
the designers and focus to risk-mitigation planning activities such as early
benchmarking or development of alternatives.
Finally, Walton [144] presents a history of M&S in DoD acquisition processes with
a focus on system-level requirement generation. The author surveyed various organizations to determine the scope of current M&S use. He finds that the effectiveness
of M&S activities during requirement definition is undocumented and areas of high
leverage are unknown. Still, Walton finds an important benefit of M&S to be that
models and simulations serve as “communication boundary objects” that facilitate the
sharing of knowledge between diverse groups of people working on the project. This
finding is important and is something that the model-based requirement methodology
is designed to emphasize.
1.3
Scope
The requirement definition and allocation process occurs within the context of a larger
system development life cycle. Figure 1-4 depicts the so-called “vee model” or “Vmodel” of the development life cycle. This particular diagram incorporates elements
31
from Forsberg and Mooz [47, 48], the INCOSE Systems Engineering Handbook [106],
Blanchard [8], and de Weck [105].
Figure 1-4: V-model of the system development life cycle (diagram based on [47, 48,
8, 105]). The scope of this thesis is primarily requirement definition and allocation.
System development begins in the upper left with mission requirement definition
and concludes at the upper right with full-scale utilization, maintenance, and support.
The left side of the V consists of translating the high-level objectives expressed by the
mission requirements into a detailed “build-to” design. Fabrication and integration
happens on the right side of the V, along with verification and validation as the
system takes shape.
Figure 1-4 also shows feedback to earlier steps. This iteration frequently occurs in
earlier requirement definition and allocation phases as users and designers trade between desired capabilities and technical constraints. Occasionally, iteration also takes
32
place after integration (shown by the gray arrows). This is undesirable because it significantly increases costs due to re-work. As discussed previously in Section 1.2.1,
much of the problem can be traced to poor requirement definition. In the words
of Blanchard, “Traditionally, the requirements have not been well defined from the
beginning, resulting in some rather extensive and costly efforts during the final integration and test activity” [8]. Therefore the primary focus of this thesis is on the
requirement definition and allocation phase of the project life cycle, with the aim reducing the amount of iteration and increasing the efficiency of the remaining feedback
loops.
1.4
Research Objectives
The objective of this thesis is to develop a model-driven approach to requirement definition that addresses the underlying challenges identified in Section 1.2. The goal of
the methodology is to map science objectives to a set of top-level engineering requirements. Furthermore, we seek to do so in a manner that enables traceability in the
requirement hierarchy and facilitates informed trades across the science/engineering
interface. This primary objective motivates several sub-goals as follows,
• To create an optimization method to efficiently and exhaustively identify sets of
engineering requirements that produce the desired level of science performance.
We aim to develop computational tools that provide a mathematical relationship
between the user (science) domain and designer (engineering) domain.
• To develop a “model agnostic” approach that is applicable to a variety of instrument systems. “Instruments,” as a category of systems, are incredibly diverse.
The term applies to imagers, spectrometers, interferometers, etc. Consequently,
the models used to simulate instrument performance can take on a variety of
forms (e.g. continuous or discrete, frequency domain or time domain, linear or
non-linear, etc). We therefore seek to develop an approach that imposes as few
limitations on the model as possible in order to achieve broad applicability.
33
• To facilitate trades across the science/engineering interface in the event of science requirement creep or engineering push-back. Requirements are rarely
static. As a program evolves, technical concerns or other unanticipated changes
mare require a re-evaluation of the requirements. We aim to provide a quantitative framework for analyzing such trade-offs.
• To demonstrate the approach through a case study of the Regolith X-ray Imaging Spectrometer (REXIS) instrument. By applying the methodology to an
actual flight instrument, we aim to show its utility for future requirement definition and allocation activities in real-world settings.
1.5
Thesis Roadmap
The following chapter presents a review of literature relevant to this work. Existing
requirement definition methodologies are considered, along with model-based design
methodologies and requirement approaches adopted by specific instrument projects.
Chapter 3 presents the model-based requirement definition methodology in detail.
At the heart of this four-step process is a new heuristic optimization technique called
the Level Set Genetic Algorithm (LSGA) designed to carry out an efficient and exhaustive search of the feasible requirement space to find designs that meet science
objectives. Chapter 3 also introduces a relatively star tracker sample problem that
is used to illustrate the concepts and tools being discussed. Chapter 4 applies the
model-based requirement definition methodology to REXIS, an X-ray spectrometer
under development at MIT. More complex than the simple star tracker, REXIS serves
as a real-world case study for the methodology and shows how LSGA can be used
to intelligently trade between engineering and science requirements. Finally, Chapter 5 summarizes the work, highlights the main thesis contributions, and proposes
directions for future research.
34
Chapter 2
Literature Review
Having defined the research objectives, we now discuss the context of the proposed
work within the body of literature. Prior work is grouped into three major areas
that are considered below in turn. First, we outline “standard” requirement definition processes as practiced by large organizations. Second, we summarize several
model-based design methodologies and highlight their relevance to the problem of requirement definition. Third, we review several instrument projects and consider the
requirement definition processes used in those specific cases. The section concludes
with the identification of the literature gap filled by this work.
2.1
Requirement Definition Processes
This section summarizes the general requirements definition processes that are contained within the high-level systems engineering standards used by large organizations. The requirements process used by the National Aeronautics and Space Administration (NASA) is described in Section 4.2 of the NASA Systems Engineering Handbook (NASA/SP-2007-6105 Rev 1) [1]. NASA identifies two inputs to this process:
1) top-level stakeholder requirements and expectations and 2) the system concept of
operations (CONOPS). The outputs are requirements of various types (functional,
performance, safety, etc.) and an accompanying requirements verification matrix.
The handbook discusses in general terms how requirements are decomposed and
35
allocated to lower levels of design, with emphasis on bi-directional traceability. Functional analysis—i.e. the “systematic process of identifying, describing, and relating
the functions a system must perform to fulfill its goals and objectives”—is used to
decompose requirements. The main steps in functional analysis are to,
• Translate top-level requirements into functions that must be performed to accomplish the requirements.
• Decompose and allocate the functions to lower levels of the product breakdown
structure.
• Identify and describe functional and subsystem interfaces.
Each system requirement is analyzed to identify the functions that must be performed
to meet the requirement and this is repeated from the top down. The process can be
aided by visual tools like functional flow block diagrams (FFBDs) and N2 diagrams.
The International Council on Systems Engineering (INCOSE) provides a handbook (INCOSE-TP-2003-002-03) [106] that, like the NASA one, describes system
engineering activities that occur throughout the life cycle. Technical requirements
generation is detailed in Section 4.3 and begins with input stakeholder requirements,
solution constraints, and operational use scenarios. The outputs are the requirements
themselves—here defined as the “technical description of the characteristics the future
system must have in order to meet stakeholder requirements”. The main activities of
the INCOSE requirements process include,
• Determining the levels and measures of performance (MOP) for the top-level
system functional requirements required to satisfy the system measures of effectiveness (MOE).
• Defining the system boundary by clearly identifying system elements under
design control of the project team.
• Identifying all environmental factors (natural or induced) that may affect system
performance.
36
• Identifying design constraints including physical limitations and defined interfaces with interacting systems or host platforms.
There are two standards that describe the requirements definition processes used
by the European Space Agency (ESA). Both are published by the European Cooperation for Space Standardization (ECSS). The first is the System Engineering General
Requirements standard (ECSS-E-ST-10C) [44], which serves a similar purpose to
the NASA and INCOSE handbooks in laying out a high-level system engineering
process that can be adapted to any project. Requirement development (here called
“requirement engineering”) and the related topic of functional analysis (called simply “analysis”) are outlined in sections 5.2 and 5.3, respectively. The procedures are
similar to those outlined in the previous standards. There is a subsection devoted to
requirement allocation, however the prescription is overly general:
“The system engineering organization shall ensure that the system requirements (and their verification methods) are allocated to lower levels
and included in the specifications of the related products.”
The second standard is the Technical Requirements Specification (ECSS-E-ST-10-06C) [45],
which describes various types of requirements and the manner in which they ought
to be written.
2.2
Model-Based Design Methodologies
The past two decades have given rise to powerful methodologies that use modeling
and simulation to aid in the design of complex, multidisciplinary engineering systems.
Several of the more relevant approaches are discussed in this section with an eye
toward identifying where MBRD fits into this landscape.
Model-Based Systems Engineering (MBSE) is a term that encompasses a variety
of tools and practices that use models as the basis for system engineering activities.
Estefan [41] presents an excellent survey of model-based methodologies and explains
MBSE as follows:
37
In a nutshell, model-based engineering (MBE) is about elevating models
in the engineering process to a central and governing role in the specification, design, integration, validation, and operation of a system. For
many organizations, this is a paradigm shift from traditional documentbased and acquisition lifecycle model approaches, many of which follow
a “pure” waterfall model of system definition, system design, and design
qualification.
It is worth mentioning that MBSE is often discussed along with Systems Modeling
Language (SysML), an industry standard visual modeling language used to represent complex systems through all phases of the design process including specification,
analysis, design, verification, and validation [57]. SysML is in turn based on the Unified Modeling Language (UML), which was originally developed to support software
system engineering.
JPL has developed an MBSE methodology called State Analysis (SA) [66, 41].
It is based on a model- and state-based control architecture that uses models to
represent requirements directly. SA is most relevant to software engineering and
is motivated by a need to improve communication between system engineers (who
specify requirements) and software engineers (who implement the code).
The MBSE label can be applied to much of the research taking place in the MIT
Space Systems Laboratory (MIT-SSL). One example is the work of Robinson [111] on
the use of an integrated evolutionary modeling (IEM) framework to aid in the design
of small satellites. This approach uses an executable model of the satellite that
matures along with the spacecraft hardware. The model is then used throughout the
satellite life cycle to support verification and validation.
The DoD has a MBSE-like process known as Simulation Based Acquisition (SBA).
Walton [144] discusses SBA as part of his survey on the use of modeling and simulation
to support early phase systems engineering, including requirements definition. Like
MBSE, SBA focuses on using modeling and simulation through the product life cycle.
In particular the DoD is interested in model re-use and leveraging M&S to replace
expensive testing procedures.
38
Integrated modeling has been used frequently for the design of astronomical telescopes, both in space and on the ground. Designing a high performance telescope
requires careful consideration of the optical, structural, thermal, and control aspects
of the design, therefore it is not surprising that multidisciplinary analysis would find
a role. Miller et al. [90] developed a comprehensive framework for the multidisciplinary modeling and analysis of space telescopes called DOCS (Dynamics-OpticsControls-Structures). The methodology consists of subsystem modeling, model assembly, model reduction and conditioning, initial performance assessment, sensitivity analysis, uncertainty analysis, redesign, design optimization, and isoperformance
analysis.
Schnetler and Taylor [119] apply the same general approach to the European Extremely Large Telescope (E-ELT) EAGLE instrument, a multi-object infrared spectrometer. The authors discuss the details of their model and a more general approach
(adapted from Miller et al.) to using integrated models throughout the development
cycle.
The Space Systems, Policy and Architecture Research Consortium (SSPARC) was
a research effort in the early 2000s whose goals were to create new methods for space
system architecting and design, including the incorporation of risk, uncertainty, and
flexibility [59]. The result was a set of new design methodologies that are relevant to
this thesis; they include GINA, MMDOSA, and MATE-CON, which are discussed in
more detail below.
The Generalized Information Network Analysis (GINA) methodology for satellite
systems was developed by Shaw [127, 128] and is based on the premise that practically any satellite system fulfills a mission relating to communication, sensing, or
navigation. The system can therefore be treated as a generalized information network
and certain unambiguous metrics for cost, capability, performance, and adaptability
can be defined. GINA serves more as a method of organizing models and classifying various inputs and outputs, rather than a stand-alone design method per se. It
can be used to develop models that are then exercised using other methods such as
MATE-CON and MMDOSA.
39
Multiple Attribute Tradespace Exploration with Concurrent Design (MATE-CON)
is a design process described by Diller [38] and Ross [112, 113]. It begins with multiattribute tradespace exploration (MATE) for the solicitation and capturing of stakeholder preferences. With these preferences, architectures can be enumerated and the
ones that maximize utility are advanced to the concurrent design (CON) process.
During concurrent design, an integrated concurrent engineering (ICE) environment is
employed to rapidly iterate and converge on a solution that ensures that user needs
are satisfied.
Another design process to emerge from SSPARC is the Multi-Objective Multidisciplinary Design Optimization Systems Architecting (MMDOSA) methodology. Developed by Jilla [68, 69], MMDOSA applies the formalism of multidisciplinary design
optimization (MDO) to the problem of distributed satellite systems (DSS). Steps
include creating a model, performing single variable studies to understand its behavior, formulating and executing a design optimization (single- or multi-objective),
interpreting the results via sensitivity analysis, and iterating as necessary.
MMDOSA is a straightforward application of multidisciplinary design optimization (MDO), an expansive topic that is described in two papers that bookend its modern development. The first is a 1991 white paper by the AIAA Technical Committee
on MDO [100] that identifies the fundamental need for MDO, its role in aerospace
systems, and optimization methods that are at its core. The second paper is a 2010
assessment by Agte et al. [2] that traces MDO over the intervening two decades. The
original nonlinear programming (NLP) formalism of MDO is (re)introduced and the
authors then review numerous state-of-the-art algorithms.
Of particular importance to this thesis is the MDO method known as isoperformance. Formalized by de Weck, Jones, and Miller [23, 25, 24] after groundwork laid
by Gutierrez [58], isoperformance is a method that seeks to satisfy performance instead of blindly maximizing it. The goal is to identify a family of configurations in
the design space that all map to equal, satisfactory performance targets in the objective space. Once this isoperforming set is located in the design space, secondary
considerations such as cost or risk are used to choose among the alternatives. The
40
isoperformance methodology was revisited by Evans, who investigated contours of
other quantities besides performance, adopting the terminology “isoparametric” to
describe the broader application of de Weck’s original algorithms [42].
2.3
Project-Specific Approaches to Requirements
This section describes the way in which several remote sensing projects—proposed,
in design, and fielded—approached requirements definition and science/engineering
trade-offs. The systems highlighted here for discussion were chosen specifically because they leveraged model-based methods and therefore provide context for the
proposed research.
The Space Interferometry Mission (SIM) is space-based optical wavelength Michelson stellar interferometer that was under development from 1996 to 2005 [22]. SIM
was to conduct ultra-high precision astrometry measurements to achieve a variety of
scientific goals, including the characterization of known exoplanet systems and the
discovery of Earth-mass planets in the habitable zones of nearby stars [143]. SIM is
interesting from the perspective of this thesis because 1) the astrometric measurements require significant amounts of post-processing and 2) the project allocated
requirements using an error budget informed by modeling and simulation.
On the first point, Unwin [142] provides an overview of the measurement process and overall observing campaign. Astrometric measurements are differential in
nature—i.e. they are made between objects—and the SIM science is contained in the
time-evolution of the astrometric signal. The astrometric signals are two-dimensional
on the sky so each science measurement requires a later repeated measurement with
the baseline approximately orthogonal to the first. Scheduling the observations must
take into account this fact, and the desire to minimize systematic errors.
The SIM astrometric data set is built up over time using a sequence of measurements carried out using an “orange peel” pattern over the sky. Once the orange peel
is complete, post-processing solves for five astrometric parameters for each star and
a number of instrument parameters. The later step effectively calibrates the instru41
ment, removing systematic noise sources (biases) such as a field-dependent error term
over the science field of regard. This external calibration procedure is described in
greater detail by Shaklan et al. [126] and Unwin [141].
All of the error sources, including the field-dependent term, are captured by the
SIM astrometric error budget (AEB). The SIM mission is characterized by extremely
tight optical requirements (maximum path-length errors are on the order of picometers), therefore careful accounting of the various noise contributions was critical. Described by Nemati and Morales [98], the AEB specifies the allowable error after postprocessing, including the calibrations mentioned above. The AEB uses relatively
simple operators to roll up the error terms (e.g. product, ratio, RSS). However, the
terms themselves are produced using a combination of numerical modeling and hardware testing. Therefore the SIM approach uses detailed modeling and simulation to
populate the AEB and to verify that the roll-up accurately captures the performance.
As models are updated and additional test results arrive, the AEB is update to ensure
that top-level requirements are still met. Errors themselves are allocated according
to their effects as predicted using modeling and testing (i.e. errors with a large contribution are given the highest AEB allocations) and by engineering judgement based
on previous experience.
The underlying data that informed the AEB terms comes from an end-to-end
simulator for SIM cleverly termed SIMsim. Meier and Folkner [88] describe the initial
development and overall structure of the model. SIMsim consists of five modules
that replicate the entire end-to-end measurement and post-processing pipeline for
SIM. It incorporates spacecraft motions, photon counting and fringe detection, laser
metrology errors and biases, calibration errors, and feed-forward dynamics, allowing
the analyst to evaluate the effects of these errors on astrometric accuracy. An entire
5-year observing campaign can be simulated with a time step of 1 millisecond, which
generates 200 TB of data and requires 2 weeks of computation.1 The general approach
adopted by the authors was to use detailed simulation to fit parameters in a simplified
1
The authors used a 14-processor 400 MHz Sun Enterprise 6000. A simplified version of the
simulation (SimSIMsim) was created using analytical approximations to some of the more time
consuming numerical steps.
42
analytical expression—and to determine which expression is right in the first place—
which then became the basis for the AEB.
Murphy and Meier [97] describe updates to SIMsim and its use to validate additional terms in the AEB, including systematic errors. SIMsim is mostly used in a
“point design” sense—i.e. simulation runs use the same fixed set of input parameters representing the baseline system design. Parametric trade studies are generally
avoided. The exception is a set of simulations in which global astrometric error is
calculated as a function of the number of star observations. The authors conducted
Monte Carlo simulations to reveal the minimum number of measurements needed to
eliminate systematic error. The interested reader is referred to Moshir et al. [95] for
a more recent update on the SIMsim and its relationship to AEB. The content of this
work is similar to the others, with updates added for thermal analysis and algorithms
to help remove systematic errors.
The European Extremely Large Telescope (E-ELT) is a next generation groundbased telescope currently nearing the end of its design phase and scheduled for first
light in 2017 [51, 99]. The telescope has a 42 m diameter primary mirror comprised
of 984 hexagonal segments each with a diameter of 1.45 m (point-to-point). It will
be built at Cerro Armazones, a mountain in the central region of the Atacama desert
in Chile.
The model-based approaches used by the E-ELT team—both at the telescope and
instrument level—share some common elements with the methodology to be developed in this thesis. At the telescope level, the E-ELT team undertook an extensive
model-driven study called the Design Reference Mission (DRM). This effort, described
in detail by Liske et al. (2011) [86, 85], used simulated E-ELT science data across a
variety of use scenarios to understand the system’s capabilities and their sensitivities to engineering parameters. Specifically, the overarching objective as stated by
DRM report [86] was to ensure that the E-ELT will meet the goals of the scientific
community. This was broken down into three sub-goals as follows:
1. To provide a quantitative assessment of the extent to which the E-ELT will be
capable of addressing key scientific questions.
43
2. To assist the project in making critical trade-off decisions by quantifying their
consequences in terms of scientific gains and losses.
3. To support the development of the E-ELT science case.
The E-ELT DRM is relevant to this thesis because the proposed MBRD methodology enables similar analysis (i.e. quantification of trade-offs between technical design
and scientific gain) and because the DRM specifically addresses questions that cannot
be answered by a straightforward signal-to-noise calculation. Like MBRD, the DRM
uses realistic simulated data to explore the connection between science and engineering. For each case study within the DRM, the authors used the following well-defined
series of steps to build the model and analyze the outputs:
1. Specify the DRM simulation goal for the given case study.
2. Define scientific figures of merit and requirements.
3. Implement the forward simulation, including data processing pipeline.
4. Define the scientific and engineering parameters (set design baseline values and
ranges for trade space exploration).
5. Perform simulations, varying scientific and engineering parameters according to
the schedule defined above.
6. Confirm that the baseline design meets science requirements.
7. Analyze the extent to which varying parameters affects whether or not science
requirement can be met.
The results of the various DRM case study simulations were collected and the
driving requirements were identified across the ensemble. E-ELT is intended to be a
general purpose observatory, so while a single science case would often only constrain
a few aspects of the telescope design, the case studies as a whole produced a large set
of demanding technical requirements.
44
The DRM incorporates many useful features of a model-based requirements methodology: the use of end-to-end models to simulate science data products, a stepby-step prescription for carrying out the analysis, and quantification of the science/engineering trade-off. However, the approach could be improved in two ways.
First, the DRM represents a “forward” sensitivity analysis in which variation of input parameters are used to investigate the resulting variation on the science figures of
merit. There is no capability to perform an inverse sensitivity analysis, i.e. prescribing
allowable errors on the science measurements and deriving the permissable range of input parameters. Second, the DRM is primarily intended for requirements verification
rather than requirements definition and allocation. As the DRM report [86] states,
“the overall conclusion is that the E-ELT system has neither been over-specified nor
under-specified: its design is well matched to the requirements of the science cases
investigated by the DRM.” Thus the DRM primarily is used in an a posteriori sense
to confirm after the fact that the telescope design will meet requirements, not to determine the technical requirements in the first place. The requirements instead were
determined by the E-ELT science working group (SWG) members based on experience and expert knowledge [39]. The SWG considered a range of science cases and
the necessary telescope characteristics for meeting scientifically useful goals. The requirements and assumptions that emerged from this process were then verified using
the DRM.
For the sake of completeness, it is worth mentioning that in addition to the end-toend models in the DRM, integrated models were used at the system level to quantify
the interaction between structural dynamics and optical performance. Müeller and
Koch (2006) [96] and Sedghi et al. (2011) [123, 122] describe a set of multidisciplinary
toolkits that simulate the behavior of the E-ELT structural dynamics, optical train,
and adaptive optics system. These models were evaluated in a parametric manner to
determine the affect of various design and operational choices (e.g. spider thickness,
secondary mirror hexapod stiffness, wind velocity, residual segmentation errors, etc.)
on performance (e.g. wavefront error).
45
2.4
Research Gap
Broadly speaking, there are no quantitative and widely applicable model-driven approaches to instrument requirement definition. This thesis develops a methodology
to fill this gap. The literature on model-based design, multidisciplinary optimization, and integrated modeling provides inspiration in this regard. Fundamentally, the
aims here are similar to those of de Weck’s isoperformance methodology and more
general concepts of “satisficing” or nominal-is-better design. While de Weck’s algorithms were tailored to linear time invariant systems and precision opto-mechanical
structures, the algorithms presented in this thesis are derivative-free and applicable
to a broader range of system models. Likewise, this thesis shares certain goals in
common with the MATE-CON methodology developed by Diller, Ross, and Hastings, namely a desire to increase communication and understanding between system
users (scientists) and system designers (engineers). While MATE-CON attempts to
bridge this divide using multi-attribute utility theory to normalize user preferences,
the methodology presented in this thesis captures preferences directly in terms of
science requirement statements. In other words, instead of normalizing objectives according to non-dimensional units of utility as in MATE-CON, the approach developed
in this work allows scientists to retain their “native” requirement language and uses
end-to-end modeling along with optimization in order to translate the requirements
into the engineering domain. The intended result is that the two communities—
scientists and engineers—can retain the language to which they are accustomed and
use optimization as a translation engine to assist in requirement allocation and trades.
46
Chapter 3
Model-Based Requirement
Definition Methodology
This chapter describes a model-based requirement definition methodology applicable to a broad range of instrument systems. The purpose of the methodology is to
systematically translate science objectives into a set of top-level engineering performance requirements that will then guide system implementation. Figure 3-1 depicts
this process at the highest level.
Figure 3-1: High-level construction of the model-based requirement definition
methodology. Inputs are science objectives and outputs are top-level engineering
requirements. Sample input and output statements are given for an asteroid remote
sensing instrument.
The input to this methodology takes the form of human-language statements
regarding what the instrument system seeks to accomplish from a science perspective.
For example, the objective may be to measure and classify the global composition
of an asteroid. After the model-based requirement definition process has run its
47
course, the output is a set of requirement statements that describe what the system
must achieve from an engineering perspective in order to meet the science objectives.
For example, the asteroid composition measurement may result in requirements on
detector temperature, data quantization, and level of stray light. The goal is to
generate the minimum number of top-level engineering requirements that define a
system capable of meeting the stated science objectives.
3.1
Overview
What follows in the rest of this chapter is a detailed description of the model-based
requirement definition methodology. We start by considering the properties of an
ideal requirements methodology and then present an overview of the activities that
constitute our four-step approach. We also introduce a case study on the design of a
star tracker. The subsequent sections go on to describe each step of the methodology
in turn, both from a conceptual perspective and using the star tracker problem as a
concrete example.
3.1.1
Desired Methodology Properties
Before outlining the steps in model-based requirement definition, we first consider
the desired properties of an ideal requirement definition methodology. These general
properties will serve as guideposts for making decisions as we develop our specific
approach. They also serve as a benchmark against which we can look back and
evaluate the methodology once it has been explained and tested.
First, we desire that any requirement definition methodology be inherently quantitative (rather than qualitative), both in the manner that requirements are derived and
in the form of the final requirement statements. According to best practices outlined
in the NASA Systems Engineering Handbook [1], requirements must be objective and
unambiguous, leaving no room for misinterpretation. Using a quantitative approach
will aid substantially in this regard. Furthermore, using the common vocabulary of
numbers and mathematics will help bridge the language gap between scientists and
48
engineers.
Second, we want our requirement definition methodology to be prescriptive in
that it should describe how to generate requirements. It may seem obvious, but we
emphasize that the methodology should provide a specific recipe that be handed to
other practitioners to define requirements for their systems. As discussed in Chapter 2, current methodologies are overly broad in this respect. The challenge, of course,
is to derive a recipe that is sufficiently flexible that it applies to a range of systems
yet not so broad that it becomes useless.
Third, the methodology should produce requirements that are traceable, in that
the driving rationale behind them should be easy to discern. Once requirements are
defined and the instrument has moved to later stages of development, there is often
push-back on the science requirements from the engineers as they refine the design
and encounter technical, cost, or schedule obstacles. Traceable requirements allow
engineers to literally trace the impact of their decisions back to science performance
(and vice versa). This is essential for facilitating science/engineering tradeoffs in an
informed manner.
Fourth, and related to the previous point, we desire that our requirements methodology span multiple domains. Specifically, it should help map requirements from
system users (e.g. scientists) to system designers/implementers (e.g. engineers). As
described in Chapter 2, this interface is particularly fraught with misunderstandings
and divergent perspectives. Therefore an ability to improve requirement definition
across this divide would reap outsized benefits compared to improving requirements
flow-down within a domain.
Lastly, the desired methodology should be model independent. By this we mean
that to the extent the methodology relies on a model, it should do so with the fewest
possible restrictions on the form the model takes. Models can be stochastic or deterministic, continuous or discrete, linear or non-linear, time domain or frequency
domain, etc. In short, instrument models are as varied as the physical phenomena
they are designed to simulate. By selecting algorithms that are tolerant to the widest
possible range of models, we ensure that the requirements methodology is generaliz49
able to various instrument systems.
We have outlined five properties of an ideal requirement definition methodology.
It is possible that more exist, however these are a useful starting point for guiding
and then evaluating the approach described in this work. The five properties are
summarized below.
• Quantitative: Requirements should be defined in terms of numerical values,
so as to be unambiguous and objective.
• Prescriptive: The methodology should describe precisely how to generate requirements, providing a step-by-step recipe that can be followed by other system
engineers.
• Traceable: Requirements should be plainly traceable to an underlying rationale or motivation. System designers (e.g. engineers) should be able to easily
trace the impact of engineering requirements push-back on system performance
(e.g. science objectives) and vice versa.
• Multi-domain: The methodology should allow for the mapping of requirements between domains (e.g. between scientists and engineers), as this interface
is often a source of misunderstanding.
• Model independent: To the extent that the methodology uses models, it
should do so in a way that places the fewest possible restrictions on the model
structure. If the methodology is to be broadly applicable then it must accommodate the range of modeling and simulation techniques that are in practice.
3.1.2
Steps for Model-Based Requirement Definition
Having outlined the methodology’s board objectives above, this section introduces
the four steps that constitute its implementation. They are shown in Figure 3-2,
which is an expanded version of Figure 3-1. Each step is summarized below and then
discussed in full detail in later sections.
The first step in model-based requirement definition is to quantify user objectives
by transforming qualitative performance aims into numerical thresholds that can serve
50
Figure 3-2: Steps in the model-based requirement definition methodology.
as the basis for computer-based optimization. Specifically, we require that the user
describe the data product, a figure of merit that indicates data product quality, and
a requirement on the figure of merit.
The second step is to create an end-to-end model that simulates operation of the
instrument. The model generates synthetic data and calculates the figure of merit
based on a set of design parameters. By “end-to-end” we mean that the model must
capture the entire measurement flow, from input at the instrument aperture to final
data product, accounting for relevant data processing that may occur along the way.
This model is a functional mapping between the science objective and engineering
requirements.
The third step is to use the end-to-end model to generate multiple sets of engineering requirements. Each requirements set specifies a system that will meet science
objectives. This is accomplished using a new heuristic optimization algorithm to find
the level set (i.e. the set of designs that meet the science requirement) over the space
of all possible designs.
51
Once multiple requirements sets have been generated, the fourth and final step is
to screen them to select a final requirement set. This selection can be accomplished
using a variety of secondary characteristics such as cost or risk. One such option is
to select the most “robust” requirements by choosing the set that shows the least
amount of sensitivity in the science performance metric. We describe a Monte Carlo
approach for doing so by calculating the relative contribution of each input parameter
to the overall variation in the science objective. The idea behind choosing the most
robust requirement set is to minimize the impact of potential science performance
creep later in the project life cycle.
The result of this process is a set of top-level engineering requirements to which
the system can be designed. The list below summarizes the steps in the methodology.
1. Quantify User Objectives: The system user, often a scientist or team of
scientists, must quantitatively define the data product (i.e. what the system
is measuring), the figure of merit (i.e. an indication of how well the system is
measuring it), and the performance requirement (i.e. the threshold value for the
figure of merit for the instrument to be useful).
2. End-to-End Model Creation: The science and engineering teams together
create an executable model that generates synthetic data products and calculates the figure of merit as a function of design parameters. By capturing the
mapping from design to performance, the model serves as a mapping between
designers and users.
3. Generate Multiple Requirement Sets: The end-to-end model is exercised
using an optimization algorithm to efficiently explore the “requirement space”
and generate a set of design-to parameters that satisfy the science performance
requirement.
4. Select a Final Requirement Set: The requirement sets identified in the
previous step are screened according to user-defined metrics such as cost or risk.
One approach is to select the requirement set with the least sensitivity, under
the assumption that robust requirements will mitigate the effect of changes later
52
in the program life cycle.
3.1.3
Assumptions
Before proceeding further, it is important to point out some assumptions that we
take for granted in developing the methodology. First, we assume that the system
architecture is fixed prior to beginning model-based requirement definition. Recall
from Figure 1-4 that the methodology is intended to apply during the requirements
allocation phase of the system development life cycle, after concept selection and prior
to detailed design. Therefore the general form of the instrument must be established,
for example through trade studies, and the concept of operations must be defined.
The methodology requires this minimal level of system maturity so that a useful
end-to-end model can be created in Step 2.
Second, this methodology applies primarily to the definition and allocation of
performance requirements. These usually drive the design and, due in part to the science/engineering divide for instrument systems, often are among the most challenging
to flow down. However, there may be other important requirements on safety, interfaces, human factors engineering, reliability, etc, that must be considered as part of
the overall requirements structure. Expansion of the methodology to other types of
requirements will be discussed as future work.
3.1.4
Star Camera Sample Problem
As a last step before describing the model-based requirement definition methodology
in full detail, we present a relatively simple problem that will be used throughout
this chapter as a means of animating the concepts described. Consider the design of
a star camera, a small space-borne telescope used to determine the orientation of a
spacecraft based on the observed pattern of detected stars.
Shown notionally in Figure 3-3, the star camera consists of a lens assembly and
a back-end electronics unit. The lens assembly—made up of several individual lens
elements, an aperture stop, and associated mounting hardware—projects an image
53
Figure 3-3: Conceptual depiction of a star camera. The instrument takes images of
the stars in its field of view and calculates the locations of centroids.
of the star field onto a focal plane array. The focal plane array, typically a charge
coupled device (CCD) or complimentary metal-oxide semiconductor (CMOS) sensor,
records a digital frame and sends it to the onboard electronics for processing. Individuals star images are extracted and centroids are calculated to determine their precise
location on the focal plane. A list of centroids is output to the spacecraft where
a transformation using knowledge of the lens optical properties converts this information into angular coordinates on the sky. By comparing the measured centroids
against a database of stored centroids, “lost-in-space” procedures such as Mortari’s
pyramid algorithm can determine the orientation of the boresight vector to a high
accuracy [93, 94].1
In our example we assume that the choice of back-end electronics, including the
focal plane, is fixed due to a desire to reuse an existing design that had flown on
a previous mission. The sensor from the heritage mission is a HAS2 CMOS imager
produced by ON Semiconductor (formerly Cypress Semiconductor) shown in Figure 31
Often the lost-in-space calculation is performed autonomously by the back-end electronics unit,
which then outputs an attitude quaternion to the spacecraft. In this case, the instrument is often
referred to as a star tracker instead of a star camera (“tracker” implying an autonomous attitude
estimation capability). Our example considers the simpler case of a star camera.
54
4. The device format is 1024 × 1024 with 18 µm pixels. The pixel well depth is
105 electrons and images are read out through an onboard 12-bit analog to digital
converter (ADC). We assume that the star camera operates at a frame rate of 5 Hz
for an integration time of approximately 200 milliseconds.
(a)
(b)
Figure 3-4: HAS2 CMOS imager showing (a) the device package and (b) the location
of the 1024 × 1024 active pixel area (white region), from [18].
With the back-end electronics fixed, the goal of the system engineer in our example is to establish requirements on the design of the lens assembly. Imaging lenses
are typically characterized in terms of two parameters: their focal length and aperture diameter. The aperture diameter quantifies the size of the entrance pupil that
accepts light into the imaging system. For a multi-element lens like the one depicted
in Figure 3-3 the entrance pupil is the image of the aperture stop through the preceding elements and does not necessarily have the same diameter as the first lens
element. There are other parameters that may help describe a lens, but these are the
predominant ones (e.g. photographic camera lenses are catalogued by focal length
and aperture size, usually represented by the dimensionless f-number) and will suffice for our simple example. With the requirements on the focal length and aperture
diameter defined, along with a few other specifications on image quality, an optical
engineer can proceed to design the curvatures, spacings, and glasses of the individual
lens elements using domain-specific optimization tools. Other assumptions for our
sample system will be described in later sections.
55
3.2
Step 1: Quantify User Objectives
We now have all the pieces in place to begin discussing model-based requirement
definition in detail. As the name implies, numerical methods lie at the core of the
methodology. Indeed, one broad aim of this research is to introduce a higher degree
of numerical formalism and procedural structure to what is often a relatively ad hoc
requirements flow-down process. Therefore the first step in model-based requirement
definition is to convert qualitative science objective into quantitative statements. The
latter can then be subjected to computer-based analysis in a manner prescribed by
later sections.
To complete this first step, the user—often a scientist or team of scientists—must
specify three quantities that together define the performance objectives of the system:
1) the data product (i.e. what the system is measuring), 2) the figure of merit (i.e.
an indication of how good the measurement is), and 3) the performance requirement
(i.e. the threshold level of acceptability for the figure of merit). These quantities then
serve the basis for numerical modeling and analysis.
3.2.1
Data Product
The data product is the final output that an instrument system generates. Simply
put, it is what the system measures—i.e. the thing that a scientist or other user
will look at to answer the fundamental questions that motivated the instrument’s
development in the first place. Examples from later chapters include a photometric
light curve showing stellar intensity over time, and abundance ratios for elements
present in asteroid regolith. The data product is chosen by the instrument scientist
because it maps directly to the higher-level objectives of the study. Photometric
light curves are interesting because they can reveal the presence of exoplanets, and
elemental abundance ratios are interesting because they give insight into the formation
of our solar system.
Importantly, when the data product is defined it must include the effects of software processing. Raw instrument measurements are often calibrated, scaled, filtered,
56
or otherwise corrected on their way to becoming final data products. These steps
can greatly influence the data quality, therefore in the context of this thesis, “data
product” refers to the final post-processed image, spectrum, map, histogram, etc of
interest to the system user.
For our star camera, the data product is simply the list of centroids produced
by the onboard software. Each measured centroid is given as a pair (x̂, ŷ) in focal
plane pixel coordinates. The user of our system in this case would be an attitude
control subsystem (ACS) engineer and these centroids will be used as the input to
their attitude estimation routine.
3.2.2
Figure of Merit
Once the data product has been clearly articulated, we must establish a metric for
its quality. This is the figure of merit (FOM): a number that describes how good a
given measurement is.2 For the star camera, consider that the lost-in-space algorithm
requires a certain minimum number of high-quality centroids to successfully determine
the star pattern. Therefore the FOM can be stated as the number of centroids at or
below some pre-defined accuracy threshold. For concreteness, let the threshold be 0.1
pixel. Our FOM can thus be stated as the number of centroids with an error of 0.1
pixel or less, denoted as N0.1 .
3.2.3
Performance Requirement
With the figure of merit defined, the user can now specify the precise FOM value
they require in order for the measurement to be considered satisfactory. This is our
definition of the performance requirement. It quantifies the level of performance with
respect to the FOM at which the measurement is acceptable to the user. Imagine that
the lost-in-space algorithm must have at least twenty centroids from the star camera
in order to correctly identify star patterns and compute the attitude. This therefore
2
For the star camera problem and in the more detailed examples that follow, we only consider a
single figure of merit. Multiple figures of merit will be discussed later in the context of future work.
57
sets the star camera performance requirement in terms of the previously-defined figure
of merit: N0.1 ≥ 20.
3.2.4
Summary
Quantifying user objectives requires articulating three definitions. First, the data
product describes what specifically the system is measuring, including the effect of
data processing. Second, the figure of merit provides a metric for gauging how good
the data product is. Third, the performance requirement specifies a minimum acceptable value for the figure of merit. These are summarized for the star camera in
Table 3.1.
Table 3.1: Quantified user objectives for the star camera sample problem.
Quantity
Data Product
Definition
Expression
A list of centroid coordinates (x̂, ŷ)
in units of focal plane pixels
Figure of Merit (FOM)
The number of centroids with N0.1
an error of 0.1 pixels or less
Performance Requirement There must be at least 20
N0.1 ≥ 20
such centroids
Looking ahead, these three quantities will allow us frame the process of allocating
engineering requirements as a numerical optimization problem. The FOM will serve
as the objective function and the performance requirement its target value. The
challenge is to find the set of engineering requirements (cast as design variables in the
optimization) that satisfies the performance requirement. To do this, we must create
an executable model that calculates the FOM as a function of engineering parameters.
This is the task of the methodology’s second step, described below.
3.3
Step 2: End-to-End Model Creation
With the user objectives now cast in terms of numerical values, the next step is to
create an executable model that captures the mapping between engineering decisions
58
(i.e. the design domain) and instrument performance (i.e. user domain). More specifically, the model will produce FOM predictions as a function of design parameters, as
shown in Figure 3-5. The model therefore serves as a forward mapping between the
engineering decisions (input) to the science performance (output). Once this forward
mapping is defined, it will be used in the next step to allocate science performance objectives into top-level engineering requirements using optimization to solve the inverse
problem.
Figure 3-5: End-to-end model as a mapping between the engineering and science
domains.
Note use of the term “end-to-end”. By this we mean that the model must encompass the entire chain of system operation, from the input of source energy at
the instrument aperture, to detection at the sensor, to digitization and processing in
the instrument electronics, to software manipulation on the ground. In other words,
the end-to-end model must be consistent with the definition of the data product in
Step 1.
3.3.1
Requirement Variables and Vectors
Care must be taken when creating the end-to-end model to ensure that it will fit
nicely into the optimization framework used in next step of the methodology. To
this end, we introduce the concept of a requirement variable. Consider the common
scenario in which the behavior of a system is described by a model f :
y = f (x) = f ([x1 x2 · · · xn ]).
59
(3.1)
Here the xi are design variables and y is the figure of merit. Design variables are parameters in the system—controllable through engineering decisions—that influence
the performance. Notice that the xi have been concatenated into a design vector
x that conveniently describes the system as a location in the N-dimensional trade
space (i.e. the span of all possible designs). Importantly, f can be wrapped in an
optimization routine that attempts to find the optimal design vector x∗ that maximizes performance. This and related activities constitute multidisciplinary design
optimization (MDO) or more broadly, model-based design (MBD).
Analogously, consider that at a higher level of abstraction, system performance
is governed not by design variables per se but rather by the set of requirements that
are imposed. In other words, system behavior could be described by a model g:
y = g(r) = g([r1 r2 · · · rn ]).
(3.2)
Here y is the same performance figure of merit while the ri are what we term requirement variables. Requirement variables are system parameters on which requirements
must be set in order to determine performance. Like design variables, we can concatenate requirement variables into a requirement vector r. And just as we can wrap
f in an optimization to find x∗ , we can also wrap g in an optimization to find r∗ .
Here r∗ is defined as the set of top-level engineering requirements that together ensure
that the instrument will meet a desired level of performance. Obtaining this vector
is the primary task of model-based requirement definition. The mathematical details
of doing so are the subject of steps 3 and 4 (Sections 3.4 and 3.5, respectively).
The difference between design variables and requirement variables is subtle and
essentially a matter of perspective. Consider the functional decomposition depicted
in Figure 3-6, adapted from Figure 1-2. The parameters defined at the top-most level
of the engineering domain are the requirement variables. Parameters defined at lower
levels are the design variables. Once the value of a requirement variable is fixed (i.e.
a requirement is written), subordinate design variables can then be traded and fixed.
In this sense requirement variables are just a special type of design variable: those
60
that exist at the first level of functional decomposition. For example, a common
requirement variable for instrument systems is focal plane temperature. One must
normally specify a temperature requirement in order to satisfy the science objectives,
after which the thermal engineer can determine which combinations of lower-level
design variables (e.g. coating emissivity, heater power, material choice, insulation
thickness, etc) meets the requirement. Requirement variables are important (and
challenging) because they encode the mapping between the science objectives and
the rest of the engineering design variables.
Figure 3-6: Requirement variables and design variables in the context of functional
decomposition and requirement flow-down.
3.3.2
Identifying Requirement Variables
A first-order question to ask when creating a model for use in design optimization is
“what are my design variables?” Equivalently, when creating a model for requirement
optimization, an initial question is “what are my requirement variables?” Once the
requirement variables are identified and the requirement vector assembled, the model
61
g in Equation 3.2 can be implemented and model-based requirement definition can
proceed.
As with formulating any modeling problem, answering the above questions involves some degree of art, experience, and engineering judgement. However, we
present a procedure that helps with this task: documenting the measurement pipeline.
By writing down all of the steps involved in creating the final data product, using
either prose or (ideally) a flow chart, it becomes much easier to identify requirement
variables.
This process will be somewhat application dependent, but the general idea is to
trace the measurement signal through the system and show how it is influenced by
physical phenomena and software manipulation along the way. Various adjustable
engineering parameters are identified in this process and become the requirement
variables. An overview of the star camera measurement pipeline is shown in Figure 3-7. The diagram shows at the highest level how incoming source energy (stellar
photons) are converted into the data product (star centroids). The pipeline is broken
down into two components: imaging system hardware that converts photons into a
raw image and onboard processing software that converts the image into centroids.
Figure 3-8 shows the imaging system block in greater detail. We decompose the
hardware into two main elements: the lens and the HAS2 CMOS sensor. Figure 38 depicts design parameters to the left in color, with dashed lines indicating the
element with which they are associated. For example, the lens is characterized by
its focal length and aperture diameter as described in Section 3.1.4, along with the
passband (i.e. the range of wavelengths accepted into the lens), transmission (i.e.
fraction of incident flux that passes through the lens), and image full-width halfmaximum (FWHM, i.e. the spread of a point source at the image plane). Parameters
labeled in orange are the requirement variables—they are the design parameters on
which we seek to levy top-level engineering requirements. The blue parameters, by
contrast, are things over which the engineers have control but for the purposes of
requirements optimization remain fixed because those design decisions have already
been made. In this example, the passband is fixed because the HAS2 imager is only
62
Figure 3-7: Overview of the star camera measurement pipeline. Star images are
converted to centroids using a combination of hardware and software elements. The
imaging system block is expanded in Figure 3-8 and centroid processing is expanded
in Figure 3-10.
sensitive over a specific range (400 nm to 900 nm) and the lens must be designed
accordingly. The transmission is assumed to be 90%, a typical achievable value for
refractive optical assemblies. The image FWHM is constrained by a desire to create
a star image that is 2 pixels (i.e. 36 µm) in diameter on the focal plane to ensure
good centroids.
Figure 3-9 goes one layer further down to reveal the contents of the HAS2 block.
Here we have two elements: the pixel array that converts incident photons into electrons, and the on-chip analog-to-digital converter (ADC) that converts the electrons
into an array of digital numbers. As before, design parameters are shown to the left.
63
Figure 3-8: Details of the imaging system block from Figure 3-7. Design parameters
are shown to the left. Orange indicates a requirement variable and blue indicates a
parameter that has been fixed. The HAS2 block is expanded in Figure 3-9.
In this case all are blue because they are fixed due to previous design choices. An integration time of 200 ms is chosen to match the control loop update rate of 5 Hz. The
temperature (20 ◦ C) and total ionizing dose (5 krad) are determined by the expected
worst-case orbital environment for the mission and a decision to omit active thermal
control. The ADC properties are set such that sensor dynamic range is maximized,
with the exception of the quantization which is inherently 12 bits by design.
Finally, Figure 3-10 shows the contents of the centroid processing software block
in Figure 3-7. We adopt the centroid algorithm proposed by Liebe [83], which starts
from a set of square windowed star images. For each window, the mean of the pixels
on the border is calculated and subtracted from each non-border pixel. Calling the
windowed image I(i, j) with row and column indices i and j respectively, the centroid
64
Figure 3-9: Expanded view of the HAS2 sensor block from Figure 3-8. Design parameters are shown to the left.
(x̂, ŷ) is calculated in pixel coordinates as the center of mass according to
x̂ =
n−1 X
n−1
X
i · I(i, j)
i=2 j=2
ŷ =
I0
n−1 X
n−1
X
j · I(i, j)
i=2 i=2
I0
(3.3)
(3.4)
where,
I0 =
n−1 X
n−1
X
I(i, j)
(3.5)
i=2 i=2
and n is the number of pixels in the window3 . In Figure 3-10 we only have one design
3
Note from the summation limits in Equations 3.3–3.5 that the pixels on the edge of the window
(i.e. in the border) are not included in the center of mass calculation.
65
parameter—the number of pixels on each side of the window—which we fix at 9 due
to the small size of the star image and hardware limits on data throughput.
Figure 3-10: Details of the centroid processing block from Figure 3-7. The algorithm
shown is from Liebe [83] and consists of subtracting the mean of the border pixels
from each window then calculating the center of mass.
With no obvious additional levels of decomposition, we conclude our exercise
of documenting the measurement pipeline and reflect on the requirement variables
identified. For this simple example we established two: the lens focal length and fnumber. These parameters constitute our requirements vector—they are the top-level
engineering design variables on which we must levy requirements. All other design
parameters were determined to be fixed as a result of system constraints or prior
66
engineering decisions. The requirement variables are summarized in Table 3.2.
Table 3.2: Requirement variables for the star camera sample problem.
r
r1
r2
3.3.3
Requirement Variable
Focal length, f
Aperture diameter, d
Description
Lens focusing distance for sources at infinity
Diameter of pupil that accepts light
Model Implementation
With the requirement variables identified we are now able to implement the model g
in Equation 3.2 that calculates the figure of merit as a function of the requirement
vector. For the star camera, the input is r = [r1 r2 ] = [f d] and the output is y = N0.1
(see Section 3.2.2), i.e.
N0.1 = g([f d]).
(3.6)
The general approach to implementing Equation 3.2 or 3.6 in model-based requirement definition is to simulate synthetic instrument measurements in accordance with
the documented data pipeline, and then compute the figure of merit directly on the
generated data product. This approach is important because it ensures that the simulated figure of merit accounts for the same data processing artifacts, calibration errors,
etc contained in the eventual real data. Furthermore, and perhaps more importantly,
it ensures that the model is a useful communication tool between scientists and engineers. Scientists will be able to see and understand precisely how the engineering
requirements affect their data and vice versa.
For the star tracker model, we use a Monte Carlo approach and simulate a large
number of star images to calculate the centroid error as a function of stellar magnitude. As the star brightness decreases (i.e. magnitude increases) the image signal
to noise ratio decreases and the centroid error worsens. After finding the magnitude
at which the centroid error is 0.1 pixels, we calculate the average number of stars
in the field of view at or above that brightness level to obtain N0.1 . The simulation
parameters are shown in Table 3.3
67
Table 3.3: Star camera model input parameters and default values.
Parameter
V
T
RTID
fframe
h
w
p
x0
y0
σr
FW
QE
Vlow
Vhigh
Voffset
Nbits
τ
λmin
λmax
FWHM
Nwin
f
d
Description
Value
Star visual magnitude
3≤V ≤8
Focal plane temperature
20 C
Total ionizing dose
5 krad (Si)
Frame rate
5 Hz
Imager height
18.43 mm
Imager width
18.43 mm
Pixel size
18 µm × 18 µm
Image center x coordinate
−p/2 ≤ x0 ≤ p/2
Image center y coordiante
−p/2 ≤ y0 ≤ p/2
Read noise
75 e− RMS/pixel
Pixel full well capacity
105 e−
Mean quantum efficiency over passband
0.3
ADC low voltage rail
0.85 V
ADC high voltage rail
2.0 V
ADC voltage offset
0.85 V
Number of ADC bits
12 bits
Average lens transmission
0.9
Passband minimum wavelength
400 nm
Passband maximum wavelength
900 nm
Image full-width half-maximum
2 pixels
Window size
9 pixels × 9 pixels
Lens focal length (requirement variable)
n/a
Lens aperture diameter (requirement variable) n/a
For each stellar magnitude V in the simulation range V ∈ [3, 8] a set of 1000
simulated star image windows is generated. The center of each star (x0 , y0 ) is sampled
from a two dimensional uniform random distribution defined over x ∈ [−p/2, p/2],
y ∈ [−p/2, p/2]. Each window is then simulated according to the following procedure.
First, we create a point spread function (PSF) the 2D Gaussian function
(x − x0 )2 (y − y0 )2
−
I(x, y) = SV · exp −
2σ 2
2σ 2
(3.7)
√
where SV is the total number of stellar photons received and σ = FWHM/2 2 ln 2.
68
The photon count SV is calculated as,
SV = S0 · 10−V /2.5 · δλ · tint · A · QE · τ
[e− ]
(3.8)
where S0 = 9.6 × 1010 photons/m2 /s/µm is the flux at zeroth magnitude [20], δλ =
λmax − λmin is the passband, tint = 1/fframe is the integration time, A = π/4 · d2 is the
effective lens aperture area, and QE and τ are as defined in Table 3.3.
The PSF from Equation 3.7 is re-sampled onto the 9 × 9 pixel window and a
Poisson random number generator is used to add shot noise. Read noise is then
added as a normal random variable with a standard deviation equivalent to σr and
dark current noise is added using Poisson statistics. The mean of the dark current
distribution is calculated using a vendor-provided equation for dark current as a
function of temperature and radiation dose [18],
D(T, RTID ) = D0 · 2(T −T0 )/∆T1 + a · RTID · 2(T −T0 )/∆T2
[e− /s/pixel]
where D0 = 300 e− /s/pixel, T0 = 30 ◦ C, ∆T1 = 5.8 ◦ C, ∆T2 = 7.1 ◦ C, and a =
325 e− /s/krad/pixel.
After thus creating a simulated window that consists of source and noise contributions, the pixel values (units of electrons) are converted to a 12-bit number using the
ADC parameters. Then the algorithm depicted in Figure 3-10 and Equations 3.3–3.5
is applied to generate centroid estimates (x̂, ŷ) for each window. These values can
be easily referenced to the original image centers (x0 , y0 ) and by taking the standard
deviation over the 1000 samples we can compute an estimate of the centroid errors
as,
σx = stdev((x̂ − x0 )/p)
[pixels RMS]
σy = stdev((ŷ − y0 )/p)
[pixels RMS]
69
where stdev is calculated using the unbiased estimator of the variance:
v
u
u
stdev(x) = t
n
1 X
(xi − x)2
n − 1 i=1
n
where
x=
1X
xi .
n i=1
Figure 3-11 shows several simulated star images that vary by V magnitude and center location. Centroid estimates (crosses) have been computed using Equations 3.3–
3.5 and are compared to the true center locations (circles). As expected, fainter stars
have worse centroid estimates than brighter ones. This is due to the fact that as
the star flux decreases, so does the SNR and consequently the centroid measurement
becomes corrupted by noisy pixel values.
Running the simulation over a range of V magnitudes, we can determine the accuracy of the centroid estimates as function of star brightness. This is illustrated by
Figure 3-12, which shows the expected trend of worsening accuracy as the stars get
fainter (i.e. V increases). We identify a critical brightness level V0.1 at which the centroid accuracy is 0.1 pixel RMS or worse. Recall from the discussion in Section 3.2.2
that this is our threshold for what constitutes a sufficiently accurate centroid for
implementing the attitude determination algorithm. For the set of parameters in
Table 3.3, a focal length of 85 mm, and an aperture diameter of 60 mm, the limiting
magnitude for 0.1 pixel centroid accuracy is V0.1 = 7.1.
Our description of the star camera simulation is nearly complete. We have calculated V0.1 by simulating many windows over a range of stellar magnitudes and
determining the point at which the centroid accuracy drops below 0.1 pixel. The
remaining task is to compute the number of stars at or above the threshold brightness level, i.e. the system figure of merit N0.1 . To do so, we turn to an expression
derived by Liebe [83] for the average number of stars in a given circular field of view.
Modifying Liebe’s original Equation 8 to account for a square field of view and the
70
V=4
V=4
1800
3
3
2000
1600
1400
2
1
1200
1
1000
0
800
−1
1600
y [pixels]
y [pixels]
1800
2
1400
1200
0
1000
800
−1
600
600
400
−2
−2
400
200
−3
200
−3
−3
−2
−1
0
x [pixels]
1
2
3
0
ADU
−3
−2
−1
(a)
0
x [pixels]
1
2
3
(b)
V=8
V=8
60
45
3
3
40
2
30
20
40
1
y [pixels]
25
0
50
2
35
1
y [pixels]
0
ADU
30
0
15
−1
20
−1
10
5
−2
10
−2
0
−3
−3
−2
−1
0
x [pixels]
1
2
3
0
−3
−5
ADU
−3
−2
−1
(c)
0
x [pixels]
1
2
3
ADU
(d)
Figure 3-11: Simulated star camera windows showing the effect of stellar magnitude
on centroid error. The true centroid (x0 , y0 ) is shown with a circle and the calculated
centroid (x̂, ŷ) is shown with a cross. In (a) and (b) stars are V = 4 while in (c) and
(d) the stars are V = 8. The true centroid location in (a) is the same as in (c), and
likewise for (b) and (d). Clearly fainter stars result in worse centroid estimates. The
images were generated using the parameters shown in Table 3.3 with f = 85 mm and
d = 60 mm.
0.1 pixel threshold magnitude we have
N0.1 = 6.57 exp (1.08V0.1 ) ·
Ω
4π
(3.9)
where N0.1 is the average number of stars in the field of view at or below magnitude
71
Centroid Accuracy versus Stellar Brightness
0.25
σ
x
σ
y
Centroid Accuracy [pixels]
0.2
0.15
0.1
0.05
0
4
5
6
Star Brightness [V mag]
7
8
Figure 3-12: Centroid accuracy as a function of star brightness, as calculated using
the star camera end-to-end model. Simulation parameters are from Table 3.3 with
f = 85 mm and d = 60 mm. The centroid accuracies for x and y are shown separately
but they are effectively the same (small deviations are due to limited Monte Carlo
samples). The dashed green line shows V0.1 , i.e. the limiting magnitude at which the
centroid error is 0.1 pixel. In this case V0.1 = 7.1.
V0.1 and Ω is the solid angle on the sky subtended by the detector. This solid angle
is given by,
Ω = 4 arcsin (sin α · sin β)
[steradians]
where α = arctan(h/2/f ) and β = arctan(w/2/f ) are the detector field of view
half-angles in the vertical and horizontal directions, respectively.
The end-to-end star camera model is now complete, providing a mapping from
requirement variables to figure of merit as described by Equations 3.2 and 3.6. To
summarize our approach, for each V magnitude we generated a large number of
simulated star windows and calculated a centroid estimate in each case. We then
calculated the standard deviation across trials to come up with centroid accuracy as
a function of star brightness. We found the limiting brightness at which the accuracy
was 0.1 pixel (V0.1 ), then determined the number of stars in the field of view at or
72
above this limit (N0.1 ).
N0.1 represents our system figure of merit. We can compute this number for a
given requirements vector and see if it meets the performance requirement N0.1 ≥ 20.
For example, Figure 3-13 shows the number of stars as a function of V magnitude
for r = [f d] = [85 mm 60 mm]. As indicated by the green dashed line, there are
approximately 51 stars at or below V0.1 = 7.1, which is much larger than the required
N0.1 value of 20. Therefore this lens is “overkill” for the desired application and may
force unnecessary design effort on the optical engineers. The challenge is therefore
find a requirements vector (or multiple requirement vectors) that satisfy performance
requirements without going too far. This task is the focus of Step 3.
Stars in the HAS2 FOV at or below a Given Brightness
140
Number of Stars in HAS2 FOV
120
100
80
60
40
20
0
4
5
6
Star Brightness [V mag]
7
8
Figure 3-13: Number of stars in the field of view as a function of brightness. Simulation parameters are from Table 3.3 with f = 85 mm and d = 60 mm. The dashed
green line shows the case of V0.1 = 7.1, for which N0.1 = 51 stars.
Before proceeding to use the model in the next step of the requirements methodology, we pause to consider the accuracy of our Monte Carlo simulation. We used
1000 simulated star windows in the above calculations but how can we be sure that
the number is sufficient? There is an error associated with the finiteness of our sample count. Figure 3-14 shows how the figure of merit N0.1 changes as the number of
73
simulated windows increases. Qualitatively, the output stabilizes after approximately
600 samples.
FOM Convergence over Many Monte Carlo Samples
Star Camera Figure of Merit, N0.1 [# of stars]
55
54
53
52
51
50
49
48
47
46
45
10
200
400
600
800
Number of Monte Carlo Samples, M
1000
Figure 3-14: Star camera simulation output versus number of Monte Carlo samples.
While Figure 3-14 tells part of the story, we desire a more quantitative measure of
the error from finite sampling. Following the treatment of Koehler et al. [74], consider
a general model used to calculate some quantity of interest φ. Furthermore, let φ̂M be
the estimate of φ using M samples in a Monte Carlo simulation. We seek to calculate
the standard deviation that one would expect across multiple such simulations using
the same number of samples. Define this as the Monte Carlo error (MCE):
r
MCE φ̂M =
h i
Var φ̂M .
(3.10)
Koehler et al. adapt Efron’s [40] “jackknife” uncertainty estimate—originally developed for bootstrap calculations—to the more general case of Monte Carlo analysis.
They consider a simulation consisting of M samples X = {X1 , X2 , . . . , XM } from
which the Monte Carlo estimate φ̂M (X) is calculated. For each m = 1, . . . , M they
calculate φ̂M −1 (X−m ), where X−m is the set X with the mth sample removed. The
74
estimate of MCE is then given by,
[ φ̂M
MCE
where
v
u
M 2
uM − 1 X
=t
φ̂M −1 (X−m ) − φ̂M −1 (X)
M m=1
(3.11)
M
1 X
φ̂M −1 (X−m ) .
φ̂M −1 (X) =
M m=1
Equation 3.11 thus provides a rigorous way of determining the error in our Monte
[ 0.1 ) as a function of the number
Carlo estimate of N0.1 . Figure 3-15 shows MCE(N
of samples M . This is useful because it allows us to determine whether there are
sufficient samples to achieve a desired accuracy. In our sample study, we wish to
know N0.1 to ±1 star (RMS), therefore according to Figure 3-15 the required minimum
number of samples is approximately 900 (we use 1000 for the sake of convenience and
margin). If the allowable MCE were lower, we could use fewer samples. For example,
M = 200 gives N0.1 within ±2 stars. While Equation 3.11 provides a general recipe for
calculating MCE in any Monte Carlo setting, the allowable error will be application
specific.
3.3.4
Summary
This concludes the third step in the model-based requirement definition methodology: the creation of an end-to-end instrument model. We prescribed some important features regarding the way in which the model is structured and implemented.
Specifically, it must take the form y = g(r) = g([r1 r2 · · · rn ]) where the input
r = [r1 r2 · · · rn ] is the requirement vector (see Section 3.3.1) and the output y is the
figure of merit (see Section 3.2.2). We outlined an approach to identifying requirement variables that relied on documenting the measurement pipeline, accounting for
the effects of data processing. The model output is produced by generating simulated
science measurements (e.g. using Monte Carlo methods) and then calculating the figure of merit on the synthetic data set. Finally, we identified and applied a useful tool
for quantifying the errors associated with Monte Carlo sampling. The entire process
75
Monte Carlo Error versus Number of Samples
10
Monte Carlo Error, MCE [# of stars]
9
8
7
6
5
4
3
2
1
0
10
200
400
600
800
Number of Monte Carlo Samples, M
1000
Figure 3-15: Monte Carlo error (MCE) estimate of N0.1 as a function of number of
samples used.
was illustrated using the star camera as an example.
This procedure is designed to produce a model with three distinctive properties.
First, the model calculates the science figure of merit as a function of engineering
requirements. Second, it does so by generating synthetic data products. Third,
the data product accounts for the effect of software processing. These features are
important, as they ensure the model can serve as a mapping between scientists and
engineers. It encodes how the top-level engineering requirements impact the science
performance and vice versa. This becomes especially important in the next two steps
of the methodology below.
3.4
Step 3: Generate Requirement Vectors
(Optimization)
The first two steps of the methodology were devoted primarily to setting up the
problem, first by defining the science objectives unambiguously and then by creating
76
a model to calculate those objectives. We now get into the “meat” of the methodology
in which we apply numerical methods to identify the requirement vector (or multiple
requirement vectors) that satisfy the performance requirement.
Step 2 was devoted to solving the forward problem of deriving a map from requirement vector r to the performance figure of merit y:
r 7→ y
Forward problem
In other words, the forward problem asks: given a set of requirement variables, what
is the expected performance? To answer this question we create an end-to-end model
g (Equation 3.2).
For the star camera problem, the requirement space—defined here as the set of
all possible requirement vectors—is relatively easy to enumerate and visualize. We
assume that technical constraints impose upper and lower bounds on the requirement
variables f and d. For the sake of concreteness, allow these feasibility limits to be
f ∈ [50 mm, 100 mm] and d ∈ [20 mm, 50 mm]. The enumeration of all possibilities
yields a contour plot of N0.1 defined over the requirement space, shown in Figure 3-16.
Performing this “full factorial” calculation of all (f, d) combinations allows us to
easily identify the set of requirement vectors that satisfy our performance requirement
N0.1 = 20 (shown as a bold red line). We could then choose a specific requirement
vector on that contour—say, [50 mm 25.7 mm] or [100 mm 46.8 mm]—on the basis
of secondary characteristics like cost or risk, and proceed with an optical design.
Full factorial enumeration is feasible for simple problems but may be prohibitive
for more realistic systems with a larger number of requirement variables or more
expensive models. For example, a system with 10 requirement variables each discretized into 20 levels and a model execution time of 1 second would require 2010 =
1.024 × 1013 seconds or roughly 325,000 years of computer time!
Clearly we require a different approach. While Step 2 focused on the forward
problem, Step 3 is devoted to solving the more difficult inverse problem of mapping
77
Figure of Merit, N0.1
50
70
80
40
45
30
Aperture Diameter, d [mm]
60
50
40
20
35
40
10
30
30
20
10
25
20
50
60
70
80
Focal Length, f [mm]
90
100
Figure 3-16: Contours of N0.1 over the requirement space defined by f and d. Recall that N0.1 is the number of stars in the field of view with centroid accuracies of
0.1 pixel or better. The bold red line corresponds to designs that satisfy the performance requirement N0.1 = 20. The plot was generated using the parameters shown
in Table 3.3. Variations in the contour are due to Monte Carlo errors from finite
sampling.
from intended performance y ∗ to the corresponding “ideal” requirement vector r∗ :
y ∗ 7→ r∗
Inverse problem
In other words, the inverse problem asks: what is the requirement vector that produces a particular desired level of performance? In our case, the answer may be
non-unique, so a more appropriate question may be: what is the set of requirement
vectors that all produce the desired level of performance? To answer this question,
we turn to optimization. What follows in the rest of this section is the development
of a novel optimization algorithm specifically intended for model-based requirement
definition. First, however, we consider a general class of goal-seeking problems and
examine existing algorithms.
78
3.4.1
Goal Programming
A general nonlinear programming (NLP) problem can be formulated as follows:
min f (x, p)
x
s.t. c(x, p) ≤ 0
(3.12)
d(x, p) = 0
xLB ≤ x ≤ xUB
Here the intent is to minimize a scalar performance objective f over a vector of design
variables x = [x1 x2 · · · xn ]. In addition to being a function of x, the objective also
depends on parameters p = [p1 p2 · · · pm ] that remain fixed during the optimization.
The optimization is subject to an inequality constraint c and an equality constraint
d. The allowable range of variation for the design variables is defined by the bounds
for all i = 1, . . . , n.
≤ xi ≤ xUB
xLB and xUB such that xLB
i
i
The optimization setting described by Equations 3.12 is one of extremes: the goal
is to find either the minimum or maximum of f over the design space.4 Imagine
instead that we desire to find a particular target value for the objective function.
In this case, the optimization becomes a goal programming problem that can be
formulated as [27],
goal f (x, p) = y ∗
(3.13)
xLB ≤ x ≤ xUB
The objective of goal programming is to find the set of design variables x∗ for which
the objective function equals the desired value y ∗ .5 This approach was introduced by
4
5
To find the maximum of f , one can minimize −f or 1/f .
Note that one can define the goal statement in four different ways [133]:
• f (x, p) = y ∗
• f (x, p) ≤ y ∗
• f (x, p) ≥ y ∗
∗
∗
• f (x, p) ∈ [yL
, yU
].
For our purposes, the equality goal statement best embodies the “satisficing” or “nominal-is-best”
property we desire for requirement definition, therefore Equation 3.13 remains the best way to
express our objective.
79
Charnes et al. in 1955 as a means of optimizing executive compensation in a linear
programming context[16]. Since then, the methodology has been developed more
fully by Ignizio [64, 65] and Lee [80], among others. As a consequence, the technique
has gained considerable traction in a variety of engineering disciplines—see Tamiz et
al. [135] for an overview.
Clearly, goal programming is closely related to the concept of “satisficing” (i.e.
to “satisfy” and “suffice”), first introduced by Simon [130]. The term describes an
observed decision making behavior in which individuals, when faced with competing alternatives, will choose the option that is “good enough” to meet their needs.
This is in contrast to strictly optimizing behavior, in which individuals seek an extreme (i.e. maximum or minimum) outcome. Similarly, Taguchi [134] categorized
utility according to three quality characteristics: “larger-is-better” (maximize output), “smaller-is-better” (minimize output), and “nominal-is-better” (seek a specific
target). More recently, de Weck et al. [23, 25, 24] have described a nominal-is-better
goal programming approach called “isoperformance” in which they identify the set of
designs that exhibit an equal level of (desired) performance. One can then selecting a
point design from among the isoperformance set on the basis of secondary characteristics like cost or risk. The authors argue that by focusing on satisfying requirements
instead of maximizing performance, cost growth can be mitigated.
Ignizio’s original formulation of the goal programming problem augmented the
raw objective function in Equation 3.13 with two non-negative “deviation variables”
p and n to form an equality constraint equation f (x) − p + n = y ∗ (here we omit
the dependence of f on the parameter vector p for clarity). The task then becomes
to minimize the quantity (n + p) using one of several canonical goal programming
methods (i.e. weighted goal programming, lexicographic goal programming, or minmax goal programming [27]). Citing a sensitivity to objective function weights as
a drawback to these classical approaches, Deb argues for the use of a more direct
80
formulation [26, 28] which we adapt for our purposes:
min |g(r, p) − y ∗ |
r
(3.14)
LB
r
UB
≤r≤r
Note that we have replaced the design vector x with the requirements vector r and the
generic objective function f with our system model g. Equation 3.14 thus captures
the essence of model-based requirement definition, which is fundamentally a goal
programming problem. Specifically, we seek the set of requirements r∗ that minimizes
the deviation from our intended level of performance y ∗ . Once we identify r∗ it will
become our set of top-level engineering requirements to which the rest of the system is
designed. We will refer back to Equation 3.14 often as we describe the methodology
further. The challenge now is to define an algorithm that can solve this equation
efficiently.
3.4.2
Genetic Algorithms
The universe of optimization algorithms for solving Equation 3.14 is potentially vast.
Fortunately, we can quickly narrow down the search based on the nature of our problem. Adopting Deb’s scheme [27], we categorize algorithms as either classical or
heuristic. A classical optimization uses deterministic rules to transition from one
iteration point to the next, typically along a locally defined search direction. In
gradient-based classical methods, the search direction is chosen based on the direction of steepest ascent or descent toward the nearest extremum, as calculated using
first or second derivatives of the objective function. Deb articulates several oft-cited
shortcomings of classical optimization algorithms:
• The speed and accuracy of convergence to an optimum depends on the initial
condition.
• Algorithms often get stuck at a point that, while locally optimal, is globally
suboptimal.
81
• Algorithms often require the existence of gradient information.
• Algorithms may not be efficient at handling discrete or mixed integer design
spaces.
• Algorithms cannot be efficiently run on a parallel computer.
Heuristic algorithms attempt to circumvent these problems by evaluating multiple points in parallel, using stochastic transition rules, avoiding the use of gradient
information, or a combination of these approaches. [our problem would seem to argue for a heuristic approach, if only because Equation 3.14 clearly lacks well-defined
derivatives at the optimum point]. There are many heuristic algorithms in practice
today, such as simulated annealing (SA) [73], particle swarm optimization (PSO) [71],
Tabu search [52, 53] and various hybrid approaches that combine two algorithms (occasionally both classical and heuristic). In this work, we focus particularly on genetic
algorithms (GAs). As a type heuristic algorithm, GAs to circumvent the problems
listed above. Moreover, and importantly for our application, certain GA variants have
an inherent notion of diversity preservation—i.e. maintaining a spread of optimal solutions across the design space—that is quite useful for model-based requirement
definition. These details will be described momentarily but first we consider the
origins and structure of GAs generally.
Genetic algorithms trace their origins to the pioneering work of Holland [60] and
were subsequently expanded by Goldberg [54], Deb [27] (particularly for the multiobjective case), and countless others. The literature of GAs is truly expansive and
the discussion that follows focuses on GA variants that are particularly relevant to
this work. Still, all GAs follow the same general evolutionary principles derived from
biology. The main idea is that solutions in the design space are represented by a
population of individuals that “evolve” as the algorithm proceeds. An individual’s
fitness is determined by the value of the objective function at its location in the design
space. Over time, those individuals with the highest fitness will pass their “genes”
on to future generations and the population will converge to the global optimum.
82
Figure 3-17: Operation of a general genetic algorithm (GA), adapted from [27].
Figure 3-17 shows the operation of a generic GA. The first step is to initialize the
starting population r0 and evaluate it according to the objective function. Next, a
fitness level is assigned to each individual and the algorithm checks to see if a convergence criterion is met. Common criteria include whether the maximum allowable
number of generations has been reached or whether the fitness is within some userdefined tolerance. If the algorithm continues, the population undergoes three genetic
operations. First, a selection operator chooses a set of high fitness individuals to serve
as the parent population. Second, the parents are “mated” pairwise by a crossover
operator that produces a set of offspring. Like biological systems, the offspring take
characteristics from their parents. And because of the selection operator has chosen
high fitness individuals, the offspring will on average be more fit than the population
that started the iteration. The third step is to mutate some fraction of the offspring
83
to maintain diversity. This allows the algorithm to more efficiently explore the design
space. Within this general recipe, specific GAs are distinguished by the particular
operators that control fitness assignment, selection, crossover, and mutation.
To illustrate this most basic GA in action, consider a simple test function f (x) =
x21 + x22 . For the sake of concreteness, say that we are interested in locating the value
f (x) = 0.8 over (0 ≤ x1 ≤ 1, 0 ≤ x2 ≤ 1). The optimization problem is then,
min |f (x) − 0.8|
x
0 ≤ x1 ≤ 1
(3.15)
0 ≤ x2 ≤ 1
Figure 3-18 shows how an initial population of 20 individuals evolves over multiple
generations. The columns show two separate optimization runs, differing only by the
choice of initial population (created using a uniform random number generator with
two different seeds). As the algorithm iterates, the population converges toward the
intended contour (bold red line) and at the beginning we see some level of diversity
between individuals. Eventually, however, all individuals collapse to a single point
in the design space. Furthermore, the location of this final point varies between
the two runs. Trials over multiple optimization runs with different initial conditions
show that, while the algorithm always converges to the intended level of 0.8, the final
location along the f (x) = 0.8 curve is random.
3.4.3
Level Sets and Diversity Preservation
One could argue that we should be satisfied by the above result. After all, the
standard GA does indeed find a solution in every case. From the perspective of
defining engineering requirements, however, this might be too limiting. For a given
objective y ∗ there well could be multiple sets of requirements r∗1 , r∗2 , . . . r∗n that
all produce the same level of scientific performance. Furthermore, these various r∗i
could each reside at vastly different locations in the requirement space and have
correspondingly different impacts on system design. For example, r∗1 might place
84
Initial Population
1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
x2
x
2
Initial Population
1
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
0.2
0.4
0.6
0.8
0
0
1
0.2
0.4
x1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.4
0.6
0.8
0
0
1
0.2
0.4
x1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
1
0.8
1
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.4
0.8
Generation 15
1
x2
x
2
Generation 15
0.2
0.6
x1
1
0
0
1
0.5
0.4
0.2
0.8
Generation 3
1
x2
x
2
Generation 3
1
0
0
0.6
x1
0.6
0.8
0
0
1
x1
0.2
0.4
0.6
x1
Figure 3-18: Attempts at solving Equation 3.15 using a basic GA with two different
initial populations. The desired level of 0.8 is indicated by the bold red line. In both
cases the population collapses to a single point.
85
demanding requirements on the thermal subsystem while placing loose requirements
on the mechanical subsystem; r∗2 might be the opposite, and so on. In this case, a
more useful algorithm would produce a diverse set of requirement vectors that all
satisfy the performance requirement y ∗ . Calling this set R and adopting the more
general notation of Equation 3.14, we seek
R = {r∗ | g(r∗ , p) = y ∗ }
(3.16)
Note that this problem is equivalent to finding the level set of the function g at
the value y ∗ . The level set of a scalar function f : Rn → R corresponding to the level
c is defined as the set of points {x ∈ Rn | f (x) = c}. For the simple test function
p
f = x21 + x22 above, we can find the level set for f = 0.8 explicitly as x2 = 0.8 − x21
and graph the resulting curve in 2D space (see the bold red line in Figure 3-18).
However, for the general case of Equation 3.16 the solution is not as straightforward.
GAs have the potential to help in this regard and their parallel nature is especially
attractive if we desire a locus of points in R. But clearly the basic GA approach
outlined in Figure 3-17 and tested in Figure 3-18 will not work. We seek the entire
level set but traditional GAs collapse to a single extremum. In other words, we
desire some degree of diversity preservation in which the spread of the population is
maintained over the design space while simultaneously converging to g = y ∗ . Below
we propose a new GA variant that accomplishes exactly this. First, we describe GAs
that include diversity preservation in promising ways and discuss why a new approach
is required.
A number of GAs have been developed to incorporate diversity preservation
schemes. Such algorithms are typically motivated by the desire to find several extrema
of a multi-modal function. A common technique is the so-called fitness sharing or
niche method. Originally developed by Goldberg and Richardson [56], fitness sharing
is based on the idea that each optimum in a multi-modal function can only accommodate a finite number of individuals who are forced to “share” the fitness in that
region. The approach is inspired by the tendency of biological species to specialize
86
into subpopulations or “niches” that occupy different parts of the environment (e.g.
different elevations of a mountain) so as to avoid competing for resources. In biology,
the size of a niche is proportional to the resource available and likewise in fitness
sharing GAs, the niche count is limited by the relative height of the optimum. If a
niche is full, additional solutions are prevented from entering and will be forced to
populate another optimum.
The clear challenge in using fitness sharing is defining the size of a niche. Goldberg
and Richardson propose a sharing function that parameterizes niche size as a function
of distance between two solutions and a user-defined normalization parameter σshare .
The niche size, and therefore the solution quality, is highly dependent on the selection
of this parameter. Deb and Goldberg [32] suggest a method of choosing σshare based
dividing the n-dimensional search space into q hyperspherical niches, where q is the
number of expected optima. Alternatively, Beasley et al. [7] describe a sequential
niche technique in which the fitness function is degraded near previously identified
optima. This penalizes individuals in regions of the design space that have already
been explored and encourages the discovery of new optima elsewhere. Other niching
approaches have been described by Horn et al. [63], Miller and Shaw [89], Gan and
Warwick [49], and Li et al. [82].
The above techniques all require a user-defined parameter to set the allowable
spacing between solutions, making the algorithms difficult to apply in real world
problems where the nature of the design space is not known a priori. Consequently,
several adaptive or parameter-free approaches to preserving diversity have been proposed. In their work on multiobjective GAs, Fonseca and Fleming [43] calculate σshare
in each iteration. They do so by discretizing the objective space near the Pareto frontier (i.e. the locus of non-dominated solutions) into hypercubes, the number of which
corresponds to the number of points in the evolving population. By reasoning that
side length of each n-dimensional hypercube should be σshare the authors are able to
calculate the sharing parameter by solving for the roots of a nth order polynomial.
Deb and his students took a different approach to diversity preservation in their
pioneering work on an elitist non-dominated sorting GA for multiobjective optimiza87
tion [30, 37]. Termed NSGA-II, their approach was to define a “crowding distance”
around points as they evolve. An n-dimensional cuboid is defined around a given
individual using the closest neighbors along each of the objectives as vertices. The
average side length of this cuboid is the crowding distance of that individual. It provides an estimate of the density in that region of the objective space (a larger crowding
distance represents a smaller density of solutions). The NSGA-II algorithm first sorts
solutions into a series of “fronts” using a fast non-dominated sorting routine. Better
fronts are those that reside closer to the utopian point6 and therefore provide the
best approximation of the actual Pareto frontier. Points within a given front are then
sorted by crowding distance. The offspring population is determined based on a combination of front and crowding distance. The interested reader is referred to Deb [27]
and Deb et al. [37] for details but the overall effect is that solutions simultaneously
evolve both toward and along the Pareto front, resulting in a well-dispersed optimal
population.
Of the diversity-preserving GAs discussed, NSGA-II is the most conceptually similar to what we seek for solving the level set problem in Equation 3.16. It offers a
parameter-free approach to ensuring a spread of solutions along the Pareto front.
However, NSGA-II was designed specifically for finding non-dominated solutions in a
multiobjective optimization context and cannot be immediately used for the (scalar)
level set problem. Specifically, both the non-dominated sorting step and crowding
distance calculation are based on a multi-dimensional objective space with a well
defined utopian point. The level set problem, on the other hand, deals with a multidimensional design space where the concept of a utopian point ceases to exist (indeed,
a level set could even be a closed loop). As such, we have developed a new genetic
algorithm—inspired in part by NSGA-II but in other ways quite different—aimed
specifically at level set problems. It is described in the following section.
6
In multiobjective optimization, the “utopian point” is the corner of the objective space at
which the ideal design would reside. For example, in a 2D objective space defined by cost and
performance, the utopian point would be at the corner where cost is at a minimum and performance
is at a maximum. Utopian points are generally unobtainable due to constraints or design variable
side bounds, giving rise instead to a Pareto front of non-dominated solutions.
88
3.4.4
Level Set Genetic Algorithm (LSGA)
The purpose of the level set genetic algorithm (LSGA) is to solve problems in the
form of Equation 3.16 and produce the level set R. Recall that in the context of
model-based requirement definition, R is the set of requirement vectors r∗ that result
in the desired system performance y ∗ . De Weck [23] developed algorithms for the case
of linear time-invariant (LTI) systems, however we desire a flexible approach that is
applicable to more general problems such as those found in model-based requirement
definition.
LSGA operates by evolving a population over the design space in search of the
level set. The user must first specify optimization parameters—some common to
all GAs and some unique to LSGA—that govern the algorithm’s execution. Let N
be the population size and n be the number of dimensions in the design space, i.e.
x = [x1 . . . xn ]. For reasons that will be discussed below, let NLS be the intended
number of points in the level set R. Broadly speaking, this corresponds to the desired
resolution or granularity over the level set. Note that N and NLS can be different. As
was the case above, y ∗ is the desired performance level. We introduce a parameter
τ which is the allowable tolerance for inclusion in the level set (i.e. performance
values within y ∗ ± τ are considered satisfactory). To produce successive generations,
LSGA uses a crossover operator and a mutation operator. These are described in
greater detail below and for now we simply point out that the user must specify
four parameters that define the relative strength of these operations: the crossover
probability pc , the crossover distribution index ηc , the mutation probability pm , and
the mutation distribution index ηm . Finally, the user must provide design variable
UB
side bounds xLB = [xLB
. . . xLB
= [xUB
. . . xUB
1
n ] and x
1
n ] along with the number of
generations to use, Ngen . These parameters are summarized in Table 3.4.
Design Variable Normalization
The main LSGA iteration loop is described below. However, first we briefly consider
a few topics that are relevant to the upcoming discussion, beginning with design
89
Table 3.4: Summary of LSGA optimization parameters.
Parameter
N
n
NLS
y∗
τ
pc
ηc
pm
ηm
xLB
xUB
Ngen
Description
Population size
Number of design variables
Desired number of entries in the level set
Desired performance level
Level set convergence tolerance
Crossover probability
Crossover distribution index
Mutation probability
Mutation distribution index
Design variable lower bounds
Design variable upper bounds
Number of generations to iterate
variable normalization.
Generally, the raw design variables can take on arbitrary units and side bounds,
resulting in different scaling on each axis of the design space. In the star camera
example, the focal length (x1 ) varies between xLB
= 50 mm and xUB
= 100 mm,
1
1
UB
while the f-number (x2 ) varies between xLB
2 = 1.4 and x2 = 3.0. One could imagine
more extreme scaling discrepancies in other problems. A common way of dealing with
this issue is to normalize the design variables to a standard interval, for example [-1,1]
or [0,1]. In our case we choose the latter. If the raw design variables x are defined
on [xLB , xUB ], we can define the normalized design variables x0 through the simple
scaling,
x0 = fscale (x) =
x − xLB
xUB − xLB
x0 ∈ [0, 1].
(3.17)
Likewise, if we are given normalized design variables we can convert back to raw
design variables through the mapping,
x = funscale (x0 ) = xUB − xLB · x0 + xLB
x ∈ [xLB , xUB ].
(3.18)
The various LSGA operations described below (crowding calculation, diversity
calculation, crossover, mutation, etc) all use normalized design variables x0 . It is
90
only when evaluating the fitness function that we must convert back to raw design
variables x. That said, for the sake of simplicity we will use the non-prime notation x
even when using normalized variables. The distinction between normalized and raw
variables will be made explicit in cases where the interpretation is ambiguous.
Crowding Metric
LSGA assigns a crowding metric C to each point in the design space to quantify the
relative density of solutions at that location. This metric is defined as the number of
solutions that lie inside a hypersphere of radius rc centered on the solution of interest.
In an ideally dispersed population of N points spread over a trade space volume V ,
the hypersphere of each individual on average would occupy a volume of VN = V /N .
The trade space with normalized design variables has a volume of unity, therefore
VN = 1/N . For two-dimensional problems, the “volume” VN is actually the area of a
circle and the solution for rc is simple:
1
N
1
π · rc2 =
N
r
VN =
⇒ rc =
1
πN
A similar calculation is possible for three-dimensional problems using the volume
formula for a sphere. For n-dimensional problems, the volume of a hypersphere of
radius r is [146],
Z
Vn =
ρ
Sn ρn−1 dρ =
0
Sn · rn
n
(3.19)
where Sn is the n-dimensional surface area of a hypersphere of unit radius,
n
2π 2
Sn =
Γ n2
(3.20)
and Γ is the gamma function. Combining Equations 3.19 and 3.20 and using the
identity Γ(1 + n2 ) = n2 Γ( n2 ), we have the n-dimensional volume of a hypersphere with
91
radius r as,
n
π2
rn .
Vn =
Γ 1 + n2
Setting this equal to 1/N and solving for r gives the desired radius:
"
#1/n
1 Γ 1 + n2
rc = √
N
π
(3.21)
Thus the crowding metric C for each point in the population is calculated by determining the number of other points that lie within the radius given by Equation 3.21.
Note that rc is constant across iterations, so it can be calculated at the start of LSGA
and used throughout the optimization. Figure 3-19 shows rc and C for an initial
population of 40 individuals in a two-dimensional problem.
Initial Population
1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
x2
x2
Initial Population
1
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
0.2
0.4
0.6
0.8
1
0
0
0.2
0.4
x1
0.6
0.8
1
x1
(a)
(b)
Figure 3-19: Crowding metric for an initial population. Circles with radius rc are
shown in (a) with gray dashed lines. The corresponding crowding metric C is represented in (b) as a magenta circle whose radius is inversely proportional to the C value
of that point. Thus smaller circles correspond to more crowded regions of the design
space (i.e. larger values of C). In this example n = 2, N = 40, and rc = 0.089.
92
Diversity Metric
As we proceed, it will be useful to have a metric D that quantifies the diversity of
the level set. By diversity we mean a measure of how different, on average, the points
are in terms of location in the design space. There are two components to diversity:
the spread of points (i.e. a wide distribution over the design space is preferable to
points that are clustered in one region) and the spacing of points (i.e. even spacing is
preferable to points that are clumped together). An ideal level has both properties:
evenly spaced points spread across a large area of the design space.
We consider spread first. The maximum extent of a distribution of points can be
used to define a rectangle (or its n-dimensional equivalent) in the design space. Figure 3-20 shows an example for the 2D case. The area of the shaded rectangle defined
max
min
min
by the minimum and maximum points along each axis is A = xmax
−
x
x
−
x
.
1
1
2
2
Ideally the points would span the entire design space, which leads to a natural definition of the spread metric as the fractional area covered by the population,
Dspread =
Atotal − A
Atotal
(3.22)
where Atotal is the total area of the design space. Notice that a highly spread population will have Dspread approaching zero while a tightly packed population will have
Dspread closer to unity. For normalized variables, the design space always has a unit
area so Equation 3.22 reduces to Dspread = 1 − A. Generalizing the area formula to n
dimensions we have the final spread metric
Dspread = 1 −
n
Y
xmax
− xmin
i
i
Dspread ∈ [0, 1]
(3.23)
i=1
where xmax
is the maximum value of the ith component over the population and
i
likewise for xmin
i :
n
o
(1)
(2)
(N )
xmax
=
max
x
,
x
,
.
.
.
,
x
i
i
i
i
n
o
(1)
(2)
(N )
xmin
=
min
x
,
x
,
.
.
.
,
x
.
i
i
i
i
93
(3.24)
1
0.9
0.8
xmax
2
0.7
0.6
0.5
0.4
0.3
0.2
xmin
2
0.1
xmax
1
xmin
1
0
0
0.2
0.4
0.6
0.8
1
Figure 3-20: Calculating the spread metric Dspread as the fraction of the design space
covered by the population.
Now consider the other aspect of diversity: spacing. For the ith individual in the
population we can calculate the distance to the closest other individual as
v
u n 2
uX (i)
(j)
(i)
(j)
di = min kx − x k = min t
xk − x k
j6=i
j6=i
(3.25)
k=1
where k · k is the Euclidean norm. Clearly, for an ideally spaced population the di for
all points would be equal. Therefore a natural definition of the spacing metric is,
N
Dspace
where d = (1/N )
PN
i=1
1 X
=
|di − d|
N d i=1
(3.26)
di . According to this metric, a population with perfectly even
spacing would have di = d for all points and consequently Dspace = 0.
The maximum value of Dspace occurs when one point is at a given location in the
design space and N − 1 points are all at a different location some distance away. This
situation is depicted in Figure 3-21. Here d1 = k∆xk while for the rest of the points
i = {2, . . . , N }, di = 0. Thus in this case d = k∆xk/N and we can calculate an upper
94
bound on Dspace as,
N
1 X
|di − d|
N d i=1
1
k∆xk
k∆xk
k∆xk
k∆xk −
+ 0 −
+ · · · + 0 −
=
k∆xk N N N 1
N −1
=1−
+
N
N
N −1
(3.27)
=2
N
max
=
Dspace
We would like to normalize Dspace so that it always lies on the interval [0, 1]. The
lower bound is already 0 as described above and we can ensure an upper bound of 1
by dividing Equation 3.26 by Equation 3.27 to get
N
Dspace =
X
1
|di − d|
2(N − 1)d i=1
Dspace ∈ [0, 1]
(3.28)
1
0.9
x1
0.8
0.7
0.6
k∆xk
0.5
0.4
0.3
0.2
x2 , . . . , xN
0.1
0
0
0.2
0.4
0.6
0.8
1
Figure 3-21: Calculating the worst case spacing metric Dspace . Individual i = 1 is
located at (0.2, 0.9) while individuals i = {2, . . . , N } are all located at (0.8, 0.2).
Putting the pieces together, we add the spread and spacing metrics to get a
95
combined diversity metric
D = Dspread + Dspace
D ∈ [0, 2]
(3.29)
where Dspread is defined in Equation 3.23 and Dspace is defined in Equation 3.28.
Figure 3-22 shows how D changes based on the combination of spread and spacing in
a population.
Main Loop
Having introduced the crowding and diversity metrics, we now describe the main loop
of LSGA, which is shown in Algorithm 1. The procedure begins by creating an initial
parent population PP at random locations in the design space. The initial fitness FP
for each individual is calculated by evaluating Equation 3.14 over every member of
the population, i.e.
FP = |g(x, p) − y ∗ | ∀ x ∈ PP .
The fitness metric captures how far a given solution is from the desired level. We also
initialize an empty level set PLS that will be filled over successive iterations of LSGA.
Algorithm 1 Level Set Genetic Algorithm (LSGA)
PP = random(xLB ≤ x ≤ xUB )
Initialize parent population at random locations
FP = fitness(PP )
Calculate fitness of initial population
PLS = { }
Initialize level set population (empty)
for i = 1, . . . , Ngen do
CP = crowding(PP ∪ PLS )
Calculate crowding metric for parent population
RP = rank(FP , CP )
Assign rank based on fitness and crowding
[PC , FC ] = offspring(PP , RP )
Create child population
PPC = PP ∪ PC ,
Combine parent and child populations (size 2N )
FPC = FP ∪ FC
PLS = extract(PPC )
Extract and add new members of level set
CPC = crowding(PPC ∪ PLS )
Calculate crowding for combined population
RPC = rank(FPC , CPC )
Assign rank based on fitness and crowding
PP = choose N best individuals in PPC
New parent population (size N )
end for
R = PLS
Assign output
96
D = 1.1782
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
0.2
0.4
0.6
D = 0.67048
1
x2
x2
1
0.8
0
0
1
0.2
0.4
x1
(a) Poor spread, poor spacing.
0.8
0.8
0.7
0.7
0.6
0.6
x2
x2
0.9
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.2
0.4
0.6
x
1
D = 0.21893
1
0.9
0
0
0.8
(b) Good spread, poor spacing.
D = 0.99464
1
0.6
x1
0.8
1
0
0
0.2
0.4
0.6
0.8
1
x
1
1
(c) Poor spread, good spacing.
(d) Good spread, good spacing.
Figure 3-22: Effect of spacing and spread on the population diversity metric D. In
all plots N = 5.
With the initialization tasks complete, LSGA begins in earnest. A single generation of the algorithm is depicted in Figure 3-23. The first step is to calculate CP , the
crowding metric for each member of the parent of the parent population. Recall that
C for a given individual is the number of other points within a radius rc defined by
Equation 3.21. Importantly, we include points in the level set in this count. Doing so
97
encourages the population to evolve away from regions where the level set has already
been found.
Figure 3-23: One generation of LSGA.
Once the crowding metric has been calculated, solutions are ranked based on a
combination of fitness and crowding. The ranking procedure is shown in Algorithm 2
and works by sorting the population twice: once according to fitness and once according to crowding. The rank is then assigned by adding the individual’s position in the
sorted fitness list to their position in the sorted crowding list. Having a higher rank
(smaller number) is favorable. For example, if two individuals have similar fitness
but different crowding, the one with the better (smaller) crowding metric will have a
better rank. Similarly, if two individuals have similar crowding metrics but different
fitness, the more fit individual (closer to the desired level) will have a better rank. Intuitively this is desirable, as individuals close to the desired level but in non-crowded
parts of the design space will be given the highest rankings. Furthermore, the ranking
is relative so there are no parameters to adjust as the population evolves.
Once the parents are ranked, the next step is to create an offspring population
through the evolutionary processes of selection, crossover, and mutation that together
are shown in Algorithm 3. The role of selection is to choose individuals of above average fitness for participation in the mating (crossover) operation and thereby increase
98
Algorithm 2 Ranking within LSGA
procedure rank(FP , CP )
f = sort FP ascending
Sort by fitness
c = sort CP ascending
Sort by crowding
for i = 1, . . . , N do
RP [i] = position in f + position in c
Assign combined rank (smaller is better)
end for
end procedure
the chance that desirable qualities will be propagated to future generations. Goldberg
and Deb showed that the so-called tournament selection operator has convergence
that is superior or equivalent to other approaches in the literature [55]. With this in
mind, LSGA uses binary tournament selection. Parents are chosen randomly two at
a time for participation in a “tournament”. The individual with the better rank wins
the tournament and is placed into the mating pool Ppool . There are N tournaments
total and each individual participates exactly twice, therefore the final mating pool
of N individuals contains two copies of the best individuals. Other individuals are
represented either once (mid-range ranks) or not at all (worst ranks).
Algorithm 3 Creating child population within LSGA
procedure offspring(PP , RP )
Ppool = selection(PP , RP )
Binary tournament selection
PC = crossover(Ppool )
Simulated binary crossover (SBX)
PC = mutation(PC )
Polynomial mutation
FC = fitness(PC )
Calculate fitness of children (typically expensive)
end procedure
After selection, individuals in the mating pool produce children through the
crossover operator. The user-defined crossover probability pc governs the likelihood
of this process on a point-by-point basis, with pc % individuals undergoing crossover
and (1 − pc )% copied directly to the offspring population unchanged.
The first GAs used binary strings to encode real-valued design variables. These
“chromosomes” could then be combined through a crossover operator in which bits
(i.e. “genes”) are exchanged between two individuals starting from a single point chosen randomly along the string. The binary encoding approach leads to shortcomings
99
for real-valued problems, including an artificial quantization of the design space and
difficult transitions between Hamming cliffs (e.g. from 0111 to 1000) [27].
Consequently, Deb et al. devised the simulated binary crossover (SBX) operator to
emulate the behavior of single-point binary crossover for real-valued problems [29, 36].
We use their approach in LSGA, which creates two children z(1) and z(2) from two
parents x(1) and x(2) . The authors first define a “spread factor” βi as the ratio of
differences between the offspring and parents
z (2) − z (1) i βi = i(2)
x − x(1) i
i
i = 1, . . . , n.
(3.30)
They then derive the following probability distribution that has a “search power”
similar to that of single-point binary crossover,
P (βi ) =


0.5(ηc + 1)βiηc

0.5(ηc + 1)
if βi ≤ 1
1
βiηc +2
(3.31)
otherwise.
In this equation, ηc is a non-negative real number called the crossover distribution
index that defines the “spread” in the offspring compared to the parent. Larger values
of ηc correspond to children that are relatively close to their parents, while smaller
values correspond to children that are relatively far. For a uniform random number
ui ∈ [0, 1] one can find the value βi∗ at which the area under the probability curve
from 0 to βi∗ is equal to ui :
Z
βi∗
P (β)dβ = ui
0
Solving this expression yields [29]


(2ui ) ηc1+1
βi∗ = η 1+1

c
1

2(1−ui )
if ui ≤ 0.5
(3.32)
otherwise.
Finally, after calculating the βi∗ using the above probability distribution, the vector
100
components of the children can be calculated as,
(1)
zi
(2)
zi
h
(1)
βi∗ ) xi
(2)
βi∗ ) xi
i
+ (1 −
= 0.5 (1 +
h
i
(1)
(2)
= 0.5 (1 − βi∗ ) xi + (1 + βi∗ ) xi .
(3.33)
To summarize, the SBX crossover operation to create a pair of children z(1) and
z(2) from parents x(1) and x(2) is a four step process. First, randomly choose pc %
of the mating pool for crossover and copy the remaining (1 − pc )% directly into the
offspring population without modification. Second, for each individual participating
in crossover, generate uniform random numbers ui ∈ [0, 1]. Third, calculate βi∗ using
Equation 3.32. Fourth, create the offspring using Equation 3.33. Note that this
approach can be easily adapted for discrete variables in mixed integer programming
problems by replacing the continuous probability distribution in Equation 3.31 with
a discrete equivalent. This has been demonstrated by Deb and Goyal [33, 34, 35].
The last step in creating offspring is mutation. The idea behind mutation is to
inject additional randomness into the population so as to avoid traps at local extrema
and explore more of the design space. For the case of binary-coded GAs, mutation is
accomplished by toggling random bits in the chromosome. For real-valued GAs, Deb
and Goyal [33] devised a polynomial mutation operator that is analogous to SBX. We
use their approach here to generate mutated offspring e
z from the original version z.
The authors present the probability distribution
P (δi ) = 0.5(ηm + 1)(1 − |δ|)ηm
δi ∈ [−1, 1]
(3.34)
where ηm is the mutation distribution index. As with the crossover distribution index,
larger values of ηm correspond to smaller amounts of mutation “spread” and vice versa.
Following the same steps used above, we select a uniform random number ui ∈ [0, 1]
and integrate Equation 3.34 from 0 to δi∗ such that the area under the curve equals
101
ui . Doing so gives [33]
δi∗ =


(2ui )1/(ηm +1) − 1
if ui < 0.5

1 − [2 (1 − ui )]1/(ηm +1)
if ui ≥ 0.5.
(3.35)
The vector components of the mutated offspring can then be calculated as,
∗
zei = zi + xUB
− xLB
δi
i
i
(3.36)
Note that the specified mutation probability pm is used to randomly select pm %
of the offspring population to undergo mutation. The remaining (1 − pm )% are unaffected. To summarize the process, we first generate generate uniform random numbers
ui ∈ [0, 1] for each individual participating in mutation. Second, Equation 3.35 is used
to calculate δi∗ . Third, the mutated offspring are created using Equation 3.36. As
with SBX, this mutation procedure can be adapted to integer-valued variables by
using the discrete version of the probability distribution in Equation 3.34.
Now that the child population PC has been created, the final step before returning
to the top-level LSGA procedure is to calculate the fitness of the new individuals.
Typically this will be the most computationally expensive part of the algorithm since
the system model g must be evaluated once for each of the N offspring:
FC = |g(z, p) − y ∗ | ∀ z ∈ PC .
The next step in the LSGA main loop (Algorithm 1) is to extract any members
of the population that are eligible for inclusion in the level set, i.e. solutions with
fitness scores less than the user-defined tolerance τ . The extraction process operates
on the combined parent and child populations PPC = PP ∪ PC and is shown below in
Algorithm 4.
Any individual in the combined population with a fitness less than τ is considered
a candidate for the level set. We denote the level set PLS and the level set candidates
∗
PLS
. The level set has a maximum size NLS specified by the user, therefore whenever
102
new candidates are extracted there are three possibilities. First, there could be enough
room in the level set to accommodate all of the candidates. Second, there could be
enough room to only accommodate some of the candidates. Third, the level set may
completely full and unable to accommodate any candidates. The extraction procedure
deals with each case separately.
If the level set has enough room for all of the candidates (case 1), the course of
action is straightforward: every candidate is simply accepted into PLS . If the level
set can only accommodate a fraction of the candidates (case 2), then the candidates
are added one-by-one into PLS until a size of NLS is achieved. Candidates are added
based on the extent to which they improve the diversity metric D for the level set.
∗
[1] is added to
The procedure for doing so is as follows. The first candidate PLS
the level set and the diversity metric for this “augmented” level set is calculated as
∗
D+1 = D (PLS + PLS
[1]). The same is done for the remaining candidates, producing
∗
D+2 = D (PLS + PLS
[2]) and so on. The candidate that produces the best level set
diversity (i.e. lowest D+i value) is added to the level set and removed from the list of
candidates. This repeats until the level set is full. Any candidates left over after this
process are handled under the final case, described below.
If the level set is full (case 3), the candidates are used to increase the diversity of
the level set rather than its size. Despite the presence of the crowding metric to encourage spread, the initial level set inevitably has some undesirable clumping. Every
time a new candidate becomes available, the current (full) level set is interrogated
to see if replacing any of the existing members by the new member will improve di+
∗
versity. This is done by first creating the augmented level set PLS
= PLS + PLS
[i].
The pairwise minimum distance metric di is then calculated for each of the NLS + 1
+
members in PLS
. The pair of solutions with closest spacing will have the two smallest
(identical) di values. Call this closest pair x1 and x2 . We wish to remove one of these
individuals, since by definition they are the biggest contributors to clumping. We
just need to decide which one to remove. This is done by calculating the diversity if
+
one or the other were excluded from the augmented level set, i.e. D−1 = D PLS
− x1
+
and D−2 = D PLS
− x2 . If D−1 is better (smaller) than D−2 then x1 is removed
103
Algorithm 4 Extracting the level set within LSGA
procedure extract(PPC )
∗
PLS
= {PPC | FPC < τ }
Candidates for inclusion in the level set
∗
∗
NLS
= size of PLS
Number of candidates
Nempty = NLS − (current size of PLS )
Number of empty slots in the level set
∗
if Nempty ≥ NLS
then
Case 1: Level set has room for all candidates
∗
PLS = PLS ∪ PLS
Simply add the candidates to the level set
∗
else if 1 ≤ Nempty < NLS
then
Case 2: Level set has room for some candidates
while (size of PLS ) < NLS do
+
∗
[i]
PLS
= PLS + PLS
+
D+i = D PLS
Add one candidate to the level set
Calculate the diversity metric (Equation 3.29)
∗
x = candidate with smallest D+i
PLS = PLS + x
∗
PLS
=
∗
PLS
−x
Find candidate producing best diversity
∗
Add that candidate to the level set
∗
Remove that candidate from candidate set
end while
else if Nempty = 0 then
for i = 1,
+
PLS
=
Case 3: Level set is full
∗
. . . , NLS
do
∗
PLS + PLS [i]
Calculate di for all x ∈
Add one candidate to the level set
+
PLS
Minimum pairwise distance (Equation 3.25)
Find x1 and x2 with smallest di
+
D−1 = D PLS
− x1
+
D−2 = D PLS
− x2
+
Calculate D for PLS
excluding x2
if D−1 < D−2 then
If excluding x1 results in better diversity...
PLS =
+
PLS
Identify the pair with the closest spacing
+
Calculate D for PLS
excluding x1
− x1
...then remove x1 from the level set
else if D−2 < D−1 then
PLS =
+
PLS
If excluding x2 results in better diversity...
− x2
...then remove x2 from the level set
end if
end for
end if
for i = 1, . . . ,
All cases:
∗
NLS
do
PPC [i] = random(x
LB
Replace members of PPC that have been extracted
≤x
(i)
≤x
UB
)
Add replacement individuals at random locations
end for
end procedure
104
and vice versa.7 Figure 3-24 shows an example of the procedure. As LSGA iterates,
this replacement behavior will cause the level set diversity to increase, ultimately
resulting in an evenly spaced population of points distributed widely over the design
space. This is an important feature of the algorithm that we will discuss in greater
detail below.
D = 1.0971
1
Final level set
0.9
0.8
0.8
0.7
0.7
0.6
0.6
x2
x2
0.9
0.5
0.4
0.5
0.4
Closest pair
0.3
0.3
0.2
0.2
0.1
0.1
0
0
D = 0.83514
1
Starting level set
New candidate
Removed from level set
0.2
0.4
0.6
0.8
1
0
0
0.2
0.4
0.6
0.8
1
x1
x1
(a)
(b)
Figure 3-24: Replacing members of an already full level set to improve diversity. (a)
Starting level set and new candidate. (b) Final level set after accepting the new
candidate and removing one of the closest pair. Note the improvement (decrease) in
D.
Regardless of what happens to the level set—adding members, replacing mem∗
bers, or a combination of the two—the level set candidates PLS
are removed from
the combined population PPC . Therefore the final step of the extraction procedure
is to replace these missing individuals with new ones at random locations in the design space. The random placement adds a mutation-like spread that encourages the
exploration over the entire design space as the algorithm iterates.
At this point in the LSGA main loop, most of the key activities have occurred.
What remains is to produce the next-generation parent population PP (size N ) from
7
Note that there some special cases with points near the edge of the design space in which
spread
spread
D−1 < D−2 and yet the spread actually decreases when x1 is removed (i.e. D−1
> D−2
).
Therefore, before removing a solution, we also check that the spread does not decrease. If it does,
we remove the other solution instead, thus retaining the spread.
105
the current combined population PPC (size 2N ). This is done by computing the
crowding metric and rank for every individual, then choosing the N best to serve
as parents. This operation is an elite-preserving mechanism that ensures that the
highest ranked individuals (i.e. the “elites”) are preserved in the next generation.
Elite-preserving operators are beneficial in GAs because they improve performance
by propagating high performing solutions and guarantee that the fitness of the best
solutions do not deteriorate [27, 114]. The elite-preserving approach used in LSGA
is inspired by that used by Deb et al. in NSGA-II [30, 37].
Examples
Having described LSGA in detail above, we now apply the algorithm to a few test
functions and discuss the results. First consider the function introduced in Equation 3.15 and used several times throughout this section,
f1 (x) = x21 + x22
0 ≤ x1 ≤ 1
0 ≤ x2 ≤ 1
where we seek the level R set that corresponds to f1 (x) = 0.8, i.e.
R = {x∗ | f1 (x∗ ) = 0.8} .
(3.37)
LSGA was run for 50 generations with N = 40, NLS = 40, τ = 0.008 (i.e. 1% of
the level), pc = 0.9, ηc = 20, pm = 0.5, and ηm = 20. The final results are shown in
Figure 3-25b below.
Clearly LSGA does a good job of finding an evenly spaced level set over the entire
design space by the end of the 50 generations. As discussed above, LSGA essentially
operates in two phases. The first phase is when the level set population is smaller
than the user-defined value of NLS and the algorithm is focused simply on finding new
solutions to grow the level set. During this phase, all candidates are accepted into the
level set, even if they happen to be quite close to previously found solutions. The first
106
Gen 16, NLS = 40, D = 0.98266
Gen 50, NLS = 40, D = 0.28635
1
1
Level Set
0.9
0.8
0.8
0.7
0.7
0.6
0.6
x2
x2
Level Set
0.9
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
0.2
0.4
0.6
0.8
1
x1
0
0
0.2
0.4
0.6
0.8
1
x1
(a) Level set at generation 16 of LSGA.
(b) Level set at generation 50 of LSGA.
Figure 3-25: Comparing the solution found by LSGA (a) when the level set first
becomes full in generation 16 and (b) at the end of the 50 generation run.
phase ends when the level set is full, which for this example occurs at generation 16.
Figure 3-25a shows the level set at this point and clearly there are improvements
to be made with regard to both spacing and spread. This is the purpose of the
second phase. The algorithm continues to search for new level set solutions, still
guided toward unexplored regions of the design space due to the crowding operator.
Solutions that improve the spread and spacing are substituted into the existing level
set and the diversity improves over time. By generation 50, the level set has a nice
distribution (Figure 3-25b).
Figure 3-26 shows a pair of metrics as the LSGA iterates. The left panel shows the
diversity metric D for each generation. The first member of the level set is extracted
in generation 4, hence D = 0 until then. At generation 16 we can see the point
at which LSGA switches from the first phase (level set finding) to the second (level
set improvement). Maximum spread is achieved at generation 18, after which the
spacing continues to improve. The right panel of Figure 3-26 shows the minimum di
for each generation. In other words, it shows the distance between the two points in
the level set that are closest together. After reaching a minimum by the end of the
107
first phase (generation 16) due to points that are nearly overlapping, the minimum
spacing begins to increase during the second phase. By the end of 50 generations di
has stabilized, indicating a population with mostly even spread.
Minimium di versus Generation
Diversity versus Generation
1.4
0.7
D
Dspace
Dspread
1.2
0.6
0.5
Minimum di
Diversity Metric
1
0.8
0.6
0.4
0.3
0.4
0.2
0.2
0.1
0
0
5
10
15
20
25
30
Generation
35
40
45
50
(a)
0
0
5
10
15
20
25
30
Generation
35
40
45
50
(b)
Figure 3-26: (a) Diversity metric and (b) minimum pairwise distance metric over 50
generations of LSGA.
One could reasonably ask the question, why implement the second phase of LSGA?
Why not simply accept all new candidates into the level set and allow the set to grow,
rather than focusing on improving the diversity? Figure 3-27 provides an answer.
The left panel shows the current LSGA implementation in which the level set is
limited to a maximum size and the diversity enhancement operator helps ensure a
good distribution of points. The right panel shows the proposed alternative in which
there is no restriction on the level set size and no explicit diversity operator. Clearly
the alternative algorithm has detected more points in the level set. However, the
diversity metric is actually worse. This is because many of the points are essentially
overlapping and the larger population fails to produce additional information about
the underlying shape of the level curve. The computational expense is the same in
both cases and up until generation 16 (when the level set becomes full), the results
are identical. The original LSGA devotes generations 17 through 50 to improving the
diversity of the level set, while the alternative version devotes those generations to
adding members. Finally, note that the metrics in Figure 3-26 provide a convenient
108
means of monitoring the state of algorithm convergence. We will use such plots to
analyze certain LSGA runs presented later.
Gen 50, NLS = 40, D = 0.28635
Gen 50, NLS = 171, D = 0.60458
1
1
Level Set
0.9
0.8
0.8
0.7
0.7
0.6
0.6
x2
x2
Level Set
0.9
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
0.2
0.4
0.6
0.8
0
0
1
0.2
0.4
x1
(a) Original LSGA.
0.6
0.8
1
x1
(b) Modified LSGA (no diversity operator).
Figure 3-27: LSGA results after 50 generations with (a) diversity enhancement (maximum level set size is NLS = 40) and (b) no diversity enhancement (no maximum
level set size).
We now explore the performance of LSGA under a few other test functions that
exhibit more complicated behavior than f1 . Consider,
1
1
f2 (x) = x31 + 3x1 x2 + x22 + 12x2 + 515
3
2
− 10 ≤ x1 ≤ 20
(3.38)
− 60 ≤ x2 ≤ 10
where we seek the level f2 (x) = 500. LSGA is run for 50 generations using the same
parameters as in the first test case except for the tolerance, which is set to τ = 5 (i.e.
1% of the level). The original function and the final level set are shown in Figure 3-28.
The algorithm performs well despite the complex shape of the level curves.
109
Objective Function f2 (x)
10
Gen 50, NLS = 40, D = 0.1984
10
500
10
Level Set
00
00
30
00
15
0
00
20
50
0
−10
0
00
25
500
−10
00
10
0
50
2
x
x2
0
50
500
100
−20
0
150
−20
0
−30
−30
0
100
1500
0
−40
0
0
10
−50
0
15
500
0
50
00
25
0
00
0
−60
−10
−5
0
5
x
10
0
20
−50
−40
15
20
−60
−10
−5
0
5
x
1
10
15
20
1
(a)
(b)
Figure 3-28: (a) Test function f2 (x) and (b) LSGA results after 50 generations.
Next consider a new test function,
f3 (x) = x21 + 5x22 − 2x31 x2 + x41 + 2x42
− 5 ≤ x1 ≤ 5
(3.39)
− 5 ≤ x2 ≤ 5
where we seek the level f3 (x) = 5. LSGA is again run for 50 generations using the
same set of optimization parameters. In this case we again adopt a tolerance of 1% of
the level (τ = 0.05). The original function and final level set are shown in Figure 3-29.
Again, LSGA performs well, this time despite the presence of two completely separate
level curves in diagonally opposite regions of the design space.
Finally, consider one last test function, this time defined over R3 as
f4 (x) = (x1 − 1)2 + (x2 + 1)2 − x1 x2 + 4x1 x22 − x3
− 1 ≤ x1 ≤ 1
− 1 ≤ x2 ≤ 1
− 6 ≤ x3 ≤ 2
110
(3.40)
Objective Function f3 (x)
Gen 50, NLS = 40, D = 0.3172
5
5
10
25
10
20
4
10
Level Set
5
15
4
5
5
3
3
1
1
1
2
10
2
2
5
x
1
0
1
5
x2
1
−1
10
1
−1
0
−2
−2
1
−3
1
1
−3
5
5
−4
5
−5
−5
15
10
10
20
25
10
−4
−3
−2
−1
0
x1
1
2
3
4
5
−4
−5
−5
−4
−3
−2
−1
(a)
0
x1
1
2
3
4
5
(b)
Figure 3-29: (a) Test function f3 (x) and (b) LSGA results after 50 generations.
where we seek the level f4 (x) = 5. Because f4 is a scalar function in three dimensions,
it is difficult to plot all level surfaces at once. Instead, we plot only the level surface
of interest by setting Equation 3.40 equal to 5 and solving for x3 ,
5 = (x1 − 1)2 + (x2 + 1)2 − x1 x2 + 4x1 x22 − x3
⇒ x3 = (x1 − 1)2 + (x2 + 1)2 − x1 x2 + 4x1 x22 − 5 ≡ z(x1 , x2 )
Figure 3-30a shows the level surface z(x1 , x2 ) corresponding to f4 (x) = 5. LSGA was
run for 50 generations using the same set of optimization parameters, except this time
the population has been increased to 100 to help with the higher dimensionality of the
problem. The tolerance remains at 1% (τ = 0.05). Results are shown in Figure 3-30b,
demonstrating LSGA’s ability to succeed in higher dimensional problems. Plotting
the diversity metric in Figure 3-31 confirms good convergence after 50 generations.
3.4.5
Star Camera Requirement Vectors
The previous section described the level set genetic algorithm (LSGA) in detail. This
algorithm is the centerpiece of the model-based requirement definition methodology.
111
(a)
(b)
Figure 3-30: (a) Test function level surface f4 (x) = 5 and (b) LSGA results after 50
generations.
Diversity versus Generation
1.4
D
Dspace
Dspread
1.2
Diversity Metric
1
0.8
0.6
0.4
0.2
0
0
5
10
15
20
25
30
Generation
35
40
45
50
Figure 3-31: Diversity metric for f4 (x) over 50 generations of LSGA.
Once the mapping between requirements and system performance has been captured
by an appropriate model (Equation 3.2), LSGA allows the user to identify a diverse
collection requirement vectors (i.e. the “level set”) all of which satisfy the desired
performance objective.
We now apply this approach to the star camera sample problem introduced pre112
viously. Recall from Section 3.2 that the performance figure of merit is N0.1 or the
number of stars in the field of view with a centroid accuracy of 0.1 pixels or better.
Our instrument performance requirement is that N0.1 ≥ 20 stars. In Section 3.3 we
developed an end-to-end model that calculates N0.1 as a function of the focal length f
and aperture diameter d. We are now well positioned to apply LSGA and generate a
level set of requirement vectors R that define acceptable values of f and d.
LSGA was run over 50 generations with a population size of 40. In this case
we seek 10 sets of requirements, so NLS = 10. We use a 1% convergence tolerance
(τ = 0.2) and otherwise the LSGA parameters are the same as in previous examples.
Figure 3-32 shows the LSGA output overlaid on the full factorial results computed
previously. Each point represents a set of requirements on f and d that will allow
the system to meet the performance objective N0.1 ≤ 20. The values are listed in
Table 3.5.
Figure of Merit, N0.1
50
Level Set
70
80
40
45
30
Aperture Diameter, d [mm]
60
50
40
20
35
40
10
30
30
20
10
25
20
50
60
70
80
Focal Length, f [mm]
90
100
Figure 3-32: LSGA results for the star camera sample problem.
Finally, Figure 3-33 shows the LSGA diversity metrics over the course of the
optimization run. Evidently, the algorithm was well converged after roughly 30 generations. The maximum spread occurred even earlier. Note that the spread metric
113
Table 3.5: Set of requirement vectors, all of which satisfy the performance requirement
N0.1 = 20.
Vector
r1
r2
r3
r4
r5
r6
r7
r8
r9
r10
f
50.0
54.3
60.4
66.5
71.7
76.6
82.7
87.4
93.5
100.0
d
25.7
29.1
30.7
33.0
35.4
37.9
40.0
41.9
44.2
47.0
(green line in Figure 3-33) plateaus at a value of approximately 0.3. This is because
once the end points at f = 50 mm and f = 100 mm are reached, the vertical (d axis)
extent of spread can grow no larger. Therefore the fact that Dspread reaches a plateau
is not a concern, especially considering that the final level set spans the range of focal
lengths.
Diversity versus Generation
1.4
D
Dspace
Dspread
1.2
Diversity Metric
1
0.8
0.6
0.4
0.2
0
0
5
10
15
20
25
30
Generation
35
40
45
50
Figure 3-33: Diversity metrics for the star camera sample problem.
114
3.4.6
Summary
Step 3 of the methodology is devoted to the core task of generating requirements.
We cast this process as an optimization problem in which we seek multiple sets
of top-level engineering requirements (i.e. multiple “requirement vectors”) that all
satisfy the science performance objective. To solve the problem, we introduced a new
genetic algorithm termed LSGA. Through a series of test functions and the more
realistic star camera sample problem, LSGA proved effective in producing a diverse
set of requirement vectors. The remaining task is now to select a single requirement
vector out of the group. This is the purpose of Step 4.
3.5
Step 4: Select a Requirement Vector
Having derived the set of requirement vectors in Step 3, we now turn to the task of
intelligently selecting the final requirement vector. Once chosen, this set of top-level
engineering requirements will serve as the basis for design work at the subsystem level
using domain-specific tools.
Requirement selection can occur according to a variety of user-defined criteria.
Cost would be a natural choice if appropriate cost-estimating relationships were available. Risk or technical maturity would be other sensible options. An alternative
metric is to select the most robust requirement set, i.e. the set of requirements for
which local perturbations have the smallest impact on system performance. This set
of requirements will be the most resilient to changes later in the program life cycle
and therefore can be viewed as the choice that minimizes risk to the project.
3.5.1
Local Sensitivity Metrics
In order to identify the most robust requirement vector out of the collection R =
{r1 , . . . , rn }, we require a local sensitivity metric with which we can rank each
ri . Once the ranking is complete, we can then simply choose the r with the lowest
sensitivity. This task falls under broad field of sensitivity analysis, for which the
115
literature is extensive. The general objective in such a setting is to exercise a model
y = f (x) = f (x1 , . . . , xn ) to determine the variation in output y given uncertainty
in the inputs xi . One of the simplest and most common measures of local sensitivity
is the partial derivative of the output with respect to a given input,
∂y Si =
∂xi x0
(3.41)
where in this case the derivative has been evaluated at a point x0 . The Si can be
calculated directly or using a numerical technique such as finite difference approximation. This sensitivity measure can be normalized as,
Si0
(xi )0 ∂y =
·
y0
∂xi x0
(3.42)
where y0 = f (x0 ) and (xi )0 is the ith component of x0 . This form casts the sensitivity
in relative terms and may be more useful when comparing effects across inputs with
different units. Alternatively, one could modify Equation 3.42 as proposed by Saltelli
et al. [117] to produce the “sigma-normalized” sensitivity measure
Siσ
σxi ∂y =
·
σy ∂xi x0
(3.43)
where σxi and σy are the standard deviations of the input and output, respectively.
Finally, de Weck and Jones present a metric for calculating the overall local uncertainty as the sum of (normalized) uncertainties from the individual inputs [24]
n X
∂y
δx
S=
i
∂xi .
(3.44)
i=1
In this case, the δxi are analogous to σxi and serve as uncertainty bounds on the input
parameters (i.e. xi = xi ± δxi ). A similar metric was employed by Gutierrez [58] and
Uebelhart [139].
116
3.5.2
Derivative-Free Local Sensitivity Metric
The sensitivity metrics above could be useful for our purposes in the sense that
they are local.8 However, they all rely on derivatives. For many applications this
would not be a problem, but in our case we have taken pains thus far to avoid
restrictions on the type of models that can be used in the methodology. Recall that
we pursued heuristic optimization in Step 3 to accommodate models that may not
have well-defined derivatives. Therefore we seek a local yet derivative-free approach
to sensitivity analysis.
Instead of probing local sensitivities using derivatives, we adopt a Monte Carlo
approach. For the sake of concreteness, consider the sensitivity in the neighborhood
of x0 ≡ r2 = [54.3 29.1] from the star camera example. We create a set of m samples
around x0 by drawing from a normal distribution whose standard deviation is related
to the average pairwise spacing between points. Each sample is given by,
(m)
xm = [x1
(m)
x2 ]
(3.45)
where
(m)
∼ N [(x1 )0 , σ1 ]
(m)
∼ N [(x2 )0 , σ2 ].
x1
x2
Here N [µ, σ] indicates a value drawn from a normal distribution with mean µ and
standard deviation σ. We define the values of σ1 and σ2 in the following way:
LB
σ1 = σ 0 · xUB
1 − x1
LB
σ2 = σ 0 · xUB
2 − x2
(3.46)
where
σ 0 = d/3.
8
There exist global sensitivity metrics that we discuss later, however they are not relevant in this
setting in which the goal is to distinguish points in one part of the requirement/design space from
those in other parts.
117
Recall that d is the mean of the minimum pairwise distance values di and therefore
represents the average spacing between points in the final level set. The quantity σ 0 is
calculated in terms of normalized units (i.e. in the x0 ∈ [0, 1] space, see Equation 3.17)
and is calculated such that the 3σ radius of the cloud of Monte Carlo samples corresponds roughly to the distance between solutions in the level set. Because this
sensitivity analysis uses raw (non-normalized) variables, we must scale σ 0 along each
axis using Equation 3.46.
Figure of Merit, N0.1
50
Level Set
Monte Carlo Samples
70
80
40
45
60
Aperture Diameter, d [mm]
30
40 50
35
20
40
10
30
30
10
20
25
20
50
60
70
80
Focal Length, f [mm]
90
100
Figure 3-34: 500 Monte Carlo samples distributed in the neighborhood of x0 =
[54.3 29.1] to be used for sensitivity analysis.
Figure 3-34 shows 500 realizations generated according to Equation 3.45. Clearly
the chosen values for σ1 and σ2 do an appropriate job of sampling the local neighborhood around the point of interest. For each sample, we calculate the figure of merit
y = N0.1 . Figure 3-35 visualizes the results using scatter plots, which are an intuitively appealing way of understanding the relative sensitivities for each parameter.
Standardized regression coefficients (SRCs) provide a convenient and informative
means of quantifying the extent of sensitivity seen in the scatter plots. The first in
118
Sensitivity to Aperture Diameter (d)
30
30
28
28
26
26
Figure of Merit, N0.1
Figure of Merit, N0.1
Sensitivity to Focal Length (f)
24
22
20
24
22
20
18
18
16
16
14
14
48
50
52
54
56
Focal Length, f [mm]
58
60
26
27
28
29
30
Aperture Diameter, d [mm]
31
32
33
Figure 3-35: Scatter plots of y = N0.1 versus x1 = f (left) and x2 = d (right) for
500 Monte Carlo samples in the neighborhood of x0 = [54.3 29.1]. Red lines are
calculated using a multiple linear regression.
deriving them step is to compute a multiple linear regression of the form,
yi = b0 +
k
X
(i)
bj x j
(3.47)
j=1
where bj are the regression coefficients, k is the number of parameters (k = 2 in
our case), and yi is the figure of merit value corresponding to the ith sample xi =
i
h
(i)
(i)
x1 , . . . , xk . We package the m Monte Carlo samples and the resulting output
values into a matrix X and vector y,

(1)
1 x1
(1)
x2
···
(1)
xk




(2)
(2)
(2) 
 1 x1
x2
· · · xk 

X=
 ..
..
..
..  ,
..
.
.
.
.
. 


(m)
(m)
(m)
1 x1
x2
· · · xk

y1

 
 
 y2 

y=
 ..  .
 . 
 
ym
(3.48)
The regression can then be carried out using ordinary least squares to minimize the
sum of squared errors between the Monte Carlo outputs y and the output of the
119
model in Equation 3.47. This is done using the Moore-Penrose pseudoinverse
 
b̂
 0
 
b̂ 
b =  1  = (X T X)−1 X T y.
b
 .. 
.
 
b̂k
Importantly, the estimated regression coefficients b̂i are the slopes of the trend lines
that relate output variation to input variation. For the current example, b1 = 0.10
and b2 = −23.29, and the associated trend lines are shown in red in Figure 3-35.
The slopes b̂i are close to the sensitivity metric we seek. They are a derivative-free
analog to Equation 3.41 and provide a means of calculating the output variation in a
region around a point of interest. The final SRCs are obtained by normalizing each
b̂i by the input and output standard deviations to give,
β̂i =
σxi
· b̂i .
σy
(3.49)
The SRCs (β̂i ) have several properties that make them a very useful local sensitivity metric. First, Saltelli [118] show that for linear models and as the number of
Monte Carlo samples goes to infinity, the β̂i are equivalent to the sigma-normalized
derivatives shown in Equation 3.43. Thus the SRCs serve as an indication of the
main effect of a parameter. Each β̂i measures the relative change in output given a
change in the input parameter xi . Regardless of the linearity in our analysis model,
P
the quantity R2 = ki=0 (β̂i )2 ∈ [0, 1] indicates the fraction of variation in the output
explained by the regression. Thus 1 − R2 provides a measure of the variation that
we are unable to trace to specific input parameters. Possible sources of unexplained
variation include model nonlinearity, intrinsic variability in the model (e.g. if it uses
an inner Monte Carlo loop like our star camera simulation), and interaction effects
between parameters. Finally, by examining the individual (β̂i )2 terms we can quantify
proportion of variance allocated to each input parameter.
SRCs are therefore useful in decomposing the variance among input parameters
120
and identifying additional variance intrinsic to the analysis model. For the case of
Figure 3-35 we have,
β̂1 = 0.13
β̂2 = −0.93
(β̂1 )2 = 0.02
(β̂2 )2 = 0.86
R2 = 0.88.
Interpreting these results, the linear regression accounts for roughly 88% of the variance in the output. Of this variance, the vast majority is due to the second parameter
(aperture diameter, d). As was the case with the scatter plots, this result is intuitively appealing given that the full factorial contour plot visible in Figure 3-34 shows
a much higher sensitivity along the d axis compared to the f axis.
The quantities (β̂i )2 and R2 are normalized by the total output variance σy2 and
therefore always reside in the interval [0,1]. If we multiply each quantity by σy2
we obtain sensitivity metrics that are more useful for comparing variances between
members of the level set. Specifically, let
2
Si = β̂i σy
and
S=
=
k
X
(3.50)
Si
i=1
k X
(3.51)
β̂i σy
2
.
i=1
Here Si are the “absolute” variance contributions from each parameter xi and S is
a measure of the total variance in a given solution due all of the parameters. In
this sense, we have a nice analogy between the relative and absolute variance terms
121
defined thus far:
σy2 ←→ R2
2
Si ←→ β̂i
S ←→ 1 − R2 .
Another interesting fact is that we can use the definition of β̂i in Equation 3.49
re-write S as
S=
k X
β̂i σy
2
i=1
=
⇒S=
k X
σx
i=1
k X
2
i
σy
b̂i · σy
b̂i · σxi
2
i=1
which—recalling the equivalence between the slopes b̂i and partial derivatives ∂y/∂xi —
is analogous to de Weck’s total uncertainty metric in Equation 3.44.
The quantities Si in Equation 3.51) are our final sensitivity metrics. They represent the variance contributions (β̂i )2 scaled by the total output variance σy2 .
3.5.3
Choosing A Robust Requirement Set
We now have the tools needed to intelligently select a final requirement set. Figure 336 shows the Si values for each candidate solution in the level set. The analysis shows
that overall, r10 is the least sensitive requirement vector. This matches our intuition
from looking at the full factorial results, as the contours tend to spread out for
increasing focal lengths, indicating lower slopes and reduced sensitivity. We therefore
select r10 as our top-level requirement set and can proceed with more detailed levels
of design.
122
Sensitivity Analysis
9
σy2
S1 (f )
S2 (d)
8
Relative Sensitivity
7
6
5
4
3
2
1
0
1
2
3
4
5
6
7
Requirement Vector
8
9
10
Figure 3-36: Star camera sensitivity analysis results.
123
124
Chapter 4
REXIS
The previous chapter described model-based error budgeting conceptually and with
simple examples. We now turn to a case study that walks through the application of
this methodology to an actual instrument system. This chapter is devoted to REXIS,
an X-ray imaging spectrometer that will fly on an upcoming NASA asteroid sample
return mission.
4.1
Background
The Regolith X-ray Imaging Spectrometer (REXIS) is one of five instruments aboard
the Origins, Spectral Interpretation, Resource Identification, and Security–Regolith
Explorer (OSIRIS-REx) mission. Scheduled for launch in September 2016 and developed under NASA’s New Frontiers program, OSIRIS-REx will travel to the nearEarth carbonaceous asteroid 101955 Bennu, conduct a multi-spectral characterization using REXIS and the other onboard remote sensing instruments, and collect
approximately 60 g of asteroid regolith for return to Earth [79]. The OSIRIS-REx
spacecraft is a cube measuring roughly 2 meters on a side, and REXIS is mounted to
the asteroid-facing instrument deck as shown in Figure 4-1. A solar X-ray monitor
(SXM) mounted to the Sun-facing side provides a real time measurement of the solar
spectrum, which is necessary for accurate calibration of the asteroid spectral data.
REXIS functions as both a spectrometer and imager. As a spectrometer, REXIS
125
Figure 4-1: Location of REXIS and the SXM on the OSIRIS-REx spacecraft. REXIS
is on the instrument deck (left), which faces the asteroid during observations. The
SXM (right) is on a bracket pointed in the +x direction, which is toward the Sun
during observations.
will collect global spectral measurements on bulk composition, revealing the mean
elemental abundances and allowing Bennu to be classified among major meteorite
groups. As an imager, REXIS will map elemental abundances on 50 m scales at the
asteroid surface, which provides context to the regolith sample and complements visible and infrared maps produced by the other instruments.1 REXIS is a collaboration
between the MIT Space Systems Laboratory (SSL), the MIT Kavli Institute for Astrophysics and Space Research (MKI), the MIT Department of Earth, Atmospheric,
and Planetary Sciences (EAPS), and Harvard-Smithsonian Center for Astrophysics
(CfA), with additional participation from MIT Lincoln Laboratory (LL) and Aurora
Flight Sciences (AFS).
The primary REXIS science objective is to determine Bennu’s surface elemental
composition and classify the asteroid according to established meteorite groupings.
1
As a student collaboration experiment, the REXIS measurements are not part of the formal
OSIRIS-REx science mission goals.
126
Visible and infrared reflectance spectra from ground-based measurements indicate
that Bennu is most likely analogous to the so-called CI or CM carbonaceous chondrites [17]. Chondrites are scientifically important because they provide a window
into the chemical makeup and physical processes prevalent in the early Solar System.
For example, based on a bulk composition matching the Sun’s photosphere, the presence of interstellar and circumstellar grains, and evidence of thermal processing over a
range of temperatures, it is likely that chondrites originated in the solar nebula—the
primordial disk of gas and dust out of which the planets formed [121]. Chondrites
therefore provide valuable clues for understanding how our solar system developed
and how planet formation may occur in the other solar systems being found during
the current wave of exoplanet discoveries.
A simulated Bennu spectrum is shown in Figure 4-2. REXIS determines Bennu’s
surface composition by measuring the strength of X-ray fluorescence lines corresponding to four elements of interest: iron (Fe), magnesium (Mg), sulfur (S) and silicon
(Si). Through a sequence of processing steps, the measured line strengths are converted to a set of weight ratios of iron, magnesium, and sulfur to silicon (denoted here
as Fe/Si, Mg/Si, and S/Si). Comparing the measured weight ratios with those in a
database of meteorite specimens allows Bennu to be classified according to groupings
and subtypes established by the scientific community, including the important chondrite group discussed above. In order for this characterization to be feasible, however,
REXIS must measure the spectrum with sufficient accuracy. We will return to this
important point in later sections.
4.1.1
Instrument Overview
REXIS consists of two major elements: a relatively large (5.5 kg, 31 × 15 × 15 cm3 )
asteroid-facing imaging spectrometer and a much smaller (500 g, 5 × 5 × 5 cm3 )
Sun-facing solar X-ray monitor (SXM). Both are shown in Figure 4-3.
The spectrometer is essentially a soft X-ray (0.3–10.0 keV) telescope. In contrast
to the large and complex grazing incidence focusing optics used by other X-ray telescopes such as Chandra [145], XMM-Newton [3], Swift XRT [13], and NuSTAR [75],
127
Asteroid Spectrum for 4 MK Sun
4
10
Fe−L
Mg−K
2
photons/s/keV
10
Si−K
S−K
0
10
−2
10
−4
10
1
2
3
4
Energy [keV]
5
6
7
Figure 4-2: Simulated X-ray fluorescence spectrum for the asteroid Bennu (data
courtesy of Branden Allen).
REXIS uses a much simpler non-focusing coded aperture imaging approach [15, 131].
At the top of the telescope, X-rays impinge on a flat mask made up of random
open/closed pixels within a circular aperture. Underneath the mask lies the focal
plane, consisting of four charged coupled devices (CCDs) arranged in a 2 × 2 tiled
array (Figure 4-4). The X-ray patten that falls on the focal plane is a convolution of
the source illumination and the mask pixel pattern. This pattern is recorded by the
CCDs and can be processed later using knowledge of the mask pattern to create a
reconstructed image of the source.
The CCDs are model CCID-41 devices manufactured by MIT Lincoln Laboratory [115]. Each detector has 24 µm × 24 µm pixels arranged in an active area of
1024 rows by 1024 columns.2 Adjacent to the active area is a non-imaging frame store
region used to hold charge during serial read-out. The CCID-41 is derived from the
CCID-17 used on the Chandra AXAF CCD Imaging Spectrometer (ACIS) [12], and
2
Each CCID-41 actually has 1026 rows but the bottom two rows are only to provide a buffer for
aligning the frame store cover and are not recommended for imaging [11].
128
Figure 4-3: REXIS spectrometer and solar X-ray monitor (SXM).
was previously flown on the Astro-E2/Suzaku3 mission in the X-ray Imaging Spectrometer (XIS) instrument [147, 76]. The fundamental enhancement of the CCID-41s
was the addition of a charge injection register at the top of the imager, which is used
to mitigate the effects of charge transfer inefficiency (CTI) caused by displacement
damage from radiation (energetic photons) [11]. The CCID-41s are back-illuminated
for improved quantum efficiency (QE) and reduced dark current.
To achieve sufficient spectral resolution, the focal plane temperature must be
maintained at approximately -60 ◦ Cor less. This is accomplished through a combination of passive cooling and thermal isolation. The detector assembly is connected to
a side-mounted radiator via a copper thermal strap, which conducts heat away from
the focal plane and radiates it to deep space. Two sets of standoffs—one set made
from low conductivity Torlon and another made of titanium—provide thermal isola3
Suzaku was known as Astro-E2 prior to launch in 2005 and some publications use the original
name.
129
Figure 4-4: Geometry of the REXIS focal plane with four CCID-41s in a 2 × 2
configuration. The gray area is the exposed imaging region while the blue area is
masked by a metal cover plate.
tion between the detectors and the relatively warm electronics box (approximately
50 ◦ C).
The SXM is a small spectrometer in its own right, mounted to the Sun-facing
gusset of the spacecraft (see Figure 4-1) to measure the solar spectrum. The solar
measurement is done by an Amptek XR-100SDD silicon drift diode (SDD). Essentially
a one-pixel X-ray spectrometer, the SDD features a silicon detector and thermoelectric
cooler (TEC) inside a hermetically sealed package. A feedback circuit is used to
modulate the TEC, keeping the detector at a low temperature and thereby improving
spectral resolution.
In summary, REXIS consists of an asteroid-facing imaging spectrometer and a
smaller Sun-facing solar X-ray monitor (SXM). The main imaging spectrometer uses
an array of four back-illuminated CCDs that are passively cooled to achieve sufficient
spectral resolution in the soft X-ray band (0.3–10 keV). Table 4.1 summarizes the
REXIS baseline design.
130
Table 4.1: Summary of the main REXIS spectrometer design parameters.
Design parameter
Energy passband
Focal length
Mask coded fraction
Mask throughput
Focal plane
CCD type
CCD active pixels
CCD pixel size
CCD operating temperature
Integration time
Observation period
Mass
4.1.2
Value
0.3 – 10 keV
20 cm
50 %
42 %
Four CCDs in a 2 × 2 array
CCID-41 (back-illuminated)
1024 rows × 1024 columns
24 µm
-60 C
4s
423 hours
5.5 kg (not to exceed)
Concept of Operations
After a roughly two-year cruise phase, OSIRIS-REx will rendezvous with Bennu, conduct a series of initial surveys, then enter a polar orbit along the asteroid’s terminator
at mean altitude of 730 km above the surface. REXIS will collect data from this vantage point, first for 11 hours a day for a period of 21 days, then for 12 hours a day
for a period of 16 days. The expected cumulative observation time is therefore 423
hours.
Figure 4-5 depicts a notional view of REXIS science observations at Bennu. The
Sun emits X-rays that impinge on the asteroid surface. Some of the incoming Xray photons interact with the asteroid regolith at an atomic level and cause the
emission of secondary X-ray fluorescence (XRF) photons at energies corresponding
to the elements that are present. The XRF photons are sampled by the REXIS focal
plane, along with photons that have been scattered off the astroid surface and photons
from the cosmic X-ray background (CXB). Simultaneously, the SXM conducts a direct
solar measurement to characterize the input spectrum to Bennu.
131
Figure 4-5: Notional view of REXIS operations from a terminator orbit around Bennu
(not to scale). Solar X-rays cause fluorescence of the asteroid regolith, which is
detected by REXIS along with the CXB. The SXM simultaneously measures the
solar X-ray spectrum.
4.1.3
Overview of CCD X-Ray Spectroscopy
Each X-ray photon that interacts with the CCD generates a cloud of charge in the
depletion region of the detector. The energy needed to liberate an electron-hole pair
from silicon is approximately w = 3.7 eV/electron, therefore the incident photon’s
energy is encoded by the amount of charge created by each interaction event:
Ne =
E
w
(4.1)
where Ne is the number of electrons liberated and E is the energy of the incident
X-ray. This direct mapping between photon energy and generated charge is the basic
principle behind CCD X-ray spectroscopy. Of course, it is based on the assumption
that each event corresponds to a single photon. Multiple photons per event, a condition known as pileup, is an undesirable state that destroys the event energy mapping.
132
Pileup is avoided by ensuring that the incoming photon rate is low enough that multiple events per pixel are all but impossible. In practice this imposes an upper limit
on the CCD integration time, which for REXIS is set at 4 seconds.
Once a charge cloud is formed in the CCD, the electrons that make up the cloud are
collected at pixel gates and ultimately digitized into analog-to-digital units (ADU).
Data volume limitations prevent the transmission of the entire pixel array to Earth.
Instead, an event list is compiled that contains the number of ADUs or pulse height
of each event, along with the event’s row/column position on the CCD and its event
grade, a code used to describe the splitting of events across multiple pixels.
An X-ray spectrum is simply a histogram that sorts events by ADU, or equivalently after calibration, by energy. Figure 4-6 shows a sample CCD spectrum for
monochromatic 5.898 keV photons. The primary photopeak has a non-zero width
due to Fano noise, electronic read noise, dark current, and other effects discussed in
Section 4.3.1. The photopeak’s full-width half-maximum (FWHM) at a given energy
is the spectral resolution and is a measure of the instrument’s sensitivity. In addition
to the primary photopeak, the spectrum has a Si fluorescence peak, a Si K-escape
peak, and a noisy continuum. All of these features affect the ability of REXIS to
accurately distinguish elemental lines in the asteroid spectrum.
The energy scale of the spectrum is determined by the gain (eV per ADU) and
offset (eV at zero ADU) of the instrument. Equation 4.1 indicates that the gain
is theoretically a constant number that captures the linear conversion from pulse
height to energy. In practice, however, the gain is a function of temperature, the
radiation environment, and electronics design. As such, REXIS has several internal
55
Fe calibration sources to measure gain and offset during the mission.
55
Fe is a
standard reference isotope in X-ray spectroscopy that emits Mn-Kα and Mn-Kβ lines
at 5.898 keV and 6.490 keV, respectively. The REXIS sources illuminate a subset
of detector pixels and are sized to produce a total of 0.42 counts per second during
observations of Bennu [9]. Because the energies of the lines are well known and
constant, the measured
55
Fe pulse heights can be used to solve for the gain and
offset. A similar approach was used by the X-Ray Spectrometer (XRS) on the Near133
30
25
Counts
20
15
10
5
0
1
2
3
4
Energy [keV]
5
6
7
Figure 4-6: Sample spectrum showing the response of a CCD to monochromatic
photons at 5.898 keV.
Earth Asteroid Rendezvous (NEAR) mission [84] and the Suzaku XIS instrument [91].
Table 4.2 summarizes the relevant line energies for REXIS.
4.2
User Objectives
The first step in model-based requirement definition is to define the user objectives
in quantitative terms. Recall from Chapter 3 that this involves describing the data
product, the science figure of merit, and the performance requirement. The REXIS
data product is the globally averaged spectrum of Bennu in units of counts per energy
bin (eV). Figure 4-7 shows two simulated Bennu spectra. These spectra differ in terms
of their spectral resolution, which was defined in the previous section. The left figure
has better spectral resolution (i.e. smaller FWHM) and clearly the fluorescence lines
are more easily distinguishable from each other. Spectral resolution will therefore
serve as our figure of merit. It captures, in a single number, the performance of the
instrument and the quality of the spectral measurement.
134
Table 4.2: X-ray lines measured by REXIS. The Fe-L, Mg-K, Si-K, and S-K complexes
comprise the asteroid observation. The Mn-Kα and Mn-Kβ complexes originate from
the internal 55 Fe calibration sources. The intensity of each line is listed as a fraction
relative to the total from all lines in a given energy shell. Effective energies for each
complex are the weighted averages of lines listed. All data are from [137] except Mg
relative intensity data (†), which are from [77].
Complex Line
Fe-L
Lα1,2
Lβ1
Mg-K
Kα1,2
Kβ1
Si-K
Kα1
Kα2
Kβ1
S-K
Kα1
Kα2
Kβ1
Mn-Kα
Kα1
Kα2
Mn-Kβ
Kβ1
Energy [keV]
0.7050
0.7185
1.2536
1.3022
1.7400
1.7394
1.8359
2.3078
2.3066
2.4640
5.8988
5.8877
6.4905
Intensity
0.63
0.37
0.99†
0.01†
0.66
0.33
0.01
0.65
0.32
0.03
0.60
0.30
0.10
Simulated Spectrum (-100 C)
6
1.7408
2.3121
5.8951
6.4905
Simulated Spectrum (-20 C)
10
Fe−L
Mg−K
Si−K
S−K
Mn−K
5
10
Fe−L
Mg−K
Si−K
S−K
Mn−K
5
10
4
4
10
Counts
10
Counts
1.2511
6
10
3
10
2
3
10
2
10
10
1
1
10
10
0
10
Effective Energy [keV]
0.7100
0
1
2
3
4
Energy [keV]
5
6
10
7
1
2
3
4
Energy [keV]
5
6
7
Figure 4-7: Simulated REXIS spectra for 1 hour of observation at −100◦ C (left)
and −20◦ C (right). The spectral resolution is better (smaller FWHM) at lower
temperature. A gaussian fit to the Mn-Kα line (5.9 keV) is shown as a solid black
line.
The final quantity to define is the performance requirement, i.e. the threshold
spectral resolution REXIS must achieve in order to be successful. Based on the
135
required separation between lines to allow adequate identification and analysis, the
REXIS science team at HCO derived a required spectral resolution at each line of
interest. These values are shown in Table 4.3. With the user objectives quantified in
terms of data product (asteroid spectrum), figure of merit (spectral resolution), and
performance requirement (Table 4.3), we can proceed to the next step of model-based
requirement definition and create an end-to-end model of the REXIS spectrometer.
Table 4.3: REXIS spectral resolution requirements at each elemental line of interest.
Line
Energy [keV]
Fe-L
0.7100
Mg-K
1.2511
Si-K
1.7508
S-K
2.3121
4.3
Spectral Resolution (FWHM) [keV]
0.198
0.204
0.210
0.227
End-to-End Model
The goal of the second step in model-based requirement definition is to create an
end-to-end model that captures the mapping between engineering parameters and
the science figure of merit. For REXIS, this means that the output of our end-to-end
model will be the spectral resolution. As for inputs, Bautz and Nousek [6] present
the following equation for the spectral resolution σ of a CCD:
σ=
p
(w · σr )2 + (F · w · E)
(4.2)
where w is the electron-hole pair libration energy for silicon, σr is the read noise, F is
the Fano factor, and E is the energy of interest. Naively, we could use Equation 4.2 as
our model and be finished. The authors, however, point out that this equation only
holds for front-illuminated CCDs and that the results differ “considerably” for backilluminated devices such as the ones to be flown on REXIS. Furthermore, a simple
expression like Equation 4.2 typically fails to take into account important processes
such as internal calibration, data fitting, quantization, and other “real-world” artifacts
136
that affect the final spectral resolution. A new model was created to capture these
effects and its development is the subject of the rest of this section.
4.3.1
Measurement Pipeline
We begin development of the end-to-end model by documenting the REXIS measurement pipeline. This is a useful practice generally, as it allows both scientists
and engineers to visualize the data flows and the various parameters that affect data
quality along the measurement and processing chain. It also serves as a template for
implementing the simulation modules.
The REXIS pipeline is depicted below, beginning with solar and CXB X-rays and
ending with spectral resolution for the lines of interest. Flow charts in Figures 4-8
through 4-14 capture the transformation of energy or data (black lines) by various
hardware and software elements (blue boxes). A thick border indicates that the subprocesses within a given box are broken out and described in more detail in another
figure.
Using orange text and arrows, the flow charts also call out design parameters
that are considered to drive the instrument performance and therefore constitute
the requirement variables. Identifying and requirement variables involves engineering
judgement that a given design decision influences data quality, based on the knowledge
and experience of team members. These parameters will become inputs to the endto-end model and form the requirement space over which the optimization in Step 3
occurs.
A high-level view of the REXIS pipeline is shown in Figure 4-8. As discussed
previously, X-ray photons generated by the Sun impinge on the surface of Bennu
causing X-ray fluorescence of the regolith. In addition, X-ray photons and photons in
the ultraviolet (UV), visible, and infrared (IR) bands are reflected and scattered from
the asteroid. Incoming photons (fluoresced and reflected/scattered) enter the REXIS
aperture along with CXB X-ray photons. These are transformed by the REXIS
spectrometer into digital data (Figure 4-9) and then transmitted to the ground for
further processing into weight ratios (Figure 4-14).
137
Figure 4-8: Overview of the REXIS measurement pipeline. REXIS converts incoming
X-ray photons from Bennu and the Sun into a digitized X-ray event list and solar
histogram, both of which are sent to Earth. Processing on the ground produces
element weight ratios, the primary spectral mode data product. REXIS instrument
operation is shown in Figure 4-9 and ground processing is shown in Figure 4-14.
Spectrometer
The various measurement processes that occur within the REXIS spectrometer are
shown in Figure 4-9. The incoming photons are first attenuated by the mask (42% transmission) before hitting the CCDs below. Some of the incident photons strike the
inside of the spectrometer, fluorescing additional photons that may reach the CCD.
For this reason and for thermal reasons, surfaces of the aluminum structure that face
the CCD are coated with gold. Detecting aluminum in the asteroid is a secondary
science objective and the gold coating eliminates internal fluorescence at the Al-Kα
138
line (1.487 keV) that would otherwise corrupt the measurement. Most of the photons
fluorescing from the structure will instead be at Au-Mα (2.123 keV) and at a much
lower intensity.
Figure 4-9: Measurement processes within the REXIS spectrometer for converting
incoming X-ray photons to digital data for transmission to the ground. Details of the
CCDs are shown in Figure 4-10 and details of the electronics are shown in Figure 4-12.
Photons from the asteroid, CXB, internal fluorescence, and the
55
Fe sources all
strike the CCDs (Figure 4-10) and are converted into charge clouds in the detector
pixels. The
55
Fe calibration count rate is precisely tuned by selecting the source
activity level, geometry, and placement [9]. The pixels are read out by on-board
electronics (Figure 4-12), which digitize the signal and perform other operations to
assemble the event list for transmission to the ground via OSIRIS-REx .
139
CCDs
The CCDs are at the heart of REXIS. They are highly sensitive and perform the
intricate task of transforming X-ray photons into pixelated packets of electrical charge.
This process is shown in Figure 4-10. Experience has shown that contamination is
likely to develop on the surface of the CCDs. Contamination can arise from two main
sources: particulate matter that physically blocks light from reaching the CCD and
molecules that alter the CCD’s quantum efficiency curve.
Figure 4-10: Charge measurement process within the CCDs.
Photons next encounter the optical blocking filter (OBF), a 220 nm thick layer of
aluminum that has been deposited directly onto the CCD. As its name implies, the
140
purpose of the OBF is to prevent optical photons from being detected by the CCD
and corrupting the spectral measurement. In practice, some optical photons will still
pass through the OBF and cause a small increase in the background signal. Therefore
the level of optical attenuation is considered a requirement variable that governs the
level of “light leak” noise.
After passing through the OBF, photons interact with the active area of the
CCD, liberating electron hole-pairs in the silicon and generating charge clouds that
propagate toward the pixel gates for measurement (recall Section 4.1.3). Furthermore,
incoming photons above 1.74 keV (Si-Kα) may generate secondary fluorescing photons
inside the detector, which will create additional silicon-related spectral features in the
final measurement. The physics of charge cloud generation and propagation is highly
dependent on photon energy and the thicknesses and material properties of the layers
that comprise the CCD. These topics are discussed extensively in Section 4.3.2, which
describes the CCD model used in the end-to-end science simulation.
In addition to source and silicon-fluoresced photons, the CCD pixels will detect
dark current electrons arising from thermally induced electron hole-pair creation in
the bulk silicon [67]. Dark current is a function of both temperature and radiation
exposure, which are both treated here as requirement variables. The detector temperature is controlled through careful thermal design (e.g. the thermal strap, radiator,
insulation, coatings, etc.) and radiation exposure is a function of radiation cover
thickness.
After each exposure, charge from the active area is transferred row-by-row into
one of two frame store regions, then read out through serial registers using one of
four output nodes (A, B, C, D in Figure 4-11). The output nodes effectively divide
each CCD into four 256 pixel-wide strips that are each read out through a different
node. Every shift of a frame store row down into the serial registered is referred to as
a “parallel transfer”. Likewise, each sideways shift as pixels are read by the output
nodes is a “serial transfer”.
CCD charge transfer is often conceptualized as a bucket brigade, where charge
from one pixel is moved from one pixel (bucket) to the next on its way to the output
141
Figure 4-11: CCID-41 showing the imaging array (i.e. active area), frame store
regions, and quadrants mapping to output nodes (adapted from [11]). Charge follows
the path of the red arrows.
node. The detector’s charge transfer inefficiency (CTI) quantifies the amount of
charge left behind during transfers from one pixel to the next. As CTI increases over
time due to increased focal plane temperature or radiation damage, more charge is
left behind during the transfer process and spectral resolution suffers. While CTI is
a function of radiation exposure and temperature, it is treated here as a requirement
variable because it can be controlled through other means such as spaced-row charge
injection, demonstrated in the Suzaku mission ([108, 4]) and to be implemented on
REXIS.
Finally, read noise is introduced into the measurement by the output node. It is
primarily due to on-chip amplifier noise but can also include other noise terms that
are independent of signal level (see Janesick, Section 2.2.2 [67]). Often as low as 1 to
2 electrons RMS per pixel, the read noise is considered a requirement variable that
drives implementation of the detector electronics.
142
Electronics
Once the CCDs have collected charge, it is the task of the on-board electronics to
digitize the signal and create an event list for transmission to Earth. This process is
shown in Figure 4-12. First, a 16-bit analog-to-digital converter (ADC) converts the
voltage level in each pixel to analog-to-digital units (ADU). In addition to physical
pixel values, the raw frame contains non-physical “overclock” pixel values representing
a static offset that has been applied to all pixels in the frame. This level subtracted
from each raw frame on a per-node basis, which acts as a type of normalization that
removes time-varying offset changes from the data.
Next, the bias map is subtracted pixel-by-pixel from the resulting image. The
bias map is generated automatically when the CCDs are powered on for the first
time. Twelve 4-second exposures are taken, then the median value in each pixel is
calculated to produce the bias map. The radiation cover will be closed at this point,
so no asteroid X-rays are present in the bias map. It measures the dark current level
and any fixed pattern noise in the detectors.
After overclock and bias correction, the frame is processed by an event finding and
grading routine. A pixel is considered to have registered a photon strike (i.e. event) if
its value exceeds a pre-set event threshold (ET) and is greater than any of the pixels
immediately adjacent. Events are assigned a “grade” based on which pixels in a 3 ×
3 island around the event are above the split threshold (ST). Note that both the ET
and ST are defined based on the expected noise levels in the corrected frame. The
grade is a number from 0 to 7 that encodes the shape of the event and indicates which
pixel values should be summed to form the event pulse height. REXIS uses a grading
scheme similar to that used in the ASCA mission [136], shown in Figure 4-13.
The event list therefore contains two entries per event: the pulse height (energy)
in ADU, and the grade. Because REXIS has limited data volume, the raw/corrected
frames are discarded and only the event list is sent to Earth. As an additional data
saving measure, REXIS features an event filtering scheme that dynamically discards
low energy events when the count rate exceeds a given threshold. This only becomes
143
Figure 4-12: Event list creation within the REXIS on-board electronics.
important during solar flares but is mentioned here for completeness.
The final step performed by the electronics is truncation of the pulse height values.
Removing least significant bits (LSBs) from the pulse heights (which natively have
16 bits) reduces the data volume. Nine bits are currently allocated for pulse heights,
however it is treated as a requirement variable because of the trade-off between the
extent of truncation and spectral resolution.
144
Figure 4-13: REXIS event grade definitions (courtesy of Branden Allen).
Ground Processing
Data processing on the ground plays a critical role in generating the final asteroid
spectrum and calculating the FWHM values. As depicted in Figure 4-14, the first
step is to sort and co-add the event data by solar state. This is done to isolate flares
from the quiet Sun state, and apply the correct downstream data corrections.
The next step is to create a histogram showing the number of occurrences versus
pulse height. This is essentially an “unscaled” spectrum. The pulse heights are in
units of ADU, however the spectral lines (where the number of incident events is
large) will appear in the histogram as well defined peaks.
Converting from a histogram (counts versus ADU) to a spectrum (counts versus
energy) is accomplished using the signal from the onboard 55 Fe sources. As mentioned
previously, these sources emit X-rays at 5.90 keV (Mn-Kα) and 6.49 keV (Mn-Kβ).
By fitting these known calibration lines, we obtain the gain (eV/ADU) that is used
145
Figure 4-14: Processing of raw science telemetry to create the asteroid spectrum and
FWHM values.
to scale the horizontal axis of the histogram to produce the spectrum. The final step
in calculating the FWHM values is fitting a Gaussian curve to each line of interest.
Doing so produces a fitted mean µ and standard deviation σ, and the FWHM can be
√
immediately calculated as FWHM = 2 2 ln 2 σ.
146
4.3.2
Model Implementation
The REXIS measurement pipeline described in the previous section has been coded
into a computer simulation that emulates REXIS science operations in an end-to-end
manner. Implemented in Matlab, the code simulates X-ray interaction in the CCDs,
detector readout, the addition of various noises, digitization, event finding, gain correction, formation of astroid spectra, and calculation of the FWHM value. Most
importantly, the simulator is parametric. The user passes a parameter structure into
the model, which defines all properties of the instrument. This allows rapid evaluation of various configurations across the design space, which is essential for employing
the LSGA optimization algorithm in later steps of the requirement methodology.
Figure 4-15 shows a block diagram depicting the simulation’s overall structure.
It is implemented as a series of functions that model the physical, electrical, and
software phenomena taking place within the instrument. The user specifies a set
of system and environmental parameters (rexis parameters.m), which are used
to initialize the incident photon distribution and other aspects of the simulation
(rexis initialize.m). The main simulation routine (rexis.m) begins with the creation of a bias map (rexis bias map.m) and continues with the main CCD simulation code (rexis ccd sim.m). Once raw pixels have been generated, the CCD noise
process module adds various noise sources (rexis ccd noise.m). The noisy pixels
are then used to calculate X-ray events, which are compiled into a raw event list
(rexis event list.m). Next, the pulse heights in the event list are used to create
an event histogram (rexis histogram.m). The final step is to convert the histogram
into a spectrum and to determine the FWHM of the line of interest to obtain the
spectral resolution (rexis fwhm.m). The following sections describe each module in
greater detail.
Input Parameters
Input parameters are specified by user through the routine rexis parameters.m.
These parameters describe the configuration of the instrument, environmental condi147
Figure 4-15: REXIS simulation block diagram.
tions, and the X-ray source. Table 4.4 lists the parameters along with their default
values. Note that not all parameters are considered variable for the purposes of
model-based requirement definition. The subset of variable parameters (i.e. requirement variables) will be described below in the context of optimization.
Simulation Initialization
Once the parameter structure has been defined, the next step prior to entering the
main REXIS simulation is to perform certain initialization tasks in rexis initialize.m.
The primary task is to create a list of photons to be passed into the CCD simulator. Monochromatic X-ray inputs are straightforward, as the photon list in that case
consists of a simple vector of replicated energy values (one entry per photon). For a
broadband input (e.g. an asteroid spectrum), the photon list is similar, except now the
number of photons at each energy is populated according to the input distribution.
148
Table 4.4: Input parameters for the REXIS simulation.
Input Parameter
Frame integration time
Total integration time
Lower energy limit
Upper energy limit
Energy quantization
Bias frames
Line of interest
Line count rate
55
Fe count rate
Light leak
Pixel size
Pixel count
Grasp (asteroid)
MBE thickness
Temperature
Total ionizing dose
Read noise
CTI
Flight Value
4 seconds
423 hours
0.5 keV
7.5 keV
9 bits
12
2.31 keV
6.0076 × 10−2 counts/s
0.42 counts/s
1 e− /s/pixel
24 µm × 24 µm
1024 × 1024
3.85 cm2 -Sr
20 nm
-60 ◦ C
0.5 krad(Si)
7 e− /pixel/read
1.6 × 10−5
Comments
Total mission integration
Number of frames for bias map
S-K
Asteroid signal
Internal calibration signal
Light leaking through OBF
CCID-41
CCID-41
Collecting area times FOV
Thickness of field-free layer
Bias Map Creation
The first step within the main simulation routine is to generate a bias map. Recall
from Section 4.3.1 that the bias map is generated by taking the median pixel value over
twelve 4-second exposures. The same is done in simulation, emulating the concept
of operations for the REXIS mission in which a bias map is created onboard prior to
collecting science data.
CCD Simulation
The CCD simulation module (rexis ccd sim.m) is the largest and most complex code
block in the end-to-end simulation. It takes an input X-ray spectrum and creates a
simulated CCD pixel map, accounting for charge cloud generation and diffusion in
the various layers of the CCD. The code is based heavily on an IDL routine for ACIS
simulations by L. K. Townsley et al [138] at Pennsylvania State University (PSU),
who in turn based their work on charge diffusion models from Pavlov and Nousek [107]
149
and Hopkinson [62].
The basic approach of the CCD simulator is to propagate a population of Xray photons through the different layers of the CCD, keeping track of the energy,
fluorescence, and charge generation along the way. This was done on a photonby-photon basis in the PSU code. The REXIS implementation takes advantage of
Matlab’s code vectorization capabilities to propagate X-rays in parallel, reducing
computation time.
Figure 4-16 shows a simplified model of the CCID-41 layer structure used in the
CCD simulator. Photons enter the top (z = 0) and propagate “downward” along the
+z axis. The main active area is the 45 µm thick depletion region. When photons
are absorbed in this layer, the resulting charge is swept by an electric field downward
toward the pixel gates where the charge is collected. Although pixel gates have a
complex geometry, they are modeled here as a homogenous slab of silicon for reasons
of computational efficiency. Between the depletion region and the gate structures lies
an insulating layer made of a SiO2 + Si3 N4 + SiO2 sandwich. The sandwich layer
thicknesses are 0.06, 0.04, and 0.015 µm, respectively starting from the layer closest
to the depletion region. Immediately above the depletion region is a thin field-free
epitaxial layer whose purpose is to prevent charge from leaking upward out of the
depletion region and generating dark current. The damage layer on top of the CCD
is left over from the backside thinning process. The PSU simulation includes channel
stops at pixel boundaries to account for the left/right grade distribution observed in
front-illuminated CCD data. The channel stops were not shown by Townsley et al.
to have a significant impact on the grade distribution for back-illuminated CCDs, so
they are omitted from the REXIS model.
A simulation run begins by initializing the photon population based on a usersupplied table that lists the number of incident photons as a function of energy.
Starting photon positions are defined as a 2D uniform random variable over the
detector array active area. Each photon is assigned a direction unit vector that
is initialized to [0 0 -1], i.e. pointed directly “down” into the CCD. Photons are
then allowed to propagate to a depth z based on the energy-dependent absorption
150
Figure 4-16: Simplified CCID-41 layer structure (not to scale). Photons enter from
the top of the figure and propagate downward. The hashed line at the top surface of
the epitaxial layer indicates a one-sided reflective boundary condition: photons may
enter the layer from above but charge may not escape in the −z direction. Thickness
values are from K. Ryu at MIT Lincoln Laboratory.
coefficient α of the surrounding material,
1
z = − ln(R(0, 1)).
α
(4.3)
where R(0, 1) is a uniform random variable defined on (0,1). If the layer thickness
is less than the propagation distance z, the photon is moved into the adjacent layer
and propagation continues. If the layer thickness is greater than z, the photon is
absorbed. Absorption coefficients for Si, SiO2 , and the SiO2 + Si3 N4 + SiO2 insulator
sandwich were measured by Prigozhin et al. [110] and made available by L. Townsley
and P. Broos (see Figure 4-17).
If an incident photon has energy E0 at or above the K-shell binding energy for
silicon (1839 eV), a secondary Si-Kα (1739.6 eV) or Si-Kβ (1835.9 eV) fluorescent
photon will be generated with a probability of 0.043. If this happens, the original
photon is absorbed at a reduced energy of E = E0 − ESi-Kα or E = E0 − ESi-Kβ and
the fluoresced photon propagates from the absorption point in a random direction.
151
Absorption Coefficients
2
10
Si
SiO2
Sandwich
1
Absorption Coefficient [µm−1]
10
0
10
−1
10
−2
10
−3
10
0
1
2
3
4
5
6
7
8
9
Energy [keV]
10
11
12
13
14
15
Figure 4-17: Measured absorption coefficients for Si, SiO2 , and the SiO2 +Si3 N4 +SiO2
insulator sandwich. Data courtesy of L. Townsley and P. Broos (available at
http://www2.astro.psu.edu/users/townsley/simulator/index.html), originally taken
by G. Prigozhin et al [110].
Each photon absorbed by the CCD, whether external or fluoresced, generates a
charge cloud in the material. The initial cloud radius ri is given by,
ri = 0.0062Ef1.75
µm
(4.4)
where Ef is calculated as the initial photon energy Ei plus Fano noise:
p
Ef = Ei + N(0, 1) F · w · Ei · λ.
(4.5)
Here N(0, 1) is a normally distributed random variable with zero mean and unity
standard deviation, F = 0.115 is the Fano factor, and w is the energy needed to
liberate an electron-hole pair in the material where the photon was absorbed. If
a charge cloud overlaps a CCD layer boundary, the cloud is divided into sub-clouds
whose associated photon energies Ef are split according to the volume fraction in each
layer. λ is a tuning parameter used to adjust the FWHM value to match experimental
152
results. A similar approach was used by Townsley et al. in adjusting their CCD
simulation and the details are discussed in Section 4.3.3.
Note that the electron-hole pair libration energy w has a weak temperature dependence, which we include in the simulation for the sake of completeness. Scholze
et al. determined that w = 3.66 eV for Si at room temperature and varies linearly
with temperature [120] . More helpfully for our purposes, Lowe and Sareen measured
w over a range of temperatures. They found a slope of -0.0131% K−1 and a value of
w = 3.73 at 155 K, which gives the following expression for w(T ), used in the REXIS
simulation:
w(T ) = −1.31 × 10−4 T + 3.7503
(4.6)
Recall from Equation 4.1 that the amount of charge in each cloud Q0 is proportional to the energy of the incident photon. Adopting notation borrowed from
Townsley et al. and used in this section, the charge is calculated as,
Q0 = Ef /w∗ .
(4.7)
Here we use an effective electron-hole pair libration energy w∗ . Prigozhin et al. [109]
saw in the ACIS calibration data that for SiO2 , the effective conversion factor was
w∗ ≈ 52 eV/e due to a loss of approximately 65% of the generated electrons (the
expected conversion for SiO2 is w = 17 eV/e). The theoretical conversion factor
is still used for the Fano noise calculation in Equation 4.5. Table 4.5 shows the
theoretical and effective libration energies used in the code. We follow Townsley et
al. in assuming that only the SiO2 layer of the insulating sandwich produces events.
Table 4.5: Electron-hole pair libration energies used in the CCD simulation code.
w is the theoretical conversion used to calculate Fano noise and w∗ is the effective
conversion measured by Prigozhin et al [109] used to calculate the initial charge.
Material
Si
SiO2
SiO2 + Si3 N4 + SiO2
w [eV/e] w ∗ [eV/e]
eqn. 4.6
eqn. 4.6
17
52
17
52
153
After determining the initial radii (Equation 4.4) and charge (Equation 4.7), the
code calculates the radial expansion of each cloud using the appropriate equations for
charge diffusion and recombination from Pavlov and Nousek [107]. The diffusion behavior is dependent on the electric field in the layer of interest, therefore the depletion
and epitaxial (field-free) regions must be treated differently.
CCD Noise Processes
After the raw CCD pixel windows have been generated, a code module is called to
add the effect of dark current noise, optical light leak, charge transfer inefficiency
(CTI), and read noise to each pixel. In general, this is implemented by simulating the
physical processes behind each noise source in a manner consistent with Figure 4-10.
Dark current arises in the CCD pixels due to thermally induced electron holepair creation in the detector layers. Janesick [67] (Chapter 7) identifies four primary
regions of the CCD that contribute to dark current generation (see Figure 4-16 for
reference): 1) the “frontside” Si/SiO2 interface between the depletion region and the
insulator, 2) the bulk depletion region, 3) the neutral bulk material (i.e. the undepleted epitaxial layer), and 4) the “backside” Si/SiO2 interface between the damage
layer and the epitaxial layer. The surface regions (1) and (4) tend to dominate the
generation of dark current, however certain mitigations are available. Westhoff et al.
[147] describe how the frontside contribution can be suppressed through a pixel clocking technique known as surface inversion. They also discuss suppression of backside
dark current through a variety of passivation techniques that prevent charge from
leaking out of the depletion region. The CCID-41s flown on Suzaku used a charge
chemisorption (CC) passivation process that placed a layer of silver and hafnium oxide (HfO2 ) on the top surface of the detector [81]. The REXIS CCID-41s, on the
other hand, use a newer molecular beam epitaxy (MBE) process that results in lower
dark current.
For the REXIS simulation, we adopt the dark current equation derived by Janesick:
D(T ) = 2.5 × 1015 · PS · DD · T 3/2 · e−Eg /(2kT )
154
(4.8)
where D is the average dark current [e− /s/pixel] at temperature T [K], PS is the
pixel area [cm2 ], DD is the dark current density (also called the “dark current figure
of merit”) at room temperature (300 K) [nA/cm2 ], Eg is the silicon bandgap energy
[eV], and k is Boltzmann’s constant (8.62 × 10−5 eV/K). The MBE CCID-41s used
in REXIS have a dark current density in the hundreds of pA/cm2 at 300 K4 . The
precise value of DD is determined by a fit to experimental data, as discussed below in
Section 4.3.3. Note that the silicon bandgap energy is itself a function of temperature,
given by the empirical formula
Eg (T ) = 1.1557 −
7.021 × 10−4 T 2
.
1108 + T
(4.9)
As Equation 4.8 shows, dark current is a function of the CCD operating temperature, which motivates the passive cooling system REXIS employs. Dark current is
also a function of radiation dose and researchers at Lincoln Laboratory have noticed
an additional radiation-induced dark current of approximately 3 e− /pixel/rad(Si) at
-10 ◦ C5 . We can scale the radiation-induced dark current at -10 ◦ Cto a general function of temperature by solving for the room temperature dark current density DD .
Starting with Equation 4.8,
rad
D(−10 C ◦ C) = 3 e− /pixel/rad(Si) = 2.5 × 1015 · PS · DD
· T 3/2 · e−Eg /(2kT )
where the standard room temperature dark current density DD [nA/cm2 ] has been rerad
placed by a radiation-induced room temperature dark current density DD
[nA/cm2 /rad(Si)].
rad
Solving for DD
gives,
rad
DD
=3·
1
1
·
· T −3/2 · eEg /(2kT ) .
15
2.5 × 10
PS
rad
Substituting T = −10 ◦ Cand Eg (−10 C ◦ C) = 1.1202 eV gives DD
= 2.582 pA/cm2 /rad(Si).
Adding the radiation-induced dark current to the baseline dark current using Equa4
5
Personal communication with Barry Burke ([email protected]).
Personal communication with Jim Gregory ([email protected]).
155
tion 4.8 gives a formula for dark current as a function of both temperature and
radiation dose:
rad
D(T, R) = 2.5 × 1015 · PS · DD + DD
R · T 3/2 · e−Eg /(2kT )
(4.10)
where R is the total ionizing dose in [rad(Si)] and the other terms are as defined
previously.
In the REXIS simulation, the mean dark current value is calculated at the userspecified temperature and radiation dose using Equations 4.10 and 4.9, then multiplied by the integration time to get a dark electron count [e− /pixel]. This value is
used as the input to a Poisson random number generator that outputs a simulated
dark current level for each pixel the 3 × 3 event windows. The dark current windows
are then added pixel-by-pixel to the raw event windows.
The next step in the noise module is to add the effect of optical light that has
leaked through the optical blocking filter (OBF). The light leak flux L [e− /s/pixel]
is a user-supplied input and is considered a design-to parameter because it uniquely
drives the thickness of the deposited aluminum. L is multiplied by the integration
time to get the average number of light leak electrons per pixel. As in the dark current
case, a Poisson random number is then used to generate simulated light leak levels,
which are added pixel-by-pixel to the event windows.
At this stage in the simulation, the pixels contain electrons from detected X-ray
events, dark current, and light leak. The next step is to simulate the transfer of these
electrons to the output nodes for readout. Figure 4-11 shows the direction of charge
transfer with red arrows. Each pixel is shifted “down” using Np parallel transfers,
then “sideways” using Ns serial transfers until the charge reaches one of the four
output nodes in each CCD. The values of Np and Ns depend on where a given pixel
resides. For example, pixels in the top row undergo 2048 total parallel transfers to
traverse the imaging area and frame store. Figure 4-18 shows Np and Ns for each
pixel in the 2048 × 2048 detector array.
A small amount of charge is lost each time electrons are transferred from one pixel
156
Number of Serial Transfers per Pixel
Number of Parallel Transfers per Pixel
256
1
2048
1
256
512
512
768
1024
1024
1280
1536
1536
1792
2048
1
512
1024
1536
2048
1025 2048
1
(a)
512
1024
1536
2048
1
(b)
Figure 4-18: Number of parallel transfers Np (a) and serial transfers Ns (b) per pixel
in the REXIS four-CCD detector plane. The CCD boundaries are shown with dashed
white lines and the node locations (16 total) are denoted by white circles. The CCDs
are shown here in the same orientation as in Figure 4-4.
to another. This loss as a fraction of pixel charge level is quantified by the detector’s
charge transfer inefficiency (CTI), the value of which for the CCID-41 is better than
1 × 10−5 [11]. Given an initial charge in pixel (i, j) of S0 , the final charge Sf after Np
parallel transfers and Ns serial transfers is,
Sf (i, j) = S0 (i, j) · (1 − CTI)Np (i,j) · (1 − CTI)Ns (i,j)
(4.11)
where Np and Ns are a function of location in the focal plane array, as shown in
Figure 4-18. The simulation uses Equation 4.11 and knowledge of each pixel’s location
in the detector array to calculate the final charge per pixel after parallel and serial
transfer.
Now that our simulated pixels have been “transferred” to the output nodes, the
next step is to simulate noise added during readout. In practice, the read noise σr
[e− RMS] is calculated as the standard deviation of the overclock levels. Recall from
Figure 4-12 that the mean overclock level is subtracted from each pixel on a per node
basis. Therefore the most realistic way of simulating read noise would be to generate
an array of overclock pixel values, add Gaussian random noise, average the overclock
157
pixels, and subtract the result from each “real” pixel. As a computationally less
intensive but mathematically equivalent alternative, we simply add Gaussian noise
with a mean of zero and variance of σr2 to each pixel. The read noise value is a
user-supplied input.
Event List Creation
At this point, the simulation has generated a series of 3 × 3 pixel windows, each
of which represents the local area of the CCD on which an X-ray event has been
detected. The windows include dark current, read noise, light leak, and CTI effects
introduced by rexis ccd noise.m described above. Figure 4-19 shows three sample
windows generated by the REXIS simulator.
Raw Pulse Height: 687
Raw Pulse Height: 658
Raw Pulse Height: 663
600
500
300
200
100
0
Figure 4-19: Sample CCD windows output by the REXIS simulation. Each window
is 3 × 3 pixels and the values are summed to give raw event pulse heights in ADU.
The next step is to generate an X-ray event list in a manner that emulates the
processing that will take place within the REXIS electronics. This is the goal of
rexis event list.m. The first step is to sum pixel values according to the rubric
shown in Figure ?? to produce pulse heights in units of ADU. These values are then
truncated by removing the number least significant bits (LSBs) that results in an
overall data size corresponding to the user-defined energy quantization value. For
example, if the pulse heights have a native size of 12 bits and the energy quantization
is 9 bits, then 3 LSBs are removed. The resultant event list is a synthetic version of
the raw data expected to be sent by REXIS to the ground during nominal operations
at Bennu.
158
ADU
400
Histogram Creation
The next step in simulating REXIS data is creation of a pulse height histogram from
the event list (rexis histogram.m). Up until this point, all of the modules have
replicated processes that occur in space on board the REXIS spectrometer. This step
and the one that follow take place on the ground as part of the science data pipeline.
Simulated Histogram
5
10
Mn-Kα
5.90 keV
4
10
3
Counts
10
2
10
1
10
0
10
0
50
100
150
Pulse Heights [ADU]
200
250
Figure 4-20: Simulated REXIS histogram. A Gaussian has been fit to the Mn-Kα
peak at 5.90 keV.
The raw histogram is created by plotting the number of counts received in each
pulse height bin. The number of bins is determined by the pulse height quantization
specified in the flight software. With the baseline 9-bit pulse height encoding used by
REXIS, there are 512 bins (only the first 250 bins are shown in Figure 4-20 since the
counts above that point are all zero). The REXIS passband goes from 0.5 to 7.5 keV,
therefore 512 bins corresponds to a pulse height quantization of approximately 13.7 eV
in the energy domain.
159
Spectrum Creation and FWHM Fitting
Once the raw histogram is created, the next step is to scale the energy axis using the
gain (eV/ADU). This takes place in rexis fwhm.m and is accomplished by fitting the
Mn-Kα line at 5.90 keV emitted by the onboard
55
Fe sources. The fitting process is
shown in Figure 4-20. Using a Gaussian fit to obtain the mean of the peak along the
ADU axis (often called the centroid ), the gain can be immediately calculated since
the energy value is known. In this example, the Mn-Kα centroid occurs at 196 ADU
which yields a gain value of approximately 30 eV/ADU.
Simulated Spectrum
5
10
Fe−L
Mg−K
Si−K
S−K
Mn−K
S-K
2.31 keV
4
10
3
Counts
10
2
10
1
10
0
10
1
2
3
4
Energy [keV]
5
6
7
Figure 4-21: Simulated REXIS spectrum. The energy scale is set by the Mn-Kα
identified in Figure 4-20 and a Gaussian has been fit to the S-K peak at 2.31 keV
Scaling the histogram by the derived gain produces a spectrum of counts as a
function of energy, as shown in Figure 4-21. The final step in the simulation is to
calculate the FWHM value at the S-K line (2.31 keV). We do this by fitting the line
to a Gaussian and extracting the FWHM. This value is the spectral resolution of
REXIS at the energy of interest and is the final output of the end-to-end simulation.
160
Radiation Environment and Effects
Our description of the REXIS end-to-end model is now complete. Before continuing
on to describe the model validation, we pause to discuss the radiation environment
that REXIS will encounter. Radiation damage from charged particles can be grouped
into three categories: total ionizing dose (TID, also called ionization damage), displacement damage (also called total non-ionizing dose or bulk damage), and single
event effects (SEE) [78]. Single event effects (e.g. latchups and single event upsets)
are more relevant for integrated circuits than CCDs, therefore we focus on the former
two phenomena.
Ionization damage occurs when charged particles impinging upon a CCD generate
electron-hole pairs in the silicon [67]. One result of this process is the creation of
“surface states” at the Si/SiO2 interface in the CCD, causing additional dark current.
Ionization damage grows over time as TID accumulates in a device. The primary
sources of ionizing radiation for REXIS will be protons and electrons in the Van
Allen belts, and solar event protons. Ionizing radiation is measured in terms of total
ionizing dose (TID) using units of rad, the average energy imparted per unit of mass.
For silicon, 1 rad(Si) is equivalent to 100 ergs of energy absorbed by 1 g of material.
Displacement damage occurs when charged particles, typically at higher energies,
physically displace atoms from a CCD’s silicon crystal lattice. This causes “traps”
that prevent charge from moving in the device. The result is increased CTI and spikes
in the dark current (i.e. hot pixels). Like ionizing damage, displacement damage accumulates over time when a device is exposed to radiation. The main contributor
of REXIS non-ionizing dose is solar particle events (solar protons in particular) [78].
Non-ionizing radiation is measured in terms of particle fluence at a reference energy. For example, the REXIS CCDs may be exposed to a non-ionizing dose of
1010 protons/cm2 at 50 MeV. This is not to say that REXIS will see a radiation
environment that consists exclusively of 50 MeV protons. Indeed, REXIS will encounter charged particles from 0.1 MeV to 1 GeV. Instead, it means that if one were
to integrate the non-ionizing dose over all energies, the result would be equivalent to
161
1010 p+ /cm2 at 50 MeV.
It is possible to convert from fluence at one energy to the equivalent fluence at
another energy using the non-ionizing energy loss (NIEL), a quantity that describes
the “stopping power” of a material as a function of energy. Specifically, the NIEL is
the amount of energy lost by a particle as it passes through a certain material. Given
a known fluence Φ(E1 ) at energy E1 , the fluence at energy E2 is computed as,
Φ(E2 ) =
NIEL(E1 )
· Φ(E1 ).
NIEL(E2 )
(4.12)
The NIEL is typically given in units of MeV-cm2 /g and has been calculated for various
materials used in common applications, including silicon [70].
The ionizing and non-ionizing radiation doses for the REXIS CCD were calculated
at Goddard Space Flight Center using NOVICE, a software package for simulating
particle transport. The simulations account for the space environment, instrument
geometry, and material properties. Figure 4-22 shows the predicted doses. A radiation
cover was not included in the simulation model, however results are shown as a
function of spherical shielding thickness. While the spherical shielding model does not
match the REXIS geometry precisely, it provides a first order estimate of the impact of
the radiation cover. With the current radiation cover thickness of 4 mm, the predicted
ionizing and displacement damage doses are 0.5 krad(Si) and 2.38 × 109 p+ /cm2 ,
respectively.
Given the predicted REXIS radiation environment, we can derive an expected
CTI value for flight. In 2004, members of the XIS/Suzaku team exposed a backilluminated CCID-41—the same type of device used by REXIS—to a radiation source
and measured the resulting change in CTI [5]. They found that irradiation by 40 MeV
protons at a fluence of 2 ± 0.2 × 109 p+ /cm2 caused an initial change in CTI of
8.8 ± 0.7 × 10−5 . The use of spaced-row charge injection—a capability also found in
REXIS—decreased the change in CTI and resulted in a final value of 1.6 ± 0.7 × 10−5 .
Fortuitously, the XIS/Suzaku test fluence and REXIS flight fluence are nearly the
same. Therefore it seems reasonable to adopt a baseline value of 1.6 × 10−5 for the
162
Total Ionizing Dose
3
40 MeV Equivalent Fluence [p+/cm2]
Total ionizing dose [krad(Si)]
2.5
2
1.5
1
0.5
0
0
Displacement Damage Equivalent Fluence
11
10
10
10
9
1
2
3
4
5
6
7
Shielding Thickness [mm]
8
9
10
(a)
10
0
1
2
3
4
5
6
7
Shielding Thickness [mm]
8
9
10
(b)
Figure 4-22: Simulated REXIS radiation environment showing (a) total ionizing dose
and (b) displacement damage equivalent fluence. Predictions with a 4 mm thick
radiation cover are 0.5 krad(Si) and 2.38 × 109 p+ /cm2 , respectively (dashed green
lines). Simulation data were provided by the NASA Goddard Space Flight Center.
REXIS CTI (assuming the use of charge injection, which is the current plan).
4.3.3
Model Validation
Now that we have described the end-to-end REXIS simulation in detail above, we turn
to important topic of validation. This is a critical activity in any model development
effort. In order for the model to be useful for requirement definition, we must ensure
that the outputs match physical reality. Also, because the REXIS simulation uses
a Monte Carlo approach, we must understand the impact of a limited number of
samples on output variability. These tasks are the subject of this section.
Measurement Convergence
The accuracy of the FWHM measurement depends in part on the number of events
that have been recorded. As integrations are carried out over the course of the
mission, counts are accumulated and the statistics of the histogram improve. The
measurement accuracy after 200 hours of integration, for example, would be expected
to be much higher than the accuracy obtained after only 2 hours. This can be seen
in Figure 4-23a, which shows the predicted FWHM at S-K (2.31 keV) for integration
163
times ranging from 1 to 423 hours (the expected mission total).
FWHM Error versus Integration Time
FWHM Convergence versus Integration Time
12
168
11
166
10
9
FWHM Error [eV]
FWHM at 2.31 keV [eV]
164
162
160
158
8
7
6
5
4
3
156
2
154
1
152
0
50
100
150
200
250
300
Total Integration Time [hours]
350
400
(a)
0
0
50
100
150
200
250
300
Total Integration Time [hours]
350
400
(b)
Figure 4-23: REXIS simulation convergence showing (a) FWHM value and (b)
FWHM standard error as a function of integration time.
Clearly, the FWHM calculation stabilizes as more samples are added via longer
integration. This fact can be quantified using the standard error of the FWHM, as
shown in Figure 4-23b. It was calculated by employing the jackknife Monte Carlo error
(MCE) estimator introduced in Chapter 3 (equation 3.11). According to Figure 423b, the 423 hour mission duration is more than enough integration time to achieve a
FWHM accuracy below 1 eV. REXIS will receive approximately 216 counts per hour
from Bennu in the S-K line, therefore the total expected flux at that energy is over
91,000 photons. Even 100 hours of integration time (21,600 counts) would appear to
provide sufficiently low FWHM error.
Dark Current
We now consider tuning the model that captures the dark current dependency on temperature. Recall that the dark current model given by Equation 4.8 is parameterized
by a single quantity DD , the so-called “dark current density” at room temperature
(300 K). The dark current behavior is completely described by this parameter, which
is shown in Figure 4-24. Here we plot experimental CCID-41 dark current data as
a function of temperature. The raw data were provided by Steve Kissel in the MIT
164
Kavli Institute (MKI) CCD laboratory. The points are then fit to the dark current
model using non-linear least squares with DD as the free parameter. The resulting
dark current density value of DD = 0.7476 nA/cm2 is used in the REXIS simulation.
Dark Current Model Validation
12
Experimental Data
Model
Dark Current [e−/s/pixel]
10
8
6
4
2
0
−95
−90
−85
−80
−75 −70 −65 −60
Detector Temperature [C]
−55
−50
−45
Figure 4-24: REXIS dark current model validation. Experimental data are courtesy
of S. Kissel using device 51-1-6-3. Error bars are ±5% and the fitted dark current
density value is DD = 0.7476 nA/cm2 .
Spectral Resolution Energy Dependence
The spectral resolution of REXIS varies as a function of energy: low energy X-rays
are detected with a higher resolution (smaller FWHM) than high energy X-rays. An
important step in validating the REXIS model is showing that it can reproduce this
trend in a manner that agrees with experimental data. Accordingly, Figure 4-25
shows experimentally determined FWHM values for a CCID-41 detector along with
those predicted by the REXIS simulation.
The experimental setup used at MKI differs slightly from the REXIS flight configuration. Specifically, the electronics used are from the Astro-E/Suzaku mission and
a single CCD is tested instead of the four-CCD configuration used by REXIS. The
165
Spectral Resolution versus X-Ray Energy
120
Experimental Data
Model
110
FWHM [eV]
100
90
80
70
60
50
500
1000
1500
2000
Energy [eV]
Figure 4-25: Spectral resolution (FWHM) as a function of X-ray energy. Experimental
data are courtesy of S. Kissel using device 53-1-6-3. Error bars are omitted because
the random error for each point is less than 1 eV.
test parameters are listed in Table 4.6 and were used in the REXIS simulation to create the results shown. Note that this illustrates an advantage of parametric models.
One can change the parameters to match experimental conditions, validate the model
using available data over a range of inputs, then change the parameters back to the
flight values when executing analysis of the instrument in its mission configuration.
Looking at Figure 4-25, the REXIS model captures the general trend of the experimental data well. Each data point is the result of at least 1 hour of integration at
a flux rate on the order of 100 counts/cm2 /s or, equivalently, 2 million counts total.
This is equivalent to approximately 10,000 hours of integration in Figure 4-23b, at
which point the FWHM standard error is much less than 1 keV. Accordingly, we omit
error bars from Figure 4-25. While the statistical error is low, there are still systematic errors in the data that result in a spread of points around the model-predicted
value. These errors include temperature fluctuations and changes in pixel clock phasing and voltages. These are artifacts of the Kavli test apparatus and are not expected
to be present in flight.
166
Table 4.6: CCD test parameters compared to REXIS flight parameters. Parameters
marked with (†) are for the FWHM vs. energy validation test (Figure 4-25) while
those marked with (‡) are for the FWHM vs. temperature validation test (Figure 426). Parameters not listed here are the same as in Table 4.4.
Parameter
Frame integration time
Total integration time
Energy quantization
Line of interest
Line count rate
Light leak
Number of CCDs
MBE thickness
Temperature
Total ionizing dose
Read noise
CTI
Device ID
Flight Value
4
423
9
2.31
0.06
1
4
20
-60
0.5
7
1.6 × 10−5
TBD
Test Value
8
∼1
12
varies† / 5.90‡
600
0
1
10† / 20‡
-80† / varies‡
0
2
10−6
53-1-6-3† / 53-1-8-4‡
Units
seconds
hours
bits
keV
counts/s
e− /s/pixel
nm
C
krad(Si)
e− /pixel/read
◦
Finally, obtaining the results shown in Figure 4-25 required adjusting the FWHM
tuning parameter λ introduced in Equation 4.5. Proposed by Townsley et al, the
parameter accounts for un-modeled noise terms that broaden the spectral line and
increase the FWHM value [138]. Non-linear least squares was used to produce a bestfit value of λ = 2.3. This is similar to what Townsley et al. found for the line width
tuning parameter in their CCD model.
Spectral Resolution Temperature Dependence
Another important system parameter influencing spectral resolution is temperature.
The FWHM is generally smaller (i.e. better spectral resolution) at lower temperatures
due to decreased amounts of dark current, fewer charge traps, and other temperaturedependent noise effects. Figure 4-26 shows experimental data for the FWHM at 5.9
keV (Mn-Kα) versus CCD temperature. As before, these data were collected by Steve
Kissel at MKI.
The model output shows good agreement with the data. As before, we used
167
Spectral Resolution versus Temperature
180
175
FWHM at 5.9 keV [eV]
170
Experimental Data (Node A)
Experimental Data (Node B)
Experimental Data (Node C)
Experimental Data (Node D)
Model
165
160
155
150
145
140
135
130
−80
−75
−70
Temperature [C]
−65
−60
Figure 4-26: Spectral resolution (FWHM) at 5.9 keV as a function of focal plane
temperature. Experimental data are courtesy of S. Kissel using device 53-1-8-4. Error
bars are omitted because the random error for each point is less than 1 eV
a parameter λ to tune the model and bring it into agreement with the measured
points. In this case, however, it became necessary to determine λ as a function of
temperature. This may be due to a temperature dependence of the electron-hole pair
liberation energy for SiO2 and the insulating sandwich material that is currently unmodeled (instead, constant values given by Table 4.5 are used). The following power
law relationship was used to fit the temperature data,
λ(T ) =


a (T + 80)k + b if T ≥ −80 ◦ C
if T < −80 ◦ C

b
where T is the temperature in Celsius. Note that b corresponds to the value of λ at
-80 ◦ C. Non-linear least squares was used to determine the following values for the
fitting constants: a = 1.15 × 10−5 , k = 3.54, and b = 1.30. These values for a and k
are used in the simulation runs presented below. The value of b, on the other hand,
is set at 2.3, which was obtained from device 53-1-6-3 (see previous section). We use
168
this value for b because the dark current parameter DD was also established using
device 53-1-6-3.
4.3.4
Baseline Performance and Single-Axis Studies
With model validation and tuning complete, we can now run the simulation with
the baseline flight parameter values discussed in the previous sections and shown in
Table 4.4. Doing so yields a predicted FWHM at 2.31 keV (S-K) of 160 eV. Thus the
current REXIS design is predicted to achieve the required S-K spectral resolution of
227 eV with considerable margin.
Figure 4-27 visualizes the baseline REXIS requirement set using a “radar plot”.
The radial position of each vertex represents the value of a given requirement axis.
The lower and upper bounds are the same as those given in Table 4.7 however the
distances have been normalized between 0 and 1 over the interval shown in each axis.
The axes are oriented so that requirement values that are more difficult to achieve are
closer to the center of the plot and requirement values that are easier to achieve are
closer to the edge of the plot. For example, along the temperature axis, a requirement
of -110◦ C would be at the center while a requirement of -40◦ C would be at the edge.
Prior to applying LSGA optimization and generating requirement sets, we perturb
one variable at a time around the baseline requirement vector. The variables, their
baseline values, and their bounds are shown in Table 4.7. The goal is to gain an
intuitive sense for how the requirement variables affect the spectral resolution.
Table 4.7: REXIS requirement variables, baseline values, and lower and upper bounds.
Variable
Temperature
Light leak
Ionizing dose
Read noise
CTI
Pulse height encoding
LB
-110
0
0.25
2
10−6
7
Baseline
-60
1
0.5
7
1.6 × 10−5
9
169
UB
-40
10
20
20
10−4
12
Units
Type
C
continuous
photons/s/pixel continuous
krad(Si)
continuous
−
e RMS/pixel
continuous
loss/transfer
continuous
bits
discrete
◦
Temperature
[−110, −40]
C
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Figure 4-27: Radar plot of REXIS baseline requirement variables. The brackets under
each variable name give the lower and upper bounds used in LSGA (see Table 4.7).
This baseline set of requirements produces a spectral resolution (FWHM) at 2.31 keV
of 160 eV, below the required value of 227 eV
The results of varying each parameter one at a time are shown in Figure 4-28. The
temperature results (top left panel) show, as expected, that the spectral resolution
worsens for increasing temperature, and dramatically so as the temperature rises
above -70◦ C. This is due to the dark current, which is an exponential function of
temperature. For extremely low temperatures, other error terms such as read noise
dominate, hence the long tail for which spectral resolution remains low and constant.
As the temperature increases, dark current becomes a more important contribution.
The top right panel of Figure 4-28 shows the effect of light leak on the spectral
resolution. As the amount of light leaking through the OBF increases, the spectral
resolution worsens. This makes intuitive sense, as light leak is an additional Poisson
noise source that corrupts the pulse heights in the output histogram.
The middle left panel of Figure 4-28 shows spectral resolution as a function of
170
Spectral Resolution vs. Temperature
Spectral Resolution vs. Light Leak
280
Prediction
Requirement
Current
260
FWHM at 2.31 keV [eV]
FWHM at 2.31 keV [eV]
260
280
240
220
200
180
160
140
−110
Prediction
Requirement
Current
240
220
200
180
160
−100
−90
−80
−70
Temperature [C]
−60
−50
140
0
−40
2
Spectral Resolution vs. Radiation (TID)
FWHM at 2.31 keV [eV]
FWHM at 2.31 keV [eV]
260
220
200
180
160
220
200
180
160
2
4
6
8
10
12
14
Radiation (TID) [rad(Si)]
16
18
140
2
20
Spectral Resolution vs. CTI
6
8
10
12
14
Read Noise [e− RMS/pixel]
16
18
20
Spectral Resolution vs. Pulse Height Encoding
260
FWHM at 2.31 keV [eV]
FWHM at 2.31 keV [eV]
4
280
Prediction
Requirement
Current
240
220
200
180
160
140
Prediction
Requirement
Current
240
280
260
10
280
Prediction
Requirement
Current
240
140
8
Spectral Resolution vs. Read Noise
280
260
4
6
Light Leak [pht/s/pixel]
Prediction
Requirement
Current
240
220
200
180
160
2
4
6
CTI [loss/transfer]
8
140
7
10
−5
x 10
8
9
10
Pulse Height Encoding [bits]
11
12
Figure 4-28: REXIS spectral resolution as a function of temperature, light leak,
total ionizing dose, read noise, charge transfer inefficiency, and pulse height encoding.
Except for the one shown varying, all parameters are set at values given by Table 4.4.
171
total ionizing dose (TID). The plot shows a degradation of spectral resolution as the
dose increases. This is due to the fact that TID increases the dark current in the
detector. The effect is linear in TID, hence the roughly linear trend, however when
coupled with temperature variations, the effect can be much more pronounced (see
radar plots in the following section).
The middle right panel of Figure 4-28 shows spectral resolution as a function of
the read noise. As the read noise grows larger, the spectral resolution gets worse.
This is due to the fact that read noise is a Gaussian random noise process added
to the pixel values. When the pixels are summed to get pulse heights, the noise is
included and the distribution of the pulse height histogram broadens.
The bottom left panel of Figure 4-28 shows changes in spectral resolution versus
charge transfer inefficiency (CTI). The effect of CTI is to destroy charge in a pixel as
it is transferred in the CCD. This is a function of the number of pixels the charge must
traverse as it is read out. Because X-ray events hit the CCD at random locations,
CTI essentially destroys a random amount of charge in each pixel. As CTI increases,
more charge is destroyed and the noise contribution increases. In this case the noise
is distributed as a uniform random variable but the trend is similar to the read noise
case.
Finally, the bottom right panel of Figure 4-28 shows the spectral resolution as a
function of the number of bits used to encode the pulse height. There appears to be
a threshold number of bits that must be used in order to avoid a quantization noise
contribution. With 9 bits or more, the FWHM values are stable, indicating that other
noise terms will dominate. When fewer than 9 bits are used, the relatively coarse
histogram binning will corrupt the measurement, as evidenced by higher FWHM
values.
4.4
Requirement Sets through LSGA
The REXIS end-to-end simulation was run using LSGA to find the level set of designs
that satisfy the spectral resolution requirement for the S-K line (FWHM = 227 eV).
172
The optimization used a population of 300 over 30 generations and a level set size of
50 solutions. The converging diversity metric is shown below in Figure 4-29.
Diversity Metric versus Generation
0.7
D
Dspace
Dspread
0.6
Diversity Metric
0.5
0.4
0.3
0.2
0.1
0
1
5
10
15
Generation
20
25
30
Figure 4-29: LSGA diversity metric for REXIS optimization run. In this case the
temperature is -65◦ C, the CTI is 5.3 × 10−5 , the read noise is 9.5 e− RMS/pixel, the
ionizing radiation dose is 7.3 krad(Si), and the light leak is 6.7 photons/s/pixel.
LSGA returns solutions spread out over the requirement space, as intended. Figure 4-30 shows three possible requirement sets out of the fifty that were generated by
LSGA. Each requirement set meets the science objective of 227 eV spectral resolution
at 2.31 keV through different means. Each set provides a different way of allocating the science performance among the top-level engineering requirements. The first
requirement set (blue) is a relatively “balanced” in the sense that all of the requirements are near the midpoint of their feasible bounds. The requirement set may be
slightly skewed toward higher temperature and a higher number of bits but this is
the requirement set that is closest to the midpoint of the requirement space out of
all the others. The second requirement set (green) skews toward higher CTI values.
To keep the overall science performance equal, slack has been taken up in the other
requirement variables, most notably temperature, which is significantly lower than in
the first requirement set. The third requirement set (red) skews toward higher read
173
Temperature
[−110, −40]
C
Reqt Set 1
Reqt Set 2
Reqt Set 3
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Figure 4-30: Three sample REXIS requirement sets produced by LSGA. Each requirement set produces a spectral resolution of 227 eV at 2.31 keV (S-K).
noise values. This requires a significant tightening of the other requirements, such as
radiation dose and CTI. In general, this demonstrates the “water bed” effect of requirement distributions. As one requirement becomes relaxed, the other requirements
must become more stringent in order to maintain the same level of performance. Once
these various requirement sets have been mapped out via LSGA, they can be traded
on the basis of cost, design difficulty, schedule, or other considerations.
The extent of coverage over the requirement space is shown in Figure 4-31. Each
panel shows two requirement vectors produced by LSGA, one at each end of the
allowed range for a given requirement of interest. Each panel therefore represents two
corner cases on opposite ends of a diagonal through the 6-dimensional hypercube that
constitutes the requirement space. For example, the panel in the upper left shows
two possible requirement sets, one for which the temperature requirement is most
constrained (-110◦ C) and another for which it is most relaxed (-40◦ C).
174
Temperature
[−110, −40]
C
Temperature
[−110, −40]
C
Min Temperature
Max Temperature
Min Light Leak
Max Light Leak
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Temperature
[−110, −40]
C
Read Noise
[2, 20]
e− RMS
Temperature
[−110, −40]
C
Min Radiation
Max Radiation
Min Read Noise
Max Read Noise
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Read Noise
[2, 20]
e− RMS
Temperature
[−110, −40]
C
Temperature
[−110, −40]
C
Min CTI
Max CTI
Min Energy Bits
Max Energy Bits
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Read Noise
[2, 20]
e− RMS
Figure 4-31: Radar plots showing the corner cases along each axis of the requirement
space. Each plot shows the requirement sets where the variable of interest is at a
minimum (blue lines) and at a maximum (red lines). Variable ranges are given by
Table 4.7.
175
4.5
Requirement Trades
The REXIS System Requirements Review (SRR) occurred in early 2012 and instrument development is well under way (CDR occurred in February 2014 and the engineering model is undergoing integration and test currently). As such, the REXIS
top-level engineering requirements are set. The requirements sets obtained via LSGA
are an interesting point of reference, but they will not influence the REXIS flight
design. However, the LSGA requirements can be used to consider three scenarios
that illustrate the utility of a model-based requirement approach. First, in light of
the current gap between the REXIS predicted performance (160 eV) and the required
performance (227 eV), we consider how much the REXIS design can be relaxed and
still meet the science objective. Second, we consider the case of engineering push-back,
in which an engineering requirement cannot be achieved and other requirements must
be adjusted to compensate. Third, we consider the case of science creep, in which the
science team desires an increase in performance and the change must be absorbed by
the engineering requirements.
Relaxing current requirements
Recall that the predicted REXIS performance (FWHM of 160 eV at 2.31 keV) is
significantly higher than the science requirement (227 eV). In this sense, one could
argue that REXIS is “over designed” and that the engineering requirements are being
overly constrained. Therefore we can ask the following question: how can we relax
the engineering requirements to bring the predicted performance of REXIS more in
line with the required performance?
The database of requirement sets produced by LSGA can help in this regard.
One option would be to adopt the set that represents a “uniform” relaxation of
requirements across all subsystems. In other words, the 67 eV of margin between
160 eV and 227 eV would be allocated roughly evenly (percentage wise) among the
different engineering requirements. This can be seen in Figure 4-32, in which the new
more relaxed requirement set appears in the radar plot as an expanded version of
176
the original requirement set. The requirement values associated with each FWHM
level in Figure 4-32 are listed in Table 4.8. Calculating the percent change for each
requirement shows that the margin is indeed apportioned roughly equally.
Temperature
[−110, −40]
C
FWHM = 160 eV
FWHM = 227 eV
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Figure 4-32: Allocating performance margin equally among engineering requirements.
The baseline requirement vector is shown in blue and the “relaxed” requirement vector
is shown in read.
Table 4.8: Requirement values for Figure 4-32.
FWHM:
Variable
Units
160 eV
◦
Temperature
C
-60
Light leak
photons/s/pixel
1
Ionizing dose krad(Si)
0.5
Read noise
e− RMS/pixel
7
CTI
loss/transfer
1.6 × 10−5
Pulse height bits
9
FWHM:
227 eV
-47
1.2
0.6
8.5
1.93 × 10−5
7
∆
+13
+0.2
+0.1
+1.5
+0.33 × 10−5
-2
%∆
+22%
+20%
+20%
+21%
+21%
-22%
Instead of allocating the performance margin equally, it may be desirable to
177
limit requirement changes to a single subsystem. Presumably, some requirements
can change due to design flexibility while others must remain fixed because of irreversible design decision that have been locked in. Alternatively or in addition, it may
be desirable to limit the scope of changes by only allowing a single requirement to
vary (e.g. if the desire is to limit changes to a single subsystem). We can therefore
search the database to find new requirements that meet the more relaxed science objective while only changing one parameter at a time. Figure 4-33 shows the possible
engineering requirement relaxation options within this one-at-a-time approach. For
example, if the focal plane temperature is changed from -60◦ C to -46.5◦ C (i.e. a 23%
increase) then the spectral resolution goes from 160 eV to 227 eV. The requirement
values are shown in Table 4.9.
Table 4.9: Requirement values for Figure 4-33. Note that requirements are being
changed one at a time in this case.
FWHM:
Variable
Units
160 eV
◦
Temperature
C
-60
Light leak
photons/s/pixel
1
Ionizing dose krad(Si)
0.5
−
Read noise
e RMS/pixel
7
FWHM:
227 eV
-46.5
7.8
12
16.5
∆
+13.5
+6.8
+11.5
+9.5
%∆
+23%
+680%
+2300%
+135%
Engineering push-back
We now consider the second scenario, the case of engineering push-back. Begin by
assuming that the initial requirements are Requirement Set 1 from Figure 4-30 above:
the focal plane temperature is -63◦ C, light leak is 4 photons/s/pixel, the ionizing
radiation dose is 6.5 krad(Si), the read noise is 9 e− RMS, the CTI is 4.4 × 10−5 ,
and the pulse height quantization is 11 bits. Now imagine that testing has revealed
that the baseline light leak requirement of 4 photons/s/pixel cannot be achieved. The
engineers ask: what is the worst that the light leak can be without hurting science?
As before, we can turn to the LSGA-generated requirement database for assistance. Because one requirement is becoming less stringent (light leak), the other
178
Temperature
[−110, −40]
C
Temperature
[−110, −40]
C
FWHM = 160 eV
FWHM = 227 eV
FWHM = 160 eV
FWHM = 227 eV
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Read Noise
[2, 20]
e− RMS
(a) Temperature requirement change
Temperature
[−110, −40]
C
(b) Light leak requirement change
Temperature
[−110, −40]
C
FWHM = 160 eV
FWHM = 227 eV
FWHM = 160 eV
FWHM = 227 eV
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Read Noise
[2, 20]
e− RMS
(c) Radiation requirement change
(d) Read noise requirement change
Figure 4-33: Changes the REXIS engineering requirements that would result in the
required science performance. The requirements shown in blue lead to a FWHM of
160 eV while the red requirements produce a FWHM of 227 eV.
requirements must “take up the slack” in order to keep the performance unchanged.
The LSGA results help in this regard and provide alternative solutions. Figure 4-34a
shows the baseline requirement set along with two options for increasing the light
leak. In both options, the light leak requirement has been relaxed at the expense of
other requirements, which generally become more strict. The exception is the energy
quantization, the requirement on which happens to be less stringent (i.e. lower bit
allocations) in this case. The other requirement axes—temperature, CTI, read noise,
and radiation—demonstrate lower values in order to compensate for the degraded
179
light leak specification.
Temperature
[−110, −40]
C
Temperature
[−110, −40]
C
Baseline
Option 1
Option 2
Baseline
Option 1
Option 2
Fixed
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Read Noise
[2, 20]
e− RMS
(a)
(b)
Figure 4-34: Options for increasing the light leak requirement at the expense of
other requirements. Panel (a) assumes all requirements can be changed. Panel (b)
assumes that the read noise and energy quantization requirement are fixed, while the
temperature, radiation, and CTI requirements can change.
Of course, it is unlikely that all requirements will be changeable. Some requirements may be fixed due to irreversible engineering decisions that have been made
previously. This is especially if true requirement changes are sought late in the
project life cycle. Figure 4-34b captures this more realistic scenario. The goal is
still to relax the light leak requirement at the expense of other subsystems, but in
this case we assume that the read noise and energy quantization requirements are
fixed (as indicated by x’s in the radar plot). This could be, for example, because the
electronics have been implemented, making it overly costly to revisit the read noise
and quantization specification. That leaves only the radiation, CTI, and temperature
requirements available to compensate for the increased levels of light leak. Note that
in option 1, the change in temperature is relatively large, therefore the changes in
radiation level and CTI are relatively small. Conversely, in option 2, the change in
temperature is smaller but the changes in radiation and CTI are relatively large.
180
Science requirement creep
Finally, consider the third scenario outlined above: science requirement creep. Imagine that at some point after the initial requirements have been set, the science team
wishes to increase the performance of the instrument to maximize the scientific impact of the mission. For concreteness, assume that the top-level requirement goes
from 227 eV to 200 eV. The question we would like to answer is: how can the impact
of this change be isolated to one subsystem and which engineering requirement should
be adjusted?
Figure 4-35 helps answer this question. Specifically, it shows solutions that will
increase the performance to the new required level while changing only one requirement. The requirement values are listed in Table 4-35. For example, decreasing the
temperature by 9.5% will result in the desired performance improvement. Likewise if
the ionizing dose were decreased by 85% or if the read noise were decreased by 44%.
This gives the system engineer a sense of the relative sensitivity of performance to
each degree of freedom. Once the options are mapped out in the manner described,
the new requirement sets can be traded on the basis of cost, risk, schedule, or design
difficulty.
Table 4.10: Requirement values for Figure 4-35. Note that parameters are changing
one at a time.
FWHM:
Variable
Units
227 eV
◦
Temperature
C
-63
Ionizing dose krad(Si)
6.5
Read noise
e− RMS/pixel
9
4.6
FWHM:
200 eV
-72
1.0
5
∆
-6
-5.5
-4
%∆
-9.5%
-85%
-44%
Summary
This chapter presented REXIS as a case study for demonstrating how model-based
requirement definition would be applied to a complex instrument. We began with a
181
Temperature
[−110, −40]
C
Baseline
∆ Temp
∆ TID
∆ Read
Light Leak
[0, 10]
pht/s/pixel
Energy Bits
[12, 7]
bits
Radiation
[0.25, 20.00]
krad(Si)
CTI
[1e−06, 1e−04]
loss/transfer
Read Noise
[2, 20]
e− RMS
Figure 4-35: Changing requirements one at a time to increase the science performance.
The baseline requirements give a FWHM of 227 eV while the revised requirements
(shown as ∆Temp, ∆TID, and ∆Read) give a FWHM of 200 eV.
description of REXIS, its concept of operation, and a general overview of the principles behind CCD X-ray spectroscopy. We then applied each step of the model-based
requirement definition methodology in turn. First, the user objectives were quantified
in terms of spectral resolution (full-width half maximum, FWHM) at the S-K line energy of 2.31 keV. A requirement of FWHM ≤ 227 eV was specified. Next, we detailed
the implementation of a high fidelity end-to-end model that simulated all aspects of
the REXIS measurement pipeline, from photon interaction to the CCD to creation
of the final calibrated data product. After validating the model by comparing the
simulated output with experimental data, we evaluated the current REXIS baseline
and determined that the expected performance is 160 eV at 2.31 keV, better than the
requirement. With the model in place, LSGA was used to generate a set of alternative
REXIS requirements across the possible requirement space. Finally, these alterna182
tive requirements were traded from the standpoint of three hypothetical scenarios.
First, we considered ways to relax the current REXIS requirements to absorb the
margin that the model shows. Second, we looked at how to re-allocate requirements
in the event of engineering push-back. Finally, we examined requirement changes to
accommodate an increase in the science performance objective.
183
184
Chapter 5
Conclusions
5.1
Thesis Summary
The goal of this work is to improve the state of the art in requirement definition
for instrument systems by introducing a model-centric approach. The methodology
transforms the requirement allocation process into a multidisciplinary optimization
and sensitivity analysis problem over a space of top-level engineering requirements.
Using a new genetic algorithm (LSGA), we demonstrate the ability to locate the
collection of requirement sets that satisfy science performance objectives. After identifying the candidate requirement sets, we conducted a sensitivity analysis to select
the final requirement set based on which one was most robust to change.
Chapter 1 discussed the importance of sound requirement definition practices.
From a theoretical perspective, much of a system’s cost is “locked in” early in the life
cycle. Requirement instability or “requirement creep” results in engineering re-work
that can have negative cost and schedule impacts. This was highlighted from a practical perspective through examples from several recent civil and defense instrument
programs. Chapter 1 also identified two reasons why requirement definition is often
difficult. First, and perhaps most importantly, there exists a divide between scientists and engineers (i.e. users and designers) that makes communication and hence
requirement definition challenging. Second, many traditional approaches to requirement allocation (e.g. error budget spreadsheets) have significant drawbacks. The
185
model-based requirement methodology was created to address these challenges.
Chapter 2 provided an overview of literature in three areas relevant to this work:
requirement definition processes, model-based design methodologies, and projectspecific approaches to requirements. The formal requirement definition processes
of NASA, INCOSE, and ESA were considered and found to be overly broad for the
purpose of instrument development. They lacked specific guidance on how to tackle
the challenging yet important task of translating science goals into top-level engineering requirements. A variety of model-based approaches were discussed, including
heuristic methods for multiobjective optimization and the “isoperformance” analysis
setting. Both informed the methodology in this thesis. Finally, Chapter 2 identified
what other instrument systems used for requirement definition and highlighted both
the promise and drawbacks of existing approaches.
Chapter 3 describes the model-based requirement definition methodology in full
detail. After presenting a sample problem on the design of a star camera lens, the
full four-step methodology is explained. Step one is to quantify the user (scientist)
objectives by defining the data product, figure of merit, and the top-level performance
requirement. Step two uses that information, along with a diagram of the measurement pipeline, to create a parametric end-to-end that serves as a mapping between
the science domain (model output) and engineering domain (model input). In step
three of the methodology, this model is exercised using a new level set finding genetic
algorithm (LSGA) to find the set of engineering requirement vectors that meet the
science objectives. Finally, step four screens the available requirement vectors—all of
which meet science objectives—on the basis of robustness. This is done using a local
derivative-free sensitivity analysis based on standardized regression coefficients.
Chapter 4 applied the method to the Regolith X-Ray Imaging Spectrometer
(REXIS) instrument, currently under development at MIT. The instrument performance was quantified in terms of spectral resolution at energies corresponding to
particular elements of interest. Next, we crated an end-to-end simulation to model
the detailed physics of x-ray interaction in the CCDs. The simulation also included
software processing to handle gain correcting using an on-board reference signal. We
186
exercised the REXIS model using LSGA to find a family of 100 requirement vectors that meet requirements, then used local sensitivity analysis to select the most
robust requirement set. We also demonstrated an ability to quickly re-compute the
requirement set given changes in the desired science performance.
Finally, this chapter summarizes the thesis and highlights the primary research
contributions. The final section of this chapter discusses interesting avenues for future
research based on this work.
5.2
Contributions
This research contributes at the nexus of model-based design, genetic algorithms,
requirement definition, instrument systems, and program management. Specific contributions include the following:
1. Developed a methodology for requirement definition as an optimization and sensitivity analysis problem. The approach developed here transforms what is typically an iterative, human intensive process into a modelcentric one. In doing so, the methodology is able to leverage model-based design
tools such as level set optimization and sensitivity analysis to make rational,
informed, quantitative, and traceable decisions to derive top-level engineering
requirements.
2. Created a new genetic algorithm for finding level sets in the design/requirement space. This work created a new genetic algorithm called
LSGA specifically designed to find level sets (i.e. contours at a specific level
of performance) in the requirement space. The algorithm is a heuristic analog
of the derivative-based methods developed by de Weck [23, 25]. Part developing the new algorithm was the introduction of a normalized diversity metric
quantifying both the spread and spacing of solutions in the level set. We also
defined a parameter-free crowding metric for ensuring diversity while propagating the algorithm. The LSGA algorithm has promise for applications beyond
187
model-based requirement definition and is suitable for any design optimization
problem requiring identification of a level set in a derivative-free manner.
3. Introduced a method for evaluating the impact of science changes.
Model-based requirement definition allows for rapid re-evaluation if scientific
preferences change. Producing a new top-level set of engineering requirements
is as simple as running the LSGA and sensitivity analysis again. Even if the
changes are not implemented, at least one can quantify what the impacts will
be and weigh the costs against benefits.
4. Embraced a model-free approach. The tools that scientists and engineers
use to simulate instrument systems are as diverse as the various physical phenomena the instruments are designed to measure. Models can be continuous or
discrete, deterministic or stochastic, linear or non-linear, steady-stay or transient, etc. As such, the approach presented by this thesis was specific designed
to be applicable to a variety of models. LSGA and the regression-based sensitivity analysis were developed so that they can be used with models in a “black
box” sense, i.e. without restrictions on the model’s inner workings.
5. Addressed drawbacks in traditional error budgeting. This work demonstrates a requirement approach that avoids the pitfalls of traditional manual
error budgeting. Specifically, by using a model we, 1) allow for the effects of
data processing to be included, 2) account for interactions between design variables, 3) allow for time-varying analysis, and 4) explore the range of possible
designs/requirements rather than perturbing a point design.
6. Introduced sensitivity analysis to avoid requirement instability. The
model-based requirement methodology uses local sensitivity analysis to quantify
the robustness of selected requirements to changes in engineering capabilities.
As discussed with historical examples in Chapter 1, requirement instability can
be a significant contributor to cost and schedule growth. By choosing the most
robust requirement set, the project is able to minimize the impact from changes
188
to requirements later in the life cycle and hence reduce programmatic risk.
7. Introduced a local derivative-free sensitivity analysis. This thesis introduced a regression-based local sensitivity analysis as an alternative to traditional derivative-based approaches. The methodology places a collection of
normally distributed random samples in the neighborhood of a point of interest. The samples are used to generate model outputs, which are then passed
through a multiple linear regression to produce standardized regression coefficients. These coefficients serve as main effect indicators and allow the analyst to
decompose the total output variance into relative contributions from different
design/requirement variables.
5.3
Future Work
This research points to a number of possible future directions. Below are several
suggestions for follow-on research or applications.
Applications during Other Life Cycle Phases
The methodology presented in this thesis was developed for use during the requirement definition and allocation phase of the system life cycle (see Figure 1-4). However,
it may also be useful in other phases. For example, consider an instrument that has
just begun its operational phase and that is performing at an unexpectedly low level
(i.e. the data product figure of merit is less than the science requirement). In this
case, LSGA could be run using the observed performance value instead of the required
value. The resulting requirement set would tell you the range of possible values for
the various top-level parameters that explain the unexpected results. One could then
use the set of values and their sensitivity metrics to determine which parameters must
be changed to most efficiently make up for the lost performance.
189
Lower-Level Requirement Allocation
Importantly, the methodology could be applicable within engineering subsystems—
i.e. at a lower level of functional decomposition than the original scope shown in
Figure 1-2—instead of between scientists and engineers as originally proposed. Indeed, this was the case in the star camera example from Chapter 3. The “user”
was an attitude control subsystem (ACS) engineer who required a certain number
of star centroids to meet their performance objective. The “designer” was an optical engineer who was tasked with producing a system that could meet those goals.
Model-based requirement definition was used to flow down one engineering requirement (ACS) into other engineering requirements (optical). It would be possible to
extend this chain further, iteratively flowing down requirements at increasingly lower
levels of decomposition within the same system or subsystem.
Application to Non-Instrument Systems
The genesis of model-based requirement definition was instrument systems, which
are uniquely compelling due to the challenges of interfacing scientists and engineers
who normally operate in separate domains. However, the methodology is broadly
applicable to any system in which one can generate an end-to-end model that captures
the mapping between engineering requirements and system performance. The tools
developed here are particularly useful for problems that lack well-behaved analytical,
linear, or differentiable solutions.
Integration with Model-Based System Engineering (MBSE) Tools
While this methodology was formulated without reference to MBSE tools so as the be
as general as possible, the approach presented here is certainly compatible with such
tools. One could envision the requirement set being housed in an SysML database
linked to the analysis packages described above. If an update to system parameters
were to occur, the optimization and sensitivity analysis tasks could be re-run and the
requirements automatically updated accordingly. Therefore model-based requirement
190
definition is inherently compatible with MBSE approaches and would benefit from
closer integration in the future.
Alternative Heuristic Methods
Another possible area of future research is the creation of alternative heuristic methods to solve level set problems in a derivative-free manner. Options include variants
of simulated annealing (SA) [73] or particle swarm optimization (PSO) [71]. For
PSO, a crowding operator similar to the on developed for LSGA may be applicable.
Such an operator would serve to push members of the swarm away from each other
and thus maintain diversity in the population. A potential challenge is balancing
this “repelling” operator with the other terms in the PSO velocity equation (current
velocity, memory influence, and swarm influence). For SA, a crowding operator could
be included explicitly, but a more elegant approach may be to minimize the entropy
(disorder) of the level set as it accumulates. The ideal level set consists of solutions
distributed uniformly in space and entropy would be at a minimum in this case. Since
the notion of entropy is inherently part of the SA optimization, this could be a natural
metric to use.
Improvements to Local Sensitivity Analysis
The regression-based local sensitivity analysis presented in this work considered only
a simple linear combination of design variables. The standardized regression coefficients served as an indication of how much each design variable contributed to the
overall variation in the output. A straightforward extension of this approach allows
for consideration of interaction effects between design variables. Specifically, one
would include terms in the regression equation proportional to the product of two
design variables. This would add n2 terms to the regression equation and the standardized coefficients of these terms would quantify the extent to which interaction
effects contribute to the output variance. Higher order combinations (e.g. interaction
effects between three design variables) could be computed similarly.
191
Global Sensitivity Analysis
The methodology presented here uses a local sensitivity analysis to screen candidate
requirement sets with the goal of choosing the most robust one. A local analysis
is appropriate in this setting because we seek to distinguish among design points
in different ares of the trade space. A useful extension would be to add a global
sensitivity analysis step prior to the LSGA optimization. Global approaches such
as the Elementary Effects method [92, 14], the Fourier Amplitude Sensitivity Test
(FAST) [21], or the Sobol method [132, 116] could be used to quantify how much
parameter contributes to output uncertainty globally. Importantly, such an analysis
would allow us to identify those variables in the model that, when allowed to vary
over their range of values, produce little or no effect on the output. Those variables
could then be fixed during LSGA optimization to avoid wasted expense. This analysis
would also serve as a useful check against excessively fine model fidelity. If it was
revealed that certain variables produce no effect on the output, one could conclude
that they are ineffective at capturing the phenomena they are intended to model.
192
Appendix A
Multiobjective Considerations
The discussion of level set optimization and the LSGA algorithm in Chapter 3 dealt
only with scalar functions of multiple variables. This appendix extends the concepts
introduced previously to the case of multiple objectives. First, we illustrate the way
in which LSGA can be adapted for multiple objective problems, as one may expect
to find in practice. Then, we compare LSGA to the popular NSGA-II algorithm.
A.1
Multiobjective LSGA
Many remote sensing instruments have multiple scientific objectives and therefore
multiple performance requirements. With a few minor modifications, LSGA can solve
these problems as well, making it suitable for ascertaining the top-level engineering
requirements that satisfy multiple science objectives simultaneously. Consider two
objectives, each a function of three variables:
f1 (x) = x21 + x22 − x3
(A.1)
f2 (x) = x21 − x42 + x3
(A.2)
where x = [x1 x2 x3 ]. Now imagine that we seek performance levels of f1 (x) = −1
and f2 (x) = 2. Using Equations A.1 and A.2, we can solve for the level surface of
193
each function,
f1 (x) = −1
f2 (x) = 2
x3 ≡ y(x1 , x2 ) = x21 + x22 + 1
(A.3)
x3 ≡ z(x1 , x2 ) = −x21 + x42 + 2
(A.4)
Figure A-1 shows the level surfaces y(x1 , x2 ) and z(x1 , x2 ) as graphs over R2 . Satisfying the objective f1 (x) = −1 is equivalent to residing on y(x1 , x2 ) while satisfying
f2 (x) = 2 is equivalent to residing on z(x1 , x2 ). Satisfying both objectives simultaneously is equivalent to residing on the intersection between the two level surfaces,
which is the red line shown in Figure A-1c.
(a)
(b)
(c)
Figure A-1: Level surfaces for (a) f1 (x) = −1 and (b) f2 (x) = 2. The intersection of
these two surfaces is shown as a solid red line in (c).
To accommodate multiple objectives, LSGA was modified slightly from the version
presented in Chapter 3. First, the ranking operator was changed to account for
multiple objectives. Recall that the original procedure for ranking involved sorting
individuals by their (single) fitness score F. The position in the fitness rank list was
then added to the position in the crowding rank list and the combined rank was used
to evaluate overall fitness (see Algorithm 2). With multiple objectives we now have
multiple fitness scores F (i) . Therefore the fitness sorting procedure is carried out over
each objective and the combined ranking is a weighted sum of the single-objective
ranks. The details are shown in Algorithm 5.
In addition to the ranking procedure, the level set extraction procedure was modified for multiobjective LSGA. Recall that in single-objective LSGA, an individual at
194
Algorithm 5 Ranking within multiobjective LSGA
(1)
(2)
(M )
procedure rank(FP , FP , ... , FP
, CP )
for i = 1, . . . , M do
fi = sort
(i)
FP
M objectives
ascending
Sort by fitness
end for
c = sort CP ascending
Sort by crowding
for i = 1, . . . , N do
N individuals in population
RP [i] = (position in f1 )/M + . . .
Assign combined rank (smaller is better)
(position in f2 )/M + . . .
...
(position in fM )/M + . . .
position in c
end for
end procedure
location x is removed from the population and added to the level set whenever it satisfies the criterion |f (x) − c| < τ , where c is the desired level and τ is the convergence
tolerance. For the multiobjective case, individuals are extracted when the satisfy the
logical conjunction of each single objective criteria:
|f1 (x) − c1 | < τ1
and . . .
|f2 (x) − c2 | < τ2
..
.
and . . .
|fM (x) − cM | < τM
Multiobjective LSGA was run on Equation A.3 over 200 generations with a population size of 300. The desired level set size was 40 individuals, the crossover probability was 0.9, the mutation probability was 1/3, and both the crossover and mutation
distribution indices were 20. The results are shown in Figure A-2.
A.2
Comparison to NSGA-II
One of the most popular multiobjective genetic algorithms is NSGA-II, developed by
Kalyanmoy Deb and his students [30, 37]. NSGA-II and LSGA are fundamentally
195
Figure A-2: Multiobjective LSGA results.
different optimization settings. NSGA-II attempts to find a Pareto front of nondominated solutions in the objective space. LSGA, on the other hand, attempts to
find a locus of “performance satisficing” solutions in the parameter (design) space.
Still, the two concepts are related and the goal of this section is to explore the
connection between them.
Consider a two objective minimization sample problem “Min-Ex” introduced by
Deb [27],
Min-Ex:



Minimize






 Minimize



subject to






f1 (x) = x1
f2 (x) =
1+x2
x1
(A.5)
0.1 ≤ x1 ≤ 1
0 ≤ x2 ≤ 5
Rearranging the objective functions yields the following expression that relates the
two,
f2 =
1 + x2
f1
(A.6)
As Deb points out, the boundaries of the feasible objective space correspond to the
196
minimum and maximum values of the numerator on the right hand side, which occurs
for x∗2 = 0 and x∗2 = 5.
To examine the connection between LSGA and NSGA-II, we first use LSGA to
calculate the level sets for f1 (x) = 0.5 and f2 (x) = 5. Because the objective functions
are simple, we can calculate the level curves analytically as,
f1 (x) = 0.5
⇒
f2 (x) = 5
⇒
x1 = 0.5
(A.7)
x2 = 5x1 − 1
The analytical level curves along with the results from LSGA are shown in Figure A-3.
Level Set f1 (x) = 0.5
Level Set f2 (x) = 5
5
5
Analytical
LSGA
Analytical
LSGA
4
3
3
x2
x2
4
2
2
1
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
x1
(a)
(b)
Figure A-3: Level curves calculated analytically from Equation A.7 and using LSGA.
Next, the Pareto front was calculated using the controlled elitism variant of NSGAII [31], as implemented in the Matlab function gamultiobj. The results are shown
in Figure A-4 along with the analytical Pareto front f2 = 1/f1 . The LSGA level
set results were then used to evaluate each objective function and, as expected, the
computed objective function values lie along lines at the specified levels f1 = 0.5 and
f2 = 5 in objective space.
LSGA thus successfully finds points along the Pareto front found by NSGA-II. In
this case, LSGA was used to find two points (f1 = 0.5 and f2 = 5), however one could
197
10
9
Pareto front
NSGA−II results
LSGA results (f = 0.5)
8
LSGA results (f2 = 5)
1
7
f2
6
5
4
3
2
1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
f1
Figure A-4: Comparison of NSGA-II and LSGA.
find an arbitrary number of points by running LSGA multiple times with different
target levels
198
Bibliography
[1] National Aeronautics and Space Administration. Systems Engineering Handbook. NASA/SP-2007-6105 Rev 1, December 2007.
[2] Jeremy Agte, Olivier de Weck, Jaroslaw Sobieszczanski-Sobieski, Paul Andersen, Alan Morris, and Martin Spieck. MDO: assessment and direction for
advancement–an opinion of one international group. Structural and Multidisciplinary Optimization, 40:17–33, 2010.
[3] Bernd Aschenbach, Ulrich G. Briel, Frank Haberl, Heinrich W. Braeuninger,
Wolfgang Burkert, Andreas Oppitz, Philippe Gondoin, and David H. Lumb.
Imaging performance of the XMM-Newton x-ray telescopes. Proceedings of
SPIE, 4012, July 2000.
[4] M. W. Bautz, S. E. Kissel, G. Y. Prigozhin, B. LaMarr, B. E. Burke, and J. A.
Gregory. Progress in X-ray CCD Sensor Performance for the Astro-E2 X-ray
Imaging Spectrometer. Proceedings of SPIE, 5501:111–122, 2004.
[5] Mark Bautz, Steve Kissel, Beveryl MaMarr, and Gregory Prigozhin. ASTRO-E2
Memo: Characteristics of BI CCD. Technical Report OSIRIS-REx-PLAN-0014,
MIT Center for Space Research, March 2004.
[6] Marshall W. Bautz and John A. Nousek. Science Instrument (SI) Calibration
Report for the AXAF CCD Imaging Spectrometer (ACIS). Technical report,
Center for Space Research (Massachusetts Institute of Technology) and Department of Astronomy & Astrophysics (Pennsylvania State University), January
1999.
[7] David Beasley, David R. Bull, and Ralph R. Martin. A Sequential Niche
Technique for Multimodal Function Optimization. Evolutionary Computation,
1(2):101–125, 1993.
[8] Benjamin S. Blanchard. System Engineering Management. John Wiley & Sons,
Inc., Hoboken, NJ, 3rd edition, 2008.
[9] Harrison L. Bralower. Mechanical Design, Calibration, and Environmental Protection of the REXIS DAM. Master’s thesis, Massachusetts Institute of Technology, Department of Mechanical Engineering, September 2013.
199
[10] Turner Brinton. Air Force’s 1st Dedicated SBIRS Satellite Makes Orbit. Space
News International, May 9 2011.
[11] Barry E. Burke. Documentation for the CCID-41. Technical report, Lincoln
Laboratory, Massachusetts Institute of Technology, March 2004.
[12] Barry E. Burke, J. A. Gregory, M. W. Bautz, G. Y. Prigozhin, S. E. Kissel,
Bernard B. Kosicki, Andrew H. Loomis, and Douglas J. Young. Soft-X-Ray
CCD Imagers for AXAF. IEEE Transactions on Electron Devices, 44(10):1633–
1642, October 1997.
[13] David N. Burrows, J. E. Hill, J. A. Nousek, J. A. Kennea, A. Wells, J. P. Osborne, A. F. Abbey, A. Beardmore, K. Mukerjee, A. D. T. Short, G. Chincarini,
S. Campana, O. Citterio, A. Moretti, C. Pagani, G. Tagliaferri, P. Giommi,
M. Capalbi, F. Tamburelli, L. Angelini, G. Cusumano, H. W. Brauninger,
W. Burkert, and G. D. Hartner. The Swift X-Ray Telescope. Space Science
Reviews, 120(3–4):165–195, October 2005.
[14] Francesca Campolongo, Jessica Cariboni, and Andrea Saltelli. An effective sscreening design for sensitivity analysis of large models. Environmental Modelling
and Software, 22:1509–1518, 2007.
[15] E. Caroli, J. B. Stephen, G. Di Cocco, L. Natalucci, and A. Spizzichino. Coded
Aperture Imaging in X- and Gamma-Ray Astronomy. Space Science Reviews,
45:349–403, 1987.
[16] A. Charnes, W. W. Cooper, and R. O. Ferguson. Optimal Estimation of Executive Compensation by Linear Programming. Management Science, 1(2):138–
151, 1955.
[17] Beth Ellen Clark, Richard P. Binzel, Ellen S. Howell, Edward A. Cloutis, Maureen Ockert-Bell, Phil Christensen, Maria Antonietta Barucci, Francesca DeMeo, Dante S. Lauretta, Harold Connolly, Alicia Soderberg, Carl Hergenrother,
Lucy Lim, Josh Emery, and Michael Mueller. Asteroid (101955) 1999 RQ36:
Spectroscopy from 0.4 to 2.4µm and meteorite analogs. Icarus, 216(2):462–475,
2011.
[18] Cypress Semiconductor Corporation. HAS2 Detailed Specification - ICD. 00154123 Rev A, September 2009.
[19] National Research Council. Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force Acquisition.
Technical report, Committee on Pre-Milestone A Systems Engineering / Air
Force Studies Board, 2008.
[20] Arthur N. Cox. Allen’s Astrophysical Quantities. Springer-Verlag, New York,
New York, fourth edition, 2000.
200
[21] R. Cukier, H. Levine, and K. Shuler. Nonlinear sensitivity analysis of multiparameter model systems. Journal of Computational Physics, 26:1–42, 1978.
[22] John C. Davidson et al., editors. SIM Lite Astrometric Observatory: From
Earth-Like Planets to Dark Matter. Number 400-1360. NASA JPL, 2009.
[23] Olivier L. de Weck. Multivariable Isoperformance Methodology for Precision
Opto-Mechancial Systems. PhD thesis, Massachusetts Institute of Technology,
Department of Aeronautics and Astronautics, 2001.
[24] Olivier L. de Weck and Marshall B. Jones. Isoperformance: Analysis and Design
of Complex Systems with Desired Outcomes. Systems Engineering, 9(1):45–61,
2006.
[25] Olivier L. de Weck, David W. Miller, and Gary E. Mosier. Multivariable Isoperformance Methodology for Precision Opto-Mechanical Systems.
AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, AIAA-2002-1420, 2002.
[26] Kalyanmoy Deb. Solving Goal Programming Problems using Multi-Objective
Genetic Algorithms. IEEE Proceedings of the Congres on Evolutionary Computation, 1:77–84, 1999.
[27] Kalyanmoy Deb. Multi-Objective Optimization using Evolutionary Algorithms.
John Wiley & Sons, Ltd, Chichester, England, 2001.
[28] Kalyanmoy Deb. Nonlinear Goal Programming using Mulit-Objective Genetic
Algorithms. The Journal of the Operational Research Society, 52(3):291–302,
2001.
[29] Kalyanmoy Deb and Ram B. Agrawal. Simulated binary crossover for continuous search space. Complex Systems, 9(2):115–148, 1995.
[30] Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. A Fast Elitist Non-Dominated Sorting Genetic Algorithm for Multi-Objective Optimization: NSGA-II. Kanpur Genetic Algorithms Laboratory (KanGAL) Technical
Report, 200001, 2000.
[31] Kalyanmoy Deb and Tushar Goel. Controlled elitist non-dominated sorting
genetic algorithms for better convergence. In Evolutionary Multi-Criterion Optimization, pages 67–81. Springer-Verlag, 2001.
[32] Kalyanmoy Deb and David E. Goldberg. An investigation of niche and species
formation in genetic function optimization. In Proceedings of the Third International Conference on Genetic Algorithms, pages 42–50, San Mateo, California,
1989. Morgan Kaufmann Publishers.
201
[33] Kalyanmoy Deb and Mayank Goyal. A Combined Genetic Adaptive Search
(GeneAS) for Engineering Design. Computer Science and Informatics, 26(4):30–
45, 1996.
[34] Kalyanmoy Deb and Mayank Goyal. Optimizing Engineering Designs Using a
Combined Genetic Search. In Proceedings of the 7th International Conference
on Genetic Algorithms, pages 521–528, East Lansing, Michigan, 1997. Morgan
Kaufmann Publishers.
[35] Kalyanmoy Deb and Mayank Goyal. A Flexible Optimization Procedure for Mechanical Component Design Based on Genetic Adaptive Search. Transactions
of the ASME: Journal of Mechanical Design, 120(2):162–164, 1998.
[36] Kalyanmoy Deb and Amarendra Kumar. Real-coded genetic algorithms with
simulated binary crossover: Studies on multi-modal and multi-objective problems. Complex Systems, 9(6):431–454, 1995.
[37] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Myarivan. A Fast
and Elitist Multiobjective Genetic Algorithm: NSGA-II. IEEE Transactions
on Evolutionary Computation, 6(2):182–197, 2002.
[38] Nathan P. Diller. Utilizing Multiple Attribute Tradespace Exoploration with
Concurrent Design for Creating Aerospace Systems Requirements. SM thesis,
Massachusetts Institute of Technology, Department of Aeronautics and Astronautics, June 2002.
[39] E-ELT Science Working Group. Science Cases and Requirements for the ESO
ELT. Program report, European Organisation for Astronomical Research in
the Southern Hemisphere (ESO), April 2006.
[40] Bradley Efron. Jackknife-After-Bootstrap Standard Errors and Influence Functions. Jornal of the Royal Statistical Society, Series B, 54:83–111, 1992.
[41] Jeff A. Estefan. Survey of Model-Based Systems Engineering (MBSE) Methodologies. Technical Report INCOSE-TD-2007-003-02, International Council on
Systems Engineering (INCOSE), Seattle, WA, June 2008.
[42] John Warner Evans. The Design of Systems with Desired Outcomes: An
Isoparametric Approach Using Adaptive Search. PhD thesis, George Mason
University, Department of Systems Engineering and Operations Research, 2009.
[43] Carlos M. Fonseca and Peter J. Fleming. Genetic Algorithms for Multiobjective
Optimization: Formulation, Discussion and Generalization. In Proceedings of
the Fifth International Conference on Genetic Algorithms, pages 416–423, San
Mateo, California, 1993. Morgan Kaufmann Publishers.
[44] European Cooperation for Space Standardization. Space Engineering: System
Engineering General Requirements. ECSS-E-ST-10C, March 2006.
202
[45] European Cooperation for Space Standardization. Space Engineering: Technical
Requirements Specification. ECSS-E-ST-10-06C, March 2006.
[46] Defense Science Board / Air Force Scientific Advisory Board Joint Task Force.
Acquisition of National Security Space Programs. Technical report, Office of
the Undersecretary of Defense for Acquisition, Technology, and Logistics, May
2003.
[47] Kevin Forsberg and Harold Mooz. The Relationship of Systems Engineering to
the Project Cycle. Engineering Management Journal, 4(3):36–43, 1992.
[48] Kevin Forsberg and Harold Mooz. Application of the “Vee” to Incremental
and Evolutionary Development. Proceedings of the Fifth Annual International
Symposium of the National Council on Systems Engineering, July 1995. St.
Louis, MO.
[49] Justin Gan and Kevin Warwick. Dynamic Niche Clustering: A Fuzzy Variable Radius Niching Technique for Multimodal Optimisation in GAs. IEEE
Proceedings of the Congress on Evolutionary Computation, 1:215–222, 2001.
[50] Gerard Gilmore. European Extremely Large Telescope: some history and the
scientific community’s preferences for wavelength. Proc. SPIE, 6986(07), 2008.
[51] Roberto Gilmozzi and Jason Spyromilio. The 42m European ELT: Status. Proc.
SPIE, 7012(19), 2008.
[52] Fred Glover. Tabu Search – Part I. ORSA Journal on Computing, 1(3):190–206,
1989.
[53] Fred Glover. Tabu Search – Part II. ORSA Journal on Computing, 2(1):4–32,
1990.
[54] David E. Goldberg. Genetic Algorithms for Search, Optimization, and Machine
Learning. Addison-Wesley, Reading, Massachusetts, 1989.
[55] David E. Goldberg and Kalyanmoy Deb. A Comparative Analysis of Selection
Schemes Used in Genetic Algorithms. Foundations of Genetic Algorithms 1
(FOGA-1), pages 69–93, 1991.
[56] David E. Goldberg and Jon Richardson. Genetic algorithms with sharing for
multimodal function optimization. In Proceedings of the Second International
Conference on Genetic Algorithms, pages 41–49, Pittsburgh, Pennsylvania,
1987. Lauwrence Erlbaum Associates.
[57] Object Management Group. Systems Modeling Language Standard v1.3.
http://www.omg.org/spec/SysML/1.3/, June 2012.
[58] Homero L. Gutierrez. Performance Assessment and Enhancement of Precision Controlled Structures During Conceptual Design. PhD dissertation, Massachusetts Institute of Technology, February 1999.
203
[59] Daniel E. Hastings and Hugh McManus. Thrust 2 and 3 Final Report. Technical report, Space Systems, Policy, and Architecture Research Consortium
(SSPARC), October 2004.
[60] John H. Holland. Adaptation in Natural and Artificial Systems. University of
Michigan Press, Ann Arbor, Michigan, 1975.
[61] Winford E. Holland and Bette A. Stead. Exploring the Scientist-Engineer Conflict: A Form and Content Analysis of Their Written Communication. Journal
of Business Communication, 9(3):25–35, 1972.
[62] Gordon R. Hopkinson. Analytic modeling of charge diffusion in charge-coupleddevice imagers. Optical Engineering, 26(8):766–772, 1987.
[63] Jeffrey Horn, Nicholas Nafpliotis, and David E. Goldberg. A Niched Pareto Genetic Algorithm for Multiob jective Optimization. In Proceedings of the First
IEEE Conference on Evolutionary Computation, volume 1, pages 82–87, Orlando, FL, June 1994.
[64] James P. Ignizio. A Review of Goal Programming: A Tool for Multiobjective
Analysis. The Journal of the Operational Research Society, 29(11):1109–1119,
1978.
[65] James P. Ignizio. Generalized Goal Programming: An Overview. Computers &
Operations Research, 11(4):277–289, 1983.
[66] Michel D. Ingham, Robert D. Rasmussen, Matthew B. Bennett, and Alex C.
Moncada. Engineering Complex Enbedded Systems with State Analysis and
the Mission Data System. Journal of Aerospace Computing, Information, and
Communication, 2, December 2005.
[67] James R. Janesick. Scientific Charge-Coupled Devices. SPIE Press, Bellingham,
Washington, 2000.
[68] Cyrus D. Jilla. A Multiobjective, Multidisciplinary Design Optimization Methodology for the Conceptual Design of Distributed Satellite Systems. PhD thesis,
Massachusetts Institute of Technology, Department of Aeronautics and Astronautics, 2002.
[69] Cyrus D. Jilla and David W. Miller. Multi-Objective, Multidisciplinary Design Optimization Methodology for Distributed Satellite Systems. Journal of
Spacecraft and Rockets, 41(1):39–50, 2004.
[70] Insoo Jun, Michael A. Xapsos, Scott R. Messenger, Edward A. Burke, Robert J.
Walters, Geoff P. Summers, and Thomas Jordan. Proton Nonionizing Energy
Loss (NIEL) for Device Applications. IEEE Transactions on Nuclear Science,
50(6):1924–1928, December 2003.
204
[71] J. Kennedy and R. Eberhart. Particle Swarm Optimization. Proceedings of the
IEEE International Conference on Neural Networks, pages 1942–680, 1995.
[72] Sarah Killcoyne and John Boyle. Managing Chaos: Lessons Learned Developing
Software in the Life Sciences. IEEE Computing in Science & Engineering,
11(6):20–29, 2009.
[73] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by Simulated
Annealing. Science, 220(4589):671–680, 1983.
[74] Elizabeth Koehler, Elizabeth Brown, and Sebastien J.-P. A. Haneuse. On the
Assessment of Monte Carlo Error in Simulation-Based Statistical Analyses. The
American Statistician, 63(2):155–162, 2009.
[75] Jason E. Koglin, HongJun An, Kenneth L. Blaedel, Nicholai F. Brejnholt,
Finn E. Christensen, William W. Craig, Todd A Decker, Charles J. Hailey,
Layon C. Hale, Fiona A Harrion, Carsten P. Jensen, Kristin K. Madsen, Kaya
Mori, Michael J. Pivovaroff, Gordon Tajiri, and William W. Zhang. NuSTAR
hard x-ray optics design and performance. Proceedings of SPIR, 7437, August
2009.
[76] Katsuji Koyama et al. X-Ray Imaging Spectrometer (XIS) on Board Suzaku.
Publications of the Astronomical Society of Japan, 59:S23–S33, January 2007.
[77] M. O. Krause and J. G. Ferreira. K x-ray emission spectra of Mg and Al.
Journal of Physics B: Atomic and Molecular Physics, 8(12):2007–2014, 1975.
[78] Ray Ladbury et al. Radiation Hardness Assurance Plan. Technical Report
OSIRIS-REx-PLAN-0014, Origins, Spectral Interpretation, Resource Identification, Security, and Regolith Explorer (OSIRIS-REx) Project, April 2012.
[79] Dante S. Lauretta et al. An Overview of the OSIRIS-REx Asteroid Sample
Return Mission. 43rd Lunar and Planetary Science Conference, 1659(2491),
March 2012.
[80] S. M. Lee. Goal Programming for Decision Analysis. Auerbach Publishers,
Philadelphia, Pennyslvania, 1972.
[81] Michael Lesser and Venkatraman Iyer. Enhancing back illuminated performance
of astronomical CCDs. Proceedings of SPIE, 3355:446–456, March 1998.
[82] Jian-Ping Li, Marton E. Balazs, Geoffrey T. Parks, and P. John Clarkson. A
Species Conserving Genetic Algorithm for Multimodal Function Optimization.
Evolutionary Computation, 10(3):207–234, 2002.
[83] Carl C. Liebe. Accuracy Performance of Star Trackers—A Tutorial. IEEE
Transactions on Aerospace and Electronic Systems, 38(2):587–599, 2002.
205
[84] Lucy F. Lim and Larry R. Nittler. Elemental composition of 433 Eros: New
calibration of the NEAR-Shoemaker XRS data. Icarus, 200:129–146, 2009.
[85] J. Liske, M. Kissley-Patig, and R. Gilmozzi. The E-ELT Design Reference Mission: Technical Data Used for Simulations. Program Report E-TRE-ESO-0800718 Issue 1, European Organisation for Astronomical Research in the Southern
Hemisphere (ESO), June 2010.
[86] J. Liske, M. Kissley-Patig, and R. Gilmozzi. The E-ELT Design Reference Mission. Program Report E-TRE-ESO-080-0717 Issue 2, European Organisation
for Astronomical Research in the Southern Hemisphere (ESO), January 2011.
[87] Mark S. Avnet. Socio-Cognitive Analysis of Engineering System Design: Shared
Knowledge, Process, and Product. PhD thesis, Massachusetts Institute of Technology, Engineering Systems Devision, 2009.
[88] David L. Meier and William M. Folkner. SIMsim: An End-to-End Simulation
of The Space Interferometer Mission. Proc. SPIE, 4852:131–142, 2003.
[89] Brad L. Miller and Michael J. Shaw. Genetic Algorithms with Dynamic Niche
Sharing for Multimodal Function Optimization. Proceedings of IEEE International Conference on Evolutionary Computation, pages 786–791, May 1996.
[90] David W. Miller, Olivier L. de Weck, and Gary E. Mosier. Framework for
Multidisciplinary Integrated Modeling and Analysis of Space Telescopes. SPIE
Workshop on Integrated Modeling of Telescopes (Lund, Sweden), 4757, 2002.
[91] Eric Miller. X-Ray Imaging Spectrometer (XIS) CCD Performance Monitoring.
http://space.mit.edu/XIS/monitor/ccdperf/, October 2013.
[92] Max D. Morris. Factorial sampling plans for preliminary computational experiments. Technometrics, 33:161–174, 1991.
[93] Daniele Mortari. Search-less algorithm for star pattern recognition. Jornal of
Astronautical Sciences, 45(2):179–194, 1997.
[94] Daniele Mortari, John L. Junkins, and Malak A. Samaan. Lost-in-space pyramid
algorithm for robust star pattern recognition. Proceedings of the AAS Guidance
and Control Conference, 107(4):49–68, 2001.
[95] Mehrdad Moshir, David W. Murphy, David L. Meier, and Mark H. Millman.
Systems engineering and application of system performance modeling in SIM
Lite mission. Proc. SPIE, 7734(1H), 2010.
[96] Michael Müller and Franz Koch. System analysis tools for an ELT at ESO.
Proc. SPIE, 6271(1E), 2006.
[97] David W. Murphy and David L. Meier. SIMsim: an end-to-end simulator for
the SIM Mission. Proc. SPIE, 5497:577–585, 2004.
206
[98] Bijan Nemati and Mauricio J. Morales. The astrometric error budget. In
John M. Davidson, editor, SIM Lite Astrometric Observatory: From EarthLike Planets to Dark Matter, chapter 18, pages 183–194. NASA JPL 400-1360,
Pasadena, 2009.
[99] European
Southern
Observatory.
The
European
http://www.eso.org/sci/facilities/eelt/. Accessed 10-May-2012.
ELT.
[100] American Institute of Aeronautics and Astronautics. White Paper on Current
State of the Art. Technical Committee on Multidisciplinary Design Optimization, January 1991.
[101] Government Accountability Office. Improvements Needed in Space Systems
Acquisition Management Policy. GAO Report, GAO-03-107, September 2003.
[102] Government Accountability Office. Stronger Development Practices and Investment Planning Needed to Address Continuing Problems. GAO Report,
GAO-05-891, July 2005.
[103] Government Accountability Office.
Assessments of Selected Large-Scale
Projects. GAO Report, GAO-11-239, March 2011.
[104] Government Accountability Office. DOD Delivering New Generations of Satellites but Space System Acquisition Challenges Remain. GAO Report, GAO-11590, May 2011.
[105] Olivier L. de Weck. 16.842 Fundamentals of Systems Engineering. Course
lecture notes, Massachusetts Institute of Technology, September 2009.
[106] International Council on Systems Engineering. Systems Engineering Handbook
(version 3). INCOSE-TP-2003-002-03, June 2006.
[107] George G. Pavlov and John A. Nousek. Charge diffusion in CCD X-ray detectors. Nuclear Instruments and Methods in Physics Research A, 428:348–366,
1999.
[108] Gregory Prigozhin, Barry Burke, Marshall Bautz, Steve Kissel, Beverly LaMarr,
and Marat Freytsis. X-ray CCD with charge injection structure. Proceedings of
SPIE, 5501:357–365, 2004.
[109] Gregory Y. Prigozhin, Stephen Jones, Marshall W. Bautz, George R. Ricker,
and Stefan Kraft. The physics of the low-energy tail in the ACIS CCD The
spectral redistribution function. Nuclear Instruments and Methods in Physics
Research A, 39:582–591, 2000.
[110] Gregory Y. Prigozhin, Jonathan Woo, James A. Gregory, Andy H. Loomis,
Marshall W. Bautz, George R. Ricker, and Stefan Kraft. X-ray absorption
near edge structure in the quantum efficiency of x-ray charge-coupled devices.
Optical Engineering, 37(10):2848–2854, October 1998.
207
[111] Joseph B. Robinson. An integrated evolutionary model approach to small satellite engineering. SM thesis, Massachusetts Institute of Technology, Department
of Aeronautics and Astronautics, May 2010.
[112] Adam M. Ross. Multi-attribute tradespce exploration with concurrent design
as a value-centric framework for space system architecture and design. SM
thesis, Massachusetts Institute of Technology, Department of Aeronautics and
Astronautics, June 2003.
[113] Adam M. Ross, Daniel E. Hastings, Joyce M. Warmkessel, and Nathan P. Diller.
Multi-Attribute Tradespace Exploration as Front End for Effective Space System Design. Journal of Spacecraft and Rockets, 41(1), 2004.
[114] Günter Rudolph. Convergence of Evolutionary Algorithms in General Search
Spaces. In Proceedings of the Third IEEE Conference on Evolutionary Computation, pages 51–54, Nagoya, Japan, 1996.
[115] Kevin K. Ryu, Barry E. Burke, Hary R. Clark, Renee D. Lambert, Peter
O’Brien, Vyshnavi Suntharamingam, Christopher M. Wrad, Keith Warner,
Mark W. Bautz, Richard P. Binzel, Steve E. Kissel, and Rebecca A. Masterson.
Development of CCDs for REXIS on OSIRIS-REx. Proceedings of SPIE, 2014.
In press.
[116] Andrea Saltelli. Making best use of model valuations to compute sensitivity
indices. Computer Physics Communications, 145:280–297, 2002.
[117] Andrea Saltelli, Marco Ratto, Terry Andres, Francesca Campolongo, Jessica
Cariboni, Debora Gatelli, Michaela Saisana, and Stefano Tarantola. Global
Sensitivity Analysis. The Primer. John Wiley & Sons, Chichester, England,
2008.
[118] Andrea Saltelli, Marco Ratto, Stefano Tarantola, and Francesca Campolongo.
Sensitivity analysis practices: Strategies for model-based inference. Reliability
Engineering and System Safety, 91:1109–1125, 2006.
[119] Hermine Schnetler and William D. Taylor. An integrated systems approach for
the modelling of infrared instruments for the next generation telescopes. Proc.
SPIE, 7017(05), 2008.
[120] F. Scholze, H. Henneken, P. Kuschnerus, H. Rabus, M. Richter, and G. Ulm.
Determination of the electron-hole pair creation energy for semiconductors from
the spectral responsibivity of photodiodes. Nuclear Instruments and Methods
in Physics Research, 439(2–3):208–215, 2000.
[121] Edward R. D. Scott. Chondrites and the Protoplanetary Disk. Annual Review
of Earth and Planetary Sciences, 35(1):577–620, 2007.
208
[122] B. Sedghi, M. Müller, H. Bonnet, M. Esselborn, M. Le Louarn, R. Clare, and
F. Koch. E-ELT modeling and simulation toolkits: philosophy and progress
status. Proc. SPIE, 8336(06), 2011.
[123] B. Sedghi, M. Müller, F. Koch, and L. Pettazzi. Local Effects on E-ELT
Global Performance: Two Examples for Requirement Verification. Proc. SPIE,
8336(0O), 2011.
[124] Judith Segal. When Software Engineers Met Research Scientists: A Case Study.
Empirical Software Engineering, 10:517–536, 2005.
[125] Judith Segal. Scientists and software engineers: A tale of two cultures. Proceedings of the Psychology of Programming Interest Group (PPIG) Conference,
September 2008. University of Lancaster, UK.
[126] Stuart Shaklan, Mark Millman, Joseph Catanzarite, Ipek Basdogan, Miltiadis
Papalexandris, Lisa Sievers, and Raymond Swartz. Overview of SIM external
calibration. Proc. SPIE, 4852:623–633, 2003.
[127] Graeme B. Shaw. The Generalized Information Network Analysis Methodology.
PhD thesis, Massachusetts Institute of Technology, Department of Aeronautics
and Astronautics, 1998.
[128] Graeme B. Shaw, David W. Miller, and Daniel E. Hastings. Development of
the Quantitative Generalized Information Network Analysis Methodology for
Satellite Systems. Journal of Spacecraft and Rockets, 38(2):257–269, 2001.
[129] U.S. Air Force Fact Sheet.
Space-Based Infrared System.
http://www.afspc.af.mil/library/factsheets/factsheet print.asp?fsID=3675.
Accessed 21-May-2012.
[130] Herbert A. Simon. Rational Choice and the Structure of the Environment.
Psychological Review, 63(2):129–138, 1956.
[131] Gerald K. Skinner. Imaging with Coded-Aperture Masks. Nuclear Instruments
and Methods in Physics Research, 221:33–44, 1984.
[132] Ilya M. Sobol. Sensitivity Estimates for Nonlinear Mathematical Models. Mathematical Modeling and Computational Experiments, 1:407–414, 1993.
[133] R. E. Steuer. Multiple Criteria Optimization: Theory, Computation and Application. John Wiley & Sons, New York, New York, 1986.
[134] Genichi Taguchi. Introduction to quality engineering: designing quality into
products and processes. Asian Productivity Organization, Tokyo, 1986.
[135] M. Tamiz, D. F. Jones, and E. El-Darzi. A review of Goal Programming and
its applications. Annals of Operations Research, 58:39–53, 1995.
209
[136] Yasuo Tanaka, Hajime Inoue, and Stephen S. Holt. The X-Ray Astronomy
Satellite ASCA. Publications of the Astronomical Society of Japan, 46:L37–
L41, 1994.
[137] Albert Thompson, David Attwood, Eric Gullikson, Malcolm Howells, KwangJe Kim, Janos Kirz, Jeffrey Kortright, Ingolf Lindau, Yanwei Liu, Piero Pianetta, Arthur Robinson, James Scofield, James Underwood, Gwyn Williams,
and Herman Winick. X-Ray Data Booklet, Table 1-2. Center for X-Ray Optics.
Lawrence Berkeley National Laboratory, Berkeley, California, third edition, October 2009.
[138] L. K. Townsley, P. S. Broos, G. Chartas, E. Moskalenko, J. A. Nousek, and G. G.
Pavlov. Simulating CCDs for the Chandra Advanced CCD Imaging Spectrometer. Nuclear Instruments and Methods in Physics Research A, 486:716–750,
2002.
[139] Scott A. Uebelhart. Non-Deterministic Design and Analysis of Parameterized
Optical Structures during Conceptual Design. PhD dissertation, Massachusetts
Institute of Technology, June 2006.
[140] Defense Acquisition University. ACQuipedia Online Acquisition Encyclopedia.
https://acc.dau.mil/ILC KPP. Accessed 30-May-2012.
[141] Stephen C. Unwin. SIM Science Operations. Proc. SPIE, 4852:172–183, 2003.
[142] Stephen C. Unwin. Observing with sim lite. In John M. Davidson, editor,
SIM Lite Astrometric Observatory: From Earth-Like Planets to Dark Matter,
chapter 17, pages 173–182. NASA JPL 400-1360, Pasadena, 2009.
[143] Stephen C. Unwin et al. Taking the Measure of the Universe: Precision Astrometry with SIM PlanetQuest. Publications of the Astronomical Society of
the Pacific, 120(5):38–88, January 2008.
[144] Myles A. Walton. Identifying the impact of modeling and simulation in the
generation of system level requirements. SM thesis, Massachusetts Institute of
Technology, Department of Aeronautics and Astronautics, June 1999.
[145] Martin C. Weisskopf. Chandra x-ray optics. Optical Engineering, 51(1), January
2012.
[146] Eric W. Weisstein. Hypersphere. MathWorld–A Wolfram Web Resource, 2014.
http://mathworld.wolfram.com/Hypersphere.html.
[147] R. C. Westhoff, B. E. Burke, H. R. Clark, A. H. Loomis, D. J. Young, J. A. Gregory, and R. K. Reich. Low-Dark-Current, Back-Illuminated Charge-CoupledDevices. Proceedings of SPIE, 7249, January 2009.
210