Week 7 - Systems Engineering and Analysis –
Making it work together!
Buede
Chapter 10 – Interface Design
Chapter 11 – Integration and Qualification
(Side order of Rapid Development)
1
At some point in big system
building…
• Somebody has to tie it all together!
That would be
a systems
engineering
job!
We’ll revisit
this figure…
2
Both authors do talk a lot about
interfaces to users, vs systems
• E.g., in Wasson, “user” concerns are mentioned all
over:
• Ch 6 – System Acceptability
– Client is the “acquirer,” vs the real users
• Ch 10 – There is a “PERSONNEL System Element”
• Ch 11 – The operating environment depends on the
missions the User plans to accomplish.
• Ch 17 – all about use cases.
3
3
Interfaces to other “things”
• A connection for ‘hooking together’ system
components – external or internal.
• Have a logical and physical component
responsible for carrying information or
electromechanical energy from one
component to another.
•Must identify interfaces,
evaluate options, and
allocate inputs/outputs to
them.
4
Approaches to Interface
Considerations
• Buede’s Ch. 10 – somewhat limited in scope
to computer/software interfaces.
– (Good for us software types!)
• Wasson – detailed
• Schindel
– Some tips, added to these slides
• Rapid Prototyping
5
Wasson on Interfaces
•
Interfaces are all about how systems and
subsystems interact with each other and their
operating environment.
•
Interface Purposes
1.
Physically link or bind two or more system elements
2.
Adapt one or more incompatible system elements
3.
Buffer the effects of incompatible system elements
4.
Leverage human capabilities
5.
Restrain a system element and its usage
6
Broad View of Interfacing
• Wide variety of ‘interfaces’ in all systems and
products.
– Software
– Communication
– Electrical
– Mechanical
– Optical
– Acoustical
– Chemical
– Biological
– Etc…
7
Logical vs. Physical
• Logical
– Direct or indirect
association between two
entities
– Who communicates
• Physical
– How communication
happens
– ‘Specialized’
interfaces
– What scenarios
– When to communicate
8
Standardized vs Dedicated
• Standardized – standard, modular,
interchangeable interfaces – complying
with a ‘standard’ like RS-232, USB,
Ethernet, etc.
• Dedicated – Unique, dedicated interfaces –
often limiting compatibility with other
systems
9
Wasson Interface Thoughts
• Consider interactions between system and environment.
• Often focus on a few, not all interactions
• Environmental effects are natural, induced, and human-made.
10
Where are the Interfaces?
USED AT:
George Mason
University
AUTHOR: SYST 520 Student
PROJECT: Automatic Teller Machine
DATE: 09/28/99
REV:
x
NOTES: 1 2 3 4 5 6 7 8 9 10
Customer Needs
Main Menu
A-11
A-1
Activity Selection,
Account Type,
Deposit Type,
Deposit of Funds,
Trans Amount,
Source Account,
Dest Account,
Source of Payment,
Payment on Account,
Request to Cancel,
Choice to End
Request for Unique ID,
Request for Activity,
Request for Account Type,
Request for Deposit Type,
Physical Means for Insert,
Receipt, Request for Amount,
Request Denied, Cust Cash,
Request for Source Account,
Request for Dest Account
Customers
NODE:
Clim
Safety
Regulations
DATE CONTEXT:
READER
None
Banking
Policies
General ID,
Unique ID
Perform
Customer
Activities
Choice, ATM Reset,
No Input Device,
Request for ID #2,
Request for ID #3,
Customer Alert, Apology,
Request for Paymt Source
Americans with
Disabilities Act
WORKING
DRAFT
RECOMMENDED
PUBLICATION
Transaction,
Request for Fmax,
Request for Status Inf..,
Input Not Working,
Request for Funds,
Request for Receipts,
Break-in Attempted
At:
Inputs
Controls
Outputs
Employee
ID Code
Provide
ATM Services
Audible
Alarm,
Operation
Terminated
Provide Bank
Information to
ATM
A-0
Cust Status Inf..,
Fmax
A-12
Maintain ATM
A-13
Request to Open,
ATM Cash,
Blank Receipts,
Initialization,
Diagnostic Test,
ATM Fixes,
Request to Close
ATM System
TITLE: Operational Phase External Systems
Access Opportunity,
ATM Opened,
Cust Deposits,
Cust Payments,
Test Results,
Fixes Applied
ATM Closed
Bank
Computer
Rob
ATM
A-14
Break-in
Attempt
Bank
Employees
NUMBER:
Robber
P. 1
11
Where are the Interfaces ?
USED AT:
George Mason
University
AUTHOR: SYST 520 Student
PROJECT: Automatic Teller Machine
x
DATE: 08/07/00
REV:
NOTES: 1 2 3 4 5 6 7 8 9 10
Employee General ID,
ID Code Unique ID
Americans with
Disabilities Act
Safety
Regulations
Customer
Valid
Provide
Access to
ATM
Cust Activity,
Cust A/C Type,
Deposit Entered,
Cust Amount,
Trans Source,
Trans Dest,
Paymt Source,
Payment Entered,
Cancel Entered,
Choice Entered
Accept
Customer
Requests and
Provide
Feedback
ID Received
Request for
Unique ID
Choice, ATM Reset,
Apology, Request
for Paymt Source
Employee
Valid
ID Validation
A2
Need for Fmax,
Trans Complete,
Receipts<25
Determine
ATM
Responses
DATE CONTEXT:
READER
Request for Activity,
Request for Account Type,
Request for Deposit Type,
Physical Means for Insert,
Receipt, Request for Amount,
Request Denied, Cust Cash,
Request for Source Account,
Request for Dest Account
Banking
Policies
Calculations
for
Withdrawal
A1
Activity Selection,
Account Type,
Deposit Type,
Deposit of Funds,
Trans Amount,
Source Account,
Dest Account,
Source of Payment,
Payment on Account,
Request to Cancel,
Choice to End
Clim
WORKING
DRAFT
RECOMMENDED
PUBLICATION
No Input Device,
Request for ID #2,
Request for ID #3,
Customer Alert
Request for Unique ID,
Request for Activity,
Request for Account Type,
Request for Deposit Type,
Physical Means for Insert,
Receipt, Request for Amount,
Request Denied, Cust Cash,
Request for Source Account,
Request for Dest Account
At:
Inputs
Controls
Outputs
Choice, ATM Reset,
No Input Device,
Request for ID #2,
Request for ID #3,
Customer Alert, Apology,
Request for Paymt Source
Account
FMax
Main Menu
Input not
Available
Creq>Cleft
Account Balance
Request to Open,
ATM Cash,
Blank Receipts,
Initialization,
Diagnostic Test,
ATM Fixes,
Request to Close
Cust Status Inf..,
Fmax
A3
Transaction,
Request for Fmax,
Request for Status Inf..,
Input Not Working,
Request for Funds,
Request for Receipts,
Break-in Attempted
Communicate
with Bank
Computer
A4
Balance Inf.,
Paymt Source Entered,
Payment Received,
Ptrns>Fmax,
Cancel Received,
Choice Received
Break-in
Attempt
Activity Selected,
A/C Type Entered,
Deposit Type Entered
Deposit Received,
Amount Entered,
Source A/C Entered,
Dest A/C Entered,
Ftrns>Fmax
Access Opportunity,
ATM Opened,
Cust Deposits,
Cust Payments,
Test Results,
Fixes Applied
ATM Closed
Enable
Re-Supply and
Maintenance
Need to Open,
Paymts Inserted,
Deposits Inserted,
Diagnostics,
Fixes to ATM
Need to Close
A5
Respond to
Hostile
Situations
A6
Audible
Alarm,
Operation
Terminated
Attempted Break-in
NODE:
A0
TITLE:
Provide ATM Services
NUMBER:
P. 3
12
Interface
Design
Process
The same SE process !!
Figure 10.7
Define Interface Requirements
Identify the Items to Be Transported by the Interface
Define the Operational Concept
Bound the Problem with an External Systems Diagram
Define the Objectives Hierarchy
Write the Requirements
Select a High-Level Interface Architecture of Interface
Identify Several Candidate Architectures
Define Trial Interfaces for Each Candidate
Evaluate Alternatives against Requirements
Choose High-Level Interface Architecture
Develop Functional Architecture of Interface
Specify Functional Decomposition
Add Inputs and Outputs
Add Fault Detection and Recovery Functions
Develop Physical Architecture of Interface
Identify Candidates based upon High-Level Architecture
Eliminate Infeasible Candidates
Develop Operational Architecture of Interface
Allocate Functions to Components of the Interface
Analyze Behavior and Performance of Alternatives
Select Alternative
Document Design and Obtain Approval
Add Functions to Components Connected to Interface as Needed
14
Benefits of Standards
• Interchangeability - ability to interchange
components with different performance and cost
characteristics
• Interoperability - the system can now operate
with a wider variety of external systems, systems
that have also adopted the same conventions
• Portability - systems can be moved and operate on
other systems
• Reduced cost and risk for equivalent performance
• Increased life cycle is possible when long-lived
standards are adopted
16
(Bill) Schindel Interface Thoughts
• List of all I/O that pass through it
• List of functions or behaviors at the
interface
• Association with a system that owns the
interface
• SOA – technologies or systems that
support the interface
• An interface does not reside ‘between’
two systems.
17
Schindel Interface Thoughts, cntd
• List of all I/O that pass through it
• List of functions or behaviors at the interface
• Association with a system that owns the interface
• SOA – technologies or systems that support the interface
• An interface does not reside ‘between’ two systems.
Additional Thoughts:
All I-C-O to a function must be associated with an
interface.
In addition to the function/behavior, also list the
physical characteristics.
18
Interface Examples
19
Interface Template
Functional
or Physical
Element
Signals
Input
or
Output
Interface
Behavior
Key
Interface
Attributes
SOA
Physical
Description
Applicable
Standards
(Wasson 43.1)
20
Defining and Managing Interfaces
• IRS – Interface Requirements
Specification (not always necessary)
• ICWG – Interface Control Working Group
• ICD – Interface Control Document (Hdwe)
• IDD – Interface Design Description
(Sftwe)
30
Schindel Interface Thoughts - Reprise
• List of all I/O that pass through it
• List of functions or behaviors at the
interface
• Association with a system that owns the
interface
• SOA – technologies or systems that
support the interface
• An interface does not reside ‘between’
two systems.
31
Key
Interface
Attributes
32
Interface Definition and Control
Challenges – Pg 519
33
Oasis Example
Source
Building
Building
User
User
Support
Repair
Repair
Repair
Repair
Support
User
User
User
User
User
User
User
User
User
Repair
Home Office
Home Office
Interface
Electrical Energy
Network
Selection
Money
Soda
Repair
Materials
Diagnostic Tests
Place into Service
Temperature Setting
Out-of-Service
Money Received
Request Selection
Selection not Available
Soda Dispensing
Fill Status Indicator
Money
Change
Soda
Diagnosis Response
Soda Quantity Notification
Out-of-Service Notification
What is Good ?
In/Out
In
In/Out
In
In
In
In
In
In
In
In
Out
Out
Out
Out
Out
Out
Out
Out
Out
Out
Out
Out
Frequency
continuous
as needed
as needed
as needed
as needed
as needed
as needed
as needed
as needed
as needed
as needed
as needed
as needed
as needed
as needed
as needed
on command
as needed
as needed
on command
as needed
as needed
Owner
Building
Oasis
User
User
User
User
User
User
User
User
Oasis
Oasis
Oasis
Oasis
Oasis
Oasis
Oasis
Oasis
Oasis
Repair Tech
Oasis
Oasis
SOA
Electrical
Internet
Keypad
Cash System
Manual
Manual
Manual
Maint Keypad
Manual
Maint Keypad
Status Indicator
Status Indicator
Status Indicator
Status Indicator
Status Indicator
Status Indicator
Cash System
Cash System
Cash System
Status Indicator
Internet
Internet
What is Bad ?
34
Context of ‘Rapid Development’
• ‘For us – “Agile” and “Lean”
• Designing Products in Half the Time’,
Reinertsen
• Keys –
• Functional decomposition and allocation
– Modularity – more rapid development, but more
modules, more interfaces.
• Performance margin in each module.
– More margin, fewer changes, but higher cost.
35
‘Rapid Development’-2
• Keys –
• Interface Design
– Link modules and functions together.
– Stable, robust, standard, simple.
– Robust – performance, electrically,
mechanically, environmentally, etc.
– Standard – faster, cheaper.
36
‘Rapid Development’-3
• Goal –
• Design ‘Architectures’ fast, early.
• Modular design with known interfaces.
• Flow down requirements to interfaces.
• Then - Work on modules independently,
concurrently.
37
‘Rapid Development’-4
• Standard Interface– Need ‘interface’ to carry 110 VAC power from
a power source to electrical system under
development.
• Standard interface – buy at hardware store
(COTS) (commercial off the shelf).
• Custom interface – design, develop,
manufacture, safety/life testing, etc.
38
Interface Thoughts
• An interface is a property of a system
component, it does not reside between
two systems. – Bill Schindel
• The distinction between :
– The Interface,
– The Physical Component,
– The ‘Standard’ involved, if any.
A car battery is a standard interface to
provide electrical power to the car….(?)
39
A Car Battery
• What are the external
systems ?
• What are the inputs and
outputs ?
• An interface at each
one !
40
Interface Implications
• More interfaces – more modular,
upgradeable, testable, but more expense.
• But, systems tend to fail at an interface
– solder joint, connector, bolted joint, data
transfer between modules, etc.
• May ‘over-design’ each module
41
The ATM Functional Architecture
USED AT:
George Mason
University
AUTHOR: SYST 520 Student
PROJECT: Automatic Teller Machine
x
DATE: 08/07/00
REV:
NOTES: 1 2 3 4 5 6 7 8 9 10
Employee General ID,
ID Code Unique ID
Americans with
Disabilities Act
Safety
Regulations
Customer
Valid
Provide
Access to
ATM
Cust Activity,
Cust A/C Type,
Deposit Entered,
Cust Amount,
Trans Source,
Trans Dest,
Paymt Source,
Payment Entered,
Cancel Entered,
Choice Entered
Accept
Customer
Requests and
Provide
Feedback
ID Received
Request for
Unique ID
Choice, ATM Reset,
Apology, Request
for Paymt Source
Employee
Valid
ID Validation
A2
Need for Fmax,
Trans Complete,
Receipts<25
Determine
ATM
Responses
DATE CONTEXT:
READER
Request for Activity,
Request for Account Type,
Request for Deposit Type,
Physical Means for Insert,
Receipt, Request for Amount,
Request Denied, Cust Cash,
Request for Source Account,
Request for Dest Account
Banking
Policies
Calculations
for
Withdrawal
A1
Activity Selection,
Account Type,
Deposit Type,
Deposit of Funds,
Trans Amount,
Source Account,
Dest Account,
Source of Payment,
Payment on Account,
Request to Cancel,
Choice to End
Clim
WORKING
DRAFT
RECOMMENDED
PUBLICATION
No Input Device,
Request for ID #2,
Request for ID #3,
Customer Alert
Request for Unique ID,
Request for Activity,
Request for Account Type,
Request for Deposit Type,
Physical Means for Insert,
Receipt, Request for Amount,
Request Denied, Cust Cash,
Request for Source Account,
Request for Dest Account
Choice, ATM Reset,
No Input Device,
Request for ID #2,
Request for ID #3,
Customer Alert, Apology,
Request for Paymt Source
Account
FMax
Main Menu
Input not
Available
Creq>Cleft
Account Balance
Request to Open,
ATM Cash,
Blank Receipts,
Initialization,
Diagnostic Test,
ATM Fixes,
Request to Close
Cust Status Inf..,
Fmax
A3
Transaction,
Request for Fmax,
Request for Status Inf..,
Input Not Working,
Request for Funds,
Request for Receipts,
Break-in Attempted
Communicate
with Bank
Computer
A4
Balance Inf.,
Paymt Source Entered,
Payment Received,
Ptrns>Fmax,
Cancel Received,
Choice Received
Break-in
Attempt
Activity Selected,
A/C Type Entered,
Deposit Type Entered
Deposit Received,
Amount Entered,
Source A/C Entered,
Dest A/C Entered,
Ftrns>Fmax
Access Opportunity,
ATM Opened,
Cust Deposits,
Cust Payments,
Test Results,
Fixes Applied
ATM Closed
Enable
Re-Supply and
Maintenance
Need to Open,
Paymts Inserted,
Deposits Inserted,
Diagnostics,
Fixes to ATM
Need to Close
A5
Respond to
Hostile
Situations
A6
Audible
Alarm,
Operation
Terminated
Attempted Break-in
NODE:
A0
TITLE:
Provide ATM Services
NUMBER:
P. 3
• Define the Interface at each I-C-O.
• Define the logical/functional behavior.
• Define the physical and standard/custom instantiations.
42
Class Exercise – to think about
•
Consider the following systems. Further
consider the typical subsystems which make
them up and identify interfaces necessary to
support the system.
•
Use the Interface Template as a guide to
describe the interface behavior and
characteristics. Note that several signals may
pass through the same interface.
1.
2.
3.
4.
5.
6.
7.
Elevator System (*)
Desktop PC
Machine Tool
This Classroom
Digital Camera
Your Car
Your Project System …..
43
Interface Template
Functional
or Physical
Element
Signals
Input
or
Output
Interface
Behavior
Key
Interface
Attributes
SOA
Physical
Description
Applicable
Standards
(Wasson 43.1)
44
Week 7 - Systems Engineering
and Analysis, cntd
Buede Chapter 11 –
Verification/Validation,
Integration and Qualification
45
Definitions
• Integration is the process of assembling the
system from its components, which must be
assembled from their configuration items (CIs)
• Qualification is the process of verifying and
validating the system design and then obtaining
the stakeholders’ acceptance of the design.
• Qualification = ‘Testing’.
– Verification is the determination that the system was
built right (more of a process focus)
– Validation determines that the right system was built
(more of a product focus)
46
1
Verification, 2
Conceptual
Validation Requirements Validity
Validity
and
Operational Concept
Acceptance
Originating Requirements
Stakeholders’
Needs
6
Acceptability
5
Operational
Validity
System Delivered
System Requirements
3
Design
Validity
Element Specs
Developmental
Verification
Elements Delivered
4
Segment Specs
Component Specs
Verification = Built Right
Validation = Right System
Acceptance = Stakeholder OK
CI Specs
Segments Delivered
Components Delivered
CIs Delivered
Systems Engineering
SE Vee
Design
Engineering
Time
Figure 11.1
47
Verification/Validation/Acceptance
• Do not happen sequentially
• Do not only happen late in the SE process
50
PUBLICATION
Derived &
Originating
Requirements
Qualification Procedures,
Activities, & Models
Major
Integration
Functions for
Component
Integration
Component Level
Design Documents
"Built-to"
CIs
Perform
Component
Integration &
Verification
Originating
& System
Requirements
Documents
Subsystem Level
Design Documents
Component
Verification
Changes
"Built-to"
Components
A221
Perform
Subsystem
Integration &
Verification
SubsystemGenerated
Component
Regression
Qualification
Perform
System
Integration &
Verification
NODE:
SystemGenerated
Subsystem
Regression
Qualification
AUTHOR: Dennis Buede
PROJECT: Engineering of a System
System
Verification
Document
Verification
Data
A223
Subsystem
Test
Results
DATE: 11/04/98
REV:
NOTES: 1 2 3 4 5 6 7 8 9 10
TITLE:
A22 Component Level
Subsystem
Verification
Changes
Component
Test
Results
"Built-to"
Subsystems
SystemGenerated
Component
Regression
Qualification
CI Test
Results
Subsystem
Verification
Documents
Component
Verification
Documents
Verification
Document
A222
USED AT:
GMU Systems
Engineering
Program
Verification
Changes
CI Verification
Changes
x
Conduct Integration & Verification
WORKING
DRAFT
RECOMMENDED
PUBLICATION
NUMBER:
Qualification Procedures,
Activities, & Models
Design Documents
DATE CONTEXT:
READER
System-Level
Reqression
Qualification
P. 16
"Built-to"
CIs
SubsystemGenerated
Component
Regression
Qualification
SystemGenerated
Component
Regression
Qualification
Inspect
&
Verify CI
CI Test
Results
Deficient
CI
A2211
Discrepancy
Reports
Identify & Fix
Correctable
CI
Deficiencies
A2212
Uncorrected CI
Corrected CI
Impact
Statement
Acceptable
Impact
Assess
Impact of
Uncorrectable
CI
Deficiencies
CI
Engineering
Changes
A2213
Unacceptable
Impact
Cleared
CI
Redesign
CI
Redesigned CI
CI Verification
Changes
Baseline
Changes
A2214
Modify CI
Baseline
Approval to
Continue
Integration
Component
Verification
Documents
A2215
Cleared CI
Integrate
with
Next CI
"Built-to"
Components
A2216
NODE:
Figure 11.4
TITLE:
The Textbook Approach is ‘Bottom Up’
A221
Perform Component Integration & Verification
NUMBER:
P. 17
51
Approaches to Integration
1. Top Down
2. Bottom Up (*)
3. Big Bang
54
Top Down Integration
Integration begins with a major or top-level module.
All modules called from the top-level module are simulated by “stubs”
(shell or model replica).
Once the top-level module is qualified, actual modules replace the stubs
until the entire system has been qualified.
This is most useful for systems using large amounts of COTS components.
Phase Integration: Integration is done from the top down to the lowest level; one peel of the onion at a
time.
Incremental Integration: Integration is done for a specific module from top to bottom; one slice of the
system at a time.
Advantage
•
Early demonstration of the system is allowed.
•
Representation of the test cases is easier.
•
More productive if major flaws occur toward the top of system.
Disadvantage
•
Stubs have to be developed.
•
Representation of test cases in the stubs may be difficult.
•
Observation of test output may be artificial and difficult.
•
This requires a hierarchical system architecture.
Table 11.1
55
Bottom-up Integration
Integration begins with the elementary pieces (or CIs) of the system.
After each CI is tested, components comprising multiple CIs are tested.
This process continues until the entire system is assembled and tested.
This is the traditional systems engineering integration approach.
Phase Integration: At any point in the integration, all of the subsystems are at the same
stage of integration testing.
Incremental Integration: proceeds one slice of the system at a time.
Advantage
•
It is easier to detect flaws in the tiniest pieces of the system.
•
Test conditions are easier to create.
•
Observation of the test results is easier.
Disadvantage
•
“Scaffold” systems must be produced to support pieces as they are integrated.
•
System’s control structure cannot be tested until the end.
•
Major errors in the system design are typically not caught until the end.
•
System does not exist until the last integration test is completed.
•
This requires a hierarchical system architecture.
Table 11.1
56
Big Bang Integration
Untested CIs are assembled and the combination is tested.
This is a commonly used and unsuccessful approach.
Advantage
• Immediate feedback on the status of system elements is
provided.
• Little or no pretest planning is required.
• Little or no training is required.
Disadvantage
• Source of errors is difficult to trace.
• Many errors are never detected.
• Test it until it ‘works’ then stop.
• Errors found by customers
Table 11.1
57
Bottom-Up
Integration of
RC Car
Controller
Directional Interface
Interface Module
Motion Interface
Structural Design
Controller Processor
Processor Logic (Software)
CPU Module
Integrated Controller
Subsystem
Signal Transmission
Structural Design
Power Control
Power Confirmation
Power Module
Battery Interface
Fully Integrated
Car/Controller
System
Car
Signal Receiver
Signal Processor
CPU Module
Processor Logic (Software)
Steering
Alignment
Motion Module
Locomotion
Structural Design
Integrated Car
Subsystem
Structural Design
Power Control
Power Confirmation
Power Module
Battery Interface
Phase 1
Phase 2
Phase 3
Phase 4
58
Exercise
1.
: Discussion of Examples
How did they do integration on the Denver
Airport project.
2. Even when large amounts of time/money are
spent on integration/qualification, how are
mistakes still made – Genesis and other space
vehicle failures ?
3. Why is the Big Bang approach so popular or often
used?
4. What integration approach would you recommend
for your project ?
59
Qualification Planning During Design
• The design of the qualification
system is not only important in terms
of finding and defining faults and
errors but also in guiding designers to
preclude them from introducing
faults in the first place. Buede
60
Qualification –
The ‘Readers Digest’ Edition
1. If you can't test it, don't build it.
2. If you don't test it, rip it out.
•
Boris Beizer, Second edition, chapter 13, section 3.2.5. ,
"Software testing techniques" by Boris Beizer , ISBN:
0442206720
61
One Way to Look at it
USED AT:
George Mason
University
AUTHOR: SYST 520 Student
PROJECT: Automatic Teller Machine
DATE: 09/28/99
REV:
x
NOTES: 1 2 3 4 5 6 7 8 9 10
Customer Needs
Main Menu
A-11
Activity Selection,
Account Type,
Deposit Type,
Deposit of Funds,
Trans Amount,
Source Account,
Dest Account,
Source of Payment,
Payment on Account,
Request to Cancel,
Choice to End
Request for Unique ID,
Request for Activity,
Request for Account Type,
Request for Deposit Type,
Physical Means for Insert,
Receipt, Request for Amount,
Request Denied, Cust Cash,
Request for Source Account,
Request for Dest Account
Customers
A-1
None
Banking
Policies
Transaction,
Request for Fmax,
Request for Status Inf..,
Input Not Working,
Request for Funds,
Request for Receipts,
Break-in Attempted
Audible
Alarm,
Operation
Terminated
Provide Bank
Information to
ATM
A-0
Cust Status Inf..,
Fmax
A-12
Maintain ATM
A-13
Rob
ATM
Qualify ATM
Access Opportunity,
ATM Opened,
Machine
Request to Open,
ATM Cash,
Blank Receipts,
Initialization,
Diagnostic Test,
ATM Fixes,
Request to Close
ATM System
TITLE:
Employee
ID Code
Provide
ATM Services
Develop a Systems
Model for this
Function
NODE:
Clim
Safety
Regulations
DATE CONTEXT:
READER
General ID,
Unique ID
Perform
Customer
Activities
Choice, ATM Reset,
No Input Device,
Request for ID #2,
Request for ID #3,
Customer Alert, Apology,
Request for Paymt Source
Americans with
Disabilities Act
WORKING
DRAFT
RECOMMENDED
PUBLICATION
Operational Phase External Systems
Cust Deposits,
Cust Payments,
Test Results,
Fixes Applied
ATM Closed
Bank
Computer
A-14
Break-in
Attempt
Bank
Employees
NUMBER:
Robber
P. 1
62
Circuit
Board
Testing
Qualification an Afterthought
63
Circuit
Board
Testing
Qualification planned and designed
64
Failure Definitions
• Failure: deviation in behavior between the system and its
requirements. Since the system does not maintain a copy of
its requirements, a failure is not observable by the system.
• Error: a subset of the system state, which may lead to a
failure. The system can monitor its own state, so errors are
observable in principle. Failures are inferred when errors
are observed. Since a system is usually not able to monitor
its entire state continuously, not all errors are observable.
As a result, not all failures are going to be detected
(inferred).
• Fault: defects (bugs) in the system that can cause an error.
Faults can be permanent (e.g., a failure of system component
that requires replacement) or temporary due to either an
internal malfunction or external transient. Temporary faults
may not cause a sufficiently noticeable error or may cause a
permanent fault in addition to a temporary error.
65
Levels of Bug/Fault
Severity and Consequences
• Mild
• Moderate
• Annoying
• Disturbing
• Serious
• Very Serious
Levels of coming up
“Martin Short”
• Extreme
• Intolerable
• Catastrophic
• Infectious
66
Boris Beizer’s 3 Laws of Software Testing
• First Law: The Pesticide Paradox – Every method you use to
prevent or find bugs leaves a residue of subtler bugs against
which those methods are ineffective.
• Second Law: The Complexity Barrier – Software complexity
(and therefore that of bugs) grows to the limits of our
ability to manage that complexity.
– Test developers are no ‘smarter’ than code developers. Errors on both
sides.
• Third Law – Code migrates to data.
– More and more of the control and processing functions are stored in
tables. Control tables having hidden programming languages,
generalized packages with parametric data to the code, etc.
– Because of this law there is increasing awareness that bugs in code are
only half the battle and that data problems should be given equal
attention.
67
Rate of Finding Bugs
100% Bugs
Found
Harder to Find
Very Difficult to Find,
Intermittent / Infrequent
Time or Configuration Based
Easy to Find,
Obvious,
Frequent
Time – Months and Years
68
Qualification Planning Functions
• Plan the qualification process
– Acceptance test
– Validation test (Built right system ? – Product)
– Verification test (System built right? – Process)
• Plan the qualification approaches
• Plan qualification activities
• Plan specific tests
Table 11.2
70
What do we need to Qualify ?
• Remember we defined ‘qualification’
requirements. (The ‘what’ to qualify)
• All Input and Output requirements ?
• All Technology and Systemwide
requirements ?
71
Plan the qualification process
· Review system objectives
· Identify qualification system objectives
· Identify pass/fail thresholds
· Define qualification operational concepts
· Define qualification requirements
· Define qualification functional architecture
· Define qualification generic physical architecture
· Generate qualification coverage matrices (allocate
requirements to functional architecture and functions to the
generic physical architecture)
· Identify risks and mitigation strategies
· Create master qualification plan
Table 11.2
72
Plan the qualification approaches
· Define qualification resources and
organizations (instantiated physical
architecture)
· Assign qualification activities to
organizations
· Allocate qualification activities to
resources
· Develop qualification schedules consistent
with development schedule
Table 11.2
73
Plan qualification activities
· Develop detailed derived qualification
requirements
· Develop functional architectures for qualification
components
· Generate coverage matrices (allocate derived
requirements to functional architectures and
functions to physical architectures)
· Write activity-level qualification plans for each
qualification component
· Assign qualification responsibilities
Table 11.2
74
Plan specific tests
· Identify required stimulation data for each
activity
· Create test scenarios
· Write test procedures
· Write analysis procedures
· Define test and analysis schedules
Table 11.2
75
Qualification Methods
Method
Inspection
(static test)
Description
Compare system
attributes to requirements.
Analysis and
simulation
Use models that represent
some aspect of the
system. Examples of
models might address
system’s environment,
system process, system
failures.
Use calibrated
instruments to measure
system’s outputs.
Examples of calibrated
instruments are
oscilloscope, voltmeter,
LAN analyzer.
Exercise system in front of
unbiased reviewers in
expected system
environment.
Instrumented
test
Demonstratio
n or field test
Table 11.3
Used During:
During all segments of
verification, validation, and
acceptance testing for
requirements that can be
addressed by human
examination.
Used throughout
qualification, but emphasis
is early in verification and
during acceptance.
Often used in conjunction
with demonstration.
Verification testing.
Primarily used for
validation and acceptance
testing.
Most Effective When:
Success or failure can be judged by
humans; examples include inspection of
physical attributes, code walkthroughs, and
evaluation of user’s manuals.
Physical elements are not yet available.
Expense prohibits instrumented test, and
demonstration is not sufficient.
Issue involves all or most of the system’s
life span.
Issue cannot be tested (e.g., survive
nuclear blast).
Engineering test models through system
elements are available.
Detailed information is required to
understand and trace failures.
Life and reliability data is needed for
analysis and simulation.
Complete instrumented test is too
expensive.
High-level data/information is needed to
corroborate results from analysis and
simulation or instrumented test.
76
Testing Methods
Functional testing Test conditions are set up to ensure that the correct outputs are
produced, based upon the inputs of the test conditions. Focus is on
whether the outputs are correct given the inputs (also called black box
testing).
Structural testing
Examines the structure of the system and its proper functioning.
Includes such elements as performance, recovery, stress, security,
safety, availability. Some of the key elements are described below.
Performance
Examination of the system performance under a range of nominal
conditions, ensures system is operational as well.
Recovery
Various failure modes are created and the system’s ability to return to
an operational mode is determined.
Interface
Examination of all interface conditions associated with the system’s
reception of inputs and sending of outputs.
Stress testing
Above-normal loads are placed on the system to ensure that the
system can handle them; these above-normal loads are increased to
determine the system’s breaking point; these tests may proceed for a
long period of time in an environment as close to real as possible.
Table 11.4
77
Black & White Box Testing
Black box Outputs are determined correct or incorrect based upon inputs; inner
workings of the module are ignored. Both positive and negative testing
testing
have to be employed. This approach is scalable to system-level testing.
 Positive testing pulls the test data and sequences from the
requirements documents.
 Negative testing attempts to find input sequences missed in the
requirements documents and then determine how the module reacts.
Crash testing is an example.
White box Inner workings of the module are examined as part of the testing to
ensure proper functioning. Usually used at the CI level of testing; this
testing
method becomes impractical at the system level.
 Path testing addresses each possible simple functionality and is
based upon a prescribed set of inputs.
 Path domain testing partitions the input space and then examines the
outputs for each partition of the input space.
 Mutation analysis injects predefined errors and tests the error
detection and recovery functionalities.
Table 11.6
78
Acceptance Testing
• Final step in qualification
• Acceptance testing is conducted by
Stakeholders.
• Acceptance criteria must be defined early.
• Outcome : accept, accept with changes, not
accept.
• Key issue - How much to test
and what assumptions.
79
Another Way to Look at it
USED AT:
George Mason
University
AUTHOR: SYST 520 Student
PROJECT: Automatic Teller Machine
DATE: 09/28/99
REV:
x
NOTES: 1 2 3 4 5 6 7 8 9 10
Customer Needs
Main Menu
Clim
Safety
Regulations
A-11
None
Banking
Policies
Activity Selection,
Account Type,
Deposit Type,
Deposit of Funds,
Trans Amount,
Source Account,
Dest Account,
Source of Payment,
Payment on Account,
Request to Cancel,
Choice to End
Request for Unique ID,
Request for Activity,
Request for Account Type,
Request for Deposit Type,
Physical Means for Insert,
Receipt, Request for Amount,
Request Denied, Cust Cash,
Request for Source Account,
Request for Dest Account
Transaction,
Request for Fmax,
Request for Status Inf..,
Input Not Working,
Request for Funds,
Request for Receipts,
Break-in Attempted
Employee
ID Code
Provide
ATM Services
Audible
Alarm,
Operation
Terminated
Provide Bank
Information to
ATM
A-0
Cust Status Inf..,
Fmax
A-12
Maintain ATM
Qualify ATM
Machine
Request to Open,
ATM Cash,
Blank Receipts,
Initialization,
Diagnostic Test,
ATM Fixes,
Request to Close
•This becomes our new ‘system of interest’
•Its external systems are the ATM, Customer, etc.
•We have some requirements from the ATM itself
Customers
ATM System
•Build a new SE
model for it
•Places
needs and requirements
back on the ATM
NODE:
TITLE:
A-1
DATE CONTEXT:
READER
General ID,
Unique ID
Perform
Customer
Activities
Choice, ATM Reset,
No Input Device,
Request for ID #2,
Request for ID #3,
Customer Alert, Apology,
Request for Paymt Source
Americans with
Disabilities Act
WORKING
DRAFT
RECOMMENDED
PUBLICATION
Operational Phase External Systems
A-13
Access Opportunity,
ATM Opened,
Cust Deposits,
Cust Payments,
Test Results,
Fixes Applied
ATM Closed
Bank
Computer
Rob
ATM
A-14
Break-in
Attempt
Bank
Employees
NUMBER:
Robber
P. 1
81
The Qualification System as an
External System
• I/C/O from ATM function to Qualification
function.
• Can model and decompose the Qualification
function.
82
RC Car
Example
83
Test Equipment and Resources
Requirement ID
[ORG-SW: 2.6.3.006]
Requirement Description
The speed of the car shall be 15mph or more. The
design goal is 20mph.
Equipment
Controller
Car
Speed Radar
o
Controller
Car
o Fully Charged
Battery
o
Controller
Car
o Speed Radar
o Stop Watch
o
Car Springs
Certified Test
instrument for
measuring force
o Test fixture for
springs
o
o
o
[ORG-SW: 2.6.3.007]
[DER: 2.12.001]
[ORG-SW: 2.6.3.008]
Resources
Tester
Open Track
o 30 minutes of
testing
Test completed
By 11/10/07
Tester
20 minutes of
testing at full
throttle condition
Test completed
By 10/28/07
Tester
20 minutes of
testing at full
throttle condition
Test completed
By 11/10/07
Tester
Statistical
Sampling of 30
Piece capability
Test completed
By 10/15/07
o
The battery life of the car under constant full throttle
shall be 10 minutes or more. The design goal is 15
minutes.
o
The R/C car shall accelerate to maximum speed within
2 sec of engaging full throttle position.
o
The suspension of the car shall have a spring constant k
between 50lb/in and 60lb/in.
o
o
o
o
o
o
Timetable
o
o
84
Discussion
• What would a qualification plan look like
for an ATM system?
• What would a qualification plan look like
for an ATM manufacturing system?
• What would a qualification plan look like
for an elevator?
85
Discussion
• What qualification approach and plan is
needed for your project ?
• Do most engineers view qualification as a
key part of the development process ?
• What are some of the reasons that make
qualification process difficult to do well ?
86