Test Integration
Strategies
Brian Nielsen
[email protected]
Center of Embedded Software Systems
Aalborg University, Denmark
1010111011010101
1011010101110111
C SS
V - Model
requirements
acceptance test spec
acceptance
test
system test spec
system
test
specification
architecture
spec
integration test spec
module test spec
detailed design
implementation
code
integration
test
module
test
unit
test spec
unit-test
CISS
Unit test
Component Hierarchy
Driver: a code fragment that
invokes the U-ut Simulate
A
•Supplies input data
•Invokes Unit
•Collects and compares results
•~xUnit testing frame work
•Function call
•Method invocation
•Message passing
•Shared variables
test driver
Unit
under test
test stub
test stub
B
E
C
F
D
G
•C-function
•Abstract Data Type
•Object
•Process / thread
Stub: a code fragment that
simulates a missing unit
•Simulate
•Pre-programmed return values
•Simplified impl.
CISS
Integration Testing
• Goal: Systematic construction of program
structure and testing for interfacing errors
• Loss of data/information across interfaces
• int f(){ … g(y,x)…; }
Int g(int x,y) {};
• Side-effects (shared data-structures)
• Accumulation of errors (imprecision)
• Incremental
• Consider required cost and thoroughness
• Mostly blackbox, supplemented with whitebox tests to ensure coverage
CISS
Big-bang integration
Component Hierarchy
1. Test all components in isolation
2. Integrate all in one step
A
B
E
C
F
D
G
Integration Order
Test
A
-
Test
E
Stubs and drivers needed!
Difficult to localize cause of errors
Difficult to distinguish interface faults
Test
B
Test
F
Test
C
Test
D
Test
G
Test
A,B,C,D
E,F,G
CISS
Buttom-Up integration
Component Hierarchy
1. Test lower levels first (utility routines)
2. Implement driver (easy) to invoke module
3. Use tested modules as stubs
E
A
B
C
F
D
G
Integration Order
Test
E
Test
B,E,F
Test
F
Test
C
Test
G
+No stubs needed!
-Mundane low-level utilities tested first, but most
important top level modules tested late
•
Critical errors found late
Test
•
Main design problems are at higherA,B,C,D
levels
E,F,G
•
Timing determined by high-level
modules
Test
D,G
CISS
Top-Down integration
Component Hierarchy
1. Test higher levels first (controlling modules)
2. Implement stubs to deliver responses
3. Use tested modules as drivers
B
A
C
E
D
F
G
Breadth-First Integration Order
Test
A
Test
A,B,C,D
Test
A,B,C,D
E,F,G
Depth-First Integration Order
Test
A
Test
A,B
Test
A,B,E
+No drivers needed!
-Many stubs may be needeed
-Stubs may be problematic to write
Must provide values as expected by tester
May require frequent alteration
Test
A,B,E,F
Test
A,B,E,F,C
Test
A,B,E,F,C,D
Test
A,B,E,F,C
,D,G
CISS
Sandwich integration
Component Hierarchy
1. Combination of B-up and T-Down
2. Divide system into upper,
middle (target), and lower levels
3. B-up in low-levels
4. T-Down in upper levels
A
B
E
C
F
D
G
Integration Order
Test
E
Test
B,E,F
+ combines B-up and T-Dn
+ Integration possible early
-Individual components not tested thoroughly
before integration, especially middle level
Test
F
Test
G
Test
A
Test
D,G
Test
A,B,C,D
E,F,G
CISS
System Test (Pfleger)
•
•
Unit & integration: code implements design correctly
System testing: system does what customers expect
•
Constructed software tested against established
requirements during requirements analysis
•
Validation criteria satisfied?
•
Are all user-visible functional, behavioral, performance
requirements satisfied?
System
functional
requirements
Integrated
modules
Function
test
functioning
system
Requirements
For other
Quality aspects
Customer
Requirements
Specification
Non-functional
test
•Performance
•Stress / load
•Reliability,
•recoverability
•Security
•Installability,
•…
Installation/
Production
test
Acceptance
test
verified
system
User
Environment
System
In
use
accepted,
validated
system
CISS
Acceptance Test (Pfleger)
•
•
•
•
•
•
•
•
•
By customers,
User’s understanding of requirements
Customer writes, conducts, and
evaluates acceptance tests
Functional Acceptance Test (FAT)
Production Acceptance Test (PAT)
Techniques
Benchmark tests
Pilot tests ~ experimental, limited edition
–
–
Alpha test: in-house, controlled environment,
developer supervised
Beta test: “live” test at customer sites, unsupervised
Parallel test: run concurrently with old system
CISS