Intelligent Interleaving of Scenarios:
A Novel Approach to System Level Test
Generation
Shady Copty, Itai Jaeger(*),
Yoav Katz, Michael Vinov
IBM Research Laboratory in Haifa
Outline

System-level stimuli generation challenges

Interaction based testcase generation

X-Gen: a system-level testcase generator

Scenario interleaving in X-Gen

Real life experience examples
– eServer, Cell BE, Xbox CPU chip

Summary
1
System verification
System verification is aimed
at validating the
integration of several
previously verified cores
in a relatively short time
2
Challenges in Stimuli Generation for
System Level Functional Verification

Specifying system level scenarios in an abstract form
–While generating the required low level stimuli

Generating coordinated system-level stimuli
–Projected to each and every core in the system

Effectively handling configuration changes
–2-way system vs. 8-way system

Adapting to core modifications and new cores
–PCI  PCIe
3
The concept of interaction

Capturing the essence of the system-level
functionality
–
–
–
–
–
CPU-to-memory read/write
CPU-to-CPU inter processor interrupt
CPU initiated MMIO
IO initiated DMA
…

Interaction: acts, actors

Independent of the system’s configuration

Agnostic to specific core details
4
The concept of interaction - example
A DMA Interaction
 A CPU stores to the doorbell
register of the DMA engine
 The data is moved from the
IO port to memory
CPU#2
CPU#1
 The DMA engine interrupts
the initiating CPU
CPU#3
CPU#4
Bridge
DMA
Engine
PLB
Memory
Interrupt
Controller
PHB
IO
BFM#1
IO
BFM#2
5
Interaction based scenarios
Verification goal
Test plan definition
Interaction based scenario
Stress the system bus through address
contention on requests from multiple devices
Generate multiple IO reads, DMAs, and
processor accesses to the same address
Generate contention on the system bus
6
X-Gen : a model-based systemlevel stimuli generator
Interactions
(Transactions)
Specifying
an
Abstract
System Model
interaction-based
Configuration
scenarios
Components
(Cores)
(Topology)
X-Gen Engine
Test Template
*.rqst
Test Case
CSP Solver
7
A system-level test template



Interaction as the basic
building block
Request file
Control over:
– Participating cores
– Interaction properties
All of
One of
A hierarchy if higher level
statements
– One-of: weighted
random choice
Weight: 70
Weight: 30
– Repeat
Load / store
DMA
– All-of
Target: mem2
Addr: 0x12??
Repeat x10
interrupt
Priority: 12-14
8
A system-level test case
Hubs, bridges, adaptors
• Translation tables
• Data descriptors for
send / receive operations
• Internal registers initializations
Memory
• Data initializations
Processors
• Instructions
• Registers initializations
• Translation tables
CPU#2
CPU#1
CPU#3
CPU#4
Bridge
DMA
Engine
PLB
Behaviorals (BFMs)
• Internal memory initializations
• Commands, such as:
•Send an interrupt
•Initiate a DMA
Memory
Interrupt
Controller
PHB
IO
BFM#1
IO
BFM#2
9
Combining scenarios


Stressing the system through parallel
execution of several system level scenarios
For example:
– Read/write cache coherency from multiple
processors
– MMIO / DMA to IO port
– Inter processor interrupts
– Create address contention between DMAs and
processor accesses
– Send interrupts to processors contending on the
same cache line
10
Scenario interleaving in X-Gen
IO.rqst
CPU.rqst
- Repeat 80
- CPU_to_memory_data_tarnsfer
- Repeat 70
- IO_to_memory_DMA
X-Gen
Total: 150
interactions
Accessing the
same address
CPU_to_memory_data_tarnsfer
IO_to_memory_DMA
CPU_to_memory_data_tarnsfer
CPU_to_memory_data_tarnsfer
IO_to_memory_DMA
IO_to_memory_DMA
…
11
eServer system verification:
Partitioned system-level test
template library
Maintenance
saving:
Cache coherency
scenarios
PCI bridge
avoiding
stressexponential
scenarios
test template growth
X-Gen
Intervention
scenarios
InfiniBand
message passing
scenarios
12
Xbox 360 chip verification :
Reusing scenarios
Xbox specific
IO scenarios
Heavy vIP reuse:
eServer MP/MT
scenarios
generating complex
system-level testcases
within weeks from
verification launch
X-Gen
13
Cell BE verification:
Verifying pervasive functions
Power
management
function
Performance
management
function
General cell
scenarios
Main scenario
Slow down unit YYY
Main
Main scenario
scenario
Main
scenario
Power
down unit XXX
Power
down unit XXX
Main scenario
Main
Main scenario
scenario
Main
scenario
Resume
frequency on unit YYY
Restart
Restart unit
unitXXX
XXX
Main
Main scenario
scenario……
X-Gen
14
Summary


Interleaving transaction-level scenarios is crucial for
system-level verification
Partitioning the test plan according to different system
functions
– Avoids costly maintenance of test templates
– Promotes reuse of verification IP
– Fast and effective verification of pervasive functions in
conjunction with “mainstream” scenarios

Implemented in X-Gen
– IBM’s system-level testcase generator
– Used in the verification of eServer, Xbox & Cell systems
15
Thank You !!!
Traditional Approach #1:
Combining Lower Level Drivers

Advantages
– Simple
– Quickly adapts to new
cores
– Reuse of core VIP

Driver
Driver
Driver
IP1
IP2
IP2
Driver
BUS
Driver
Driver
Disadvantages
– Not system level
verification
• No coordinated stimuli
• No system level
scenarios
17
Traditional approach #2:
Transaction Based Verification

Advantages
–
–
–

System level
abstraction
Allows complex
coordinated
stimuli
Allows reuse of test
specification
Disadvantages
–
–
–
Transactor code is
monolithic
Difficult to adapt to
configuration
changes
Difficult to adapt to
new
components / changed
functionality
Driver
Transactor
Transactor
IP1
IP2
IP2
BUS
18
Request files: a dual effort
methodology
The Bug
Request file
Specifying a scenario

Interactions as building blocks

Restrict actors, properties

Inter-interaction relations
Intelligent background noise

Built-in testing knowledge

User direction
Read: 80
Write: 20
All of
One of
CPU load /
store
Address
Collision: 65%
DMA
transfer
Repeat x10
Interrupt
19