Benchmarking
for [Physical] Synthesis
Igor Markov and Prabhakar Kudva
The Univ. of Michigan / IBM
In This Talk …
Benchmarking vs benchmarks
 Benchmarking exposes new research Qs
 Why industry should care about
benchmarking
 What is (and is not) being done
to improve benchmarking infrastructure
 Not in this talk, but in a focus group

Incentives for verifying published work
 How to accelerate a culture change

Benchmarking

Design benchmarks


Objectives (QOR metrics) and constraints






Algorithms, methodologies; Implementations
Solvers: ditto
Empirical and theoretical analyses, e.g.,


Data model / representation; Instances
Hard vs easy benchmarks (regardless of size)
Correlation between different objectives
Upper / lower bounds for QOR, statistical behavior, etc
Dualism between benchmarks and solvers
For more details, see http://gigascale.org/bookshelf
Industrial Benchmarking

Growing size & complexity of VLSI chips

Design objectives
 Area

/ power / yield / etc
Design constraints
 Timing
/ FP + fixed-die partitions / fixed IPs /
routability / pin access / signal integrity…
Can the same algo excel in all contexts?
 Sophistication of layout and logic motivate
open benchmarking for Synthesis and P&R

Design Types

ASICs




SoCs




Lots of fixed I/Os, few macros, millions of standard cells
Design densities : 40-80% (IBM)
Flat and hierarchical designs
Many more macro blocks, cores
Datapaths + control logic
Can have very low design densities : < 20%
Micro-Processor (P) Random Logic Macros(RLM)




Hierarchical partitions are LS+P&R instances (5-30K)
High placement densities : 80%-98% (low whitespace)
Many fixed I/Os, relatively few standard cells
Note: “Partitioning w Terminals” DAC`99, ISPD `99, ASPDAC`00
Why Invest in Benchmarking

Academia




Industry




Benchmarks can identify / capture
new research problems
Empirical validation of novel research
Open-source tools/BMs can be analyzed and tweaked
Evaluation and transfer of academic research
Support for executive decisions
(which tools are relatively week & must be improved)
Open-source tools/BMs can be analyzed and tweaked
When is an EDA problem (not) solved?


Are there good solver implementations?
Can they “solve” existing benchmarks?
Participation / Leadership Necessary
Activity 1: Benchmarking platform / flows
 Activity 2: Establishing common evaluators

Static timing analysis
 Congestion / yield prediction
 Power estimation

Activity 3: Standard-cell libraries
 Activity 4: Large designs w bells & whistles
 Activity 5: Automation of benchmarking

Activity 1: Benchmarking Platform

Benchmarking “platform”: a reasonable subset of







data model
specific data representations (e.g., file formats)
access mechanisms (e.g., APIs)
reference implementation (e.g., a design database)
design examples in compatible formats
Base platforms available (next slide)
More participation necessary


regular discussions
additional tasks / features outlined
Common Methodology Platform
Synthesis
(SIS, MVSIS…)
Blif  Bookshelf format
Common Model
(Open Access?)
Placement
(Capo, Dragon,
Feng Shui, mPl,…)
Blue Flow exists, Common model hooks: To be Done
Placement Utilities
http://vlsicad.eecs.umich.edu/BK/PlaceUtils/
 Accept input in the GSRC Bookshelf format
 Format converters




LEF/DEF  Bookshelf
Bookshelf  Kraftwerk (DAC98 BP, E&J)
BLIF(SIS)  Bookshelf
Evaluators, checkers,
postprocessors and plotters

Contributions in these categories are welcome
Placement Utilities (cont’d)

Wirelength Calculator (HPWL)


Independent evaluation of placement results
Placement Plotter
Saves gnuplot scripts ( .eps, .gif, …)
 Multiple views (cells only, cells+nets, rows,…)


Probabilistic Congestion Maps (Lou et al.)
Gnuplot scripts
 Matlab scripts

 better

graphics, including 3-d fly-by views
.xpm files ( .gif, .jpg, .eps, …)
Placement Utilities (cont’d)
Legality checker
 Simple legalizer
 Layout Generator

Given a netlist, creates a row structure
 Tunable %whitespace, aspect ratio, etc


All available in binaries/PERL at
http://vlsicad.eecs.umich.edu/BK/PlaceUtils/

Most source codes are shipped w Capo
Activity 2: Creating Evaluators

Contribute measures/analysis tools for:
Timing Analysis
 Congestion/Yield
 Power
 Area
 Noise….

Benchmarking Needs for Timing Opt.

A common, reusable STA methodology
High-quality, open-source infrastructure
 False paths; realistic gate/delay models


Metrics validated against phys. synthesis
The simpler the better,
but must be good predictors
 Buffer insertion profoundly impacts layout

 The
use of linear wirelength in timing-driven layout
assumes buffers insertion (min-cut vs quadratic)

Apparently, synthesis is affected too
Vertical Benchmarks

“Tool flow”



Vertical benchmarks


Two or more EDA tools, chained sequentially
(potentially, part of a complete design cycle)
Sample contexts: physical synthesis, place & route,
retiming followed by sequential verification
Multiple, redundant snapshots of a tool flow
sufficient info for detailed analysis of tool performance
Herman Schmit @CMU is maintaining
a resp. slot in the VLSI CAD Bookshelf




See http://gigascale.org/bookself
Include flat gate-level netlists
Library information ( < 250nm)
Realistic timing & fixed-die constraints
Infrastructure Needs

Need common evaluators of delay / power


To avoid inconsistent / outdated results
Relevant initiatives from Si2
OLA (Open Library Architecture)
 OpenAccess
 For more info, see http://www.si2.org

Still: no reliable public STA tool
 Sought: OA-based utilities for timing/layout

Activity 3: Standard-cell Libraries

Libraries carry technology information



Some benchmarks in the Bookshelf
use 0.25m and 0.35m libraries




Impact of wirelength delays
increases in recent technology generations
Cell characteristics must be compatible
Geometry info is there, + timing (in some cases)
Cadence test library?
Artisan libraries?
Use commercial tools to create libraries

Prolific, Cadabra,…
Activity 4: Need New Benchmarks
To Confirm / Defeat Tool Tuning


Data on tuning from the ISPD03 paper
“Benchmarking for Placement”, Adya et al.
Observe that







Capo does well on Cadence-Capo, grid-like circuits
Dragon does well on IBM-Place (IBM-Dragon)
FengShui does well on MCNC benchmarks
mPL does well on PEKO
This is hardly a coincidence
Motivation for more / better benchmarks
P.S. Most differences above have been explained,
all placers above have been improved
Activity 4: Large Benchmark Creation

www.opencores.org has large designs
May be a good starting point –
use vendor tools to create blif files
(+post results)
 Note: there may be different ways to convert


A group of design houses
(IBM, Intel, LSI, HP)
is planning a release of new large
gate-level benchmarks for layout

Probably no logic information
Activity 5: Benchmarking Automation



Rigorous benchmarking is laborious. Risk of errors is high
How do we keep things simple / accessible?
Encapsulate software management in an ASP








Web uploads for binaries and source in tar.gz w Makefiles
Web uploads for benchmarks
GUI interface for NxM simulations; tables created automatically
GUI interface for composing tool-flows; flows can be saved/reused
Distributed back-end includes job scheduling
Email notification of job completion
All files created are available on the Web (permissions & policies)
Anyone can re-run / study your experiment or interface with it
Follow-on Action Plan

Looking for volunteers to -test Bookshelf.exe



Create a joint benchmarking group
from industry and academia



Particularly, in the context of synthesis & verification
Contact: Igor [email protected]
Contact: Prabhakar [email protected]
Regular discussions
Development based
on common infrastructure