Cool FPV Tricks: Reaching
Deep Bounds With Not-QuiteFormal Methods
Erik Seligman
CS 510, Lecture 13, February 2009
Agenda





Motivation: FPV and Deep Bounds
0in’s ‘Dynamic Formal’ FPV
Synopsys ‘Hybrid Formal’ FPV
When FPV Is Not FPV
Some Meta-Comments
Agenda





Motivation: FPV and Deep Bounds
0in’s ‘Dynamic Formal’ FPV
Synopsys ‘Hybrid Formal’ FPV
When FPV Is Not FPV
Some Meta-Comments
Bounded FPV
 Majority of FPV work done at low bound
• Around 50-100 cycles
• Many modules testable this way
• Limits of engines: exponential blowup
 Good in many (most?) cases
• Verification of isolated modules
• Tricky bugs often don’t need much depth
– with right input & constraints
• Cases where deep queues/counters not critical
 Double-check with coverage points!
Problem: FPV Bounds

Bounded FPV is ignoring possibilities
•

Known industry cases of “deep” errors
Deep bugs suspected in some cases
•
Example: 2 independent 13-bit ctrs
–
–
•
•
How many combinations possible?
How many cycles needed?
Bugs may be lurking beyond FPV bounds
Maybe too much logic to solve with pruning
 Strategies desired for hi-bound FPV
Compromise: Abstraction
 Counters, state machines: free outputs
• As we saw in memory controller example
All values possible
• Useful in many cases
• But at significant cost
• Realism of cases: values jump around
• No coverage for abstracted logic
Compromise: Initial State
 Set initial state to something ‘interesting’
• Fill queues, set high counters, etc.
 FPV no longer exhaustive
• Errors from different initial state not covered
• Focus is bug hunting in suspicious area
 Can we do better?
Limits of Pruning
 Pruning often helps complexity issues
• In best case, bounded  full proof
• Or can deepen bounds
 But it can’t help in worst cases
• Maybe long, complex interactions of parts
– Think about 2-counter example again
• Lots of logic may be involved
Key Insight: Leverage
Simulation
 Simulation env has design knowledge
• Usually set up to create interesting states
• Wide variety of states possible in sim
 Can we leverage simulation for FPV?
• Simple example: Snapshot a simulation
state, use as FPV input
• Are there bigger opportunities?
Agenda





Motivation: FPV and Deep Bounds
0in’s ‘Dynamic Formal’ FPV
Synopsys ‘Hybrid Formal’ FPV
When FPV Is Not FPV
Some Meta-Comments
0in Dynamic Formal FPV
 Monitor simulation runs
• Tool runs as plugin to simulator
 Identify “interesting” states
• Hit cover point
• Transition counter, state machine, etc.
 Launch many low-bound FPV runs
Ordinary FPV & State Space
• Small proof radius covered after reset
Dynamic Formal Approach
• Many mini-FPV runs launched from sim
• Smaller proof radius for each FPV run
Advantages/Disadvantages
 Advantages
• Based on real tests  likely good examples
• Produce realistic counterexamples since
most cycles are from simulation
• Finds deep bugs “almost” tested in sim
 Disadvantages?
Advantages/Disadvantages
 Advantages
• Based on real tests  likely good examples
• Produce realistic counterexamples since most
cycles are from simulation
Finds deep bugs “almost” tested in sim
•
 Disadvantages?
• Depends on setting up simulation env
– Not usable really early or arbitrary hierarchy
• Can’t find counterexamples distant from tests
– Misses major selling point of FPV: corner cases not
conceived by designer
• Tool pain: FPV & sim tools interact?
– Harder than it sounds!
Agenda





Motivation: FPV and Deep Bounds
0in’s ‘Dynamic Formal’ FPV
Synopsys ‘Hybrid Formal’ FPV
When FPV Is Not FPV
Some Meta-Comments
Synopsys Magellan Approach
 Similar motivations to 0in
 Recognize problem: simulation env
• Few real designers use ‘raw’ sim tool
• Many layers of scripts & wrappers
• Really painful to enable 0in-like method
 Solution: Random Simulation
• Still uses simulation engine
• Randomly tries to simulate to get cover pts
– No use of real tests like 0in
Hybrid Formal Approach
• Lots of random simulation paths
• Many mini-FPV runs launched from sim
• Smaller proof radius for each FPV run
Advantages/Disadvantages
 Advantages
• Not limited by tests from validation team
• Can theoretically find very deep bugs
• Independent of simulation environment
 Disadvantages?
Advantages/Disadvantages
 Advantages
• Not limited by tests from validation team
• Can theoretically find very deep bugs
• Independent of simulation environment
 Disadvantages?
• Very random: need to get lucky for bug
– Loses both comprehensiveness & test-guidance
– Good cover points may mitigate
• Still have sim/fpv tool integration issues
– Many “simulation-synthesis mismatches”: time delays,
system functions, non-det behavior, X/Z vals, etc.
Agenda





Motivation: FPV and Deep Bounds
0in’s ‘Dynamic Formal’ FPV
Synopsys ‘Hybrid Formal’ FPV
When FPV Is Not FPV
Some Meta-Comments
Can We Use A NonSimulation Approach?
 FPV user suspects deep error
• But proof bounds not enough
• Standard complexity techniques not working
 What does FPV user know?
• Probably has likely scenarios in mind
• But FPV tool isn’t getting there
 So why not use user-guided FPV?
• Try to manually get FPV tool close to error
• Then launch bounded, comprehensive run
User-Guided FPV
 Create cover points for reset state
• Suspected intermediate state on way to err
 Ask FPV tool to reach cover point
• CEX at cover == reset state for next
• Save state manually
– Nicer if tool help, but not seen in current tools
 After cover pt, use real bounded FPV
• Run isn’t full formal proof of properties
• But is comprehensive bug hunt from there
User Guided Approach
• Use FPV engine to visit cover point
• Launch bounded FPV from targeted pt
Advantages/Disadvantages
 Advantages
• Completely eliminates sim engine issues
• Can theoretically find very deep bugs
• Takes advantage of user knowledge
 Disadvantages?
Advantages/Disadvantages
 Advantages
• Completely eliminates sim engine issues
• Can theoretically find very deep bugs
• Takes advantage of user knowledge
 Disadvantages?
• Highly dependent on user intuition
– Loses ‘automatic search’ aspect of 0in/Synopsys
• Problem areas must be specifically targeted
Agenda





Motivation: FPV and Deep Bounds
0in’s ‘Dynamic Formal’ FPV
Synopsys ‘Hybrid Formal’ FPV
When FPV Is Not FPV
Some Meta-Comments
Deep Bounds: Why ‘Sexy’?
 Much effort in industry/academia
 Recognition of limits of bounded FPV
 Potential for fame and fortune
• Really cool to claim find of 10000-cycledeep error nobody noticed!
• Common view: One such error justifies cost
of full deployment + BMW for the validator
 But is this really the key FPV problem?
Where Do FPV users Spend
Their Time?
 Intuitive explanation, take with grain of salt
 70% “Wiggling”
• Early attempts with quick counterexamples
• Root-causing CEXs, adding assumptions, & finding occasional
shallow bugs
• Many FPV efforts never get beyond this stage
 20% “Solidifying”
• Have bounded proofs on most assertions
• Incrementally deepening bounds, checking covers
 10% “Heroics”
• Extra, expert efforts to get full proofs or deep proofs in
suspected problem areas
• Trying exotic FPV technologies
Where Should FPV Research
Be Concentrated?
 Vast majority of FPV effort is “wiggling”
• Assuming previous slide roughly accurate!
 Then next largest chunk is “solidifying”
• Understaning, improving low bounds
• Basic complexity issues
 What are FPVers actually doing?
• Running low-bound proof attempts
• Root-causing failures to add assumptions
• Attacking basic complexity issues
 (IMHO) The real problem is Usability
What Would An FPV
Breakthrough Look Like?
 Need quantum leaps in productivity of
“wiggling” stage
• Secondary priority: addressing complexity
 Key functionality I want to see
• Viewing and understanding counterexamples
• More & better live what-if experiments
• Intuitive user interaction
• Really fast incremental responsiveness
• Support for interactive complexity analysis
My Advice To Academia
 Better engines & deeper proofs are nice
• But only leveraged by small group of engineers who
progress to “heroic” FPV stage
 Best impact: where engineers spend time
• Wiggling stage: quickly understanding cex
• Solidifying stage: root-causing & fixing complexity
 Focus on usability, interactivity, and
incremental resposiveness!
References / Further Reading
 http://www.edacafe.com/Vision/200208/d
esign3.html
 http://drona.csa.iisc.ernet.in/~deepakd/tal
ks/formal-iisc-0306.pdf
 http://www.te.rl.ac.uk/europractice/vendor
s/magellan_ds.pdf