Verification
Verification (CS340 John Knight 2004)
1
Place your School Logo Here
Validation vs. Verification

Validation:



“Did We Build The Right Thing?”
Does The Software Do What User Wants?
Verification:


“Did We Build The Thing Right?”
Does The System Implement The Specification?
Development
Idea
Validation
Software
Specification
Verification
Actual
Software
Validation
Verification (CS340 John Knight 2004)
2
Place your School Logo Here
Verification Techniques

Inspection:



Static Analysis:



Have A Program Examine The Work Product
Summary
Testing:



Examine The Work Product By Eye
Detailed Discussion—Laboratory Topic
For An Executable Work Product, Try It Out
Summary
Formal Proof:


Use Mathematics To Demonstrate The Desired Property
Summary
Inspections are the best return on investment.
Always include inspection in your software processes.
Verification (CS340 John Knight 2004)
3
Place your School Logo Here
Reviews, Walkthroughs, &
Inspections

Very Effective In Practice
Applicable To All Work
products
Important Observation:
You can inspect all of a
program but you cannot
test all of a program.
A Variety Of Techniques Tried:



Reviews
Walkthroughs
Inspections
All Based On Same Premise:
Humans looking at work products
find defects quite well.

Verification (CS340 John Knight 2004)
4
Place your School Logo Here
Fagan Inspections




Developed By Michael Fagan At IBM
Extremely Successful, Very Popular
Hard To Do Well, Easy To Botch
Steps:







Planning
Overview
Preparation
Inspection
Rework
Follow-Up
- Fairly Brief
- About One Hour
- Several Hours Over Several Days
- About A Two-Hour Meeting
- Depends On What Is Found
- Check That Necessary Changes Made
People:





Moderator
Author
Reader
Inspectors
Specialists
Verification (CS340 John Knight 2004)
- Manages Process
- Developer Of Artifact
- Provides Inspection Focus
- General Participants
- Specific Technical Areas
5
Place your School Logo Here
Fagan Inspections
Author’s
Overview






Specifications
Designs
Implementations
Test Plans
Process Concepts
Etc.
Inspection
Inspector’s
Reading
Inspector’s
Reading
Author’s
Rework
Inspector’s
Reading
Verification (CS340 John Knight 2004)
6
Place your School Logo Here
Inspection Activities

Goal:




Checklist-driven:




Detect Defects In Work Product
Defects Might Be Any Deviation From Desired Quality
Not Just Source Code "Bugs"
Usually A Corporate Standard Set Of Checks
Determines Specialists Who Have To Inspect
Inspectors Assumed To Have Read/Inspected Work
Product
Moderator:




Schedules Inspection
Determines Length
Monitors/Manages Meeting
Determines Which Part Of Artifact To Inspect
Verification (CS340 John Knight 2004)
7
Place your School Logo Here
Inspection Activities

Reader:


Author:




Provides Overview
Answers Questions
Undertakes Rework
Inspectors:




“Guides” Inspection By Reading Aloud, Line-by-line
Apply Product Knowledge
Language/Tools Knowledge
Technical Experience
Specialists—Attention To:





Portability
Maintenance
Real-time
Use Of Language
Etc.
Verification (CS340 John Knight 2004)
8
Place your School Logo Here
Review—Issues With Fagan
Inspections




Why The Overview? Documents?
No Enforced Preparation—People Tend Not To Bother...
Inspection Based On Paper Documents
Not Rigorous:





No Matching Of Inspector Skill Levels:


Driven By The Clock
Dominated By Strong-Willed Individuals
No Precise Format
No Precise Schedule
Senior Engineers Argue About Comments And Spelling
No Conclusions Ensured
Remains A Controversial Technology
Done Well, Inspection Is The Best Value For Money Available
Verification (CS340 John Knight 2004)
9
Place your School Logo Here
Active Reviews




Developed By Parnas And Weiss At NRL
Attempt To Deal With Problems Of Fagan Inspections
Sophisticated Technique
Some Key Points:





Inspectors Work On A Single Technical Area—Their Area Of
Expertise
Inspectors Work Alone
Authors Supply Questions To Inspectors To Check
Comprehension
Results Collected And Product Reworked As Necessary
Found To Be Very Effective At NRL On A7
Verification (CS340 John Knight 2004)
10
Place your School Logo Here
Phased Inspections








Developed By Knight And Myers At UVa
Attempt To Deal With All The Problems Of Fagan Inspections
Goal—A Very Rigorous, Repeatable Software Inspection Process
Premise:
There is far too much to check in a single inspection.
Inspections should be in tightly focused, separate steps.
Steps Are Called Phases—Each Phase Building On Prior Phases
Two Types Of Phase—Single- And Multi-Inspector
Use Questions And Isolated Inspection From Active Reviews
Multi-inspector Phase:



Several Inspectors Work In Parallel—Try To Find All Defects In Certain
Class
Inspectors Meet To Reconcile Results
Experiments At UVa Indicate Considerable Benefits In Approach
Verification (CS340 John Knight 2004)
11
Place your School Logo Here
Phased Inspections
Phase 1
Phase 2
Phase N
Inspector(s) A
--Check
Compliance
With
Specific
Quality
Goal
Inspector(s) B
--Check
Compliance
With
Specific
Quality
Goal
Inspector(s) I
--Check
Compliance
With
Specific
Quality
Goal
Signature
Signatures
Assume All
Prior Checks
Assume All
Prior Checks
Computer Support
Verification (CS340 John Knight 2004)
12
Place your School Logo Here
Types Of Phases
(a) Individual
Specific
Quality
Check
(b) Group
Specific
Quality
Check
Specific
Quality
Check
Reconciliation
Specific
Quality
Check
Verification (CS340 John Knight 2004)
13
Place your School Logo Here
Phased Inspection—Example
Internal
Document
Inspection
Local
Programming
Practices
Inspection
Source Code
Format
Inspaction
Defect
Inspection
By Individual
Global
Programming
Practices
Inspection
Design
Documentation
Inspection
Defect
Inspection
By Individual
Reconciliation
Defect
Inspection
By Individual
Verification (CS340 John Knight 2004)
14
Place your School Logo Here
Cleanroom Development




Developed By Mills At IBM
Premise:
Provide a software development environment, a “cleanroom”,
that would not permit defects to enter the software.
Draws On Analogy With Hardware Engineering
Sophisticated Technique—Some Key Points:







Software Developed Without Execution
Authors Not Permitted To Execute Software
Some Variants Do Not Permit Compilation
Software Tested By Independent Group
Test Results Returned To Authors For Software Repair
Quality Assessment By Statistical Estimation
Results Include:


Better Productivity (Actually, Much Better)
Better Quality (Actually, Much Better)
Verification (CS340 John Knight 2004)
15
Place your School Logo Here
(Insert Course #) Inspections


Goal Is To Try Inspection Exercise
Three-phase Phased Inspection:







Source-code Layout
Programming Practices
Functional Correctness
Author-supplied Questions For Comprehension
Serious Measurement Of Activities
Laboratory Activity To Be Presentation Of Results
Post-lab Is To Be Inspection Report
Read the lab manual for lab 8.
Verification (CS340 John Knight 2004)
16
Place your School Logo Here
Static Analysis
Work Product
(Any Type, Not
Just Source Code
Static Analyzer
Report
Parse Input
Create Internal Representation
Conduct Check(s)
Produce Report


Examine Work Product Looking For Legal But Probably
Erroneous Things
Typical Uses:



Set/Use Anomalies
Coding Standard Checking
Design-rule Checking
Verification (CS340 John Knight 2004)
17
Place your School Logo Here
Difficulties With Static Analysis


Not A Panacea—Many Things Cannot Be Checked:
if a > b then
c := 1;
...
if d < e then
f := g / c;
-- Is c initialized on every execution?
Program Complexity:
...
very_large_function(a, b, c, d, e, f, g, h);
-- What is the effect of the function on the prgrm variables?



Language Specific, Difficult To Construct—Hence Not As Common As They
Should Be
Often Supplement Compilers (Finish The Compiler’s Job), E.g. Lint.
Very High Performance System Developed By Dave Evans—Splint
Verification (CS340 John Knight 2004)
18
Place your School Logo Here
Testing Summary




Testing As Verification Technique
Testing And Validation
Testing Compared To Debugging
Testing Phases:








- Integration
- Regression
- System
Black Box vs. White (Clear) Box Testing
Coverage:


Unit
Acceptance
Functional
Statement
Branch
Path
Automation Of Test Execution
Checking Correctness Of Output
Verification (CS340 John Knight 2004)
19
Place your School Logo Here
Testing In Engineering
AEROSPACE ENGINEERING
SOFTWARE ENGINEERING
Hydraulic Rams
Structural Analysis By Finite
Element Technique
Almost No Comprehensive
Analysis Possible
Testing Confirms Analysis
Testing Establishes Conclusion
Verification (CS340 John Knight 2004)
20
Place your School Logo Here
Some Testing Terminology

Confusing Terms—Use Them Properly:





Test Oracle—Hardest Part Of Testing:





Beethoven
Exxon
Integration Strategies:


Error And Erroneous State
Fault And Latent Fault
Failure
Commission vs. Omission
Bottom Up
Top Down
Test “Case”?
Software Reliability?
Verification (CS340 John Knight 2004)
21
Place your School Logo Here
Functional Testing



Black Box Approach
Function/Test Case Matrix
Coverage Metric:






All Functions Covered Singly
Function Covered In Combinations
Equivalence Partitioning
Boundary Value Analysis
Cause-Effect Graphing
Special Cases:


Derive Cases From Conditions In Software
Base Cases On Intuition
Verification (CS340 John Knight 2004)
22
Place your School Logo Here
Structural Testing



White/Glass Box Approach
Base Test Case Selection On Software Structure
Control Flow Based Testing—Try To Meet Coverage
Metrics:





Statement Coverage
Branch Coverage (All Flavors)
Path Coverage
How Can These Be Measured?
Mutation Testing:




Test The Test Cases
Benign Mutants
Killing Mutants
Other Defects?
Verification (CS340 John Knight 2004)
23
Place your School Logo Here
Testing Special Types Of
Program

Object-Oriented Programs:





User Interfaces:


How Should The Input Be Generated?
Concurrent Programs:



Tricky...
How Do You Test A Template?
Programs Manipulate Objects, Not Classes (Or Templates)
What Can You “Trust” When Inheriting A “Tested” Class?
Order Of Events
Test Harnesses
Real-Time Programs:


Correctness Includes Timing
Timing Affected By Measurement....
Verification (CS340 John Knight 2004)
24
Place your School Logo Here