Evaluating Testing Methods by
Delivered Reliability
Frankl, Hamlet, Littlewood, Strigini
IEEE TOSE Aug98
841f07frankl6sep11
11
Testing
 Debug
 Operational
841f07frankl6sep11
2
Fault Detection
Probability
 Probability
of a testing methodology finding
a fault (if it existed)
841f07frankl6sep11
3
Partition vs Random
841f07frankl6sep11
4
Tests, Specifications, meets

Test or test case
– single value of program input
– functional program - one input produces an output

Specification - S
– set of input-output pairs

Program meets specification
– iff for all x in spec, actual output matches spec output
841f07frankl6sep11
5
Q: probability
distribution
Q
- probability distribution over input
domain
– Q:D -> [0,1] and S Q(t) = 1
841f07frankl6sep11
6
Q : Failure Probability
Q
- failure probability for a randomly
drawn point is S Q*d
– Where d(t) = 1 if f and 0 if s
– and f-phi(failure) and s-sigma(success)
 How
does this relate to our notation?
841f07frankl6sep11
7
Reliability
 R(N)
= (1- Q)N
841f07frankl6sep11
8
Assumptions of initial model
841f07frankl6sep11
9
Terms
q
d
841f07frankl6sep11
10
3.2 SFR, w/o
subdomains
= StinF V(t)
 P(Q=0) = 1-(1-d)T
 P(Q=q) = (1-d)T
 E(Q) = 0* P(Q=0) +q* P(Q=q)

= q(1-d)T
d
841f07frankl6sep11
11
3.3 single failure
region, with sub
 Debug
with n subdomains
– E(Q) = qP(1-di)T/n
 Operational
– E(Q) = q(1-q)T
Assumptions: multiple failure model
15
Thurs, Sep 13
 Read
next sections of article – 3.4
841f07frankl6sep11
16