An Approach for Selecting Tests with Provable Guarantees
Mahadevan Subramaniam and Bo Guo
University of Nebraska at Omaha
Multi-version QA
Changes
Requirements
Development
Team
bugs
Testing
Team
No bug
claimed
R1
Development
Team
codes
bugs
Testing
Team
No bug
claimed
Modified codes
Two or Three weeks
Testing is very costly and time consuming [NIST]
1) in critical path for release
2) development deadline slips can reduce test time
3) development team needs time to fix bugs
2
UNO
11/30/2012
R2
Multi-version QA Scope

Predictive Changes: new tests to validate new features
Corrective Changes: Regression testing of existing features

Problems




3
Regression Test Minimization – remove redundant tests
Regression Test Prioritization – fault-finding tests first
Regression Test Selection (RTS) – select relevant tests
UNO
11/30/2012
Regression Test Selection -- Motivation
Regression testing is most frequent [NIST] and has significant
impact on product quality.


System behavior re-validated in practice using large test suites.

Re-running the entire test suite each time is impractical.
Regression test selection identifies tests from a test suite to
validate a change in an evolution step.

4
UNO
11/30/2012
Current State of Art – Code based
5
UNO
11/30/2012
Current State of Art – Model-based
6
UNO
11/30/2012
Current State of Art
 Store all test traces on original version
 Static analysis
 Compare new and modified Control Flow Graphs
 If a trace includes the differences, select test including the trace.
 Use data flow graphs and program dependency graphs
7
UNO
11/30/2012
Problems
 Storing all traces is expensive
 Control flow graphs comparing may involve untested
parts of programs
 No guarantee on the regression test suite
 Relevant tests may be skipped (Incomplete)
 Irrelevant tests may be selected
8
UNO
11/30/2012
Goal
 Develop an automatic approach for regression test selection
to create a regression test suite with provable guarantees
and without using any history information.
 Main Challenges
 Provably predict test execution behavior without actually
executing the test ?
 Keep regression test selection economical
 Selection cost + regression test cost <= brute-force run cost.
9
UNO
11/30/2012
Model-based Proposal
Key ideas
 Analyze test descriptions to provably predict their behavior.
 Test descriptions based on EFSMs and GUI event diagrams have
sufficient information in terms of input and output events.
 Exploit overlap among descriptions to keep regression
economical [See tech report]
10
UNO
11/30/2012
EFSM – extended finite state machine
 List of states
The machine is in only one state at a time
 Triggering condition for each transition
s0
Initially a = 0; state is s0
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
1
6
s1
2
3
3
5
s2
4
11
UNO
11/30/2012
Overview of the Approach
 Analyze a given test description to determine if the test
execution will exercise a given change and select the test.
 Identify descriptions having sufficient information about their execution
(fully-observable tests)
 Analyze such descriptions to determine if the change will be executed.
 Patch other descriptions to obtain information about their execution.
 Changes add / delete / replace transitions.
 Test description is a sequence of input and output messages
with parameters with values over booleans, integers, arrays,
queues, and records.
12
UNO
11/30/2012
Overview of the Approach (Contd.)
 From a given test description and a change
 find sets of matching transitions that can process inputs in the description
 descriptions having a matching change transition are candidates
 determine feasibility of an execution path formed by matching transitions
 automatically attempt to patch infeasible paths to obtain an execution path
 Test selected if the change transition appears in the feasible
execution path.
13
UNO
11/30/2012
A Simple Example
s0
6
1
Initially a = 0; state is s0
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
3
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
s1
7
3
5
s2
4
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
Not
Selected
Tests
λ1: open(100)/ack(100), deposit(50)/ack(150), close/ack(150)
λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150)
λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50)
14
UNO
11/30/2012
A Simple Example – λ2
λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150)
Φ=[
{1}
{2}
{ 3, 7 }
{6}
]
Find transitions matching each input of description λ2
s0
1
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
15
UNO
6
s1
2
3
7
3
5
s2
4
11/30/2012
A Simple Example – λ2 (conti…)
λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150)
Φ=[
{1}
{2}
{3, 7}
{6}
First test input of λ2 must be processed by transition 1
Transition 2 can immediately follow 1 to process next input of λ2
s0
1
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
16
UNO
6
s1
2
3
7
3
5
s2
4
11/30/2012
]
A Simple Example – λ2 (conti…)
Selected
λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150)
Φ=[
{1}
{2}
{3, 7}
{6}
Transition 7 can immediately follow 2 since (160 > 0) ˄ (160 > 150)
satisfiable.
s0
1
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
17
UNO
6
s1
2
3
7
3
5
s2
4
11/30/2012
]
A Simple Example – λ3
λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50)
Φ=[
{1}
{2}
{3, 7}
{3, 7}
{6} ]
First two inputs processed by transition 1 followed by transition 2.
Transition 7 cannot immediately follow 2 since (100 > 0) ^ (100 > 150)
is unsatisfiable.
s0
1
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
18
UNO
6
s1
2
3
7
3
5
s2
4
11/30/2012
A Simple Example – λ3(conti..)
λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50)
Φ=[
{1}
{2}
{3, 7}
{3, 7}
{6} ]
But 3 can immediately follow 2 since (100 > 0) ˄ (100 <= 150) is
satisfiable.
s0
1
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
19
UNO
6
s1
2
3
7
3
5
s2
4
11/30/2012
A Simple Example – λ3(conti..)
λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50)
Φ=[
{1}
{2}
{3, 7}
?
{3, 7}
{6} ]
Can we patch
λ3 to obtain a
path with
transition 7?
Neither 7 nor 3 can immediately follow transition 3.
7 can follow 3 after some transitions not using test inputs.
s0
1
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
20
UNO
6
s1
2
3
7
3
5
s2
4
11/30/2012
A Simple Example – λ3(conti..)
Selected
λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50)
Φ=[
{1}
{2}
{3, 7}
?
{3, 7}
{6} ]
4 cannot patch since (50 > 0) ^ (50 < 50) is unsatisfiable.
5 can patch since (50 >= 50) is satisfiable and 7 can immediately
follow 5.
s0
1
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
21
UNO
6
s1
2
3
7
3
5
s2
4
11/30/2012
Salient Aspects
 Regression test selection for test descriptions with rich data types.
 Fully-observable tests: descriptions with sufficient information.
 A simple structural invariant to identify fully-observable tests.
 Automatically analyze failure of invariant to patch test descriptions.
 Procedures to select tests guaranteed to exercise changes to EFSMs.
 Experiments with 10 web services and protocols
 Study costs of running the full test case and selective test cases.
 Proposed approach reduces test running times in all examples.
22
UNO
11/30/2012
Outline
 Preliminaries
 Fully-Observable Tests
 Tests with Non-Observable Regions
 Experiments
 Conclusion
23
UNO
11/30/2012
EFSMs
 E = (Ii, Oi, Si, Vi, Ti)
 Ii, Oi : input and output messages
 Si: local states
 Vi: Data and queue variables
 Ti: Deterministic transition relation
 Transition t: mj, Pt, st  qt, ml, At
 mi ml: parameterized input and output messages
 Pt: a predicate over variables from Vi
 st, qt : states from Si
 At: ordered sequence of assignments to variables from Vi
 Explicit transition is transition having both input and output
messages
24
UNO
11/30/2012
Test Descriptions and Changes
 Test description
 λ = < g0, [i1/o1, i2/o2, …, in/on] >
 g0 is concrete initial global state
 Finite sequence of input/output elements ik/ok
 Test run
 rλ= g0t0…tmgm…g0 is an EFSM run produced by applying λ to the EFSM
in state g0.
 All the global states in rλ are concrete global states.
 Changes to the EFSM
 Performed at the transition level
 δ = < sign, tn >, sign { +, - } (addition, deletion)
 δ = < to, tn >
(replacement)
25
UNO
11/30/2012
Outline
 Preliminaries
 Fully-Observable Tests
 Tests with Non-Observable Regions
 Experiments
 Conclusion
26
UNO
11/30/2012
Transitions Matching a Test Description
 Transition matches a test if an input-output element of test
 is an instance of input-output message of the transition and
 satisfies the input condition of the transition.
 Transition 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a)
matches withdraw(160)/ack(150).
 Φ: sequence of sets of transitions point-wise matching the
sequence of input-output elements in a test description.
 Φ = [ {1} {2} {3, 7} {6} ] for test λ2
 : sequence of transitions selected point-wise from the sets.
  =[1236]
27
UNO
11/30/2012
(
Fully-Observable Tests
n-1
=
˅ ˄ Pos([t1,…, tk], g0)  Pre(tk+1)
ρϵΦ
k=0
  has a disjunct for each sequence of transitions ρ in Φ.
 kth conjunct states that execution path with k transitions can be
extended using k+1st transition.
 Test is fully-observable if its invariant  is a satisfiable formula
 A disjunct with value true denotes execution path for a satisfiable 
 At most one disjunct can evaluate to true since EFSMs are deterministic
28
UNO
11/30/2012
Identifying Fully-Observable Tests
λ2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150)
ρ1: conjunct 0: (a0 == 0) ˄ (100 > 0)
ρ1: conjunct 1: (a0 == 0) ˄ (100 > 0) ˄ (a1 == a0+100) ˄ (50 > 0)
ρ1: conjunct 2: (a0 == 0) ˄ (100 > 0) ˄ (a1 == a0+100) ˄ (50 > 0) ˄ (a2 == a1+50)
˄ (160 > 0) ˄ (160 <= a2)
ρ2: conjunct 2: (a0 == 0) ˄ (100 > 0) ˄ (a1 == a0+100) ˄ (50 > 0) ˄ (a2 == a1+50)
˄ (160 > 0) ˄ (160 > a2)
Initially a = 0; state is s0
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
1
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
29
UNO
Φ = [ {1} {2} {3, 7} {6} ]
3
2
6
7
11/30/2012
Outline
 Preliminaries
 Fully-Observable Tests
 Tests with Non-Observable Regions
 Preliminary Experiments
 Conclusion
30
UNO
11/30/2012
Tests with Non-Observable Regions
 Identify a non-observable region (tk, tk+1) for a conjunct that fails
to extend to transition tk+1
 EFSM paths from output state of tk to input state of tk+1.
 All intermediate transitions in all paths have no test inputs.
 Eliminating regions without loops
 Merge the intermediate transitions with tk to tmerge
 Patch conjunct C = Pos([t1,…, tmerge], g0)  Pre(tk+1) and
check if satisfiable.
 Fails if patched conjuncts for all paths in region unsatisfiable.
31
UNO
11/30/2012
Patching Failures – Example
λ3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50)
s0
1
1
ignore
2
3
5
?
7
32
UNO
s1
2
?
7
6
3
7
3
5
s2
4
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
11/30/2012
Regions with Terminating Loops
 Regions having loops cannot be eliminated by simple merging.
 Each loop iteration requires additional test input.
 Merge intermediate transitions in the path until the loop end-
points and attempt to construct a path by executing loop
transitions.
 Process terminates since all loops are terminating over
concrete global states.
33
UNO
11/30/2012
Patching Involving Loop – Example
λ4: open(100)/ack(100), deposit(50)/ack(150), withdraw(50)/ack(100), withdraw(60)/ack(40),
withdraw(60)/ack(50), close/ack(50)
s0
Φ = [ {1} {2} {3, 7} {3, 7} {3, 7} {6} ]
1
1
ignore
2
?
3
5
?
3
4
?
5
34
UNO
7
6
s1
2
3
7
3
5
s2
7
Patch fails
?
7
4
1. open(v), v > 0, s0  s1, {a = a + v}, ack(a),
2. deposit(v), v > 0, s1  s1, {a = a + v}, ack(a),
3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a),
4. a > 0  a < 50, s2  s2, {a = a + 1},
5. a >= 50, s2  s1, {},
6. close, true, s1  s0, {}, ack(a).
7. withdraw(v), v > 0  v > a, s1  s1, {}, ack(a).
11/30/2012
Outline
 Preliminaries
 Fully-Observable Tests
 Tests with Non-Observable Regions
 Experiments
 Conclusion
35
UNO
11/30/2012
Experiments
Regression Cost Model [Leung and White 91]

C1: Cost of running the full test suite

C2: Cost of running the selected tests

C3: Cost of analyzing the regression test selection
-
36
>
Atm: automatic teller machine; Thp: third party call; Bnk: back web services; Ven: a vending machine
(Cmp, Tcp, Cnf) : completion, two-phase commit, and conference protocol
Time Savings across Multiple Changes
37
UNO
11/30/2012
Test Selected across Multiple Changes
38
UNO
11/30/2012
SPG with Code-based approach
P1:
s0
E
t1: incr1(p)
/{x = p}
incr1 (x) {
P1
p1
while(++x <= 0)
{
}
return x;
s1
}
t3: jump()
[x+1>0]
/{x=x+1;
return(x)}
F
T
s1
s1
t2: while()
[x+1<=0]
/{x=x+1}
X
39
UNO
11/30/2012
SPG with Code-based approach
P1:
E
E
incr1 (x) {
P1
P1
p1
while(++x <= 0)
{
while(++x <= 0)
{}
}
return x;
p2
s1
F
T
F
F
P2
T
s1
T
s1
}
X
X
x = 0 : (E, P1) (P1, s1)
x = 0 : (E, P1) (P1, s1)
40
UNO
11/30/2012
SPG with Code-based approach
s0
P1:
s0
incr1 (x) {
p1
while(++x <= 0)
{
while(++x <= 0)
{}
}
return x;
p2
s1
}
t3: jump()
[x+1>0]
/{x=x+1;
return(x)}
t1: incr1(p)
/{x = p}
t3: jump()
[x+1>0]
/{x=x+1;
return(x)}
t1: incr1(p)
/{x = p}
s1
s1
t4: while(),
[x+1<=0]
/{x=x+1}
s2
t2: while()
[x+1<=0]
/{x=x+1}
t6: jump(),
[x+1>0]
/{x=x+1}
t5: while()
[x+1<=0]
/{x=x+1}
Incr1(0)/null, jump()/return() : t1,t3
Incr1(0)/null, jump()/return() : t1,t3
41
UNO
11/30/2012
SPG with Code-based approach
E
E
P3:
incr1 (x) {
P1
P1
T
p1
while(++x <= 0)
{
while(++x <= 0)
{}
return x;
}
return x;
p2
s2
s1
F
F
P2
T
F
T
s1
s2
s1
}
X
X
x = -1 : (E, P1) (P1, P1) (P1, s1) (s1, X)
x = -1 : (E, P1) (P1, P2) (P2, s2) (s2, X)
42
UNO
11/30/2012
SPG with Code-based approach
s0
s0
P3:
incr1 (x) {
p1
while(++x <= 0)
{
while(++x <= 0)
{}
return x;
}
return x;
p2
s2
s1
t3: jump()
[x+1>0]
/{x=x+1;
return(x)}
t1: incr1(p)
/{x = p}
s1
t3: jump()
[x+1>0]
/{x=x+1;
return(x)}
t1: incr1(p)
/{x = p}
s1
t4: while(),
[x+1<=0]
/{x=x+1}
s2
t2: while()
[x+1<=0]
/{x=x+1}
}
t6: jump(),
[x+1>0]
/{x=x+1;
return(x)}
t5: while()
[x+1<=0]
/{x=x+1}
incr1(-1)/null, while()/null, jump()/return(0) : t1, t2, t3
incr1(-1)/null, while()/null, jump()/return(0) : t1, t4, t6
43
UNO
11/30/2012
Conclusions
 Proposed an approach for regression test selection for high-
level EFSM test descriptions supporting common data types.
 Test descriptions are automatically analyzed to identify tests
that exercise a given change.
 Procedures to identify fully-observable tests, patching non-
observable regions, and reducing the test suites are described.
 Initial results of our experiments are promising.
 Comparison with Code-based approach.
44
UNO
11/30/2012
Pre- and Post-Images of Transitions
 Pre-image of t: a symbolic global state that triggers the
transition t.
 3. withdraw(v), v > 0  v <= a, s1  s2, {a = a – v}, ack(a)
 Pre(3) = < s1, ( v > 0 ^ v <= a0 ^ IQ0.hd == withdraw(v) )
 Post-image of t: a symbolic global state that is obtained after
executing t.
 Pos(3) = < s2, (v > 0 ^ v <=a0 ^ IQ1 == deq(IQ0) ^ IQ0.hd ==
withdraw(v) ^ OQ1 == <OQ0, ack(v)> ^ a1 == a0 – v ) >
 Executable path: a transition sequence [t1,…, tn] from a given
global state g if Pos([t1,…tn], g) is satisfiable.
n-1
Pos([t1,..,tn], g) = pred ˄ (
45
UNO
˄ Pos(t ) ˄ Pre(t
i
i=1
i+1))
˄ Pos(tn)
11/30/2012