Semi-Proving: an Integrated Method
Based on Global Symbolic Evaluation
and Metamorphic Testing
T.Y. Chen
T.H. Tse and Zhiquan Zhou
(speaker)
Swinburne University of
Technology, Australia
The University of
Hong Kong
1
Presentation Outline




Conventional Program Testing and Proving
Metamorphic Testing
Our method: Semi-Proving
Summary.
2
Presentation Outline




Conventional Program Testing and Proving
Metamorphic Testing
Our method: Semi-Proving
Summary.
3
Conventional Program
Testing and Proving
Given a bijective function f ;
A Program: F_Sort (a1, a2, ..., an), n  2
Output: (a1’, a2’, ..., an’), such that
1. (a1’, a2’, ..., an’) is a permutation of (a1, a2, ..., an)
2. f (a1’)  f (a2’)  ...  f (an’).
4
Conventional Program
Testing and Proving

Testing
1. Design test cases: e.g. (2, 6, 3) for n=3
2. Run: F_Sort (2, 6, 3) = (6, 3, 2)
3. Check: f (6) < f (3) < f (2) ?
5
Conventional Program
Testing and Proving

Proving correctness
1. F_Sort terminates for any valid input;
2. The output is correct.
6
Conventional Program
Testing and Proving

Proving properties
F_Sort (a1, a2, ..., an) = (a1’, a2’, ..., an’)
Permutation.
7
Metamorphic Testing

Metamorphic Testing

Employing relationships between different executions
Fact: different permutations will produce same output
F_Sort (a1, a2, a3)
=
F_Sort (a3, a1, a2)
“ Metamorphic Relation ” ·
8
Metamorphic Testing
Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}
Metamorphic Testing:
1. F_Sort (2, 6, 3) = (6, 3, 2)
||
No matter whether an oracle is
available or not;
Very useful when the oracle
cannot be found.
2. F_Sort (3, 2, 6) = (6, 3, 2) PASS
9
Metamorphic Testing
Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}
Metamorphic Testing:
1. F_Sort (2, 6, 3) = (6, 3, 2)
||
2. F_Sort (3, 2, 6) = (3, 6, 2) Failure.
10
Presentation Outline




Conventional Program Testing and Proving
Metamorphic Testing
Semi-Proving: Verifying Metamorphic
Relations
Summary.
11
Semi-Proving: Verifying
Metamorphic Relations

Objective:

If the program does not satisfy a metamorphic relation
on some inputs, locate these inputs;

Otherwise prove the satisfaction of the metamorphic
relation over all inputs.
12
Semi-Proving: Verifying
Metamorphic Relations

Why called “Semi”?

Proving necessary properties, which may not be
sufficient for program correctness

Characteristics of Semi-Proving

Multiple symbolic executions

Testing and proving.
13
Semi-Proving: Verifying
Metamorphic Relations
double GetMid (double x1, double x2, double x3)
{
double mid;
mid = x3;
if (x2 < x3)
if (x1 < x2)
mid = x2;
else {
if (x1 < x3)
mid = x1;
}
else
if (x1 > x2)
mid = x2;
else if (x1 > x3)
mid = x1;
return mid;
}
14
Semi-Proving: Verifying
Metamorphic Relations

Specification

“GetMid (X, Y, Z)” returns the median of (X, Y, Z)

E.g. GetMid (3, 4, 1): “3”.
15
Semi-Proving: Verifying
Metamorphic Relations

Verifying “GetMid” by Semi-Proving

Identify a Metamorphic Relation
GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) )
any numbers
any permutation
Purpose: to verify
16
Semi-Proving: Verifying
Metamorphic Relations

Basic concepts

Transposition
• simple permutation that exchanges two elements
(1, 2, 3)  (2, 1, 3) ......... 1
(1, 2, 3)  (1, 3, 2) ......... 2
17
Semi-Proving: Verifying
Metamorphic Relations

Basic concepts

Composition of Transpositions
A tuple (1, 2, 3)
A permutation (2, 3, 1)
2
1
(1, 2, 3) 
(2, 1, 3)  (2, 3, 1)
18
Semi-Proving: Verifying
Metamorphic Relations

Result from Group Theory

Any permutation of (X, Y, Z) can be achieved by
compositions of transpositions (X, Z, Y) and (Y, X, Z).
19
Semi-Proving: Verifying
Metamorphic Relations

Purpose


GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) )
Any permutation.
Only need to verify:
• GetMid (X, Y, Z) = GetMid (X, Z, Y)
• GetMid (X, Y, Z) = GetMid (Y, X, Z)
20
Semi-Proving: Verifying
Metamorphic Relations

Purpose

GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) )

Only need to verify:
• GetMid (X, Y, Z) = GetMid (X, Z, Y)
• GetMid (X, Y, Z) = GetMid (Y, X, Z)
21
Semi-Proving: Verifying
Metamorphic Relations

Global Symbolic Evaluation on GetMid (X, Y, Z)

Execute all the possible paths.
22
Semi-Proving: Verifying
Metamorphic Relations
double GetMid (double x1, double x2, double x3)
{
double mid;
mid = x3;
if (x2 < x3)
if (x1 < x2)
mid = x2;
else {
if (x1 < x3)
mid = x1;
}
else
if (x1 > x2)
mid = x2;
else if (x1 > x3)
mid = x1;
return mid;
}
23
Semi-Proving: Verifying
Metamorphic Relations
GetMid (X, Y, Z) =
X when C1 is true
Y when C2 is true
Z when C3 is true
C1: (Y  X < Z) OR (Z < X  Y)
Path Conditions
C2: (X < Y < Z)
OR (Z  Y < X)
C3: (Y < Z  X) OR (X  Z  Y)
24
Semi-Proving: Verifying
Metamorphic Relations
GetMid (X, Y, Z) =
?
X when C1 is true
Y when C2 is true
Z when C3 is true
GetMid (X, Z, Y)
25
Semi-Proving: Verifying
Metamorphic Relations
GetMid (X, Y, Z) =
?
GetMid (X, Z, Y) =
X when C1 is true
Y when C2 is true
Z when C3 is true
X when C4 is true
PASS
Z when C5 is true
Y when C6 is true
C4: (Z  X < Y) OR (Y < X  Z)
C5: (X < Z < Y) OR (Y  Z < X)
C6: (Z < Y  X) OR (X  Y 26
 Z)
Semi-Proving: Verifying
Metamorphic Relations
GetMid (X, Y, Z) =
?
GetMid (X, Z, Y) =
X when C1 is true
Y when C2 is true
Z when C3 is true
X when C4 is true
Z when C5 is true
PASS
Y when C6 is true
C4: (Z  X < Y) OR (Y < X  Z)
C1: (Y  X < Z) OR (Z < X  Y)
 Contradiction
&
C5: (X < Z < Y) OR (Y  Z < X)
C6: (Z < Y  X) OR (X  Y 27
 Z)
Semi-Proving: Verifying
Metamorphic Relations
GetMid (X, Y, Z) =
?
GetMid (X, Z, Y) =
X when C1 is true
Y when C2 is true
Z when C3 is true
X when C4 is true
Z when C5 is true
Y when C6 is true
C4: (Z  X < Y) OR (Y < X  Z)
X=Y<Z OR Z<Y=X
C1: (Y <= X < Z) OR (Z < X <= Y)
&
C5: (X < Z < Y) OR (Y  Z < X)
C6: (Z < Y  X) OR (X  Y 28
 Z)
Semi-Proving: Verifying
Metamorphic Relations
GetMid (X, Y, Z) =
?
GetMid (X, Z, Y) =
X when C1 is true
Y when C2 is true
Z when C3 is true
Yes. X=Y
X when C4 is true
Z when C5 is true
Y when C6 is true
PASS
C4: (Z  X < Y) OR (Y < X  Z)
X=Y<Z OR Z<Y=X
C1: (Y <= X < Z) OR (Z < X <= Y)
&
C5: (X < Z < Y) OR (Y  Z < X)
C6: (Z < Y  X) OR (X  Y 29
 Z)
Semi-Proving: Verifying
Metamorphic Relations
GetMid (X, Y, Z) =
verified
?
X when C1 is true
Y when C2 is true
Z when C3 is true
GetMid (X, Z, Y)
30
Semi-Proving: Verifying
Metamorphic Relations

Conclusion
GetMid (X, Y, Z) =
?
X when C1 is true
Y when C2 is true
Z when C3 is true
GetMid (X, Z, Y)
31
Semi-Proving: Verifying
Metamorphic Relations

Conclusion
GetMid (X, Y, Z) =
?
X when C1 is true
Y when C2 is true
Z when C3 is true
GetMid (X, Z, Y)
32
Semi-Proving: Verifying
Metamorphic Relations

Conclusion
GetMid (X, Y, Z) =
X when C1 is true
Y when C2 is true
Z when C3 is true
GetMid (X, Z, Y)
33
Semi-Proving: Verifying
Metamorphic Relations

Conclusion
GetMid (X, Y, Z) =
GetMid (X, Z, Y)
X when C1 is true
Y when C2 is true
Z when C3 is true
GetMid (Y, X, Z)
Composition of transpositions
GetMid (X, Y, Z) = GetMid ( Permute(X, Y, Z) )
Any
Any.
34
Semi-Proving:
Detecting Program Faults

Detecting Program Faults ·
35
double GetMid (double x1, double x2, double x3)
{
double mid;
mid = x3;
if (x2 < x3)
if (x1 < x2)
mid = x2;
else {
if (x1 < x3)
mid = x1;
}
else
if (x1 > x2)
mid = x2;
else if (x1 > x3)
mid = x1;
return mid;
}
36
Semi-Proving:
Detecting Program Faults
Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y)
||
X
when Y  X < Z
||
?
Y
when (Z < Y  X ) OR (Y  Z AND X  Z)
AND
37
Semi-Proving:
Detecting Program Faults
Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y)
||
X
when Y  X < Z
||
?
Y
when (Z < Y  X ) OR (Y  Z AND X  Z)
AND
(Y=X<Z) OR (Y<X<Z)
38
Semi-Proving:
Detecting Program Faults
Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y)
||
X
when Y  X < Z
failure
?

||
Y
when (Z < Y  X ) OR (Y  Z AND X  Z)
AND
(Y=X<Z)
Can identify
all the OR (Y<X<Z)
failure-causing
Failure-causing input
inputs.
39
Summary
A proving technique: all the paths
 A testing technique:




failure-causing inputs
selected path(s)
Characteristics



Metamorphic relations
Multiple symbolic executions
Employing global symbolic evaluation and
constraint solving.
40
Questions are welcome
41