Software Testing
Syntax-based Testing
Syntax Coverage
Four Structures for
Modeling Software
Graphs
Logic
Input Space
Syntax
Applied to
Applied
to
Source
Specs
Source
Specs
Design
Use cases
Applied
to
FSMs
DNF
Source
Models
Integ
Input
Applying Syntax-based Testing to Programs
• Syntax-based criteria originated with
programs and have been used most with
programs
• BNF criteria are most commonly used to
test compilers
• Mutation testing criteria are most
commonly used for unit testing and
integration testing of classes
Instantiating Grammar-Based Testing
Grammar-Based Testing
Program-based
5.2
Grammar
Integration
String
mutation
String
mutation
• Program mutation
• Valid strings
• Mutants are not tests
• Must kill mutants
• Compiler testing
• Valid and invalid strings
Model-Based
Input-Based
String
mutation
• FSMs
• Model checking
• Valid strings
• Traces are tests
• Test how classes interact
• Valid strings
• Mutants are not tests
• Must kill mutants
• Includes OO
Grammar
String
mutation
• Input validation
testing
• XML and others
• Invalid strings
• No ground strings
• Mutants are tests
• Input validation testing
• XML and others
• Valid strings
4
Program-based Grammars
• The original and most widely known
application of syntax-based testing is
to modify programs
• Operators modify a ground string
(program under test) to create mutant
programs
• Mutant programs must compile
correctly (valid strings)
5
Program-based Grammars
• Mutants are not tests, but used
to find tests
• Once mutants are defined, tests
must be found to cause mutants
to fail when executed
• This is called “killing mutants”
6
Mutation Testing
• Ground string: A string in the
grammar
–The term “ground” is used as a
reference to algebraic ground terms
• Mutation Operator : A rule that
specifies syntactic variations of strings
generated from a grammar
• Mutant : The result of one application
of a mutation operator
Mutant operators: Examples
Mutant operator
In P
In mutant
Variable replacement z=x*y+1;
x=x*y+1;
z=x*x+1;
Relational operator
replacement
if (x<y)
if(x>y)
if(x<=y)
Off-by-1
z=x*y+1;
z=x*(y+1)+1;
z=(x+1)*y+1;
Replacement by 0
z=x*y+1;
z=0*y+1;
z=0;
Arithmetic operator
replacement
z=x*y+1;
z=x*y-1;
z=x+y-1;
What is program mutation?
 P’ is known as a mutant of P.
 There might be a test t in T such that
P(t)≠P’(t). In this case we say that t
distinguishes P’ from P. Or, that t has
killed P’.
 There might be not be any test t in T
such that P(t)≠P’(t). In this case we say
that T is unable to distinguish P and P’.
Hence P’ is considered live in the test
process.
Killing Mutants
Given a mutant m  M for a ground string
program P and a test t, t is said to kill m if and
only if the output of t on P is different from the
output of t on m.
• Testers can keep adding tests until all mutants
have been killed
– Dead mutant : A test case has killed it
– Stillborn mutant : Syntactically illegal
– Trivial mutant : Almost every test can kill it
– Equivalent mutant : No test can kill it (equivalent to
original program)
Error detection using mutation
 Consider the following function
foo that is required to return the
sum of two integers x and y.
int foo(int x, y){
return (x-y);
}
Error detection using mutation
 Now suppose that foo has been tested using
a test set T that contains two tests:
T={ t1: <x=1, y=0>, t2: <x=-1, y=0>}
 First note that foo behaves perfectly fine on
each test in, i.e. foo returns the expected
value for each test case in T. Also, T is
adequate with respect to all control and
data flow based test adequacy criteria.
Error detection using mutation
Let us evaluate the adequacy of T using mutation.
Suppose that the following three mutants are
generated from foo.
M1:
int foo(int x, y){
return (x+y);
}
M2:
int foo(int x, y){
return (x-0);
}
M3:
int foo(int x, y){
return (0+y);
}
 Note that M1 is obtained by replacing
the - operator by a + operator, M2 by
replacing y by 0, and M3 by replacing
x by 0.
Error detection using mutation
Next we execute each mutant against tests in T
until the mutant is distinguished or we have
exhausted all tests. Here is what we get.
T={ t1: <x=1, y=0>, t2: <x=-1, y=0>}
Test (t) foo(t)
M1(t)
M2(t)
M3(t)
t1
t2
1
-1
1
-1
0
0
Live
Live
Killed
1
-1
Questions About Mutation
• Should more than one operator be applied
at the same time ?
–Should a mutated string contain one
mutated element or several?
–Almost certainly not – multiple
mutations can interfere with each other
–Extensive experience with programbased mutation indicates not
Mutation testing
Consider the following program P
1. main(argc,argv)
2. int argc, r, i;
3. char *argv[];
4. { r = 1;
5. for i = 2 to 3 do
6. if (atoi(argv[i]) > atoi(argv[r])) r = i;
7. printf(“Value of the rank is %d \n”, r);
8. exit(0); }
16
Mutation testing
• Test Case 1:
input: 1 2 3
output: Value of the rank is 3
• Test Case 2:
input: 1 2 1
output: Values of the rank is 2
• Test Case 3:
input: 3 1 2
output: Value of the rank is 1
17
Mutation testing
18
Mutant 1: Change line 5 to for i = 1 to 3 do
Mutant 2: Change line 6 to if (i > atoi(argv[r])) r = i;
Mutant 3: Change line 6 to if (atoi(argv[i]) >= atoi(argv[r]))
r = i;
Mutant 4: Change line 6 to if (atoi(argv[r]) > atoi(argv[r])) r
= i;
Execute modified programs against the test suite, you
will get the results:
Mutants 1 & 3: Programs will pass the test suite, i.e.,
mutants 1 & 3 are not killable
Mutant 2: Program will fail test cases 2
Mutant 1: Program will fail test case 1 and test cases 2
Mutation score is 50%, assuming mutants 1 & 3 nonequivalent
Logic Mutation Testing Process
1) Systematically create program variations
(mutants) each containing a single typical
logic fault
if (a && b)

if (a || b)
2) For each mutant, find an input that
distinguishes the original program from the
mutant program (killing)
- a && b and a || b evaluate to different truth
values
Program-based Grammars
Original Method
int Min (int A, int B)
{
int minVal;
minVal = A;
if (B < A)
{
minVal = B;
}
return (minVal);
} // end Min
With Embedded Mutants
int Min (int A, int B)
{
int minVal;
minVal = A;
∆ 1 minVal = B;
if (B < A)
∆ 2 if (B > A)
∆ 3 if (B < minVal)
{
minVal = B;
∆4
Bomb ();
∆5
minVal = A;
∆6
minVal = failOnZero (B);
}
return (minVal);
} // end Min
Replace one variable
with another
Changes operator
Immediate runtime
failure … if reached
Immediate runtime
failure if B==0 else
does nothing
Syntax-Based Coverage Criteria
1) Strongly Killing Mutants:
Given a mutant m  M for a program P and a
test t, t is said to strongly kill m if and only if the
output of t on P is different from the output of t
on m
2) Weakly Killing Mutants:
Given a mutant m  M that modifies a location
l in a program P, and a test t, t is said to
weakly kill m if and only if the state of the
execution of P on t is different from the state of
the execution of m immediately on t after l
Weak Mutation Example
• Mutant 1 in the Min( ) example is:
minVal = A;
∆ 1 minVal = B;
if (B < A)
minVal = B;
• The complete test specification to kill mutant 1:
• (A = 5, B = 3) will weakly kill mutant 1, but not
strongly
Equivalent Mutation Example
• Mutant 3 in the Min() example is equivalent:
minVal = A;
if (B < A)
∆ 3 if (B < minVal)
• The infection condition is “(B < A) != (B <
minVal)”
• However, the previous statement was “minVal =
A”
– Substituing, we get: “(B < A) != (B < A)”
• Thus no input can kill this mutant
Strong Versus Weak Mutation
(X = -6) will kill mutant
4 under weak mutation
1 boolean isEven (int X)
2 {
3
if (X < 0)
4
X = 0 - X;
∆4
X = 0;
5
if (float) (X/2) == ((float) X) / 2.0
6
return (true);
7
else
8
return (false);
9
}
Testing Programs with Mutation
Prog
Input test
method
Define
threshold
Create
mutants
Run
equivalence
detector
Automated
steps
no
Threshold
reached
?
Fix
P
no
P (T)
correct
?
Generate
test cases
Run T
on P
Run mutants:
• schema-based
• weak
• selective
Eliminate
ineffective
TCs
yes
26
Mutation Operators for Java
1. ABS –– Absolute Value Insertion:
Each arithmetic expression (and subexpression) is modified by the functions
abs(), negAbs(), and failOnZero().
2. AOR –– Arithmetic Operator Replacement:
Each occurrence of one of the arithmetic operators +,－,*,／, and % is
replaced by each of the other operators. In addition, each is replaced by the
special mutation operators leftOp, and rightOp.
3. ROR –– Relational Operator Replacement:
Each occurrence of one of the relational operators (<, ≤, >, ≥, =, ≠) is replaced
by each of the other operators and by falseOp and trueOp.
Mutation Operators for Java (2)
4. COR –– Conditional Operator Replacement:
Each occurrence of one of the logical operators (and - &&, or - || , and with no
conditional evaluation - &, or with no conditional evaluation - |, not equivalent
- ^) is replaced by each of the other operators; in addition, each is replaced by
falseOp, trueOp, leftOp, and rightOp.
5. SOR –– Shift Operator Replacement:
Each occurrence of one of the shift operators <<, >>, and >>> is replaced by
each of the other operators. In addition, each is replaced by the special
mutation operator leftOp.
6. LOR –– Logical Operator Replacement:
Each occurrence of one of the logical operators (bitwise and - &, bitwise or
- |, exclusive or - ^) is replaced by each of the other operators; in addition,
each is replaced by leftOp and rightOp.
Mutation Operators for Java (3)
7. ASR –– Assignment Operator Replacement:
Each occurrence of one of the assignment operators (+=, -=, *=, /=, %=, &=,
|=, ^=, <<=, >>=, >>>=) is replaced by each of the other operators.
8. UOI –– Unary Operator Insertion:
Each unary operator (arithmetic +, arithmetic -, conditional !, logical ~) is
inserted in front of each expression of the correct type.
9. UOD –– Unary Operator Deletion:
Each unary operator (arithmetic +, arithmetic -, conditional !, logical~) is
deleted.
Mutation Operators for Java (4)
10. SVR –– Scalar Variable Replacement:
Each variable reference is replaced by every other variable of the appropriate
type that is declared in the current scope.
Binary search example
int search ( int key, int [] elemArray){
int bottom = 0;
int top = elemArray.length - 1;
int mid;
int result = -1;
while ( bottom <= top ) {
mid = (top + bottom) / 2;
if (elemArray [mid] == key){
result = mid;
return result;
} // if part
else
{
if (elemArray [mid] < key)
bottom = mid + 1;
else
top = mid - 1;
}
} //while loop
return result;}