Syntax-based test coverage
(part 2)
- grammar-based testing Introduction
Basic grammar concepts
Testing criteria
Grammar-based mutation testing
!
!
!
!
Verificação e Validação de Software!
Departamento de Informática!
Faculdade de Ciências da Universidade de Lisboa!
!
Eduardo Marques, [email protected]
Grammar-based testing
Grammar-based testing
Inputs for the SUT are many times (or can be seen as) defined by a
grammar.
Testing the SUT with valid inputs: exercise productions of the grammar
according to some criterion.
Testing the SUT with invalid inputs: consider grammar-based mutations
are also considered to test the SUT with invalid inputs.
Examples
XML program data (in particular XML data with associated XML schemas)
Programs written in a programming language (to test compilers) !
Assorted data formats and instances of them (JSON, ASN.1, …)
…
2
Example grammar
simple grammar for a toy language of arithmetic expressions in
BNF notation
production!
rule
root symbol
expr
id
num
op
letter
digit
non-terminal!
symbols
3
::=
::=
::=
::=
::=
::=
id | num | expr op expr!
letter | letter id !
digit | digit num!
“+”|”-”|”*”|”/”!
“a”|...|“z”!
“0”|”1”|”2”|...|”9”
terminal !
symbols
6 non-terminal symbols
40 (4+26+10) terminal symbols
47 production rules (3+2+2+4+26+10)
Example grammar & derivations
expr
id
num
op
letter
digit
::=
::=
::=
::=
::=
::=
id | num | expr op expr!
letter | letter id !
digit | digit num!
“+”|”-”|”*”|”/”!
“a”|...|“z”!
“0”|”1”|”2”|...|”9”
syntax tree for
ab+12
expr
a
!
expr => id => letter => “a”!
expr
op
expr
43!
expr => num => digit num => “4” num !
“+”
id
num
=> “4” digit => “4” “3”!
ab+12 !
letter id
digit num
expr => expr op expr => expr “+” expr !
=> ... => “a””b” “+” “1” “2”
“1” digit
“a” letter
sample derivations
4
“b”
“2”
Coverage criteria
Terminal Symbol Coverage (TSC)
TR contains each terminal in the grammar.
One test case per terminal.
Toy language example: 40 test requirements
Production Coverage (PDC)
TR contains each production rule in the grammar.
One test case per production (hence PDC subsumes TSC).
Toy language example: 47 test requirements
Derivation Coverage (DC)
TR contains every possible derivation of the grammar.
One test case per derivation required.
Not practical - TR usually infinite as in our toy language.
When applicable, DC subsumes PDC.
5
Exercise 1
expr
id
num
op
letter
digit
::=
::=
::=
::=
::=
::=
id | num | expr op expr!
letter | letter id !
digit | digit num!
“+”|”-”|”*”|”/”!
“a”|...|“z”!
“0”|”1”|”2”|...|”9”
Suppose we wish to test a parser or interpreter for the example
toy language.
1) Define a test set (i.e., a set of grammar derivations) that
satisfies TSC but not PDC.
2) Complete the previous test set to satisfy PDC.
6
Exercise 1 (resolution)
expr
id
num
op
letter
digit
::=
::=
::=
::=
::=
::=
id | num | expr op expr!
letter | letter id !
digit | digit num!
“+”|”-”|”*”|”/”!
“a”|...|“z”!
“0”|”1”|”2”|...|”9”
1) We can consider the following tests for TSC:
a b ... z (26 tests)
0 1 ... 9 (10 tests)
a + b
a - b
a * b
a / b (4 tests)
PDC is not satisfied since no test case covers productions id ::= letter id
and num ::= digit num
2) To satisfy PDC we can for instance additionally consider:
ab !
7
12
Grammar-based mutation
Mutations over a grammar can be considered with the
purpose of generating invalid derivations.
Invalid derivations must be recognized as errors by the
software under test.
Example mutation operators:
Terminal and nonterminal deletion: remove a terminal or
nonterminal symbol from a production.
Terminal and Non-terminal duplication: duplicate a terminal or
nonterminal symbol in a production.
Terminal replacement: replace a terminal by another terminal.
Non-terminal replacement
8
Grammar mutations - example
expr
id
num
op
letter
digit
::=
::=
::=
::=
::=
::=
id | num | expr op expr!
letter | letter id !
digit | digit num!
“+”|”-”|”*”|”/”!
“a”|...|“z”!
“0”|”1”|”2”|...|”9”
expr
::= id | num | expr op expr |!
/* empty */ | op expr | expr expr | expr op!
id
::= letter | letter id !
num
::= digit | digit num!
mutant productions!
op
::= “+”|”-”|”*”|”/”!
resulting from nonterminal
letter ::= “a”|...|“z”!
deletion for expr productions!
digit ::= “0”|”1”|”2”|...|”9”
Example mutant derivations: 1) empty
9
2) + a
3)
a a
4)
12 +
Exercise 2: consider nonterminal duplication over expr .Describe the mutated
grammar and provide one mutant derivation per each mutated production rule.
Exercise 2 - resolution
expr
id
num
op
letter
digit
10
expr
::=
id
num
op
letter
digit
::=
::=
::=
::=
::=
::=
::=
::=
::=
::=
::=
id | num | expr op expr!
letter | letter id !
digit | digit num!
“+”|”-”|”*”|”/”!
“a”|...|“z”!
“0”|”1”|”2”|...|”9”
id | num | expr op expr |!
id id | num num | expr expr op expr |!
expr op op expr | expr op expr expr!
letter | letter id !
mutant productions!
digit | digit num!
resulting from nonterminal
“+”|”-”|”*”|”/”!
duplication
“a”|...|“z”!
“0”|”1”|”2”|...|”9”
Example derivations from mutated rules (not all of them are invalid inputs): !
1) a a (valid) 2) 12 13 (valid) 3)
a + 1 a + b 4) a + + 10 5)
a + 1 a!
The “id id” and “num num” rules always produce valid inputs. Some derivations
of “expr expr op expr” and “expr op expr expr” can be valid (e.g. a b +
c, a + b c). Finally, derivations of “expr op op expr” are always invalid.