Generating Structured Test Data with
Specific Properties using Metaheuristic and
Nested Monte-Carlo Search
Simon Poulding
Blekinge Institute of Technology
Robert Feldt
Chalmers University
Context
software testing: generating test data for software for which
inputs are highly structured
Problem
how to enable the test engineer to generate test data that
is both valid and has desirable properties?
Solution
(1) allow the test engineer to specify the construction of valid
test inputs using generators written in a general purpose
language
(2) apply metaheuristic and Monte-Carlo tree search algorithms
to optimise the generation process for the desirable
properties
GödelTest
+ Metaheuristic Search
+ Nested Monte-Carlo Search
Application: Robustness Testing
generator
@generator AZStrings begin
start() = String(mult(letter))
letter() = choose(Bool)? 'A' : 'B'
end
Julia implementation of GödelTest: https://github.com/simonpoulding/DataGenerators.jl
generator
@generator AZStrings begin
start() = String(mult(letter))
letter() = choose(Bool)? 'A' : 'B'
end
choice model
choice point
range: [0,∞)
distribution: geometric (p1)
choice point
range: [false,true]
distribution: Bernoulli (p2)
Julia implementation of GödelTest: https://github.com/simonpoulding/DataGenerators.jl
generator
@generator ABStrings begin
start() = String(mult(letter))
letter() = choose(Bool)? 'A' : 'B'
end
choice point
probability
range: [0,∞)
distribution: geometric (p1=0.5)
0
1
2
3
4
5
“AB”
“A”
“”
“BBA”
generator
@generator ABStrings begin
start() = String(mult(letter))
letter() = choose(Bool)? 'A' : 'B'
end
choice point
probability
range: [0,∞)
distribution: geometric (p1=0.1)
0
1
2
3
4
5
“BBABAB”
“BBAAABAA”
“ABBA”
“AABAAB”
(p1, p2) = (0.1, 0.2)
(p1, p2) = (0.5, 0.5)
“AAABAA”
“ABAAAAAA”
“AAA”
“BAAAAB”
“BAB”
“B”
“”
“ABBA”
GödelTest
+ Metaheuristic Search
+ Nested Monte-Carlo Search
Application: Robustness Testing
generator
GödelSequences
samples
[3,0,4,1,2]
choice model
“AAABAA”
“ABAAABAA”
“AAA”
“BAAAAB”
property metrics
parameters
(p1, p2) = (0.1, 0.2)
metaheuristic
search
height
size
target 1: size = 100 AND height = 36
target 2: size = 100 AND height = 6
Robert Feldt and Simon Poulding,
Finding Test Data with Specific Properties via Metaheuristic Search,
Proc. International Symposium on Software Reliability Engineering (ISSRE 2013)
80
Tree height
60
40
20
100
200
Tree size
Boltzmann Sampler
80
60
60
Tree height
Tree height
80
40
20
40
20
100
200
Tree size
100
Tree size
GödelTest using Differential Evolution
200
GödelTest
+ Metaheuristic Search
+ Nested Monte-Carlo Search
Application: Robustness Testing
current game state
simulation
child states
current game state
simulation
child states
current game state
simulation
child states
current game state
simulation
child states
current game state
simulation
child states
current game state
simulation
child states
0.7
current game state
child states
simulation
0.4
0.7
current game state
child states
simulation
0.4
0.7
0.9
[]
[0]
[1]
simulation
[0,0]
[1,0]
[0,0,3]
[2]
[2,1]
[2,1,2]
0.4
[0,0,3,0]
[2,1,2,1]
0.7
0.9
size
target: size = 100
Simon Poulding and Robert Feldt,
Generating structured test data with specific properties using nested Monte-Carlo search,
Proc. Genetic and Evolutionary Computation Conference (GECCO), 1279-1286, 2014
number of tree
consructions per target
structure
4000
3000
2000
1000
0
Boltzmann
NMCS 1
NMCS 2
algorithm
NMCS 4
time per target structure
(ms)
150
125
100
75
50
25
0
Boltzmann
NMCS 1
NMCS 2
algorithm
NMCS 4
median coordinate
1.00
0.75
0.50
0.25
0.00
Boltzmann NMCS 1 NMCS 2 NMCS 4 NMCS 8 NMCS 16
algorithm
height
size
target: size = 101 AND height = 8
height
8
height
8
0
0
1
101
size
‘random’ sampler
1
101
size
NMCS
GödelTest
+ Metaheuristic Search
+ Nested Monte-Carlo Search
Application: Robustness Testing
<math>
<mfrac>
<mrow>
<mn> 1 </mn>
<mo> + </mo>
<msqrt>
<mn> 5 </mn>
</msqrt>
</mrow>
<mn> 2 </mn>
</mfrac>
</math>
JEuclid
Simon Poulding and Robert Feldt
Generating Controllably Invalid and Atypical Inputs for Robustness Testing,
Proc. International Workshop on Testing Extra-Functional Properties (ITEQS 2017)
A2
modifying
choice model
A1
I2
I1
T
typical test set
mutating
choice model
normalised information
content in errors/warnings
0.015
0.01
0.005
0
T
A1
A2
T
I1
A1
I1
I2
A2
I2
Context
software testing: generating test data for software for which
inputs are highly structured
Problem
how to enable the test engineer to generate test data that
is both valid and has desirable properties?
Solution
(1) allow the test engineer to specify the construction of valid
test inputs using generators written in a general purpose
language
(2) apply metaheuristic and Monte-Carlo tree search algorithms
to optimise the generation process for the desirable
properties