Achievements, Open Problems and Challenges for
Search based Software Testing
Mark Harman!
Joint work with Yue Jia and Yuanyuan Zhang
University College London
There is a paper
accompany this keynote
Yuanyuan !
Zhang
Yue Jia!
Technical work!
and considerable help with slides
… and he’s here in Graz too
Madame Tussaud’s
Sherlock Holmes
Museum
20 mins walk
British
Museum
Marble Arch
Eros
National
Gallery
Westminster
Abbey
National
History
Museum
ICST’15
Covent Garden Market
Nelson’s Column
London Eye
St. Paul’s
RoyalCourts of Justice
Globe
Tate Modern Theatre
House of Parliament
Achievements, Open Problems and Challenges for SBST
Mark Harman
COWs
CREST Open Workshop
Roughly one per month
!
Discussion based
Recorded and archived
ICST’15
http://crest.cs.ucl.ac.uk/cow/
Achievements, Open Problems and Challenges for SBST
Mark Harman
COWs
CREST Open Workshop
Roughly one per month
!
Discussion based!
Recorded and archived
ICST’15
http://crest.cs.ucl.ac.uk/cow/
Achievements, Open Problems and Challenges for SBST
Mark Harman
COWs
CREST Open Workshop
Roughly one per month
!
Discussion based
Recorded and archived
ICST’15
http://crest.cs.ucl.ac.uk/cow/
Achievements, Open Problems and Challenges for SBST
Mark Harman
COWs
ICST’15
http://crest.cs.ucl.ac.uk/cow/
Achievements, Open Problems and Challenges for SBST
Mark Harman
COWs
#Total Registrations 1347
#Unique Attendees 623
#Unique Institutions 232
#Countries 42
#Talks 372
!
(Last updated on January 31, 2015)
!
ICST’15
http://crest.cs.ucl.ac.uk/cow/
Achievements, Open Problems and Challenges for SBST
Mark Harman
What is SBST
S
B
S
T
Search Based
Optimization
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Software
Testing
Mark Harman
What is SBST
In SBST we apply search techniques to search
large search spaces, guided by a fitness function
that captures natural counterparts as test
objectives.
Tabu Search Ant Colonies Particle Swarm Optimization
Genetic Algorithms
Hill Climbing
Genetic Programming
Simulated Annealing
Random
Greedy
LP
Estimation of Distribution Algorithms
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
What is SBSE
Search Based Software Engineering
In SBSE we apply search techniques to search
large search spaces, guided by a fitness function
that captures natural counterparts as test
objectives.
Tabu Search Ant Colonies Particle Swarm Optimization
Genetic Algorithms
Hill Climbing
Genetic Programming
Simulated Annealing
Random
Greedy
LP
Estimation of Distribution Algorithms
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
History of SBST
1842
2001
1879
“In almost every computation a great variety of
arrangements for the succession of the processes is
possible, and various considerations must influence
the selection amongst them for the purposes of a
Calculating Engine. One essential
essentialobject
objectisistoto
choose that
choose
thatarrangement
arrangementwhich
which
shall
shall
tend
tend
to reduce
to
to a minimum
reduce
to a minimum
the time necessary
the time necessary
for completing
for the
calculation.” the calculation.”
completing
Extract from ‘Note D’.
Checking a large routine
by Dr. A. Turing
In this shot paper, Turing
In this shot paper, Turing
suggested the use of
suggested the use of manually
manually constructed
constructed assertions and we
assertions and we can find
can find the origins of both
both
the origins of both software
software testing
software
testingand
andsoftware
testing and software
verification.
software
verification.
verification.
Sauder formulates the test
generation problem as one of
finding test inputs from a
search space, though the
search algorithm is random
search, making this likely to be
the first paper on Random
Random Test
Data Data
Test
Generation.
Generation.
The seminal PhD thesis
by James King
James King used automated
symbolic execution to capture
path conditions, solved using
linear programming
linear
programming
“We therefore considered various
alternatives that would not be
subject to this limitation. The most
promising of these alternatives
appears to be a conjugate gradient
algorithm (‘hill
(‘hill climbing’
climbing’program)
program)
that seeks to minimise a potential
function constructed from the
inequalities.”
At about the same time, Miller
and Spooner were also
experimenting with optimisationoptimisationbased approaches
approachesfor
for
generating test
generating
testdata
data(which
(whichthey
they
refer to as ‘test selection’ in the
sense that they ‘select’ from the
input space, which, in the more
recent literature we would refer to
as ‘test data generation’).
It appears that SBST research
lay dormant for at
approximately a decade until
the work of Korel, which
introduced a practical test
data generation approach, the
Alternating Variable
Alternating
VariableMethod
Method
(AVM), based on hill climbing.
Windows 3.1
first use
The first
useofofgenetic
genetic
algorithms for
algorithms
forsoftware
software
engineering problems is
usually attributed also to the
field of SBST, with the work
of Xanthakis et al., who
introduced a genetic
algorithm to develop whole
test suites.
The first suggestion of
search
search as
asaauniversal
universal
approach
approach to
toSoftware
Software
Engineering
Engineering
SB
SE
Analysis of Trends in SBST
The data is taken from the SBSE
Careful human-based update
100% precision and recall
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
700"
y"="0.0013x4"*"0.061x3"+"1.0008x2"*"5.8636x"+"10.443"
600"
500"
Polynomial yearly rise in
the number of papers
Search Based Software
Testing
400"
300"
200"
100"
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
2013"
2011"
2009"
2007"
2005"
2003"
2001"
1999"
1997"
1995"
1993"
1991"
1989"
1987"
1985"
1983"
1981"
1979"
1977"
0"
1975"
Accumulated*Number*of*SBST*Publica5ons*
800"
Mark Harman
2014#
2011#
2008#
2005#
2002#
1999#
1996#
The changing ratio
SBSE to SBST
1993#
1990#
1987#
1984#
1981#
1978#
1975#
0%#
10%#
20%#
30%#
SBST$
ICST’15
40%#
50%#
60%#
70%#
80%#
90%#
100%#
Other$SBSE$Publica2ons$
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST papers at ICST
Reformulating Branch Coverage as a Many-Objective Optimization Problem!
Annibale Panichella, Fitsum Meshesha Kifetew and Paolo Tonella!
!
Behind an Application Firewall, Are We Safe from SQL Injection Attacks?!
Dennis Appelt, Cu D. Nguyen, Lionel Briand!
!
Exploring Test Suite Diversification and Code Coverage in Multi-Objective Test Case Selection!
Debajyoti Mondal, Hadi Hemmati, and Stephane Durocher!
!
Guided Test Generation for Finding Worst-Case Stack Usage in Embedded Systems!
Tingting Yu and Myra B. Cohen!
!
Re-using Generators of Complex Test Data!
Simon Poulding and Robert Feldt!
!
U-Test: Evolving, Modelling and Testing Realistic Uncertain Behaviours of Cyber-Physical Systems!
Shaukat Ali and Tao Yue!
!
History-Based Test Case Prioritization for Black Box Testing using Ant Colony Optimization!
Tadahiro Noguchi, Hironori Washizaki, Yoshiaki Fukazawa, Atsutoshi Sato and Kenichiro Ota!
!
Combining Minimization and Generation for Combinatorial Testing!
Itai Segall, Rachel Tzoref-Brill and Aviad Zlotnick.
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Structural
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Structural
find tests to!
cover !
branches,!
statements &!
dataflow, etc.
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Integration
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Integration
find !
best component!
ordering
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Temporal
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Temporal
find worst case!
execution time
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
CIT
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
CIT
find 2-way, 3-way!
n-way!
interaction tests
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
SPLs
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Augment
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Augment
find new tests!
from old tests
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Regression
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Regression
find good!
subsets and !
orders of tests
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Functional
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Mutation
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
State!
based
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Model !
based
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Black box
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Failure !
Analysis
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Security
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Web/!
Services
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
Agents
ICST’15
History
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST’s Industrial Applications and Tools
Joachim Wegener and Oliver Bühler. GECCO 2004
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST’s Industrial Applications and Tools
Wasif Afzal, Richard Torkar, Robert Feldt and Greger Wikstrand. SSBSE 2010
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST’s Industrial Applications and Tools
Nikolai Tillmann, Jonathan de Halleux and Tao Xie. ASE 2014
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST Public Tools
AUSTIN applied to real-world embedded
automotive industry: Daimler, B&M
Systemtechnik. Recommended for testing C.
Kiran Lakhotia,Mark Harman,and Hamilton Gross. I&ST 2013
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST Public Tools
EvoSuite automatically generates test cases
for Java code. An excellent and high
recommended tool.
Gordon Fraser and Andrea Arcuri. ESEC/FSE 2011
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
More details in the keynote paper
in your ICST proceedings
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy
Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation
techniques (MoSBaT).
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy
Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation
techniques (MoSBaT).
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy
Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation
techniques (MoSBaT).
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy
Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation
techniques (MoSBaT).
Next
Annibale Panichella, Fitsum Meshesha Kifetew and Paolo Tonella!
Session: Reformulating Branch Coverage as a Many-Objective Optimization Problem
ICST’15
Analysis
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST for Non-Functional Properties
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
SBST for Non-Functional Properties
Number'of'SBST'Publica2on'for'Non4
Func2nal'Proper2es'
12"
10"
8"
6"
4"
2"
0"
1996$
1998$
2000$
2002$
2004$
2006$
2008$
2010$
2012$
Mark Harman,Wasif
Yue Jia
Afzal,
andRichard
Yuanyuan
Torkar,
Zhang.
andICST
Robert
2015.
Feldt.
Extends
I&ST 2009
Afzal et al 2009.
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
2014$
Scalability)
5)
6%)
Robustness)
10)
12%)
Safety)
4)
11%)
Flexibility))
3)
4%)
Energy)
consump&on)
1)
1%)
Efficiency)
8)
10%)
Execu&on)
&me)
15)
43%)
Usability)
7)
20%)
Security)
7)
20%)
Availability)
1)
1%)
QoS)
2)
6%)
Execu&on)&me)
21)
25%)
QoS)
4)
5%)
Safety)
8)
9%)
Usability)
10)
12%)
Security)
13)
15%)
The)Categories)of)NonLFunc&onal)Proper&es)from)1996)to)2014)
The)Categories)of)NonEFunc&onal)Proper&es)from)1996)to)2007)
Changing
Tiny amount
distribution
on SBET
of work
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
Search based Energy Test (SBET)
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
Search based Energy Test (SBET)
A smartphone could consume more energy
per year than a medium-sized refrigerator
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
Search based Energy Test (SBET)
A smartphone could consume more energy
per year than a medium-sized refrigerator
IT energy consumption rose 3% in 3 years
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
Search based Energy Test (SBET)
Measure energy consumption as a fitness function
Efficiency: will need to consider many different test cases
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
Search based Energy Test (SBET)
Efficiency: will need to consider many different test cases
Coarse Granularity: Energy consumed per run overall
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
Search based Energy Test (SBET)
Efficiency: will need to consider many different test cases
Coarse Granularity: Energy consumed per run overall
Hawthorne Effect: Instrumentation may affect energy consumed
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
Search based Energy Test (SBET)
Efficiency: will need to consider many different test cases
Coarse Granularity: Energy consumed per run overall
Hawthorne Effect: Instrumentation may affect energy consumed
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
39th COW - Measuring, Testing and Optimising Computational Energy Consumption
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
39th COW - Measuring, Testing and Optimising Computational Energy Consumption
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
39th COW - Measuring, Testing and Optimising Computational Energy Consumption
ICST’15
SBET
Achievements, Open Problems and Challenges for SBST
Mark Harman
Search Based Test Strategy Identification (SBTSI)
Move from finding specific inputs to finding strategies for finding inputs
General CIT Unknown CIT
Problems
Solution
A hyperheuristic SBSTI
for CIT
SBSTI
ICST’15 Achievements, Open Problems and Challenges for SBST
A co-evolutionary SBSTI
for Mutation testing
Mark Harman
A hyperheuristic SBSTI
for CIT
CIT Solutions
AETG
IPOG
GA
Simulated Annealing
Tabu
ICST’15
Hill Climbing
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A hyperheuristic SBSTI
for CIT
CIT Problem !
Characteristics
CIT Solutions
AETG
IPOG
GA
Simulated Annealing
Tabu
ICST’15
Specific structures
unconstrainted problems
constrainted problems
Hill Climbing
Weighted problems
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A hyperheuristic SBSTI
for CIT
CIT Problem !
Characteristics
CIT Solutions
AETG
IPOG
GA
Simulated Annealing
Tabu
ICST’15
Specific structures
unconstrainted problems
constrainted problems
Hill Climbing
Weighted problems
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A hyperheuristic SBSTI
for CIT
CIT Problem !
Characteristics
CIT Solutions
AETG
IPOG
GA
Simulated Annealing
Tabu
ICST’15
Specific structures
unconstrainted problems
constrainted problems
Hill Climbing
Weighted problems
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A hyperheuristic SBSTI
for CIT
CIT Problem !
Characteristics
CIT Solutions
AETG
IPOG
GA
Specific structures
unconstrainted
Unknown CITproblems
Problems
Simulated Annealing
Tabu
ICST’15
constrainted problems
Hill Climbing
Weighted problems
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A hyperheuristic SBSTI
for CIT
CIT Problem !
Characteristics
CIT Solutions
AETG
IPOG
GA
Specific structures
unconstrainted
Unknown CITproblems
Problems
Simulated Annealing
Tabu
ICST’15
constrainted problems
Hill Climbing
Weighted problems
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A hyperheuristic SBSTI
for CIT
CIT Problem !
Characteristics
CIT Solutions
AETG
IPOG
GA
General CIT
Solution
Simulated Annealing
Tabu
ICST’15
Specific structures
unconstrainted
problems
Unknown CIT
Problems
constrainted problems
Hill Climbing
Weighted problems
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
General CIT
Solution
Unknown CIT
Problems
Learning Combinatorial Interaction Test Generation Strategies using
Hyperheuristic Search. Yue Jia, Myra Cohen, Mark Harman and Justyna Petke.
ICSE 2015
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
General CIT
Solution
Unknown CIT
Problems
Learning Combinatorial Interaction Test Generation Strategies using
Hyperheuristic Search. Yue Jia, Myra Cohen, Mark Harman and Justyna Petke.
ICSE 2015
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
Predator
Prey
Testing
Bugs
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
Evolving
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
Evolving
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
ICST’15
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
A co-evolutionary approach for SBSTI
Test data
ICST’15
Higher order mutants
SBSTI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Multi-Objective Search Based Testing (MOSBAT)
increasingly prevalent
regression testing was early adopter
e.g. Yoo and Harman: ISSTA 2007
ICST’15
MOSBAT
Achievements, Open Problems and Challenges for SBST
Mark Harman
Multi-Objective Search Based Testing (MOSBAT)
ICST’15
MOSBAT
Achievements, Open Problems and Challenges for SBST
Mark Harman
Multi-Objective Search Based Testing (MOSBAT)
Coverage
Test data
Generation
test case generation
is still mostly single objective
ICST’15
MOSBAT
Achievements, Open Problems and Challenges for SBST
Mark Harman
Multi-Objective Search Based Testing (MOSBAT)
Coverage
Security
Usability
Test data
Generation
Execution
Time
ICST’15
MOSBAT
Achievements, Open Problems and Challenges for SBST
Energy
consumption
Mark Harman
Multi-Objective Search Based Testing (MOSBAT)
Multi-objective Understanding:
Debug Security policies
ICST’15
Usability
MOSBAT
Achievements, Open Problems and Challenges for SBST
Security
Mark Harman
SBST’s Challenges
We need to extend SBST to test non-functional properties. In particular, we need more work on Search Based Energy
Testing (SBET).
We need Search Based Test Strategy Identification (SBTSI).
We need more work on multi-objective test data generation
techniques (MoSBaT).
ICST’15
MOSBAT
Achievements, Open Problems and Challenges for SBST
Mark Harman
Let me ask you something …
ICST’15
MOSBAT
Achievements, Open Problems and Challenges for SBST
Mark Harman
Genetic Improvement:
searching for
improving modifications
guided by testing
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Genetic Improvement of Programs
Bowtie2
Sensitivity
Analysis
GP
Test
data
Non-functional
property Test
harness
Programs
Programs
Bowtie2
Programs
Improved
Fitness
70 times faster
30+ interventions
HC clean up: 7
slight semantic improvement
W. B. Langdon and M. Harman
Optimising Existing Software with Genetic Programming. TEC 2015
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Genetic Improvement of Programs
Sensitivity
Analysis
GP
Test
data
Programs
Fitness
Non-functional
property Test
harness
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Genetic Improvement of Programs
Cuda
Sensitivity
Analysis
GP
Test
data
Non-functional
property Test
harness
Programs
Programs
Cuda
Programs
Improved
Fitness
7 times faster
updated for new hardware
automated updating
W. B. Langdon and M. Harman
Genetically Improved CUDA C++ Software, EuroGP 2014
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Inter version transplantation
Sensitivity
Analysis
GP
Test
data
Programs
Fitness
Non-functional
property Test
harness
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Inter version transplantation
v1
MiniSat
v2
MiniSat
Sensitivity
Analysis
GP
Test
data
vn
MiniSat
Programs
Programs
MiniSat
Programs
Improved
Fitness
Non-functional
property Test
harness
Multi-doner transplant
Specialized for CIT
17% faster
Justyna Petke, Mark Harman, William B. Langdon and Westley Weimer
Using Genetic Improvement & Code Transplants to Specialise a C++ program
to a Problem Class (EuroGP’14)
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
!
ie
m
u
l
H da
O
e
C
m
C
r
E
e
G ilv
s
Mark Harman
Real world cross system transplantation
Sensitivity
Analysis
GP
Test
data
Programs
Fitness
Non-functional
property Test
harness
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Real world cross system transplantation
Doner
feature
Sensitivity
Analysis
Host
Host’
feature
GP
Test
data
Fitness
Non-functional
property Test
harness
Successfully
autotransplanted new
functionality and passed all
regression tests for 12 out of
15 real world systems
Earl T. Barr, Mark Harman, Yue Jia, Alexandru Marginean, and Justyna Petke Automated Software Transplantation (Tech Report). To appear.
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Memory speed trade offs
Sensitivity
Analysis
GP
Test
data
Programs
Fitness
Non-functional
property Test
harness
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Memory speed trade offs
System
malloc
System
Sensitivity
Analysis
optimised
malloc
GP
Test
data
Fitness
Non-functional
property Test
harness
Improve execution time by
12% or achieve a 21%
memory consumption
reduction
Fan Wu, Westley Weimer, Mark Harman, Yue Jia and Jens Krinke
Deep Parameter Optimisation
Conference on Genetic and Evolutionary Computation (GECCO'15), To appear
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Memory speed trade offs
System
malloc
System
Sensitivity
Analysis
optimised
malloc
GP
Test
data
Fitness
Non-functional
property Test
harness
Improve execution time by
12% or achieve a 21%
memory consumption
reduction
Fan Wu, Westley Weimer, Mark Harman, Yue Jia and Jens Krinke
Deep Parameter Optimisation
Conference on Genetic and Evolutionary Computation (GECCO'15), To appear
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Reducing energy consumption
Sensitivity
Analysis
GP
Test
data
Programs
Fitness
Non-functional
property Test
harness
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Reducing energy consumption
Improved MiniSat
MiniSat
CIT
MiniSat
Ensemble
CIT
Sensitivity
Analysis
GP
Test
data
MiniSat
AProVE
Improved MiniSat
Ensemble
Improved MiniSat
AProVE
Fitness
Non-functional
property Test
harness
Energy consumption can be
reduced by as much as 25%
Bobby R. Bruce Justyna Petke Mark Harman
Reducing Energy Consumption Using Genetic Improvement
Conference on Genetic and Evolutionary Computation (GECCO'15), To appear
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Grow and graft new functionality
?
Sensitivity
Analysis
GP
Test
data
Programs
Fitness
Non-functional
property Test
harness
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Grow and graft new functionality
Graft
Grow
Human!
Knowledge
GP
Test
data
Non-functional
property Test
harness
Feature
Sensitivity
Analysis
Host System
GP
Test
data
Fitness
Fitness
Non-functional
property Test
harness
Mark Harman, Yue Jia and Bill Langdon, Babel Pidgin: SBSE can grow and graft entirely new functionality into a real world system Symposium on Search-Based Software Engineering SSBSE 2014. (Challenge track)
ICST’15
Feature
GI
Achievements, Open Problems and Challenges for SBST
C
a
r
T
e
g
d
n
r
ll e wa
a
A
h
Mark Harman
k
c
!
Another keynote paper
at ASE 2012
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Pareto Front
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Pareto Front
each circle is a program found by a machine
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Pareto Front
different non
functional
properties have
different pareto
program fronts
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Failed Test Cases
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Why can’t functional properties
be optimisation objectives ?
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Optimisation
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Optimisation
2.5 times faster but
failed 1 test case?
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Optimisation
double the battery life
but failed 2 test cases?
ICST’15
GI
Achievements, Open Problems and Challenges for SBST
Mark Harman
Summary
Isn’t testing all about searching?!
!
Searching for test cases!
Searching for test application orders!
Searching for patches!
!
Searching for better programs guided by tests!
Genetic Improvement
ICST’15
Achievements, Open Problems and Challenges for SBST
Mark Harman
Summary
Isn’t testing all about searching?!
!
Searching for test cases!
Searching for test application orders!
Searching for patches!
!
Searching for better programs guided by tests!
Genetic Improvement
ICST’15
Achievements, Open Problems and Challenges for SBST
Mark Harman
Picture Copyrights
http://en.wikipedia.org/wiki/Thomas_Edison#/media/File:Edison_bulb.jpg
http://en.wikipedia.org/wiki/Colossus_computer#/media/File:Colossus.jpg
http://en.wikipedia.org/wiki/Hippie#/media/File:Woodstock-kids.jpg
http://beatles.wikia.com/wiki/The_Beatles_Wiki
ICST’15
Achievements, Open Problems and Challenges for SBST
Mark Harman