Automated Generation of Adaptive Test Plans for Self-­‐Adaptive Systems Erik Fredericks and Be'y H. C. Cheng May 19th, 2015 Motivation § 
Run-­‐9me tes9ng provides assurance for self-­‐adap9ve systems (SAS) § 
An SAS can experience uncertainty, possibly rendering test cases created at design 9me irrelevant Proteus ensures that test suites and test cases remain relevant throughout SAS execu9on § 
Background Approach Case Study Impact Related Work!
Agenda § 
Background § 
Proteus approach § 
Case study § 
§ 
Impact of run-­‐9me tes9ng § 
§ 
Discussion Discussion Related work Background Approach Case Study Impact Related Work!
Background § 
Remote data mirroring § 
Goal-­‐oriented requirements engineering § 
SoQware tes9ng Background Approach Case Study Impact Related Work!
Remote Data Mirroring Application § 
RDM [Veitch2003,Keeton2004] provides: § 
§ 
§ 
§ 
Data protec9on Prevents data loss and maintains availability Stores data in physically remote loca9ons Represented as an SAS Background Approach Case Study Impact Related Work!
Remote Data Mirroring Application Background Approach Case Study Impact Related Work!
Network Connections Background Approach Case Study Impact Related Work!
Dropped Message Background Approach Case Study Impact Related Work!
Disrupted Connection Background Approach Case Study Impact Related Work!
Reconfiguration Background Approach Case Study Impact Related Work!
Goal-­‐Oriented Requirements Engineering (A)
(B)
Achieve [Measure
(D)
Network Properties]
(J)
Achieve [Cost
Measured]
(K)
Achieve
[Activity
Measured]
(L)
Achieve
[LossRate
Measured]
Link
Sensor
Maintain
[DataAvailable]
Maintain [Operational
Costs ≤ Budget]
(E)
…
Achieve
(F)
[Network
Partitions == 0]
Achieve [Minimum
Num Links Active]
(N)
(M)
Achieve
Achieve
[Workload [Capacity
Measured] Measured]
(P)
Achieve
Achieve [Link
[Link
Deactivated]
Activated]
RDM
Sensor
Network
Actuator
(O)
Par0al KAOS [Dardenne1993,vanLamsweerde2009] goal model of RDM Background Approach Case Study Impact Related Work!
Goal-­‐Oriented Requirements Engineering (A)
Non-­‐invariant (B)
Achieve [Measure
(D)
Network Properties]
Requirement (J)
Achieve [Cost
Measured]
(K)
Achieve
[Activity
Measured]
(L)
Achieve
[LossRate
Measured]
Link
Sensor
Background Approach …
Invariant Maintain [Operational
Costs ≤ Budget]
(E)
Maintain
[DataAvailable]
Achieve
(F)
[Network
Partitions == 0]
Achieve [Minimum
Num Links Active]
(N)
(M)
Achieve
Achieve
[Workload [Capacity
Measured] Measured]
(P)
Achieve
Achieve [Link
[Link
Deactivated]
Activated]
RDM
Sensor
Network
Actuator
Agent Case Study (O)
Impact Related Work!
Utility Functions (A)
utilityGoal_B
operational_cost
budget
utilityGoal _ B = 1.0 − (B)
Achieve [Measure
(D)
Network Properties]
(J)
Achieve [Cost
Measured]
(K)
Achieve
[Activity
Measured]
(L)
Achieve
[LossRate
Measured]
Link
Sensor
Background Approach Maintain
[DataAvailable]
Maintain [Operational
Costs ≤ Budget]
(E)
…
Achieve
(F)
[Network
Partitions == 0]
Achieve [Minimum
Num Links Active]
(N)
(M)
Achieve
Achieve
[Workload [Capacity
Measured] Measured]
(P)
Achieve
Achieve [Link
[Link
Deactivated]
Activated]
RDM
Sensor
Network
Actuator
Case Study (O)
Impact Related Work!
Software Testing § 
Requirements-­‐based tes9ng § 
Validate that system under test is sa9sfying requirements [Myers2011,IEEE2010] § 
Regression tes9ng § 
Re-­‐validate system following major change to soQware [Myers2011,IEEE2010] Background Approach Case Study Impact Related Work!
Software Testing § 
Terminology § 
Test Specifica0on Test Suite 1 TC1 TC1..TC5 TC2 TC6 TC3 TC7 Test specifica9on TC4 TC8 §  Set of all possible test cases TC5 Test case §  Single test to assess all or a por9on of a requirement § 
derived for a soQware system § 
TC6 Test suite TC7 §  Subset of test cases from the TC8 test specifica9on §  Typically derived to be executed under a par9cular opera9ng context Background Approach Test Suite 2 TC1..TC5 TC6 TC8 TC9 TC10 TC10 … Case Study Impact Related Work!
Software Testing § 
Terminology § 
Test Specifica0on Test Scuite Test ase 1 TC1 §  TC1..TC5 Single test to assess all or a TC2 por9on TC6 of a requirement § 
TC7 Test specifica9on TC4 TC8 of all possible test cases §  Set derived for a soQware system Test Suite 2 § 
TC5 TC6 TC7 Test suite TC1..TC5 §  Subset TC6 of test cases from the TC8 test specifica9on TC8 §  Typically derived to be executed under a par9cular TC10 opera9ng context Background Invariant TC3 Approach Non-­‐invariant TC9 TC10 … Case Study Impact Related Work!
Proteus Approach § 
Requirements-­‐driven approach for managing run-­‐9me tes9ng § 
Defines an adap0ve test plan for each SAS configura9on § 
Each configura9on corresponds to a par9cular set of environmental condi9ons, or opera'ng context Performs a tes9ng cycle during each 9mestep of SAS execu9on § 
Background Approach Case Study Impact Related Work!
Adaptive Test Plans •  Network loss rate 1 •  ATP Data mirror capacity •  Sensor fuzz ATP 2 OC 1 OC 2 SAS Config. 1 SAS Config. 2 Background Approach Case Study ATP n … OC n SAS Config. n Impact Related Work!
Adaptive Test Plans Comprises all test suites for given opera9ng context ATP 1 ATP 2 OC 1 OC 2 SAS Config. 1 SAS Config. 2 Background Approach Case Study ATP n … OC n SAS Config. n Impact Related Work!
Adaptive Test Plans Automa9cally derived by Defined by test engineer Proteus ATP 1 TS
1.0 TS1.1 TS1.2 TS1.3 TS1.4 ATP 2 OC1 OC2 SAS Config. 1 SAS Config. 2 Background Approach Case Study ATP n … OCn SAS Config. n Impact Related Work!
Test Case Activation State § 
Test cases within test suite have an ac9va9on state: § 
§ 
§ 
§ 
ACTIVE: Executed when current test suite is performed INACTIVE: Not executed when current test suite is performed N/A:
Not executed, as it is not relevant to current opera9ng context Default test suite (TSk.0): § 
§ 
Relevant to opera9ng context: ACTIVE Irrelevant test cases labeled: N/A Background Approach Case Study Impact Related Work!
Testing Cycle § 
Tes9ng cycle at each step of SAS execu9on 1.  Execute default test suite 2.  Analyze test results 3.  Perform fine-­‐grained test case parameter adapta9on 4.  Perform coarse-­‐grained test suite adapta9on 5.  Determine if cycle is complete 1.  If complete: halt tes9ng 2.  If not complete: 1.  Execute intermediate test suite Background Approach Case Study Impact Related Work!
Test Case Fitness § 
Test case fitness (relevance) is defined as: relevanceTC = 1.0 −
valuemeasured − valueexpected
valueexpected
For example: High relevance to environment Low relevance to environment § 
§  Test case expected value = 0.50 §  Test case measured value = 0.45
§  Fitness = 0.90
Background Approach Case Study Test case expected value = 0.50 Test case measured value = 0.01 Fitness = 0.02 Impact Related Work!
Results Analysis § 
Each test case is correlated to at least one goal for valida9on § 
§ 
True posi0ve § 
§ 
§ 
Test case relevance = [0.0, Threshold) U9lity value(s) = 0.0 False posi0ve § 
§ 
§ 
Test case relevance = [Threshold, 1.0] U9lity value(s) > 0.0 True nega0ve § 
§ 
§ 
Test result validated against u9lity value Test case relevance = [Threshold, 1.0] U9lity value(s) = 0.0 False nega0ve § 
§ 
Test case relevance [0.0, Threshold) U9lity value(s) > 0.0 Background Approach Case Study No ac9on necessary Error detected in SAS , perform reconfigura9on Error detected in both SAS and test case, adapt both Error detected in test case, adapt test case Impact Related Work!
Fine-­‐grained Adaptation § 
Veritas [Fredericks2014.SEAMS] §  Adapts non-­‐invariant false posi9ve and false nega9ve test cases § 
Online evolu9onary algorithm § 
Searches for a be'er test case expected value §  Addresses system or environmental uncertainty for each opera9ng context Data mirror capacity TC1 Background 4.50 Approach TC1 Case Study 4.65 Impact Related Work!
Coarse-­‐grained Adaptation § 
Dynamically generate test suites based on test results Invariant Test Suite 1.0 Test Suite 1.1 True posi9ve TC1..TC5 TC6 TC1..TC5 TC7 True nega9ve TC7 TC8 False posi9ve TC8 TC9 Background TC9 False nega9ve Approach Case Study Impact Related Work!
End of Testing Cycle § 
Tes9ng cycle terminates when: § 
New SAS configura9on is invoked §  New tes9ng cycle ini9ated § 
§ 
All test cases result in true posi9ves If the cycle con9nues, then the dynamically-­‐generated test suite is executed instead of the default test suite Background Approach Case Study Impact Related Work!
Case Study § 
Simulated RDM network § 
§ 
§ 
§ 
[15, 30] data mirrors [100, 200] data messages 300 9mesteps Uncertainty at each 9mestep: § 
§ 
§ 
Unpredictable network link failures Randomly dropped or delayed messages Noise applied to data mirror sensors / network links Background Approach Case Study Impact Related Work!
Case Study § 
Test specifica9on: § 
36 test cases §  7 invariant §  29 non-­‐invariant
§ 
[precluded from adapta9on] [can be adapted] Compared Proteus adap9ve test plans to a manually-­‐derived Control test plan § 
Control test suite: §  All test cases from test specifica9on relevant to each opera9ng context Background Approach Case Study Impact Related Work!
Proteus and Veritas Executed false posi9ve test cases, i.e., Executed false nega9ve test cases, i.e., test case relevance = [Threshold, 1.0], test case relevance = [0.0, Threshold), u0lity value = 0.0 u0lity value > 0.0 Cumulative Number of False Negatives
Cumulative Number of False Positives
30
20
10
20
10
0
0
Control
Background Proteus
Approach Case Study Control
Proteus
Impact Related Work!
Executed irrelevant test cases, i.e., test case relevance = 0.0 Total number of executed test cases 4
10000
Cumulative Number of Executed Test Cases
Cumulative Number of Irrelevant Test Cases
Experimental Results 3
2
1
0
8000
6000
4000
2000
Control
Background Control
Proteus
Approach Case Study Impact Proteus
Related Work!
Case Study Discussion § 
Adap9ve tes9ng provided by Proteus framework supported by Veritas § 
Test suites and test cases remain relevant to changing environmental condi9ons § 
Reduces number of irrelevant test cases executed at run 9me Background Approach Case Study Impact Related Work!
Impact of Run-­‐Time Testing § 
Analyzed impact of our framework: § 
§ 
§ 
§ 
Execu9on 9me Memory overhead Requirements sa9sficement Number of reconfigura9ons Background Approach Case Study Impact Related Work!
Impact Study § 
Tested three states: § 
(S1): All run-­‐9me tes9ng ac9vi9es enabled §  Proteus+Veritas enabled § 
(S2): All run-­‐9me tes9ng ac9vi9es disabled §  Proteus+Veritas disabled § 
(S3): Run-­‐9me tes9ng framework removed §  Proteus+Veritas code / data structures removed from implementa9on Background Approach Case Study Impact Related Work!
Impact § 
Execu9on 9me § 
Measured total execu9on 9me of RDM simula9on Average execu9on 9me (seconds) Background Approach (S1) (S2) (S3) 23.030 13.901 13.785 Case Study Impact Related Work!
Impact § 
Execu9on 9me § 
Measured total execu9on 9me of RDM simula9on Average execu9on 9me (seconds) (S1) (S2) (S3) 23.030 13.901 13.785 Significant (p < 0.05) Background Approach Case Study Impact Related Work!
Impact § 
Execu9on 9me § 
Measured total execu9on 9me of RDM simula9on Average execu9on 9me (seconds) § 
(S1) (S2) (S3) 23.030 13.901 13.785 Memory footprint § 
Measured maximum memory usage of RDM simula9on Average memory usage (megabytes) Background Approach (S1) (S2) (S3) 65.234 65.332 65.020 Case Study Impact Related Work!
Impact § 
Execu9on 9me § 
Measured total execu9on 9me of RDM simula9on Average execu9on 9me (seconds) (S1) (S2) (S3) 23.030 13.901 13.785 Not significant (p > 0.05) § 
Memory footprint § 
Measured maximum memory usage of RDM simula9on Average memory usage (megabytes) Background Approach (S1) (S2) (S3) 65.234 65.332 65.020 Case Study Impact Related Work!
Impact § 
Requirements sa9sficement § 
Calculated average u9lity value over simula9on Average u9lity value Background Approach (S1) (S2) (S3) 0.7717 0.7656 0.7656 Case Study Impact Related Work!
Impact § 
Requirements sa9sficement § 
Calculated average u9lity value over simula9on Average u9lity value (S1) (S2) (S3) 0.7717 0.7656 0.7656 Not significant (p > 0.05) Background Approach Case Study Impact Related Work!
Impact § 
Requirements sa9sficement § 
Calculated average u9lity value over simula9on Average u9lity value § 
(S1) (S2) (S3) 0.7717 0.7656 0.7656 Number of reconfigura9ons § 
Averaged number of triggered RDM reconfigura9ons Average number of reconfigura9ons Background Approach (S1) (S2) (S3) 23.00 17.28 19.16 Case Study Impact Related Work!
Impact § 
Requirements sa9sficement § 
Calculated average u9lity value over simula9on Average u9lity value (S1) (S2) (S3) 0.7717 0.7656 0.7656 Not significant (p > 0.05) § 
Number of reconfigura9ons § 
Averaged number of triggered RDM reconfigura9ons Average number of reconfigura9ons Background Approach (S1) (S2) (S3) 23.00 17.28 19.16 Case Study Impact Related Work!
Discussion of Testing Impact § 
Run-­‐9me, adap9ve tes9ng only significantly impacts RDM in terms of execu9on 9me § 
§ 
Exploring paralleliza9on strategies to reduce 9me impact While not significant, a clear difference in mean u9lity values exist in requirements sa9sficement § 
Timing of measurement causes sampling 9mes to be slightly different Background Approach Case Study Impact Related Work!
Related Work § 
Search-­‐based so^ware tes0ng § 
§ 
§ 
§ 
Techniques such as evolu9onary computa9on, hill climbing, simulated annealing used for different tes9ng approaches in model tes9ng [Harman2009], regression tes9ng [Harman2012], and structural tes9ng [McMinn2011] EvoSuite [Fraser2011] and Nighthawk [Andrews2011] are evolu9onary frameworks for genera9ng test suites and instan9a9ng unit tests Veritas uses a run-­‐9me evolu9onary algorithm, whereas the other techniques focus on design 9me search Run-­‐0me tes0ng § 
§ 
§ 
Implemented using reinforcement learning [Veanes2006], recording & replaying [Tsai1990], and Markov modeling [Filieri2011] approaches Veritas combines evolu9onary search for test parameters with u9lity-­‐based valida9on Proteus maintains relevance of test suites as condi9ons change Background Approach Case Study Impact Related Work!
Related Work § 
Test suite genera0on § 
Requirements specifica9on used to generate formal grammars [Bauer1979] § 
§ 
Ar9ficial intelligence used to automa9cally generate test plans for graphical user interfaces [Memon2001] § 
§ 
Proteus generates new test suites based upon a pre-­‐defined default test suite and executes based on monitored condi9ons Proteus analyzes monitoring informa9on to select appropriate test suite Test case selec0on and priori0za0on § 
Select a representa9ve set of test cases and priori9ze their execu9on [Harman2009] § 
§ 
Proteus selects and executes tests at run 9me Tropos [Nguyen2008] uses agent-­‐based randomized tes9ng to validate mul9-­‐agent systems § 
Proteus generates test suites targeted towards specific DAS opera9ng contexts Background Approach Case Study Impact Related Work!
Acknowledgements § 
NSF BEACON Center (www.beacon-­‐center.org) § 
NSF grants CCF-­‐0820220, DBI-­‐0939454, CNS-­‐0854931, CNS-­‐1305358 § 
Ford Motor Company § 
General Motors Background Approach Case Study Impact Related Work!
Q&A Background Approach Case Study Impact Related Work!
References § 
[Veitch2003] M. Ji, A. Veitch, and J. Wilkes, “Seneca: Remote mirroring done write,” in USENIX 2003 Annual Technical Conference. Berkeley, CA, USA: USENIX Associa9on, June 2003, pp. 253–268. § 
[Keeton2004] K. Keeton, C. Santos, D. Beyer, J. Chase, and J. Wilkes, “Designing for disasters,” in Proc. of the 3rd USENIX Conference on File and Storage Technologies. Berkeley, CA, USA: USENIX Associa9on, 2004, pp. 59–62. § 
[Dardenne1993] A. Dardenne, A. Van Lamsweerde, and S. Fickas, “Goal-­‐directed re-­‐ quirements acquisi9on,” Science of computer programming, vol. 20, no. 1, pp. 3–50, 1993. § 
[vanLamsweerde2009] A. van Lamsweerde, Requirements Engineering: From System Goals to UML Models to SoLware Specifica'ons. Wiley, 2009. § 
[deGrandis2009] P. deGrandis and G. Vale'o, “Elicita9on and u9liza9on of applica9on-­‐ level u9lity func9ons,” in Proc. of the 6th Interna'onal Conference on Autonomic Compu'ng, ser. ICAC ’09. ACM, 2009, pp. 107–116. § 
[Ramirez2011] A. J. Ramirez and B. H. C. Cheng, “Automa9cally deriving u9lity func9ons for monitoring soQware requirements,” in Proc. of the 2011 Interna'onal Conference on Model Driven Engineering Languages and Systems Conference, Wellington, New Zealand, 2011, pp. 501–516. § 
[Walsh2004] W. E. Walsh, G. Tesauro, J. O. Kephart, and R. Das, “U9lity func9ons in autonomic systems,” in Proc. of the First IEEE Interna'onal Conference on Autonomic Compu'ng. IEEE Computer Society, 2004, pp. 70–77. § 
[Myers2011] G. J. Myers, C. Sandler, and T. Badge', The art of soLware tes'ng. John Wiley & Sons, 2011. [IEEE2010] IEEE, “Systems and soQware engineering – vocabulary,” ISO/IEC/IEEE 24765:2010(E), pp. 1–418, Dec 2010. § 
Background Approach Case Study Impact Related Work!
References § 
[Fredericks2014.SEAMS] E. M. Fredericks, B. DeVries, and B. H. C. Cheng, “Towards run-­‐ 9me adapta9on of test cases for self-­‐adap9ve systems in the face of uncertainty,” in Proceedings of the 9th Interna'onal Symposium on SoLware Engineering for Adap've and Self-­‐Managing Systems, ser. SEAMS ’14, 2014. § 
[Harman2009] M. Harman, S. A. Mansouri, and Y. Zhang, “Search based soQware engineering: A comprehensive analysis and review of trends techniques and applica9ons,” Department of Computer Science, King’s College London, Tech. Rep. TR-­‐09-­‐03, 2009. § 
[Fraser2011] Gordon Fraser and Andrea Arcuri. Evosuite: automa9c test suite genera9on for object-­‐oriented soQware. In Proc. of the 19th ACM SIGSOFT symposium and the 13th European conference on Founda9ons of soQware engineering, ES-­‐ EC/FSE ’11, pages 416–419, Szeged, Hungary, 2011. ACM. § 
[Andrews2011] James H. Andrews, Tim Menzies, and Felix C.H. Li. Gene9c algorithms for randomized unit tes9ng. IEEE Trans. on SoQware Engineering, 37(1):80–94, January 2011. § 
[Veanes2006] Margus Veanes, Pritam Roy, and Colin Campbell. Online tes9ng with rein-­‐ forcement learning. In Formal Approaches to SoQware Tes9ng and Run9me Verifica9on, pages 240–253. Springer, 2006. § 
[Tsai1990] J.J.-­‐P. Tsai, K.-­‐Y. Fang, Horng-­‐Yuan Chen, and Yao-­‐Dong Bi. A noninter-­‐ ference monitoring and replay mechanism for real-­‐9me soQware tes9ng and debugging. SoQware Engineering, IEEE Transac9ons on, 16(8):897–916, 1990. § 
[Filieri2011] Antonio Filieri, Carlo Ghezzi, and Giordano Tamburrelli. Run-­‐9me efficient probabilis9c model checking. In Proc. of the 33rd Interna9onal Conference on SoQware Engineering, pages 341–350, Waikiki, Honolulu, Hawaii, USA, 2011. ACM. Background Approach Case Study Impact Related Work!