Enabling continuous quality assurance and optimization
in future enterprise cloud service brokers
Verification & Testing: SAP HANA Case Study
Anthony J H Simons (University of Sheffield)
Wolfgang K R Schwach (SAP Karlsruhe)
Call: FP7-ICT-2011-8
Grant agreement no: 318392
www.broker-cloud.eu
Overview
•
•
•
•
•
Verification and Testing Toolset
Service Specification
Validation and Verification
Test Generation and Grounding
Test Execution on SAP HANA Cloud App
– V&T tools: Anthony Simons, Uni. Sheffield
– App & grounding: Wolfgang Schwach, SAP
31/07/2017
Anthony J H Simons, University of Sheffield
2
Verification & Testing Tools
31/07/2017
Anthony J H Simons, University of Sheffield
3
Specification
• Why?
– Gentle pressure towards common standards
– Provide basis for formal service certification
• What?
–
–
–
–
Describe service interfaces and behaviour
State-based model of service API
IOPE model of detailed behaviour
XML language  Java metamodel
31/07/2017
Anthony J H Simons, University of Sheffield
4
Holiday Booking Service
select/fail
back/ok
delete/ok
Delete
Days
save/fail
save/duplicate
save/overflow
View
Days
bookDays/ok
select/high
select/low
back/ok
save/ok
choose/one
choose/four
choose/seven
31/07/2017
Anthony J H Simons, University of Sheffield
Book
Days
choose/one
choose/four
choose/seven
5
XML Specification – FSM
31/07/2017
Anthony J H Simons, University of Sheffield
6
XML Specification – IOPE
31/07/2017
Anthony J H Simons, University of Sheffield
7
Validation & Verification
• Why?
– Designers write incomplete specifications
– Designers write incorrect specifications
• How?
– Validation tool explores state-transition diagram
• Reports unhandled events, missing transitions
– Verification tool simulates all input/memory space
• Reports blocking/nondeterministic operations
• Blocking: input exists which triggers no operation
• Nondeterminism: some input triggers many operations
31/07/2017
Anthony J H Simons, University of Sheffield
8
Verification Fault Analysis
off-by-one
indexing
error
31/07/2017
Anthony J H Simons, University of Sheffield
9
Test Generation
• Why?
– More thorough coverage than manual testing
– Assures full compliance to the specification
• How?
– Generation tool explores the whole specification
– Creates all interleaved sequences up to depth k
– Reduces test-suite size by optimising
• Removes infeasible sequences (blocked in the spec)
• Removes redundant sequences (equivalence-classes)
• Compresses executable suite (multi-objective tests)
31/07/2017
Anthony J H Simons, University of Sheffield
10
Generated Test Suite
multiobjective
test
31/07/2017
Anthony J H Simons, University of Sheffield
11
Test Grounding
• Why?
– High-level test suites are technology-neutral
– Platform-specific translations required
• How?
– Platform providers offer bespoke groundings
– Broker@Cloud framework provides hooks
• XML test suite parsed as a Java metamodel instance
• Visitor Design Pattern is used to generate concrete tests
• Bespoke grounding to e.g. Java, SOAP, REST, Selenium
31/07/2017
Anthony J H Simons, University of Sheffield
12
Example Java Grounding
auto-gen
JUnit test
assertions
31/07/2017
Anthony J H Simons, University of Sheffield
13
SAP HANA Case Study
• User may book up to 5 days vacation
– blocks of days can be booked, selected, deleted
– prevents overlapping blocks, more than 5 days
• App designed using the SAP UI5 toolkit
– visually pleasing point-and-click interface
– operations nominally match the earlier specification
– but a rich-client web app API is hard to test
• Deployed on the SAP HANA cloud platform
– temporarily available at URI: https://brokercloudd050960trial.dispatcher.hanatrial.ondemand.com/
31/07/2017
Anthony J H Simons, University of Sheffield
14
App Running on HANA
31/07/2017
Anthony J H Simons, University of Sheffield
15
Selenium Test Driver
• Selenium: a tool for web-testing
– normally used to capture-and-record a user's pointand-click gestures interactively
– can replay recorded test scripts, stored as HTML
tables, to run web app regression tests
• Leverage to test a rich-client cloud app
– given the Selenium DOM for the web app
– translate high-level test suite into Selenium test scripts
– use the Selenium engine to execute the test scripts
31/07/2017
Anthony J H Simons, University of Sheffield
16
Selenium Grounding Rules
31/07/2017
Anthony J H Simons, University of Sheffield
17
Executable Test Scripts
31/07/2017
Anthony J H Simons, University of Sheffield
18
Executing the Tests
31/07/2017
Anthony J H Simons, University of Sheffield
19
Subtle Faults Detected
• Functionality
– service failed to notify duplicate holiday booking
as a separate error case
– service failed to distinguish overflow and duplicate
holiday booking responses
• Security
– service executed operations in extra states, when
the operations should be ignored
– service revealed response information, when the
request should be ignored
31/07/2017
Anthony J H Simons, University of Sheffield
20
Beyond State-of-the-Art
• Extends leading SXM theory for MBT
– Adds reasoning about the full SXM model
– Adds validation and full verification capability
– Adds test optimisation capability
• Exceeds all industrial testing methods
– Full coverage of the specification achieved
– All positive and negative test cases created
– Identifies top 10% uniquely discriminating tests
31/07/2017
Anthony J H Simons, University of Sheffield
21
Questions
• Acknowledgement
FP7-ICT-2011-8 grant
agreement no: 318392
• Project website
www.broker-cloud.eu
31/07/2017
Anthony J H Simons, University of Sheffield
22
References
•
•
•
•
•
Broker@Cloud D20.1 State of the art and research baseline,
http:www.broker-cloud.eu, 2013.
Broker@Cloud D20.3 Requirements for continuous quality assurance
and optimisation in cloud brokerage, http:www.broker-cloud.eu, 2013.
AJH Simons and RE Lefticaru, Specification, verification and testing of
brokered software services in the cloud, IEEE ICST 2015, Graz
(submitted, under review)
AJH Simons and RE Lefticaru, Stream X-Machine based verification
and testing of cloud services, 2nd Int. Workshop on Cloud Service
Brokerage, ESOCC, Manchester, 2 Sept., CCIS, 2014.
M Kiran, A Friesen, AJH Simons and WKR Schwach, Model-based
testing in cloud brokerage scenarios, 1st Int. Workshop on Cloud
Service Brokerage, ICSOC, Berlin, 2 Dec., LNCS, 2013.
31/07/2017
Anthony J H Simons, University of Sheffield
23