Distributed Load Testing of CMIS
Alfresco Benchmark Framework 2.0
Derek Hulley
Senior Developer
Michael Suzuki
Senior Developer
Contents
•
•
•
•
Essential Features
Framework Architecture
Building and Deploying a Test
Alfresco-specific Tests
• Creating Test Users
• CMIS Test
• Deployment
• Execution
• Results and Analysis
Key Features of the Framework
•
Load Tests are First Class projects
•
•
•
•
•
Central Control
•
•
•
Scale up the load drivers
Preset test defaults and override only as required
Scalability
•
•
•
Full Maven project lifecycle
Java
Automated testing
GitHub: derekhulley/alfresco-benchmark
NoSQL storage of results
Independent load drivers compete for work
Deep Analysis
•
•
Errors, messages and results are all retained
Every event execution is recorded and kept
Framework Architecture
Benchmark Server
Benchmark Driver
Benchmark Driver (xN)
Benchmark Driver (xN)
Tomcat 7
Extras
(Selenium)
Rest API
Services
Test
Tomcat 7
UI
Rest API
Services
MongoDB
Config Data
MongoDB
Test Data
Load Balancer
Servers / APIs
Servers / APIs
Servers / APIs
Writing a New Test
• Maven archetype
•
https://wiki.alfresco.com/wiki/Benchmark_Framework_2.0#Developing_Tests
01-Using-the-archetype.wmv
02-running-new-project-locally.wmv
• Import into Eclipse
• Build and run
• Deploy and Execute
• Then
• Phase out sample code
• Phase in real test code
CMIS Load Test: Infrastructure
BM Server Ireland
http://54.247.12.28:9080/alfrescobenchmark-server
BM Driver Ireland 1
54.74.38.161:9080
(Sign Up, CMIS, Sample)
Alfresco 5.0.a CE
Config
Data
Test
Data
BM Driver Sydney 1
54.206.178.67:9080
BM Driver Virginia 1
54.167.71.95:9080
(CMIS)
(CMIS)
Preparing for the Tests
• Deploy and run Signup Test
•
alfresco-benchmark-tests-ent-signup-2.0.war
05-Creating-users-in-Alfresco.wmv
• Permissions and test directories
•
06-Prepare-the-Alfresco-server.wmv
CMIS Load Test: Scenarios
•
•
GitHub: Search ‘benchmark-cmis’
• Pull requests for new scenarios welcome
• Repository-agnostic
Scenario 01
•
•
•
Scenario 02
•
•
•
•
•
•
Get test folder
Create folder
Upload file
Download file
Delete folder
Scenario 03
•
•
•
•
•
•
Find a random folder
List folder contents
Get test folder
Create folder
Upload file
Search in folder
Delete folder
Scenario weightings adjustable
CMIS Load Test: Sweet Spot
• Load driver “sweet spot”: Ireland
• IRELAND_03: 240 sessions at 0.25s intervals
•
07-Finding-the-sweet-spot.wmv
Alfresco
Driver
Data Comparison
CMIS Scenario
(Ireland-Ireland vs Virginia-Ireland)
CMIS.IRELAND_05
CMIS.VIRGINIA_03
xN
155
962
6.21
cmis.scenario.01.listFolderContents
81
643
7.94
cmis.scenario.02.createTestFolder
31
209
6.74
cmis.scenario.02.deleteTestFolder
130
279
2.15
cmis.scenario.02.downloadFile
26
391
15.04
cmis.scenario.02.retrieveTestFolder
23
345
15.00
159
772
4.86
cmis.scenario.03.createTestFolder
31
194
6.26
cmis.scenario.03.deleteTestFolder
53
230
4.34
cmis.scenario.03.retrieveTestFolder
24
367
15.29
cmis.scenario.03.searchInFolder
25
187
7.48
cmis.startSession
14
412
29.43
cmis.scenario.01.findFolder
cmis.scenario.02.uploadFile
• Raw results
•
•
CMIS-IRELAND_05.xlsx
CMIS-VIRGINIA_03.xlsx
• Compared