Web Broadcasting Performance
Test
Mirjam Labrenz
Structure
1.
2.
3.
4.
Remember what the project is
Test on a local machine
Test on a local machine with external server
Test on external machines (Jenkins)
Remember what the project is
• A prototype that can:
– Broadcast platform
– Perform subscriptions (DIP, OPC)
– Uses:
•
•
•
•
•
Websocket
Atmosphere framework
Tomcat 7
Cache based on Hazelcast and Spring framework
Highcharts to show the statistics
Remember what the project is
First test – First problem
Client
Server
local
local
Dual Core
Dual Core
8 GB RAM
8 GB RAM
64-bit
64-bit
Wasync client
Tomcat 7 Server
• Problem:
– Max. 150 clients
• Solution
– Changing the NIO configuration of Tomcat 7
– Now max. 2700 clients
First Test with external server
Client
Server
local
extern
Dual Core
Hudson slave - Broadcast01
8 GB RAM
2 GB RAM
64-bit
Java 1.7
Wasync client
Tomcat 7 Server
NIO Connector
First Test - external server
Number of
messages
Number of
clients
Connection Ramp up Interval
(per 10 clients) in ms
Interval between
messages in ms
All messages
arrived
200
1
50
100

200
10
50
100

200
25
50
100

200
50
50
100

200
100
50
100

200
200
50
100

200
300
50
100

200
400
50
100

200
500
50
100

200
600
50
200

200
700
50
800

200
800
50
800

200
1000
50
1000

200
1500
50
2000

200
2000
50
3500

200
2500
50
3500

200
2720
50
3500

Second Test - external server
Client
Server
local
extern
Dual Core
Hudson slave - Broadcast01
8 GB RAM
2 GB RAM
64-bit
Java 1.7
Wasync client
Tomcat 7 Server
NIO Connector
Second Test - external server
Test combinations
Number of
messages
Number of
clients
Connection Ramp up Interval
(per 10 clients) in ms
Interval between
messages in ms
200
50
800
100,200,500,1000,2
000,5000
200
100
800
100,200,500,1000,2
000,5000
200
200
800
100,200,500,1000,2
000,5000
200
500
800
100,200,500,1000,2
000,5000
200
1000
800
100,200,500,1000,2
000,5000
200
2000
800
100,200,500,1000,2
000,5000
Second Test - external server
Average sending Time
Second Test - external server
Average sending Time
Second Test - external server
Average sending Time
Second Test - external server
Average sending Time
Second Test - external server
Average sending Time
Second Test - external server
Average sending Time - Boxplot
Second Test - external server
Variance
Second Test - external server
Variance
Second Test - external server
Variance
Second Test - external server
Variance
Second Test - external server
Variance - Boxplot
Second Test - external server
Standard deviation
Second Test - external server
Standard deviation
Second Test - external server
Standard deviation
Second Test - external server
Standard deviation
Second Test - external server
Standard deviation - Boxplot
Big test – external server
Number of
messages
Number of
clients
Connection Ramp up Interval
(per 10 clients) in ms
Interval between
messages in ms
All messages
arrived
100.000
5
50
1000

• Ready to move on
• Project on 3 external machines (Jenkins)
First test - Jenkins
Client 1
Client 2
Client 3
Server
Extern
Extern
extern
extern
cvl-ci-32-1
cvl-ci-32-1
cvw-ci-32-1
Hudson slave Broadcast01
GB RAM
GB RAM
GB RAM
2 GB RAM
32-bit
32-bit
32-bit
Java 1.7
Linux
Linux
Windows
Wasync client
Wasync client
Wasync client
Tomcat 7
Server
NIO Connector
First test - Jenkins
• Problems
– reached the max. number of threads / file handles
on the Linux machines
couldn’t create all the clients
no messages arrived
• Solution:
– increasing max. number of Threads
• Still not all the messages arrived
– decided to use only Windows machines at the
moment
Second Tests - Jenkins
Client 1
Client 2
Client 3
Server
extern
extern
extern
extern
cvw-bcast01
cvw-bcast02
cvw-ci-64-1
Hudson slave Broadcast01
2 GB RAM
2 GB RAM
2 GB RAM
2 GB RAM
64-bit
Java 1.7
Windows
Windows
Windows
Wasync client
Wasync client
Wasync client
Tomcat 7
Server
NIO Connector
Second Tests – Jenkins
Test combinations
Number of
clients
Number of
messages
Connection Ramp up Interval
(per 10 clients) in ms
Interval between
messages in ms
All clients created
50
10, 50, 100,
250, 500
200
1000
bcast01 
Bcast02 
ci-64-1  
100
10, 50, 100,
250, 500
200
1000
bcast01 
bcast02 
ci-64-1 
250
10, 50, 100,
250, 500
200
1000
bcast01 
bcast02 
ci-64-1 
500
10, 50, 100,
250, 500
200
1000
bcast01 
bcast02 
ci-64-1 
750
10, 50, 100,
250, 500
200
1000
bcast01 
bcast02 
ci-64-1 
1000
10, 50, 100,
250, 500
200
1000
bcast01 
bcast02 
ci-64-1 
Second Tests – Jenkins
Problems
• until 50 clients all messages arrived
• from 100 to 1000 clients more and more
messages lost
– for example
• 100 clients, 10 messages sent, 9 arrived
• 1000 clients, 10 messages sent, 1 arrived
Second Tests – Jenkins
Problems
Second Tests – Jenkins
Solution
• Problem was that the client hadn’t enough
time to get the messages
 Increasing the waiting time for the clients
Latest tests - Jenkins
Client 1
Client 2
Client 3
Server
extern
extern
extern
extern
cvw-bcast01
cvw-bcast02
cvw-bcast03
Hudson slave Broadcast01
2 GB RAM
2 GB RAM
8 GB RAM
2 GB RAM
64-bit
64-bit
64-bit
Java 1.7
Windows
Windows
Windows
Wasync client
Wasync client
Wasync client
Tomcat 7
Server
NIO Connector
Latest tests – Jenkins
Test combinations
Number of clients
Number of
messages
Connection Ramp up Interval (per
10 clients) in ms
Interval between
messages in ms
50
100
200
200,500,1000
200
100
200
200,500,1000
500
100
200
200,500,1000
Latest tests – Jenkins
Average sending Time
Latest tests – Jenkins
Average sending Time
Latest tests – Jenkins
Average sending Time
Latest tests – Jenkins
Average sending Time
Latest tests – Jenkins
Average sending Time
Latest tests – Jenkins
Average sending Time - Boxplot
Latest tests – Jenkins
Variance
Latest tests – Jenkins
Variance
Latest tests – Jenkins
Variance
Latest tests – Jenkins
Variance
Latest tests – Jenkins
Variance
Latest tests – Jenkins
Variance - Boxplot
Latest tests – Jenkins
Standard deviation
Latest tests – Jenkins
Standard deviation
Latest tests – Jenkins
Standard deviation
Latest tests – Jenkins
Standard deviation
Latest tests – Jenkins
Standard deviation
Latest tests – Jenkins
Standard deviation - Boxplot